Recent

Author Topic: Visual Planit JSON datastore issues  (Read 4727 times)

JD

  • Hero Member
  • *****
  • Posts: 1848
Visual Planit JSON datastore issues
« on: September 29, 2018, 10:00:52 pm »
Hi there everyone,

Don't know if anybody can help me with this or maybe it is only the maintainer of the Visual Planit package that can fix it.

I've been using Visual Planit and its Buf datastore in my project for a couple of years. I now want to replace it with the package's JSON datastore. I just updated to Visual Planit 1.12.

I noticed that the JSON datastore reads its data from a file already saved onto the disk. I however am interacting with a remote REST server. Nothing is retained on the client-side in my project.  The server sends me a JSON string and then I am compelled to save the data to disk before loading it into the datastore. 

My question is: is it possible to load the data into the JSON datastore without first saving it onto the disk?

My workflow also  requires that I also update the agenda and then send it back to the REST server so that the remote database can be updated.

In addition, the JSON datastore mangles French special characters after the data is loaded into it (see screenshot).

How do I fix it so that the JSON datastore respects UTF8 text?

Thanks a lot,

JD
« Last Edit: September 30, 2018, 11:36:52 am by JD »
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

wp

  • Hero Member
  • *****
  • Posts: 11855
Re: Visual Planit JSON datastore issues
« Reply #1 on: September 30, 2018, 01:04:39 am »
Please try the current TVPlanit trunk. I don't have experience with a REST server, but maybe the new version has what you want:
  • The TVpJSONDatastore has a new propert JSONStoreType: when it is jstFile then data are read from/written to a file, as before. But when it is jstString then data are read from/written to a string exposed in property JSONString.
  • Before connecting the datastore read the JSON string from the server and assign it to this property; then connect the datastore (make sure to have AutoConnect = false).
  • When the datastore disconnects read the corresponding value again and send it to the server.
As for the UTF8 issue, I must say that it works for me. However, I now made the options of the JSONObject accessible to the DataStore: Set the properties UseUTF8, Comment, Strict, IgnoreTrailingComma accordingly before connecting the datastore. Maybe you have to set UseUTF8 to false - it was forced to true in the previous version. Alternatively you can keep it at its default and convert the JSON string to UTF8 before you pass it to the datastore.

JD

  • Hero Member
  • *****
  • Posts: 1848
Re: Visual Planit JSON datastore issues
« Reply #2 on: September 30, 2018, 01:58:17 pm »
Thanks a million wp. It worked perfectly. All the manual mapping I used to do to map database fields to the Buf datastore fields is now redundant as the REST server's JSON string can now be loaded directly into the JSON datastore.  :D :D :D

I haven't tested it thoroughly though and I would like to know if I can track delta changes in the JSON data with the JSON  datastore. Put differently, I want to update the remote database with ONLY the most recent changes made to the JSON data by the clients.

If not, I would have to

a) batch repost ALL the events data to the remote database; changed or not
b) develop a mechanism for identifying only changed data and post only that.

Thanks a lot,

JD
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

JD

  • Hero Member
  • *****
  • Posts: 1848
Re: Visual Planit JSON datastore issues
« Reply #3 on: October 01, 2018, 09:05:51 am »
Bump.  ;D
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

wp

  • Hero Member
  • *****
  • Posts: 11855
Re: Visual Planit JSON datastore issues
« Reply #4 on: October 01, 2018, 05:22:16 pm »
As far as I understand (I do maintain this library but I'm not its author...), there are two ways how the datastores work:
  • a flat-file mode which is used for the ini, xml, json stores - they store the events, contacts etc. in a local file, load the file into internal lists, change or delete some records in these lists and finally write everything back to file.
  • a database mode which is used by the bufdataset, sqlite etc stores - they load all records of a specified time frame (DayBuffer) into the internal lists, add or modify the records there and with every modification a flag "Changed" is set. Correspondingly records are marked for deletion. When the data are posted only the records with the "Changed" flag are written which is reset afterwards. Correspondignly those records marked for deletion are deleted from the dataset.
The second mode looks like what you need, but not exactly because after posting (which occurs when the data input dialogs close) the "Changed" and "to-be-deleted" flags are lost - the JSON string would still contain all records (except for the deleted ones).

I probably have to keep all records in memory and write the JSON string only when the datastore disconnects, as it happens now,  but include only the changed records. But what to do with those marked for deletion? How do you interact with the REST server if you want to delete records?

JD

  • Hero Member
  • *****
  • Posts: 1848
Re: Visual Planit JSON datastore issues
« Reply #5 on: October 02, 2018, 12:22:48 pm »
@ wp

Thanks a lot for your suggestions. I realize that I am looking for an ideal between the convenience of using pure JSON (VpJSONDatastore) for retrieval of data and the dataset like qualities of VpBufDSDatastore for the updating of data.

I am currently taking a look at the mORMot REST Server based example for Visual Planit (VpMORMotDs) created by DonAlfredo. That looks closer to what I want to achieve. I'll try and see if I can adapt it for my needs. I'll be sure to keep you posted.

Cheers,

JD
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

 

TinyPortal © 2005-2018