Recent

Author Topic: Database seeder  (Read 5111 times)

tudi_x

  • Hero Member
  • *****
  • Posts: 532
Database seeder
« on: March 17, 2018, 11:27:08 am »
hi,
would you know if some database seeder code / classes would exist for Pascal?
(https://en.wikipedia.org/wiki/Database_seeding)

thank you
Lazarus 2.0.2 64b on Debian LXDE 10

JD

  • Hero Member
  • *****
  • Posts: 1848
Re: Database seeder
« Reply #1 on: March 17, 2018, 02:03:01 pm »
DBeaver 5.0 https://dbeaver.jkiss.org/ can generate mock data for tables

JD
« Last Edit: March 17, 2018, 02:05:13 pm by JD »
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Database seeder
« Reply #2 on: March 17, 2018, 02:49:16 pm »
would you know if some database seeder code / classes would exist for Pascal?
Are you talking about mock data ???

The option in DBeaver is indeed nice but the data really doesn't mean anything. You could just generate a random string yourself.

I have a test-project that downloads real "mock" data from https://randomuser.me. It's not finished yet and the downloading and creating of the database could be moved to a separate class but it already works quite nice. (attached)

It is self-contained. It will download openssl (if needed) for Windows so if you're not on Windows you might need to strip that function-call. Furthermore it will download test-data (200 persons, with picture url) and read it in a new database SQLite3 (you do need to add the dll yourself at the moment).

JD

  • Hero Member
  • *****
  • Posts: 1848
Re: Database seeder
« Reply #3 on: March 17, 2018, 03:09:11 pm »
would you know if some database seeder code / classes would exist for Pascal?
Are you talking about mock data ???

The option in DBeaver is indeed nice but the data really doesn't mean anything. You could just generate a random string yourself.

I have a test-project that downloads real "mock" data from https://randomuser.me. It's not finished yet and the downloading and creating of the database could be moved to a separate class but it already works quite nice. (attached)

It is self-contained. It will download openssl (if needed) for Windows so if you're not on Windows you might need to strip that function-call. Furthermore it will download test-data (200 persons, with picture url) and read it in a new database SQLite3 (you do need to add the dll yourself at the moment).

Thanks for the link and your example project rvk. I did not know about that one & furthermore the data format is quite flexible.  :D

JD
« Last Edit: March 17, 2018, 03:17:26 pm by JD »
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

tudi_x

  • Hero Member
  • *****
  • Posts: 532
Re: Database seeder
« Reply #4 on: March 17, 2018, 03:26:27 pm »
thank you guys!

i will adopt the SSL download approach for future.
basically based on a data definition in first part of JSON, a thread would create table if it does not exist.
second part of JSON could be the data to seed.
same data, same table would be consistent for DevOps.
Lazarus 2.0.2 64b on Debian LXDE 10

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Database seeder
« Reply #5 on: March 17, 2018, 03:52:35 pm »
thank you guys!

i will adopt the SSL download approach for future.
basically based on a data definition in first part of JSON, a thread would create table if it does not exist.
second part of JSON could be the data to seed.
same data, same table would be consistent for DevOps.
that is called a DDL file, In short you write a text file that has all the commands to create the tables and insert their data why do you need a json file ? If you do need a json have checked the database import package that comes with lazarus for a control that can import json data?
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

tudi_x

  • Hero Member
  • *****
  • Posts: 532
Re: Database seeder
« Reply #6 on: March 17, 2018, 06:16:43 pm »
i need a JSON in order to be able to send the commands on a REST Endpoint to the app that resides in a container.

i searched for the database import with no luck.
could you please point me in the right direction or a wiki entry?
thank you
Lazarus 2.0.2 64b on Debian LXDE 10

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Database seeder
« Reply #7 on: March 17, 2018, 06:39:38 pm »
i need a JSON in order to be able to send the commands on a REST Endpoint to the app that resides in a container.
so send a json with the sql command as a value to a property/field. I hypothesize that it will be  around 20% smaller.
i searched for the database import with no luck.
could you please point me in the right direction or a wiki entry?
thank you
seems that my memory failed me I was referring to http://wiki.freepascal.org/lazdbexport there is a link at the bottom https://bitbucket.org/reiniero/smalltools/src which will give you a starting point.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

tudi_x

  • Hero Member
  • *****
  • Posts: 532
Re: Database seeder
« Reply #8 on: March 17, 2018, 06:55:30 pm »
do not want to send SQL creation commands as the structure might be coming from apps or users that are not aware of SQL.
also the user would need to know what type of RDBMS is deployed which is too much sometimes (the app holding the REST would do the discovery solely).

thank you for the link.

« Last Edit: March 18, 2018, 10:32:05 am by tudi_x »
Lazarus 2.0.2 64b on Debian LXDE 10

JD

  • Hero Member
  • *****
  • Posts: 1848
Re: Database seeder
« Reply #9 on: March 18, 2018, 09:03:57 am »
do not want to tie to SQL as the structure might be coming from apps or users that are not aware of SQL.
also the user would need to know what type of RDBMS is deployed which is too much sometimes (the app holding the REST would do the discovery solely).

thank you for the link.

I send JSON data to databases (Firebird and PostgreSQL) behind a remote REST server that I built using Lazarus. The REST server uses SQLdb to convert the data into SQL parameters for predefined INSERT, UPDATE, DELETE statements. I then call SQLDb's ExecSQL to post the data into the databases.

That is the beauty of having an application server. It does the work and the user/client app does not need to know anything about how the data is processed by the server.

If you don't like that method, you can send JSON data directly to a stored procedure in the database. The stored procedure will parse the JSON data and update the database accordingly.

Another option, if you are using PostgreSQL or MariaDB, is to use their JSON functionalities to parse the sent data and update the database.

Cheers,

JD
« Last Edit: March 18, 2018, 09:06:03 am by JD »
Windows - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe),
Linux Mint - Lazarus 2.1/FPC 3.2 (built using fpcupdeluxe)

mORMot; Zeos 8; SQLite, PostgreSQL & MariaDB; VirtualTreeView

 

TinyPortal © 2005-2018