Lazarus

Programming => LCL => Topic started by: penpen on March 14, 2019, 04:02:08 pm

Title: Tdbf is raising exception "out of memory"
Post by: penpen on March 14, 2019, 04:02:08 pm
Hi,
when I connect a .dbf database file and try to read a memo field that is empty I always get an outofmemory exception because it just keeps reading "forever".

This is the command I use:

MyString := dbf2.FieldByName('MemoField').AsString;

How can I check if the field is empty, without reading it ?
dbf2.FieldByName('MemoField').isNull doesnt work unfortunately.

Hope you can help

penpen
Title: Re: Tdbf is raising exception "out of memory"
Post by: Handoko on March 14, 2019, 06:43:09 pm
I cannot reproduce the issue. I've just made a test and S := MyDbf.FieldByName('MemoField').AsString; worked correctly even the field is empty. So I'm sure the bug you got was not on the line you showed us.

I've been using TDbf for years on a small program I wrote and use almost everyday, so far it's working good.

Please show us the whole source code so we can inspect it and tell you where you did it wrong.
Title: Re: Tdbf is raising exception "out of memory"
Post by: penpen on March 14, 2019, 08:56:35 pm
Thank you for your input.

I figured the problem is with the database. I cant share that with you, but it has some Memo Fields where dbf lib just wont stop reading till the "out of memory" exception is raised. It also does that when I call  bytes := MyDbf.FieldByName('MemoField').AsBytes

Some other empty memo fields in the same .dbf file however return NIL.

PS:
I can kind of workaround that using
try
 S := MyDbf.FieldByName('MemoField').AsString;
except
end;
But once these outofmemory memo fields come the program slows down dramatically because its reading many MB of nothing.

Is there a decent way I can debug this ?
Title: Re: Tdbf is raising exception "out of memory"
Post by: sstvmaster on March 14, 2019, 09:34:25 pm
Did you tried a Third party software to test your dbf?

You can check using this:

- https://sourceforge.net/projects/gtkdbfeditor/files/gtkdbfeditor/gtkdbfeditor-1.0.4/
- http://www.alexnolan.net/software/dbf.htm

Maybe this editor can read the memo field
Title: Re: Tdbf is raising exception "out of memory"
Post by: Cyrax on March 14, 2019, 09:52:02 pm
Thank you for your input.

I figured the problem is with the database. I cant share that with you, but it has some Memo Fields where dbf lib just wont stop reading till the "out of memory" exception is raised. It also does that when I call  bytes := MyDbf.FieldByName('MemoField').AsBytes

Some other empty memo fields in the same .dbf file however return NIL.

PS:
I can kind of workaround that using
try
 S := MyDbf.FieldByName('MemoField').AsString;
except
end;
But once these outofmemory memo fields come the program slows down dramatically because its reading many MB of nothing.

Is there a decent way I can debug this ?

You need to rebuild whole FPC suite with debug info options enabled : -gw2 -godwarfsets -godwarfmethodclassprefix -gl -O-
Title: Re: Tdbf is raising exception "out of memory"
Post by: penpen on March 14, 2019, 11:02:37 pm
Did you tried a Third party software to test your dbf?

You can check using this:

- https://sourceforge.net/projects/gtkdbfeditor/files/gtkdbfeditor/gtkdbfeditor-1.0.4/
- http://www.alexnolan.net/software/dbf.htm

Maybe this editor can read the memo field

Yes they dont crash at these fields. Except for one DBF editor that is written in lazarus, that one also crashes when reading the memos. Its this one: http://mydbfstudio.altervista.org/
Thats why I believe there is something wrong with the lib.

PS:
Actually gtkdbfeditor just shows numbers or nothing where there should be a memo.

And the other editor opens the memos fine and doesnt crash on the empty ones where lazarus crashes.

PPS:
I did select "count all" in dbfplus and got an outofmemory error as well.
http://prntscr.com/my0y90
Title: Re: Tdbf is raising exception "out of memory"
Post by: Handoko on March 15, 2019, 03:35:35 am
Have you tried to pack and rebuild the dbf using any available tools or this code below?
http://wiki.freepascal.org/Lazarus_Tdbf_Tutorial#Packing_and_rebuilding_the_tables

If the issue was caused by the damage of the dbf, packing it may solve the problem.

Or maybe you can create a new dbf with similar fields and write some code to import the data from the old dbf.
Title: Re: Tdbf is raising exception "out of memory"
Post by: penpen on March 15, 2019, 10:33:04 am
Have you tried to pack and rebuild the dbf using any available tools or this code below?
http://wiki.freepascal.org/Lazarus_Tdbf_Tutorial#Packing_and_rebuilding_the_tables

If the issue was caused by the damage of the dbf, packing it may solve the problem.

Or maybe you can create a new dbf with similar fields and write some code to import the data from the old dbf.


Just tried it. It also raises this exception "out of memory" on this line Dbf2.PackTable;
http://prntscr.com/my703p

The db is only 900kb.
Title: Re: Tdbf is raising exception "out of memory"
Post by: BrunoK on March 15, 2019, 02:08:52 pm
It is very likely that a blobfield (TMemoField is a TBlobField descendant) has been damaged in the data file during an update operation.
You ABSOLUTELY need to recreate a NEW clean table with the data that can be recovered.

You can "steal" dbf.pas TDbf.CopyFrom procedure and paste it to a father-son program where you will have to do a bit of editing.
Code: Pascal  [Select]
  1. equivalent to lines lines 1720-1733 of dbf.pas
  2.  
  3.         if not lSrcField.IsNull then
  4.         begin
  5.           if lSrcField.DataType = ftDateTime then
  6.           begin
  7.             if FCopyDateTimeAsString then
  8.             begin
  9.               lDestField.AsString := lSrcField.AsString;
  10.               if Assigned(FOnCopyDateTimeAsString) then
  11.                 FOnCopyDateTimeAsString(Self, lDestField, lSrcField)
  12.             end else
  13.               lDestField.AsDateTime := lSrcField.AsDateTime;
  14.           end
  15.           else begin
  16.             if lSrcField.DataType = ftMemo then begin // <-- begin of patch
  17.               try
  18.                lDestField.Assign(lSrcField);
  19.               except
  20.                // maybe log a message, a counter for dropped data or something
  21.                // identifying the damage record
  22.               end
  23.             end
  24.           else
  25.             lDestField.Assign(lSrcField);
  26.         end;                                                              // <-- end of patch
  27.  

If that works, Keep a backup of the old file and try using the regenerated file.
Title: Re: Tdbf is raising exception "out of memory"
Post by: penpen on March 17, 2019, 12:12:00 am
Hey. thanks for the help.

I personally dont think the records are damaged but its more of a bug in the lazarus dbf lib.
I think so since other programmes (unless written in lazarus' dbf lib) open the db just fine and read the memos with no problem.

I can "restore" it to fix it like you said and I probably will since its not very  good if my programme uses like 500mb of ram till the exception gets raised after 15 secs.
Title: Re: Tdbf is raising exception "out of memory"
Post by: wp on March 17, 2019, 12:42:29 am
Can you post the dbf file? You will have to zip it to squeeze it into the 250 KB upload limit and to bypass the filetype restriction. If it remains too big upload it to some cloud server and share the link. Or send me a PM with the link if you don't want to publish the file.