[SOLVED ]Simple Bynary files handling example

[torbente]

Hi everyone. Im trying to create a simple example for managing binary files (first time i work with binary files)
The objetive is to store a dinamyc array of records and a long string in a disk file, and of course, be able to read it again.
I used this post as starting point:
http://forum.lazarus-ide.org/index.php/topic,24597.msg148222.html#msg148222

I can not go beyond this point. I assume i need include headers or write/read every record separately, but i can not find the way.

Code: Pascal  [Select][+][-]

1. unit Unit1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. interface
6.  
7. uses
8.   Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs, Grids,
9.   StdCtrls;
10.  
11. type
12.  
13.   { TForm1 }
14.  
15.   PeopleData = Packed Record
16.     Name : String[255];
17.     LastName : String[255];
18.     Age: String[3];
19.     end;
20.  
21.   FileData = Packed Record
22.     People : Array of PeopleData;
23.     LongString : String;
24.     end;
25.  
26.   TForm1 = class(TForm)
27.     Button1: TButton;
28.     Memo1: TMemo;
29.     SGrid1: TStringGrid;
30.     procedure Button1Click(Sender: TObject);
31.     procedure FormCreate(Sender: TObject);
32.     procedure generateDefValues();
33.     Procedure loadfromfile();
34.   private
35.  
36.   public
37.  
38.   end;
39.  
40. Const
41.   ArchData = 'example.dat';
42.  
43. var
44.   Form1: TForm1;
45.   LongString : String = '';
46.   ArrPeople : Array of PeopleData;
47.   CustomFile : FileData;
48.  
49. implementation
50.  
51. {$R *.lfm}
52.  
53. { TForm1 }
54.  
55. procedure TForm1.FormCreate(Sender: TObject);
56. begin
57. Randomize;
58. if not fileexists(ArchData) then generateDefValues()
59. else loadfromfile();
60. end;
61.  
62. procedure TForm1.Button1Click(Sender: TObject); // save to file
63. var
64.   MemStr: TMemoryStream;
65. begin
66. MemStr := TMemoryStream.Create;
67.   try
68.   MemStr.Write(CustomFile, SizeOf(CustomFile));
69.   MemStr.SaveToFile(ArchData);
70.   finally
71.   MemStr.Free;
72.   end;
73. end;
74.  
75. Procedure TForm1.loadfromfile();
76. var
77.   MemStr: TMemoryStream;
78.   FileStr: TFileStream;
79. Begin
80. setlength(Customfile.People,0);
81. Customfile.LongString:='';
82. MemStr := TMemoryStream.Create;
83.    try
84.    FileStr := TFileStream.Create(ArchData, fmOpenRead);
85.    MemStr.CopyFrom(FileStr,Sizeof(FileStr));
86.    FileStr.Free;
87.    MemStr.Read(CustomFile, SizeOf(CustomFile));
88.    finally
89.    MemStr.Free;
90.    end;
91. memo1.lines.add(LongString);
92. // fill Sgrid with loaded Arrpeople
93. end;
94.  
95. Procedure TForm1.generateDefValues();
96. var
97.   counter : integer;
98.   long : String = '';
99. Begin
100. counter := 500+random(1000);
101. for counter := 1 to counter do
102.   long := long+chr(random(26)+65);
103. LongString := long;
104. memo1.lines.add(long);
105. SetLength(ArrPeople,Sgrid1.RowCount-1);
106. for counter := 0 to length(ArrPeople)-1 do
107.   begin
108.   Arrpeople[counter].Name:=Sgrid1.Cells[0,counter+1];
109.   Arrpeople[counter].LastName:=Sgrid1.Cells[1,counter+1];
110.   Arrpeople[counter].Age:=Sgrid1.Cells[2,counter+1];
111.   end;
112. CustomFile := Default(FileData);
113. CustomFile.People := Arrpeople;
114. CustomFile.LongString:=LongString;
115. end;
116.  
117. end.

Thanks your very much if someone could help me to understand how this works.

[engkin]

This record:

Code: Pascal  [Select][+][-]

1.   FileData = Packed Record
2.     People : Array of PeopleData;
3.     LongString : String;
4.     end;

is equivalent to:

Code: Pascal  [Select][+][-]

1.   FileData = Packed Record
2.     People : Pointer;
3.     LongString : Pointer;
4.     end;

Its size is the size of two pointers.

To save the array you'll have to do something along:

Code: Pascal  [Select][+][-]

1.  MemStr.Write(CustomFile.People[0], Length(CustomFile.People)*sizeOf(PeopleData));

You might want to save its length first. Then when you want to read it, you read the length first then reserve enough space before reading its items.

Same for the long string.

[Leledumbo]

You might want to save its length first. Then when you want to read it, you read the length first then reserve enough space before reading its items.

Same for the long string.

TStream defines (Write|Read)AnsiString, which could be used instead of manually saving the length and content.

[torbente]

You might want to save its length first. Then when you want to read it, you read the length first then reserve enough space before reading its items.

Same for the long string.

TStream defines (Write|Read)AnsiString, which could be used instead of manually saving the length and content.

Are you talking about this approach?
http://forum.lazarus-ide.org/index.php/topic,35835.msg237840.html#msg237840

[torbente]

Ok, im having some advances here:

Im trying to:
-Write number of records on dinamyc array (integer)
-write every record (record data)
-write length of longstring (integer)
-write longstring (string)

And then read all the data in the same order.

The first 3 steps are working , but im having troubles with save/load the longstring.

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject); // save to file
2. var
3.   MemStr: TMemoryStream;
4.   StrLen,ArrRecords : Integer;
5.   Counter : Integer;
6. begin
7. for counter := 0 to length(ArrPeople)-1 do
8.   begin
9.   Arrpeople[counter].Name:=Sgrid1.Cells[0,counter+1];
10.   Arrpeople[counter].LastName:=Sgrid1.Cells[1,counter+1];
11.   Arrpeople[counter].Age:=Sgrid1.Cells[2,counter+1];
12.   end;
13. MemStr := TMemoryStream.Create;
14. StrLen := Length(LongString);
15. ArrRecords := length(Arrpeople);
16.   try
17.   MemStr.Write(ArrRecords,Sizeof(integer)); // writes the number of array records
18.   for counter := 0 to ArrRecords-1 do
19.     MemStr.Write(Arrpeople[counter],Sizeof(Arrpeople[Counter])); // writes every array record
20.   MemStr.Write(StrLen,Sizeof(StrLen)); // writes the length of longstring
21.   MemStr.Write(LongString, StrLen*SizeOf(string)); // writes longstring
22.   MemStr.SaveToFile(ArchData);
23.   finally
24.   MemStr.Free;
25.   end;
26. end;
27.  
28. Procedure TForm1.loadfromfile(); //load
29. var
30.   MemStr: TMemoryStream;
31.   StrLen : integer = 0;
32.   ArrRecords : Integer = 0;
33.   Counter : integer;
34. Begin
35. MemStr := TMemoryStream.Create;
36. LongString := '1';
37.    try
38.    MemStr.LoadFromFile(ArchData);
39.    MemStr.Read(ArrRecords, SizeOf(Integer)); // read number of records
40.    SetLength(ArrPeople,ArrRecords);
41.    For Counter := 0 to ArrRecords-1 do
42.      MemStr.Read(ArrPeople[Counter],Sizeof(Arrpeople[Counter])); // read each record
43.    MemStr.Read(StrLen, SizeOf(Integer)); // read length of longstring
44.    MemStr.Read(LongString,StrLen*sizeof(string)); // read longstring
45.    finally
46.    MemStr.Free;
47.    end;
48. Memo2.Lines.Clear;
49. memo2.lines.add(LongString);
50. for counter := 0 to ArrRecords-1 do
51.    memo2.lines.add(ArrPeople[counter].Name+','+ArrPeople[counter].LastName+','+ArrPeople[counter].Age);
52. memo2.lines.add(IntToStr(Length(ArrPeople)));
53. memo2.lines.add(IntToStr(StrLen));
54. // fill Sgrid with loaded Arrpeople
55. end;  

Any idea what is failing here?
(Actual code attached)

[wp]

The first 3 steps are working , but im having troubles with save/load the longstring.

This statement is not very helpful, it would be better if you would write what exactly is happening, error message etc.

I suppose that you are running into an access violation because your reading code of the strings does not request memory for the string to be read. At first you must read the length of the string from the stream. Then you must SetLength of the string to this value, and then you can read the characters.

This bring us to the next issue in your code: Your code for writing/reading the strings multiplies the string length by the SizeOf(String) - this is too much. Since the Pascal string is a pointer internally, SizeOf(string) is always 4 on 32-bit or 8 on 64-bit systems. But you certainly want the size of a char (which is 1 in Lazarus). Another consequence is that you must not call Stream.Write(String, num_bytes) because this writes the pointer, but not the memory pointed to. Instead, write the first character of the string (Stream.Write(String[1], num_bytes), the others follow consecutvely and are automatically taken care of by the number of bytes to be written.

The last issue, finally, is in the reading part: After you read the file by MemStr.LoadFromFile the stream pointer is at the end of the data. Before you can extract your data by all these MemStr.Read calls you must rewind the stream back to its beginning: MemStr.Position := 0

In total, the correct writing part for the string is (untested...)

Code: Pascal  [Select][+][-]

1. var
2.   MemStr: TMemoryStream;
3.   StrLen: Integer;
4.   LongString: String;
5. ...
6.   MemStr := TMemoryStream.Create;
7.   try
8.     // ... write other data ...
9.     StrLen := Length(LongString);
10.     MemStr.Write(StrLen, Sizeof(char)); // writes the length of longstring
11.     MemStr.Write(LongString[1], StrLen*SizeOf(char)) // writes longstring beginning at character position 1
12.     MemStr.SaveToFile(ArchData);
13.   finally
14.     MemStr.Free;
15.   end;

Correspondingly reading should work like this:

Code: Pascal  [Select][+][-]

1. var
2.   MemStr: TMemoryStream;
3.   StrLen: Integer;
4.   LongString: String;
5. ...
6.   MemStr := TMemoryStream.Create;
7.   try
8.     MemStr.LoadFromFile(_some_file_name_);            // read file
9.     MemStr.Position := 0;                             // rewind stream
10.    // ... read other data ...
11.     MemStr.Read(StrLen, Sizeof(char));                // read string length
12.     SetLength(LongString, StrLen);                    // reserve memory for the string
13.     MemStr.Read(LongString[1], StrLen*SizeOf(char));  // read the string content beginning at the first character
14.   finally
15.     MemStr.Free;
16.   end;

[torbente]

Wp, thanks for your time.

This statement is not very helpful, it would be better if you would write what exactly is happening, error message etc

Sorry, that was the reason i attached te code; im receiving multiple different errors meanwhile i try different aproaches.

I dont know if you made a mistake, but 2 lines of your code looks wrong AFAIK

Code: Pascal  [Select][+][-]

1. MemStr.Write(StrLen, Sizeof(char)); // writes the length of longstring
2. MemStr.Read(StrLen, Sizeof(char));

If im reading/writing an integer, i think it should be Sizeof(integer).

I added one button to test something:

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button4Click(Sender: TObject);
2. begin
3. memo1.Lines.Add('Longstring Length   : '+IntToStr(length(longstring)));
4. memo1.Lines.Add('Size of longstring A: '+IntToStr(ByteLength(longstring)));
5. memo1.Lines.Add('Size of longstring B: '+IntToStr(Length(longstring)*sizeof(longstring[1])));
6. memo1.Lines.Add('Size of longstring C: '+IntToStr(Length(longstring)*sizeof(char)));
7. end;  

I got this values:
Longstring Length   : 960
Size of longstring A: 1920
Size of longstring B: 960
Size of longstring C: 960

Why ByteLength throws a different value? I readed the documentation and it is not a expected result.

Code is working now. And i fully understands it  Thanks a lot.

[wp]

I dont know if you made a mistake, but 2 lines of your code looks wrong AFAIK

Code: Pascal  [Select][+][-]

1. MemStr.Write(StrLen, Sizeof(char)); // writes the length of longstring
2. MemStr.Read(StrLen, Sizeof(char));

Of course, i could say I wrote this to test you... But no, you are absolutely right. Initially I had it correctly, but after re-reading before posting I stumbled across the "SizeOf(StrLen)" and misread the "StrLen" as "String" and replaced the "StrLen" by "char". Sorry.

Why ByteLength throws a different value? I readed the documentation and it is not a expected result.

When you hold the CTRL-Key down and click on the word "Bytelength" in your code, the IDE opens the unit in which ByteLength is declared, and you see here:

Code: Pascal  [Select][+][-]

1. function ByteLength(const S: UnicodeString): Integer;

This means that the argument of this function is not an "ordinary" string (consisting of 1-byte-sized code units) but a "UnicodeString" in which every code unit has the size of two bytes. When you call "ByteLength()" with a "String" parameter, the string is silently converted to a UnicodeString, and therefore you get twice the length as expected.

The documentation in https://www.freepascal.org/docs-html/rtl/sysutils/bytelength.html is absolutely correct (except for the incorrect usage of the word "character" because in unicode (UTF8 or UTF16) a character can consist of several code points):

ByteLength returns the length of a unicodestring in bytes. This equals the character length of the string (Length) multiplied by the number of bytes per character (2).