Recent

Author Topic: BlockReads Questions  (Read 2897 times)

JLWest

  • Hero Member
  • *****
  • Posts: 1293
BlockReads Questions
« on: March 21, 2019, 10:21:38 pm »

Code: Pascal  [Select][+][-]
  1. procedure TForm1.BlockReadFile;
  2.  Var
  3.   MyFile: file;
  4.   Data  : array of Byte;
  5.   count : longint;
  6.   FName : String  = 'Test.txt';
  7.   Path  : String;
  8.   Loc   : String;
  9.  begin
  10.   Loc := Application.Location;
  11.   Path := Loc +  FName;
  12.   AssignFile(MyFile, path);
  13.   Reset(MyFile, 1 { size of read chunk });
  14.   count:= system.FileSize(MyFile);
  15.   SetLength(Data, count);
  16.   try
  17.     While not eof(MYFile) do begin
  18.      BlockRead(MyFile, Data[0], Length(Data));
  19.     end;
  20.   finally
  21.     CloseFile(MyFile);
  22.   end;
  23.  end;
  24.  

Basically I'm trying to read a large text file as quick as possible.

Trying to do blockreads on a text file. Maybe this isn't possible.
Under the debugger it does one read (No matter the size of the file)  and closes the file.
 
Can't figure out how to look at the data. I'm assuming the data read is in "Data[0]".

But something is wrong.

I attached the program if it helps.
FPC 3.2.0, Lazarus IDE v2.0.4
 Windows 10 Pro 32-GB
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
4.1 TB

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: BlockReads Questions
« Reply #1 on: March 21, 2019, 10:41:02 pm »

are you receiving a EinOUtError ?

The BlockRead has an additional Parameter you can use to report back the actual amount of bytes read.

BlockRead(MyFile, Data^, Count, ReportedLoadSize);

you need to supply the variable so it can write it back

P.S.
 you can also use A TmemoryStream that will load it and it has the features you need to stream the file
letter by letter..

 Or you could use TStringList!

The only true wisdom is knowing you know nothing

Bart

  • Hero Member
  • *****
  • Posts: 5275
    • Bart en Mariska's Webstek
Re: BlockReads Questions
« Reply #2 on: March 21, 2019, 10:41:10 pm »
From the old days, this is what I remember.
I never used EOF in combination with BlockRead.
BlockRead has an optional parameter that holds the actual bytes read.
You just keep on doing BlockRead (and process the data) until it reports it read less bytes then you requested.

Untested code.

Code: Pascal  [Select][+][-]
  1.   repeat
  2.     BlockRead(InFile, Buf, BufLen, BytesRead);
  3.     BlockWrite(OutFile, Buf, BytesRead, BytesWritten);
  4.     if (BytesWritten<>BytesRead) then
  5.       FatalError('Harddisk broken?');
  6.   until (BytesRead <> BufLen);

Your BlockRead example will read just one time, because your buffer is exactly the length of the file in bytes and you request to read that amount of bytes?

And yes, it will store the bytes it read into Data[0].

Bart

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: BlockReads Questions
« Reply #3 on: March 21, 2019, 10:44:51 pm »
TStringList.LoadFromFile() is pretty quick, and all the heavy lifting is already programmed for you.
I would only try to move away from standard FCL/LCL routines if they are too slow. And you may discover your alternative(s) are even slower.

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: BlockReads Questions
« Reply #4 on: March 21, 2019, 10:56:28 pm »
it does one read (No matter the size of the file)  and closes the file.

It does one read because the file, the whole file, is read in one go. Your code is reading: system.FileSize(MyFile).

furious programming

  • Hero Member
  • *****
  • Posts: 853
Re: BlockReads Questions
« Reply #5 on: March 22, 2019, 01:11:41 am »
Basically I'm trying to read a large text file as quick as possible.

This is no optimization, since later you'll be wasting the time parsing such a raw data.

If you want to quickly load the text file into memory, use the TStringList class and the LoadFromFile method. Not only is this method already implemented what you are just inventing, in addition, it divides the data into lines during loading, so you will not have to do it yourself.
Lazarus 3.2 with FPC 3.2.2, Windows 10 — all 64-bit

Working solo on an acrade, action/adventure game in retro style (pixelart), programming the engine and shell from scratch, using Free Pascal and SDL. Release planned in 2026.

Bart

  • Hero Member
  • *****
  • Posts: 5275
    • Bart en Mariska's Webstek
Re: BlockReads Questions
« Reply #6 on: March 22, 2019, 04:45:34 pm »
It does one read because the file, the whole file, is read in one go. Your code is reading: system.FileSize(MyFile).

Didn't I already point that out?  O:-)

Bart

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: BlockReads Questions
« Reply #7 on: March 22, 2019, 06:15:50 pm »
It does one read because the file, the whole file, is read in one go. Your code is reading: system.FileSize(MyFile).

Didn't I already point that out?  O:-)

Sorry, didn't see it the first time.  :-[

 

TinyPortal © 2005-2018