Recent

Author Topic: Won't load Apt.Dat.  (Read 13136 times)

wp

  • Hero Member
  • *****
  • Posts: 5444
Re: Won't load Apt.Dat.
« Reply #30 on: January 10, 2019, 10:19:27 am »
Wow. a dynamic array. I'll give it a try. Have a few questions like How to view the data after loaded into the array.

You should use some "virtual" control which can take the data from the external data structure. VirtualTreeView is such a control, however, it takes some time to understand it. Another control which even is on the usual component palette is TListView which can be put into a virtual mode by setting is OwnerData to true and providing an event handler for OnData which assigns the external data to a TListItem.

I put together a little demo which takes the KLAX.txt file that you posted somewhere else, reads it and displays its lines in the virtual listview. There is also a second virtual listview into which records having a common ID (the leading number in each line) are collected (click on the button "Filter"). The data displayed by the second listview are taken from the main data array by means of the corresponding indexes.
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

GetMem

  • Hero Member
  • *****
  • Posts: 3414
Re: Won't load Apt.Dat.
« Reply #31 on: January 10, 2019, 10:38:44 am »
@wp
You opened Pandora's box. :D Now OP will load the big file in VTV, which of course will work, VTV easily can handle 10M records. Still wrong approach in my opinion, the array must be filtered first. He mentioned somewhere that after filtering, only a few thousand item remains, this can be easily shown in any control. I intentionally avoided VTV, although it's my favorite component.

PS: Nice example.

wp

  • Hero Member
  • *****
  • Posts: 5444
Re: Won't load Apt.Dat.
« Reply #32 on: January 10, 2019, 10:52:18 am »
You opened Pandora's box. :D Now OP will load the big file in VTV, which of course will work, VTV easily can handle 10M records. Still wrong approach in my opinion, the array must be filtered first. He mentioned somewhere that after filtering, only a few thousand item remains, this can be easily shown in any control. I intentionally avoided VTV, although it's my favorite component.
Certainly. I only wanted to demonstrate usage of TListView in virtual mode. The code is not meant to be a solution to the OP's question.
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

JLWest

  • Sr. Member
  • ****
  • Posts: 390
Re: Won't load Apt.Dat.
« Reply #33 on: January 10, 2019, 05:31:43 pm »
@GetMem #39

I can do that, I understand that one.

When I tried to run I  was getting an out of bounds error immediately. So I did the record count thinking to size in one step. However I'm not sure where or how to do that.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

GetMem

  • Hero Member
  • *****
  • Posts: 3414
Re: Won't load Apt.Dat.
« Reply #34 on: January 10, 2019, 06:13:56 pm »
Quote
When I tried to run I  was getting an out of bounds error immediately. So I did the record count thinking to size in one step. However I'm not sure where or how to do that.
GetRecordCount returns a value, namely BlockSize(see my previous example, reply 29). You must use that to set the length of the array. Please try the following function to load the array:
Code: Pascal  [Select]
  1. function LoadDataFromFile(const AFileName: String): Boolean;
  2. var
  3.   DataFile: TextFile;
  4.   Counter: Integer = 0;
  5.   Line: String;
  6. begin
  7.   Result := True;
  8.   AssignFile(DataFile, AFileName);
  9.   try
  10.     Reset(DataFile);
  11.     SetLength(Data, BlockSize); // set here
  12.     while not eof(DataFile) do
  13.     begin
  14.       Readln(DataFile, Line);
  15.       Data[Counter].FLine := Line;
  16.       Data[Counter].FIndex := Len;
  17.       Inc(Counter)
  18.     end;
  19.     CloseFile(DataFile);
  20.    except
  21.      Result := False;
  22.    end;
  23. end;

JLWest

  • Sr. Member
  • ****
  • Posts: 390
Re: Won't load Apt.Dat.
« Reply #35 on: January 10, 2019, 11:52:26 pm »
Code: Pascal  [Select]
  1.     procedure TfMain.FormShow(Sender: TObject);
  2.     begin
  3.       Timer1.Enabled := True;
  4.     end;
  5.      
  6.     procedure TfMain.Timer1Timer(Sender: TObject);
  7.     begin
  8.       Timer1.Enabled := False;
  9.       Caption := 'Loading data. Please wait...';
  10.       BlockSize  := GetRecordCount(ExtractFilePath(Application.ExeName) + 'klax.dat');
  11.       if BlockSize = 0 then
  12.       begin
  13.         ShowMessage('Error loading file.');
  14.         Exit;
  15.       end;
  16.       //...
  17.     end;  

Shouldn't  TfMain.FormShow
make the call:

 Timer1Timer(NIL);

Under the debugger Timer1Timer is not executed.

I'm getting an out of bounds error on line 114. Which makes no sense because it is a beginstatement  but I think from the description it is really a few lines down down that reads    'Reset(DataFile); ' So I'm trying to run that down.

I don't understand the array declaration and set up.

At first I thought there were 3 arrays declared in the program.

The line PData = ^TData; What dose the ^ do. Have to run that down. but this is the record definition not an array, I understand that.

 Then we have under global variables two arrays:

var
  Data: array of TData;                     // array of records
  DataCondition : array of TData;      // array of the data

Thus where do I set length on the arrays.

I don't have a problem posting my code if it will help. I have made very few changes.
I would like to understand what I'm doing and not take and use without knowledge. I believe we are trying to pack the data strings in the records, but not sure. You guys are so quick and it's difficult for me to keep up at times.

So when new to me ideas come in I have to do the research, sometimes write a demo and be able to say, Oh that's how that works. Not complaining. Have a whole library of 264 sub directories of examples and a program that will bring up the text files of the demo's.   


JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

lucamar

  • Hero Member
  • *****
  • Posts: 1046
Re: Won't load Apt.Dat.
« Reply #36 on: January 11, 2019, 12:01:17 am »
The line PData = ^TData; What dose the ^ do. Have to run that down [...]

That is a normal, everyday declaration of a pointer-to-TData type. That is basic pascal, JLWest :)
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K)Ubuntu 12..16, Windows XP SP3 (Home/Prof.) and various DOS incarnations.

JLWest

  • Sr. Member
  • ****
  • Posts: 390
Re: Won't load Apt.Dat.
« Reply #37 on: January 11, 2019, 02:52:09 am »
I have two data sets. one with 62 records and one with 7.9 mil. rrecords It's pointed at the 62 record set and reading to the end of file. Low(Data) is 0 and (High(Data) is 61.

Figure out how to load them in a listbox, verify there there and do a setlength on data to 7.9 in one chunk and try for the 7.9 million.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

lucamar

  • Hero Member
  • *****
  • Posts: 1046
Re: Won't load Apt.Dat.
« Reply #38 on: January 11, 2019, 03:26:37 am »
Let's start from the basics: you don't have ~8 million records. What you have is data for X (rather less than 8 mill I guess) airports, heliports, etc. Each airport, heliport, etc. data is a record.

Now, that data happens to be stored in lines in a text file so that all sumed up there are close to 8 mill lines occupying 280 MiB in disk. At the front of each line there is a code representing what kind of data the line contains and the codes 1, 16 & 17 represent the start of data for an airport, seaplane base or helipad, respectively, and therefore the end of the data for the previous airport, seplane base, etc.

You want the user to be able to select one airport (heliport, etc), change (some of) its data and store it again in that giant text file.

Is this correct? Because the correct representation of your problem is one big step towards the solution.

One last question: Can you spare the disk space to have a duplicated data set?
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K)Ubuntu 12..16, Windows XP SP3 (Home/Prof.) and various DOS incarnations.

JLWest

  • Sr. Member
  • ****
  • Posts: 390
Re: Won't load Apt.Dat.
« Reply #39 on: January 11, 2019, 04:48:43 am »
"Is this correct? Because the correct representation of your problem is one big step towards the solution."

Yes, Correct to that point in your narrative. 

But the lines that need to be edited come in pairs of text lines. a 1301 and a corresponding 1302 text line. The 1301 is the gate number Gate A-12 and the 1302 is the Gate Specifications. Cargo, Airline, Military, A gate, Tie down and who owns the right to park at the gate. SWA, DAL AAL. Often the gates are co-leased in large airports.

"One last question: Can you spare the disk space to have a duplicated data set?"

Sat. my spec will change on this machine to 32 gig of ram and my son installed 3 terabytes of regular non SSD disk today  to go with my 1.5 terabyte SSD. So with 4.5 terabytes I think so.
 
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

Handoko

  • Hero Member
  • *****
  • Posts: 2899
  • My goal: build my own game engine using Lazarus
Re: Won't load Apt.Dat.
« Reply #40 on: January 11, 2019, 05:53:02 am »
You've got a nice son.

i7 GTX 1080 32 GB RAM 1.5 TB SSD 3 TB HDD => very expensive >:D

GetMem

  • Hero Member
  • *****
  • Posts: 3414
Re: Won't load Apt.Dat.
« Reply #41 on: January 11, 2019, 06:21:07 am »
@JLWest
Post your dat file, the big one. Just archive it and upload somewhere, otherwise we cannot help.

Handoko

  • Hero Member
  • *****
  • Posts: 2899
  • My goal: build my own game engine using Lazarus
Re: Won't load Apt.Dat.
« Reply #42 on: January 11, 2019, 06:49:27 am »
@JLWest

Quote
As it turns out with the free edition of Dropbox you would only be able to view. no download or copy.

There are a lots of Dropbox alternatives:
https://en.wikipedia.org/wiki/Comparison_of_online_backup_services

Not all services in the list above are free, pick carefully.

If you have problem choosing one, I recommend you Box.com. I ever used its free plan for sharing 30 MB file, it's easy and worked good.

If you still have problem, you can contact me. I can temporary host your file on my blog. I have a reseller web hosting account.

JLWest

  • Sr. Member
  • ****
  • Posts: 390
Re: Won't load Apt.Dat.
« Reply #43 on: January 11, 2019, 07:04:36 am »
You've got a nice son.

i7 GTX 1080 32 GB RAM 1.5 TB SSD 3 TB HDD => very expensive >:D

No Pretty cheap. My son in the 8 grade fixed all the neighbors computers, upgrades and builds. The equipment they gave him is unbelievable. He just took 4 compatible drives from the 30 or so he has in the closet and dropped them in.

The SSD I bought and the water cooled mother board but beyond that nothing. About $1,4000 dollars. The GForce1080 was given by way of an upgrade to a GForce 1080i or Ti with 32 meg of ram by one of his friends.

My machine is pretty good but the neighbor has a dual processors 6 terabyte, 64 gig Ram with an $8,000 graphics card.

He does bitcom on the graphics.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

JLWest

  • Sr. Member
  • ****
  • Posts: 390
Re: Won't load Apt.Dat.
« Reply #44 on: January 11, 2019, 07:11:46 am »
@JLWest

Quote
As it turns out with the free edition of Dropbox you would only be able to view. no download or copy.

There are a lots of Dropbox alternatives:
https://en.wikipedia.org/wiki/Comparison_of_online_backup_services

Not all services in the list above are free, pick carefully.

Yea, I found that out and dropped Dropbox.

I'll give it a go and try to get it on the web somewhere. It will take me http://forum.lazarus.freepascal.org/index.php?action=post;quote=307694;topic=43855.30;last_msg=307695a few days.

If you have problem choosing one, I recommend you Box.com. I ever used its free plan for sharing 30 MB file, it's easy and worked good.

If you still have problem, you can contact me. I can temporary host your file on my blog. I have a reseller web hosting account.

Thank you Handako, See what I can do. This is probably going to take me a couple of days.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.