Lazarus

Free Pascal => Beginners => Topic started by: JLWest on January 08, 2019, 11:00:54 pm

Title: Won't load Apt.Dat.
Post by: JLWest on January 08, 2019, 11:00:54 pm
Tried loading the whole file into a memo box and it fails. No error message just stops as if everything is OK but when I look at the file it should have a '99' as the last record.

I can verify this by opening a DOS box and entering 'Type Apt.Dat'. Scrolls the file to the screen and the last record is in fact '99'.

The last record in the memo box is a 1302 .............

It should give a failure to load message.

Is this a bug?
Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 08, 2019, 11:07:26 pm
Sounds more like dying by exhaustion :)

How big is the file? If it's too big the memo may simply load as much as it can and stop there, although IIRC it also should spout some error.

And ... didn't we already discuss this or something like this? (I seriously don't remember)
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 08, 2019, 11:28:51 pm
The file size is 285,986 KB according to Windows Explorer.
By count 7,970,312 Records.

They are not fixed length but have variable size records with some records quite large others small.
Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 08, 2019, 11:38:49 pm
If by "records" you mean "lines" then it's too big. Remember that the base storage for memos is TStrings and its Count property is a standard (32 bits) Integer, i.e. it can contain a max of 4 MLines. The ~280MiB of data don't help either, because TStrings isn't memory efficient and it can use (depending on the operations performed) as much as four times the assumed capacity.

Humm ... Are you sure we haven't discussed all this already?
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 12:19:32 am
Yea, We have discussed all this and then Handoko wrote a program that generated a 10,000,000 record file into a Memo box. Worked fine. So I thought I would try and load the 7,970,312 Records, seemed reasonable.

I think something isn't right. He loading more than 2,000,000 records that I am trying to do. Should make up for my variable length records. And I think they should load.

I also think there is bug with this. FP should report a failure.
Title: Re: Won't load Apt.Dat.
Post by: garlar27 on January 09, 2019, 12:43:11 am
Humm ... Are you sure we haven't discussed all this already?

For what I saw there are 2 other threads 7,970,312 As it turns out. (http://forum.lazarus.freepascal.org/index.php/topic,43806.0.html) (on: 2019-01-04, 15:34:24) and Just advise (http://forum.lazarus.freepascal.org/index.php/topic,43733.0.html) (on: 2018-12-29, 21:32:43). I don't know if there are more...
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 01:03:11 am
"Yea, We have discussed all this and then Handoko wrote a program that generated a 10,000,000 record file (Text Lines)  into a Memo box. Worked fine. So I thought I would try and load the 7,970,312 (Text Lines) Records, seemed reasonable.

I think something isn't right. He loading more than 2,000,000 records that I am trying to do. Should make up for my variable length records. And I think they should load.

I also think there is bug with this. FP should report a failure."

As stated above. It has been discussed. What I'm asking now is:

Is this a bug. Should it be reported?


If Handoko can generate 10,000,000 records into a memo box but you can't load 7.9 million something is wrong.
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 09, 2019, 01:34:08 am
Hi

I think you hitting the limitation of a memo, the actual amount of memory a memo utilizes can double/triple/quadruple in size during memory maintenance.

You should also notice the memo filling up slows down after a few thousand additions, as the manager is handling larger and larger chunks of memory...

Attached a quick test, that will attempt to fill a memo with 8000000 largest lines one by one,
It will give you a runnign progress on the length of the memo used.

If you open taskmanager and monitor it when you run it; you should see the memory temporarily jump.

I have also created multiple debug modes, so run it in the debug mode; as runnign it in normal or release various checks will be disabled.

Hope it helps

ps. not the most eligent or efficient code. No checks etc....
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 02:51:15 am
Thanks I'll test it. Mine didn't really slow down until about 5mil and then just a little.

I did find a free editor that can handle very large text files. And it does it with ease. It's called Kainet Editor. It's very basic but fast and a way to check results with large file. Notepad and Notepad++ can't handle the records but Kainet with ease.

I recommend it if anyone is interested.

Thanks Josh
Title: Re: Won't load Apt.Dat.
Post by: Handoko on January 09, 2019, 07:00:00 am
It seems you misunderstood and didn't get to points I want to tell you on my previous posts.

The workarounds I suggest are:

For memory friendly
Load a chunk of data into memory, do some processing, store the result, free the data from memory and continue to do the same on the next chunk.

For user friendly
You mentioned you want to load the items into a ListBox, do you show the ListBox to user for selecting some items?
Selecting one or several items from a list box that contains thousands of items is a bad user-experience design. You should separate them into several categories.

Write your own component
It's a possible solution but I do not recommend. Because it is harder you can imagine.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 07:15:41 am
After about an hour got an out of memory message and continue abort box.

I think I now know how to write this program.

1. build a double hash file for the Airports, Sea Planes Bases and Heliports. 35,000 plus records. They would look like:

Airport      SRCD     EREC
KPHX      721478   751523
KPDX      751524   762621

Load this into a listbox and allow double click of a airport.

Open the 7.9 million file read to the first record (SRCD) and load to the last record (ERCD) of the airport you are working with. You don't have that much in memory.

When your done write them out.

Pick another airport or close the file and quit. Tricky, may have to close the file and start  all over at the first record again depending on airport picked.

I would like to eliminate the page up and page down buttons and maybe the up and down arrows.  What would be best is to control how far they could move the focus in the list box.

I would need to stop them at the SRCD vector  and EREC  vector.

Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 07:38:15 am
@Handako
For user friendly
You mentioned you want to load the items into a ListBox, do you show the ListBox to user for selecting some items?
Selecting one or several items from a list box that contains thousands of items is a bad user-experience design. You should separate them into several categories.


For user friendly Agree: Have to load all of KPDX and then have a working listbox with just the records to be modified. Good call. Putting it back to gatherer will be a little tricky.

I went back and figured out that you weren't loading that many records. Actually, I modified your code to read and load in a memo and I got close. Because at the time I couldn't load the file in any editor I couldn't tell where in the file it quit but I think about 300,000 from the end from what the counters showed. I think I was out of memory so I ordered more memory today. My kid who is something of a guru with computers says it won't help on a 32 bit program.

Looks like a big turn and burn to get to a 64 bit compiler and it may not help.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 09, 2019, 07:54:08 am
@JLWest

I do not understand why you and others keep insisting on using Memos, Listboxes for processing data. This is ridiculous. You should switch to dynamic arrays, they are fast and can handle much more then 7M records. Loading 7M lines to a memo make no sense anyways, the user cannot process that much information. Lots of bad advice in my opinion.
Although I'm not familiar with the exact requirements, you should try something like this:
Code: Pascal  [Select][+][-]
  1. type
  2.   PData = ^TData;
  3.   TData = record
  4.     FLine: String;
  5.     FIndex: Integer;
  6.     //other data if needed
  7.   end;
  8.  
  9. var
  10.   Data: array of TData;
  11.   DataCondition: array of TData;
  12.  
  13. function LoadDataFromFile(const AFileName: String): Boolean;
  14. var
  15.   DataFile: TextFile;
  16.   Len: Integer;
  17.   Line: String;
  18. begin
  19.   Result := True;
  20.   AssignFile(DataFile, AFileName);
  21.   try
  22.     Reset(DataFile);
  23.     while not eof(DataFile) do
  24.     begin
  25.       Readln(DataFile, Line);
  26.       Len := Length(Data);
  27.       SetLength(Data, Len + 1);
  28.       Data[Len].FLine := Line;
  29.       Data[Len].FIndex := Len;
  30.     end;
  31.     CloseFile(DataFile);
  32.    except
  33.      Result := False;
  34.    end;
  35. end;
  36.  
  37. procedure TForm1.Button1Click(Sender: TObject);
  38. var
  39.   Len, I: Integer;
  40. begin
  41.   Data := nil;
  42.   DataCondition := nil;
  43.   if not LoadDataFromFile(ExtractFilePath(Application.Exename) + 'klax.dat') then
  44.   begin
  45.     MessageDlg('Cannot load file! ', mtError, [mbOk], 0);
  46.     Exit;
  47.   end;
  48.   //for some reason you wish to start from the last record, I saw it in one of your previous thread
  49.   for I := High(Data) downto Low(Data) do
  50.   begin
  51.     //check for some conditions, if the conditions are met copy the record/line to another array DataCondition(Listbox2,3 equivalent)
  52.    //you can define more arrays/records if needed)
  53.     Len := Length(DataCondition);
  54.     SetLength(DataCondition, Len + 1);
  55.     DataCondition[Len].FLine := Data[I].FLine;
  56.     DataCondition[Len].FIndex := Data[I].FIndex; //keep the original index for later reference, maybe you wish to delete from the file
  57.   end;
  58.   //do more stuff here, if you have to show to the user informations from DataCondition,
  59.   //you can add it to a listbox or even better to a virtualtreeview
  60. end;
  61.  

Title: Re: Won't load Apt.Dat.
Post by: wp on January 09, 2019, 09:11:48 am
@JLWest

I do not understand why you and others keep insisting on using Memos, Listboxes for processing data. This is ridiculous. You should switch to dynamic arrays, they are fast and can handle much more then 7M records. Loading 7M lines to a memo make no sense anyways, the user cannot process that much information. Lots of bad advice in my opinion.
Although I'm not familiar with the exact requirements, you should try something like this:
Absolutely. However, in this sample code, the array length is incremented by 1. This means that the array must be copied to another place whenever a new line is read which will become very slow after some time in case of large files. I would pre-allocate a larger block of array items and use a line counter to set the final dimension at the end:

Code: Text  [Select][+][-]
  1. type
  2.   PData = ^TData;
  3.   TData = record
  4.     FLine: String;
  5.     FIndex: Integer;
  6.     //other data if needed
  7.   end;
  8.  
  9. function LoadDataFromFile(const AFileName: String): Boolean;
  10. const
  11.   BLOCK_SIZE = 1000;  // Experiment with this value
  12. var
  13.   DataFile: TextFile;
  14.   Line: String;
  15.   Counter: Integer = 0;
  16. begin
  17.   Result := True;
  18.   AssignFile(DataFile, AFileName);
  19.   try
  20.     Reset(DataFile);
  21.     while not eof(DataFile) do
  22.     begin
  23.       Readln(DataFile, Line);
  24.       if Counter mod BLOCK_SIZE = 0 then
  25.         SetLength(Data, Length(Data) + BLOCK_SIZE);
  26.       Data[Counter].FLine := Line;
  27.       Data[Counter].FIndex := Counter;
  28.       inc(Counter);
  29.     end;
  30.     CloseFile(DataFile);
  31.     SetLength(Data, Counter);
  32.    except
  33.      Result := False;
  34.    end;
  35. end;
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 09, 2019, 09:47:20 am
@wp
Quote
This means that the array must be copied to another place whenever a new line is read which will become very slow after some time in case of large files. I would pre-allocate a larger block of array items and use a line counter to set the final dimension at the end
OK. Good point :), however after running a few tests the results are not so convincing. For 10M line text file(Lazarus 32bit, FPC 3.0.4, win10):
My method: 28125 ms
Your method: 27530 ms
Your version is still better but not by much. Please run a few tests yourself, I'm curious what the results are at your side.


Title: Re: Won't load Apt.Dat.
Post by: wp on January 09, 2019, 10:31:07 am
In order to have something comparable, can you post how you create the 10M lines file?

Your results indicate that file access seems to be the limiting factor. There has been a recent discussion in which Bart showed that using the correct size of text buffers can speed up conventional file I/O considerably (https://forum.lazarus.freepascal.org/index.php/topic,42629.msg297970.html).
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 09, 2019, 10:41:20 am
Quote
In order to have something comparable, can you post how you create the 10M lines file?
http://forum.lazarus.freepascal.org/index.php/topic,43806.msg307101.html#msg307101
Title: Re: Won't load Apt.Dat.
Post by: wp on January 09, 2019, 11:29:18 am
Standard I/O
Code: [Select]
BLOCK_SIZE      Time to read
-----------     ------------
         1          42.8 s
      1000          42.5 s
    10,000          36.1 s
   100,000          11.7 s
 1,000,000           8.5 s
10,000,000           8.3 s

ReadLn only          7.8 s

Using SetTextBuf with varied BUFFER_SIZE:
Code: [Select]
BUFFER_SIZE    BLOCK_SIZE     Time to read
-----------    ----------     ------------
 1 kB          10,000,000         5.1 s
 1 MB          10,000,000         4.0 s
16 MB          10,000,000         3.9 s

 1 kB          ReadLn only        4.7 s
 1 MB          ReadLn only        3.6 s
16 MB          ReadLn only        3.6 s
Title: Re: Won't load Apt.Dat.
Post by: CCRDude on January 09, 2019, 01:01:55 pm
Some thoughts...

No screen is capable of displaying 10 million datasets at once.
No user is capable of viewing 10 million datasets at once.
Which means they don't need to be in memory at the same time.

So whatever kind of display there is for these, think about pagination. Display the visible chunks, make navigation between these chunks as easy as possible.

And once you're there - think about using databases, since they totally simplify (and speed up if you use proper indizes) that kind of use in the background.
Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 09, 2019, 01:21:20 pm
Pick another airport or close the file and quit. Tricky, may have to close the file and start  all over at the first record again depending on airport picked.

You can avoid that by using streams and doing a Seek to the point you want to start reading. Use a buffered stream, else your program will slow to a crawl very quickly.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 09, 2019, 02:05:42 pm
@wp
It looks like for BLOCK_SIZE larger then 100,000 the speed increase is significant, however you can overshoot the length of the array with a large amount. I find a solution which can count the records fast(3 seconds on my computer), then I can set the length to the exact value:
Code: Pascal  [Select][+][-]
  1. function GetRecordCount(const AFileName: string): Integer;
  2. var
  3.   DataFile: TextFile;  
  4. begin
  5.   Result := 0;
  6.   AssignFile(DataFile, AFileName);
  7.   try
  8.     Reset(DataFile);  
  9.     while not eof(DataFile) do
  10.     begin
  11.       Readln(DataFile);//this line has changed, basically it reads nothing but is needed otherwise we go into an endless loop
  12.       Inc(Result);
  13.     end;
  14.     CloseFile(DataFile);
  15.   except
  16.     //...
  17.   end;
  18. end;
  19.  


PS: SetTextBuff indeed helps, the whole process(counting + loading the data) takes 5-6 seconds on my computer.
PS1: I think OP has enough info now to code his application.
Title: Re: Won't load Apt.Dat.
Post by: howardpc on January 09, 2019, 02:14:52 pm
No screen is capable of displaying 10 million datasets at once.
No user is capable of viewing 10 million datasets at once.
Which means they don't need to be in memory at the same time.
As CCRDude points out, when trying to edit huge amounts of data, it is essential to decouple the data location, data editing and data display functions.
Which of the 10 million lines require editing?Just a few?How can you locate them without having to display them? What identifies them? Or perhaps you should report to the user that the file has no data that needs editing?
If the file does contain relevant line(s) that require editing, your display function should limit itself just to that line or lines.
And your edit function should know how to reinsert edited lines into the original text file at the correct location.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 08:00:23 pm
Wow. a dynamic array. I'll give it a try. Have a few questions like How to view the data after loaded into the array. I guess you could load the ones you need into a listbox maybe?


@howardpc
Yes I agree, no one can handle 7.9 million records. My original thinking was to load all records into a listbox. Find the ones I needed and place them into a second listbox and then allow editing and put them back into the 1st listbox and write out to file.

@GetMem
Oh I know exactly how many records are in the file. 7,970, 312. So can i do this:

Block_Size = 7970312;
SetLength(Data, Block_Size);

I can locate the lines needed for editing with a recordID of 1301 and 1302 First field in the record. They come as a pair in the record set of say KPDX, Portland Or within the Apt.Dat file.

The 1301 is the gate number 'Gate-A12' just like when  you go to the airport and the second record defines what type of gate, Cargo, Airline, Military and what ICAO Operations can use the gate, DAL SWA AAL. Almost all gate at a large airport are co-leased by several ICAO Operators.
 

I don't know if the question was directed to me but I didn't create the file. It comes with X-Plane 11. You can get it by downloading a Demo copy of X-Plane 11 for free and saving off the file and deleting X-Plane 11. It's a big download. Or try it out. It's interesting, it's not really a game.

Think of the movie "Scully' and they are flying the simulators to duplicate his landing on the Hudson. That's X-Plane 11. Of course I don't have the commercial version but the retail version.

I'll have some questions but I think I can download the code provided and get an array loaded.

Thanks All
 
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 09, 2019, 11:55:26 pm
@wp
It looks like for BLOCK_SIZE larger then 100,000 the speed increase is significant, however you can overshoot the length of the array with a large amount. I find a solution which can count the records fast(3 seconds on my computer), then I can set the length to the exact value:
Code: Pascal  [Select][+][-]
  1. function GetRecordCount(const AFileName: string): Integer;
  2. var
  3.   DataFile: TextFile;  
  4. begin
  5.   Result := 0;
  6.   AssignFile(DataFile, AFileName);
  7.   try
  8.     Reset(DataFile);  
  9.     while not eof(DataFile) do
  10.     begin
  11.       Readln(DataFile);//this line has changed, basically it reads nothing but is needed otherwise we go into an endless loop
  12.       Inc(Result);
  13.     end;
  14.     CloseFile(DataFile);
  15.   except
  16.     //...
  17.   end;
  18. end;
  19.  


PS: SetTextBuff indeed helps, the whole process(counting + loading the data) takes 5-6 seconds on my computer.
PS1: I think OP has enough info now to code his application.

I put this in FormCreaete and on my machine by the time the form shows it has read the file and posted the number to BlockSize which is a global. I didn't time it but about 2 seconds.

Can't seem to get it to load the array.
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 10, 2019, 01:01:37 am
Hi

Your not reading any data into any array

I have not tested it at all, just typed it; so may not work straight off, just a concept..

Code: [Select]
// set some const and variables up
const  MyMaxArraySize=8000000;
Var  s:string;
     MyFileHandle:textfile;
     MyCounter:longint=0;
     myline:string='';
     MyArray:Array of String;
     MyFileName:AnsiString;


// code to read in a text file into dynamic array, and adjust array after
// some minimal test during routine

// please note I would not do this iin the on create event; if this routine takes some time
// to complete on an old slow machine, with standard hd, users will not see anything until it has finished.
// best to add into form show event; and have a global variable so that it is only run once.
// that way your application will show; whilst its working.


Setlength(MyArray,MyMaxArraySize);
if fileexists(MyFileName) Then
begin
  Assignfile(MyFileHandle,MyFileName);
  Reset(MyFileHandle);
  myGetOut:=False;
  While ((not eof(MyFileHandle)) and (MyGetOut=False)) do
  begin
    readln(MyFileHandle,myline);
    if myline<>'' then
    begin
      myArray[MyCounter]:=s;
      inc(MyCounter);
      If MyCounter>MyMaxArraySize then
      begin
        showmessage('Too Much Data, exceeded :'+inttostr(MyMaxArraySize))
        MyGetOut:=True;
      End;
    End;
  end;
  closefile(MyFileHandle);
  setlength(MyArray,MyCounter);
end;                           
Title: Re: Won't load Apt.Dat.
Post by: wp on January 10, 2019, 01:03:46 am
Or use the code posted in reply #12 or #13
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 10, 2019, 01:18:53 am
Oops

Should have looked back a page...

Too many different topics on same subject; making it awkward to follow...

This is massive dataset; do we know how this is going to made into a user-workable UI yet?
I think the UI should be thought out first; as this will guide the route of how to handle it.

What spec machine is this designed to run on?



Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 10, 2019, 02:57:29 am
Yea, Some of that has been thought out ahead of time. It will run on Windows 7 and above. Designed to run on a Windows maching simular to my spec's at the bottom of the post.

Getting it loaded is the problem right this min. The current proposal is a dynamic array.

Each text line in the file has a record indicator, '1', '16' and '17' in the first column denotes an airport, seaplane base or helipad. Within those records in the fifth field is the name of the airport. Followed by 2 or 3,000 records of lights, runways, taxiways, painted lines, signs - most of which I don't have to deal with.

What I need to edit is the 1301 (RampStarts) and 1302(GateAssigments) records. Maybe 30 for a small airport to 400 records for an International. LAX or KJFK, Chicago O'Hare. 

Load those into a listbox and allow editing. Then dump the file to disk. The idea is the file would remain open but only the 1301 and 1302 would be visible and in memory.


Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 10, 2019, 03:05:31 am
Or use the code posted in reply #12 or #13

Yes, I'm using #12, #13 and the record count function. The record count function works great and and returns the correct number of records.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 10, 2019, 05:42:58 am
@JLWest
Quote
I put this in FormCreaete and on my machine by the time the form shows it has read the file and posted the number to BlockSize which is a global. I didn't time it but about 2 seconds.
Don't do that, on a slower computer it might take much longer to count the records. Please drop a TTimer to your form, set Interval to 100, Enabled to false, create an OnTimer event then:
Code: Pascal  [Select][+][-]
  1. procedure TfMain.FormShow(Sender: TObject);
  2. begin
  3.   Timer1.Enabled := True;
  4. end;
  5.  
  6. procedure TfMain.Timer1Timer(Sender: TObject);
  7. begin
  8.   Timer1.Enabled := False;
  9.   Caption := 'Loading data. Please wait...';
  10.   BlockSize  := GetRecordCount(ExtractFilePath(Application.ExeName) + 'klax.dat');
  11.   if BlockSize = 0 then
  12.   begin
  13.     ShowMessage('Error loading file.');
  14.     Exit;
  15.   end;
  16.   //...
  17. end;  

Quote
Getting it loaded is the problem right this min. The current proposal is a dynamic array.
What is the exact problem? Do you get an error? Please be more specific. If there is no error, add the following code(after the array is loaded):
Code: Pascal  [Select][+][-]
  1.   for I := High(Data) downto High(Data) - 100 do
  2.      ListBox1.Items.Add(DataCondition[Len].FLine);
  3.  
You should see the last 100 records in the listbox.



Title: Re: Won't load Apt.Dat.
Post by: wp on January 10, 2019, 10:19:27 am
Wow. a dynamic array. I'll give it a try. Have a few questions like How to view the data after loaded into the array.

You should use some "virtual" control which can take the data from the external data structure. VirtualTreeView is such a control, however, it takes some time to understand it. Another control which even is on the usual component palette is TListView which can be put into a virtual mode by setting is OwnerData to true and providing an event handler for OnData which assigns the external data to a TListItem.

I put together a little demo which takes the KLAX.txt file that you posted somewhere else, reads it and displays its lines in the virtual listview. There is also a second virtual listview into which records having a common ID (the leading number in each line) are collected (click on the button "Filter"). The data displayed by the second listview are taken from the main data array by means of the corresponding indexes.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 10, 2019, 10:38:44 am
@wp
You opened Pandora's box. :D Now OP will load the big file in VTV, which of course will work, VTV easily can handle 10M records. Still wrong approach in my opinion, the array must be filtered first. He mentioned somewhere that after filtering, only a few thousand item remains, this can be easily shown in any control. I intentionally avoided VTV, although it's my favorite component.

PS: Nice example.
Title: Re: Won't load Apt.Dat.
Post by: wp on January 10, 2019, 10:52:18 am
You opened Pandora's box. :D Now OP will load the big file in VTV, which of course will work, VTV easily can handle 10M records. Still wrong approach in my opinion, the array must be filtered first. He mentioned somewhere that after filtering, only a few thousand item remains, this can be easily shown in any control. I intentionally avoided VTV, although it's my favorite component.
Certainly. I only wanted to demonstrate usage of TListView in virtual mode. The code is not meant to be a solution to the OP's question.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 10, 2019, 05:31:43 pm
@GetMem #39

I can do that, I understand that one.

When I tried to run I  was getting an out of bounds error immediately. So I did the record count thinking to size in one step. However I'm not sure where or how to do that.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 10, 2019, 06:13:56 pm
Quote
When I tried to run I  was getting an out of bounds error immediately. So I did the record count thinking to size in one step. However I'm not sure where or how to do that.
GetRecordCount returns a value, namely BlockSize(see my previous example, reply 29). You must use that to set the length of the array. Please try the following function to load the array:
Code: Pascal  [Select][+][-]
  1. function LoadDataFromFile(const AFileName: String): Boolean;
  2. var
  3.   DataFile: TextFile;
  4.   Counter: Integer = 0;
  5.   Line: String;
  6. begin
  7.   Result := True;
  8.   AssignFile(DataFile, AFileName);
  9.   try
  10.     Reset(DataFile);
  11.     SetLength(Data, BlockSize); // set here
  12.     while not eof(DataFile) do
  13.     begin
  14.       Readln(DataFile, Line);
  15.       Data[Counter].FLine := Line;
  16.       Data[Counter].FIndex := Len;
  17.       Inc(Counter)
  18.     end;
  19.     CloseFile(DataFile);
  20.    except
  21.      Result := False;
  22.    end;
  23. end;
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 10, 2019, 11:52:26 pm
Code: Pascal  [Select][+][-]
  1.     procedure TfMain.FormShow(Sender: TObject);
  2.     begin
  3.       Timer1.Enabled := True;
  4.     end;
  5.      
  6.     procedure TfMain.Timer1Timer(Sender: TObject);
  7.     begin
  8.       Timer1.Enabled := False;
  9.       Caption := 'Loading data. Please wait...';
  10.       BlockSize  := GetRecordCount(ExtractFilePath(Application.ExeName) + 'klax.dat');
  11.       if BlockSize = 0 then
  12.       begin
  13.         ShowMessage('Error loading file.');
  14.         Exit;
  15.       end;
  16.       //...
  17.     end;  

Shouldn't  TfMain.FormShow
make the call:

 Timer1Timer(NIL);

Under the debugger Timer1Timer is not executed.

I'm getting an out of bounds error on line 114. Which makes no sense because it is a beginstatement  but I think from the description it is really a few lines down down that reads    'Reset(DataFile); ' So I'm trying to run that down.

I don't understand the array declaration and set up.

At first I thought there were 3 arrays declared in the program.

The line PData = ^TData; What dose the ^ do. Have to run that down. but this is the record definition not an array, I understand that.

 Then we have under global variables two arrays:

var
  Data: array of TData;                     // array of records
  DataCondition : array of TData;      // array of the data

Thus where do I set length on the arrays.

I don't have a problem posting my code if it will help. I have made very few changes.
I would like to understand what I'm doing and not take and use without knowledge. I believe we are trying to pack the data strings in the records, but not sure. You guys are so quick and it's difficult for me to keep up at times.

So when new to me ideas come in I have to do the research, sometimes write a demo and be able to say, Oh that's how that works. Not complaining. Have a whole library of 264 sub directories of examples and a program that will bring up the text files of the demo's.   


Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 11, 2019, 12:01:17 am
The line PData = ^TData; What dose the ^ do. Have to run that down [...]

That is a normal, everyday declaration of a pointer-to-TData type. That is basic pascal, JLWest :)
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 02:52:09 am
I have two data sets. one with 62 records and one with 7.9 mil. rrecords It's pointed at the 62 record set and reading to the end of file. Low(Data) is 0 and (High(Data) is 61.

Figure out how to load them in a listbox, verify there there and do a setlength on data to 7.9 in one chunk and try for the 7.9 million.
Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 11, 2019, 03:26:37 am
Let's start from the basics: you don't have ~8 million records. What you have is data for X (rather less than 8 mill I guess) airports, heliports, etc. Each airport, heliport, etc. data is a record.

Now, that data happens to be stored in lines in a text file so that all sumed up there are close to 8 mill lines occupying 280 MiB in disk. At the front of each line there is a code representing what kind of data the line contains and the codes 1, 16 & 17 represent the start of data for an airport, seaplane base or helipad, respectively, and therefore the end of the data for the previous airport, seplane base, etc.

You want the user to be able to select one airport (heliport, etc), change (some of) its data and store it again in that giant text file.

Is this correct? Because the correct representation of your problem is one big step towards the solution.

One last question: Can you spare the disk space to have a duplicated data set?
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 04:48:43 am
"Is this correct? Because the correct representation of your problem is one big step towards the solution."

Yes, Correct to that point in your narrative. 

But the lines that need to be edited come in pairs of text lines. a 1301 and a corresponding 1302 text line. The 1301 is the gate number Gate A-12 and the 1302 is the Gate Specifications. Cargo, Airline, Military, A gate, Tie down and who owns the right to park at the gate. SWA, DAL AAL. Often the gates are co-leased in large airports.

"One last question: Can you spare the disk space to have a duplicated data set?"

Sat. my spec will change on this machine to 32 gig of ram and my son installed 3 terabytes of regular non SSD disk today  to go with my 1.5 terabyte SSD. So with 4.5 terabytes I think so.
 
Title: Re: Won't load Apt.Dat.
Post by: Handoko on January 11, 2019, 05:53:02 am
You've got a nice son.

i7 GTX 1080 32 GB RAM 1.5 TB SSD 3 TB HDD => very expensive >:D
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 11, 2019, 06:21:07 am
@JLWest
Post your dat file, the big one. Just archive it and upload somewhere, otherwise we cannot help.
Title: Re: Won't load Apt.Dat.
Post by: Handoko on January 11, 2019, 06:49:27 am
@JLWest

Quote
As it turns out with the free edition of Dropbox you would only be able to view. no download or copy.

There are a lots of Dropbox alternatives:
https://en.wikipedia.org/wiki/Comparison_of_online_backup_services

Not all services in the list above are free, pick carefully.

If you have problem choosing one, I recommend you Box.com. I ever used its free plan for sharing 30 MB file, it's easy and worked good.

If you still have problem, you can contact me. I can temporary host your file on my blog. I have a reseller web hosting account.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 07:04:36 am
You've got a nice son.

i7 GTX 1080 32 GB RAM 1.5 TB SSD 3 TB HDD => very expensive >:D

No Pretty cheap. My son in the 8 grade fixed all the neighbors computers, upgrades and builds. The equipment they gave him is unbelievable. He just took 4 compatible drives from the 30 or so he has in the closet and dropped them in.

The SSD I bought and the water cooled mother board but beyond that nothing. About $1,4000 dollars. The GForce1080 was given by way of an upgrade to a GForce 1080i or Ti with 32 meg of ram by one of his friends.

My machine is pretty good but the neighbor has a dual processors 6 terabyte, 64 gig Ram with an $8,000 graphics card.

He does bitcom on the graphics.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 07:11:46 am
@JLWest

Quote
As it turns out with the free edition of Dropbox you would only be able to view. no download or copy.

There are a lots of Dropbox alternatives:
https://en.wikipedia.org/wiki/Comparison_of_online_backup_services

Not all services in the list above are free, pick carefully.

Yea, I found that out and dropped Dropbox.

I'll give it a go and try to get it on the web somewhere. It will take me http://forum.lazarus.freepascal.org/index.php?action=post;quote=307694;topic=43855.30;last_msg=307695a few days.

If you have problem choosing one, I recommend you Box.com. I ever used its free plan for sharing 30 MB file, it's easy and worked good.

If you still have problem, you can contact me. I can temporary host your file on my blog. I have a reseller web hosting account.

Thank you Handako, See what I can do. This is probably going to take me a couple of days.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 11, 2019, 07:21:55 am
@JLWest
Quote
This is probably going to take me a couple of days.
:o  Don't forget that the dat file is mostly text, archived should be under 100MB. You can even send it to me via mail.   
Title: Re: Won't load Apt.Dat.
Post by: wp on January 11, 2019, 10:21:06 am
@wp
You opened Pandora's box. :D Now OP will load the big file in VTV, which of course will work, VTV easily can handle 10M records. Still wrong approach in my opinion, the array must be filtered first. He mentioned somewhere that after filtering, only a few thousand item remains, this can be easily shown in any control.
Loading only the filtered data into memory will make it more difficult to write a valid file back after editing some records. In order to save an edited record you must read the original file line by line and write each line immediately to the new file; when the line which was edited is read the modified content must be written to the new file instead of the original line.

Or, during first reading of the file you collect an index of the stream positions of the start of all lines. After editing a specific line you look up in the index at which stream position this line begins in the original file, read everything en-block before that line and write it to a new file, write the new line, then jump to the next line in the original file and read and write the rest of the file. The index must be recreated after the new file is written.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 11, 2019, 10:50:06 am
What I meant to say by "Still wrong approach in my opinion, the array must be filtered first" is: load the whole file into a dynamic array, filter it, then show the filtered records to the user(Listbox, VTV, whatever). In my #12 reply, I even store the index of the large array for future reference.
Title: Re: Won't load Apt.Dat.
Post by: wp on January 11, 2019, 11:14:46 am
OK, I misunderstood this. But I think the idea of filtering the file during reading to load only the records needed is good. It is mentioned somewhere that the file has several hundreds of MB in size. Depending on the system and what else is running this may be too much for 32 bit systems. And maybe there are files which are even larger, who knows? Of course when the OP allows his program to be 64bit then this point can be neglegted.

P.S. I find it interesting that we reached the point to discuss out-of-memory issues again after burying DOS.
Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 11, 2019, 01:38:39 pm
Yes, Correct to that point in your narrative. 

But the lines that need to be edited come in pairs of text lines. a 1301 and a corresponding 1302 text line. The 1301 is the gate number Gate A-12 and the 1302 is the Gate Specifications. Cargo, Airline, Military, A gate, Tie down and who owns the right to park at the gate. SWA, DAL AAL. Often the gates are co-leased in large airports.

That second point isn't really important to the problem at hand, which is how to avoid having to handle all 280 MiB. If the rest is correct then a pretty easy solution is to divide the huge file into smaller files and an index.

The basic process would be: read all lines pertaining to an airport and save them to a new file with just the data for that airport and save the base airport line (those coded with 1, 16, 17) to an index file. Then on to the next airport, etc.

Once you have all those smaller fles you cn load the index, allow the user to select and load (and parse?), for modification, only the file corresponding to that airport. Finally, add an order in your program to regenerate the APP.DAT, which can be easily done by concatenating all those smaller files in the same order they were added to the index file, if the order is important.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 02:40:28 pm
@JLWest
Quote
This is probably going to take me a couple of days.
:o  Don't forget that the data file is mostly text, archived should be under 100MB. You can even send it to me via mail.

I'll zip and send it.

Oh, by the way, I looked at the VTV. What a nice object. but, the OP won't load the file into the VTV. I'll load the big file into the dynamic array then what I need (1 airport into the VTV and then the records for editing into the lower box of the VTV.

  Zipped it's 48,755KB. I'll try to send.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 03:06:26 pm
@ lucmar

Yes, the order is important. Lamair Research or X-Plane reads the file to verify integrity when you start the program. If it's wrong they give  a message of corrupted file and exit to the Desktop. In my case I always have a backup of X-Plane, but I think there is a way to download a new file from the site.

There are 35,000+ airports, Seaplane bases and Helipads. The only way I would feel comfortable with this approach is to beak each out in a separate file: F00001.txt, F00002.txt . . .F35187.txt.

And it's maybe the way to go. 
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 11, 2019, 03:17:40 pm
@JLWest
I received an empty mail, no attachment.

PS: If you have a gmail account you can add the file to the google drive then paste the link here.
Title: Re: Won't load Apt.Dat.
Post by: lucamar on January 11, 2019, 03:25:09 pm
There are 35,000+ airports, Seaplane bases and Helipads. The only way I would feel comfortable with this approach is to beak each out in a separate file: F00001.txt, F00002.txt . . .F35187.txt.

And it's maybe the way to go.

Yes, that's what I sugested: dividing the big file into small one airport/heli/seaplane base files, whatever the name. Then all you have to do is select the appropiate file to get all data for airport/heli/seaplane base and regenerating the big file is just a matter of concatenating those files.
Title: Re: Won't load Apt.Dat.
Post by: 440bx on January 11, 2019, 04:57:52 pm
Lamair Research or X-Plane reads the file to verify integrity when you start the program. If it's wrong they give  a message of corrupted file and exit to the Desktop.

<snip>

There are 35,000+ airports, Seaplane bases and Helipads. The only way I would feel comfortable with this approach is to beak each out in a separate file: F00001.txt, F00002.txt . . .F35187.txt.
Breaking the large file into separate files will make update and maintenance of your program rather cumbersome.  If you do that, every time there is a new version of the file, you'll have to break it into separate files for your program to use them.  That's a headache you don't want.

You should do it the way Lamair research is doing it, which most likely is, they map (CreateFile/CreateFileMapping/MapViewOfFile) the file, which is extremely quick, then scan it to ensure it is as required.  On your machine, scanning the file would most likely not even be noticeable.

If I knew of a Lazarus component that does that (map the file for you), I would suggest it but, I know next to nothing about what components are available, much less what they do.  Hopefully someone else can fill that gap.

In your case, your first step would be the same, create a file mapping of the file, the second step would be to scan the file to create an array of pointer indexes based on whatever information you want to show (the "filter" box in wp's example.)  This second step, you do it using the technique wp posted an example for in this thread, that is using a component such as VTV (which wp used) so you don't copy thousands (or more) strings into the component that displays them.

You can keep the code very simple if you scan the file twice.  The first scan to determine how many records/lines you need to keep track of, which would allow you to set the length of the dynamic array upfront before you proceed to load it with the pointers to the strings/lines.  The second scan to load the array.  That keeps memory management as simple as it gets.

The basic concept is very simple, you load the file into memory (mapping it is the fastest way also, memory efficient if you set it to PAGE_READONLY) then, create an index based on what the user (you in this case) wants to look at (as wp's example does.) 

On your machine, the above would happen in less than the blink of an eye.

HTH.

ETA:
correction: the component wp used is a ListView placed in virtual mode, not VTV.   See wp's post below for full detail.


Title: Re: Won't load Apt.Dat.
Post by: wp on January 11, 2019, 05:30:56 pm
using a component such as VTV (which wp used)
I think it's worth to put this right: I did not use VTV - which is hard to learn - but a plain old TListView. Using virtual mode in TListView is just three simple actions:
Title: Re: Won't load Apt.Dat.
Post by: 440bx on January 11, 2019, 06:05:32 pm
using a component such as VTV (which wp used)
I think it's worth to put this right: I did not use VTV - which is hard to learn - but a plain old TListView. Using virtual mode in TListView is just three simple actions:
  • Activate virtual mode (ListView.OwnerData := true)
  • Define how many items are contained in the List (ListView.Items.Count := <number>);
  • Write an event handler OnData to populate the TListItem passed as a parameter with the strings to be displayed
Thank you for clarifying that.  The comments about VTV after the example you posted left me with that mistaken impression.    Virtual mode is definitely the way to go which, I believe, was the point you were making with the example you posted.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 08:17:23 pm
@JLWest
I received an empty mail, no attachment.

PS: If you have a gmail account you can add the file to the google drive then paste the link here.
I couldn't figure out how to attach a file. So I tried to abandon the email.
The size of the file 48,755 KB using 7zip.

I sent it to  Handoko as I have his direct E-mail and I think it went thru. He said he could host.

Is there a way I could move the file to one of my empty drives (yes obviously to that), share the drive and publish something so you could download the file.

Right now I setlength to the 7.9 and I'm getting an error. I was loading the small data set 62 records. Need to go back and make sure.
 
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 11, 2019, 08:36:26 pm
I'm able to load the small data set 62 records into the array and display in a listbox.

So I'm doing the setlength in the wrong place I guess.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 12, 2019, 04:07:29 am
@JLWest
I received an empty mail, no attachment.

PS: If you have a gmail account you can add the file to the google drive then paste the link here.

I sorry I didn't see the notification. I was running down the Copyright of the file but think I have that to bed. I think I do have a gmail account. I have to look it up or get Google to give it to me. But they never throw anything away.


News: During the copyright flap I had the program running in the background and I think I loaded the 7.9 unwashed and unloved records. I can't setlength on the array in one chunk without getting an error. So I did it the way you wrote it. But I think it all loaded.

On to google
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 12, 2019, 05:47:44 am
https://drive.google.com/open?id=1u1g5NXJ-qjvb_eigt-9pjBaaYU3eDzza

The Apt.dat zip resides on google drive.
Title: Re: Won't load Apt.Dat.
Post by: wp on January 12, 2019, 07:06:27 am
My demo program of reply #30 loads and displays all records of apt.dat within 10 seconds. Just change the const FILENAME near line 50 to the path to apt.dat.
Title: Re: Won't load Apt.Dat.
Post by: balazsszekely on January 12, 2019, 09:14:17 am
The file is loaded in 2 seconds at my side.  :) On a slower computer some feedback(progress) to the user would be nice.  @JLWest please study @wp's code. It will load the data into a dynamic array, then display in a ListView. All you have to do is add the filter.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 12, 2019, 10:02:57 pm
My demo program of reply #30 loads and displays all records of apt.dat within 10 seconds. Just change the const FILENAME near line 50 to the path to apt.dat.

Oh that would be on the small data set. Yea.

will try #30 and #34.

As I said I'm actually loadind the big nail but not seting the array in one set.

Say what! Is the last record '99'? I got it loaded I think but not in 10 seconds.
I'll go back and incorporate #30. I think it must have taken 20 to 30 min on my machine. Maybe I need a tuneup.

Title: Re: Won't load Apt.Dat.
Post by: wp on January 12, 2019, 10:40:36 pm
Is the last record '99'?
The (0-based) index of the last record is 7,970,311 and the first number (I call it "ID") is 99.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 12, 2019, 10:42:43 pm
You loaded it.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 13, 2019, 10:10:16 pm
Modified code to run Example #34. Loads the 62 records set fine and shows the 99 record in a edit box.

Pointed it at the the 7.9 and. It gives the correct record count (BlockSize).

A few seconds after pressing the load button it lights the top right X (exit) red reports a not responding in the title bar a second or two later and I think out to lunch.

Guess what, I'll be. It was running while I was typing this. Went over to close thinking to make a final change. and I have the big nail loaded. 99 in the TEdit.text and the correct number of records reported in the label2.caption which match the Blocksize.

I don't know how long it took, a bit, 5 to 8 min maybe. Haven't been able to  make the timer work and need to research.

Now how to manage all that data edit and save.
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 14, 2019, 02:09:54 am
Well I bit the bullet and downloaded it,
Had an hr or two spare and came up with this, no special load routine just standard file i/o

If you click the R:\AIRDATA\apt.dat label at the top; you can point it to the location of the file.

It read in analyzes and the data.
You can select via country then city, or from a complete list.

I have done some filtering and checking for duplicates which effects the speed a little.

Once you have selected the airport, the airport details shoudl popup; as all this is in memory.
In the name of the airport details box at the end is the actual filelinenumber that any airport details would reside; so
you would then read the main apt.,dat file until you got there and then scan forward to the details you after.
I have not added this in, but should be easy enough.


I have not fully tested, just a test to see how complicated this actually is..

Also not the neatest code;

WIth the data loaded and everything poulate, should take a total of about 25MB.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 14, 2019, 03:47:21 am
@ Josh

Don't fully understand what your saying but I'll run it up and try and figure it out.


Thanks.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 14, 2019, 05:52:27 am
@Josh

Had a response typed in and got logged out and lost the typing. Hate that about this system. So here goes again.

Do you by any chance like a little color on you screens?

I can't tell whch file you are pointed at, but I tested it on my test file, worked fine and then the 9.7mil file. Half way thru it took a big hit on performance and I'm not sure if it loaded all the records.

The last record in a Apt.dat file is just '99'.

I'll add a counter and display the last record to a label1.caption to check.

In the lower maroon box there is a number at the top right 14,666. I think that may be the entries in the box which is the airports. Should be 35,000+.

You display City country which is a problem. Most large cities have several airports. Phoenix KPHX, Dear Valley  Busiest GA airport in the world, Gelipse field, Mesa, Chandler, Scottsdale. In order to sort the mess avaiation use ICAO Codes. Guarantee unique code by country. US code starts with a K or a P in Alaska. KPHX Phoenix, KLAS Los Vegas, PANC Anchorage. Canada with a 'C'  CYUL Monteral.

Where I'm loading all 9.7mil records in a dynamic array they are stored in an array of records.
Fline is the text record or text line. I added RCDID which is = to Copy2Space(Fline); and ICAO which is the 5th field in of the RCDID '1', airport, '16' seaplane base and '17' heliport. It is the ICAO code "KPHX'. Thinking it will make searching thre array of records easier. but it does create a redundancy of data. The other two important ones '1300' and '1301' are the ones I need to edit. All the rest are just baggage I have to carry.

 type
 PData = ^TData;
  TData = record
  FLine  : String;
  FIndex : Integer;
  RCDId  : String;
  //other data if needed
      end;
   
Unique way of setting the filename path. Took me a bit to figure out. Not a bad idea. Do you really have an R drive?

Thanks
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 14, 2019, 10:26:20 am
DO you only access by ACAO code?

as line 407400 in aptdat

I do not see ACAO code
1   1212 1 0 XLRBANE BANESTI PRAHOVA
1302 city BANESTI
1302 country ROMANIA
1302 datum_lat 45.068673
1302 datum_lon 25.806558
1302 state PRAHOVA
1302 transition_alt 3000
1302 flatten 1
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 14, 2019, 01:42:36 pm
Hi
Altered during lunch.

Modified code for just acao, visible.
Storing the 1300 and 1301 in memory, when you select a ACAO code, if it contains 1300/1301 then they appear in the grid, you can then edit in grid, the center memo keeps a track of what you have altered ( visually at the moment); but would reaaly be stored internally.

As ACAO is in Listbox, you can when selectiiing them just press the keys for KLAX; to save scrolling.

Yes I have an R drive it is used for 'Rubbish' stuff, that is never backed up and does not matter if it gets deleted.
My Projects are on P Drive
Lazarus Installs are on L Drive.

All seperate physical drives.
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 14, 2019, 02:16:21 pm
Just moded the upload.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 14, 2019, 05:55:59 pm
Yes we only access by ICAO. Although in X-Plane when you select an airport to start from  you can type out a city and sometimes get the right airport. But it's a poor search engine and usually comes back with nothing.

Here is the first record in the live real data file:



1    460 1 0 LELL Sabadell

The '1' is the record ID. Indicate a land based airport. (About 30 different  Record Id).

'460' I think is the average elevation of the airport. Other records give the elevation of the touchdown zone elevation of each runway.
 
'1' and '0' are what they call depreciated fields. Fancy term meaning not used for anything in X-Plane. Maybe in other applications. This file has been around for a long time and it is is my understanding it is used and adapted for use in many commercial applications. 

LELL is the IACO code for the airport at Sabadell.
It is always the fifth field in the record in the '1', '16' and '17' record.

Sabadell Name of City where Airport is located.

On the record you posted:
 
1   1212 1 0 XLRBANE BANESTI PRAHOVA

The ICAO committee has allowed legacy codes exceeding 4 characters which is the standard now. XLRBANE is the ICAO code. It is the 5th field in.

Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 14, 2019, 07:15:00 pm
It's actually very neat. I think I would add a TEdit and enter the ICAO code for the airport I want.

The lower two boxes are good. I would set focus on the 1300 record and the corresponding 1301 record should get focus also. Easy enough as you indexed them.
 
A double click on any 1300 would move both records (1300 and matching 1301) to a restructured blue box for editing. This part is a little tricky on how you would present it, put it back together.

Notice how a lot of the 1300 say 'HeavyjetsProps' that all computer generated. If you really go to KLAX and look there are a lot of gates that can't take a heavy Jet, A380, B747. That's what wrong with the data.And the 1301 are even less useful. C airline. Who? AAL DAL SWA, CAP BAW.
Title: Re: Won't load Apt.Dat.
Post by: Josh on January 15, 2019, 12:47:06 am
Hi

Added in an edit box that simulates the same as would do on the Listbox.

If you change the data in the grids at the bottom, it will list the changes in the memo field.

I have added in a SAVE DATA button that only appears when you have altered something.

It will then create a new file with the changes in it called apt_NEW.dat.

I have checked, and when I read in the new file, the changes are indeed there.

You should be able to create a 'form' based on the stringgrid pretty easy, you just need to extract the data from the grid line column 2, column 1 holds the line in the file. So its hould be easy enough to rebuild the line. If you examine the onediting done event of the grid, you will see how it stores the changes, it can hold upto 2000 line changes in one go.

Also added in the search for the 1,16,17 fields for icao codes.


Note I can only spend limited time on this, it shoudl hopefully give you some ideas though.
Its not the neatest code, and more checks should be added in for ranges and file i/o errors etc.
Title: Re: Won't load Apt.Dat.
Post by: JLWest on January 15, 2019, 01:27:53 am
You have done more than I would have expected. The changes I mentioned I intended to make. After all I need to understand the program, know how it works and how to make changes. I need to decide how to incorporate this and get a working product.

It's close, I think.

Thanks.
TinyPortal © 2005-2018