Recent

Author Topic: Simple file with only one single record, but of HUGE size  (Read 9842 times)

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Simple file with only one single record, but of HUGE size
« Reply #30 on: December 14, 2018, 07:30:47 pm »
The same test with an even older machine, a Toshiba Satellite A60.
OS: Ubuntu 11.04
CPU: Pentium IV - 3.0 Ghz
Mem: 512 Mb

Output:
Code: [Select]
Creating a file of 1000000000 bytes...
Test 1 (loop and random data only): 0:20.937
Test 2 (block): 0:34.086
Test 3 (single bytes): 1:20.634

The older the machine, the faster Test 3 goes !? >:D
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

Handoko

  • Hero Member
  • *****
  • Posts: 5130
  • My goal: build my own game engine using Lazarus
Re: Simple file with only one single record, but of HUGE size
« Reply #31 on: December 14, 2018, 07:38:29 pm »
The time of Test 3 differs greatly I think because it depends more on hard drive performance and how badly fragmented the content is. In theory, Test 3 performs 1000 millions disk operations (move the disk head and write the data). Test 2 performs only 100 disk operations (because the buffer size is 10 millions).

The same test with an even older machine, a Toshiba Satellite A60.
OS: Ubuntu 11.04
CPU: Pentium IV - 3.0 Ghz
Mem: 512 Mb

Output:
Code: [Select]
Creating a file of 1000000000 bytes...
Test 1 (loop and random data only): 0:20.937
Test 2 (block): 0:34.086
Test 3 (single bytes): 1:20.634

The older the machine, the faster Test 3 goes !? >:D

AFAIK, Linux file system is less defragment-prone.
« Last Edit: December 14, 2018, 07:51:20 pm by Handoko »

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Simple file with only one single record, but of HUGE size
« Reply #32 on: December 15, 2018, 02:26:23 am »
In theory, Test 3 performs 1000 millions disk operations (move the disk head and write the data).
It has an internal memory specifically to prevent this crisis. A similar cache/buffer is also provided by the OS to prevent a slowdown caused by transferring data to/from the HD.

Handoko

  • Hero Member
  • *****
  • Posts: 5130
  • My goal: build my own game engine using Lazarus
Re: Simple file with only one single record, but of HUGE size
« Reply #33 on: December 15, 2018, 03:00:10 am »
Yes, that's why Test 3 is not 10 millions times slower than Test 2. OS and hard drive have their own cache but still it is better if the application can have its own buffer.

garlar27

  • Hero Member
  • *****
  • Posts: 652
Re: Simple file with only one single record, but of HUGE size
« Reply #34 on: February 05, 2019, 11:31:36 pm »
At the moment all is at a the state of planning, the machine for that setup is not unpacked yet. Well, after setting it up and after the planned config has gone through some minor tests (the "final" filesize may be huge, but nevertheless at first I have to know that the underlying software is correctly running), after all that and after the first produced files filled with real data, then I'll drop a note here.
I can't promise a fixed date, but it may be in the first half of January.

@Epand, how did it go? I assume you must be very busy...

Epand

  • New Member
  • *
  • Posts: 25
Re: Simple file with only one single record, but of HUGE size
« Reply #35 on: February 06, 2019, 10:56:02 am »
@Epand, how did it go? I assume you must be very busy...
Hello garlar27
Hello to other posters in this thread

Please rest assured that I did not forget to post whether this project is making progress.
The data which will fill the huge file, these data are provided by a numeric algorithm/procedure I am just working on. I would not define that numeric algorithm as a problem, but the point is, that its computation by a single-thread program  takes a very LONG time before the algorithm reaches a certain limit - reaching the algorithms goal is necessary for filling the discussed file.

Therefore I want to port this numeric part to multi-threading before the rest of this project goes ahead; if the numerical part - the computation of its mathematical formula - takes too long then filling the ensuing file would also take more time, the latter I have to avoid. The porting to multi-threading is a bit tricky because of the mathematical formula to be multi-threaded.

The whole project consists of two interrelated parts, step1 is programming the multi-threading of the underlying mathematical formula, step2 is filling the discussed file.
At the moment I'm still at step1, but I do not forget to drop a note if step2 is under way and running.

Have a nice time
Michael
 
« Last Edit: February 06, 2019, 02:05:33 pm by Epand »

garlar27

  • Hero Member
  • *****
  • Posts: 652
Re: Simple file with only one single record, but of HUGE size
« Reply #36 on: February 06, 2019, 05:28:43 pm »
Is good to hear that your project keep going.

Well, it looks like you have two nice projects in your hands.

Enjoy them!!!

By the way, once I heard a colonel say on a documentary "every plan is perfect until the first shot" and I thought in system programming sometimes we have the same problem "everything's perfect until you start coding"  :D

Good luck!!

 

TinyPortal © 2005-2018