Recent

Author Topic: Synapse: get file date before download  (Read 4131 times)

Caravelle

  • New member
  • *
  • Posts: 46
Synapse: get file date before download
« on: January 11, 2014, 03:05:17 pm »
I am using the latest versions of Lazarus and FPC on Vista 32.

I have managed to write a working downloader using the Synapse httpsend function (it seems really counter-intuitive to use something with the word "send" in it to receive, but maybe it's just me !).  I need to download 4 or 5 files which are updated on the server once or sometimes twice a week.  They are not all necessarily updated at the same time, and the filenames do not change with a version change.  Currently, running my program sets off a long (15 minutes plus) process which can result in a set of files being downloaded some or all of which I may already have.

What I need is a way of reading the timestamps of the files which are (potentially) to be downloaded, so that my program can skip them if they have not been updated since the last download.  I know precisely what the names of the files are, there is no need to get a listing of all files available, but maybe  the answer is to obtain a full listing and parse it ?  In either case, I don't know how to read the timestamps while the files are still on the server - all I can do is read them after download and so avoid processing them any further if I have already done so.

I should be grateful for any help you can give.  Yes, I have tried looking at the Synapse documentation, and searching here, but I cannot see anything relevant.

[edit]  I have tried using AceFTP to visit the site where the files are held to no avail - it does not seem to respond to FTP.

Thanks

Caravelle
« Last Edit: January 11, 2014, 03:26:20 pm by Caravelle »

Caravelle

  • New member
  • *
  • Posts: 46
Re: Synapse: get file date before download
« Reply #1 on: January 17, 2014, 05:07:10 pm »
OK, is what I want to do impossible?   Or could it be done with something other than synapse ?

Thanks

Caravelle

taazz

  • Hero Member
  • *****
  • Posts: 5363
Re: Synapse: get file date before download
« Reply #2 on: January 17, 2014, 05:12:18 pm »
it is impossible http does not support file attributes. You can how ever add a second smaller text that has only a line or two that have the information you need eg version of the file or the date the update was created and download that instead of the complete download.

This is common practice.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

engkin

  • Hero Member
  • *****
  • Posts: 2237
Re: Synapse: get file date before download
« Reply #3 on: January 17, 2014, 06:11:37 pm »
Two HTTP header fields might help, one is in the request: If-Modified-Since, and the other is in the response: Last-Modified. You can learn more about them by reading: Header Field Definitions - RFC2616. You don't need to read it all, just read 14.25 If-Modified-Since and 14.29 Last-Modified. There is a summary on Wikipedia

Caravelle

  • New member
  • *
  • Posts: 46
Re: Synapse: get file date before download
« Reply #4 on: January 18, 2014, 05:11:47 pm »
Thanks to both.   

Taazz   Neither the files to download nor the website which hosts them are under my control.  I can't add any files. 

engkin  Oddly enough, I discovered what you say by chance yesterday - searching the net etc is really a matter of knowing what search terms to use and I finally hit on them.  I had been looking at http://forum.lazarus.freepascal.org/index.php/topic,21337.0.html and noticed that
Code: [Select]
if http.HTTPMethod('HEAD', fURL) was used to fetch the length of a file in order to set up a progress bar.   That led me on to explore HTTP headers...

I revised my own program to follow the same scheme as the one in that post, giving me a useful progress bar, and added a new bit after the code for obtaining the file length:

Code: [Select]
if http.HTTPMethod('HEAD', fURL) then
        begin
          for i := 0 to http.Headers.Count-1 do
          begin
            s := UpperCase(http.Headers[i]);
            if pos('CONTENT-LENGTH:', s) > 0 then
              TotalBytes := StrToIntDef(copy(s, pos(':', s) + 1, length(s)), 0);
              // Start of new code
              if pos('LAST-MODIFIED:', s) > 0 then
              LastModDate := copy(s, pos(':', s) + 1, length(s)), 0);    //LastModDate is a string
              Form1.memProgress.Lines.Append(s);  //memProgress is a temporary TMemo which reports what the program is doing as it goes along
              // End of new code
          end;
          http.Headers.Clear;
        end;

I'm hoping that the LAST-MODIFIED date will be enough.  My program uses an .ini file containing various more or less constant strings like the URL, filenames and paths which might conceivably change - I can edit the ini file a lot faster than rewriting the program.  I can store the LAST-MODIFIED date of each zip file as a simple string in the ini file, and only download the zip file again if checking the header as above reveals a non-matching date.  IF-MODIFIED-SINCE looks like it would work neatly but with the present state of my knowledge of the subject I'm not sure how I would persuade the server to return the info. And as I am already reading the header for the file length, I might as well stick to the devil I know.  Logically I should read LAST-MODIFIED first and bail out if I don't need to go any further. 

 STOP ! while typing this I have suddenly realised (doh!  :-[ ) that CONTENT-LENGTH would probably do the job - if the files have been changed at all they are almost certain to have grown.  More data is continually being added to the mdb files within the zips, that's precisely why we need to download updates.  Sometimes the obvious answer is staring you in the face...

Thank you again.

Caravelle