Recent

Author Topic: Internet Tools  (Read 65387 times)

RomanMoz

  • Newbie
  • Posts: 6
Re: Internet Tools, now with more http
« Reply #60 on: August 23, 2017, 03:16:47 pm »
Thank you!

But really I don't know how to implement this into pascal code  :(

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Internet Tools, now with more http
« Reply #61 on: August 24, 2017, 10:09:44 am »
Docs and Wiki show json parsing. Forum also shows a lot after entering 'json array' into search.
http://wiki.lazarus.freepascal.org/fcl-json
http://lazplanet.blogspot.rs/2014/09/a-simple-json-parsing-example.html
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Internet Tools, now with more http
« Reply #62 on: August 24, 2017, 02:48:55 pm »
Thank you!

But really I don't know how to implement this into pascal code  :(

That xidelcgi page is pretty much the same as  process function in simpleinternet.

Code: [Select]
process(input, query).toJoinedString
for .. in .. to iterate over the values

RomanMoz

  • Newbie
  • Posts: 6
Re: Internet Tools
« Reply #63 on: August 24, 2017, 10:21:12 pm »
Thanks a lot!

Now it solved...

ArtLogi

  • Full Member
  • ***
  • Posts: 184
Re: Internet Tools
« Reply #64 on: September 30, 2017, 01:55:18 am »
Hmm... I'm trying to get this working with Laz 1.6 and FPC3.0.0.0 on/to Win7, I downloaded the Internettools.zip from homepage and have been trying to include the package to my project all imaginable ways, but it seems always throw some errors while building the package itself.

....Those red IF_DEF swamps at the beginning of every .pas mainly seems to throw red.

If I try to compile example 1 from: http://www.benibela.de/documentation/internettools/
Code: Pascal  [Select][+][-]
  1. program TEST;
  2.        uses xquery;
  3.  
  4. begin
  5.   writeln(query('doc("http://example.org")//title').toString);
  6. end.  
Which all files from ZIP I should include to the same folder as the program TEST.pas is.  xquery.pas and... simpleinternet.pas and ... ... ..

..Or is there a "quaranteed" stable version, the funtionality I need is almost 7-bit ASCII web scrabing of few tens of lines per a day. What I understood these IFDEFs are mainly for UTF-8 quirks??
« Last Edit: September 30, 2017, 02:23:15 am by ArtLogi »
While Record is a drawer and method is a clerk, when both are combined to same space it forms an concept of office, which is alias for a great suffering.

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Internet Tools
« Reply #65 on: October 04, 2017, 08:00:19 pm »
Hmm... I'm trying to get this working with Laz 1.6 and FPC3.0.0.0 on/to Win7, I downloaded the Internettools.zip from homepage and have been trying to include the package to my project all imaginable ways, but it seems always throw some errors while building the package itself.

....Those red IF_DEF swamps at the beginning of every .pas mainly seems to throw red.

What kind of errors?

Which all files from ZIP I should include to the same folder as the program TEST.pas is.  xquery.pas and... simpleinternet.pas and ... ... ..

Pretty much everything from the data and internet subdirectory. And the FLRE files, unless you disable that ifdef

..Or is there a "quaranteed" stable version, the funtionality I need is almost 7-bit ASCII web scrabing of few tens of lines per a day. What I understood these IFDEFs are mainly for UTF-8 quirks??

The zip file should be stable.

The ifdefs are mostly for backward compatibility, because I used different utf8/regex libraries. 

For fpc 3 I am merging the utf8 and non-utf8 packages, so that there will only be one version

RomanMoz

  • Newbie
  • Posts: 6
Re: Internet Tools
« Reply #66 on: January 11, 2018, 03:05:11 pm »
Good day!

How to add Http headers in httpRequest post-method query? Is any the same like Headers.Add in synapse?

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Internet Tools
« Reply #67 on: January 12, 2018, 01:33:13 am »
Good day!

How to add Http headers in httpRequest post-method query? Is any the same like Headers.Add in synapse?

Yes, defaultInternet.additionalHeaders.add

RomanMoz

  • Newbie
  • Posts: 6
Re: Internet Tools
« Reply #68 on: January 12, 2018, 09:32:54 am »
Thanks
Works fine

M.A.R.C.

  • Jr. Member
  • **
  • Posts: 68
Re: Internet Tools
« Reply #69 on: March 27, 2018, 07:46:21 am »
Hi @BeniBela,

I'm using InternetTools for download some files from a server, sometimes the server is very slow and the program stops responding to close command.

I want to know if there is some way to stop a retrieve process. I use strSaveToFile in a Thread.Execute:

Code: Pascal  [Select][+][-]
  1.   if (not Terminated) then
  2.     try
  3.       strSaveToFile(FFileDestiny, retrieve(FUrlSource));
  4.     finally
  5.       // Free Memory from InternetTools
  6.       freeThreadVars;
  7.     end;

Also, there is some way to get the progress of the download process?

Thank you.

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Internet Tools
« Reply #70 on: March 27, 2018, 11:59:54 pm »
There is the defaultInternet.onProgress event to monitor the progress.

But there nothing to stop the download. Perhaps you can raise an exception from the progress event




TRon

  • Hero Member
  • *****
  • Posts: 2435
Re: Internet Tools
« Reply #71 on: June 27, 2020, 08:52:25 pm »
Downloaded the sources from https://packages.lazarus-ide.org/. Working as expected except for:

Code: Pascal  [Select][+][-]
  1. program internet_tools_test2;
  2. {$MODE OBJFPC}{$H+}
  3. uses
  4.   xquery; // , simpleinternet, simplehtmltreeparser;
  5. begin
  6.   WriteLn('hello');
  7. end.
  8.  
Which produces the following trace:
Code: [Select]
hello
Heap dump by heaptrc unit of /home/tron/projects/12345/internet_tools_test2
10116 memory blocks allocated : 2265956/2281112
9999 memory blocks freed     : 2262624/2277656
117 unfreed memory blocks : 3332
True heap size : 753664
True free heap : 737536
Should be : 738976
Call trace for block $76F8BA00 size 16
  $001011E4  TNAMESPACE__MAKE,  line 934 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F90B80 size 28
  $001011C0  TNAMESPACE__MAKE,  line 933 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F8B980 size 16
  $00100C08  TNAMESPACECACHE__CREATE,  line 887 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $001010B4  NAMESPACECACHE,  line 919 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $00101178  TNAMESPACE__MAKE,  line 930 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F8B900 size 16
  $00100C08  TNAMESPACECACHE__CREATE,  line 887 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $001010B4  NAMESPACECACHE,  line 919 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $00101178  TNAMESPACE__MAKE,  line 930 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F90AF0 size 36
  $00100C08  TNAMESPACECACHE__CREATE,  line 887 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $001010B4  NAMESPACECACHE,  line 919 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $00101178  TNAMESPACE__MAKE,  line 930 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F8B880 size 24
  $00100C08  TNAMESPACECACHE__CREATE,  line 887 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $001010B4  NAMESPACECACHE,  line 919 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $00101178  TNAMESPACE__MAKE,  line 930 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F8B800 size 24
  $001A2070  TFLRECACHEHASHMAP__RESIZE,  line 21183 of /apps/fpc/3.2.0/unit3/BeBe/FLRE.pas
  $001A1A08  TFLRECACHEHASHMAP__CREATE,  line 21105 of /apps/fpc/3.2.0/unit3/BeBe/FLRE.pas
  $000FDFA8  TXQHASHMAPSTROWNING$2$CRC42249A36__CREATE,  line 566 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $00100C08  TNAMESPACECACHE__CREATE,  line 887 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $001010B4  NAMESPACECACHE,  line 919 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $00101178  TNAMESPACE__MAKE,  line 930 of /apps/fpc/3.2.0/unit3/BeBe/simplehtmltreeparser.pas
  $000A9B30  XQUERY_$$_init$,  line 8420 of /apps/fpc/3.2.0/unit3/BeBe/xquery.pas
  $00032880  fpc_initializeunits,  line 969 of /home/Pierre/pas/release-build/release_3_2_0/fpcsrc/rtl/inc/system.inc
Call trace for block $76F8B780 size 24
  $001A2044  TFLRECACHEHASHMAP__RESIZE,  line 21182 of /apps/fpc/3.2.0/unit3/BeBe/FLRE.pas
^C
As you can see it goes on for quite a while, so I stopped it.

I'm am quite possibly doing something wrong or out of the ordinary there but I have no idea what or how to solve.
« Last Edit: June 27, 2020, 09:45:30 pm by TRon »

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Internet Tools
« Reply #72 on: June 27, 2020, 11:42:26 pm »
hello,
what is your O.S, Lazarus and fpc versions ?
you can  have a look here  (fpc 3.0.4)

Friendly, J.P
« Last Edit: June 27, 2020, 11:45:20 pm by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

TRon

  • Hero Member
  • *****
  • Posts: 2435
Re: Internet Tools
« Reply #73 on: June 28, 2020, 12:10:44 am »
hello,
Hi Jurassic Pork. Thank you for your reply regarding this (particular) matter.

Quote
what is your O.S, Lazarus and fpc versions ?
OS is Raspbian (Debian buster derivative, 32-bit),

Lazarus version is in theory non-existent (i am using FPC command-line compiler) but I have 2.0, 2.0.8 as well as a  trunk version installed.

FPC used for compilation is v3.2.0 (issue also shows itself with v3.0.4, although the exact trace might differ). I haven't tried v3.0 or v2.6.4 yet as these old versions (although installed) would work a bit counterproductive for me atm)

Quote
you can  have a look here  (fpc 3.0.4)
Thank you for the link.

Only that thread seems to discuss installing internettools by means of using Lazarus. I have no such intention :)

fwiw internettools works splendid for me and I am able to do all kind of neat things with it.

The snippet that I posted is the most basic example I was able to come up with that is able to show the issue.

I just seem unable to get rid of the memory leaks.

edit: my internettoolsconfig.inc looks like:
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. //Here you can choose which regex library to use
  4. {$if not defined(USE_SOROKINS_REGEX) and not defined(USE_SOROKINS_DREGEX) and not defined(USE_FLRE) and not defined(USE_FLRE_WITH_CACHE)}
  5. {.$DEFINE USE_SOROKINS_REGEX}         //Sorokins' regex library included in FPC. It is quite reliable
  6. //{$DEFINE USE_SOROKINS_DREGEX}        //my copy of Sorokins' library, for older FPC versions that did not include it
  7. //{$DEFINE USE_FLRE}                   //Bero's "Fast Light Regular Expressions" library. It is required for proper UTF-8 support in regex
  8. {$DEFINE USE_FLRE_WITH_CACHE}          //As above, but parse every regex only once
  9. {$endif}
  10.  
  11.  
  12.  
  13. {$IFDEF USE_FLRE_WITH_CACHE}{$DEFINE USE_FLRE}{$ENDIF}
  14.  
  15.  
  16.  
  17.  
  18.  
  19. //internal flags
  20. {$mode objfpc} {$H+}
  21. {$modeswitch advancedrecords}
  22.  
« Last Edit: June 28, 2020, 12:24:02 am by TRon »

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Internet Tools
« Reply #74 on: June 28, 2020, 06:58:25 pm »
Downloaded the sources from https://packages.lazarus-ide.org/. Working as expected except for:

That is an outdated version. I do not know who put it on packages.lazarus-ide.org or how to update it there.

I have not updated the zip file on my homepage for a while either. So currently, it is the best to use the source repository on github or hg.benibela.de/internettools/


 

TinyPortal © 2005-2018