Recent

Author Topic: TIOStream, TXMLInputSource wait if input is empty  (Read 3470 times)

nurettin

  • New member
  • *
  • Posts: 8
TIOStream, TXMLInputSource wait if input is empty
« on: September 24, 2017, 06:29:52 pm »
So I have some message passing between a main process and subprocess for a win/linux/mac app;

Subprocess reads xml from stdin, validates it using dtd and outputs some data to stdout or stderr like this;

Code: Pascal  [Select][+][-]
  1.     SSTDIN := TIOStream.Create(iosInput);
  2.     xmlsrc := TXMLInputSource.Create(SSTDIN);
  3.     xmlparser := TDOMParser.Create;
  4.     xmlparser.Options.Validate := True;
  5.  
  6.     running := true;
  7.     while running do
  8.     begin
  9.       xmlparser.OnError := @(command.OnError);
  10.       xmlparser.Parse(xmlsrc, xml);
  11.       // running := not xml says stop
  12.     end;
  13.  

When TIOStream finishes reading the first xml packet from pipe, everything parses OK. But then it doesn't stop there and tries to read from stdin again, and that causes a parse error because stdin is empty.

How do I make TIOStream block when there is no more data? What is a canonical way of doing this?
« Last Edit: September 24, 2017, 06:40:37 pm by nurettin »

molly

  • Hero Member
  • *****
  • Posts: 2330
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #1 on: September 24, 2017, 06:42:54 pm »
I've not checked but since IOStream is just another stream, can't you use the streams position and size to determine if the IOstream still contains data that requires parsing or not ?

nurettin

  • New member
  • *
  • Posts: 8
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #2 on: September 24, 2017, 07:36:38 pm »
@molly: I can, sure, but then how do I wait on input? Polling?

molly

  • Hero Member
  • *****
  • Posts: 2330
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #3 on: September 24, 2017, 07:50:25 pm »
Oh, wait. I did not read the first part of your question correctly  :-[

Not all data is transfered in one go ?

In that case i think there is not a good solution when using this approach, although it would be welcome if i could be proven wrong.

How do you 'launch' the other process btw ? because if you you use tprocess then you could probably peek at how output is being processed (and try to adapt that for your input channel), or use the same approach.

As another question: are the other processes also written by you (in pascal ?), because in that case you could perhaps get away with sending a eof between different runs or close and re-open the channel.

Usually you would use IPC to communicate between different processes.


Ok, for historical reason i've just striked the above, because i'm really not reading correctly. sorry about that.

Have you tried with a TBufStream ? It offers a buffer capacity, bufferpos and buffersize. In theory that could perhaps be a solution to your problem.

fwiw: i'm having a hard time reproducing a program that outputs things in chunks as per your conditions so that input would dry out momentarily. are you perhaps piping ?
« Last Edit: September 24, 2017, 08:18:36 pm by molly »

nurettin

  • New member
  • *
  • Posts: 8
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #4 on: September 24, 2017, 08:16:44 pm »
The problem is me. I'm super new, have limited understanding of everything and shouldn't be coding at all but lazarus is awesome so I have to use it and there seems to be a lot of ways of doing one thing. (SimpleIPCServer, Parse STDIN, sockets, indy, ...)

At the moment I'm just testing subprocess through named pipes like this:

Code: Pascal  [Select][+][-]
  1. mkfifo /tmp/somepipe
  2. cat /tmp/somepipe | ./subprocess
  3. subprocess waits... (how?)
  4. cat some.xml | /tmp/somepipe
  5. subprocess awakens!, parses correctly, tries to parse from empty stream again! (why??)
  6.  

I want to transfer large amounts of data, sometimes gigabytes through the pipe back and forth, so from past experience I just thought pipes are super efficient. At least on linux and mac.


molly

  • Hero Member
  • *****
  • Posts: 2330
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #5 on: September 24, 2017, 08:19:38 pm »
Ah, piping it is ! that answered one of my questions   :)

In that case, have a look at unit pipes with all it's classes and functionality.

PS: in case seeking cross-platform compatibility then the link i posted earlier to processing command output seems to be better suited. Linux units have functionality to work with named pipes but is ofc not cross-platform. According to the mailing list (long time ago, so things might have changed), using named pipes can be problematic.
« Last Edit: September 24, 2017, 08:40:39 pm by molly »

nurettin

  • New member
  • *
  • Posts: 8
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #6 on: September 25, 2017, 07:09:44 pm »
XMLInputSource will not work with TBufStream. It throws an exception saying reading from TBufstream is not supported. That's my real problem. I will have to do the buffering myself or do some sort of loop sleep(100) to keep the child process waiting for non-empty input if I want to pipe intermittent xml files between two processes.

molly

  • Hero Member
  • *****
  • Posts: 2330
Re: TIOStream, TXMLInputSource wait if input is empty
« Reply #7 on: September 29, 2017, 10:20:50 am »
Sorry, my bad. I should have been more explicit. Try TReadBufStream. TBufStream is the ancestor class.
Code: [Select]
  Buffer : TReadBufStream; // Buffer declaration
  Buffer := TReadBufStream.Create(SSTDIN, 10);  // 10 bytes buffer
  xmlsrc := TXMLInputSource.Create(Buffer);

But... that won't probably help you out either as it only solves the part that the 'reader' isn't thrown off when no output is generated (or takes a long time).

You would not be able to distinguish between two files because afaik the only way to do that would be to close the output en re-open it again (on the program that feeds the pipe/reader, and yes that seems to work), but closing the output would be your only way of being able to tell that the process ended. Unfortunately the parser also does not react to any control characters or provides any other means to stop parsing after last closing tag so it will happily parse along and detect the second xml tag (and throw an exception because of that).

afaik that leaves one other option open, but it will be slow. Read the buffer into a temporary stream and analyze the contents to be able to distinguish between different files and create a new parser (after having destroyed the old one) when a new files has 'arrived' and feed the new parser this buffer.
« Last Edit: September 29, 2017, 11:39:49 am by molly »

 

TinyPortal © 2005-2018