Subscribe to a web stream and listen to incoming streams

[p-samuel 2]

I have an url where I can listen to incoming streams whenever a new notification is published. It works perfectly using the cmd curl command but I can't reproduce the same in delphi. I tried using CreateProcess and read the output stream of the process but was worthless. Some has any idea how to make this?
 

curl -s ntfy.sh/something-very-strange/json

Each line is a response from the server. When a "GET" request is sent it opens a connection pool and doesn't closes it, sending packages streams to the subscriber,  just like a web-socket, until the subscriber ends the process.

My goal is to be able to connect and subscribe to this url through delphi, maybe using some component, process or any other tool which allows me to do so.

Edited November 11, 2022 by p-samuel
Grammar

[Fr0sT.Brutal 904]

Create socket, write to socket, read from socket.

What's your concrete problem?

[p-samuel 2]

Hello @Fr0sT.Brutalthanks for your interest.

This is not a websocket, it's a http url. I am trying to read the stream of responses when I send a GET request, but I'm afraid this is not possible to be done in Delphi by native components such as RestResquest or Indy:

In Python would sound like this:
 

resp = requests.get("https://ntfy.sh/something-very-strange/json", stream=True)
for line in resp.iter_lines():
  if line:
    print(line)

 

[Der schöne Günther 336]

In Delphi, you will usually take a StreamReader and use its ReadLine() method which will block.

[Fr0sT.Brutal 904]

1 hour ago, p-samuel said:

I'm afraid this is not possible to be done in Delphi by native components such as RestResquest or Indy:

Rest won't help here but there are THTTPClient, Indy, not talking about many 3rd party libs. At last you can use TNetClient and send manually crafted request

Edited November 11, 2022 by Fr0sT.Brutal

[Vincent Gsell 14]

Warning : Very "quick and dirty as a "One min programming challenge ";)

 

 

 

-> joke appart :

- you can mimic the pythonic "resp.iter_line" : Just wrap in a TThread descendant all what is done in screen capture. (Put the http and idssl component in the TThread descendant)

- And on each "get stream" (onWork summoning), qualify the "new line", and send it on your main thread. 

- Avoid use of AnonymousThread here, it was just to make it quicky'n dirty.

Edited November 11, 2022 by Vincent Gsell

[p-samuel 2]

@Vincent Gsell This worked perfectly! I just needed to make some adjustments but it worked just like a glove! Thanks by heart!

[Remy Lebeau 1636]

8 hours ago, p-samuel said:

This is not a websocket, it's a http url. I am trying to read the stream of responses when I send a GET request, but I'm afraid this is not possible to be done in Delphi by native components such as RestResquest or Indy:

It would help if you could show the raw HTTP response data that is actually being transmitted.  curl is only outputting the JSON data, not the full HTTP data (you can enable its verbose output to see that).  Streaming events like you have described could be transmitted in many different formats.

 

Depending on the format used, using Indy's TIdHTTP component, for instance, you might be able to either:

 

- use its OnChunkReceived event

 

- enable its hoNoReadMultipartMIME or hoNoReadChunked option and then read the lines manually from Indy's socket directly

 

- use a TIdEventStream as the target to receive the data, using its OnWrite event to access the lines.

 

It is hard to say for sure without more information.

[Remy Lebeau 1636]

4 hours ago, Vincent Gsell said:

- you can mimic the pythonic "resp.iter_line" : Just wrap in a TThread descendant all what is done in screen capture. (Put the http and idssl component in the TThread descendant)

- And on each "get stream" (onWork summoning), qualify the "new line", and send it on your main thread. 

- Avoid use of AnonymousThread here, it was just to make it quicky'n dirty.

Since the data is streaming, there is no guarantee that the data that is available during each firing of the TIdHTTP.OnWork event will represent a complete line of data.  It could be partial data.  At the very least, I would not suggest using the Memo's LoadFromStream() method, which replaces the entire contents of the Memo on each call.  It would be better to convert each event's data to a raw string and then append it to (not replace) the Memo's current text, which can be done efficiently using its SelStart/SelLength/SelText properties.

 

But, more importantly, the TMemoryStream that you are having TIdHTTP write to will just keep receiving more and more data that you are never discarding from memory, so the stream's memory usage will keep growing over time, until memory runs out eventually.  If you want to process the data in chunks as it is arriving, I would suggest using a TIdEventStream instead, replacing the TIdHTTP.OnWork event handler with a TIdEventStream.OnWrite event handler instead.  You will be given access to the actual bytes per event as a dynamic array, which means you can simply increment the array's refcount, send the array pointer to the main thread, and then decrement the refcount.

[p-samuel 2]

@Remy Lebeau Here is a snapshot of the  response in verbose. I just omitted some sensitive information.

[Remy Lebeau 1636]

18 minutes ago, p-samuel said:

@Remy Lebeau Here is a snapshot of the  response in verbose. I just omitted some sensitive information.

That is only part of the response, and not even the relevant part, eihter.  That snippet shows that you are requesting an unencrypted HTTP url and being redirected to an encrypted HTTPS url, but it is not showing the contents of the subsequent HTTPS request or its response, which contains the actual event data.

Edited November 11, 2022 by Remy Lebeau

[p-samuel 2]

@Remy Lebeau, I don't know any other way of debugging subsequent requests and displaying verbose information apart from using -v in curl so far. @Vincent Gsell's answer helped me on giving me a north where should I go, but I'll also try taking a deep look on the information that you shared.  I really appreciate the help of you all. Delphi is a quite hard language and documentation concerning these components is quite hard to find.

Thank you all guys!

[p-samuel 2]

@Remy Lebeau, debugging is quite simple using chrome's dev tools. As you have said before, it seems a better option using TIdEventStream as the data itself is transmitted by events, like a websocket.



It's a plug and play. I didn't know Indy was so easy to setup this. 

type
  TNotityProviderIndy = class
  private
    FIOHandlerSSL: TIdSSLIOHandlerSocketOpenSSL;
    FIdHTTP: TIdHTTP;
    FIdEventStream: TIdEventStream;
    procedure OnWriteEvent(const ABuffer: TIdBytes; AOffset, ACount: Longint; var VResult: Longint);
  public
    constructor Create;
  end;

implementation

{ TNotityProviderIndy }

constructor TNotityProviderIndy.Create;
begin
  FIdHTTP := TIdHTTP.Create(nil);
  FIOHandlerSSL := TIdSSLIOHandlerSocketOpenSSL.Create;
  FIdEventStream := TIdEventStream.Create;

  FIOHandlerSSL.SSLOptions.Method := sslvTLSv1_2;
  FIdHTTP.IOHandler := FIOHandlerSSL;
  FIdHTTP.Request.Accept := 'text/event-stream';
  FIdHTTP.Request.CacheControl := 'no-store';
  FIdHTTP.HTTPOptions := [hoNoReadMultipartMIME, hoNoReadChunked];
  FIdEventStream.OnWrite := OnWriteEvent;

end;

procedure TNotityProviderIndy.OnWriteEvent(const ABuffer: TIdBytes; AOffset, ACount: Longint; var VResult: Longint);
begin
  Writeln(IndyTextEncoding_UTF8.GetString(ABuffer));
end;

Thanks for the info!
 

Edited November 14, 2022 by p-samuel

[Remy Lebeau 1636]

18 hours ago, p-samuel said:

@Remy Lebeau, debugging is quite simple using chrome's dev tools. As you have said before, it seems a better option using TIdEventStream as the data itself is transmitted by events, like a websocket.

The screenshot you provided appears to be an actual websocket event.

Quote

It's a plug and play. I didn't know Indy was so easy to setup this. 

'text/event-stream' is just a stream of linebreak-delimited events.  TIdHTTP is not currently designed to handle that kind of stream.  Using TIdEventStream is going to give you arbitrary blocks of data as they are read off the socket, which you are assuming are complete events, but that is not guaranteed (or even likely) to happen. To be safe, you would have to buffer the stream data, break it up on linebreaks as they become available, and decode only complete events as UTF-8.

 

The code you have presented has the potential to decode incomplete events.

 

In this type of situation, it would be useful if TIdHTTP had a flag available to tell it not to read the response body at all and just exit, letting the caller then read the body from the IOHandler directly as needed.  Similar to the existing hoNoReadMultipartMIME and hoNoReadChunked flags, but for other kinds of responses.  Such a flag does not exist at this time.

Edited November 14, 2022 by Remy Lebeau

[Fr0sT.Brutal 904]

10 hours ago, Remy Lebeau said:

'text/event-stream' is just a stream of linebreak-delimited events.  TIdHTTP is not currently designed to handle that kind of stream

The actual stream is still not shown to us but there's chunked encoding header so probably the response is valid HTTP chunked stream which Indy should handle without issues

[Remy Lebeau 1636]

8 hours ago, Fr0sT.Brutal said:

The actual stream is still not shown to us but there's chunked encoding header so probably the response is valid HTTP chunked stream which Indy should handle without issues

If that is the case, if each event is its own HTTP chunk, then you can use the TIdHTTP.OnChunkReceived event to handle the stream events.  At the moment, you still have to provide a TStream in order for the OnChunkReceived event to fire (I may change that in the future), but you can use TIdEventStream with no event hanlders assigned so that the chunks written to the stream are ignored.