Angus Robertson

BTW, the Json you receive from a websocket or by HTTP may not contain any line endings, so using ReceiveStr is a bad idea, it may not return complete lines, that might explain why your json was corrupted if there was no line ending at the end.  That is why the HTTP component receives content into a stream, and that stream is passed to IcsHtmlToStr to convert into something Delphi can handle.

Angus

For HTTP, the headers contain the content type and character set so it's easy to convert to a string, including entities like £, look at TSslHttpRest.TriggerRequestDone2 in OverbyteIcsSslHttpRest.pas:

// convert response to correct codepage, including entities
   if (Pos ('text/', FContentType) = 1) or  (Pos ('json', FContentType) <> 0) or  (Pos ('javascript', FContentType) <> 0) or  (Pos ('xml', FContentType) <> 0) then begin
         FResponseRaw := IcsHtmlToStr(FResponseStream, FContentType, true);

  end;

But I have no idea if websockets provides the content type, and you've not said if you are using Unicode.

Angus

TSSLWsocket receives a stream of bytes, perhaps you are not correctly converting them into a string, might be UTF8, UTF16, ANSI or anything really.  No idea what websockets specifies.  Are you using a unicode or ANSI Delphi, the websockets project was ANSI only.

I'm not aware of any up to date websocket libraries for ICS. 

Angus

This is avaiIable in the OverbyteIcsSslJose unit:

{ digests and hashes - note all digests are binary bytes in AnsiStrings }
function IcsHMACDigest(const Data, Key: AnsiString;   HashDigest: TEvpDigest = Digest_sha256): AnsiString;
function IcsHMACDigestEx(const Data, Key: AnsiString;  HashDigest: TEvpDigest = Digest_sha256): AnsiString;
function IcsHMACDigestVerify(const Data, Key, OldDigest: AnsiString; HashDigest: TEvpDigest = Digest_sha256): Boolean;
 

With various other functions used for Json Object Signing and Encryption,

Angus

I don't see any problem with your proposed change, but don't see any great benefit either, the connection might close without OnDataAvailable being called so you still need processing in OnSessionClosed.  Or am I missing something?

Angus

Exactly like that thread from a couple of years ago.  However the real issue here is it was not necessary until Windows 1903, and Windows internet applications that worked before may now fail.

Angus

Just open a TCP connection to the server and you should get back a one line reply, that's all you need to see if it's online.  But it might be off-line by the time you send email.  And it might not accept email from you.  The ICS library now has a mail queue component that allows multiple email servers to be specified, and all those servers are attempted if any fail. 

Angus

Resorted to regedit to disable the MpsSvc service, still started when set to manual, and now my applications are listening on all their strange ports again OK. 

So the Microsoft firewall is now running continually even when turned off according to all the dialogs.   Will need to warn my customers.

Angus
 

Since I updated to Windows 10 1903 three weeks ago, some of my servers that use unusual ports have failed to start saying 'Another server is already listening on 192.168.1.124:1777' and similar.

There does not seem to be a problem listening on common ports, 21, 25, 80, 443, etc, but not 2100, 4443 etc. 

I have always had Windows firewall disabled since this is a development PC and I have lots of internet applications being developed. 

All the firewall dialogs I can find are red warning firewall is disabled.  But the Windows Defender Firewall service is running and can not be stopped, the options are greyed. 

I do actually have firewall rules set-up for some of the offending applications to listen on all ports.

So is nanny Microsoft fighting my choice to not use a firewall and running it anyway? 

I could probably change the registry setting to stop it running, but Microsoft might check that and start it anyway.

Anyone have silly problems on Windows 10 1903?

Angus
 

No, header lines will not be ignored or lost since they all end with CRLF, and a double CRLF at the end of the header.  Headers are small, and need to parsed as strings.

To receive a POST body, that function calls an event where you can accept data using Receive and write it to a stream if expecting a large upload or just process some Json or whatever. 

Angus

Thanks both, the words 'profile photo' appearing anywhere on those pages would have helped, instead of expecting me to hover over icons.  

For those interested, the drawing was done by Steve Bell 40 years ago, he was political cartoonist for The Guardian newspaper for many years, and as a journalist at the time I commissioned an A3 cartoon of a pop concert not realising that my staff and I were going to appear in it.

Angus

I've uploaded a 263x304 thumbnail photo to my profile, but I still see A appearing against my posts.  The photo, actually a drawing,  just appears cropped and massively stretched across the whole screen at the top. 

I'm missing something very obvious...

Angus

Why bother with torrents when you can download the official version from your local CDN node at 60 to 100Mbps, took a few minutes for 7GB.

http://altd.embarcadero.com/download/radstudio/10.3/delphicbuilder10_3_2_496593b.iso

MD5: 43FA7A32D9ECBB8CF14AF9519AFE066A

Angus

I've just updated SVN and the overnight zip so that ICS will build properly again, tested against Delphi Rio 10.3 Update 2, aka 10.3.2, no changes were needed. 

There are some minor fixes from this forum, with a lot of other pending units waiting mostly relating to automatic ordering of Let's Encrypt SSL certificates,  just got to fix one SSL bug in socket server.

Angus

This change is now in SVN.

Angus

The change to report exceptions rather than hide them is now in SVN.

Angus

It is never a good ideal in high performance application to update a visual component like TMemo, scrolling the screen is expensive for heavy updates and will slow down or even stall your application, likewise calling Synchronous to do that stops your thread. 

I've been slowly updating ICS samples like OverbyteIcsIpStmLogTst1.pas so that log lines build a string buffer which is written to a TMeno from a timer every one or two seconds which is fine for most purposes.  It is better to use Windows messages or even a socket to send logging to the main thread, effectively they become a non-blocking queue. 

Our tests with ICS indicate a single thread will handle at least two hundred simultaneous requests, provided don't block with SQL or start doing massive MD5sums or other horrible stuff. 

If you look at the OverbyteIcsMailQueue.pas unit, that has a thread to send emails from a queue, updating by the main thread, and shows how to use a message handler while the thread is waiting to do something. However if your thread only makes continual sync Post requests, you might get away without a specific message handler because ICS calls one itself waiting for the request to finish.

But I'd still suggest creating an array of 10, 50, etc, TSslHttpCli all sharing the same event handlers, use Tag to identify which is calling an event and test that.  This just works. 

Angus

Why do you need to run 10 threads, instead of 10 instances of TSslHttpCli in a single thread.  That is how you use Indy which is blocking, not ICS which is none-blocking mostly. 

I would first ensure none of your threads access any visual components and ideally don't use Synchronize atall.

I don't see a message loop anywhere in your thread, so nothing should work.  You can use blocking calls in a thread, since it's doing nothing else.  And a thread may be necessary if you are doing long processing like SQL database look-ups, although even then it's better to use a thread for only the SQL stuff and leave all ICS stuff in the main thread. 

Angus

Never seen the icon lost, been signing files for 15 years. Are you using the latest signtool?  No other EXE manglers?

Angus

Most better ICS applications use Async methods so are fully event driven, allowing multiple connections are the same time in a single thread.  So most of the code goes in various event 

But sync calls are easier to use from a program flow perspective and the code can either be in the event are after the sync call, although exceptions cause trouble which is why I added ResponseNoException. But sync programs are less suited to multiple connections or tasks at the same time. 

In event driven applications, you usually use messages from an event to start something new  or a triggers in a timer event to start another connection 60 seconds later of something, I do that a lot.

Angus

It is better to process the response in the onRequestDone event. 

There is a new property added a year ago ResponseNoException which will help you.

Angus

Don't know what FConnectorUtils.StreamToString does, but you always have to seek to beginning of the stream before reading it. 

There is a new function that convert an HTML page in the buffer to unicode string with the correct codepage:

FResponseRaw := IcsHtmlToStr(FResponseStream, FContentType, true);

Last boolean option converts HTML entries like &pound; and &#9741; to unicode. 

That function is used in the new TSslHttpRest unit that hides a lot of the low level stuff you are doing returning Json.

Angus

I don't run Windows Firewall anywhere, and still see rare problems, thus the retries.  Adding your EXE to the exception list may help, but not be a complete solution. 

Angus

Sorry, no simple answer, the issue might also relate to NAT translation in a router somewhere, which has to keep track of all the sessions.  I had a Netgear router in the early days of broadband that could not handle more than one file per session. 

While TIcsFtpMulti retries individual files on error, once the control connection dies nothing works, but my higher level application then retries the whole job. 

This happens with non-SSL as well.  As I said, never got the bottom of it, just have to cope with the errors.   

Angus

Problems like that are hard to track down, could be client or server, perhaps something to do with SSL session caching since each new file opens a new SSL connection.  Trying to debug is almost impossible due to the volume of data created after 145 files.   

You can try other clients to the server, or your client to a different server, you can test against the ICS FTP server on one of my pubic servers.  I download hundreds of log files daily from those servers using FTP and errors are very rare.  But I use the new (to ICS) TIcsFtpMulti component and that does automatic retries if a file download or upload fails.

Angus