Angus Robertson

Please read my previous message again.  Do not attach screen shots and copies of our files to these messages. 

Even your text file named errors has no error information, just more source code. 

If you want support, you need to report the actual compiler errors, not just send copies of code I wrote back to us.

Angus

Quote

1) To remove huge amount of logging from online scripts and bots that keeps requesting plain HTTP on port 443, and i replied with and empty HTTP, when it is was HTTP.

I get massive logs from so called internet research companies probing my servers, I have about 40 public IP4s in three subnets and any servers listening are attacked daily, the firewalls try to restrict how many ports and IPs can be accessed, but all public servers suffer.  

ICS has a TIcsBlackList component that most of my servers use, any HTTP request by IP address is immediately blocked for 24 hours and new connections closed, well known IP ranges are permanently blocked, etc.

So checking for non-SSL connections on SSL ports would be a useful addition.  I'll experiment adding it to TSslSocketServer.

Angus

If ICS is installed according to the instructions in readme9, the samples should compile.  With Delphi 7, you use dpk not dproj.

Beware De;phi 7 is no longer supported, but should still work, Delphi XE is also very old and has not been tested for many years since my license key expired and new keys are unavailable. 

Exactly what error is being returned by which sample?  Please copy the textual error from the compiler, no attached files or screen shots.

Angus

I can read your messages, but you have still not asked anything about ICS, so this will be my last message here. 

I've never heard of PPPoW so won't be reading your other message.  It is nothing to do with Delphi or ICS.

ICS has never used github.

Angus

Sorry, we rarely read files, unless the message gives information about your actual problems about ICS, then we might ask for sample code to reproduce a specific problem or logs. 

We still have no idea what questions you are asking here.

Angus

The developers working on ICS do not have any Mac hardware, therefore we can not test if it works.  Previous ICS versions have been known to work, and we have not deliberately broken MacOS support, but changes may have been introduced that have broken that support.

We do test Windows FMX, and I added new FMX samples last year to test new components.  You should start with the samples in the platform folder.

Angus

This problem is with the new library packages I produced for ICS v9, not v10, to allow fixing Posix bugs in ICS v9.  Specifically using Lib suffix: $(Auto) to avoid compiler specific packages. 

I fixed several Android bugs with the older library, but can no longer build Android or Linux with the new library.  So IcsAndroidD110.dproj was OK, IcsAndroid.dproj gives the overload error.  Both are in the SVN, the newer one is not yet zipped, pending a lot of other package changes. 

Angus

ICS has an old experimental Linux package that used to build without errors. 

I created a new Linux package, but this gives dozens of "Previous declaration was not marked with the 'overload' directive" errors.  Is there a compiler directive or something I've missed for the new package?  Ditto the Android package. 

Angus

Quote

I mean, byte arrays exist in Delphi 7

Correct, and ICS has always used byte arrays in low level functions, the MD5 digest is a byte array, although the Sha1 digest is an AnsiChar array since it was contributed. 

But there were no TBytes library functions in Delphi 20 years old, so AnsiString was commonly used.  I've added a lot of TBytes  functions to ICS in the last few years so that support for old compilers has continued, and a TBytes version of Base64 conversion recently.  But old code only gets modernised when updated for some reason. 

Angus

ICS still supports Delphi 7 and later, so needs AnsiStrings.  I am trying to modernise some code, but can not break old functions that people have used in applications for 20 years.  

I'm expecting some bad feedback as people upgrade to ICS V9.1 which has some non-backward compatible SSL/TLS changes, new units and conceptual changes.  SVN notes have all the details. 

Angus

This Websocket client problem should be fixed in SVN and the overnight zip, assuming it was an ANSI/Unicode casting problem with non-English character set conversions, which I'm unable to reproduce.  It did not affect Websocket server which had no casting, and yet always worked against our client. 

The real problem is ICS has overloaded versions of many functions for ANSI and Unicode, and the compiler does not always choose the correct version if input and output parameters don't match, fixed by using a specific ANSI function.  I'm slowly adding TBytes versions for binary data to avoid such problems.  but it's a long job. 

Angus

ICS has a new component TIcsNeighbDevices that builds a historic LAN neighbourhood MAC device and IP address table that shows MAC vendor name to help identify devices.   Runs in a thread continually checking for new devices.  ICS can be installed from Getit.

To test it, build the ICS Network Tools sample, it's very similar to the excellent Nirsoft Wireless Network Watcher tool I've run continually for a decade to monitor my LAN.

Angus

The new Websocket API is live, when using streamlined requests, it handles about 100 SQL queries a minute from a single thread, although slower if the requests are sent sequentially waiting for a reply. 

Supporting JSON-RPC 2.0 on the server should not be hard, but does need POST support, I'll look at adding it to the ICS web server database sample.  Client is a little harder since JSON-RPC 2.0 JSON-RPC uses a sequential ID to keep track of requests.

Angus

Quote

the output of the hash function is in TBytes and the encode function can take TBytes as input.

The output of the hash function is AnsiString and the input to encode is AnsiString, so the String cast is unnecessary.  I'll remove it for the new version going to SVN today.

Angus

Quote

why are you using AnsiString casts

The Base64Encode and SHA1ofStr functions were written 20 to 25 years ago before Unicode, we convert to UTF-8 before calling them if needed.

In this unit, the input to create the server key is a Base64 encoded random number we created and sent to the server earlier and a GUID so all simple ASCII.

Angus

I will need to reproduce connection failures before I look into fixing them.  So I need a Websocket server to test against. 

My only suggestion is to try the client against other servers and see if any fail, because no-one else has reported any problems. 

Angus

Are you suggesting there is a problem with the Base64Encode function creating incorrect output? 

Have you attempted to debug and fix the problem yourself, or are you expecting someone else to do it?

I've been testing the OverbyteIcsWebSocketCli unit this week and it works fine. 

Angus

As the comments in the code say, the MsUserAuthority setting depends on the account type that the credentials relate to, mainly for corporate accounts, common and consumer are for general email accounts, the older version defaulted to consumer.

Angus

The URLs used for Rest Email are at the top of the OverbyteIcsSslHttpOAuth.pas unit, and have not changed since the unit was created, except to allow <MsUserAuth> instead of consumer, are you setting MsUserAuthority correctly?

Angus

Quote

 persistent HTTP connections

No idea why this client is not using keep-alive, perhaps their REST library does not support it? 

The basic Websocket API is working, and the client can access another Websocket interface on the server so seem to know their stuff, now just need to design the pipelining, so that queued queries get returned to the correct client.

I'll look at JSON-RPC 2.0 for next time.  

Angus

Sorry, no quick solution.  There are OpenSSL providers to access private keys on TPM2 devices and to access certificates in the Windows Store (which ICS can now do), but not to access the Windows crypto signing APIs so that other dongles can be supported, at least to my knowledge.

Angus

Digital certificates are sometimes distributed on USB tokens precisely to stop them being copied by software applications.  ICS can only access private keys that are stored in local windows store. 

The USB token is not a simple storage device, it has an operating system that allows applications to pass data to the token, which is then signed by the private key and passed back to the application as a hash.  This signing is normally done by a Windows or OpenSSL API, and they need to be aware of the token and use that instead.   All token suppliers provide drivers that allow Windows applications to use their tokens, but not usually for OpenSSL. 

OpenSSL needs extra code to access USB tokens, in the obsolete versions it was called an engine, with 3.0 and later it's called a provider, but it's just another DLL.  I understand there are OpenSSL providers to allow use signing with USB tokens, but they are supplied as C source code not Windows DLLs, and I've never tried any.  

Integrating such a provider into ICS would not be quick, or maybe some-one has done it already?

Angus

Thanks for the comment, yes record ending is important, since this API has simple URL encoded arguments a single CRLF will be fine.  In fact, allowing multiple requests in a single message is probably what is really needed.  This client tends to do 10,000 odd requests in a single block during the night, takes about 25 minutes at the moment, single server thread, with a new session each time, eight per second. Will need to decide how many requests can be queued, in case they decide to try 1 million. 

The responses do include the main argument, ie {"success":true,"reccount":16,"records":[{"number":"118118","number_from":"2010-03-10", etc. I should add the API type to the outer wrapper.  Then send them back one message at a time.  

Either the server or client can ping/pong to keep the connection open, I was planning on the server doing that to avoid complexity at the client, they simply close the connection when the batch is over.

 Angus

I offer clients a REST API service to look-up telecommunication information, using the ICS web application server and MS SQL server. 

It works well, for low volumes of queries, but most users start a new SSL/TLS session for each query, which becomes a limiting factor with performance. 

So I want to offer a Websocket API as well, so one SSL/TLS session stays open, with just simple request/response packets sent. 

But how to adapt the REST HTTP request/response to Websocket?  My queries are simple URL parameters, ie codelookapi.htm?numhistory=118118.

Should the Websocket message just be the arguments or include the full or partial URL as well?  Or something else, like a command? 

Should the initial Websocket request allow arguments, or just open the connection? 

Should be Websocket response message be just a JSON block, or include a wrapper of some sort, like the HTTP response header? 

Has anyone done a similar design?  I just want to make it easy for clients using standard Websocket client libraries to integrate the new API.

Angus

Quote

Nice thing with async networking is that you can start many connections simultaneously and react for connection/timeout in event handlers.

Indeed, no need for threads in ICS. 

But Windows uses a thread to connect a TCP socket, and typically waits 30 to 40 seconds for a response before the Close event is called with an error, you can not abort it earlier.  So the socket can not be re-used for another connection immediately, unlike ICMP.   If you are checking a lot of hosts, you need a socket pool where they are not re-used until closed by Windows. 

Angus