Angus Robertson

Thanks, I already have FileZilla installed, albeit an older version, will test later and see what FileZilla has broken.

Angus

OverbyteIcsXferTst.dpr is a complete ready to build testing project, takes a couple of minutes to start downloading stuff, with logs.  I have FileZilla on one of my public servers, will test it later.

Angus

The ICS FTP components have been tested regularly against FileZilla Server since 0.9.10 beta 15 years ago as you can read in the source code, can not recall testing it for a while, since I don't recall it every being broken.  If FileZilla has re-invented the FTP protocol in some non-standard way, I'll look at it, once I see full logs from OverbyteIcsXferTst.

Angus

You should never use TDateTime for duration calculations, users can change the system time, and summer time saving changes it twice a year (unless you use UTC time). 

Always use the difference between two GetTickCount64 Int64 values.

Angus

No real idea what that response means, ICS does not share TLS sessions, they are new for each connection. 

Which ICS component are you using?  It should be TIcsFtpMulti for which there is a sample OverbyteIcsXferTst.dpr which will allow you to test again FileZilla saving a proper log that may show the error.

Angus

Can you please try and install the latest ICS from SVN or the overnight zip, we've made various C++ package changes (CPP 11.0 only) this week that should resolve the lib files not being updated, and fixes some CPP warnings.   

Same applies for any other CPP users, V8.68 is finished and will be released next week, so now is the time to test it installs correctly. 

V8.68 is a minor release, mainly install problems, added OpenSSL 3.0.1, and support for new HTTP request and response methods to help caching (Entity Tags), the REST component will now download files of any size, including resuming failed partial downloads, and HTTP error reporting is improved, as illustrated in the  previous post here that previously would have said just Abort without any explanation. 

Angus

Probably unrelated, but just been reading an article in PC Pro magazine about the new Intel 12th generation processors, which have two types of core, performance and efficiency, something ARM has had for a while. 

Only Windows 11 has the extra code to receive telemetry from the Intel Thread Director in the CPU to negotiate on which cores processes should run.  Windows 10 application performance may be more random, and different each time you run it.  Which is slightly frightening. 

So just saying modern CPUs may affect applications in ways you have not considered. VMs are even worse. 

Angus

All I can say is you must build the common and vcl packages before the design package. 

But as you say, if you don't put any components on forms you don't need any packages, just build the units that your application actually uses. 

Angus

BTW, for C++ you should be using the early version of V8.68 from SVN or the overnight zip (same wiki page as the stable version), I fixed a load of C++ package and sample issues back in October so that 10.4 and 11.0  build again, including the ones you mention, sorry only just remembered. 

Angus

Sorry, I don't support C++, I just distribute files updated by other ICS C++ users.  Hopefully one of them will be along shortly to help.

OverbyteIcsSslThrdLock.pas has long gone, just remove any reference to it.  ICS builds with Delphi 11.0 without any warning, if C++ gives warnings just ignore them.

Angus

All that happens internally within the ICS TRestOAuth component, it holds the expiry date so knows when to refresh the access token. 

Getting a new refresh token has various options since many applications are used unattended, not just Windows services, so it will notify an administrator that a new OAuth2 login us required, but API access will fail until it happens.  It is unfortunate that OAuth2 was designed without a refresh expiry date being known, so things could be planned better.

Angus

Provided you have set the RefrToken or RefreshToken property, you won't see a login window. 

It;'s expiry is unknown to the application, but can happen if the account secrets are changed, if you invalidate it through the online console, or at the whim of Google or Microsoft.  So applications need to handle token failure, as I mentioned a couple of days ago. 

The same refresh token can be used on multiple computers, at least for Gmail which is the service I use regularly as a backup when my own SMTP server is down.   This why the ICS MailQueue component handles multiple email servers, with OAuth2.

Angus

I think you are confusing the two tokens that OAuth2 should return. 

In ICS, AccToken property is the short lived token used for HTTP requests.  The RefreshToken property is a long lived token that you should store safely like a password, and may be used repeatedly by the component to refresh AccToken, in background without any interaction.  I tested GMail yesterday, and the saved refresh token meant it just worked without a new login. 

So you need to check if Azure is returning a refresh token and whether you are saving it for re-use.   You may need to provide specific scope settings to get a refresh token, see the constant OAuthUriMSRest. 

Angus

Quote

I wonder what prevents them to make it FMX compatible right away, on all platforms, I expect not much VCL code inside ?

The ICS packages all build okay with FMX and YuOpenSSL, why do you think it is incompatible?  Or at least only with lots of deprecated warnings from OpenSSL, that we never see when using the DLLs.  Just built one of the FMX samples and that works fine, once I'd commented out a couple of old lines.  Need to update those old samples.

Angus

OpenSSL has released quarterly updates for the two supported branches, 3.0.1 and 1.1.1m, Windows binaries are available from

http://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp .

OpenSSL 3.0.1 fixes a medium security risk relating to clients verifying X509 certificates from the server, a malicious server could potentially send a bad certificate that caused the client to hang or misbehave during verify.  https://www.openssl.org/news/secadv/20211214.txt

Now OpenSSL 3.0 has been available for three months, updated the main supported OpenSSL release to 3.0.1.  The samples SslInternet directory now has both OpenSSL 1.1.1m and 3.0.1, ICS will try and load OpenSSL 3.0 first, then 1.1.1 if not found, unless the global variable GSSLEAY_DLL_IgnoreNew is set true before OpenSSL is loaded. Likewise GSSLEAY_DLL_IgnoreOld may be set true to ignore 1.1.1 and fail unless 3.0 is available. This is available from SVN and the overnight zip. 

Note the binaries are now digitally signed by 'Magenta Systems Ltd' instead of 'Open Source Developer, François PIETTE' due to the massive cost of renewing the open source certificate.  Developers can always resign the DLLs with their own signing certificate to remove the Magenta name.

Separately YuOpenSSL has released both these versions as commercial DCUs allowing applications to be used with OpenSSL without needing separate DLLs.

Angus

I only removed TSuperWriterSock last year, perhaps there are references to it in an old HPP file built from an earlier ICS version? 

We don't distribute HPP files, maybe deleting it would cause it to be rebuilt correctly - but I know almost nothing about C++.

I'd prefer not to restore unused code. 

Angus

I assume you are referring to an OAuth2 login page appearing in a browser during authentication. 

For Google, the Refresh Token you receive after an interactive OAuth2 login remains effective for several months or longer, generally, so can be used by services, believe this is the case with Microsoft as well, but don't recall from testing.  Your service can email an admin to update the token manually if it expires.  You must store the Refresh Token securely as if it were a password, because that's really what it is, it is used by OAuth2 to get a new 12 hour or something Application Token which is the one you use to access APIs. 

If you set AuthType to OAuthTypeMan, an event will be called in which you can send the email.  The event could launch an interactive application, if the service is running on PC with someone watching the screen. 

Angus

I only remember one security patch for Delphi, well over 10 years ago, relating to a graphics function that did not check a PChar string length properly.  Have there been any others?

Angus

Sorry, never done any testing with THttpTunnelWSocket, it does seem to be used by the FTP components as a proxy option, but not tested that feature for a long time either. 

Angus

UDP application have to be designed to work around data loss, that is not the fault of a low level UDP component.  If one loses packets and another does not, the first is probably less efficient. 

UDP is the basis of HTTP/3 and QUIC, which is reliable. 

Angus

If you have to send a response on SQL completion, you have to process all JSON in that request before sending a response, so a queue is not that useful, 

So this is down to your server design, if you are listening for connections that is a server, and you have still not explained how you are doing that, it is not a 'web site', it's a server.   Another web server may be sending your server requests, but that is not relevant.

Angus

You have still not clarified how exactly you are receiving the JSON, just a vague 'Cliente Server'. 

But if a new request is stopping an old request, it sounds like you have a single listener for receiving requests, whereas any proper server would accept multiple requests and handle each one separately, sometimes in a thread, but not necessarily with good program design and the ICS internet components. 

While a FIFO queue can help, there is a problem if you need to send a response for the SQL update status, particularly if you are continually opening and closing SQL connections. 

None of this design is really relevant to how many JSON records you receive, you just process them in one go, ideally with one SQL update.

Angus

How are you receiving the JSON, with a web server or TCP service? 

I have a similar application that accepts data in various ways and writes to a SQL database, I use a FIFO queue (a stringlist) for the SQL stored procedures, so any that arrive faster than the SQL can accept them (about 20 per second) get queued.  If the volume is really high,  you can open multiple connections to the SQL server and do some in parallel.

Angus

So you are looking for the friendly names for serial COM ports?  There are various methods to find them, depending on how were installed:

1 - Enumerate HLM\HARDWARE\DEVICEMAP\SERIALCOMM registry
2 - Enumerate Device Installation Class 'Ports' which finds most serial ports including USB
3 - Optionally add Disabled 'Ports' (hardware not currently installed)
4 - Enumerate Device Installation Class 'CNCPorts' (used by com0com serial port emulator)
5 - Enumerate Device Installation Class 'Modem' which finds USB and internal modems

I have an old component that does all this, never got around to publishing it, maybe in December.

Angus

You would be better starting with the OverbyteIcsHttpRestTst.dpr sample, even if you don't actually make use of the REST features. 

The TSslHttpRest component is a high level version of THttpCli that bundles all the extra components for extra functionality, including SSL configuration and certificate validation with a  root bundle, SSL session caching, content compression, content code page decoding, persistent cookies, Json handling, logging, client SSL certificate.

You ignore SslContext and SslVerifyPeer, and just set CertVerMethod to CertVerBundle or CertVerWinStore and all the chain checking is done for you. 

You still need to implement your SSL pinning in your own onHandshakeDone event, checking whatever aspect of the server certificate you have been given.  My very brief reading about HPKP suggests you hard code a hash of the public key, not a hash of the certificate, so you will need to calculate that yourself, ICS does not have a method to get that hash. 

But all browsers have abandoned HPKP so I'm not sure what you are trying to achieve. 

If you are trying to ensure your SSL connection has not been intercepted by an SSL proxy or firewall that has generated a fake local SSL certificate, you can check the server certificate is signed by the expected root authority, although the chain should fail verify anyway if you are using CertVerBundle with the ICS built-in bundles since they won't include the private root certificate used to fake the SSL certificates in the proxy or firewall. 

Beware this is my understanding of how SSL certificates are faked, I've no direct experience of such corporate environments.

Angus