Angus Robertson

So effectively you want to use an external pool of TSslHttpCli objects from within your threads, rather than creating them as needed within the threads? 

So why use TSslHttpCli  in the thread, why not just use one of the pool objects asynchronously, waiting in the thread for a semaphore to be set on competition? 

Windows actually creates a thread for async winsock operations, so there is no reason to use TSslHttpCli in the thread with all the messy stuff that goes with it. 

One of the ICS samples uses a pool of components and a queue to download all the elements on a web page.

Angus

Thanks for the code, added to my local copy, will be in SVN later this week.

Angus

It looks like all the OAuth2 stuff has worked, but the account you authenticated does not have access to POP3 mail. Could be different scopes are required for Exchange, I only test against consumer accounts and servers like office365.com. 

The error for graph.microsoft.com is attempting to get your profile and email address, which works with Google but not currently Microsoft, I could not find scopes that gave access to all the APIs I needed.  It's not fatal.

Angus

To use OAuth2 with the POP3 component, you need to also use the TIcsRestEmail which handles all the OAuth2 stuff, look at the OverbyteIcsSslMailRcv sample which has all the extra code needed. 

There will be significantly improved OAuth2 support with a new embedded browser window later this week.

Angus

Can you reproduce this problem in any of the ICS sample applications?  Never seen it before. 

Generally, ICS handles loading and unloading OpenSSL itself, some application use LoadSsl to load it early to check for errors or version, but it's not necessary.

Angus

The main issue in implementing your dual protocol concept is pre-reading the initial data received from the client, and then resetting so that it's read a second time after initialising SSL.  That will be messy with our event driven structure.   

Angus

My own servers listen happily on multiple ports and addresses using IcsHosts without a problem.  This is my main web server:

Socket 1 State: Listening Only IPv4 on 217.146.102.150 port 80
Socket 2 State: Listening Only IPv6 on 2a00:1940:2:2::150 port 80
Socket 3 State: Listening Only IPv4 on 217.146.102.150 port 443 SSL
Socket 4 State: Listening Only IPv6 on 2a00:1940:2:2::150 port 443 SSL
Socket 5 State: Listening Only IPv4 on 217.146.102.155 port 80
Socket 6 State: Listening Only IPv6 on 2a00:1940:2:2::155 port 80
Socket 7 State: Listening Only IPv4 on 217.146.102.155 port 443 SSL
Socket 8 State: Listening Only IPv6 on 2a00:1940:2:2::155 port 443 SSL
Socket 9 State: Listening Only IPv6 on 2a00:1940:2:2::250 port 80
Socket 10 State: Listening Only IPv6 on 2a00:1940:2:2::250 port 443 SSL
Socket 11 State: Listening Only IPv4 on 217.146.102.153 port 80
Socket 12 State: Listening Only IPv6 on 2a00:1940:2:2::153 port 80
Socket 13 State: Listening Only IPv4 on 217.146.102.153 port 443 SSL
Socket 14 State: Listening Only IPv6 on 2a00:1940:2:2::153 port 443 SSL

Hosts=www.telecom-tariffs.co.uk,www.telecom-tariffs.uk,telecom-tariffs.co.uk,telecom-tariffs.uk
BindIpAddr=217.146.102.150
BindIpAddr2=2a00:1940:2:2::150
BindNonPort=80
BindSslPort=443

(lots more)

And different Let's Encrypt certificates on each address. 

Angus

I accept your concept is possible, but would be messy to implement in ICS, and add code that would be of no use to the vast majority of users but would end up in all server applications.

I suggest you derive a new component from TSSLWSocketServer, and implement it yourself.  If there is interest from other developers for this functionality, it could be added to ICS.

Angus

Technically, it is possible to recognise a non-SSL connection is being made to an SSL port, OpenSSL specifically checks if an HTTP header is being received rather than a HELLO packet and raises an error.   And hackers often do this, attempting to made non-SSL connections to port 443, no idea why.

But to fall back from SSL to non-SSL would require the co-operation of both client and server, a non-SSL client would never attempt to connect to port 443, unless incorrectly configured. So I'm not sure what scenario you are anticipating.  Perhaps some industrial environment where you use a special port 8080 or something for ease of configuration of both protocols on the same port? 

This is hardly a widely needed feature, so development would be hard to justify, except commercially.

Angus

There are ifdefs relating to other zlib related files, and those changed in V8.70 to support native Delphi zlib, but OverbyteIcsZlibHigh is always used unconditionally in any unit that needs ZLIB support.  But I may have screwed something up, I'll do more testing later in the week. 

Angus

Those literals and functions are in OverbyteIcsZlibHigh.pas.

That unit was heavily rewritten last month for the last release, but has existed for many years.

Angus

If you build the OverbyteIcsHttpRestTst sample application and access your JSON URL, you'll see the sample parses the JSON into a ListView, you can click on an object and it will expand that into another window, mostly done in the DisplayJson procedure using SuperObject DataType.

Angus

All I can say is it builds here without errors on 11.2, do it almost daily.

I rarely build the non-FMX packages, but if you are not using FMX they should be fine.

Angus

Your main problem is trying to use old software in a world where security changes need newer software. 

The SSL error you got is almost certainly because the application was using obsolete SSL protocols that are no longer supported, only TLSv1.2 and TLSv1.3 are acceptable today, and the latter needs software released in the last two years, 

V8.58 is four years old and will be packaged with obsolete versions of OpenSSL and default protocols.  You should be using V8.70. 

Also we have not updated or tested the C++ samples for 10 years, so they need updating to use the latest protocols, you need to compare the Delphi samples and see what changes have been made in 10 years.   Sorry, the ICS authors don't support C++, that can only be done by users of the component.

Angus

The component probably was in use 15 to 20 years ago, it looks like clean code, but compilers change. 

That User Made page is all very old projects for old versions of ICS, several of them are now part of ICS.

Angus

But what error are you getting, and when, compile or runtime?. 

That ancient program was built for an earlier version of ICS, but should still work once the unit names are corrected. 

You may want to change all strings to AnsiStrings and Char to AnsiChar since you are using a unicode compiler.

Or are you expecting someone to correct, build and debug it for you?  

Angus

ISuperObject documentation and examples are very poor, but it's a reliable and fast library...

There is an example in the OverbyteIcsSslX509Certs.pas unit, search for SA([]) which creates an empty array, then you can add elements with a blank name to create a simple array.

Angus

Conditionally added CRLF to a header line is certainly possible, but it would be better done when the properties are set. 

I'm planning a new multi-threaded web server which will be an opportunity to clean up legacy problems with the existing server that has evolved over 25 years. 

Angus

The USP of LZMA in 7Zip was higher compression size and better decompression speed, against a slower compression speed, which is usually done rather less frequently. 

ZLIB deflate as used by ZIP and HTTP compression is speed, not minimal size, with options for both.  The ZLIB library in Delphi is optimised C code which will be faster than the Pascal conversion of LZMA, I'm sure the DLL version will be faster, but then we are into DLL hell. 

Angus

ICS V8.70 has been released at: http://wiki.overbyte.eu/wiki/index.php/ICS_Download

ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10 Seattle, 10.1 Berlin, 10.2 Tokyo, 10.3 Rio, 10.4 Sydney and 11.0 and C++ Builder 2006 to XE3, 10.2 Tokyo, 10.3 Rio, 10.4 Sydney and 11. ICS supports VCL and FMX, Win32, Win64 and MacOS 32-bit targets. The distribution zip includes the latest OpenSSL 3.0.7 win32, with other versions of OpenSSL being available from the download page.

Major Changes in ICS V8.70 include:

1 - V8.70 has various minor improvements providing better compatibility with modern compilers such as more unicode overloads to avoid ANSI string warnings and casts, and more use of TBytes to avoid ANSI strings. Updated various samples to use TIcsRestEmail to support OAuth2 authentication for GMail and Outlook that no longer allow old authentication protocols.

2 - The TIcsFileCopy, TIcsFtpMulti and TIcsHttpMulti file transfer components now support file zipping and unzipping using System.Zip in recent Delphi compilers, instead of the obsolete VclZip which is no longer available. Before a file copy or FTP upload, files may be automatically zipped, useful for large log files, after a file copy, FTP or HTTP download, files may be unzipped in various ways.

3 - Added support to TIcsFileCopy to copy file names longer than 259 characters by adding \\?\ to the start of long names passed to Windows APIs, if supported by the disk file system, unicode APIs only. Fixed a problem deleting empty directories after copying. Fixed a problem with BuildDirList2 with COMPILER16_UP.

4 - The OverbyteIcsXferTst sample has a new tabs, 'Single File Copy' to test the CopyOneFile method and 'Zip/Unzipping Files' to test zipping and unzipping that has always been supported by the components but not this demo.

5 - Allow content compression for HTTP and FTP using System.Zlib in newer versions of Delphi instead of the OverbyteIcsZLibObj unit to avoid duplication. Only Delphi 11.1 and later have the same ZLIB 1.2.12 as ICS, so will automatically used System.Zip.  Beware a new version of OverbyteIcsDefs.inc is required to allow ZLIB to work correctly, otherwise it will default to using the DLL which is unlikely to be available, it is not in the distribution.  So either install the new inc file and customise it, or copy the ZLIB changes to your own inc file.

6 - In TWsocket, added ReceiveTB(var Data : TBytes; MaxLen : Integer = -1): Integer; where MaxLen is optional, to receive TCP data into a TBytes dynamic array of bytes. Also ReceiveFromTB and ReceiveFrom6TB for UDP datagrams. The last release added similar SendTB functions, so buffer pointers and ANSI strings can now be avoided.

7 - Added UTF-8 support to TIcsIpStrmLog, to convert received lines from UTF-8 to Unicode with unicode compilers (as String) and converts sent data to UTF-8. Changed FRxBuffer to TBytes, use SendTB and ReceiveTB methods with TBytes.

8 - Updated OpenSSL to 3.0.7 and 1.1.1s.  OpenSSL 3.0.6 was withdrawn shortly after  release, we never distributed it.

9 - In OverbyteIcsSslHttpOAuth, added an OAuth2 and Rest Email Microsoft User Authority property to access different user authorities, defaults to 'consumers' but can be changed to 'common' or an Azure Active Directory tenant GUID for corporate accounts.

10 - Added TIcsRestEmail to support OAuth2 authentication to the OverbyteIcsSslMultiWebServ, OverbyteIcsSslMultiFtpServ and OverbyteIcsDDWebService samples, since GMail and Outlook that no longer allow old authentication protocols.

11 - In the TIcsInetAlive component, added a new method AliveMethEither so internet alive checking works if either ping or HTTP works, instead of one
or the other.

More detailed release notes are at http://wiki.overbyte.eu/wiki/index.php/ICS_V8.70

Angus

OpenSSL has released new versions of the two supported branches, 3.0.7 and 1.1.1s, Windows binaries are available from

http://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp .

OpenSSL 3.0.7 fixes two serious security bugs in 3.0.0 and later relating to verifying X509 certificates with email punycode (non-ASCII characters) name constraint checking, that can cause a crash.  These bugs can only happen after a certificate chain has been successfully checked, mainly in client applications (or servers that request and verify client certificates) so should not happen with self signed certificates.  Another security bug fixed related to using null encryption, which ICS doesn't use. Also, the RIPEMD160 hash no longer requires the legacy provider.
 
OpenSSL 1.1.1s has general bug fixes.

Separately YuOpenSSL has released both these versions as commercial DCUs allowing applications to be used with OpenSSL without needing separate DLLs.

 
Angus

The SendHeader function does not add any line endings to PersisentHeaders, each header line requires a line ending, including the last header.  Changing that now would break all existing web servers.

Angus

SendHeader is really an internal function, it is called during most of the other Send/Answer methods, and will send any PersistentHeader property lines, together with headers specified by the Send/Answer functions, that may also add their own headers.  Only use SendHeader if you are using low level functions like Send and SendStream.

You can use the onRespHdr event to log the headers actually being sent, to check they are correct.

Angus

There are various methods for sending web server responses, SendDocument has a CustomHeaders property where you put complete header lines, while AnswerPage, AnswerStream, AnswerString, etc have a Header property that does the same.  You don't need to use an event.

Angus

The missing path "\$(Platform)" error in D11.2 effected many other components, it was nothing to do with HTMLViewer specifically,

Angus