Angus Robertson

To sum up, you are unable to make the ICS proxy work with a specific web server or web broker, really no idea what that is.

How do you know this is an ICS problem?  Have you tried using the ICS proxy with other more modern web servers like Apache, as I have it, 

Your service example is undocumented, no setup instructions, I've no idea what is supposed to do, and I don't have time to debug it for you.

Angus

I've tested the ICS proxy as a reverse HTTP proxy for HTTPS<>HTTP, I've emailed you the URL, everything seems to work fine with a browser, with web pages of varying sizes.

The servicetest zip you posted seems to use web broker, never used that, and nothing to do with ICS.  I'm afraid I don't have the spare time to build, install and test such applications that are unconnected to ICS. 

Angus

I only managed to tick a few boxes, guess I've managed without all those tools for 25 years. My experience is the IDE is more stable to less third party stuff you install.

Angus

There are no simple solutions to supporting multiple cloud providers.

The free ICS component library has the HTTP REST and OAuth2 support you need, and has components to access cloud email at Google and MS, so should not be hard to add support for cloud storage, but someone will have to dedicate a few days to supporting and testing each cloud service. 

Angus

Each of these cloud providers has its own REST API to access storage services.  

You can sign up to each of them, read loads of API documentation and write code using various REST components for each one, and hope they don't change the API too often.

Or search GetIt for commercial components that have done all this hard work, and hope the authors keep them up to date as APIs change, TMS Software and /n Software come to mind, never used them myself. 

Angus

I'll be testing using my own public proxy server, remembered I do have one http server, svn.overbyte.be.  But I have some stuff to catch up with first.

Angus

Thanks, I'll look at reproducing it on my servers, main problem is all my web sites are https only and force http to https, so needs something set-up for testing.

Angus

To fix a bug I must be able to reproduce it.  Sure the logs sure some different numbers, but I've no idea what data or requests you are sending, no headers logged or anything useful for debugging. 

Forward proxy is intended to direct to another web site, such as this one that generates a lot of data, very useful for testing in a live environment, the internals of the proxy moving data are the same for forward and reverse HTTP proxy. 

Angus

ICS V9.2 has been released at: https://wiki.overbyte.eu/wiki/index.php/ICS_Download

ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10, 10.1, 10.2, 10.3, 10.4, 11 and 12 and C++ Builder 10.4, 11 and 12. ICS supports VCL and FMX, Win32, Win64 and MacOS 32-bit targets. Beware Mac OS-X and C++ have not been tested recently due to lack of support from such users.

The distribution zip includes the latest OpenSSL 3.0.14. 3.2.2 and 3.3.1, for Win32 and Win64.

Changes in ICS V9.2 include:

1 - V9.2 is a minor release, fixing a few issues introduced in the last major release, and other bugs located since. There are no breaking or installation changes from V9.1, but if updating from earlier releases please read https://wiki.overbyte.eu/wiki/index.php/Updating_projects_to_V9.1

2 - TIcsMailQueue can now queue a prepared EML file created by another application, or perhaps received by the SMTP Server.  Added optional SkipEmpty argument to StartMailQu method so queue is not started unless there are pending emails waiting to be sent. The sample has 'Send Prepared EML File' to queue an existing EML file rather then preparing email with properties.

3 - Improved email MIME decoding by supporting embedded boundaries, usually for multipart/alternative parts, within a multipart/mixed message, using code written 20 years ago but suppressed for some reason.  Previously these parts were sometimes left encoded within a part. There is a new property LooseRFC to allow decoding if the boundaries in the body are missing the two required hyphens, usually because the boundary also begins with hyphens. TMimeDecodeEx should now always return the body if no MIME parts are found, and TPartInfo has PLevel which is Part Level, and PInfo which is displayable part information for logs.  The MimeDDemo sample has various improvements to test these features.

4 - Fixed a nasty Win64 problem reading EC certificates from the Windows Store, which may have caused server crashes, also reproducible in the PemTool listing the Windows Store.  This was due to Win64 bad initialisation of a buffer used for a Crypto API call that failed.

5 - The HTTP client now checks the URL always has / at start of the path, ie add it for test.com?query.  In the REST client, added a sanity check for RawParameters to encode any spaces, which can break the HTTP request.  After a file download completed, check actual file size against response size. The multipart/form-data MimeBoundary no longer includes extra -- at start that are required preceding boundaries within parts, some web servers may have  been unable to decode our MIME encoding.

6 - The HTTP server has a new method AnswerRedirect for various redirection responses to a new URL. When accessing the default document in a path without a trailing path delimited /, redirect using 301 to the correct path with delimiter instead of adding it locally and displaying the document which will then incorrectly link to pages in a higher level directory. Using the THttpOption hoAddMissPath redirects if the default document is missing perhaps a template or virtual document. Fixed a bug where authenticated POST/PUT requests always returned a 404 error. Added AnswerBodyTB client response with TBytes binary, similar to AnswerString, tested in the sample by supporting favicon.ico request.   Check if the request HTTP version gets corrupted due to spaces in the URL, which are not allowed.  The SslMultiWebServ sample has new web pages to test POST/PUT and template authentication.

7 - TSslX509Certs has a new function CertResetDomain to reset a certificate order state to None, if the order process stalls or gets confused due to errors. If AcmeV2StartChallgs fails because there are no pending challenges, reset to order to None so it starts again next time and does not loop.

8 - ICS not longer tries to load OpenSSL RAND_screen function that may be missing from recent DLLs.

9 - TIcsHttpMulti fixes a bug introduced in V8.66 that stopped the application setting authentication, rather than adding it to the URL, and a Win64 free stream bug.

10 - TIcsIpStrmLog correctly counts failed client connection attempts if ping is not used first to check the remote IP address. The sample has a new client Retry Attempts box to test this.

11 - Updated the Snippets sample to use authentication to access some the hardcoded URLs, which started failing after authentication was added to test web server bugs (see above).

12 - Added OverbyteIcsHttpThrd sample to show how to use TSslHttpRest component in a multi-threaded program.

13 - Improved Posix support for Linux and Android, not tested or supported yet.  Beware SSL does not correctly load for Posix at the moment.

14 - Added support for a new feature release of OpenSSL 3.3 with {$DEFINE OpenSSL_33} in the Defs.inc file, ICS includes  new versions of the active versions, 3.3.1, 3.2.2 and 3.0.14, but no longer includes 3.1 since there are two newer feature versions.

15 - Updated the 'ICS Intermediate Short' SSL certificates, used by ICS to generate temporary server certificates to allow SSL servers to run until a Let's Encrypt or commercial certificate is installed. It now expires after 200 days, 21st December 2024, after which self signed certificates will be used instead, unless a newer 'short' is installed.

16 - Only Delphi 10.41  and 10.42 (10.4 with updates 1 or 2) will install correctly with the new install packages, the original RTM version does not support the package LIB suffix: $(Auto) so you must change it manually for each package to 21.0.

The release notes for V9.2 are at https://wiki.overbyte.eu/wiki/index.php/ICS_V9.2

All ICS active samples are available as prebuilt executables, to allow ease of testing without needing to install ICS and build them all. There are four separate zip files split into clients, servers, tools and miscellaneous samples which can be downloaded from https://wiki.overbyte.eu/wiki/index.php/ICS_Samples

Angus

Nothing obvious missing, why did you create a certificate in the Windows store instead of a PEM/PFX that ICS would create for you?  I find it better to test with real IP addresses, your server won't start if there are any web servers running on 80/443. 

The two logs look the same except for a couple of values, not sure what they are supposed to show. 

All my mail travels through TIcsProxy, and I use the Forward Proxy on my public servers to access my local servers remotely, so the component is well tested, and rarely touched to break it

Angus

TStringList handles most requirements for CSV handling with the CommaText method, you can change the delimiter if needed.

You read one line at a time into TStringList.CommaText  and get one line per column, or vice versa to create a CSV record.  The file can be read or written with a second TStringList or sequentially with normal file handling code if large. 

Angus

What specific new features and performance improvements are you looking for?  What is the business case for the major effort a migration would involve?  This is many weeks or months of work, and retesting.  

All your other questions have been covered in detail over the last 15 years as others have migrated from non-unicode compilers to modern Delphi.  

I've just released a new version of one of my products built with Delphi 2007 because of third party component problems with Delphi 11.  The end users don't care.

Angus

OpenSSL has released new versions of the active versions, 3.3.1, 3.2.2 and 3.0.14 which have several low priority security fixes.

Windows binaries are available in SVN and the overnight zip file and separately from https://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp

In addition to the three DLL files, the zips include compiled RES resource files that contain the same DLLs, text files and version information, see the RC file. The RES file may be linked into application EXE files and code then used to extract the DLLs from the resource to a temporary directory to avoid distributing them separately.

ICS V9.1 and later optionally support loading the resource file.

These OpenSSL versions are included with ICS V9.2 due for release in a few days, but available from SVN and the overnight zip.

Separately, YuOpenSSL has released 3.0.14 as commercial DCUs allowing applications to be used with OpenSSL without needing separate DLLs.

Note: OpenSSL also released 3.1.6, but this ceases support in less than a year and has two newer releases, so we've stopped building binaries for it. 3.0.14 is a long term support release so is still being built.

Angus

Your management may need to concede that the components offered by Embarcadero are often missing features in third party components, which is why there are so many of them.

Angus

You are not using the modern REST component TSslHttpRest, but the older harder to use component TSslHttpCli, your choice, but you have to live with more complex code.

The point here is that you are responsible for determining whether you successfully connected to the HTTP server, and cleanly close everything if not, and try again.

ICS is event driven, it does not wait x seconds and then decide to clean everything up for you, not even the sync modes.

There will be no breaking changes to the way the HTTP client handles errors, without weeks of discussion.  You need to handle this yourself.

Angus

This was fixed for V9.2, in SVN and the overnight zip.  Are you using one of those?

Angus

If your error handling code does not correctly clean up after errors, you can check ,

Before starting a new request, you can check State <> wsClosed before starting a request and close it.  Beware, Close may not be instant, if Windows is still attempting to connect.

Angus

X509_get_X509_PUBKEY was added to OverbyteIcsLIBEAY;.pas for V9.1, so you probably have mixed versions, with old and new units. 

Angus

To my eye, the two procedures seem to be identical, so I can only assume the different result is due to different parameters to the two procedures.  

BTW, since you are using the RawParams option in RestRequest, RestParams are ignored. 

Angus

Authentication should now be fixed in SVN and the overnight zip.  

Angus

AnswerBodyTB is now in SVN and overnight zip updated.  

Angus

Quote

How to download DLLs from internet from a https site, without SSL DLLs present

That was fixed in V9.1, or at least there are now settings that allow the HTTP clients to be used without SSL being available, for http:// only.

I go to extreme lengths to keep ICS backward compatible, I know how horrible it is updating to new compilers and components when you get blizzards or IDE error messages on opening old projects, it's why some of my projects are still Delphi 2007. 

Rarely backward compatible is not possible, but I did spend a week or two writing documentation and updating samples explaining how to update to V9.1. 

Angus

The default defines are designed so that users get new applications working easily and safely, without them failing immediately because the OpenSSL DLLs can not be found.  Or needing to copy those DLLs into every single different directory to which modern versions of Delphi build by default.

Angus

If you undefine {$DEFINE OpenSSL_Resource_Files}, ICS will look in the ICS-OpenSSL directory for the DLLs, so how does make you in control? 

If you want to load the DLLs from your own specified location, like the application directory, you need to add extra code, as has always been the case with ICS, but you'll need to change other defines as well since OpenSSL is now loaded by default before any code is run.  This is explained in depth in readme9, and other places. 

Angus

No, ICS will never contain an openssl-x64.exe file, since ICS does not use openssl.exe. 

Our OpenSSL zip distributions are separate for Win32 and Win64, so it is up to end users to do something if they want to combine them. 

SVN and the overnight zip do now correctly have the Win32 version of openssl.exe, the Win64 version was an error.

Angus