Angus Robertson

Uploading files is built into the TSslHttpRest component. 

The main OverbyteIcsHttpRestTst sample has a menu option Upload File, select MIME Multipart, specify the Upload File Name, POST or PUT, and any REST parameters needed, it just works. 

The OverbyteIcsSnippets sample has a much simpler demo where you click a single button 'HTTP POST Upload File' which runs a single function to upload a file to one of my servers. 

Angus

The latest ICS offers OAuth2 with embedded Edge or MSIE browsers or external browser and a private web server.

Pre-built samples at https://wiki.overbyte.eu/arch/icsdemos-clients.zip

Angus

Look-up base64 conversion in the help, then break the lines to 64 long and add the --- header and footer you see in a PEM certificate file.

Angus

 OpenSSL has released new versions of the three active branches.

These releases fix a low severity issue with the Win64 POLY1305 MAC implementation corrupting XMM registers ([CVE-2023-4807]) which  applies only to processors released in the last two or three years supporting AVX512-IFMA instructions.  
 

Windows binaries are available in SVN and the overnight zip file (tomorrow) and separately from https://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp

Separately, YuOpenSSL has released 3.0.11 and 1.1.1w as commercial DCUs allowing applications to be used with OpenSSL without needing separate DLLs.

Note that 1.1.1w is the last release of that version, it is now out of support unless you have an OpenSSL Premium Level Support contract for $50,000/year.

Angus

ICS will now build again with the OpenSSL DLLs.

Angus

Sorry about that, new function added this week, tested with YuOpenSSL, but obviously need to update the DLL imports as well, will be done today.  Just use the old wsocket unit until it's done.

Angus

A PEM certificate is base64 encoded lines of the binary DER, with a header and trailer added, although ICS often adds comments to the files for identification purposes.  So could just strip off the headers, remove line endings and do a comparison of the base64 text.

Is your server also ICS?  So it could be updated for Raw Public Key TLS support?

Angus

There are several global variables used to determine how OpenSSL is loaded, this chunk of code is from the OverbyteIcsHttpRestTst1 unit, but most ICS samples have something similar. 

// Avoid dynamical loading and unloading the SSL DLLs plenty of times
//  GSSLEAY_DLL_IgnoreNew := True;     { ignore OpenSSL 3.0 and later }
//  GSSLEAY_DLL_IgnoreOld := True;     { ignore OpenSSL 1.1 }
// note both not allowed true
    GSSL_DLL_DIR := FProgDir;          { only from our directory }
    GSSL_SignTest_Check := True;       { check digitally signed }
    GSSL_SignTest_Certificate := True; { check digital certificate }
    OverbyteIcsWSocket.LoadSsl;
    if NOT GSSLStaticLinked  then begin
        if NOT FileExists (GLIBEAY_DLL_FileName) then
            LogWin.Lines.Add('SSL/TLS DLL not found: ' + GLIBEAY_DLL_FileName)
        else
            LogWin.Lines.Add('SSL/TLS DLL: ' + GLIBEAY_DLL_FileName + ', Version: ' + OpenSslVersion);
    end
    else
        LogWin.Lines.Add('SSL/TLS Static Linked, Version: ' + OpenSslVersion);   

This version ensures the DLLs are only loaded from our own directory using GSSL_DLL_DIR, since Windows may have dozens of different versions of the DLLs scattered around the drive from different applications.  The code loads OpenSSL once and tells you what version it found and where, and whether YuOpenmSSL is being used which avoids all DLL problems. 

Might have to revisit the IgnoreNew/Old stuff since 1.1 is now out of support. 

Angus

This fix is now in SVN and the overnight zip, property X509PubKeyTB in TX509Base will get the certificate public in DER binary format as TBytes, from where it may be converted to hex or base64, and used for Raw Public Key certificate validation. 

At least assuming the raw key distributed is the DER format.  It seems comparing a SHA256 digest is common, rather than a long string. 

Angus

This improvement is now in SVN and the overnight zip, there is a new TOAOption of OAopAuthBasic that causes Basic Authentication to be used instead of sending the client ID and secret as parameters, Microsoft accepts both methods.

Angus

I explained how in the third line:

nStatCode := RestRequest(httpPOST, 'https://xx', False, astrJSON.Text);

You can also look at the new OverbyteIcsSnippets sample in ICS v9 which has working examples of many ICS functions, just click a button and watch it happen, although you are almost there already.

But do use the debugs options, you'd then have seen the modified Json being sent, and the problem should have been glaringly obvious.

Angus

Quote

RestParams.AddItem('raw', astrJSON.Text);

Are you intending to nest Json here, using a parameter like this will create a new Json item named raw with your Json as the value. 

Except when adding raw parameters like Json, you need a third parameter True so that the item is not escape encoded. 

If your Json is complete, you can ignore RestParams and just add it as RawParameters which is the last argument in RestRequest. 

The component has built in logging, set DebugLevel to DebugBody, assign the onHttpRestProg event, and write everything that arrives to your memo,

Angus

Sorry, single line Windows errors in isolation are no-use.  I asked you to reproduce the error in the new IcsSslMultiWebServ sample and send me the log it generates.

Note that both KeepAliveTimeSec and KeepAliveTimeXferSec close the remote client connection, the latter after five minutes by default. If you want to keep an idle remote client open for more than five minutes, you have to set both to zero.  Neither will stop the server.

Angus

ICS v9 has a lot of low level stability changes, many for Win64 applications, in particular errors during close and within finally statements, where inherited functions were sometimes skipped.

I only built my own public servers as Win64 about a year ago, which means that is when Win64 got a lot of extra live testing.   And I'm still uncertain about the OpenSSL DLLs in some very rare circumstances with Win64. 

Angus

It seems the industry has come up with RFC7250 Raw Public Key to allow authentication of TLS connections without using X509 certificates and trusted certificate chains, which seems aimed at your type of LAN application.  You are really doing the same thing, but still sending a self signed certificate you ignore.  

RPK uses TLS extensions, which are supported by the next OpenSSL release 3.2 currently in alpha testing.  I'll look at supporting RPK when it arrives.

Meanwhile, I'll add a TX509Base method to get a public key this week.

Angus

KeepAliveTimeSec only effects remote clients not the server itself.  Are you saying a remote client connection was closed after five minutes, or the web server itself stopped after that time?  There is no server timeout or close down.  

Please private email me the complete log file from the ICS sample showing this activity, not just little extracts. 

Angus

Stopping and disabling the Windows Update service worked in the past, but Microsoft may have other apps that monitor it being stopped.

Angus

KeepAliveTimeSec is an idle timeout, from when the last data was received or sent on the client connection.  It is part of http/1.1 that leaves the connection open after a request, waiting for further requests and defaults to 10 seconds in all ICS web servers, XferSecs is five minutes.  It's not changed in years, except to support Int64 ticks. 

Quote

we are using ICS_FMX-8.69-11

Is this Windows Server or MacOS?  Why FMX for Windows? 

ICS v9 has the first new FMX samples in 10 years, I suggest you build the FMX sample IcsSslMultiWebServ and see if you can reproduce the problem, if so I'll look into it.

Angus

When you say 'big performance regressions', if I read your article correctly you mean some cryptographic functions are slower in new versions, due to the provider layer that hides internal structures from being damaged by applications and allows flexibility for developers. 

The question is how many times a second are those operations performed in a typical web client or server, so what is the actual penalty?  Or is it once or twice a connection, so microseconds?

Angus

My public web servers have the line:

        Client.KeepAliveTimeSec := 120 ; // 13 July 2009 increase session timeout

and serve thousands of users a day without halting since that line was added.  I don't use KeepAliveTimeXferSec which is only effective during requests,

Both versions close the client, not the server.  Which ICS version are you using.

Angus

The OverbyteIcsOAuthFormVcl unit is very much optional, it is new in this release and not even built for Delphi 7 since it does not have the browser windows.  But a couple of samples will complain. 

SHDocVW is not in any ICS packages.

We only provide very limited for support for very old compilers, except Delphi 2007 which is widely used, my XE license has expired so I can not use it, only some of the later XE versions.

Angus

Thanks, I'll look at this when I get back to ICS next week. 

Note, there is an ICS forum for future support, but you don't need to repost this topic.

Angus

It is no secret that many or most components developers are beta testers under NDA, how else are all their components ready for each new release, or in the olde days on the component companion CD included with the final release.

This benefits everyone involved, because new versions of Delphi can be used for old projects immediately, rather than waiting weeks for developers to buy the new version, etc.

What has changed in recent years is beta testing being offered openly for paying customers, rather than by invitation only, and blogging about the next release, so it is now all more obvious.

Angus

I assume you mean the files OverbyteIcsDXeRun.dpr and OverbyteIcsDXeRun.dproj?   SHDocVW is not in either of those files so your compiler must have added it. 

TWebBrowse did keep changing in early versions of Delphi, ICS has several painful conditionals to try and make it work.  You can try removing OverbyteIcsOAuthFormVcl, and refeences to it, that might help. 

Angus

Component libraries can be published that have been tested on beta versions, provided they don't reveal any features of the beta or use any new features.

The latest ICS v9 release has packages for Delphi 12, but did not need any changes for D12 other than VER360.   Several other libraries are also available from GetIt for the beta. 

Angus