Angus Robertson

Setting up MTS/ATS for Windows is not trivial, Azure command line interface, new signing tool, Azure signing DLL, .net8 runtime, environmental variables, Azure commands, etc. 

Not helped by the Microsoft web site code and Json examples all using non-breaking spaces (xA0) which have to be replaced by real spaces to do anything useful.

Once installed, it attempts to sign my program, but fails with OAuth2 login using a browser, and with incorrect secrets, so many to choose from in Azure. 

So the documentation needs much clearer information of how Azure authentication should be set up for Windows apps.

Angus

In the progress of getting this working, setting anything up in Azure is horrible, I've got OAuth2 stuff, now waiting for company identity validation.  I've had MSDN and partner accounts for 30 years, used to get a box of CDs each quarter before the internet. 

Worth mentioning that if you have a partner MAPS account and pay annually (UK £350) to get all the MS OS and apps, you get $100 of Azure credit each month, which I assume can be used for code signing.

Two minutes later, validation pass.

Angus

I'm slowly adding TBytes versions of binary methods and properties as I update components, should be added in the next week or so.

Angus

Quote

10 hours so far and all I have had is a confirm email address

Probably handled manually in USA business hours until full launch.

Angus

I'll be happy not needing to deal with Digicert, Sectigo and co, particularly with their massive price increases in recent years.  Let's Encrypt stopped most of their web certificate business.

Microsoft does have a history of looking after developers, for many years there was a web page that allowed buying heavily discounted certificates from the major vendors at the time, it disappeared a few years ago. 

Angus

Thanks for all the useful comments, I'll try it myself when I have time later in the week.  

Got another eight months left before my code signing certificate expires, but looks like I can avoid dongle hell.  I've had an Azure account for years, although perhaps not the correct type of account. 

Angus

Generally, in the progress event, you check the LogLevel so that LoglevelProg is displayed, not logged every two seconds during the download, as you did in your example above, although does not really matter for small files. 

The component reports the HTTP reported size at the end and does not check the actual file, the snippet sample checks the real size after download.

Angus

I just tested the 'HTTP Rest Download' function in the snippets sample, that downloads a 300M file from my server, same code as yours, I get a 300M file.  How are you reporting the size downloaded?

Beware the V9.1 snippets sample won't download from my server any longer, I added server authentication to test something else, and you'll need a new snippets from SVN or the overnight zip that adds authentication as well.

Angus

Has anyone tried this service for signing applications, free until June 2024, then $9.99:

https://techcommunity.microsoft.com/t5/security-compliance-and-identity/trusted-signing-is-in-public-preview/ba-p/4103457

Angus

Beware TWinCertificate is only declared in Delphi 12, perhaps ImageHlp as well. 

But I'll look at those functions to allow verifying the signing chain using OpenSSL instead of the slow capicom.dll COM object used by my component at the moment, but not looked at the component for several years. 

Angus

This Win64 exception was due to using Free on a stream that was already closed, instead of NilAndFree which checks first, and is used elsewhere in the same function. 

Not sure why Win32 hides the bug, but Win64 does not, this was within finally/end.

Not in SVN yet, and the V9.1 HTTP snippet is now broken because last week I added authentication to the web server directory it uses to test something else, and forget snippets uses that directory.

Angus

The sample OverbyteIcsIpStmLogTst is the proper test bed for the TIcsIpStrmLog component.  For a successful remote connection, the events triggered are:

11:55:09 C[0] State: Starting
11:55:09 C[0] TCP/Client Opening Connection to [2a00:1940:0:c::128]:80
11:55:09 C[0] TCP/Client Connected OK
11:55:09 C[0] State: OK

(manually stopped)
11:55:24 C[0] State: Stopping
11:55:24 C[0] TCP/Client Disconnected
11:55:24 C[0] State: None

But for an unsuccessful connection it normally keeps trying to reconnect to the remote IP, so there is no immediate state change:

11:50:56 C[0] State: Starting
11:50:56 C[0] TCP/Client Opening Connection to [2a00:1940:0:c::128]:5000
11:51:17 C[0] TCP/Client Failed Connection - Connection timed out (#10060) - Retrying in 10 secs
11:51:27 C[0] TCP/Client Opening Connection to [2a00:1940:0:c::128]:5000

This is one of the 'features' of the component, if the client connection fails or drops it keeps trying to restore the connection, although that is exactly what you don't want! 

With RetryAttempts set to -1, the events called are:

12:06:45 C[0] State: Starting
12:06:45 C[0] TCP/Client Opening Connection to [2a00:1940:0:c::128]:6666
12:07:06 C[0] State: Stopping
12:07:06 C[0] State: None
12:07:06 C[0] TCP/Client Failed Connection - Connection timed out (#10060)
12:07:06 C[0] TCP/Client Disconnected

So you need to check Stopping and/or None for failure, there is no specific State: Failed event, unless you watch the literals for timeout out. 

Angus

Most ICS functions are async, not blocking, you missed IpLogClient.onLogProgEvent := onCliLogProgEvent; from snippets, and in the event you check IpLogClient.States[0] for logstateOK or something else if failed. 

As is often discussed here, it is rarely necessary or useful to use ICS components in a thread, but if so you can not use application.processmessages  since that is the main thread, you have to create the component within the thread Execute method, set the MultiThreaded property to true, and call the component ProcessMessages method instead.

Angus

Quote

Why TIcsIpStrmLog and not TSslWSocket directly?

The magic word was simplest, less code, fewer errors.

Quote

by using a non-blocking socket with select() or (e)poll() for the timeout.

Windows supports none blocking DNS lookups and sockets by using a thread, what is the API to stop those threads early?

Angus

The simplest way is with the TIcsIpStrmLog component, look at the doSocketRemoteClick procedure in the OverbyteIcsSnippets sample, although you can ignore SSL. 

Failing to open a normal TCP connection will timeout after about 30 to 40 seconds, you can not easily make this any shorter, and you can not re-use the socket until the connection attempt fails, even if you abort it earlier.  If you need to check a lot of ports, either use multiple components running in parallel (no threads needed for hundreds) or use the ping feature of TIcsIpStrmLog to see if at least the IP address exists before checking the port.  

Angus

{.$DEFINE OpenSSL_Resource_Files} will stop OpenSSL being linked, {$DEFINE] OpenSSL_ProgramData}  then helps determine where the DLLs are located. 

readme9.txt now has a lengthy section describing all the defines.

Angus

ICS V9.2 has started the beta process, and can be downloaded from https://svn.overbyte.be/svn/icsv9/ or the overnight zip from https://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp

This beta version of V9.2 adds a new feature release of OpenSSL 3.3.0 and fixes a number of bugs mostly introduced in V9.1, but also two long term HTTP URL in the client and server software where missing / delimiters could cause problems, and fixing server authentication issues with POST requests. Several others issues discussed on Delphi-Praxis are also fixed, but not all, yet.

This beta also includes a new 'ICS Intermediate Short' SSL certificate to replace the one in V9.0 that has just expired, it is used by ICS to generate temporary server certificates to allow SSL servers to run until a Let's Encrypt or commercial certificate is installed.  

The OverbyteIcsSslMultiWebServ sample has various improvements to test authentication more thoroughly (the DDService version is not done yet).

Only Delphi 10.41 and 10.42 (10.4 with updates 1 or 2) will install correctly with the new install packages, the original RTM version does not support the package LIB suffix: $(Auto) so you must change it manually for each package to 21.0.

OpenSSL 3.3.0 is now the default in the OverbyteIcsDefs.inc file and the ICS-OpenSSL path, ICS does not use any of the new features (nor those in 3.1 or 3.2).

Now that OpenSSL is more closely integrated with ICS, updating for security fixes will become more complicated, needing files in two or more directories to be updated.  When OpenSSL does the next batch of security fix versions (scheduled quarterly), I'll generate a zip with all the new files and directories that can be extracted over an existing ICS installation with all the new files.  

Angus

You are concerned about a one second longer download on a tiny file?   With different SSL protocols and ciphers. 

Angus

In HTTP headers, there is always a space after the colon, seems simple but sufficient to confuse servers.

Angus

TSslHttpRest is an ICS component, and to add a special header field you use the component ExtraHeaders: Strings property to add the full header and value,

ie ExtraHeaders.Add('Store-Token: 22345673301244567896663456789012');

Angus

I believe the authentication POST problem was mainly a simple literal,

    if FOutsideFlag and (not (hoAllowOutsideRoot in FOptions)) then
        Flags := hg403
    else
        Flags := hg404;

where hg404 should be hgSendDoc.  But something else is going on I'm still tracking, RequestDone should be called for a 401 error to reset the state machine, but is not, although it still seems to work. 

The biggest problem is our samples test all the authentication variations for virtual and normal pages, but not for template pages or POST pages, so that all had to be added first. 

I always test server fixes on my public servers for a day or two, so the changes won't be in SVN until later in the week.

Angus

On 3/22/2024 at 9:17 AM, Angus Robertson said:

3.3 has just entered alpha testing and adds QUIC for HTTP/3 servers.

Correction, OpenSSL 3.3 was released last week and does not add QUIC for servers, that is scheduled for OpenSSL 3.4 due in October 2024.

https://github.com/orgs/openssl/projects/11/views/3

OpenSSL 3.3 for Windows will be released later this week with ICS V9.2 beta.  But there are no new features particularly relevant to ICS.

Angus

I use been using the same two news readers for 25 years, Forte Agent and Ameol2 (very specialised for the UK).  

I read a number of uk news groups.  But there are no active Delphi groups, since Embarcadero moved away from a web forum that allowed NNTP access.

I have an account with https://www.astraweb.com/ they sell fixed usage so 25GB for $10 that will last for ever if you don't download binaries. 

One free news server is news.gmane.io that holds several mailing lists, I read OpenSSL mailing lists via it. 

Angus

On 3/27/2024 at 10:42 PM, omnibrain said:

I used the postinfo-Demo, because I think you added that specifically to test POST.

\demos-data\WebAppServerData\Templates\postinfo.htm is missing, but that doesn't matter for the test. 

Took some detective work to work out where that template has gone, it was written last autumn to test a major rejig of web server get/post parameter processing using streams for multi-gig uploads.

But the template never got added to SVN and then got lost when the samples were re-organised. 

So I've had to recreate it from my public web site: https://www.telecom-tariffs.co.uk/testing/postinfo.htm

Not in SVN yet, still looking at your real problem.

Angus

Still got no idea why are distributing openssl.exe and why this is a problem.  The directory your screen shows has the correct DLLs for the EXE. 

We don't need two separate openssl.exe files since we don't use them.

The only problem here is you decided to change the way OpenSSL is distributed for your own reasons.

Angus