ISC 9.1 as a library events not firing

[sfrazor 3]

Some background.  I'm replacing a C console WS app with a Delphi console app, eventually a DLL.  The C WS is libwebsockets.  Using ICS, SSL (YuOpenSSL), TSslcontext and TSSLWSocket.

in the constructor, I want to assign class procedures for items such as listed below.

constructor TMyWSComms.Create(AOwner: TComponent);

begin

    inherited create(AOwner);

    Sslcontext:= TSslCOntext.create;

    SslEnable:= True;

    Sslcontext.SslVerifyPeer:= True;

    addr:= 192.168.100.40;

    port:= 443;

   // overrides with class methods

   OnSessionConnected:= MyOnSessonConnected;

   OnSessionClosed:= MyOnSessonClosed;

   OnStateChanged:= MyOnStateChanged;

   OnSslVerifyPeer:= MySslVerifyPeer;   

// several more.....

end

 

Using ICS as VCL this all works fine and events fire as expected. But not as a console app, It seems I'm missing a message pump?  

With ICS  I'm running in to a couple of issues:

First, there are built in message pumps available in ICS but I'm not sure how to implement that or if that is even what I need.  If there is an example of this, please point me to it.

Second, Even though I Assign/create a SslContext I get an error on socket shutdown inside ICS saying Sslcontext is required/missing.  IJ can provide the source and line number in ICS if needed.

 

 

Last, with libwebsockets there are 3 params to set for a connection (more if you include the socket and ssl setup) 

address

port

path

 

So if I want to connect to 192.168.100.40:443/ws/interface

 

I set the above accordingly:

address:= 192.168.100.40

port: 443

path '/ws/interface'

 

where /ws/interface/ is an expected python tornado connection path.  How do I construct this using ICS?   Sounds trivial but I'm completely missing it.

 

 

 

But with ICS there is no path equivalent that I see.  

 

[Angus Robertson 577]

There is a terminology issue here, Websocket is a specific protocol based on HTTP.  TWSocket is the name of the base ICS component for all low level socket operations, and does not handle high level protocols like Websocket.

 

ICS has a TSslWebSocketCli component, there is a simple function doWebSocketClick in the OverbyteIcsSnippets sample you can build and run which accesses one of my servers, or a full example in the OverbyteIcsHttpRestTst sample that should allow you to access your site.

 

ICS has several console samples, but they are no longer in the main distribution not being needed very often, I'll have to look one out.

 

Angus

 

 

 

 

 

 

 

 

[sfrazor 3]

My apologies for the confusion.  Yes Websocket as a protocol.  I understand that.   I was originally using OverByteSimpleSslCli as an example reference and it uses TSslWSocket and TSslContext and it seemed like a great starting point.  So I just now switched to TSslWebSocketCli and can do a little refactoring.  But in regards to a console example and implementing events would be very helpful.  I'll for sure checkout the Snippets.  Thanks Angus.

[Angus Robertson 577]

There is an old console HTTP sample OverbyteIcsConHttp which is now in the archived samples download https://wiki.overbyte.eu/arch/arch-samples-V9.1.zip

 

Essentially, it just has a message loop after the request,

FHttpCli.MessageLoop;

and the RequestDone event has

FHttpCli.PostQuitMessage;

which causes the loop to break and the application to die. 

 

Alternatively, you can call ProcessMessages in your own loop checking stuff until you are ready to break. 

 

I'll make sure the next release has at least one console SSL sample.

 

BTW, you don't need an TSslContext with the new high level components, it's created internally.

 

Angus

 

 

[sfrazor 3]

OK switched to TSslWebSocketCli.  This component seems much more suitable.  Lots of extra options...  and the url setting works perfectly for concatenating my WS path together.   

 

This app is doing its own verifypeer with self signed certs.  I'm missing a setting because I cant seem to get verifypeer event to fire using TSslWebSocketCli.  It does using TSSLWSocket.   

 

How do I reference the automatically generated sslcontext you mentioned?  or am I misunderstanding.    Without assigning sslcontext, I cant set the SslContext.VerifyPeer:= True.  It seems to want me to add/assign the sslcontext. 

 

I need to disable all automatic validation and rely on the OK check in the verifypeer proc.  Plus I'm suffering the same issue with events working in my test VCL app but not being called in the console app.  Do I just call the components .messagepump  method before the wsconnect?  Going to test that in a few....

 

One issue I have to  yet investigate.  Can I queue the data and when the socket becomes writable - send it?    If not, how do I  use TSslWebSocketCli to determine if the socket is ready/writable?

 

Last, my apologies for not dropping code here.  My dev network is air-gapped so I'd have to type it in a second time.  It'll take a bit but I can if you say you need to see it.

[sfrazor 3]

Using TSslWebSocketCli,  .messagepump did not make any difference.

Also in the VCL app, after setting the url - 'wss://192.168.100.40:443/ws/data' it connects but issues an additional GET with an error reported on my server "GET /ws/data HTTP/1.1" error 404,  for some sort of synchronization?  

This component seems to be geared toward http(s) rest GET PUT etc instead of straight forward socket ssl connections via a url and read/write to the socket.  

I also noticed OnStateChange no longer passes the previous and current state during state change.  That was handy but not absolutely required IMO.

 

I think I'm missing the correct usage.  All I want to do is a WS SSL connection either via url or IP , PORT and path (preferably url), validate my cert and read/write data  to and from the socket.   All in a console app. 

[Angus Robertson 577]

The purpose of the Websocket is to exchange data with a server, so it opens a connection and does a GET to the path you specify, just as HTTP does, if you don't want that it's not Websockets.

 

I suggest you try using the OverbyteIcsHttpRestTst sample I mentioned, if that does not work you would appear to be using a custom protocol.

 

In a console application, you need to call the message pump after every line that accesses IP functions, ICS is fully event driven. The sample I mentioned makes a single HTTP command, waitis for it to finish and exits.  If you are making a series of commands, the console applications gets more complicated. 

 

Angus

 

 

 

[sfrazor 3]

Thanks Angus.  I traced the GET and its required during the http upgrade to WS.  Thanks for pointing that out.  Messagepump after the calls did the trick!

Setting the following got me to a point where VerifyPeer is is being called and I can validate the cert.
SslVerifyPeer:= MySslVerifyPeer;

sslcontext.SslVerify:= True;

SslAllowSelfSigned:= True;

CertVerifyMethod:= CertVerBundle;  <---- T his works, but is this the correct verification setting for ignoring cert validation and using VerifyPeer?  

 

I'm going through more examples today.  Is there an example that shows data placed in a buffer and be sent when the socket is writable?  LWS has a callback that pushes data whenever there is data in the buffer and the socket state is writeable.  I'm still looking through examples 🙂

 

Quote

BTW, you don't need an TSslContext with the new high level components, it's created internally.

I'm not sure I understand how this works since TSslWebSocketCli is still needing the TSslContext to be created.

Thanks again for supporting newbies!

[Angus Robertson 577]

CertVerifyMethod := CertVerNone is how you bypass internal chain verification.    

 

 TSslWebSocketCli is derived from TSslHttpRest which has an internal TSslContext, and it's that component that does all the SSL stuff.  You can use an external SslContext, see the notes in the rest component, for instance if you are using a hundred rest component instances to download lots of stuff in parallel. 

 

If you are using the real WebSocket protocol, it exchanges frames between client and server, and vice versa, and there are several methods WSSendText, WSSendBinary, WSSendFrame, etc, with the OnWSFrameRcvd event being received frames, which you need to check the data type and then process.  IsWSConnected does as it says.  

 

If you want to send raw data without using WS frames, you are using the wrong component. 

 

Angus

, 

[sfrazor 3]

On 8/23/2024 at 3:13 AM, Angus Robertson said:

CertVerifyMethod := CertVerNone is how you bypass internal chain verification.    

Angus

Using TSslWebSocketCli, after setting CertifyMethod to CertVerNone, verifypeer no longer gets executed.  But the Websocket connects and is ready.  If it completely bypasses all internal chain verification AND does not fire verifypeer (which it seems to behave the way you described), how do I do a manual cert verification? 

These are self signed certs and we validate using hashes, serialnum, pubkey(Mod N) etc...  whatever the cert provides that is unique.  

Setting the  CertifyMethod to CertVerBundle or CertVerdWinstore fires verifypeer but the certificates fail regardless of the _OK_ flag.

 

I'm looking for something that may be described as CertVerSelf.  This would fire the verifypeer to be used without other verification and the _OK_ would signify pass/fail of the cert.

Is there a method in place to accomplish this already?

 

 

Edited August 28 by sfrazor

[Angus Robertson 577]

You are trying to do something unusual, I'll need to look at the code before I can make any suggestions, may be a day or two.

 

This has been done before for light weight clients.  Also OpenSSL now supports RFC7250 to check certificates by raw public key, which I plan to support.  

 

Angus

 

[Angus Robertson 577]

The best way to handle self signed certificates is the way ICS does it, the server components use an ICS intermediate certificate to create CA signed certificate if a real one is missing.   

 

ICS includes two intermediates signed by the ICS root, a short 200 day intermediate version for which the private key is distributed, and a two year one for our use only. 

 

You can use these or create your own versions with ICS samples or components, create your own CA signed server certificates, or if using our servers let it happen automatically, and ensure your clients have the root certificate in the store, only that certificate no others are necessary.  The ICS root gets loaded automatically for all ICS applications.

 

If you want to use real self signed certificates, TX509Base has a method X509PubKeyTB that gives you the public key, save it on the server and distribute to your clients in some way (instead of a root), extract the public key from the certificate in the OnHandshakeDone event and compare it with your saved version. 

 

The only catch when using the high level components with built in chain checking is there is no chain if you disable checking, but if you enable it, our checking can not be skipped. I'll need to fix this somehow when I add public key checking.

 

Angus

 

 

 

[sfrazor 3]

On 8/23/2024 at 3:13 AM, Angus Robertson said:

You are trying to do something unusual, I'll need to look at the code before I can make any suggestions, may be a day or two.

 

This has been done before for light weight clients.  Also OpenSSL now supports RFC7250 to check certificates by raw public key, which I plan to support.  

Yes.  X509PubKeyTB Was the result of a previous thread here where I was doing the same validation with the ICS HTTPS Client.   I added the code to pull the Mod(N) from the public key. which somehow became the standard in our shop to validate self signed certs.  I ran in to this exact issue there not being able to use OnVerifyPeer as a final validation.  This time around I'll opt for a more simple validation like described above.  I did try to validate in the OnHandshakeDone but no luck with the OK flag overriding the chain validation.  In OnVerifyPeer, if the OK flag would override any internal chain validation, that would work.

 

Its ugly logic on our part.  Its like....  I don't care what the cert validation chain says, If I say its valid, its valid.  The Delphi NetHTTPClient uses this exact logic in their OnVerifyPeer override, which works great.  If its assigned, all validation relies on the OnVerifyPeer OK flag.  Also, I would think the purpose of having the VerifyPeer would be this exact case and solve RFC7250.  Maybe?

 

I'm going to press forward with my development because ICS WebSockets is very feature-rich and I'll need some of those features.  Plus, I have faith you'll present a solution soon 🙂

 

Regards

 

[Angus Robertson 577]

Effectively, when validating the chain, OpenSSL and ICS are checking the chain is signed by a public key in a certificate in the store, you are just trying to skip the certificate overhead. 

 

I've found the new OpenSSL APIs to make the server use a raw public key instead of a certificate, and likewise the client to check it, now need the time to test it all.

 

Angus

 

 

 

[Angus Robertson 577]

I've made a change to TSslHttpRest adding a new way for applications to check SSL certificate chains themselves, ignoring OpenSSL bundle checks, usually for self signed private certificates.
if CertVerMethod = CertVerOwnEvent, during OnSslHandshakeDone the component calls a new event OnSslCertVerifyEvent where the application can check the chain and change the verify result appropriately. Maybe checking certificate serials, names or public key.

 

Not tested yet, need to update the sample with the new event and some sensible code to check a self signed certificate, not until next week.  Supportng the new OpenSSL for raw public keys will take longer, need to do the server stuff first so I can test clients.

 

Angus

 

 

[Angus Robertson 577]

The new certificate chain checking event is now in SVN, look at the OverbyteIcsHttpRestTst sample.

 

Angus

 

[sfrazor 3]

Thank you!  Headed over now!