sfrazor

When calls to WSSendBinaryStream happen, I'd like to verify the contents went and are not buffered. If even does buffer (for the lack of a more precise term). if WComms.IsConnected WSSendBinaryStream(nil,Outdataa); I have events for OnSocketError and OnBGException and onWSFrameSent, checking AFrame.bytes. I am sending 3 4M files back to back. It looks as though all files were sent but after the first file sends and fires the on onWSFrameSent event, the last two files don't fire the event but I don' see any errors. Its like they are buffered or staged. If I send them again, the previous 2 files will send and fire the event, but it leaves 2 more either buffered or??? Sometimes all 3 files send, sometimes 2 etc..... But a fresh send will push the prefvious out and fire the onWSFrameSent. How can I verify all 3 frames were sent and if not, flush any remaining or resend? Regards Update: If I send the files one at a time, they transfer fine. In a loop, sending back-to-back seems to be the issue.

Angus, In a previouse quesiton I asked about high CPU usage. We found that MsgWaitForMultipleObjects worked to resolve the issue. but sleep(0) did not. I tested with single files and transfer speeds were great and CPU was normal. In my mutiple file send scenario, if I remove the MsgWaitForMultipleObjects and let the CPU run at ~50 percent in that main loop, all of the files transfer without issue. No other code changes. I'm really at a loss. Some more info... after every call to any kind of socket IO I immediately call processmessages. SInce this is a console app and will eventually be a DLL the messages need to be addressed. This is something else you pointed out to me some time back. The file send is in its own thread looking for the existence of files, reads them and sends them. Nothing magical. There is a Sleep(5000) inbetween checks.

It works really well for one file at a time. 🙂 Its like its not ready to send when I call the WSSendBinaryStream in a loop but it takes in the data in to a buffer and it sits there. It seems out of sync with the socket state. Its like I need to wait for a state like httpready or something before I call the send. The server is a python server running Flask utilizing websockets. The server has been solid for some time but that doesn't mean there is not an issue. Its like the ICS client isn't receiving an ACK from the last frame sent, if it was infact sent Or the Comms is not in the right state. Or the entire cycle is too fast to process..... Slowing it down with sleeps in between doesn't seem to have any effect..... But I'm not sure how to verify the socket is in the ready-to-send state. I don't hink its fair to ask you to reproduce the client and server. So instead just give this some thought and get back if you come up with something that would be a worth try. I'm still working at it on my end. Again I have to apologize. Everything I do is on an air-gapped network so poulling code over is often not possible. Thank! Regards

Using OverbyteIcsSnippets as an example. Websocket CLIent section, after wsconnect is successful it falls in to this loop. while NOT AbortFlag do begin Application.ProcessMessages; if Application.Terminated then Break; if IcsTestTrgTick64(TrgTimerEnd) then Break; end; Mine is a console app and I do something almost identicle to this on startup. I connect then fall in to the main thread loop waiting for a signal to end. My application runs at 48% cpu usage just like the example. I realize this is a tight loop. Running as-is the OnWSFrameRecv will receive a 50M file in a few seconds. However when I try to address the high cpu issue and add a simple sleep(5), which drops the CPU down to less than 5 percent. The 50M file receive via the OnWSFrameRecv event now slows down to 9 minutes. Its like the receive is blocking. I'm not sure why a simple sleep would do this. Is the sleep cascading in the tight loop and causing the overall main thread to sleep much longer than the 5ms maybe? I know there is missing info here. The application is quite large but the websocket connect and receive is very basic. I rebuilt excluding throttling in the library but it didn't make any difference. Is there a better way this not CPU intensive to loop and wait for the termination? The goal is to receive large files via WS and not have the appl;ication idle at this high usage. Any help would be appreciated. Regards

I was sifting through the source looking for examples of this very issue. I found MsgWaitForMultipleObjects right after you posted. I had tried sleep(0) with no success. But the MsgWaitForMultipleObjects worked and now the CPU is back down to .3 % and file transfers are back to their normal fast speed. Thanks!

First, is there specific settings when running as a DLL? What is the right way to configure the component to run as DLL? I realize this is an event driven component but I can't seem to figure out how to get the events to fire properly when not a EXE. I have a test console app(exe) and it works great doing a wss connect to the host. When building and running the DLL, after the Connect it pauses for quite some time. May be a minute or so. The onConnect event doesn't seem to fire even though the server sees the inbound connection. The Connect is in a While NOT connected. Looks something like this procedure WSComms.ConnectHost; var Retries: integer; begin Retries:= 0; WSComms.WSPingSecs := 5; WSComms.Timeou t:= 0; WSComms.ResponseNoExceptions := True; WSComms.url:= 'wss:\\192.168.1.30' While (NOT WSComms.Connected) do begin sleep(2000); WSComms.ProcessMessages; //server sees the connectoin and responds but messages/events don't fire in the DL only WSComms.Connect; WSComms.MessagePump; Inc(Retries); If Retries > 5 then Exit; end; end; procedure WSDoconnection; begin //attempt connectoin until connected WSComms.ConnectHost; While (NOT AbortFlag) do begin WSComms.ProcessMessages; end; end; So this should attempt a connection around every 2 seconds or so..... But it stays on Connect for a good minute or more and doesn't actually connect. Again, the exe version works perfectly.... So its an events issue.

I updated the above code hopefully for a little more clarity. So let me ask this a different way. In the above example WSConnect reaches the Server. The Server responds. WSConnect being a blocking call just hangs. IE no Windows messaging seems to be happening. I understanding writing a message pump, but ICS has both ProcessMessages and MessgePump build in to the library. So I am attempting to use those. OverbyteIcsSnippets1.pas does not address this as it uses Application.ProcessMessages which is not available to a DLL. I have tried various ways and places to insert ProcessMessages and MessagePump as you can see above with no luck. OnWSConnected event does not not run OnWSFrameRcvd event does not run What is the proper way to use the ICS MessagePump so t hat I can properly process messages throughout the DLL? Again, the console app works perfectly. When I switch it from a console app to DLL, WSConnect blocks and never returns, because I believe Messages aren't being processed.

Yes, but that doesn't answer the events question. How do I get events to process in a connection loop when the events don't fire. And WSConnect is the method I'm using. That loop was off the top of my head. Meant to illustrate a connection attempt every few seconds until it was connected without timing out. I'd pull over the actual code but I work on an air-gapped network so, I can't.

Some background. I'm replacing a C console WS app with a Delphi console app, eventually a DLL. The C WS is libwebsockets. Using ICS, SSL (YuOpenSSL), TSslcontext and TSSLWSocket. in the constructor, I want to assign class procedures for items such as listed below. constructor TMyWSComms.Create(AOwner: TComponent); begin inherited create(AOwner); Sslcontext:= TSslCOntext.create; SslEnable:= True; Sslcontext.SslVerifyPeer:= True; addr:= 192.168.100.40; port:= 443; // overrides with class methods OnSessionConnected:= MyOnSessonConnected; OnSessionClosed:= MyOnSessonClosed; OnStateChanged:= MyOnStateChanged; OnSslVerifyPeer:= MySslVerifyPeer; // several more..... end Using ICS as VCL this all works fine and events fire as expected. But not as a console app, It seems I'm missing a message pump? With ICS I'm running in to a couple of issues: First, there are built in message pumps available in ICS but I'm not sure how to implement that or if that is even what I need. If there is an example of this, please point me to it. Second, Even though I Assign/create a SslContext I get an error on socket shutdown inside ICS saying Sslcontext is required/missing. IJ can provide the source and line number in ICS if needed. Last, with libwebsockets there are 3 params to set for a connection (more if you include the socket and ssl setup) address port path So if I want to connect to 192.168.100.40:443/ws/interface I set the above accordingly: address:= 192.168.100.40 port: 443 path '/ws/interface' where /ws/interface/ is an expected python tornado connection path. How do I construct this using ICS? Sounds trivial but I'm completely missing it. But with ICS there is no path equivalent that I see.

Thank you! Headed over now!

Yes. X509PubKeyTB Was the result of a previous thread here where I was doing the same validation with the ICS HTTPS Client. I added the code to pull the Mod(N) from the public key. which somehow became the standard in our shop to validate self signed certs. I ran in to this exact issue there not being able to use OnVerifyPeer as a final validation. This time around I'll opt for a more simple validation like described above. I did try to validate in the OnHandshakeDone but no luck with the OK flag overriding the chain validation. In OnVerifyPeer, if the OK flag would override any internal chain validation, that would work. Its ugly logic on our part. Its like.... I don't care what the cert validation chain says, If I say its valid, its valid. The Delphi NetHTTPClient uses this exact logic in their OnVerifyPeer override, which works great. If its assigned, all validation relies on the OnVerifyPeer OK flag. Also, I would think the purpose of having the VerifyPeer would be this exact case and solve RFC7250. Maybe? I'm going to press forward with my development because ICS WebSockets is very feature-rich and I'll need some of those features. Plus, I have faith you'll present a solution soon 🙂 Regards

Angus Using TSslWebSocketCli, after setting CertifyMethod to CertVerNone, verifypeer no longer gets executed. But the Websocket connects and is ready. If it completely bypasses all internal chain verification AND does not fire verifypeer (which it seems to behave the way you described), how do I do a manual cert verification? These are self signed certs and we validate using hashes, serialnum, pubkey(Mod N) etc... whatever the cert provides that is unique. Setting the CertifyMethod to CertVerBundle or CertVerdWinstore fires verifypeer but the certificates fail regardless of the _OK_ flag. I'm looking for something that may be described as CertVerSelf. This would fire the verifypeer to be used without other verification and the _OK_ would signify pass/fail of the cert. Is there a method in place to accomplish this already?

Thanks Angus. I traced the GET and its required during the http upgrade to WS. Thanks for pointing that out. Messagepump after the calls did the trick! Setting the following got me to a point where VerifyPeer is is being called and I can validate the cert. SslVerifyPeer:= MySslVerifyPeer; sslcontext.SslVerify:= True; SslAllowSelfSigned:= True; CertVerifyMethod:= CertVerBundle; <---- T his works, but is this the correct verification setting for ignoring cert validation and using VerifyPeer? I'm going through more examples today. Is there an example that shows data placed in a buffer and be sent when the socket is writable? LWS has a callback that pushes data whenever there is data in the buffer and the socket state is writeable. I'm still looking through examples 🙂 I'm not sure I understand how this works since TSslWebSocketCli is still needing the TSslContext to be created. Thanks again for supporting newbies!

Using TSslWebSocketCli, .messagepump did not make any difference. Also in the VCL app, after setting the url - 'wss://192.168.100.40:443/ws/data' it connects but issues an additional GET with an error reported on my server "GET /ws/data HTTP/1.1" error 404, for some sort of synchronization? This component seems to be geared toward http(s) rest GET PUT etc instead of straight forward socket ssl connections via a url and read/write to the socket. I also noticed OnStateChange no longer passes the previous and current state during state change. That was handy but not absolutely required IMO. I think I'm missing the correct usage. All I want to do is a WS SSL connection either via url or IP , PORT and path (preferably url), validate my cert and read/write data to and from the socket. All in a console app.

OK switched to TSslWebSocketCli. This component seems much more suitable. Lots of extra options... and the url setting works perfectly for concatenating my WS path together. This app is doing its own verifypeer with self signed certs. I'm missing a setting because I cant seem to get verifypeer event to fire using TSslWebSocketCli. It does using TSSLWSocket. How do I reference the automatically generated sslcontext you mentioned? or am I misunderstanding. Without assigning sslcontext, I cant set the SslContext.VerifyPeer:= True. It seems to want me to add/assign the sslcontext. I need to disable all automatic validation and rely on the OK check in the verifypeer proc. Plus I'm suffering the same issue with events working in my test VCL app but not being called in the console app. Do I just call the components .messagepump method before the wsconnect? Going to test that in a few.... One issue I have to yet investigate. Can I queue the data and when the socket becomes writable - send it? If not, how do I use TSslWebSocketCli to determine if the socket is ready/writable? Last, my apologies for not dropping code here. My dev network is air-gapped so I'd have to type it in a second time. It'll take a bit but I can if you say you need to see it.

My apologies for the confusion. Yes Websocket as a protocol. I understand that. I was originally using OverByteSimpleSslCli as an example reference and it uses TSslWSocket and TSslContext and it seemed like a great starting point. So I just now switched to TSslWebSocketCli and can do a little refactoring. But in regards to a console example and implementing events would be very helpful. I'll for sure checkout the Snippets. Thanks Angus.

You would have to ask public opinion on this one. Most would probably say 'why bother doing this at all'. To a limited few it would be a blessing. I would use it in my current project depending on how large adding the resource makes the DLL's overall. A couple of things should be noted: As just mentioned to me by Fr0sT.Brutal there is no auto-run. DLLMain calls are not supported that I am aware of. So if you are counting on DLLMain, OpenSSL wont initialize. Currently when I call SSL_CTX_new it fails ONLY when called from a memoryloaded DLL. This may be the root of a failure I'm currently experiencing. I need to test that theory and discuss with Ralf (YuOpenSSL). Someone please correct me if I'm wrong. Next, and I'm saying this without solid verification, exception handling in memorymodule loaded DLL's doesn't work maybe due to .SAFESEH section of the DLL not being mapped/loaded. If this is the case Windows won't allow SEH Handlers because it believes they are not safe. It will crash the app using memorymodule. This needs to be fixed. I'm trying to understand this part now so anyone wanting to correct me please do so. I am no authority on this. and last, BTMemorymodule is only 32bit last time I looked. Has that changed? Blackbone an Dark Memory Loader both correct the above issues in C but there is no equivalent in Delphi unless that is stashed away in private repos. I'm just not sure which one would be a viable repo to fork from.... So, Angus, proceed with caution 🙂

I have used both MemoryModuyle from Fr0sT.Brutal and BTMemoryModule from Martin Offenwanger linked above. Both were stable enough for our production code recognizing there are limitations and am adapting code using ICS with YuOpenSSL. We chose the MemoryModule from Fr0sT.Brutal only because it was the first one we came across and its fairly solid. If there is an improved branch of either I'd love to have it. I could go in to a lengthy description of an issue with both if anyone is interested. The issue is similar between the two. I use both with YuOpenSSL which works very well. I highly recommend it as a OpenSSL snap-in. Including using it with ICS and Synapse (I haven't had much luck using it with Indy). I think Ralf should open a 3rd party section here for his product FWIW. @FroSt.Brutal for an abandoned project, its pretty darn good. I pulled the dev branch and made a slight mod to remove the Ordinal entry points because it was giving me an error and I just didn't need it. Regardless of the nay-sayers asking why and how come etc... there is a need. In my case I have no choice as its in legacy code that I am forced to build upon. The only issue I have with either is figuring out how to debug a DLL loaded with these memory loaders. I'm learning but thats beyond my skill set. Can anyone help? @FroSt.Brutal check your PM please. Angus, either is an excellent starting point for your memory based resource loader and the need is there. IMO.

Thank you. managing certs is all magic to me. This is definitely a learning experience. So, decode and clean up the text... Its a fairly simple server in python (tornado). However. we currently have a ticket in our backlog to rewrite with additional features. It will be ICS at some point as we've been asked to move away from tornado/python and toward RAD Studio solutions I added your Raw Public Key TLS support to the comments in the ticket for consideration. All clients ( 2 and soon to be 3) are currently TNETHttpClient based dll's and as mentioned, failover has become a priority. As also noted above, the dll plugins are memory loaded and I'm stuck with that architecture for the time being. TNetHttpClient uses exceptions to determine connection failures and rollover causes the dll's to crash when an exception is hit/raised. I am working on a replacement that uses error codes in comms instead.

ICS 8.70 and Delphi 11.3 Using TNetHTTPClient there is an event handler that allows verification of key data and setting the accept to true/false. OnValidateServerCertiicate(const Sender: TObject; const ARequest: TURLRequest; const certificate; var Accepted: boolean). I am currently working with TSslHttpRest. I am unsure how to accomplish this same validation check in ICS. If there is a demo that illustrates this, or some direction, I would appreciate the feedback.

Thanks Angus. I had to have a look at our pipeline. Its producing a PEM formatted cert(s). We have a server and a couple of apps that are using it. I'll look in to changing to DER and see how far I get. I seem to recall being able to convert to-from PEM/DER. Not something I'm familiar with. I really appreciate the work!!

There is 2 servers on the network both using the same CA for keygen during the CI/CD pipeline builds. That process is how the client apps get the pubkey imbedded when they are built. Its a one-to-many between the server and app. No other servers besides the primary and failover servers will have that pubkey and the apps will only rollover to a secondary that has the cert signed by the same CA. This effort is a progression from a previous post where DLL plugins loaded with memorymodule cause bad results when exception handling is used for comms state instead of error codes during a failover. In the case NetHttpClient, that's a show-stopper now. ICS offers a way to disable that and rely on error codes as you pointed out in a previous post plus it supports YuOpenSSL. Thus the ICS investigation. I'd have to mod our pipeline builds to use a hash instead of publickey. That also means I'd have to redistribute the current 2 rest apps (soon to be 3) to validate against a hash. I'm still investigating pulling the X509PublicKey via the pointer that is passed in via peerverify.

2) Put that root certificate in place to accessed by the client, either install it in Windows store or even better use the ICS bundle store. Unfortunately this is not an option. Thanks for the input Kass Ob. I completely understand. If this were a normal client app I would do it exactly as you mentioned. In this case we're dealing with some imbedded code and size matters. Pushing the entire cert in the binary is not preferred and there is no place to write the pem(cert) to on the client. So in order to validate the cert belongs to a specific server, the publickey is imbedded and used for validation. We could imbed the key fingerprint instead. But none of the comms components I've seen surface the fingerprint.

Thank you Angus for looking in to this. Unfortunately it seems I'm stuck with TNETHttpClient since it surfaces all aspects of the key for manual validation including the public key. CertName SerialNum Subject Issuer ProtocolName AlgSignature AlgEncrytpion Cert Start Date Cert Exp Date Keysize publicKey (RSA Modulus (n) in hex) Seems all it is missing is fingerprint. When SSL has determined the keys are a problem and the callback allows override, publickey is very popular for manual validation because its small and easy to embed. I'll investigate pulling the X509PublicKey but I don't think I'll be allowed the time. Please consider surfacing publickey in the future. 🙂

unit Unit6; interface uses Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, OverbyteIcsWndControl, OverbyteIcsHttpProt, OverbyteIcsWSocket; type TForm6 = class(TForm) Memo1: TMemo; Button1: TButton; SslHttpCli1: TSslHttpCli; SslContext1: TSslContext; procedure Button1Click(Sender: TObject); procedure SslHttpCli1SslVerifyPeer(Sender: TObject; var Ok: Integer; Cert: TX509Base); procedure SslHttpCli1SslHandshakeDone(Sender: TObject; ErrCode: Word; PeerCert: TX509Base; var Disconnect: Boolean); private { Private declarations } public { Public declarations } end; var Form6: TForm6; implementation {$R *.dfm} procedure TForm6.Button1Click(Sender: TObject); begin SslHttpCli1.URL := 'https://www.google.com'; //choose any https server SslHttpCli1.ResponseNoException := True; SslHttpCli1.Get; end; procedure TForm6.SslHttpCli1SslHandshakeDone(Sender: TObject; ErrCode: Word; PeerCert: TX509Base; var Disconnect: Boolean); var MyKey: string; begin Memo1.Lines.Add('HandshakeDone: ' + IntToStr(ErrCode)); Memo1.Lines.Add(PeerCert.GetPKeyRawText(True)); if PeerCert.GetPKeyRawText(True) = MyKey then <<<-- this is wrong and is probably the private key Disconnect := False else Disconnect := True; end; procedure TForm6.SslHttpCli1SslVerifyPeer(Sender: TObject; var Ok: Integer; Cert: TX509Base); var MyKey: string; begin Memo1.Lines.Add('VerifyPeer'); end; end. I think the question is, in either VerifyPeer or HandshakeDone, how do I retrieve the peer pubkey for validation? A hex string would be optimal. Also in the above is GetPKeyRawText(True)); for retrieving the public key or private key?