Search the Community

ICS V9.5 has been released at: https://wiki.overbyte.eu/wiki/index.php/ICS_Download ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10, 10.1, 10.2, 10.3, 10.4, 11, 12 and 13 and C++ Builder 10.4, 11, 12 and 13. ICS supports VCL and FMX, Win32 and Win64 targets. The distribution zip includes the latest OpenSSL 3.5.2, 3.4.2, 3.3.4, 3.2.5 and 3.0.17 for Win32 and Win64. Changes in ICS V9.5 include: 1 - Major improvements in ICS V9.5 include a new geographic component that has built in IP address databases for countries and ASN; server components have a new event called before a connection is accepted allowing 'firewall' rejection of connections based on IP address; the MQTT client and server components now support protocol 3.1.1; the automatic certification ordering component now supports Google Trust Services and other ACME suppliers, as well as Let's Encrypt; changes for the HTTP clients and servers to better support REST request APIs; 2 - Many of these improvements, and the delay finishing this release, relate to web server improvements needed to mitigate a nine month long attack on a public web server, that started with millions of accesses from two Far East IP addresses, progressed to accesses from VPNs at data centres worldwide, then finally to a botnet that caused access from over one million different IP addresses in 150 countries each week. It's not often a developer has first hand experience of such web server abuse, we try to plan for it, but rarely experience it directly. The ICS web server samples already had filtering by IP addresses and reverse DNS lookup and this worked for a few months with manual updating of the filtering lists, but this was time consuming. So a new GEO component was added with an IP address to country database that allowed specific countries to be blocked, then regions of the world, finally an ASN database allowed specific cloud/ISPs to be blocked. During these months, the ICS web server kept working, albeit slowing as logs tried to handle the vast volumes of IP addresses, needing rewrites of some ICS components. But everything is now stable and ICS capable of handling such heavy traffic. 3 - TWSocketServer has a new event OnClientAcceptFilter event called before the component accepts an incoming connection allowing filtering on the remote IP address so the connection is refused without any more events being called. This action is similar to a firewall refusing a connection, rather than opening and immediately closing it again. Before the event is called, a TIcsSessIpInfo record if filled with remote and local addresses and ports in binary and as strings, saving a lot of application code, the event can complete other record fields. This event can be used with the new GEO components to check countries and regions that should be blocked, and with the TIcsBlacklist component to stop those previously blocked addresses from accessing the server. The THttpSrv HTTP server has a similar OnHttpAcceptFilter event, and it will be added to other servers for the next release. 4 - Added a new TIcsGeoTools component that reads MaxMind formatted GEO database files using the MMDBReader component, and includes two small databases from db-ip.com, 'IP to Country Lite' and 'IP to ASN Lite', but can handle other MaxMind databases. Both databases can are available as resource files that can be linked into applications or loaded from a file to be shared between servers. There is also a country name database ICS-Countries.csv linked as a resource file that contains country GEO information. ASN is Autonomous System Name, an ISP or cloud name, that supplements reverse DNS (often missing) in identifying the owners of IP addresses. The databases are updated monthly by db-ip.com and can be downloaded from them, will try to keep ICS up to date. The TIcsGeoTools component is a self contained unit, IcsGeoUtils.pas with no dependencies, but is only available for Delphi 11 and later due to use of new language features. The component needs to be created in code and the databases required loaded before use, see the samples mentioned below. The main lookup methods are FindISOA2Code and FindASNCode, then FindCountry and FindRegion from an ISOA2 country code, region is a quick was to block all Asian countries for instance. The TIcsDomainNameCache and TIcsBlacklist now include ISOA2 and ASN fields that are included in responses and reports from these components. Beware block countries and regions may have unexpected consequences, for instance Let's Encrypt and Google validate SSL/TLS certificate domain names from multiple countries. The OverbyteIcsSslMultiWebServ and OverbyteIcsDDWebService samples use the databases in the new server OnHttpAcceptFilter event, and writes country and ASN to the web log file, as well as allowing hacker filtering using this information. The OverbyteIcsNetTools sample Trace Route now shows the country and ASN for each IP in the route to the destination, as well as reverse DNS, although the IP addresses allocated to network routers don't appear to totally accurate. These samples only use TIcsGeoTools if DEFINE USE_IcsGeoTools is set in Defs.inc. 5 - Added new components TIcsFilterList and TIcsIpAddrList to replace TestFilters using HackFilterList and TestIpWhiteList using WhiteIpList in sample OverbyteIcsSslMultiWebServ1.pas. TIcsFilterList reads same file hackfilterlist.txt containing key=value pairs which are used to filter incoming connections for path, remhost, country, useragent or referrer, trying to filter out abusive remote hosts. TIcsIpAddrList reads same file whiteiplist.txt which is a list of ASCII IP full or partial addresses, generally that should not be blocked by filters. The TIcsBlacklist has major changes including support for saving IPv6 addresses in binary as well as ASCII, they sort better in reports and take less memory, adding and checking an TSockAddrIn6 which avoids conversion to strings, and other improvements to handle one million IP addresses more efficiently. 6 - Since Let's Encrypt introduced the ACME (Automatic Certificate Management Environment) protocol to download free SSL/TLS certificates, other suppliers have added automated ordering using the same API, mostly with extra account information for commercial certificates. ICS has been tested successfully with free certificates from Google Trust Services, and should work with DigiCert, ZeroSSL and SSLcom, but these three are primarily commercial suppliers and need prepaid accounts, so not tested yet. Google Trust Services offers an excellent alternate to Let's Encrypt and offers almost the same free certificates up to 90 days with multiple wildcards, but allows the expiry days to be specified during ordering, down to three days. Some companies were reluctant to use Let's Encrypt when there was no alternative in case of extended down time, now Google offers that alternate. Apart from Let's Encrypt, suppliers use ACME external accounting to tie the ordering process to web site accounts, which is explained in comments in the OverbyteIcsSslX509Certs unit, more information will be added and the wiki pages updated soon. Google needs the Google Cloud CLI Windows application installing, type a few commands and you get the external account information Acme needs. The OverbyteIcsX509CertsTst sample has a major revision to support multiple account suppliers and to specify the external accounting information. The sample needs to be run on any servers that will order certificates to create the initial Acme account (except for Let's Encrypt), and includes a web server allowing test certificates to be ordered provided DNS points to a public IP on the server. Most suppliers provide a testing endpoint which is listed in OverbyteIcsX509CertsTst so you can order fake certificates to understand the process. There is now a facility to ask ICS servers to renew certificates on demand from the OverbyteIcsX509CertsTst sample, previously you had to mess with the INI file to force a new order. 7 - TWSocketServer has a lot of improvements relating to SSL/TLS certificates, many relating to new IcsHosts options to support suppliers other than Let'S Encrypt. IcsHosts has a new property AcmeSupplier as TAcmeSupplier which may be AcmeLetsEncrypt or AcmeGoogle (or several others), and property SupplierTitle to specify the account name of than supplier from a database. The supplier accounts database is generally maintained by the OverbyteIcsX509CertsTst sample, which must be used to create accounts for new suppliers, and which may be used to view certificate orders. SupplierTitle is used instead of specifying CertDirWork which will be looked up from C:\ProgramData\ICS-Acme-Accounts\ics-acme-accounts.db. By default, new work directories will be in: C:\ProgramData\ICS-Acme-Accounts\. CertDirWork is still supported, but it's recommended that applications move to using supplier accounts instead, which can be monitored using OverbyteIcsX509CertsTst. Google and other suppliers only work with supplier accounts, since information is needed that is not in IcsHosts. IcsHosts has other new properties: AcmeCertProfile to specify the type of certificate requested for Let's Encrypt, listed in FAcmeProfileNames array, default classic, optional tlsserver and shortlived (7 day, not yet available); AcmeCertValidity to specify certificate life in days, default 90, only Google at present, down to 3 days. Certificate ordering now makes use of the ACME Renewal Information API that specifies how many days before expiry a certificate should be renewed, and how often these dates should be rechecked to see if the certificate needs immediate renewal due to being revoked. This overrides CertExpireDays. Renewal Information is checked each time the certificate chain is checked, but is cached so there is usually only a server API call every six hours. Note with OCSP gone, this is now the only way to check if a certificate is revoked. Reworked certificate checking so if automatic ordering is enabled the Acme account information is looked up when the certificate is first loaded to get renewal information and maybe working directory, rather than only when time to order a new certificate, so there is more logging and error checking at load time. Temporary ICS self signed certificates are now created in GSSL_CERTS_DIR instead of TempPath. When starting a certificate order, if the challenges have been previously completed OK, collect order immediately, don't try to start them again. Let's Encrypt is implementing a change in the way new certificates are issued, which may be delayed a few seconds after the CSR is provided, rather than immediately, so the component now waits and checks every five seconds for the new certificate to be issued. This already happens for Google. Note this Let's Encrypt change means earlier ICS versions will soon fail to work. ICS now supports ordering SSL/TLS certificates with IP addresses as well as host domain names, tested with Let's Encrypt Staging but not available yet from live certificates. Testing showed a problem using SSL with IP addresses URLs relating to the Server Name Indication HELO feature which does not allow simple IP addresses which must be converted to domain names, ie 217.146.102.139 becomes 139.102.146.217.in-addr.arpa. Automatic certificate ordering in IcsHosts now has a database property CertRenewNow that if set true in the database using the OverbyteIcsX509CertsTst, will override certificate expiry checking and cause an immediate new certificate replacement order by in servers with IcsHosts the next time RecheckSslCerts is called by the server, typically every two hours. Fixed a long term problem where SSL/TLS server name SNI checking for a matching IcsHost used the certificate SANs that might have included a wild card, instead of the Hosts list of host names. If one IcsHost allowed wild cards it might have been found instead a specific IcsHost for a single host. 8 - New major versions of OpenSSL often add new functions and deprecate older functions that are then removed in a subsequent major release after applications should have been updated. ICS has added a DEFINE OpenSSL_Deprecated without which no deprecated functions should be loaded. ICS has been testing with a special build of OpenSSL 3.5 without deprecated functions and several units have now been updated to use newer 3.0 functions, so no more work should be necessary for OpenSSL 4.0 when those deprecated could disappear. The DEFINE OpenSSL_Deprecated should only be needed if your application uses old OpenSSL functions for encryption or signing. The OverbyteIcsJoseTst sample also needs OpenSSL_Deprecated for RSA string encryption, pending a rewrite without deprecated functions. ICS now only creates the C:\ProgramData\ICS-OpenSSL directory if conditionals OpenSSL_Resource_Files or OpenSSL_ProgramData are specified meaning OpenSSL files are expected there. Otherwise the developer is responsible for setting GSSL_DLL_DIR to the OpenSSL DLL directory. 9 - Updated the MQTT client and server components to support protocol 3.1.1 which is commonly used, previously we only supported 3.1. The client will connect to a v5 server by ignoring dozens of new options, but needs a lot more work, much more complicated than v3.1.1, not planning any more v5 unless there is a specific requirement. Added LogPackets property to log packets in ASCII and hex for diagnostics, UseSSL property to force client to use SSL on any port, BlankClient property (anonymous) for 3.1.1 so server allocates ClientId, but only v5 tells us that ID. BurstMode property for 3.1.1 so client does not wait for response to Connect, but publishes immediately. When Subscribing With v3.1.1, the server now returns a failure flag for permissions failure, which is returned as QoS qtFAILURE. Also improvements to the OverbyteIcsMQTTst sample, allow Username/Password to be set, so they may be left blank, ClientHost is now a drop down box, and includes test.mosquitto.org that may be used for client testing, see https://test.mosquitto.org/ for a long list of ports for different testing purposes, allow MQTT protocol to be specified, added v3.1.1 and v5, and options to test all new functions. If the server SSL port non-zero, the server will create an ICS CA signed certificate for the host name (ie localhost) if a certificate file bundle is not found. 10 - There are various WebSocket improvements. The client now has optional asynchronous connection which no longer blocks the initial WSConnect which now returns immediately and a OnWSConnected event is called when the connections is ready or fails, so should now correctly process a welcome message or packets sent immediately upon connection. The server now has a configurable delay after connection before sending a welcome message or packets, for clients that can not process them immediately. Fixed a problem that data sent immediately a new connection opened could be lost because the component had not switched to Websocket mode. Allow Sec-WebSocket-Protocol: header to added with HeaderSecWebSocketProtocol values (char, superchat, etc). Added a new OnWSFramesDone event called when a queue of frames have been sent, for flow control when sending a lot of data. Note the IcsAppMonMan.dpr sample illustrates how to use multiple WebSocket client components to contact multiple WebSocket servers and display information from them, it comes configured to view three public servers running ICS web, FTP and proxy servers. 11 - Fixed a long term problem with ECDSA binary digests, which have two formats, ASN.1 used by OpenSSL and IEEE P1363 which is shorter fixed length and often also used. Added IcsDigestAsntoIEEE and IcsDigestIEEEtoAsn to convert between the two formats, and a new EcdsaIEE flag to IcsAsymSignDigestTB, IcsAsymVerifyDigestTB, IcsJoseJWSJson, IcsJoseGetSigTB, IcsJoseCheckSigTB and IcsJoseCheckJWS to use the new format, only effective when using EC private keys. Signing Acme requests with EC keys now correctly use IEEE P1363 digests so finally work properly, been looking for this since 2018. 12 - CreateSelfSignCertEx now adds IP addresses to the correct alternate list, not allowed as common name. TSslCertTools has new certificate properties for more Distinguished Names, mainly for personal names: Street, SurName, GivenName, NameTitle, NameInitials, used when creating Certificate Requests. Using Description no longer gives an error. 13 - The HTTP clients THttpCli and TSslHttpRest have new properties RespAttachment (Boolean) and RespFileName, parsed from Content-Disposition: response header which can be used to offer to save content as a file, and RespRetryDT parsed from Retry-After: response header, when this request should next be repeated as TDateTime. ResponseNoException now defaults to True to skip exceptions for most connection errors like 404, etc, beware this default change may cause applications expecting exceptions to misbehave, either set it false or check StatusCode in RequestDone. 14 - In HTTP client TSslHttpRest, if HttpUploadStrat=HttpUploadSimple, add unofficial Content-Disposition request header that some web servers might check for an upload file name. Check for a Json response of any array only [] without objects. Allow GET and DELETE methods to use PContBodyJson, PContBodyUrlEn and PContBodyXML content types, beware web servers may not support this. 15 - The TRestParams component has a new RParamFmt property that for Json only defines whether nested objects or an array should be formatted, default is RPFmtNestObj (Nested Objects, same as previously), or RPFmtArrayVal (Array of Values) if first element is any array, or RPFmtArrayObj (Array of Objects) where each element is treated as object in the array. Note RPFmtArrayObj allows duplicate names in Add methods, since output into different objects. For instance: RPFmtNestObj: {"field1":"data1","field2":"data2","field3":[data1, data2, data3]} RPFmtArrayVal: [data1, data2, data3] RPFmtArrayObj: [{"field":"data1"},{"field":"data2"},{"field":[data1, data2, data3]}] 16 - In the HTTP servers THttpSrv and THttpAppSrv, allow the built in HTTP error response to be customised using new event OnHttpCustomError which is called by the error handlers with the error, path, and existing Body, that may be replaced or modified as required. Called for errors 301, 302, 307, 308, 400, 401, 403, 404, 416, 501. Added new hoContDispHdr Option and AttachmentTypes list of file extensions that if matched causes the server to add an Content-Disposition: attachment header with the filename, that should cause a browser to offer a 'Save As' dialog to save a binary file, rather than trying to display it. Note the default list includes .pdf so Acrobat files are saved rather than displayed. The Get and Delete methods now accept uploaded body content similarly to POST/PUT. The derived THttpAppSrv server has handlers for uploaded content, for THttpSrv you need to write your own. Added OnHttpAcceptFilter event called before TWSocketServer accepts an incoming connection allowing filtering on the remote IP address so the connection is refused without any more events being called. 17 - TWSocket has a new property SessionIpInfo which is TIcsSessIpInfo record set after connection with the local and remote IP addresses and ports from the socket, also socket type and protocol, as internal and string versions. Might be easier to use than various GetPeer methods. Set for accepted listen connections. Fixed a missing inherited DupConnected that meant counters did not get reset. The SSL/TLS Server Name extension does not allow raw IP addresses, so convert then to domain names, ie 217.146.102.139 becomes 139.102.146.217.in-addr.arpa. 18 - Added Windows memory reporting functions IcsMemInfoProg, IcsMemInfoGlob and IcsMemInfoPerf to the OverbyteIcsWinUtils unit, useful for server monitoring, used by the sample IcsAppMon.dpr. Also IcsMemWarning to check for low or critical memory problems, returns Warning at 85% physical or page file usage, critical at 95% usage (reboot probably required). 19 - ICS added OSCP (Online Certificate Status Protocol) support a few years ago, used to check if certificates have been revoked. But running the massive OCSP databases needed has proved challenging, and the industry is moving away from OCSP, Let's Encrypt stopped adding an OCSP URL to certificates in May 2025. OCSP adds quite a lot of code, so added new defines to ICS so OCSP code is only linked if using authorities that still support OCSP, see information about OverbyteIcsDefs.inc. This change effects many components that check certificates, if the defines are disabled OCSP properties are still available, but will be ineffective, removing the OCSP properties would in too many form errors. Another reason for OCSP's demise is shorter SSL/TLS certificate life, so they expire rather than needing to the revoked. From 15th March 2026, certificate life span is reduced to 200 days, from 15th March 2027 down to 100 days and finally from 15th March 2029 to 47 days, but only 10 days for domain control validated certificates, such as most free certificates which are currently 90 days maximum. ICS can already order seven day certificates from Google Trust Services, with Let's Encrypt adding this later in 2025. 20 - ICS now defaults to the latest OpenSSL version 3.5.2 which includes support for new Post Quantum Cryptography (PQC) algorithms (ML-KEM, ML-DSA and SLH-DSA) and for server side QUIC (RFC 9000). ICS has no plans for QUIC support, not yet investigated PQC, don't believe any low level changes are needed, maybe changes to the cipher lists. This is a long term support release with fixes and security updates for five years, until April 2030. ICS still includes four older OpenSSL versions, which will slowly disappear as they reach end of life, about one every six months. 21 - The OverbyteIcsDefs.inc file included in most ICS units has several new defines. DEFINE OpenSSL_36 (due Oct 2025) and OpenSSL_40 (due Apr 2026). Enabled DEFINE OpenSSL_35 for OpenSSL 3.5. DEFINE OpenSSL_OcspStaple, should SSL server staple an OCSP response to check if server certificate is revoked. Let's Encrypt stopped adding an OCSP URL to certificates in May 2025 so only enable this if using authorities that still support OCSPL, to avoid extra code being linked. DEFINE OpenSSL_OcspChains, should SSL clients checking a certificate chain check an OCSP server to see if the certificate is revoked, only happens if the certificate has an OCSP URL, undefine to remove the extra code that does OCSP checks. DEFINE OpenSSL_Deprecated, should OpenSSL deprecated functions be loaded, not needed for ICS but may be used by applications for encryption or signing. DEFINE USE_IcsGeoTools used by samples with the TIcsGeoTools component to lookup countries from IP addresses, D11 and later only. All ICS active samples are available as prebuilt executables, to allow ease of testing without needing to install ICS and build them all. There are four separate zip files split into clients, servers, tools and miscellaneous samples which can be downloaded from https://wiki.overbyte.eu/wiki/index.php/ICS_Samples

Hello, I'm using the ICS 9.0 C++ package. While testing a websocket server (TSslHttpAppSrv) communication with a client (TSslWebSocketCli), I noticed that both have trouble receiving several frames in a short period. It doesn't seems to be related to the frame size because sending empty text frames (using WSSendText) doesn't change anything. Let's suppose I want to send 10 messages from my server to my client, in my small test program I can : a) click 10 times on a "send" button. b) click one time on a "send" button but then I loop 10 times on WSSendText. In the (a) scenario everything is ok but in the (b) scenario the connection is closed at some point, attaching a TIcsLogger to the server and client does not really help as you can see below : Server side LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TriggerDataSent handle=1476 LoggerLogEvent : 11:00:15:084 00000148B13008B0 TCustomWSocket.Shutdown 1 handle=1476 "11:00:15.084" : WS ClientDisconnect : 0 Client side LoggerLogEvent : 11:00:15:084 SessionClosed Error: 0 LoggerLogEvent : 11:00:15:084 00000204E35B5B30 TCustomWSocket.Shutdown 1 handle=1468 These tests are done without SSL. Now if I activate SSL, both (a) and (b) scenarios works fine, I do not have any sudden connection close anymore. My understanding is that the extra computation introduced by the transport layer security adds a delay before sending the frame (or maybe the way frames are sent is reorganised) so the client has now time to process each frame without being flooded. I'm a bit surprised because these numbers are quite low (10 frames), I actually doesn't have to send 10 frames to create the issue 3 is enough. I have the feeling that I'm missing something, either on the server/client configuration side (a missing parameter ?) or on the feedback informations (from the logger) that I get preventing me to understand exactly what is going on. If someone has a clue on what should I do to get a better grasp on my problem it would be very much appreciated. Thanks in advance.

SVN has new versions of the Websocket components and three samples, will be zipped overnight. IcsAppMonMan which contacts multiple Websocket servers is much better on startup with async connections. Angus

In the current version (V9.4) of TSslWebSocketCli, the OnFrameRcvd event is triggered before the OnConnected event. This is the full set of modifications I made for OverbyteIcsWebSocketCli.pas, and it requires review. I added a new value to TWebSocketState: wssConnected. This state represents the situation where the server has already responded with "101 Switching Protocols", but the internal HttpState has not yet reached httpReady. In TSslWebSocketCli.IsWSConnected, HttpState need to be httpReady. I overrode StateChange to wait for this condition. I’m curious why the httpReady state is always slightly delayed. With this change, OnConnected is now triggered earlier, before OnFrameRcvd, which aligns better with the expected WebSocket event sequence. However, there is still one issue: OnFrameRcvd may be triggered before WSConnect returns. I'm not sure whether this behavior is acceptable or expected. Please advise. Using Demo: OverbyteIcsSnippets as client and OverbyteIcsSslMultiWebServ as server TWebSocketState = (wssHttp, wssConnecting, wssConnected, wssReady); function TSslWebSocketCli.WSConnect: Boolean; begin Result := False; RequestVer := '1.1'; HeaderUpgrade := ''; HeaderConnection := ''; HeaderSecWebSocketAccept := ''; HeaderSecWebSocketProtocol := ''; HeaderAccessControlAllowOrigin := ''; FWSFrameCounter := 0; if (DebugLevel >= DebugConn) then LogEvent('WebSocket: Connecting to: ' + URL); // WSState := wssConnecting; { in case server returns data quickly } Get; // sync request Result:= WSState = wssReady; end; procedure TSslWebSocketCli.StateChange(NewState: THttpState); begin inherited; if (NewState = httpReady) and (WSState = wssConnected) then begin WSState := wssReady; if Assigned(FOnWSConnected) then FOnWSConnected(Self); end; end; procedure TSslWebSocketCli.SocketDataAvailable(Sender: TObject; ErrCode: Word); var Len: Integer; { JK 28.11.2023 } BufPos : Integer; ParsedBytes : Integer; { JK 28.11.2023 end } ServerKey : String; s : AnsiString; KeyComp: Boolean; begin if (NOT (State = httpReady)) OR (WSState <> wssReady) then begin inherited SocketDataAvailable(Sender, ErrCode); // if FReceiveLen >= 2 then // LogEvent('WebSocket: Inherited Recv Raw: ' + IcsTBytesToString(FReceiveBuffer, FReceiveLen)); // !!! TEMP if (WSState <> wssConnecting) then Exit; //The protocol has been upgraded from HTTP to WebSocket. //This is when the connection is truly considered established. if (StatusCode = 101) then begin if SameText(HeaderUpgrade, 'websocket') and SameText(HeaderConnection, 'Upgrade') then begin s := AnsiString(ClientKey + '258EAFA5-E914-47DA-95CA-C5AB0DC85B11'); ServerKey := String(IcsBase64EncodeA(SHA1ofStr(s))); { V9.1 corrected string cast, V9.4 } KeyComp := (HeaderSecWebSocketAccept = ServerKey); if (NOT KeyComp) and (DebugLevel >= DebugConn) then LogEvent('WebSocket: Server Key Comparison Failed'); end else if (DebugLevel >= DebugConn) then LogEvent('WebSocket: Failed to Upgrade to WebSocket Protocol'); end else if (DebugLevel >= DebugConn) then LogEvent('WebSocket: Failed to Connect: ' + LastResponse); if KeyComp then begin if (DebugLevel >= DebugConn) then LogEvent('WebSocket: Connected OK'); //change the state to wssConnected, then wait httpReady WSState := wssConnected; { in case server returns data quickly } LastReceivedDataTickCount := IcsGetTickCount64; LastSentPingTickCount := 0; if (FWSPingSecs > 0) and (FWSPingSecs < 5) then FWSPingSecs := 5; FPeriodicTimer.Enabled := true; end else WSState := wssHttp; end; ........ OverbyteIcsWebSocketCli.pas

I'll look at your changes, but I rewrote the WSConnect function yesterday, adding an async option so it is no longer blocking, which was a serious anomaly for ICS. I've also changed the ICS server component not to send welcome or other data immediately, before the client has a chance to process the 101 command and switch to Websocket mode. Still testing all this. Angus

When using the `TSslWebSocketCli.WSConnect` method an expected Connection Refused error is thrown (See Attached). I want to hide this. How can I do so? I have tried wrapping a `try..except` block around it without any luck. I have also assigned the Error and Exception events to empty procedures.

Hi, Not really a clue but a guess reading the log above and the steps, the shut down triggered at 16kb of sending, i read this as clue, it looks like the pending response ( aka blocking request) translated into disconnect and wrongly handled as failure instead. So; 1) a question on side but might be important, were did these numbers 1476 and 1468 come from ?, did you calculate the MTU and decided it should be like that ? the lack of client side successful recv means the log entry wasn't from successful send event, track those, (but again of ICS have them, as i don't know) 2) i can't tell how ICS works for websocket, but i think Angus can tell if the blocking handled right in your version, yet he recommended that you update, then update, also reproducing such behavior should be fairly easy, @FrozenK if you can provide short project reproducing that shutdown trigger, then that behavior can be solved once and for all. 3) Have you missed around with TCP buffers sizes ? (namely sending buffer) , are you tweaking some socket options ? the default send buffer on Windows is 8kb, but this doesn't means you can't send more, it is just little more long story to explain. 4) Also, the problem in my opinion is that ACK hadn't been received and sending party reached some a limit around 16kb (limit but not receiving peer ACK), this triggered blocking request (pending), and this translated into error with lead to shut down, in other words handled, the socket is not checking if ready to send while handling WSAEWOULDBLOCK as fatal connection error, while it is not. Hope that helps.

There have been a lot of changed to the websocket components since ICS V9.0, and a major rewrite to support newer protocols is coming in a week or so with ICS V9.5. So please repeat all your tests with ICS from the overnight zip and see if you can still reproduce the same problems. I'm afraid the debug logs you show are meaningless for diagnostic purposes, they are purely for development use. I have a public websocket server that is supporting about 50 SQL REST requests each second, for an hour at a time. Angus

Hi Some protocols require Sec-WebSocket-Protocol header to be set when connecting, I made a patch to allow it in OverbyteIcsWebSocketCli.pas OverbyteIcsWebSocketCli.pas.patch

ICS V9.4 has been released at: https://wiki.overbyte.eu/wiki/index.php/ICS_Download ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10, 10.1, 10.2, 10.3, 10.4, 11 and 12 and C++ Builder 10.4, 11 and 12. ICS supports VCL and FMX, Win32 and Win64 targets. The distribution zip includes the latest OpenSSL 3.0.16, 3.2.4, 3.3.3 and 3.4.1 for Win32 and Win64. Changes in ICS V9.4 include: 1 - Completed the ICS Application Monitoring system added in V9.3, designed to locally and remotely monitor ICS servers and applications, and to locally restart applications on demand or if they crash. It comprises a small TIcsAppMonCli client monitoring component that is added to ICS applications, usually Windows Servers, but also client applications. This client component communicates with a TIcsAppMonSrv server component, usually running as a Windows Service on the same server so it is able to restart applications, but can also support clients on a LAN. The monitoring server has web and Websocket servers, allowing remote browsers to view the state of all applications being monitored by the server with a continually updated web page. There is also ICS Application Monitor - Remote Manager application that provides remote monitoring of multiple ICS Application Monitor servers using Json web and Websocket requests on a single screen. 2 - The SMTP client component fixes a bug introduced in V9.3 which could corrupt the Content-Transfer-Encoding header line. 3 - In TWSocket, fixed a potential problem using multiple threads where a new connection opened very quickly (ie localhost) and then stalled due to an unexpected connection state. Made DataToString Unicode compatible, only used for diagnostic dump logs. 4 - In the HTTP client, fixed a check for an overflowing buffer when receiving very long headers that could cause failure detecting headers end. Made several URL validation functions public: GetProtocolPort, IsSSLProtocol, IsKnownProtocol and IsKnownProtocolURL. 5 - When creating PKCS12/PFX certificates, change the 3DES cipher to AES256 if the legacy provider is not loaded. 6 - In the TIcsFtpMulti component, skip download of zero length file by creating an empty file, previously this got SSL handshake error. Don't report directories as being downloadable, they are not. If extended passive mode allowed, send EPSV ALL at start so firewalls and NAT routers can handle sessions more efficiently. Added CheckBadUnicode property defaults to false, so that checks for bad Unicode to Ansi conversions with ? are skipped, allowing more complex paths without errors. 7 - The FTP server FEAT request now returns EPRT and EPSV which have been supported for IPv6 for years, but were not advertised for IPv4. 8 - The OverbyteIcsSnippets sample adds two new simple REST snippets to Get/Post Parameters that send them to an ICS server, and the server echoes back those params so you can check what was actually sent. 9 - When loading the OpenSSL DLLs, no longer check they are digitally signed for Windows XP, 2003, Vista and 2008, they don't recognise SHA-256 code signing, never tested since no longer have those old versions available. 10 - The TRestParams method AddItemSO to add a SuperObject now has an Escape parameter defaulting to True, so non-ASCII characters are escaped by default. 11 - The Proxy component TProxyTarget now has a SocketFamily property so target connections can be restricted to TSocketFamily values. Added property SrvTotSess count of server session connections for logging. 12 - In the HTTP Application Web Server, fixed a memory leak with multiple virtual PUT and POST documents. 13 - The OverbyteIcsJoseTst sample 'Sign/Verify Data' tests now support hashes other than SHA-256, selected from the Key and Signing Hash Algorithm drop down list. Also, a private key matching that selection is created automatically, including X25519. 14 - There is a new unit OverbyteIcsWinUtils that contains Windows API functions, built from selected Magenta Systems libraries, with functions needed to build and control Windows Service applications, accessing the Windows registry, Windows firewall, Windows tasks, hardware, and with simple encryption for passwords. Most of these functions are used by the TIcsAppMonSrv server component and IcsAppMon sample, but should have much wider use for Windows Service server applications in particular, like allowing firewall access. It's planned to move most other Windows specific function here for ICS V10. 15 - In the ICMP component, fixed a problem setting property PingMsg to the text to ping. 16 - In the TIcsIpStrmLog component, added method ListenStates which for logprotUdpServer and logprotTcpServer returns a multiline string listing the IP, port, SSL and state of all socket listeners. The CurSockets property now reflects actual TCP Server clients. 17 - In the TIcsMailQueue component, don't keep retrying email that is too short to send with no body or with no sender headers. Added more error handling if the SMTP component fails to build the EML spool file. 18 - In OverbyteIcsUtils, finished the cleanup of old Base64 functions by adding new IcsBase64 functions using TBytes internally to replace old Base64 functions that used AnsiChars, with no overloaded versions for simplicity. Old Base64 versions retained as deprecated for user applications, please update to the IcsBase versions. Added IcsTBytesCompare to compare two TBytes. Added IcsOutputDebugStr for Posix and Windows. Added IcsDateToAStr and IcsDateTimeToAStr with alpha month (Jan/Feb). 19 - The ICS C++ packages for C++ 10.4 and later have been updated with the correct paths for the three supported platforms, and all build and install correctly for Win32. Win64 should also build, but not Win64x Modern which needs fixes in a future release of C++ 12. The release notes for V9.4 are at https://wiki.overbyte.eu/wiki/index.php/ICS_V9.4 All ICS active samples are available as prebuilt executables, to allow ease of testing without needing to install ICS and build them all. There are four separate zip files split into clients, servers, tools and miscellaneous samples which can be downloaded from https://wiki.overbyte.eu/wiki/index.php/ICS_Samples Angus

Hello, I'm trying to configure the most simple websocket client (with TSslWebSocketCli) and server (with TSslHttpAppSrv) in order to get them communicate with each other locally. Worth noting that I'm a beginner in the internet communication protocol world, this is why I'm trying to get the minimum working and then I will build from that. I'm using the ICS 9.0 C++ package, I believe I have a configuration problem more than a C++ problem. For the client part, I configured the following (no TLS for now) : client->SocketFamily = sfAnyIPv4; client->DebugLevel = DebugHdr; client->CertVerMethod = CertVerNone; client->SslCliSecurity = sslCliSecDefault; client->URL = "ws://127.0.0.1/"; client->WSPingSecs = 10; event handlers to get something printed in the terminal : client->OnHttpRestProg = HttpRestProg; client->OnWSFrameRcvd = FrameRcvd; client->OnWSConnected = Connected; client->OnWSDisconnected = Disconnected; For the server part, based on the Delphi samples OverbyteIcsBasicWebServer1.pas and OverbyteIcsSslMultiWebServ.pas, I configured : server->AuthRealm = "ics"; server->SocketErrs = wsErrFriendly; server->ExclusiveAddr = true; server->SessionTimeout = 300; server->MaxSessions = 100; server->OnDisplay = Display; server->SslEnable = false; server->SslCliCertMethod = sslCliCertNone; server->SslCertAutoOrder = false; server->CertExpireDays = 30; server->OcspSrvStapling = false; server->AddGetAllowedPath("/", afBeginBy, "WEB-APP"); server->OnClientConnect = ClientConnect; I added an host : server->IcsHosts->Clear(); TIcsHost *host = new TIcsHost(server->IcsHosts); host->HostEnabled = true; host->HostNames->Text = "localhost"; host->HostTag = "WEB-APP"; host->Descr = "WebApp Server"; host->BindIpAddr = "127.0.0.1"; host->BindNonPort = 80; host->BindSslPort = 0; host->WebDefDoc = "index.html"; host->WebDocDir = dir + "data"; When the client is connecting (TSslWebSocketCli.WSConnect), OnClientConnect is triggered server side, then : THttpWSSrvConn *conn = (THttpWSSrvConn *)Client; conn->WSessionCookie = "OverbyteIcsWebAppServer" + server->Port; conn->OnWSLogEvent = ClientLogEvent; conn->OnWSHandshake = ClientHandShake; conn->OnWSDisconnected = ClientDisconnected; conn->OnWSFrameRcvd = ClientFrameRcvd; conn->OnWSFrameSent = ClientFrameSent; conn->OnWSReady = ClientReady; conn->OnWSPingTimer = ClientPingTimer; conn->OnBgException = ClientBgException; Then the server handle the request (these are called : THttpConnection::ProcessRequest(), ::ProcessGet(), ::ProcessGetHeadDel(), ::SendDocument()...) and returns an "HTTP/1.1 200 OK" response to the client. From TSslWebSocketCli.WSConnect I get "WebSocket: Failed to Connect:" but the LastResponse string is empty. It looks like the server treats my websocket upgrade request as a "classic" request, so it returns an "HTTP/1.1 200 OK" response instead of the expected "HTTP/1.1 101 Switching Protocols". To make sure my client part was ok, I quickly tried it on a node.js websocket server, no problem, I get the "101 Switching Protocols" response and the websocket connection is established. If I do not use TSslWebSocketCli.WSConnect but do a "manual" GET with an upgrade request header and on server side add an url handler (TUrlHandlerIndexHtml) and send a response header through AnswerStream, the connection is established. But if I send frames from the client, OnWSFrameRcvd is not triggred server side (as expected I believe since server has no clue I handled the upgrade "myself"). Hopefully this post makes sense, in short, I don't understand why my upgrade request on url: "ws://127.0.0.1/" fails on the ICS server, any help would be appreciated. Thanks in advance.

I believe the Internet Component Suite is amazing and a godsend for Delphi. Thank you for everything you do to support this software. I wish to use WebSockets with ICS Delphi components. I wish to use this in a very large and important system (cannot name it). I am challenged to prototype this because the Overbyte demo OverbyteIcsSslMultiWebServ.dpr has so many features it is impossible to extract a simple webSocket server. I wish to demonstrate ssl/tls and non ssl/tls connections. I am not interested in echo servers/clients. I want servers/clients that can handshake and exchange messages with each either other. How can I demo this. The client side I have been able and is fairly straight forward. The server side is too complex. Please provide a simple Websocket Server application that supports ssl/tls and non ssl/tls connections. Thank you Richard Maley

I've added a new event OnWSFramesDone to the Websocket client, called when the queue of frames has been sent, for flow control when sending a lot of data. Previously, the OnWSFreameSent was called after each frame, but it was not easy to tell if this was the last queued frame. I've only tested it with a few frames of ASCII, the HttpRest sample really needs a new button to send a queue of files, which is on my list, but also needs a server able to accept those files. But I'm sure you'll tell me if it does not work. In SVN now, will be zipped overnight. Angus

I offer clients a REST API service to look-up telecommunication information, using the ICS web application server and MS SQL server. It works well, for low volumes of queries, but most users start a new SSL/TLS session for each query, which becomes a limiting factor with performance. So I want to offer a Websocket API as well, so one SSL/TLS session stays open, with just simple request/response packets sent. But how to adapt the REST HTTP request/response to Websocket? My queries are simple URL parameters, ie codelookapi.htm?numhistory=118118. Should the Websocket message just be the arguments or include the full or partial URL as well? Or something else, like a command? Should the initial Websocket request allow arguments, or just open the connection? Should be Websocket response message be just a JSON block, or include a wrapper of some sort, like the HTTP response header? Has anyone done a similar design? I just want to make it easy for clients using standard Websocket client libraries to integrate the new API. Angus

I'd make the general point that ICS is an async library, generally you should never use Sleep(), but events. If you want to delay something, use triggers within a timer. Having multiple ProcessMeesages everywhere is also bad design. Your code should be packaged into an object with events, called before a single message loop. In fact, this design would mean you can test and debug your WS code in a simple GUI before using it in a DLL. From your various comments, I gather you are writing a Websocket client DLL that sends large binary blocks of data to a server, All my testing and the ICS samples are server to client communication, although in theory the code is two-way and similar in client and server. But I simply don't have a way to easily test your requirement. Angus

Using OverbyteIcsSnippets as an example. Websocket CLIent section, after wsconnect is successful it falls in to this loop. while NOT AbortFlag do begin Application.ProcessMessages; if Application.Terminated then Break; if IcsTestTrgTick64(TrgTimerEnd) then Break; end; Mine is a console app and I do something almost identicle to this on startup. I connect then fall in to the main thread loop waiting for a signal to end. My application runs at 48% cpu usage just like the example. I realize this is a tight loop. Running as-is the OnWSFrameRecv will receive a 50M file in a few seconds. However when I try to address the high cpu issue and add a simple sleep(5), which drops the CPU down to less than 5 percent. The 50M file receive via the OnWSFrameRecv event now slows down to 9 minutes. Its like the receive is blocking. I'm not sure why a simple sleep would do this. Is the sleep cascading in the tight loop and causing the overall main thread to sleep much longer than the 5ms maybe? I know there is missing info here. The application is quite large but the websocket connect and receive is very basic. I rebuilt excluding throttling in the library but it didn't make any difference. Is there a better way this not CPU intensive to loop and wait for the termination? The goal is to receive large files via WS and not have the appl;ication idle at this high usage. Any help would be appreciated. Regards

Are you using an old version of ICS? ICS added full Websocket server and client support two years ago, and handles the upgrade negotiation process automatically. The main unit is OverbyteIcsWebSocketSrv.pas which contains a derived connection class THttpWSSrvConn that handles the Websocket protocol. There are two samples OverbyteIcsSslMultiWebServ.dpr and IcsAppMon.dpr that show how it all works. Angus

var ARequestInfo := THttpConnection(Client); if SameText(ARequestInfo.RequestConnection,'Upgrade') then ARequestInfo.Answerstring(Flags, '101', '', 'Upgrade: websocket' + #13#10 + 'Connection: Upgrade' + #13#10 + 'Sec-WebSocket-Accept: ' + WebSocketAccept, '');//WebSocketAccept is working good with idhttpserver what ever i do i can't make this work and its just the first step maybe i am trying wrong way maybe i need to use ARequestInfo.Send(TWSocketdata,len) or ARequestInfo.StartConnection; idk what i have to do to make handshake then send client alert from server

I'd first make the general comment that it is always best to develop and test the two parts of client/server applications separately, against known working versions. In this case, with the ICS OverbyteIcsSslMultiWebServ and OverbyteIcsHttpRestTst samples. Don't know if C++ allows you to build them, but the wiki site allows you to download prebuilt executable files. Your settings are missing a websocket path or page, just ws://127.0.0.1/ so you are assuming the web server default HTML page is actually a Websocket request, this was never testing with the ICS web server, perhaps my fault for not expecting anyone to try that. Since you set default page to index.html, I assume that is the websocket URL you are checking for, but you don;t show any of that code. I find it best to use a virtual path /websocket/ to clarify that such requests from HTML. Angus

I want to implement a Delphi WebSocket server to send binary data (Stream and byte array) to javascript client. This could be done after receive a client message OR EVEN as unidirectional message so no client message need to be received. Below the pseudocode. var WSServer: TWSocketServer; Case A: I want to send back the binary data after receive a message WSServer.OnClientConnect := procedure (Sender: TObject; Client: TWSocketClient; Error: Word) begin TWebSockSrvClient(Client).OnWebSocketMessage := WebSocketMessage; end; procedure WebSocketMessage(Sender: TObject; Msg: string); var MemStream: TMemoryStream; SocketConn: IWebSocketConnection; begin //create and populate MemStream if (Msg = 'A') then begin SocketConn := TWebSocketSocket(Sender).getConnection; //I need to send back the MemStream but I found only these method: //SocketConn.sendString //SocketConn.sendFrame //SocketConn.sendMessage end; end; Case B: send binary data to some client from button click procedure TForm1.btnSendBufferClick(Sender: TObject); var ArrB: TBytes; begin ArrB := [1, 2, 3]; //I tried WSServer.Client[0].SendTB(ArrB) but is doesn't work end; I check OverbyteIcsWebSocketSrv.dpr demo but only strings are send. Any advice?

There is a new ICS sample OverbyteIcsBasicWebServer.dpr in SVN, will be zipped overnight, which is a simplified version of OverbyteIcsSslMultiWebServ ignoring configuration INI files, security features, session data, most demo pages and most logging, and settings for localhost set in code, search for IcsHosts to change IP addresses, etc. But this sample will be a much easier starting point for those developing web or Websocket servers with ICS. I'm going to copy most of the code into another new ICS sample that needs a Websocket server. Angus

I am using the ICS TWSocketServer to communicate between a delphi application and a collection of React applications embedded in Edge browsers. The websocket server is running in the delphi application as are the React web applications. When one of the React applications attempts to connect with the websocket server it sends a json frame with some information needed for authentication. I am able to extract the header information but I cannot figure out how to get the json. Could someone help me? Richard Maley

If you really mean the websocket protocol and not HTTP, there are no header or bodies or protocol, once a websocket is open there is simply a two way TCP stream, and you can send what you like. That stream arrives at the server in the ClientWSFrameRcvdEvent as a string packet and TWebSocketReceivedFrame to tell what typer of data is arriving (text, binary, or closing), it's up to the developer to decide how to interpret that data. Angus

The OverbyteIcsSslMultiWebServ sample serves both web and websocket pages, since all websocket requests start with an HTTP request to port 80/443. But you can remove all the code relating to creating web pages from the sample, the OverbyteIcsSslMultiWebxx units, etc, leaving just the default page and websocket handlers ClientWSxx and WSxx which do the websocket events. I will look at a new simple web server sample, similar to the client snippet samples, but unlikely to be ready for the next release. Angus

Angus Using TSslWebSocketCli, after setting CertifyMethod to CertVerNone, verifypeer no longer gets executed. But the Websocket connects and is ready. If it completely bypasses all internal chain verification AND does not fire verifypeer (which it seems to behave the way you described), how do I do a manual cert verification? These are self signed certs and we validate using hashes, serialnum, pubkey(Mod N) etc... whatever the cert provides that is unique. Setting the CertifyMethod to CertVerBundle or CertVerdWinstore fires verifypeer but the certificates fail regardless of the _OK_ flag. I'm looking for something that may be described as CertVerSelf. This would fire the verifypeer to be used without other verification and the _OK_ would signify pass/fail of the cert. Is there a method in place to accomplish this already?