Angus Robertson

The Internet Communications Suite (ICS) has a REST client component using Json and an application web server that sends whatever you give it.  There is an ICS topic here.  

But no SOAP protocol as such, Json seems to have overtaken SOAP as much easier to use.  ICS uses SuperObject for Json and reading XML, all Delphi 2007 compatible.  Adding a SOAP wrapper to your XML objects should not be too hard. 

Angus

Servers use SSL/TLS certificates, not usually clients.  Some errors on a certificate will prevent a server accepting an SSL/TLS connection, like an invalid private key. or cause the client problems like an expired certificate, but using self signed certificates is optional in the client, you decide whether to accept them or not.

Angus

Two new zips for Win32 and Win64 versions of OpenSSL 1.1.1e can now be downloadable from the Wiki at: http://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp . The latest 1.1.1 DLLs are also included in the ICS distribution SVN and overnight zip.  This release includes one low priority security improvements and bug fixes, including one that allows IcsJoseJWKPubKey support RSA-PSS keys.

ICS applications require V8.57 or later to support OpenSSL 1.1.1e.

Changes in 1.1.1e may be found at https://www.openssl.org/news/openssl-1.1.1-notes.html 

Angus

There is nothing wrong with Delphi 7 for applications today, if anything they are less bloated than later compilers where EXEs just get bigger with no extra functionality, All my ICS development and commercial applications are Delphi 2007, but then tested and made compatible with the latest and greatest of the day.  Obviously cross platform development is another matter, but I don't do that.  And lots of new language features and library functions, which ICS seems to manage without, or has implemented it's own versions to be compatible across all compilers D7 up. 

For client/server, also look at the new ICS component TIcsIpStrmLog and sample OverbyteIcsIpStmLogTst, the component can be configured as TCP/UDP/Client/Server with SSL/TLS, and mainly just sends lines of text back and forward, with two methods to connect/listen and send and one event to receive, you can build client/server apps very quickly. 

Angus

Websockets are designed for use in browsers, as a more efficient duplex mechanism than Ajax, for interactive content like spell check and auto completion as you type.  

Don't see any purpose outside the browser environment where simple TCP client/server has been used for 30 years.  ICS has several sockets demos illustrating various client/server concepts. 

Angus

When I get feedback from people actually testing this initial implementation, I'll look at it again. 

Angus

I've updated the ICS distribution with the Websocket server implementation (HIXIE and HYBIE protocols) ported from phpws project, written by Stan Korotky <stasson@orc.ru> (eight years ago).

This demo may be tested against the  web page websocketclient.html which should be accessed from a server using the same IP address as this application, such as the ICS demo web server.  The web page uses the
WebSocket API with Javascript to send data to the server which is echoed back to the page.

Ideally, websockets should be integrated into the ICS web server so they can be used on the same ports, the request that starts them up is HTTP, and I'll do that is anyone needs it. 

Please note this is an experimental server websocket implementation, that needs a lot more testing in case the protocol has changed in the last eight years.  But it works with the latest version of Firefox as a client.

A websockets client component is more work, but using most of the same code as the server since it's a duplex protocol. 

Angus

Websockets is a client server protocol, like HTTP, except once the connection is open duplex packets can be sent back and forward.  So that packet stuff is common to client and server, and exists in the old code using ICS.  The sample application is just a server and a JavaScript client web page.  But creating an ICS client should not be hard, uses HTTP and the same functions as the server.

Angus

I will do a quick evaluation of building that old code with the latest ICS over the next few days, but I won't do anything further unless someone else sets up a websockets test environment to test it all, that often takes as long as the coding, a server and client that the ICS components can be separately tested against.

Angus

I added your change to my copy, not in SVN yet.

Angus

Thanks, now fixed.

Angus

SVN and the overnight zip have been updated with a lot of changes so ICS  supports International Domain Names for Applications (IDNA), i.e. using accents and Unicode characters in domain names.

Domain names can only contain lowercase ASCII letters and numbers and a couple of symbols, so Unicode U-Labels (nodes in a domain) must be converted to A-Labels (Punycode ASCII) with an ACE (ASCII Compatible Encoding) prefix.  So www.mâgsÿstést.eu becomes www.xn--mgsstst-pwa1e4l.eu and мособлеирц.рф becomes xn--90aijkdmaud0d.xn--p1ai.

ICS mostly does the Unicode to A-Label conversion just before looking up an IP address for a domain name (in DnsLookup) and converts back from A-Label to Unicode when doing a reverse lookup (in ReverseDnsLookup).  HTTP headers also contain A-Labels for the Host: header and the host part of URLs for proxy or relocation, but Unicode paths should be UrlEncoded by the application as now.  Not looked at SMTP yet.

The HTTP client and server, Ping, ICMP and DNS Query components all now support Unicode domain names, generally without application changes unless you want to display the A-Label name looked-up (PunycodeHost property).  DNS Query does require application changes due to all methods and properties previously being AnsiString, now String.

SSL/TLS now fully supports Unicode domain names, including displaying the Unicode version of the domain name (except for Subject and Issuer lines), and X509 automatic certificate ordering from Let's Encrypt fully supports Unicode domain names.  Certificate files are saved with Unicode names, not A-Labels.
 

For server testing I registered an eu domain which is live on one of my web sites at https://www.mâgsÿstést.eu/  and https://scrúdú.mâgsÿstést.eu/ which have ICS ordered SSL certificates.  I do have DNS for Cyrillic and Far East domains, but this web server is built with Delphi 2007 so no full Unicode. 

Angus

Quote

http://welcometothe.中国

That works OK, but redirects to an Alibaba site using western domains. 

I did try and register an accented test domain earlier with 123-Reg.  My attempt for co.uk was declined but they registered a com OK except it is missing all the accented characters, won't accept xn-- names only Unicode.   I'll try an eu instead, they must support accents.

Angus

It seems those using non-English domains hedge their bets on their sites:

Handshake done, error #0 - SSL Connected OK with TLSv1.2, cipher ECDHE-RSA-AES128-GCM-SHA256, key auth RSA, key exchange ECDH, encryption AESGCM(128), message auth AEAD
! VerifyResult: ok, Peer domain: мособлеирц.рф
3 Certificate(s) in the verify chain.
#3 Issued to (CN): mosobleirc.ru
Alt Domains (SAN): mosobleirc.ru, www.mosobleirc.ru, www.мособлеирц.рф, www.новый.мособлеирц.рф, мособлеирц.рф, новый.мособлеирц.рф
Issued by (CN): Let's Encrypt Authority X3, (O): Let's Encrypt
Expires: 11/05/2020 18:43:06, Signature: sha256WithRSAEncryption
 

Does anyone have any working Far East web sites with IDNs, Chinese, Japanese, etc, those I've tried are all dead.

Angus

Testing IDN with Windows 2019 DNS Server is proving problematic.  I've been able to set-up A records for scrúdú and xn--scrd-srab,  I would have expected DNS Manager to convert an accented domain to puncode ASCII but it actually stores scr\303\272d\303\272 in the file.  It also stores 16-bit characters.  I had to convert the punycode version manually.

Old versions of ICS are actually able to lookup scrúdú.ftptest.co.uk without any punycode, so it seems windows usea the full 8-bits for DNS queries,  Internally, we convert Unicode to ANSI before the query, so this will only work for code pages that match the DNS server. 

The real question is whether this DNS behaviour is by design or ignorance, perhaps internal networks are allow full 8-bit local names?  I've Googled a lot, but can not find any design recommendations for IDN and Windows DNS Server, indeed no mentions atall. 

I was planning on changing ICS to automatically convert IDNs in Unicode to punycode ASCII, but this would break existing internal applications using 8-bit ANSI, so perhaps it needs to be optional, I'd prefer ASCII to become the default.

Also, think the HTTP client and server need changing, because the Host: header needs to be punycode ASCII, probably Location: as well, but sure about sub directories, arguments are always character converted but does that apply to directories as well? 

Angus

Sorry, really do not have time to explore 20 year old code,  ICS still supports Delphi 7 and later, but not earlier due to lack of language features. 

Angus

Quote

https://мособлеирц.рф/

Not a single ASCII character in that name, converts to xn--90aijkdmaud0d.xn--p1ai with no single hyphens, although conversion back to Unicode does not like it. 

IDN is now working with forward DNS lookups, using the DnsLookup function, now need to look at other places that don't use that function like ping.

Angus

It is a very strange format, no real reason for re-ordering the letters. 

Angus

ICS already has the inline MIME processing used for email headers since international names are far more common than international domains, bu this needs to be done at application level, the SMTP and POP3 components don't do it internally. 

Rather than use the Windows IDN APIs, I've trying some pascal code first, multi-platform. 

Angus

Thanks for the URLs everyone, I've also setup www.éxàmplê.ftptest.co.uk although it's not working yet, will make sure it works with SSL as a server as well.

Angus

Thanks everyone, I'll add IDN support to TWSocket.DnsLookup and TWSocket.ReverseDnsLookup which fortunately have string arguments.  I'll set-up a sub-domain with accents for testing.  Then think about email. 

If anyone has any real IDN URLs  they would be useful for testing. 

Angus

Do you ever get questions about IDN support in Indy, because I don't recall it ever being mentioned in the ICS mailing list.  No point adding IDN support if no-one is going to use, except the one Delphi developer that emailed me, not about ICS.

Angus

Has anyone have a requirement to support IDNs in ICS, or have any punycode conversion functions to share with ICS?

Someone just filled in a form on my web site with email domain with an accented character, first time I've seen that.  My ICS email client can handle the MIME encoded email header for display, but is unable to reply to the email due to no IDN handling.  I guess punycode conversion would fix that, but not sure which email header fields need which encoding.

Or indeed whether we should try and handle IDN at a lower level in ICS. 

Guess I should register an accented domain name for testing.

Angus

If you can do so easily, removing ReqVerHigh and ReqVerLow from the DFMs is safest, maybe an earlier release of Delphi stored default values or you set them to 1 after the default changed to 2, when is not noted in the unit  Can not see the winsock version changing again in my life time.  

The next low level change is likely to be QUIC which Google has been developing for several years, this uses UDP for TCP like streams, so should not involve winsock, it should be supported by OpenSSL within a year or so, once the spec is finalised. 

Angus

Not sure how that could happen, THttpCli does not reference ReqVerHigh or ReqVerLow and the underlying TWSocket is not accessible for writing in current versions.  ReqVerHigh or ReqVerLow would only be saved in the DFM if you changed them from the default of 2, again unless you are using an ancient version of ICS.

Angus