[Question/Feature] SSL/TLS fallback using magic bytes

[FearDC 1]

Hi.

 

I have a simple question when it comes to ICS SSL/TLS server socket. Is it possible to fallback to non-encrypted communication between a SSL/TLS-encrypted server and a non-encrypted client when such client connects to a such server? The situation is as simple as there is SSL/TLS-encrypted server, it receives a connection from a client who does not support SSL/TLS-encryption. Could this client be able to continue the communication without SSL/TLS-encryption on same port? This does sound wrong, because SSL/TLS-encrypted communication is encrypted from the very beginning. But, there are magic bytes that could be detected before actually starting the actual SSL/TLS handshake > https://github.com/verlihub/tls-proxy/blob/master/proxy.go#L186 < This application managed to do so, so why not trying the same approach using ICS SSL/TLS server socket? I have already tried IcsHosts with BindSslPort and BindNonPort, but it does not work the way I'm expecting.

 

If this magic byte detection could be implemented into ICS, then this suite would be the first one to allow encrypted and non-encpypted communication on same port. This would become the number one solution to non-HTTP protocols that do not support redirects as easy as HTTP protocol does.

 

Regards.

Edited November 13, 2022 by FearDC

[Angus Robertson 526]

Technically, it is possible to recognise a non-SSL connection is being made to an SSL port, OpenSSL specifically checks if an HTTP header is being received rather than a HELLO packet and raises an error.   And hackers often do this, attempting to made non-SSL connections to port 443, no idea why.

 

But to fall back from SSL to non-SSL would require the co-operation of both client and server, a non-SSL client would never attempt to connect to port 443, unless incorrectly configured. So I'm not sure what scenario you are anticipating.  Perhaps some industrial environment where you use a special port 8080 or something for ease of configuration of both protocols on the same port? 

 

This is hardly a widely needed feature, so development would be hard to justify, except commercially.

 

Angus

 

[Fr0sT.Brutal 897]

2 hours ago, Angus Robertson said:

a non-SSL client would never attempt to connect to port 443

Probably a proxy with the same port for TLS and plain channel, or some other custom server.

[FearDC 1]

@Angus Robertson

I'm not using a HTTP server, this is completely different type of protocol. Lets imagine that a SSL-enabled client connects to a SSL-enabled TSSLWSocketServer, he sends the client hello part of SSL handshake - in order to recognize/use this data we still need to read the bytes from client socket, right? Then once we have read these very first bytes, we actually check two first bytes - once they are magic (means SSL/TLS handshake) we keep shaking hands with the client as in SSL handshake, while if these bytes are not magic, we could just passthrough the SSL handshake and go directly to the TCP data exchange and keep this data exchange SSL-less all the way, as if SSLEnable was set to False for this particular client only. Do you understand what I mean? I'm not talking futuristic ideas right now, this has been implemented in a real software, which allows SSL-enabled and non-SSL clients to use same TCP port in same application > https://github.com/direct-connect/go-dcpp/blob/master/hub/hub.go#L506 . In this example, server application detects those magic bytes and configures the client socket to work either in SSL or non-SSL mode all the way.

 

Ofcourse I understand that not all developers who work with TSSLWSocketServer would like this to be automated, because maybe the protocols they are working with, do actually send same type of magic bytes as part of the protocol. So this option should ofcourse be optional and disabled by default, so their projects don't get broken.

Edited November 14, 2022 by FearDC

[Angus Robertson 526]

I accept your concept is possible, but would be messy to implement in ICS, and add code that would be of no use to the vast majority of users but would end up in all server applications.

 

I suggest you derive a new component from TSSLWSocketServer, and implement it yourself.  If there is interest from other developers for this functionality, it could be added to ICS.

 

Angus

[FearDC 1]

That's what I wanted to hear, that this is possible.  This could be a great challenge for me, and a good solution for those many people asking for SSL and non-SSL on the same port.

 

In fact whole this idea started from ICSHosts collection - your latest invention to multi-listening socket. I did specify a SSL and non-SSL port just to test this, but my server application doesn't listen on both ports, instead it listens on just a single SSL-enabled port 1412 - the non-SSL port 1411 is completely omitted. I could find this piece of code:

Quote

            if (FBindNonPort > 0) or (FBindSslPort = 0) then begin
                AddBinding(FBindIpAddr, FBindNonPort, False, FBindIdxNone, FBindInfo);
                if FBindIpAddr2 <> '' then
                    AddBinding(FBindIpAddr2, FBindNonPort, False, FBindIdx2None, FBindInfo);
            end;
            if (FBindSslPort > 0) then begin
                AddBinding(FBindIpAddr, FBindSslPort, True, FBindIdxSsl, FBindInfo);
                if FBindIpAddr2 <> '' then
                    AddBinding(FBindIpAddr2, FBindSslPort, True, FBindIdx2Ssl, FBindInfo);
            end;

This basically means that setting up both SSL and non-SSL, still different ports, using ICSHosts, is not possible. Is this a bug?

 

Edited November 14, 2022 by FearDC

[Angus Robertson 526]

My own servers listen happily on multiple ports and addresses using IcsHosts without a problem.  This is my main web server:

 

Socket 1 State: Listening Only IPv4 on 217.146.102.150 port 80
Socket 2 State: Listening Only IPv6 on 2a00:1940:2:2::150 port 80
Socket 3 State: Listening Only IPv4 on 217.146.102.150 port 443 SSL
Socket 4 State: Listening Only IPv6 on 2a00:1940:2:2::150 port 443 SSL
Socket 5 State: Listening Only IPv4 on 217.146.102.155 port 80
Socket 6 State: Listening Only IPv6 on 2a00:1940:2:2::155 port 80
Socket 7 State: Listening Only IPv4 on 217.146.102.155 port 443 SSL
Socket 8 State: Listening Only IPv6 on 2a00:1940:2:2::155 port 443 SSL
Socket 9 State: Listening Only IPv6 on 2a00:1940:2:2::250 port 80
Socket 10 State: Listening Only IPv6 on 2a00:1940:2:2::250 port 443 SSL
Socket 11 State: Listening Only IPv4 on 217.146.102.153 port 80
Socket 12 State: Listening Only IPv6 on 2a00:1940:2:2::153 port 80
Socket 13 State: Listening Only IPv4 on 217.146.102.153 port 443 SSL
Socket 14 State: Listening Only IPv6 on 2a00:1940:2:2::153 port 443 SSL

 

Hosts=www.telecom-tariffs.co.uk,www.telecom-tariffs.uk,telecom-tariffs.co.uk,telecom-tariffs.uk
BindIpAddr=217.146.102.150
BindIpAddr2=2a00:1940:2:2::150
BindNonPort=80
BindSslPort=443

(lots more)

 

And different Let's Encrypt certificates on each address. 

 

Angus

[Angus Robertson 526]

The main issue in implementing your dual protocol concept is pre-reading the initial data received from the client, and then resetting so that it's read a second time after initialising SSL.  That will be messy with our event driven structure.   

 

Angus

[FearDC 1]

I did not setup ICSHosts using INI, i did set them up using Object Inspector.

Edited November 14, 2022 by FearDC

[FearDC 1]

Nevermind about ICSHosts right now, it's not what I currently want, I known that it works only with different ports. If I will get interested in ICSHosts, I will create another thread about it.

 

Now back to SSLEnable option. I managed to set it to False at runtime on DataAvailable callback, and actually got both SSL and non-SSL connection working on same port.  Now only need to find a way to detect those bytes and set SSLEnable eccordingly. To be continued...

Edited November 14, 2022 by FearDC

[Remy Lebeau 1260]

22 hours ago, FearDC said:

Is it possible to fallback to non-encrypted communication between a SSL/TLS-encrypted server and a non-encrypted client when such client connects to a such server?

I have implemented this kind of logic before, but only in plain socket code (though I suppose it can be applied to socket libraries like ICS or Indy, too).  It requires the TLS server to peek not recv the first few bytes of a new connection, and only if those bytes belong to the header of a TLS handshake then enable TLS on that connection (allowing the previously-peeked bytes to now be read by whatever TLS library you use), otherwise don't enable TLS on the connection and just read the connection raw.

[FearDC 1]

Finally got some time to get back to my project and completed the goal. Thanks for the hints @Remy Lebeau. Here are the results if anyone needs the hack without modifying TSSLWSocketServer class:

 

procedure TMainForm.SSLServerClientCreate(Sender: TObject; Client: TWSocketClient);
var
  mConn: TClient;
begin
  mConn := Client as TClient;
  mConn.OnDataAvailable := SSLServerDataAvailable;
  mConn.mTest := False;
end;

procedure TMainForm.SSLServerClientConnect(Sender: TObject; Client: TWSocketClient; Error: Word);
var
  mConn: TClient;
begin
  mConn := Client as TClient;
  mConn.SSLEnable := False;
end;

procedure TMainForm.SSLServerDataAvailable(Sender: TObject; ErrCode: Word);
var
  mConn: TClient;
  mPeek: TBytes;
  mData: String;
begin
  mConn := Sender as TClient;

  if not mConn.mTest then begin // first read
    SetLength(mPeek, 2);
    mConn.PeekData(mPeek, 2);
    mConn.mTest := True;

    if (mPeek[0] = 22) and (mPeek[1] = 3) then begin // client hello
      SetLength(mPeek, 0);
      mConn.SSLEnable := True;
      mConn.SSLContext := SSLContext;
      mConn.AcceptSSLHandshake;
      Exit;
    end;

    SetLength(mPeek, 0);
  end;

  mData := mConn.ReceiveStr;
end;

 

The only thing I'm worried about is memory leak due to not reading connected socket after data available call. At some point I had experience of that issue in another project.

 

Regards.

 

[Kas Ob. 91]

I did the same thing with in few servers, and as Remy and Angus said it should handled on lower level from the event layer, here a snippet i have in a project

    if (BytesRecv > 0) then
    begin

      if (FInSSLBuffer[0] = $16){ and (FInSSLBuffer[1] = $03)} then
      begin
        FConnecitonIsTLS := True;
        TLSInit;
        //
      end
      else
      begin
        //
        Move(FInSSLBuffer[0], FInBuffer[FInBufferCount], FInSSLBufferCount);
        BytesToProcess := FInSSLBufferCount;
        FInSSLBufferCount := 0;
      end;
    end;

Though i am not experienced in ICS, i would suggest to derive (inherit) your own server and make you detection, before triggering the event thus you will have your dual usage server, and drop in place,

 

On side note, i needed this for some reasons:

1) To remove huge amount of logging from online scripts and bots that keeps requesting plain HTTP on port 443, and i replied with and empty HTTP, when it is was HTTP.

2) I needed to add SSH and HTTP (as above), but i didn't want to go with multiple ports, so used 443 for SSH too, the same way it detected few bytes, as above it seems i used only one byte for SSL/TLS.

 

An argument for the need to have such functionality:

I needed to switch off SSL/TLS traffic and leave the plain traffic in extreme debugging/testing and watching unstable networks (WiFi) traffic on WireShark, also needed to stress the socket engine itself not the Secure layer and its CPU demanding, so for such case the ICS server could have a Boolean property to disable SSL/TLS, with default as False, and when it is True the traffic will skip all Secure layer handling, such feature will have its consequences and can be be not allowed to disable after enabling, but it is the responsibility of the developer (ICS user) to understand how and when to flip this property.

Not big or needed feature, but a feature non the less.

[Angus Robertson 526]

Quote

1) To remove huge amount of logging from online scripts and bots that keeps requesting plain HTTP on port 443, and i replied with and empty HTTP, when it is was HTTP.

I get massive logs from so called internet research companies probing my servers, I have about 40 public IP4s in three subnets and any servers listening are attacked daily, the firewalls try to restrict how many ports and IPs can be accessed, but all public servers suffer.  

 

ICS has a TIcsBlackList component that most of my servers use, any HTTP request by IP address is immediately blocked for 24 hours and new connections closed, well known IP ranges are permanently blocked, etc.

 

So checking for non-SSL connections on SSL ports would be a useful addition.  I'll experiment adding it to TSslSocketServer.

 

Angus

 

[FearDC 1]

On 11/14/2022 at 9:46 PM, Remy Lebeau said:

I have implemented this kind of logic before, but only in plain socket code (though I suppose it can be applied to socket libraries like ICS or Indy, too).  It requires the TLS server to peek not recv the first few bytes of a new connection, and only if those bytes belong to the header of a TLS handshake then enable TLS on that connection (allowing the previously-peeked bytes to now be read by whatever TLS library you use), otherwise don't enable TLS on the connection and just read the connection raw.

 

@Remy Lebeau Since ICS does not support Linux yet, I would like to use Indy instead. My application requires TLS 1.3, but I will figure this out later, since I heard that TLS 1.3 is possible with Indy using some modifications. Now to my project..

 

There is a TCP server listening on incoming connections, it checks whether client supports TLS or not using magic bytes detection as described earlier (client hello > client sends 22+03 bytes on first read), then accepting TLS or passing through, and mapping protocol data to destination server. I have tried using TIdMappedPortTCP, but without success, guess because InputBuffer has already been received by the point where OnExecute is executed. So it seems that I need to clone and modify TIdMappedPortTCP - which I did too. But in order to start proper "PassThrough := False" and "StartSSL" after "AContext.Connection.IOHandler.CheckForDataOnSource(0)" and "AContext.Connection.IOHandler.InputBuffer.PeekByte(0) == 22 and AContext.Connection.IOHandler.InputBuffer.PeekByte(1) = 03", it seems that I need to modify TIdCustomTCPServer aswell, in order to "undo" already received buffer. I'm almost there anyway, but I need your help, since you are expert in Indy. Could you please help me?

 

If something of what I said is unclear, please let me know. I will try to describe even better.

 

Regards.

Edited January 26 by FearDC

[Remy Lebeau 1260]

52 minutes ago, FearDC said:

I have tried using TIdMappedPortTCP, but without success, guess because InputBuffer has already been received by the point where OnExecute is executed.

If you want to do this TLS handshake detection in TIdMappedPortTCP, the best place to handle this is in the OnBeforeConnect event, which is fired immediately after a new client is accepted by TIdMappedPortTCP and before its associated OutboundClient is connected to the next server.

 

The OnExecute event is fired after the OutboundClient is connected to the next server and the accepted client has sent data which has been read into its InputBuffer.  By then, its too late for the TLS detection since the connection is already receiving application data after the TLS session is established.

52 minutes ago, FearDC said:

So it seems that I need to clone and modify TIdMappedPortTCP

No, you do not.

52 minutes ago, FearDC said:

But in order to start proper "PassThrough := False" and "StartSSL", it seems that I need to modify TIdCustomTCPServer aswell, in order to "undo" already received buffer.

You can't "undo" a socket read.  Nor will doing so help you with detecting the TLS handshake, anyway.  You have to peek the bytes directly from the socket itself, not from the InputBuffer that is on top of the socket.

Edited January 26 by Remy Lebeau

[FearDC 1]

Thank you @Remy Lebeau, you are very helpful as usual!

 

procedure OnServerBeforeConnect(AContext: TIdContext);
var
  mTime: Integer;
  mPeek: Array [0..1] of AnsiChar;
begin
  AContext.Binding.GetSockOpt(Id_SOL_SOCKET, Id_SO_RCVTIMEO, mTime);
  AContext.Binding.SetSockOpt(Id_SOL_SOCKET, Id_SO_RCVTIMEO, 5);

  if (Recv(AContext.Binding.Handle, mPeek, 2, MSG_PEEK) = 2) and (mPeek[0] = #22) and (mPeek[1] = #03) then // client hello
    TIdSSLIOHandlerSocketOpenSSL(AContext.Connection.IOHandler).PassThrough := False;

  AContext.Binding.SetSockOpt(Id_SOL_SOCKET, Id_SO_RCVTIMEO, mTime);
end;

 

Edited January 27 by FearDC

[Remy Lebeau 1260]

That is the basic jist of it, yes.  Though, detecting a viable handshake is slightly more complex than just testing the first 2 bytes.  I did write some code for this a long time ago, I'll have to look for it.