ssh tunnel with ssh-pascal

[dummzeuch 1505]

I'm trying to get ssh tunneling to work with ssh-pascal. The goal is to replace the currently used ssh tunnel for Remote Desktop via Putty with my own program.

 

I adapted the LocalForward demo and I can connect to and read from

http://detectportal.firefox.com/success.txt

(the original url http://git.php.net/ is apparently no longer available)

 

Then I changed the code to connect to my employer's internet facing ssh server port and forwarded a local port to the remote 3389 port of a computer in our LAN.

I can log on fine to the ssh server, even the Remote Desktop client starts, connects and gets a valid logon. But then it asks whether it should connect even though the certificate is not from a trusted certifying authority (which is normal for that connection) and pressing Yes there odes not do anything and the RDC connection times out after a while.

 

I added some debug WriteLns and found that the processing thread while executing TSshTunnel.ForwardLocalPort is waiting to read from ForwardSock here, after apparently FD_ISSET returned True:

    SetLength(Buf, FBufferSize);
    while not FCancelled do
    begin
      FD_ZERO(ReadFds);
      _FD_SET(ForwardSock, ReadFds);
      _FD_SET(ChannelSock, ReadFds);
      // wait for action
      ReturnCode := select(0, @ReadFds, nil, nil, @TimeVal);
      if ReturnCode < 0 then CheckSocketResult(ReturnCode, 'select');
      if ReturnCode = 0 then Continue;

      // we should be able to read now
      WriteLn('Checking ForwardSock');
      if FD_ISSET(ForwardSock, ReadFds) then begin
        WriteLn('Reading from ForwardSock');
        Read := recv(Forwardsock, Buf[0], FBufferSize, 0); // <=== hangs here
        if Read = SOCKET_ERROR then
          CheckSocketResult(WSAGetLastError, 'recv');
        if Read = 0 then
          raise ESshTunnelError.CreateRes(@Err_ConnClosed);
        WriteLn(Read, ' bytes read from ForwardSock');
        Total := 0;
        While (Total < Read) do
        begin
          Written := libssh2_channel_write(Channel, PAnsiChar(Buf) + Total, Read - Total);
          // Takes care of LIBSSH2_ERROR_EAGAIN
          CheckLibSsh2Result(Written, FSession, 'libssh2_channel_write');
          Inc(Total, Written);
          Writeln(Written, ' bytes written to ChannelSock');
        end;
      end;
      WriteLn('Checking ChannelSock');
      if FD_ISSET(ChannelSock, ReadFds) then begin
        WriteLn('Reading from ChannelSock');
        Read := libssh2_channel_read(channel, PAnsiChar(Buf), FBufferSize);
        if Read = LIBSSH2_ERROR_EAGAIN then
          // Go to Wait state again
          Continue;
        CheckLibSsh2Result(Read, FSession, 'libssh2_channel_read');
        WriteLn(Read, ' bytes read from ChannelSock');
        Total := 0;
        While (Total < Read) do
        begin
          Written := send(forwardsock, Buf[Total], Read - Total, 0);
          if Written = SOCKET_ERROR then
            CheckSocketResult(WSAGetLastError, 'send');
          Inc(Total, Written);
          Writeln(Written, ' bytes written to ForwardSock');
        end;
      end;
    end;

These are the last lines of the debug output:

Checking ChannelSock
Checking ForwardSock
Checking ChannelSock
Reading from ChannelSock
88 bytes read from ChannelSock
88 bytes written to ForwardSock
Checking ForwardSock
Reading from ForwardSock

Has anybody ever used this code successfully for more than requesting a simple webpage?

Edited November 6 by dummzeuch

[Kas Ob. 121]

@dummzeuch Can't test it right now, as it need more code to make it work, but

 

Don't ever use select in that way, in general all looks fine and correct, but without socket errors not the ones with API calls return, but the ones from "exceptfds", without checking for socket level errors you are driving in the dark, if error was signaled on the socket then read will hang for ever !

 

Add exceptfds, and then check for errors before read or write, as in many cases both error and read/write might be triggered and reported at the same time, in your case most likely this what did happen, an acknowledgment received to close and both readfds and exceptfds had been signaled.

[pyscripter 689]

@dummzeuch I have pushed a couple of fixes to SshTunnel.pas (issues found by Copilot!).  Could you please check whether they make a difference?  One of them might:  

 

SocketOption := 1; // Initialize SocketOption

Edited November 6 by pyscripter

[dummzeuch 1505]

13 hours ago, pyscripter said:

@dummzeuch I have pushed a couple of fixes to SshTunnel.pas (issues found by Copilot!).  Could you please check whether they make a difference?  One of them might:  

 

SocketOption := 1; // Initialize SocketOption

Unfortunately the problem persists. I'll have a deeper look into it later. Thanks a lot.

[pyscripter 689]

2 hours ago, dummzeuch said:

Unfortunately the problem persists.

The code in SshTunnel.pas is based on this libssh2 example.  Maybe I missed something, in converting it to Delphi.   It would be nice to fix it.

[dummzeuch 1505]

1 hour ago, pyscripter said:

The code in SshTunnel.pas is based on this libssh2 example.  Maybe I missed something, in converting it to Delphi.   It would be nice to fix it.

That C++ example does not use a separate thread as far as I can see. Not sure whether that makes any difference.

Edited November 7 by dummzeuch

[dummzeuch 1505]

The original code calls SetScokOpt before bind. Your code does it the other way round. Not sure whether that makes any difference either.

[dummzeuch 1505]

Turns out that reading from the local socked returned an error with error code 10054 "An existing connection was forcibly closed by the remote host" and raised an ESocketError exception. I never saw that exception because I had disabled the exception notification for this exception type in the debugger (because it happens every few seconds when debugging a IDE plugin without connection to the elc server).

 

So I figured that the remote desktop client closes the socket after login and then tries to open a new connection, but since the thread exits after that exception it cannot connect. And guess what: I just restarted the program again before clicking "Yes" and I've got a working remote desktop session now.

 

So the solution in this case is to start listening for a connection again after the previous one was closed. I'll try that later.

 

Edited November 8 by dummzeuch

[dummzeuch 1505]

Basically the problem I am facing here is the same as issue#11 reported on GitHub, only that I have two connections one after another and the person who opened that issue has them in parallel.

 

And that's rather difficult to fix based on the current structure because TSshTunnel.ForwardLocalPort is executed completely in one thread.

In order for this to work, there must be one thread that calls select on ListenSock and then starts another thread whenever there is a connection to the socket and let that thread handle the forwarding from it until that connection is closed. The original thread must then again call select on ListenSock.

This would handle multiple connections serially and in parallel.

[pyscripter 689]

1 hour ago, dummzeuch said:

This would handle multiple connections serially and in parallel.

A Pull Request implementing an alternative design that resolves this issue would be much appreciated.

Edited November 8 by pyscripter

[dummzeuch 1505]

25 minutes ago, pyscripter said:

A Pull Request implementing an alternative design that resolves this issue would be much appreciated. 

I can't promise anything, but if I get it to work, I'll submit one.

[pyscripter 689]

@dummzeuch  I think handling connections sequentially should be quite easy, without resorting to multiple threads.  Once one is closed you wait for the next one. 

Please bear in mind, that there are limitations when using libssh2 from multiple threads:

 

Quote

 

Using libssh2 in a multithreaded application requires some caution. While libssh2 itself can be used in a multithreaded environment, there are important limitations to be aware of:

1. Initialization: You need to ensure that libssh2_init() is called in a thread-safe manner. This means it should be called before any threads are created that will use libssh2¹.

2. Session Handling: You cannot use a single session (or channels for a single session) across multiple threads simultaneously. Doing so can lead to internal state corruption². Each thread should have its own session if they need to perform SSH operations concurrently.

3. Thread Safety: The underlying cryptographic libraries (like OpenSSL or GnuTLS) need to be initialized in a thread-safe manner. This often involves setting up mutexes for the cryptographic operations¹.

If you follow these guidelines, you should be able to use libssh2 in a multithreaded application without running into major issues. Do you have a specific use case or problem you’re encountering with libssh2?

 

 

Points 1 and 3 are the same, since the crypto functions are initialized by libssh2_init.   Ssh_pascal handles this in a thread safe manner.  The second point, I think means that you either have a session per thread or you protect with a lock, access to the same session and its channels,

 

Edited November 8 by pyscripter

[dummzeuch 1505]

23 hours ago, pyscripter said:

A Pull Request implementing an alternative design that resolves this issue would be much appreciated.

I have submitted 3 pull requests (none solving that problem yet), but I have been doing something wrong: The branches from which I created the pull requests contain more commits than expected.

[pyscripter 689]

1 hour ago, dummzeuch said:

but I have been doing something wrong

Probably not.  When you add commits to the same branch, the commits are added to the PR.  If you want to have separate PRs you need different branches for each change. Anyway your changes have been merged and you are listed as a contributor.

Edited November 9 by pyscripter

[dummzeuch 1505]

2 hours ago, pyscripter said:

Probably not.  When you add commits to the same branch, the commits are added to the PR.  If you want to have separate PRs you need different branches for each change. Anyway your changes have been merged and you are listed as a contributor.

I used 3 different branches, but I guess the mistake was merging those changes into my master branch and creating new branches for the other changes from my master branch which already contained the changes from the first branch. I'll try it differently next time.

 

Thanks for merging.

[pyscripter 689]

@dummzeuchI have updated the build scripts and the binaries which do not need OpenSSL.

 

Could you please try the new binaries and see whether you still get the error you mentioned?  These binaries work well with ssh-agent password-less authentication, whilst the Php ones don't.

 

Also would it be possible to post a test project that requires multiple connections, so that I can have a go at fixing the issue?

Edited November 9 by pyscripter

[dummzeuch 1505]

13 hours ago, pyscripter said:

@dummzeuchI have updated the build scripts and the binaries which do not need OpenSSL.

 

Could you please try the new binaries and see whether you still get the error you mentioned?  These binaries work well with ssh-agent password-less authentication, whilst the Php ones don't.

I just tried sshexec on my home Linux "server" and it worked now. It also automatically took my ssh key from Pageant.

 

13 hours ago, pyscripter said:

Also would it be possible to post a test project that requires multiple connections, so that I can have a go at fixing the issue?

Hm, I just tried to simply request a web page through the tunnel twice in a row, but apparently the browser keeps the connection open even if I close that tab. I had to wait for a timeout. for a request to fail. But closing the whole browser window rather than just the tab worked. The connection will be closed and when retrying in a new window the request will fail.

 

[dummzeuch 1505]

@pyscripter see my latest pull request for some helpful debug output options

 

This currently requires a hack using TMethod because LocalForward.Main is not an object. Would it be OK, if I made it a method of a TLocalForward object and create that in the Main procedure?

Edited November 10 by dummzeuch

[pyscripter 689]

@dummzeuch Please see my latest commit.  TSshTunnel.ForwardLocalPort has been refactored and can now handle multiple connections sequentially.  So your use case may be working.  Could you please test.  

 

I will also see whether the connections can be handled in parallel.  Is there a use case for that?  Can you have multiple connections simultaneously on the same port? 

[dummzeuch 1505]

7 hours ago, pyscripter said:

@dummzeuch Please see my latest commit.  TSshTunnel.ForwardLocalPort has been refactored and can now handle multiple connections sequentially.  So your use case may be working.  Could you please test.  

I'll test your changes as soon as I am awake properly and had breakfast.

 

edit: It seems to work.

I can connect to the web server using multiple browser instances sequentially without restarting the LocalForward demo. I haven't looked at the code yet.

 

I'll try to change my Remote Desktop forwarding tool to use your new code.

edit: Nope, doesn't work. After logon, while the Remote Desktop confirmation dialog is being shown An exception "Windows socket error: An existing connection was forcibly closed by the remote host (10054), API 'recv'" occurs in SshTunnel, Line 170, which aborts the listening the thread. Afterwards of course no further connections are being accepted.

7 hours ago, pyscripter said:

I will also see whether the connections can be handled in parallel.  Is there a use case for that?  Can you have multiple connections simultaneously on the same port?

A web server has multiple connections on the same port, doesn't it? So does ssh itself. Or a database server.

A simple use case would be if you want to test some page on a web server using different browsers in parallel.

Edited November 11 by dummzeuch

[pyscripter 689]

In the latest commit, multiple connections are handled in separate threads. So, Multiple Connections on LocalForward · Issue #11 · pyscripter/Ssh-Pascal is fixed.  Seems to work fine, but could you please test it? 

Edited Tuesday at 10:34 AM by pyscripter

[dummzeuch 1505]

2 hours ago, pyscripter said:

In the latest commit, multiple connections are handled in separate threads. So, Multiple Connections on LocalForward · Issue #11 · pyscripter/Ssh-Pascal is fixed.  Seems to work fine, but could you please test it? 

Hm, I must have broken something. It compiles, but it doesn't work at all. I'll have a closer look later.

[dummzeuch 1505]

22 hours ago, pyscripter said:

In the latest commit, multiple connections are handled in separate threads. So, Multiple Connections on LocalForward · Issue #11 · pyscripter/Ssh-Pascal is fixed.  Seems to work fine, but could you please test it? 

I found my bug: I had for testing purposes changed the RemoteHost to one that requires different credentials, so log on failed.

Now, having changed that back, the tunneling Remote Desktop works as expected. So your change works for me too.

 

Thanks a lot!

 

As for parallel connections:

The unmodified LocalForward demo works for two different browsers in parallel. But after closing the second browser I get ESshError "LibSSh2 error: Unable to send channel-open request (-7), on API "libssh2_channeld_direct_tcpip_ex" in SshTunnel, line 170. This then happens every time I try to connect to http://localhost:12345/success.txt So there still seems to be something amiss.

 

[pyscripter 689]

3 hours ago, dummzeuch said:

As for parallel connections:

The unmodified LocalForward demo works for two different browsers in parallel. But after closing the second browser I get ESshError "LibSSh2 error: Unable to send channel-open request (-7), on API "libssh2_channeld_direct_tcpip_ex" in SshTunnel, line 170. This then happens every time I try to connect to http://localhost:12345/success.txt So there still seems to be something amiss.

I committed a change trying to fix this.  Could you please try once more.

[dummzeuch 1505]

3 hours ago, pyscripter said:

I committed a change trying to fix this.  Could you please try once more.

No dice either. Still the same error message. Sorry, I am too tired to be of much help here.