dummzeuch 1621 Posted Thursday at 08:49 AM (edited) I am using ISshExec.Exec from @pyscripter's ssh-pascal library for running a command that calculates checksums on all files in a directory tree. Since this is potentially a large tree and some of the files are huge, this takes quite a while. Unfortunately Exec is synchronously waiting for the output of the command which creates two issues: My program has to wait for the command to finish before it can get any output. For the huge files calculating the checksum takes longer than the standard timeout which raises an exception in the Exec method (actually in ReadStringFromChannel) and stops reading (the command continues running though) Is there any existing way to get the output asynchronously? EDIT: I just found that my ssh-pascal library was outdated, in particular the reading code has changed significantly. Maybe my second point has become moot. I'll check. EDIT2: No, the timeout still occurs. EDIT3: Hm, now I can't reproduce the timeout. Maybe I didn't rebuild the executable correctly before trying, so it was still using the old code. Edited Thursday at 09:34 AM by dummzeuch Share this post Link to post
Kas Ob. 133 Posted Thursday at 10:46 AM I am intrigued here !, And really want to ask this , Did you asked any AI for solution ? Because the solution is very simple unless there is something missing or the ssh library is broken, it could be broken on both client and server, but what concern me is what server SSH is being used, and few numbers will be great to understand this. Anyway, my first suggestion is to hack your way into it, well it is a workaround to guarantee the TCP socket connection is alive and being pinged, just add another channel for port forwarding if that is allowed on server side, this should solve your problem and enforce ping for that channel on that socket to be communicated hence stopping the timeout, well unless the timeout is coming from different cause and has nothing to do with SSH per se. Also what are the setting on server for keepalive interval and max count ? Share this post Link to post
pyscripter 742 Posted Thursday at 11:24 AM (edited) ISshExec.Exec was substantially revised and now works in non-blocking mode. Are you using the latest source? Are you using the latest linssh2 binaries? You can execute ISshExec in a thread. Note that then it can be cancelled from the main thread using ISshExec.Cancel. Also you memory leak in TSshExec.Exec · Issue #18 · pyscripter/Ssh-Pascal has been fixed. Can you confirm that? Edited Thursday at 11:29 AM by pyscripter Share this post Link to post
dummzeuch 1621 Posted Thursday at 11:39 AM My problem is not that the main thread is blocked by the call but that I get the output of the command only after it has finished (which can take 30 minutes). During that time the output is received and stored in a memory stream to be converted to a string. Access to this partial output not available. I know that I could adapt the source code to change that behavior. But I was asking whether I may just be missing something (maybe a different interface?) in the library that already exists. @pyscripter Yes, I am (now) using the latest sources. No I haven't updated the binaries, but the problem is in the Delphi code. (Just in case you missed my edits in the original question: The timeout no longer occurs.) Share this post Link to post
pyscripter 742 Posted Thursday at 12:20 PM (edited) 44 minutes ago, dummzeuch said: I get the output of the command only after it has finished This is how it is designed to work. I will add an event that is triggered whenever output is added. Add an event to access partial output of ISshExec.Exec · Issue #20 · pyscripter/Ssh-Pascal Edited Thursday at 12:23 PM by pyscripter Share this post Link to post
dummzeuch 1621 Posted Thursday at 12:27 PM I'll probably implement something like this (if I get the time to finish this project). I'll of course submit a patch in that case. 1 Share this post Link to post
dummzeuch 1621 Posted Thursday at 12:47 PM (edited) Here you go: type TOnSshExecDataRecieve = procedure(const AData: TBytes; ADataLen: Int64) of object; // ... { Execute commands on the host and get Output/Errorcode back } ISshExec = interface ['{CA97A730-667A-4800-AF5D-77D5A4DDB192}'] procedure SetBufferSize(Size: Int64); procedure Cancel; // ---- change procedure SetOnStdErrReceive(ACallback: TOnSshExecDataRecieve); procedure SetOnStdOutReceive(ACallback: TOnSshExecDataRecieve); // ---- endchange procedure Exec(const Command: string; var Output, ErrOutput: string; var ExitCode: Integer); property BufferSize: Int64 write SetBufferSize; end; // ... type TSshExec = class(TInterfacedObject, ISshExec) private FSession : ISshSession; FBufferSize: Int64; FCancelled: Boolean; // ---- change FOnStdErrReceive: TOnSshExecDataRecieve; FOnStdOutReceive: TOnSshExecDataRecieve; procedure doOnStdErrReceive(const AData: TBytes; ADataLen: Int64); procedure doOnStdOutReceive(const AData: TBytes; ADataLen: Int64); // ---- end change procedure SetBufferSize(Size: Int64); procedure Cancel; procedure Exec(const Command: string; var Output, ErrOutput: string; var ExitCode: Integer); // ---- change procedure SetOnStdErrReceive(ACallback: TOnSshExecDataRecieve); procedure SetOnStdOutReceive(ACallback: TOnSshExecDataRecieve); // ---- end change public constructor Create(Session: ISshSession); end; // ... // ---- change procedure TSshExec.doOnStdErrReceive(const AData: TBytes; ADataLen: Int64); begin if Assigned(FOnStdErrReceive) then FOnStdErrReceive(AData, ADataLen); end; procedure TSshExec.doOnStdOutReceive(const AData: TBytes; ADataLen: Int64); begin if Assigned(FOnStdOutReceive) then FOnStdOutReceive(AData, ADataLen); end; // ---- end change procedure TSshExec.Exec(const Command: string; var Output, ErrOutput: string; var ExitCode: Integer); var Channel: PLIBSSH2_CHANNEL; M: TMarshaller; ReadBuffer, OutBuffer, ErrBuffer: TBytes; StdStream, ErrStream: TBytesStream; TimeVal: TTimeVal; ReadFds: TFdSet; BytesRead: ssize_t; ReturnCode: integer; OldBlocking: Boolean; begin if FSession.SessionState <> session_Authorized then raise ESshError.CreateRes(@Err_SessionAuth); FCancelled := False; Channel := libssh2_channel_open_session(FSession.Addr); if Channel = nil then CheckLibSsh2Result(libssh2_session_last_errno(FSession.Addr), FSession, 'libssh2_channel_open_session'); TimeVal.tv_sec := 1; // check for cancel every one second TimeVal.tv_usec := 0; StdStream := TBytesStream.Create(OutBuffer); ErrStream := TBytesStream.Create(ErrBuffer); SetLength(ReadBuffer, FBufferSize); OldBlocking := FSession.Blocking; FSession.Blocking := False; try Repeat ReturnCode := libssh2_channel_exec(Channel, M.AsAnsi(Command, FSession.CodePage).ToPointer); CheckLibSsh2Result(ReturnCode, FSession, 'libssh2_channel_exec'); Until ReturnCode <> LIBSSH2_ERROR_EAGAIN; // Stop waiting if cancelled of Channel is sent EOF while not FCancelled do begin // Wait until there is something to read on the Channel Repeat FD_ZERO(ReadFds); _FD_SET(FSession.Socket, ReadFds); ReturnCode := select(0, @ReadFds, nil, nil, @TimeVal); if ReturnCode < 0 then CheckSocketResult(WSAGetLastError, 'select'); if libssh2_channel_eof(Channel) = 1 then Break; Until (ReturnCode > 0) or FCancelled; try // Standard output BytesRead := libssh2_channel_read(Channel, PAnsiChar(ReadBuffer), FBufferSize); CheckLibSsh2Result(BytesRead, FSession, 'libssh2_channel_read_ex'); // ---- change if BytesRead > 0 then begin StdStream.WriteBuffer(ReadBuffer, BytesRead); DoOnStdOutReceive(ReadBuffer, BytesRead); end; // ---- end change // Error output BytesRead := libssh2_channel_read_stderr(Channel, PAnsiChar(ReadBuffer), FBufferSize); CheckLibSsh2Result(BytesRead, FSession, 'libssh2_channel_read_ex'); // ---- change if BytesRead > 0 then begin ErrStream.WriteBuffer(ReadBuffer, BytesRead); DoOnStdErrReceive(ReadBuffer, BytesRead); end; // ---- end change except on E: Exception do begin OutputDebugString(PChar(E.Message)); Break; end; end; // BytesRead will be either > 0 or LIBSSH2_ERROR_EAGAIN until // the command is processed if BytesRead = 0 then Break; end; Output := AnsiToUnicode(PAnsiChar(StdStream.Memory), StdStream.Size, FSession.CodePage); ErrOutput := AnsiToUnicode(PAnsiChar(ErrStream.Memory), ErrStream.Size, FSession.CodePage); // libssh2_channel_close sends SSH_MSG_CLOSE to the host libssh2_channel_close(Channel); if FCancelled then ExitCode := 130 // ^C on Linux else Exitcode := libssh2_channel_get_exit_status(Channel); finally StdStream.Free; ErrStream.Free; libssh2_channel_free(Channel); FSession.Blocking := OldBlocking; end; end; // ---- change procedure TSshExec.SetOnStdErrReceive(ACallback: TOnSshExecDataRecieve); begin FOnStdErrReceive := ACallback end; procedure TSshExec.SetOnStdOutReceive(ACallback: TOnSshExecDataRecieve); begin FOnStdOutReceive := ACallback; end; // ---- end change This passes the raw bytes received to the callback. Maybe it is possible to convert them to a string first? Or maybe call every time a line feed is received? Also, maybe the callbacks are enough so collecting the whole output is no longer necessary. I'll see how far I get with this. Edited Thursday at 12:50 PM by dummzeuch 1 Share this post Link to post
pyscripter 742 Posted Thursday at 02:12 PM (edited) 1 hour ago, dummzeuch said: This passes the raw bytes received to the callback. Maybe it is possible to convert them to a string first? Or maybe call every time a line feed is received? Also, maybe the callbacks are enough so collecting the whole output is no longer necessary. This is roughly what I had in mind. Suggestions: Have one event with an Enumerated parameter TExecOutput = (eoStdout, eoStdErr) Pass to the callback only the newly added bytes in a TBytes. They can be easily converted to strings using the ISshClient encoding. Keep the output as is. The user has a choice of not providing a callback. The overhead is small. Edited Thursday at 02:13 PM by pyscripter Share this post Link to post