Jump to content
al17nichols

how to flush buffers on Indy components

Recommended Posts

Posted (edited)

What is the best way to flush the buffer on both the indy client tidtcpclient and server tidtcpserver components?  Also, is there an acknowledgement when the flush is complete.  This includes flushing read and write buffers for both components. 

 

Edited by al17nichols
clarity

Share this post


Link to post
18 hours ago, al17nichols said:

What is the best way to flush the buffer on both the indy client tidtcpclient and server tidtcpserver components?

I don't really understand what you are asking for, but why would you ever need something like this?  Please provide a use-case example.

18 hours ago, al17nichols said:

Also, is there an acknowledgement when the flush is complete.

No (if such a flush were possible).

18 hours ago, al17nichols said:

This includes flushing read and write buffers for both components.

By default, there is no write buffering beyond the send buffer that is built-in to the underlying socket at the OS layer.  You can't "flush" that buffer, but you can reduce its size with the SO_SNDBUF socket option, or disable/avoid it with the TCP_NODELAY socket option (Indy's UseNagle property).  On top of that, Indy does provide an application-level send buffer via the IOHandler.WriteBuffer...() methods.  That buffer is flushed to the socket when the specified threshold is reached, or when WriteBufferFlush() or WriteBufferClose() is called. That buffer can be discarded with WriteBufferCancel().

 

Read buffering is a bit trickier. There is the receive buffer that is built-in to the underlying socket at the OS layer. You can't "flush" that buffer, only reduce its size with the SO_RCVBUF socket option. On top of that is the application-level buffering that Indy implements via the IOHandler.InputBuffer property, where the IOHandler.RecvBufferSize property controls how many bytes Indy attempts to read from the socket's receive buffer into the InputBuffer at a time.  Once data is in the InputBuffer, you can "flush" that buffer by calling its Clear() or Remove() method (in the latter case, you could use the IOHandler.Discard() method instead).

 

So, what is it that you really want to "flush", exactly? What is the scenario that you feel "flushing" is the solution for?

Share this post


Link to post

Thanks for your reply.  A scenario can occur when receiving data that an exception occurs or something happens that disrupts the flow of data.  I want to flush the remaining data that is being received so that I can reread the data.  I was doing a file transfer and the transfer was disrupted.  I read from the socket and there is more data coming from the buffer.  I have to close the socket and reopen to resync communication.  I hope this makes sense.

 

Share this post


Link to post
Posted (edited)

I don't know if its the right way but with Indy10 I use:
 

unit osCustomConnection;

interface

uses
  Windows,

  IdGlobal,
  IdBuffer,
  IdIOHandler,
  IdTCPClient,
  IdTCPConnection;

type
  TCustomConnection = class
  private
    FDevice: TIdTCPClient;
  private
    function GetConnected: Boolean;
    function GetHost: string;
    function GetPort: Integer;
    function GetInputBuffer: TIdBuffer;
  private
    procedure SetHost(const Value: string);
    procedure SetPort(Value: Integer);
  public
    procedure Connect(const ATimeout: Integer = IdTimeoutDefault);
    procedure Disconnect;
    function ReadFromStack(const ARaiseExceptionIfDisconnected: Boolean = True; ATimeout: Integer = IdTimeoutDefault; const ARaiseExceptionOnTimeout: Boolean = True): Integer;
    procedure WriteBuffer(const ABuffer; AByteCount: Integer; const AWriteNow: Boolean = False);
  public
	function FlushInputBuffer: Integer;  
  public
    property Connected: Boolean read GetConnected;
    property Host: string read GetHost write SetHost;
    property Port: Integer read GetPort write SetPort;
    property InputBuffer: TIdBuffer read GetInputBuffer;
  public
    constructor Create;
    destructor Destroy; override;
  end;

  TIdIOHandlerHelper = class(TIdIOHandler)
  public
    function ReadFromSource(ARaiseExceptionIfDisconnected: Boolean; ATimeout: Integer; ARaiseExceptionOnTimeout: Boolean): Integer;
  end;

implementation

{ TCustomConnection }

procedure TCustomConnection.Connect(const ATimeout: Integer);
begin
  FDevice.ConnectTimeout := ATimeOut;
  FDevice.Connect;
end;

constructor TCustomConnection.Create;
begin
  FDevice := nil;

  FDevice := TIdTCPClient.Create(nil);
end;

destructor TCustomConnection.Destroy;
begin
  FDevice.Free;

  inherited;
end;

procedure TCustomConnection.Disconnect;
begin
  FDevice.Disconnect;
end;

function TCustomConnection.GetConnected: Boolean;
begin
  Result := FDevice.Connected;
end;

function TCustomConnection.GetHost: string;
begin
  Result := FDevice.Host;
end;

function TCustomConnection.GetInputBuffer: TIdBuffer;
begin
  Result := FDevice.IOHandler.InputBuffer;
end;

function TCustomConnection.GetPort: Integer;
begin
  Result := FDevice.Port;
end;

function TCustomConnection.ReadFromStack(const ARaiseExceptionIfDisconnected: Boolean; ATimeout: Integer; const ARaiseExceptionOnTimeout: Boolean): Integer;
begin
  Result := TIdIOHandlerHelper(FDevice.IOHandler).ReadFromSource(ARaiseExceptionIfDisconnected, ATimeout, ARaiseExceptionOnTimeout);
end;

procedure TCustomConnection.SetHost(const Value: string);
begin
  FDevice.Host := Value;
end;

procedure TCustomConnection.SetPort(Value: Integer);
begin
  FDevice.Port := Value;
end;

procedure TCustomConnection.WriteBuffer(const ABuffer; AByteCount: Integer; const AWriteNow: Boolean);
begin
  if AByteCount = 0 then Exit;
  FDevice.IOHandler.WriteDirect(TIdBytes(@ABuffer), AByteCount);
end;

// flushes communications device input buffer
function TCustomConnection.FlushInputBuffer: Integer;
begin
  if not Connected then
    Result := 0
  else
  begin
    ReadFromStack(False, 1, False);
    Result := InputBuffer.Size;
    InputBuffer.Clear;
  end;
end;

{ TIdIOHandlerHelper }

function TIdIOHandlerHelper.ReadFromSource(ARaiseExceptionIfDisconnected: Boolean; ATimeout: Integer; ARaiseExceptionOnTimeout: Boolean): Integer;
begin
  Result := inherited ReadFromSource(ARaiseExceptionIfDisconnected, ATimeout, ARaiseExceptionIfDisconnected);
end;

end.

It's a part of a more complex class (reduced to be simple to be read) but works.

 

I use Indy10 TCP to capture encoded/encrypted/compressed continuous flow of data and so I've moved to public ReadFromSource,

which call private TIdIOHandler.ReadFromSource function, to manage better the flow in a capture/decode/decrypt/uncompress thread.

FlusInputBuffer calls a ReadFromStack to capture pending data en delete InputBuffer.Size return how many bytes were flushed.

Edited by shineworld

Share this post


Link to post
8 hours ago, al17nichols said:

I was doing a file transfer and the transfer was disrupted.  I read from the socket and there is more data coming from the buffer.  I have to close the socket and reopen to resync communication.

I don't get it. If you close a socket, you automatically discard all the buffers.

Share this post


Link to post
Posted (edited)

The desire is not to close the socket, but remove all of the bytes like a resync so that we can start communicating again.  The fact that I do not want to close and reopen the socket was not clear, but I do not want to close, but I do want to restart the communication.

 

 

Edited by al17nichols
clarity

Share this post


Link to post

Can you just read up remains to a junk buffer and then start new conversation?

Also, what does "the transfer was disrupted" mean? Some troubles at client side? Because if there's something wrong at server side, there will be no "data coming".

Share this post


Link to post
On 8/2/2022 at 4:00 PM, al17nichols said:

A scenario can occur when receiving data that an exception occurs or something happens that disrupts the flow of data.  I want to flush the remaining data that is being received so that I can reread the data.

Unless the data is structured in such a way that you can definitively identify, from ANY point within the data, where the data ends and the next piece of communication begins, then "flushing" the existing data to resume communications is usually not a viable option.  Especially if the next piece of communication has already been received and buffered and waiting for your code to read it from the buffer.

Quote

I was doing a file transfer and the transfer was disrupted.

Are you using a standard protocol like FTP for the transfer, or are you using a custom protocol?

Quote

I read from the socket and there is more data coming from the buffer. I have to close the socket and reopen to resync communication.

Unless the exception is a socket error, then you don't know the actual state of the socket, or how the data was buffered by the socket or by Indy.  So, once the exception occurs, usually the only sane thing to do is to just disconnect and reconnect to restart communications fresh, yes.  This is true for any TCP library, it is not limited to just Indy.  Since you know how much data you transferred before the exception, you can resume the transfer from that position, if your system allows for resuming.

Edited by Remy Lebeau

Share this post


Link to post
On 8/2/2022 at 10:58 PM, shineworld said:

I don't know if its the right way but with Indy10 I use:

FYI, your use of the TIdIOHandler.ReadFromSource() method is just mirroring what the public TIdIOHandler.CheckForDataOnSource() method does internally, so you don't really need your TIdIOHandlerHelper class at all, just use CheckForDataOnSource(1) instead of ReadFromStack(False, 1, False).

  • Like 1

Share this post


Link to post
On 8/3/2022 at 7:08 AM, al17nichols said:

The desire is not to close the socket, but remove all of the bytes like a resync so that we can start communicating again.  The fact that I do not want to close and reopen the socket was not clear, but I do not want to close, but I do want to restart the communication.

Unless your communication protocol is specifically designed to allow resyncing communications without closing the socket, then this is usually not a viable option.  What does your protocol actually look like? How are your messages delimited? What does your "disrupted flow of data" look like? At the very least, it is possible to discard bytes only until the next message delimiter is reached.  But you won't know if you discarded a (partial) message that was important to your communications, so resyncing via a reconnect is usually the best option.

Edited by Remy Lebeau

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×