Download file from AWS S3 to stream

[M. Toussaint 1]

Hi,

From our application we use several procedures and functions to upload files to AWS S3 and also download text-like files from AWS S3.

 

We use the function below to get the contents of text-like files (.SVG, .TXT, .CSV) from S3, but we have no idea how to (download and) process binary files.
 

function S3Get_Object_AsString(const bucket, key: string): string;
var
  pyKey            : TPythonDelphiVar;
  pyBucket         : TPythonDelphiVar;
  pyObj            : TPythonDelphiVar;
  PyEngine         : IPyEngineAndGIL;
  pyError          : TPythonDelphiVar;

  Script          : TStringList;
  cModuleName     : AnsiString;

begin
  pyKey         := TPythonDelphiVar.Create(nil);
  pyBucket      := TPythonDelphiVar.Create(nil);
  pyObj         := TPythonDelphiVar.Create(nil);
  pyError       := TPythonDelphiVar.Create(nil);

  script := TStringList.Create;
  try
    PyEngine    := TPyEngineAndGIL.Create;
    cModuleName := AnsiString(TGUID.NewGuid.ToString);


    pyBucket.Engine   := PyEngine.getPyEngine;
    pyBucket.VarName  := 'BUCKET';
    pyBucket.Module   := cModuleName;
    pyBucket.Initialize;
    pyBucket.Value    := bucket;

    pyKey.Engine  := PyEngine.getPyEngine;
    pyKey.VarName := 'S3KEY';
    pyKey.Module  := cModuleName;
    pyKey.Initialize;
    pyKey.Value   := key;

    pyObj.Engine   := PyEngine.getPyEngine;
    pyObj.VarName  := 'OBJ';
    pyObj.Module   := cModuleName;
    pyObj.Initialize;

    pyError.Engine  := PyEngine.getPyEngine;
    pyError.VarName := 'ERROR';
    pyError.Module  := cModuleName;
    pyError.Initialize;
    pyError.Value   := '';

    InitBotoClient(Script);

    Script.Add('import json ');
    Script.Add('from botocore.exceptions import ClientError ');

    Script.Add('s3_client = client(''s3'')');

    Script.Add('try: ' );
    Script.Add('    s3_object = s3_client.get_object(Bucket=BUCKET.VALUE, Key=S3KEY.VALUE)');
    Script.Add('    OBJ.VALUE = s3_object[''Body''].read()');
    Script.Add('except ClientError as e: ');
    Script.Add('    print("Get Object failed:" + e.response[''Error''][''Message''])');
    Script.Add('else:');
    Script.Add('    print("Get Object:" + BUCKET.VALUE + "-" + S3KEY.VALUE)');


    PyEngine.getPyEngine.ExecModule := cModuleName;
    PyEngine.getPyEngine.ExecStrings(Script);

    if pyError.ValueAsString <> '' then
    begin
      //Log(pyError.ValueAsString  + 'bucket: ' + bucket + ' key:' + key , 'E');
      result := '';
    end
    else
    begin
      if VarIsStr(pyObj.Value) then result := pyObj.Value else result := '';
    end;


  finally
    Script.Free;
    pyBucket.Finalize;
    pyKey.Finalize;
    pyObj.Finalize;
    pyError.Finalize;

    pyBucket.Free;
    pyKey.Free;
    pyObj.Free;
    pyError.Free;
  end;
end;

 

So far we found an alternative for downloading a file object from S3, but how to process this in Delphi?

    Script.Add('import json ');
    Script.Add('import io ');
    Script.Add('from botocore.exceptions import ClientError ');

    Script.Add('s3_client = client(''s3'')');
    Script.Add('io_stream = io.BytesIO()');

    Script.Add('try: ' );
    // Script.Add('    s3_object = s3_client.get_object(Bucket=BUCKET.VALUE, Key=S3KEY.VALUE, )');
    // Script.Add('    OBJ.VALUE = s3_object[''Body''].read()');
    // Use download_fileobj to gat a file-like object
    Script.Add('   s3_client.download_fileobj(Bucket=BUCKET.VALUE, Key=S3KEY.VALUE, Fileobj=io_stream)');
    Script.Add('   io_stream.seek(0)');
    Script.Add('   OBJ.VALUE = io_stream.getvalue()');
    Script.Add('except ClientError as e: ');
    Script.Add('    print("Get Object failed:" + e.response[''Error''][''Message''])');
    Script.Add('else:');
    Script.Add('    print("Get Object:" + BUCKET.VALUE + "-" + S3KEY.VALUE)');

 

[darnocian 86]

I have a few points:

1. it may be easier to use the Appercept AWS SDK. You don't need to bring Python into the picture in order to use it. Understandably, the SDK is currently only available in Delphi Enterprise. 

2. in the code above, as you are downloading binary, I'd say it would be easier just write to file from python, and then read it back in from Delphi. I have not checked the marshalling code in the Py4D layer, but I assume that is what you are after. It depends on how you intend to use it, if the file is really big, do you want the whole thing in memory?

3. in python, I think you could also do

Quote

pbarray = io_stream.getvalue()

 and in delphi,

Quote

var barray = PyEngine.GetVarAsVariant('pbarray');

I suspect this might mean there could be 2 copies of the byte array in memory temporary, which may not be ideal. 

 

disclaimer: I havn't tested the above, but guessing it may get towards the right direction.

Edited September 6, 2023 by darnocian