M. Toussaint 1 Posted September 4, 2023 Hi, From our application we use several procedures and functions to upload files to AWS S3 and also download text-like files from AWS S3. We use the function below to get the contents of text-like files (.SVG, .TXT, .CSV) from S3, but we have no idea how to (download and) process binary files. function S3Get_Object_AsString(const bucket, key: string): string; var pyKey : TPythonDelphiVar; pyBucket : TPythonDelphiVar; pyObj : TPythonDelphiVar; PyEngine : IPyEngineAndGIL; pyError : TPythonDelphiVar; Script : TStringList; cModuleName : AnsiString; begin pyKey := TPythonDelphiVar.Create(nil); pyBucket := TPythonDelphiVar.Create(nil); pyObj := TPythonDelphiVar.Create(nil); pyError := TPythonDelphiVar.Create(nil); script := TStringList.Create; try PyEngine := TPyEngineAndGIL.Create; cModuleName := AnsiString(TGUID.NewGuid.ToString); pyBucket.Engine := PyEngine.getPyEngine; pyBucket.VarName := 'BUCKET'; pyBucket.Module := cModuleName; pyBucket.Initialize; pyBucket.Value := bucket; pyKey.Engine := PyEngine.getPyEngine; pyKey.VarName := 'S3KEY'; pyKey.Module := cModuleName; pyKey.Initialize; pyKey.Value := key; pyObj.Engine := PyEngine.getPyEngine; pyObj.VarName := 'OBJ'; pyObj.Module := cModuleName; pyObj.Initialize; pyError.Engine := PyEngine.getPyEngine; pyError.VarName := 'ERROR'; pyError.Module := cModuleName; pyError.Initialize; pyError.Value := ''; InitBotoClient(Script); Script.Add('import json '); Script.Add('from botocore.exceptions import ClientError '); Script.Add('s3_client = client(''s3'')'); Script.Add('try: ' ); Script.Add(' s3_object = s3_client.get_object(Bucket=BUCKET.VALUE, Key=S3KEY.VALUE)'); Script.Add(' OBJ.VALUE = s3_object[''Body''].read()'); Script.Add('except ClientError as e: '); Script.Add(' print("Get Object failed:" + e.response[''Error''][''Message''])'); Script.Add('else:'); Script.Add(' print("Get Object:" + BUCKET.VALUE + "-" + S3KEY.VALUE)'); PyEngine.getPyEngine.ExecModule := cModuleName; PyEngine.getPyEngine.ExecStrings(Script); if pyError.ValueAsString <> '' then begin //Log(pyError.ValueAsString + 'bucket: ' + bucket + ' key:' + key , 'E'); result := ''; end else begin if VarIsStr(pyObj.Value) then result := pyObj.Value else result := ''; end; finally Script.Free; pyBucket.Finalize; pyKey.Finalize; pyObj.Finalize; pyError.Finalize; pyBucket.Free; pyKey.Free; pyObj.Free; pyError.Free; end; end; So far we found an alternative for downloading a file object from S3, but how to process this in Delphi? Script.Add('import json '); Script.Add('import io '); Script.Add('from botocore.exceptions import ClientError '); Script.Add('s3_client = client(''s3'')'); Script.Add('io_stream = io.BytesIO()'); Script.Add('try: ' ); // Script.Add(' s3_object = s3_client.get_object(Bucket=BUCKET.VALUE, Key=S3KEY.VALUE, )'); // Script.Add(' OBJ.VALUE = s3_object[''Body''].read()'); // Use download_fileobj to gat a file-like object Script.Add(' s3_client.download_fileobj(Bucket=BUCKET.VALUE, Key=S3KEY.VALUE, Fileobj=io_stream)'); Script.Add(' io_stream.seek(0)'); Script.Add(' OBJ.VALUE = io_stream.getvalue()'); Script.Add('except ClientError as e: '); Script.Add(' print("Get Object failed:" + e.response[''Error''][''Message''])'); Script.Add('else:'); Script.Add(' print("Get Object:" + BUCKET.VALUE + "-" + S3KEY.VALUE)'); Share this post Link to post
darnocian 77 Posted September 6, 2023 (edited) I have a few points: 1. it may be easier to use the Appercept AWS SDK. You don't need to bring Python into the picture in order to use it. Understandably, the SDK is currently only available in Delphi Enterprise. 2. in the code above, as you are downloading binary, I'd say it would be easier just write to file from python, and then read it back in from Delphi. I have not checked the marshalling code in the Py4D layer, but I assume that is what you are after. It depends on how you intend to use it, if the file is really big, do you want the whole thing in memory? 3. in python, I think you could also do Quote pbarray = io_stream.getvalue() and in delphi, Quote var barray = PyEngine.GetVarAsVariant('pbarray'); I suspect this might mean there could be 2 copies of the byte array in memory temporary, which may not be ideal. disclaimer: I havn't tested the above, but guessing it may get towards the right direction. Edited September 6, 2023 by darnocian Share this post Link to post