TIdHttp - incorrect UTF-8

[polasss 0]

I save a JSON file on Dropbox, after reading it I don't get it in the correct UTF-8 state as it left, please advise.
I am sending Czech characters

{"auth":{"text1":"cz_čřšť","text2":"ýáíé","int":10}}

I will get 

{"auth":{"text1":"cz_èø","text2":"ýáíé","int":10}}

POST code

procedure TForm2.btn_Upload_JsonClick(Sender: TObject);
const
  API_URL = 'https://content.dropboxapi.com/2/files/upload';
var
  IdHTTP: TIdHTTP;
  Res : string;
  Ssl: TIdSSLIOHandlerSocketOpenSSL;
  Json : string;
  Req_Json:TStringStream;
begin
  Json := '{"auth":{"text1":"cz_čřšť","text2":"ýáíé","int":10}}';
  Req_Json:=TstringStream.Create(Json);
  Req_Json.Position:=0;

  IdHTTP := TIdHTTP.Create(nil);
  try
    Memo1.Lines.Clear;
    IdHTTP.HandleRedirects := True;
    ssl := TIdSSLIOHandlerSocketOpenSSL.Create();
    ssl.SSLOptions.Method := sslvTLSv1_2;
    ssl.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
    ssl.SSLOptions.Mode := TidSSLMode.sslmUnassigned;
    ssl.SSLOptions.VerifyMode := [];
    ssl.SSLOptions.VerifyDepth := 0;
    ssl.host := '';

    IdHTTP.IOHandler := ssl;
    IdHTTP.Request.CustomHeaders.Values['Authorization'] := 'Bearer ' + myAccessToken;
    IdHTTP.Request.CustomHeaders.Values['Dropbox-API-Arg'] :=
           '{ "autorename": false, "mode": "add", "mute": false, "path": "/CUZK/test3.json", "strict_conflict": false }';
    IdHTTP.Request.CustomHeaders.Values['Content-Type']  := 'application/octet-stream';
    IdHTTP.Request.CharSet:='utf-8';

    Memo1.Lines.Add('** IdHTTP.Request.CustomHeaders.Text **');
    Memo1.Lines.Add(IdHTTP.Request.CustomHeaders.Text);
    Memo1.Lines.Add('** ssl.OnStatusInfoEx **');
    ssl.OnStatusInfoEx := IdSSLIOHandlerSocketOpenSSL1StatusInfoEx;

    (**)

      Res := IdHTTP.Post(API_URL, Req_Json);
      memo1.Lines.Add('------------------------------------');
      Memo1.Lines.Add(Res);

  finally
      IdHTTP.Free;
      Req_Json.Free;
  end;

GET CODE

procedure TForm2.btn_Download_File_Click(Sender: TObject);
const
  API_URL = 'https://content.dropboxapi.com/2/files/download';
var
  Source: TMemoryStream;
  IdHTTP: TIdHTTP;
  Res : string;
  Ssl: TIdSSLIOHandlerSocketOpenSSL;
begin

  IdHTTP := TIdHTTP.Create(nil);
  try
    IdHTTP.HandleRedirects := True;
    ssl := TIdSSLIOHandlerSocketOpenSSL.Create();
    ssl.SSLOptions.Method := sslvTLSv1_2;
    Ssl.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
    ssl.SSLOptions.Mode := TidSSLMode.sslmUnassigned;
    ssl.SSLOptions.VerifyMode := [];
    ssl.SSLOptions.VerifyDepth := 0;
    ssl.host := '';

    Source := TMemoryStream.Create;

    IdHTTP.IOHandler := ssl;
    IdHTTP.Request.CustomHeaders.Values['Authorization'] := 'Bearer ' + myAccessToken;
    IdHTTP.Request.CustomHeaders.Values['Dropbox-API-Arg'] := '{ "path": "/CUZK/test3.json" }';
    IdHTTP.Request.CharSet := 'utf-8';

      Res := ansitoutf8(IdHTTP.Get(API_URL));
      memo1.Lines.Add('------------------------------------');
      Memo1.Lines.Add(Res);

  finally
      IdHTTP.Free;
      Source.Free;
  end;

end;

 

Edited November 19, 2022 by polasss

[aehimself 388]

If you create a StringStream from an input string, it uses TEncoding.Default. What if you override this by

 TStringStream.Create(Json, TEncoding.UTF8, False);

?

[Remy Lebeau 1263]

On 11/19/2022 at 1:35 AM, polasss said:

POST code

You did not say which version of Delphi you are using.  If it is Delphi 2009+, then as @aehimself mentioned, you need to specify the correct TEncoding for the TStringStream to use, eg:

Req_Json := TStringStream.Create(Json, TEncoding.UTF8);

If you are using an older version, then you are responsible for populating the TStringStream (better to use TMemoryStream in that case) with proper UTF-8 data yourself, eg by calling  UTF8Encode() (since pre-2009 versions of Delphi did not have a native UTF-8 string type yet).

 

That being said, there are some other issues with your code...

 

You are leaking the TIdSSLIOHandlerSocketOpenSSL object, as you are not assigning an Owner to it, and not Free'ing it.  You should assign the TIdHTTP object as its Owner, eg:

ssl := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP);

Also, on these statements:

ssl.SSLOptions.Method := sslvTLSv1_2;
ssl.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];

You DO NOT need to set the Method property, the SSLVersions setter will handle that for you (not to mention, you are setting it to the wrong value, anyway).

 

Also, DO NOT use the TIdHTTP.Request.CustomHeaders property to specify a 'Content-Type' header.  Use the TIdHTTP.Request.ContentType property instead:

IdHTTP.Request.ContentType := 'application/octet-stream';

But why are you using 'application/octet-stream' instead of 'application/json'?

On 11/19/2022 at 1:35 AM, polasss said:

GET code

Same notes about the SSL object here, too.

 

Also, why are you creating a TMemoryStream that you are not using for anything?

 

Also, again, which version of Delphi are you using?  If you are using a pre-2009 version, the string returned by TIdHTTP.Get() will be the raw bytes of the downloaded data.  But in Delphi 2009+, the returned string will be a UTF-16 UnicodeString.  DO NOT use AnsiToUtf8() to convert that to UTF-8.  Simply assign it to a UTF8String instead, or at least use UTF8Encode().  However, you are assigning the converted result to another native string, which will undo the conversion in D2009+.  So, you are best off just displaying the original string as-is, whatever format it is in.

 

That being said, if the downloaded content is UTF-8 to begin with, you are better off downloading it into your TMemoryStream instead of to a string, and then you can copy the content of the TMemoryStream into a UTF8String using SetString(), and display it as-is.  Or save the TMemoryStream to a file as-is. Etc.