Control TDataSet strings encoding (auto encode to utf-8)

[Max Terentiev 0]

Hi,

 

I moving my project from D2007 to Rio. My project uses ElevateSoftware DBISAM database engine. I was very happy with it but it's not support unicode and I got problems with saving string data from new unicode-enabled delphi controls to TDBISAMTable: i got ???? instead of unicode symbols. I can handle this problem by manually encode/decode string fields values to/from UTF8. But I need to do it in several thousands places in my code :((((

 

Maybe somebody already found elegant solution for this problem ? Maybe overriding some methods in TDataSet ?

 

I found TStringField.Transliterate and TDataSet.Translate it's almost that I need but function

 

function TDataSet.Translate(Src, Dest: PAnsiChar; ToOem: Boolean): Integer;

 

have PAnsiChar params and I have data loss (???? instead of unicode) here, so I can't use it for background UTF8 conversions. But I need something similar !

 

Thanks for help !

Edited June 1, 2020 by Max Terentiev

[rcaspers 0]

maybe you can override a TStringField and tell your program to use that as the default.

i, on the other hand am very curious for the solution you seem to have found

 

René

 

  TUnicodeStringField = class(TStringField)
  protected
    function GetAsString: string; override;
    procedure SetAsString(const Value: string); override;
  end;

implementation

{ TUnicodeStringField }

function TUnicodeStringField.GetAsString: string;
begin
  // your solution here
end;

procedure TUnicodeStringField.SetAsString(const Value: string);
begin
  inherited;
  // your solution here
end;

    DefaultFieldClasses[ftString] := TUnicodeStringField;

 

Edited July 9, 2020 by rcaspers

[aehimself 396]

@rcaspers I had no idea you can "override" default TField types. Would have made my life a lot easier a couple of years ago!

 

@Max Terentiev You can easily do the conversion by TEncoding.Convert, GetBytes and GetString. However, the easiest solution would be to upgrade the component you are using to one, which actually supports Unicode. If they are all TDataSet descendants, you should face no issues.

If upgrading is not an option, I'd create a TFieldHelper with an .AsUnicodeString property.

Getter would read the fields value as a byte array, and return it as a String with TEnoding.Unicode.GetString(fieldbytearray);

Setter would accept a string, convert it to a byte array with TEncoding.Unicode.GetBytes and strore this instead of the string.

[Max Terentiev 0]

I solve problem by overriding setting/getting fields content like this:

 

TUTF8StringField = class(TStringField)
private
  FAutoUtf8Conversion:     Boolean;
protected
  function                 GetAsString: string; override;
  function                 GetAsVariant: Variant; override;
  procedure                SetAsString(const Value: string); override;
public
  constructor              Create(AOwner: TComponent); override;
published
  property                 AutoUtf8Conversion: Boolean read FAutoUtf8Conversion write FAutoUtf8Conversion;
end;
//=============================================================

 

constructor TUTF8StringField.Create(AOwner: TComponent);
begin
  FAutoUtf8Conversion:=false;
  inherited Create(AOwner);
end;
//------------------------------------------------------------------------------

function TUTF8StringField.GetAsString: string;
begin
  if FAutoUtf8Conversion then
    Result:=UTF8ToString(GetAsAnsiString)
  else
    Result:=inherited GetAsString;
end;
//------------------------------------------------------------------------------

function TUTF8StringField.GetAsVariant: Variant;
begin
  if FAutoUtf8Conversion then
    Result:=UTF8ToString(AnsiString(inherited GetAsVariant))
  else
    Result:=inherited GetAsVariant;
end;
//------------------------------------------------------------------------------

procedure TUTF8StringField.SetAsString(const Value: string);
begin
  if FAutoUtf8Conversion then
    inherited SetAsAnsiString(UTF8Encode(Value))
  else
    Inherited SetAsString(Value);
end;
//------------------------------------------------------------------------------