base64 encode/decode TStringDynArray

[borni69 1]

Hi all

We have a TOML template configuration  system  we use to hold config data.

 

When I load a template into  Delphi  I use  TStringDynArray  to hold the data

var

 FLines            : TStringDynArray;

....

     FLines        :=  TFile.ReadAllLines(afilepath,Tencoding.UTF8 );

 

our TOML parser also use TStringDynArray to scan the files and set the different values.

 

after load we would like to put the data into a Memcache, so next time we don't need to load the data from disc.

 

Is there a way to base64encode / decode  TStringDynArray  direct ?

 

Today we change the data to a normal string like below..

 

               TemplateLines:='';
               for I := 0 to length(FLines)-1 do
               begin
                TemplateLines:=TemplateLines+FLines[i]+sLineBreak;
               end;


               b64data := SettingTemplateClass.encodeBase64(TemplateLines);
               SettingTemplateClass.memcacheClass.store( memcached_key , b64data );

So we need either to change the string back from string to  TStringDynArray  or base64 encode/decode  TStringDynArray direct and save it to memcache  ?

 

Any suggestion recommendation.

 

 

 

 

[borni69 1]

We see we can use SplitString to go from string to  TStringDynArray



  b64data := SettingTemplateClass.memcacheClass.lookup(memcached_key);
  TemplateLines :=   TNetEncoding.Base64.Decode( b64data);
  FLines := SplitString(TemplateLines,sLineBreak);

So maybe the  best way is to store the string in memcache  and not the TStringDynArray    

 

all suggestion are welcome...

[David Heffernan 2345]

What's wrong with storing the strings as a dynamic array of string as you already do? Where does base64 come into this?

[borni69 1]

We had some problem earlier with some encoding and memcache so today we store all object in memcache as base64.

 

[David Heffernan 2345]

2 minutes ago, borni69 said:

We had some problem earlier with some encoding and memcache so today we store all object in memcache as base64.

 

You won't have any problems with encoding Delphi string objects. What you do now seems to be to as follows:

 

1. Read a UTF-8 encoded file and convert that to UTF-16 encoded array of string.

2. Create a new string object by concatenating all the strings from the array created in step 1.

3. Base64 encode the string from step 2, which creates another UTF-16 encoded string, but a larger one, because that's the nature of base64.

 

This looks very inefficient to me.

 

I don't know what your memcache code works with, but it looks like it works with strings. And strings don't have encoding problems, they are encoded as UTF-16. It sounds very much like base64 has no purpose here. It's just an extra step with added CPU time and inefficient memory use. And it can't have anything to do with encoding, because base64 just a bijection between binary data and text data. And here we have text data all the way so base64 must be being misused here. Put another way, you use base64 when you have binary data and want to store it in something (file, variable etc.) that expects text. You have text all the way so there should be no base64.

 

It seems to me that you should just load the file into a string and stuff that into your memcache, which is expecting a string anyway. Use TFile.ReadAllText(afilepath,Tencoding.UTF8) to do that and store that string to your memcache. Then when you need to retrieve and parse it, pull out the string and use SplitString.

Even better would be if your TOML parser could work on a stream rather than requiring an array of string. But that's another story.

[borni69 1]

Thanks for following up, I understand you recomandation, and we will try..

So to store it as a string we still need to do 

 

TemplateLines:='';
for I := 0 to length(FLines)-1 do
begin
TemplateLines:=TemplateLines+FLines[i]+sLineBreak;
end;

or is there better ways to do this...

 

we will try to skip the base64

[Lajos Juhász 293]

7 minutes ago, borni69 said:

or is there better ways to do this...

You can read a file as a string:

 

TemplateLines := TFile.ReadAllText(afilepath,Tencoding.UTF8 );

[David Heffernan 2345]

28 minutes ago, borni69 said:

Thanks for following up, I understand you recomandation, and we will try..

So to store it as a string we still need to do 

 

TemplateLines:='';
for I := 0 to length(FLines)-1 do
begin
TemplateLines:=TemplateLines+FLines[i]+sLineBreak;
end;

or is there better ways to do this...

 

we will try to skip the base64

See the penultimate para in my post where I explain exactly how to do this

[borni69 1]

Thanks  🙂

 

 

[borni69 1]

Quote

 

Thanks again

 

Even better would be if your TOML parser could work on a stream rather than requiring an array of string. But that's another story.

 

 

I guess I could change my TOML parser to also use  a stream instead.

 

procedure TtomlMalReader.scanner;
var
 i : Integer;
 aline       : String;
 LocalC      : Char;
begin
  for i := 0 to length(malLines)-1 do
    begin
     aline := malLines[i];
     LineNumber:=i+1;
     for LocalC in aline  do
       begin
         c:=localc;
         validateCharacter;
         if error > ''  then
          begin
           error_line:=i+1;
           break;
          end;

       end;
     if error > ''  then break;
     validateNewLine;
    end;
end;

 

 

As you can see I loop all lines and validate the c : char; character in each line..  

 

So maybe I should use a TStringStream instead  is that what you ment ?   will also try to skip base64

 

and thanks again for your support.

 

B

 

 

[mvanrijnen 123]

18 hours ago, borni69 said:

We had some problem earlier with some encoding and memcache so today we store all object in memcache as base64.

 

Don't know your "memcache" class, but why not use a simple dictionairy ? 

 

[borni69 1]

Still have issue when storing to memcache when string is not base64 encoded

 

this is my code Writeln to memcache

       tcp.IOHandler.InputBuffer.clear;

       command :='set '+key+' 0 0 '+length(Value).ToString+' noreply';
          try
           tcp.Socket.Writeln(command,IndyTextEncoding_UTF8);
           tcp.Socket.Writeln(Value,IndyTextEncoding_UTF8);
          except
             on e: Exception do
            begin
               reportError(e.Message);
               exit;
            end;
          end;
   end;

 

all Norwegian character like æøå  ÆØÅ   become ???  ???   memcache is running on Linux docker.

 

b

 

 

 

 

[David Heffernan 2345]

Only you know what your memcache is doing, but I'd guess that it uses an 8 bit encoding, perhaps AnsiString. You need to understand a problem before trying to solve it.

[borni69 1]

Thanks,

 

Yes you are rigth  it look like Ansistring 8 bit , and it also looks like my encoding send with it is not working for memcache..  😞

IndyTextEncoding_UTF8

tcp.Socket.Writeln(Value,IndyTextEncoding_UTF8);

 

So I guess I  Either I need to convert this delphi unicode UTF-16 string TemplateLines

               TemplateLines:=  TFile.ReadAllText(afilepath,Tencoding.UTF8 );
to Ansistring

or

continue with base64encode

 

Are there an easy way in Delphi to convert a Delphi UTF-16 string to Ansistring ?    Sorry if this is a dumb question...

 

 

 

 

 

 

 

[David Heffernan 2345]

Your problem is with this memcache class. That's what you need to fix. Shouldn't be using AnsiString. 

[Remy Lebeau 1394]

4 hours ago, borni69 said:

Are there an easy way in Delphi to convert a Delphi UTF-16 string to Ansistring ?

Simply assign the (Unicode)String directly to the AnsiString, the RTL will handle the conversion for you.  But like David said, you really shouldn't be using `AnsiString`, at least outside of direct interop with ANSI-based APIs.  Stick with (Unicode)String natively for all of your processing, converting to/from bytes only when the string data needs to leave/enter your code (socket I/O, streaming, etc).

 

[borni69 1]

OK thanks all for your help, I will give it a try...

[borni69 1]

 

 

23 hours ago, mvanrijnen said:

Don't know your "memcache" class, but why not use a simple dictionairy ? 

 

Thanks for the Idea, and  we are considering this.

But we have some consideration.

 

Our Delphi app is an API server ,  ISAPI webbroker module running on Apache linux docker several container instances behind a load balancer.   

We can share a dictionary between the threads  using

  DataSync := TMultiReadExclusiveWriteSynchronizer.Create;

 

We use MPM_event on apache with this setup per production server

<IfDefine TSprod>
    ServerLimit         8
    MaxRequestWorkers   200
    ThreadsPerChild     25
</IfDefine>

 

And for setup above  this will give 8 dictionary per server holding almost the same information, and 25 threads will share same dictionary this will be super fast but consume more memory than memcache.

 

We could consider reducing ServerLimit and increase ThreadsPerChild for this setup to not use so much memory per server this will increase threads using same dict.

 

We will test a little more on this.

 

B

 

 

 

Edited June 11, 2021 by borni69