borni69

Still have issue when storing to memcache when string is not base64 encoded

this is my code Writeln to memcache

       tcp.IOHandler.InputBuffer.clear;

       command :='set '+key+' 0 0 '+length(Value).ToString+' noreply';
          try
           tcp.Socket.Writeln(command,IndyTextEncoding_UTF8);
           tcp.Socket.Writeln(Value,IndyTextEncoding_UTF8);
          except
             on e: Exception do
            begin
               reportError(e.Message);
               exit;
            end;
          end;
   end;

all Norwegian character like æøå  ÆØÅ   become ???  ???   memcache is running on Linux docker.

b

Quote

 

Thanks again

 

Even better would be if your TOML parser could work on a stream rather than requiring an array of string. But that's another story.

 

I guess I could change my TOML parser to also use  a stream instead.

procedure TtomlMalReader.scanner;
var
 i : Integer;
 aline       : String;
 LocalC      : Char;
begin
  for i := 0 to length(malLines)-1 do
    begin
     aline := malLines[i];
     LineNumber:=i+1;
     for LocalC in aline  do
       begin
         c:=localc;
         validateCharacter;
         if error > ''  then
          begin
           error_line:=i+1;
           break;
          end;

       end;
     if error > ''  then break;
     validateNewLine;
    end;
end;

As you can see I loop all lines and validate the c : char; character in each line..  

So maybe I should use a TStringStream instead  is that what you ment ?   will also try to skip base64

and thanks again for your support.

B

Thanks  🙂

Thanks for following up, I understand you recomandation, and we will try..

So to store it as a string we still need to do 

TemplateLines:='';
for I := 0 to length(FLines)-1 do
begin
TemplateLines:=TemplateLines+FLines[i]+sLineBreak;
end;

or is there better ways to do this...

we will try to skip the base64

We had some problem earlier with some encoding and memcache so today we store all object in memcache as base64.

We see we can use SplitString to go from string to  TStringDynArray



  b64data := SettingTemplateClass.memcacheClass.lookup(memcached_key);
  TemplateLines :=   TNetEncoding.Base64.Decode( b64data);
  FLines := SplitString(TemplateLines,sLineBreak);

So maybe the  best way is to store the string in memcache  and not the TStringDynArray    

all suggestion are welcome...

Hi all

We have a TOML template configuration  system  we use to hold config data.

When I load a template into  Delphi  I use  TStringDynArray  to hold the data

var

 FLines            : TStringDynArray;

....

     FLines        :=  TFile.ReadAllLines(afilepath,Tencoding.UTF8 );

our TOML parser also use TStringDynArray to scan the files and set the different values.

after load we would like to put the data into a Memcache, so next time we don't need to load the data from disc.

Is there a way to base64encode / decode  TStringDynArray  direct ?

Today we change the data to a normal string like below..

               TemplateLines:='';
               for I := 0 to length(FLines)-1 do
               begin
                TemplateLines:=TemplateLines+FLines[i]+sLineBreak;
               end;


               b64data := SettingTemplateClass.encodeBase64(TemplateLines);
               SettingTemplateClass.memcacheClass.store( memcached_key , b64data );

So we need either to change the string back from string to  TStringDynArray  or base64 encode/decode  TStringDynArray direct and save it to memcache  ?

Any suggestion recommendation.

I found it, need to run it from Delphi not just build it..

B

Hi , I am a little new on using Delphi app on OSX. But I will give it a try.

I have installed x-code

I have set up PAserver

So now I can build a terminal app, and it works from my mac  :-), so far so good  .

But if I create a visual  FMX app  called test and only add one text box to it.  then I build it .

On my mac I find it using my terminal  and type  ./test

the App start but keyboard is not working...   If I select edit box and type I can see the text is going into the terminal..

I guess I am doing something wrong here  ? 

any help would be great thanks.

Updated,

I will try to see what changes this is making to the system..

thanks..

var
 acontentType : String;
 abase64      : String;
 afileName    : String;
 FileContent  : TmemoryStream;
 abuffer      : Tbytes;

begin
  // Pdf proxy

   acontentType := Request.ContentFields.Values['contenttype'];
   abase64      := Request.ContentFields.Values['base64'];
   afileName    := Request.ContentFields.Values['fileName'];



 if ( acontentType > '') and ( abase64 > '') and ( afileName > '') then
 begin
      FileContent      := TmemoryStream.Create;
      abuffer :=TNetEncoding.Base64.DecodeStringToBytes(abase64);
      FileContent.write(abuffer, length(abuffer)  );
      FileContent.Position := 0; // Go to the start of the stream

      Response.SetCustomHeader('Content-Disposition', 'attachment; filename='+ TNetEncoding.URL.Encode(afileName));
      Response.ContentStream := FileContent;
      Response.ContentType := 'application/pdf';
      Response.SendResponse;
 end
  else
    Response.Content:=unitGetJsonText.createJsonStatusResult
                          ( Response , TjsonStatus.error ,'error missing content type / base64 /afileName','');
end;

Hi , We  have a Delphi webbroker system that receives pdf files from a web client in a base64 format.

The server is just receiving them and returning them, it's  like a proxy.

We do this , because the PDF is created on the client site, but we can not download directly from the client, so after creation we send it to the server and back.

This works perfectly for smaller pdf

But when the base64 is getting bigger we are getting out of memory.

The code we use are 

procedure TWebModule1.WebModule1pdfproxyAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
var
 acontentType : String;
 abase64      : String;
 afileName    : String;
 FileContent  : TmemoryStream;

begin
  // Pdf proxy
   acontentType := Request.ContentFields.Values['contenttype'];
   abase64      := Request.ContentFields.Values['base64'];
   afileName    := Request.ContentFields.Values['fileName'];

 if ( acontentType > '') and ( abase64 > '') and ( afileName > '') then
 begin
      FileContent      := TmemoryStream.Create;
      FileContent.write(TNetEncoding.Base64.DecodeStringToBytes(abase64),
        length(TNetEncoding.Base64.DecodeStringToBytes(abase64))  );
      FileContent.Position := 0; // Go to the start of the stream

      Response.SetCustomHeader('Content-Disposition', 'attachment; filename='+ TNetEncoding.URL.Encode(afileName));
      Response.ContentStream := FileContent;
      Response.ContentType := 'application/pdf';
      Response.SendResponse;
 end
  else
    Response.Content:=unitGetJsonText.createJsonStatusResult
                          ( Response , TjsonStatus.error ,'error missing content type / base64 /afileName','');
end;

Would there be a way to store the incoming data  as a file and then return it or other ideas for handle a case like this ?

Thanks in advance

Hi Again and thanks..  

I have never used TMultiReadExclusiveWriteSynchronizer , but it looks straight forward.

I created a test app and it seems to work, so I guess it is as simple as code below. except I will use a Dictionary.

unit WebModuleUnit1;

interface

uses
  System.SysUtils, System.Classes, Web.HTTPApp;

type
  TWebModule1 = class(TWebModule)
    procedure WebModule1DefaultHandlerAction(Sender: TObject;
      Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  WebModuleClass: TComponentClass = TWebModule1;
  Data: TStringList;
  DataSync: TMultiReadExclusiveWriteSynchronizer;

implementation

{%CLASSGROUP 'System.Classes.TPersistent'}

{$R *.dfm}

procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
  var
   i : Integer;
begin
 if Request.QueryFields.Values['add'] > ''  then
 begin
   // Add line
    DataSync.BeginWrite;
      try
        Data.Add ( Request.QueryFields.Values['add']);
      finally
        DataSync.EndWrite;
      end;


 end;

  Response.Content :=
    '<html>' +
    '<head><title>Web Server Application</title></head>' +
    '<body>';

     DataSync.BeginRead;
      try
        for I := 0 to data.Count-1 do
          begin
             Response.Content := Response.Content+data[i]+'</br>'
          end;
      finally
        DataSync.EndRead;
      end;



     Response.Content := Response.Content+' </body>' +    '</html>';
end;

initialization
   // test:='Test';
  Data := TStringList.Create;
  DataSync := TMultiReadExclusiveWriteSynchronizer.Create;
finalization
  Data.Clear;
  Data.Free;
  DataSync.Free;
end.

end.

thanks

Thank you so much both for your reply.  :- )

I will try to set this up and test it.  🙂

Bernt

Hi All

We are  working on a rest server  ISAPI mod created in Delphi 10.4.1  running on Linux.

On Apache we use MPM event and have   

MaxRequestWorkers set to  160, meaning we have upto 160 threads per server

We have a database containing our language files . It's around 1300 records supporting 9 languages.

Field structure is 
 Id
 Key
 Text1
 Text2
 Text3
 ..

All our templates  have a “Key” where text should be and  we replace it when parsing on the server. We find the correct key in our DB using a local index and replace it with the correct text based on the user's text setting.

On  WebModuleOnCreate we load this table into a TFDMemTable , and it just stays open in that thread.

So every thread have a  TFDMemTable DB in memory containing all 1300 records

We set up TFDMemTable this way today

FFDMem_LAN_tekst := TFDMemTable.Create(nil);
  with FFDMem_LAN_tekst.Indexes.Add do begin
   Name := 'ixlan_key';
   Fields := 'key';
   Active := True;
  end;
  FFDMem_LAN_tekst.IndexesActive := True;
  FFDMem_LAN_tekst.IndexName := 'ixlan_key';
  FFDMem_LAN_tekst.Close;

We load the data using TFDCommand

  FDCommand_LAN_tekst  := TFDCommand.Create(nil);
  FDCommand_LAN_tekst.Connection:=FParam_con;
  FDCommand_LAN_tekst.CommandText.Add(' SELECT * FROM lan where deleted=0 ');
  FDAdapter_LAN_tekst        := TFDTableAdapter.Create(nil);
  FDAdapter_LAN_tekst.SelectCommand:= FDCommand_LAN_tekst;
  FFDMem_LAN_tekst.Adapter:= FDAdapter_LAN_tekst;
  FFDMem_LAN_tekst.Close;
  FFDMem_LAN_tekst.Open;
  FFDMem_LAN_tekst.FetchAll;

This is super speedy and we have been using this method for a long time, but of course 1300 lines is a lot of memory in every thread.

We are now considering moving this from TFDMemTable  to a TDictionary   like

var
  lanDict: TDictionary<string, TjsonObject>

Where the first string is key  and the second is a TjsonObject containing text1 , text2…

The text is not often updated , but it can happen one or two times in a month, so we could restart App server if text is updated.

So the question is.

Would it be possible to have this TDictionary as a global variable shared by all threads  loaded on startup ? 

If so  

Where in the webbroker app would you recommend to put the  loading code ?

Where would you put the variabel ?

Thanks in advance 🙂

Thanks all for helping.

We have now fixed the problem.

Sorry If i confused you guys in the start with not be clear about the problem. I did not understand it myself. And started in wrong end.

Learned a lot about unicode / utf8 last days.

The problem was the Angular client app that did not handle a few  control character in Json. It was  correct sent from Delphi in Tjsonstring  the character was  #11  #3   #5   .

We have lopped all fields in  database  and found a few  places with  this character, all of them did not work client site. 

After a data review, we have removed them with a script in db .

All is working fine now.

the DB is like 20 gb and there was a total of 355 fields with wrong character, not so many.

all are  old registration's , more that 2 years ago.

We will make a check for this character when new registration occur in future.  

The angular team will also check why these character make the client crash.

Thanks

B

ok got it..

thanks

outtext:='';
 str:=memo1.Lines.Text;
 for ch  in  str  do
   begin
     valid:=true;
       if TCharacter.IsControl(ch) then
        begin
          valid:=false;
          CharacterControl:=ord(ch);
          if CharacterControl in [9,10,13] then valid:=true;
        end;
     if valid then
     begin
      outtext:= outtext+ch;
     end;
   end;

   memo2.Lines.Text := outtext;

For me this test code seems to do the job..  and now all needed characters are send out..

will test it a little more.

thanks..

Hi,

some json issue  is

if the frontend get byte like 0b  #11  in Json  it crash..

when we build the Json in Delphi  before sent to Server  we use TJSONString.create( delphi widestring ).

Maybe the issue is FrontEnd not handeling the json correct  or TJSONString.create in Delphi not handle some control  characters..

Bu now I consider to use  like recommended in an earlier thread   

if TCharacter.IsControl(ch) then only  accept #9   #10  #13 else block  Control characters.

 all other characters will be sent out..

B

Some more information

We store all in  MYSQL   UTF8

We also use UTF 8   in our webbroker APP 

The system is  running on  linux mod files connecting to DB with Firedac, all read  and write  text is done by Aswidestring.

The problem is.

Sometimes we get some character from client that can make some json issue on angular when sent back to client, this is what we try to fix.

On 600 000 records  we have  approx 1  problem...

We could og course just make a fix for the problem we have today with this character #11, but we try to fix it for other character coming later. 

The problem appears when client copy paste text from other system like words..

So now we try to figure out what character's to remove before we save text to DB.

Not sure if this make it more clear..

B

I tested yesterday  I see that   #127..#160  have som valid characters and also exist in our DB...

I will have another look..

Will do it like this...

function StripCharsInSet(s:string; c:CharSet):string;
  var i,j:Integer;
  begin
     SetLength(result,Length(s));
     j:=0;
     for i:=1 to Length(s) do
       if not (s[i] in c) then
        begin
         inc(j);
         result[j]:=s[i];
        end;
     SetLength(result,j);
  end;





Begin
 //test 
 memo2.Lines.Text := StripCharsInSet( memo1.Lines.Text ,[#0..#9,#11,#12,#14..#31,#127..#160] );

...

Found the  snippet on this page

https://stackoverflow.com/questions/5650532/delphi-strip-out-all-non-standard-text-characers-from-string

https://www.utf8-chartable.de/unicode-utf8-table.pl?utf8=dec

B

ok Thanks will reconsider..

Bernt

Line break will work, since

1)  TjsonString.create('') will change them first to  :   \r\n   , and now they are valid.

2) remove unwanted characters

Bernt

I agree with you...

And after this back and forward discussion in this thread I think I have a solution.

I will only send out characters  and commands that are handled by TjsonString.create() 

So I think my code will be something like this  

PS:  there is  0b 0b  before E-K in rawText

 rawText := 'E-K ble æøå  Test // 98';
  ajson := TjsonObject.Create;
  try
   ajson.AddPair('text',TJSONString.Create(rawText));
   RawtextOut := ajson.tostring;
  finally
    ajson.Free;
  end;

  textOut:='';
  for ch in RawtextOut  do
  begin
   if (ch >= #32)  then
   textOut := textOut+ch

  end;

  memo1.Lines.Text := textOut;

result

{"text":"E-K ble æøå  Test \/\/ 98"}  characters I dont want are removed...

also line break tabs will be handled correct  ..    \r\n

Thanks for all your help..

B

Thanks looks like  

TCharacter.IsValid(s[i]))

is not suported in 10.4