borni69

Thanks, Yes you are rigth it look like Ansistring 8 bit , and it also looks like my encoding send with it is not working for memcache.. 😞 IndyTextEncoding_UTF8 tcp.Socket.Writeln(Value,IndyTextEncoding_UTF8); So I guess I Either I need to convert this delphi unicode UTF-16 string TemplateLines TemplateLines:= TFile.ReadAllText(afilepath,Tencoding.UTF8 ); to Ansistring or continue with base64encode Are there an easy way in Delphi to convert a Delphi UTF-16 string to Ansistring ? Sorry if this is a dumb question...

Still have issue when storing to memcache when string is not base64 encoded this is my code Writeln to memcache tcp.IOHandler.InputBuffer.clear; command :='set '+key+' 0 0 '+length(Value).ToString+' noreply'; try tcp.Socket.Writeln(command,IndyTextEncoding_UTF8); tcp.Socket.Writeln(Value,IndyTextEncoding_UTF8); except on e: Exception do begin reportError(e.Message); exit; end; end; end; all Norwegian character like æøå ÆØÅ become ??? ??? memcache is running on Linux docker. b

I guess I could change my TOML parser to also use a stream instead. procedure TtomlMalReader.scanner; var i : Integer; aline : String; LocalC : Char; begin for i := 0 to length(malLines)-1 do begin aline := malLines[i]; LineNumber:=i+1; for LocalC in aline do begin c:=localc; validateCharacter; if error > '' then begin error_line:=i+1; break; end; end; if error > '' then break; validateNewLine; end; end; As you can see I loop all lines and validate the c : char; character in each line.. So maybe I should use a TStringStream instead is that what you ment ? will also try to skip base64 and thanks again for your support. B

Thanks 🙂

Thanks for following up, I understand you recomandation, and we will try.. So to store it as a string we still need to do TemplateLines:=''; for I := 0 to length(FLines)-1 do begin TemplateLines:=TemplateLines+FLines[i]+sLineBreak; end; or is there better ways to do this... we will try to skip the base64

We had some problem earlier with some encoding and memcache so today we store all object in memcache as base64.

We see we can use SplitString to go from string to TStringDynArray  b64data := SettingTemplateClass.memcacheClass.lookup(memcached_key); TemplateLines := TNetEncoding.Base64.Decode( b64data); FLines := SplitString(TemplateLines,sLineBreak); So maybe the best way is to store the string in memcache and not the TStringDynArray all suggestion are welcome...

Hi , I am a little new on using Delphi app on OSX. But I will give it a try. I have installed x-code I have set up PAserver So now I can build a terminal app, and it works from my mac :-), so far so good . But if I create a visual FMX app called test and only add one text box to it. then I build it . On my mac I find it using my terminal and type ./test the App start but keyboard is not working... If I select edit box and type I can see the text is going into the terminal.. I guess I am doing something wrong here ? any help would be great thanks.

I found it, need to run it from Delphi not just build it.. B

Hi , We have a Delphi webbroker system that receives pdf files from a web client in a base64 format. The server is just receiving them and returning them, it's like a proxy. We do this , because the PDF is created on the client site, but we can not download directly from the client, so after creation we send it to the server and back. This works perfectly for smaller pdf But when the base64 is getting bigger we are getting out of memory. The code we use are procedure TWebModule1.WebModule1pdfproxyAction(Sender: TObject; Request: TWebRequest; Response: TWebResponse; var Handled: Boolean); var acontentType : String; abase64 : String; afileName : String; FileContent : TmemoryStream; begin // Pdf proxy acontentType := Request.ContentFields.Values['contenttype']; abase64 := Request.ContentFields.Values['base64']; afileName := Request.ContentFields.Values['fileName']; if ( acontentType > '') and ( abase64 > '') and ( afileName > '') then begin FileContent := TmemoryStream.Create; FileContent.write(TNetEncoding.Base64.DecodeStringToBytes(abase64), length(TNetEncoding.Base64.DecodeStringToBytes(abase64)) ); FileContent.Position := 0; // Go to the start of the stream Response.SetCustomHeader('Content-Disposition', 'attachment; filename='+ TNetEncoding.URL.Encode(afileName)); Response.ContentStream := FileContent; Response.ContentType := 'application/pdf'; Response.SendResponse; end else Response.Content:=unitGetJsonText.createJsonStatusResult ( Response , TjsonStatus.error ,'error missing content type / base64 /afileName',''); end; Would there be a way to store the incoming data as a file and then return it or other ideas for handle a case like this ? Thanks in advance

Updated, I will try to see what changes this is making to the system.. thanks.. var acontentType : String; abase64 : String; afileName : String; FileContent : TmemoryStream; abuffer : Tbytes; begin // Pdf proxy acontentType := Request.ContentFields.Values['contenttype']; abase64 := Request.ContentFields.Values['base64']; afileName := Request.ContentFields.Values['fileName']; if ( acontentType > '') and ( abase64 > '') and ( afileName > '') then begin FileContent := TmemoryStream.Create; abuffer :=TNetEncoding.Base64.DecodeStringToBytes(abase64); FileContent.write(abuffer, length(abuffer) ); FileContent.Position := 0; // Go to the start of the stream Response.SetCustomHeader('Content-Disposition', 'attachment; filename='+ TNetEncoding.URL.Encode(afileName)); Response.ContentStream := FileContent; Response.ContentType := 'application/pdf'; Response.SendResponse; end else Response.Content:=unitGetJsonText.createJsonStatusResult ( Response , TjsonStatus.error ,'error missing content type / base64 /afileName',''); end;

Hi All We are working on a rest server ISAPI mod created in Delphi 10.4.1 running on Linux. On Apache we use MPM event and have MaxRequestWorkers set to 160, meaning we have upto 160 threads per server We have a database containing our language files . It's around 1300 records supporting 9 languages. Field structure is Id Key Text1 Text2 Text3 .. All our templates have a “Key” where text should be and we replace it when parsing on the server. We find the correct key in our DB using a local index and replace it with the correct text based on the user's text setting. On WebModuleOnCreate we load this table into a TFDMemTable , and it just stays open in that thread. So every thread have a TFDMemTable DB in memory containing all 1300 records We set up TFDMemTable this way today FFDMem_LAN_tekst := TFDMemTable.Create(nil); with FFDMem_LAN_tekst.Indexes.Add do begin Name := 'ixlan_key'; Fields := 'key'; Active := True; end; FFDMem_LAN_tekst.IndexesActive := True; FFDMem_LAN_tekst.IndexName := 'ixlan_key'; FFDMem_LAN_tekst.Close; We load the data using TFDCommand FDCommand_LAN_tekst := TFDCommand.Create(nil); FDCommand_LAN_tekst.Connection:=FParam_con; FDCommand_LAN_tekst.CommandText.Add(' SELECT * FROM lan where deleted=0 '); FDAdapter_LAN_tekst := TFDTableAdapter.Create(nil); FDAdapter_LAN_tekst.SelectCommand:= FDCommand_LAN_tekst; FFDMem_LAN_tekst.Adapter:= FDAdapter_LAN_tekst; FFDMem_LAN_tekst.Close; FFDMem_LAN_tekst.Open; FFDMem_LAN_tekst.FetchAll; This is super speedy and we have been using this method for a long time, but of course 1300 lines is a lot of memory in every thread. We are now considering moving this from TFDMemTable to a TDictionary like var lanDict: TDictionary<string, TjsonObject> Where the first string is key and the second is a TjsonObject containing text1 , text2… The text is not often updated , but it can happen one or two times in a month, so we could restart App server if text is updated. So the question is. Would it be possible to have this TDictionary as a global variable shared by all threads loaded on startup ? If so Where in the webbroker app would you recommend to put the loading code ? Where would you put the variabel ? Thanks in advance 🙂

Hi Again and thanks.. I have never used TMultiReadExclusiveWriteSynchronizer , but it looks straight forward. I created a test app and it seems to work, so I guess it is as simple as code below. except I will use a Dictionary. unit WebModuleUnit1; interface uses System.SysUtils, System.Classes, Web.HTTPApp; type TWebModule1 = class(TWebModule) procedure WebModule1DefaultHandlerAction(Sender: TObject; Request: TWebRequest; Response: TWebResponse; var Handled: Boolean); private { Private declarations } public { Public declarations } end; var WebModuleClass: TComponentClass = TWebModule1; Data: TStringList; DataSync: TMultiReadExclusiveWriteSynchronizer; implementation {%CLASSGROUP 'System.Classes.TPersistent'} {$R *.dfm} procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject; Request: TWebRequest; Response: TWebResponse; var Handled: Boolean); var i : Integer; begin if Request.QueryFields.Values['add'] > '' then begin // Add line DataSync.BeginWrite; try Data.Add ( Request.QueryFields.Values['add']); finally DataSync.EndWrite; end; end; Response.Content := '<html>' + '<head><title>Web Server Application</title></head>' + '<body>'; DataSync.BeginRead; try for I := 0 to data.Count-1 do begin Response.Content := Response.Content+data[i]+'</br>' end; finally DataSync.EndRead; end; Response.Content := Response.Content+' </body>' + '</html>'; end; initialization // test:='Test'; Data := TStringList.Create; DataSync := TMultiReadExclusiveWriteSynchronizer.Create; finalization Data.Clear; Data.Free; DataSync.Free; end. end. thanks

Thank you so much both for your reply. :- ) I will try to set this up and test it. 🙂 Bernt

Hei, how can I remove non-utf8 characters from a utf8 string. Any examples Thanks B

Thanks all for helping. We have now fixed the problem. Sorry If i confused you guys in the start with not be clear about the problem. I did not understand it myself. And started in wrong end. Learned a lot about unicode / utf8 last days. The problem was the Angular client app that did not handle a few control character in Json. It was correct sent from Delphi in Tjsonstring the character was #11 #3 #5 . We have lopped all fields in database and found a few places with this character, all of them did not work client site. After a data review, we have removed them with a script in db . All is working fine now. the DB is like 20 gb and there was a total of 355 fields with wrong character, not so many. all are old registration's , more that 2 years ago. We will make a check for this character when new registration occur in future. The angular team will also check why these character make the client crash. Thanks B

ok got it.. thanks

outtext:=''; str:=memo1.Lines.Text; for ch in str do begin valid:=true; if TCharacter.IsControl(ch) then begin valid:=false; CharacterControl:=ord(ch); if CharacterControl in [9,10,13] then valid:=true; end; if valid then begin outtext:= outtext+ch; end; end; memo2.Lines.Text := outtext; For me this test code seems to do the job.. and now all needed characters are send out.. will test it a little more. thanks..

Hi, some json issue is if the frontend get byte like 0b #11 in Json it crash.. when we build the Json in Delphi before sent to Server we use TJSONString.create( delphi widestring ). Maybe the issue is FrontEnd not handeling the json correct or TJSONString.create in Delphi not handle some control characters.. Bu now I consider to use like recommended in an earlier thread if TCharacter.IsControl(ch) then only accept #9 #10 #13 else block Control characters. all other characters will be sent out.. B

Some more information We store all in MYSQL UTF8 We also use UTF 8 in our webbroker APP The system is running on linux mod files connecting to DB with Firedac, all read and write text is done by Aswidestring. The problem is. Sometimes we get some character from client that can make some json issue on angular when sent back to client, this is what we try to fix. On 600 000 records we have approx 1 problem... We could og course just make a fix for the problem we have today with this character #11, but we try to fix it for other character coming later. The problem appears when client copy paste text from other system like words.. So now we try to figure out what character's to remove before we save text to DB. Not sure if this make it more clear.. B

I tested yesterday I see that #127..#160 have som valid characters and also exist in our DB... I will have another look..

Will do it like this... function StripCharsInSet(s:string; c:CharSet):string; var i,j:Integer; begin SetLength(result,Length(s)); j:=0; for i:=1 to Length(s) do if not (s[i] in c) then begin inc(j); result[j]:=s[i]; end; SetLength(result,j); end; Begin //test memo2.Lines.Text := StripCharsInSet( memo1.Lines.Text ,[#0..#9,#11,#12,#14..#31,#127..#160] ); ... Found the snippet on this page https://stackoverflow.com/questions/5650532/delphi-strip-out-all-non-standard-text-characers-from-string https://www.utf8-chartable.de/unicode-utf8-table.pl?utf8=dec B

ok Thanks will reconsider.. Bernt

Line break will work, since 1) TjsonString.create('') will change them first to : \r\n , and now they are valid. 2) remove unwanted characters Bernt

I agree with you... And after this back and forward discussion in this thread I think I have a solution. I will only send out characters and commands that are handled by TjsonString.create() So I think my code will be something like this PS: there is 0b 0b before E-K in rawText rawText := 'E-K ble æøå Test // 98'; ajson := TjsonObject.Create; try ajson.AddPair('text',TJSONString.Create(rawText)); RawtextOut := ajson.tostring; finally ajson.Free; end; textOut:=''; for ch in RawtextOut do begin if (ch >= #32) then textOut := textOut+ch end; memo1.Lines.Text := textOut; result {"text":"E-K ble æøå Test \/\/ 98"} characters I dont want are removed... also line break tabs will be handled correct .. \r\n Thanks for all your help.. B