Webbroker and global variable question shared dictionary between threads

[borni69 1]

Hi All

 

We are  working on a rest server  ISAPI mod created in Delphi 10.4.1  running on Linux.

 

On Apache we use MPM event and have   

MaxRequestWorkers set to  160, meaning we have upto 160 threads per server

 

We have a database containing our language files . It's around 1300 records supporting 9 languages.

Field structure is 
 Id
 Key
 Text1
 Text2
 Text3
 ..

 

All our templates  have a “Key” where text should be and  we replace it when parsing on the server. We find the correct key in our DB using a local index and replace it with the correct text based on the user's text setting.

 

On  WebModuleOnCreate we load this table into a TFDMemTable , and it just stays open in that thread.

 

So every thread have a  TFDMemTable DB in memory containing all 1300 records

 

We set up TFDMemTable this way today

FFDMem_LAN_tekst := TFDMemTable.Create(nil);
  with FFDMem_LAN_tekst.Indexes.Add do begin
   Name := 'ixlan_key';
   Fields := 'key';
   Active := True;
  end;
  FFDMem_LAN_tekst.IndexesActive := True;
  FFDMem_LAN_tekst.IndexName := 'ixlan_key';
  FFDMem_LAN_tekst.Close;

 

We load the data using TFDCommand

 

  FDCommand_LAN_tekst  := TFDCommand.Create(nil);
  FDCommand_LAN_tekst.Connection:=FParam_con;
  FDCommand_LAN_tekst.CommandText.Add(' SELECT * FROM lan where deleted=0 ');
  FDAdapter_LAN_tekst        := TFDTableAdapter.Create(nil);
  FDAdapter_LAN_tekst.SelectCommand:= FDCommand_LAN_tekst;
  FFDMem_LAN_tekst.Adapter:= FDAdapter_LAN_tekst;
  FFDMem_LAN_tekst.Close;
  FFDMem_LAN_tekst.Open;
  FFDMem_LAN_tekst.FetchAll;

 

This is super speedy and we have been using this method for a long time, but of course 1300 lines is a lot of memory in every thread.

 

We are now considering moving this from TFDMemTable  to a TDictionary   like

var
  lanDict: TDictionary<string, TjsonObject>

 

Where the first string is key  and the second is a TjsonObject containing text1 , text2…

 

The text is not often updated , but it can happen one or two times in a month, so we could restart App server if text is updated.

 

So the question is.

 

Would it be possible to have this TDictionary as a global variable shared by all threads  loaded on startup ? 

 

If so  

Where in the webbroker app would you recommend to put the  loading code ?

Where would you put the variabel ?

 

 

Thanks in advance 🙂

Edited February 21, 2021 by borni69

[Guest]

I have something similar (but not WebBroker).

My workers have to wait for the initialization of the "data packet" (dictionary) using a flag, Initialized.

I protect it (the flag too of course) with a critical section.

I can update it "anytime" (Write CS) and read a lot more (Read CS).

I can even update it on a DB "event".

Packed into something i can even manage a dictionary of dictionaries.

 

Getting all the Enter's and Leave's in place was a bit tedious back when, though.

 

HTH

[Dalija Prasnikar 1341]

16 hours ago, borni69 said:

Would it be possible to have this TDictionary as a global variable shared by all threads  loaded on startup ? 

If the dictionary is populated before threads start using it, and you are not writing anything afterwards, then you can safely share that dictionary between threads for reading without any additional protection. If you will be writing to it while threads are running, then you need to protect all reads and all writes with some locking mechanism.

[Dalija Prasnikar 1341]

12 minutes ago, Dany Marmur said:

I can update it "anytime" (Write CS) and read a lot more (Read CS).

What do you mean by Write CS and Read CS?

 

The way I read it, it sounds like you have two critical sections protecting same data. You should have only one.

[borni69 1]

Thank you so much both for your reply.  :- )

 

I will try to set this up and test it.  🙂

 

Bernt

[Anders Melander 1638]

22 minutes ago, Dalija Prasnikar said:

you need to protect all reads and all writes with some locking mechanism

For example a TMultiReadExclusiveWriteSynchronizer.

[Guest]

@Dalija Prasnikar i meant one TMultiReadExclusiveWriteSyncronizer. I was on my mobile device. Sry.

[borni69 1]

Hi Again and thanks..  

 

I have never used TMultiReadExclusiveWriteSynchronizer , but it looks straight forward.

 

I created a test app and it seems to work, so I guess it is as simple as code below. except I will use a Dictionary.

 

 

unit WebModuleUnit1;

interface

uses
  System.SysUtils, System.Classes, Web.HTTPApp;

type
  TWebModule1 = class(TWebModule)
    procedure WebModule1DefaultHandlerAction(Sender: TObject;
      Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  WebModuleClass: TComponentClass = TWebModule1;
  Data: TStringList;
  DataSync: TMultiReadExclusiveWriteSynchronizer;

implementation

{%CLASSGROUP 'System.Classes.TPersistent'}

{$R *.dfm}

procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
  var
   i : Integer;
begin
 if Request.QueryFields.Values['add'] > ''  then
 begin
   // Add line
    DataSync.BeginWrite;
      try
        Data.Add ( Request.QueryFields.Values['add']);
      finally
        DataSync.EndWrite;
      end;


 end;

  Response.Content :=
    '<html>' +
    '<head><title>Web Server Application</title></head>' +
    '<body>';

     DataSync.BeginRead;
      try
        for I := 0 to data.Count-1 do
          begin
             Response.Content := Response.Content+data[i]+'</br>'
          end;
      finally
        DataSync.EndRead;
      end;



     Response.Content := Response.Content+' </body>' +    '</html>';
end;

initialization
   // test:='Test';
  Data := TStringList.Create;
  DataSync := TMultiReadExclusiveWriteSynchronizer.Create;
finalization
  Data.Clear;
  Data.Free;
  DataSync.Free;
end.

end.

 

 

thanks