Using dxgettext on Windows 10

[dummzeuch 1515]

The dxGetText installer available from SourceForge has been quite outdated for a while. Via this StackOverflow answer I got this link, where somebody actually went through the trouble to update the tools and create ...

https://blog.dummzeuch.de/2018/12/02/using-dxgettext-on-windows-10/

[Achim Kalwa 61]

Did you had success in contacting Lars Dybdal?

I'm using gnugettext.pas and the dxGetText shell extensions a lot (Extract translation to template, Merge template). Depending on the some config options, several GetText-tools spawn other tools like ggMerge.exe calling msgremove.exe, but using "bash.exe" as environment, which requires cygwin.dll to run on windows. I guess when porting GetText to Windows, the need for bash.exe was ported, too. However, I don't see a reason why bash.exe needs to be called at all. Using cmd.exe (from comspec environment variable) or plain CreateProcess() can give the same results.

 

A very useful option in xgettext.pas to *not* include lines numbers (very helpful when storing po/pot in version control) in the po-file is not available in the configuration dialog.

And why does extracting translations from source create a .po file instead of a .pot file? In my local fork I've addressed some of this issues and more, i.e. ignore component captions if they are the same as the component name like Label1=Label1, Button5=Button5, Action1=Action1, respecting Windows fonts, removed XPMan.pas use Delphi's manifest, etc.

 

Thomas, do you have write access to the dxGetText project on sourceforge? I would like to send you some patches then.

 

Achim

[dummzeuch 1515]

31 minutes ago, Achim Kalwa said:

Did you had success in contacting Lars Dybdal?

Yes, I did get a very short response:

But apart from saying that dybdahl.dk has been closed, he didn't react to any of my follow ups.

 

31 minutes ago, Achim Kalwa said:

Thomas, do you have write access to the dxGetText project on sourceforge? I would like to send you some patches then.

Yes, for now I still have, but Lars has blocked my write access before when he didn't like what I committed. OTOH it doesn't look as if he is interested in the project much any more.

 

But do send me the patches, I'll do what I can.

 

twm

 

 

[Sue King 5]

I have a very difficult to find (for me) issue with a version of gnugettext.pas shown as $LastChangedRevision 220 $ with Thomas Mueller and Olivier Sannier in the contributors' list that does not occur in a version with the same revision, and without those contributors.  I'm not sure where either of file came from, but thought I had them from the Sourceforge repository in 2017.   There are a number of different versions in the repository, and all of them with $DEFINEs of the form dx_xxxx seem to cause the problem.

 

The reason it is difficult to find is that it only happens in programs that use some of the Nexus DB components, and it results in the component going into an endless loop when a stopping as some threads do not terminate.

 

I'm just looking at this again after a long period of ignoring it by using an older version of gnugettext.

 

This is with 10B.

 

The test application works fine on its own.  As soon as I add gnugettext to the uses clause, it fails.  The app does not actually call anything in gnugettext.

 

I assume the only way to narrow it down is to change the working version and add the changes that are in later versions and see which one causes the problem.

 

Which of the many different versions in the repository should I aim for ?

 

Regards

Sue

[dummzeuch 1515]

1 hour ago, Sue King said:

Which of the many different versions in the repository should I aim for ?

If you have a version that works, and one that doesn't, I would take the list of changes to the file and, like in a binary search, try to find the revision were the problem starts to occur.

That should be doable if the problem is reproducible.

[Sue King 5]

My previous suspicions that it is related to load strings seem to be borne out.  I noticed that the Nexus code does use resource strings.

 

I can make the latest version r115 work by commenting out one line :

 

HookLoadResString.Enable;

 

Prior to that, r47 worked as is.  I could get r75 to work by adding the the lines referencing CreatorThread.

r76 introduces GetResString and re-adding CreatorThread references did not help, hence looking for other changes I could make.

 

What is the purpose of using Windows LoadResStringW (for unicode) instead of the Borland version ? What does 'gettext enabled version' mean ?

 

Sue

[dummzeuch 1515]

I can't currently look at the code and I don't remember. I'll try to find out next week when I'm back from vacation.

[pyscripter 694]

5 hours ago, Sue King said:

 

What is the purpose of using Windows LoadResStringW (for unicode) instead of the Borland version ? What does 'gettext enabled version' mean ?

Automatic translation of resource strings.

[Sue King 5]

3 minutes ago, pyscripter said:

Automatic translation of resource strings. 

Do you mean so that if there isn't something for gettext and there is a resource translation, there is only one call ?

[Sue King 5]

29 minutes ago, dummzeuch said:

I can't currently look at the code and I don't remember. I'll try to find out next week when I'm back from vacation. 

Thanks.  It isn't urgent although I would like to get something resolved and move on to the next challenge - dxgettext and FireMonkey.😕

[pyscripter 694]

8 minutes ago, Sue King said:

Do you mean so that if there isn't something for gettext and there is a resource translation, there is only one call ? 

  {$ifdef UNICODE}
  HookLoadResString:=THook.Create (@system.LoadResString, @LoadResStringW);
  {$else}
  HookLoadResString:=THook.Create (@system.LoadResString, @LoadResStringA);
  {$endif}

 

The system functions are replaced with gettext versions that return translated strings.  So wherever you use resource strings translated values will appear.

[Sue King 5]

I don't quite understand where the gettext version is used in this code.

3 minutes ago, pyscripter said:

  {$ifdef UNICODE}
  HookLoadResString:=THook.Create (@system.LoadResString, @LoadResStringW);
  {$else}
  HookLoadResString:=THook.Create (@system.LoadResString, @LoadResStringA);
  {$endif}

 

The system functions are replaced with gettext versions that return translated strings.  So wherever you use resource strings translated values will appear.

I interpreted it to mean that LoadResString uses Windows LoadResStringW (for example) instead of the Borland implementation to get the string, and that string is then passed to gettext.  Is this not correct ?

[pyscripter 694]

LoadResStringW is not a Windows function.  It s a function defined with the gettext unit.   The THook is used to direct calls from System.LoadResString to gettext's LoadResStringW. It also redirects a bunch of other functions related to resource strings.

 

Say for example this is your program:

 

resourcestring

  Test = 'Hello"

...

ShowMessage(Test);

 

Your program will show the translation of Test, if it exists in the translated messages for the current language and if not it will show 'Hello'.

 

 

Edited January 17, 2019 by pyscripter

[Sue King 5]

That makes sense.

 

I thought it was a Windows function because at some stage the IDE took me to WinAPI.Windows, but it doesn't now.

 

Thanks

[dummzeuch 1515]

OK, I'm back home with a computer. What is the current status of the problem @Sue King ? Did you find the cause? Can you give me some more hints where to look?

 

Edit: Yeah, right. I have got a computer, but it doesn't start. 😞

Why do these bloody things always break when you just want to get something done? (Because that's the only time you turn them on, obviously...)

Edited January 23, 2019 by dummzeuch

[Sue King 5]

I haven't done any more on it.  I thought I would leave it for a bit and come back to it.

 

I can use the latest version providing I disable

HookLoadResString.Enable;

 

I'll have another look today.  This is the first time I've needed to look at hooking, so it is new territory for me.

 

The actual code looks OK, but something must be going on that isn't.

[Sue King 5]

A bit more information.   The test app I was using was a sample demo from Nexus that I'd modified to test some units I'd written.  To eliminate my code, I went back to the original demo.  The problem still occurs, but it may have shed more light on what is happening.

 

In gnugettext.pas, I enabled the HookLoadResString and changed  TGnuGettextInstance.LoadResString  to

Result := 'fred';

This resulted in the app closing properly.

 

Result := ResourceStringGettext('fred');

Resulted in the problem.

 

Putting in some outputdebugstrings, it appears that it is going into a loop calling LoadResStringW.  Looking at the Nexus code in the debugger I think at this stage it might be raising a silent exception which is calling LoadResStringW and maybe that is generating an exception somewhere and it is going into a never-ending loop.  More time in debugging may pin it down a bit more.

 

Unfortunately my day was unexpectedly interrupted and I haven't been able to do more than this.  I may not have more time until mid-next week to do much on it.  Things will be a bit unpredicable for the next few days, but if there is anything you can suggest for me to do, I'll do my best to find the time as I'd like to get to the bottom of it.

 

Sue

 

 

 

 

 

[dummzeuch 1515]

I suspect that Nexus is also hooking LoadResString and this might conflict with the hooking gnugettext does. But since I don't use Nexus (and therefore don't have the source code), I can't verify this.

[Sue King 5]

Nexus does not hook that function.  I have checked with the Nexus team.

 

I have found a work around in gnugettext.pas.  In TGnuGettextInstance.ResourceStringGettext add a test for refcount = 0 so that it starts with

 

if (MsgId='') or (ResourceStringDomainListCS=nil) or (ResourceStringDomainListCS.RefCount=0) then begin

  Result := MsgId;

  exit;

end;

 

[dummzeuch 1515]

RefCount should always be 0 if the class is not assigned to an interface variable, which it isn't. So that change basically means:

if true then begin
  Result := MsgId;
  Exit;
end;

So, that's not a solution to the actual problem.

 

Did you try to set a breakpoint in the finalization section and single step through it to find the position where it hangs? (Or maybe see that it doesn't hang there but somewhere else?)

 

I looked at the code and can't find anything wrong with it. It does use various instances of TMultiReadExclusiveWriteSynchronizer which, if I remember correctly, was broken in several RTL versions. So using that class might not have been the best decision. But on the other hand, my memory might fail me and anyway those bugs should have been fixed long ago.

 

Which Delphi version are we talking about? 10.1, as your profile suggests?

[dummzeuch 1515]

As a test whether a bug in TMultiReadExclusiveWriteSynchronizer is to be blamed, I have just replaced it with the TMultiReadSingleWrite class from DWScript. Here is the code:

 

https://download.dummzeuch.de/twm/gnugettext-pas-with-TMultiReadSingleWrite.zip

 

Could you please download and try it?

It's only active if the symbol dx_UseDWSMultiReadSigleWrite is defined (which I have done in the code linked above).

 

If we believe Eric Grange (which I tend to do), it is also much faster, but that was with the implementation from 2013, so might no longer be true.

Also, it is not reentrant. I have not checked whether that might be a problem with dxgettext.

Edited January 27, 2019 by dummzeuch

[Sue King 5]

Thanks for the explanation about why the changed test isn't a fix.

 

From what I can see in the debugger, it is not an error in finalization - the process is not being terminated when the error occurs.

 

The loop involves repeatedly calling ResourceStringGettext.

 

The test app has a class that uses a winsock.  Something is happening when the winsock is closed that causes a loop, possibly an exception getting a string.  The workarounds I've tried mean I can open and close the socket multiple times in the app.  Without it, the app hangs on the first close.

 

This version does not help.

[dummzeuch 1515]

OK, I don't think I can do more without access to the sources. This is now down to guesswork.

Have you tried to stop execution in the debugger when the program hangs on shutdown? There should either be an infinite loop or an infinite wait for an event, mutex or critical section. The call stack should give a hint.

Edited January 27, 2019 by dummzeuch

[Sue King 5]

I've spent some hours in the debugger trying to see what is going on.  Unfortunately the Nexus code is multi-threaded and it makes it hard to follow what is going on.

 

Before I go back to them, I've played around with the code inTGnuGettextInstance.dgettext.  It seems as though calling utf8encode, and then utf8decode can trigger the looping, but it may be that something is happening that I don't understand.

 

That is, changing to code to add

 

var

  lResult: UnicodeString;

  lUTF8String: UTF8String;

begin

  ...

  end else begin

    lUTF8String := utf8encode(szMsgId);

    lResult := UTF8Decode(lUTF8String);

  Result := szMsgId;

...

 

causes the loops.  While running, if I jump over the utf8decode in the debugger (4 times), it then completes.  If utf8decode is not called it works fine.

 

I noticed that while it is looping, the message is [unknown].  When I skip the utf8decode with the debugger, it shows [Operation aborted] the first 2 times through, and then '[unknown]' the next 2 times, and then doesn't need to call it again as it has completed whatever it is doing successfully.

 

If this doesn't give you any clues, then I'll talk to Nexus.  I wanted to do all I could before raising it with them.  At the moment, I've think I reached the end of my knowledge with trying to debug this.  The call stack in the debugger doesn't appear helpful.

 

Thanks for looking at this.

 

 

 

 

[Sue King 5]

I've asked on the Nexus forum about this, posting this call stack from the debugger.

 

gnugettext.TGnuGettextInstance.LoadResString($676444)
gnugettext.LoadResStringW($676444)
nxllUtils.nxRaiseLastOSError(True,0)
nxtwWinsockTransport.TnxWinsockClientConnection.DoRecv((no value),32)
nxtwWinsockTransport.TnxWinsockClientConnection.Read((nxptBasePooledTransport.TnxBasePooledTransport.btMessageReceived,$7E790030))
nxptBasePooledTransport.TnxBaseCallbackThread.InnerExecute
nxllThread.TnxInternalInitThread.DoExecute
nxllThread.TnxThread.Execute
nxptBasePooledTransport.TnxBaseCallbackThread.Execute
:0047a030 ThreadProc + $4C
:00409e7e ThreadWrapper + $2A
:76d1fe09 KERNEL32.BaseThreadInitThunk + 0x19
:77d0662d ntdll.RtlGetAppContainerNamedObjectPath + 0xed
:77d065fd ntdll.RtlGetAppContainerNamedObjectPath + 0xbd

 

This sequence is repeated for each time through the endless loop.  The translated text first time through is '[unknown]'.  If I don't actually

 

The suggestion was that maybe there is something in LoadResString that causes issues when called in the context of a different thread.  The Nexus code I'm trying to use has multiple threads involved so maybe there is a race condition somewhere.