pyscripter

Pcre-Jit-Delphi Delphi wraps the PCRE library into the System.RegularExpressions unit. This unit provides a high-level, easy-to-use interface to PCRE. A limitation however is that it does not provide access to the Just-In-Time (JIT) compiler of PCRE. PCRE JIT can improve the performance of regular expression matching dramatically and is particularly useful, when you apply the same regular expression repetitively. The purpose of the Pcre-Jit-Delphi project is to patch the Delphi RTL in order to provide access to JIT. Please visit the project page at Github to see how you can use PCRE JIT in your projects. Benchmark The console project Benchmark.dpr in Demos folder compares the performance of the built-in Delphi regular expressions library, with the those of using Study without JIT and Study with JIT on a commonly used regular expression benchmark. Here are the results I got from the 64 bit version. Time | Match count ============================================================================== Delphi's own TRegEx: /Twain/ : 7.00 ms | 811 /(?i)Twain/ : 41.00 ms | 965 /[a-z]shing/ : 384.00 ms | 1540 /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ : 461.00 ms | 262 /\b\w+nn\b/ : 588.00 ms | 262 /[a-q][^u-z]{13}x/ : 539.00 ms | 4094 /Tom|Sawyer|Huckleberry|Finn/ : 757.00 ms | 2598 /(?i)Tom|Sawyer|Huckleberry|Finn/ : 861.00 ms | 4152 /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ : 2615.00 ms | 2598 /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ : 2766.00 ms | 1976 /Tom.{10,25}river|river.{10,25}Tom/ : 455.00 ms | 2 /[a-zA-Z]+ing/ : 807.00 ms | 78423 /\s[a-zA-Z]{0,12}ing\s/ : 560.00 ms | 49659 /([A-Za-z]awyer|[A-Za-z]inn)\s/ : 789.00 ms | 209 /["'][^"']{0,30}[?!\.]["']/ : 321.00 ms | 8885 Total Time: 11963.00 ms ============================================================================== Delphi's own TRegEx with Study: /Twain/ : 6.00 ms | 811 /(?i)Twain/ : 41.00 ms | 965 /[a-z]shing/ : 316.00 ms | 1540 /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ : 21.00 ms | 262 /\b\w+nn\b/ : 581.00 ms | 262 /[a-q][^u-z]{13}x/ : 413.00 ms | 4094 /Tom|Sawyer|Huckleberry|Finn/ : 28.00 ms | 2598 /(?i)Tom|Sawyer|Huckleberry|Finn/ : 217.00 ms | 4152 /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ : 2632.00 ms | 2598 /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ : 2785.00 ms | 1976 /Tom.{10,25}river|river.{10,25}Tom/ : 50.00 ms | 2 /[a-zA-Z]+ing/ : 759.00 ms | 78423 /\s[a-zA-Z]{0,12}ing\s/ : 563.00 ms | 49659 /([A-Za-z]awyer|[A-Za-z]inn)\s/ : 699.00 ms | 209 /["'][^"']{0,30}[?!\.]["']/ : 52.00 ms | 8885 Total Time: 9179.00 ms ============================================================================== Delphi's own TRegEx with JIT: /Twain/ : 11.00 ms | 811 /(?i)Twain/ : 14.00 ms | 965 /[a-z]shing/ : 12.00 ms | 1540 /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ : 3.00 ms | 262 /\b\w+nn\b/ : 126.00 ms | 262 /[a-q][^u-z]{13}x/ : 154.00 ms | 4094 /Tom|Sawyer|Huckleberry|Finn/ : 22.00 ms | 2598 /(?i)Tom|Sawyer|Huckleberry|Finn/ : 61.00 ms | 4152 /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ : 277.00 ms | 2598 /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ : 346.00 ms | 1976 /Tom.{10,25}river|river.{10,25}Tom/ : 12.00 ms | 2 /[a-zA-Z]+ing/ : 84.00 ms | 78423 /\s[a-zA-Z]{0,12}ing\s/ : 156.00 ms | 49659 /([A-Za-z]awyer|[A-Za-z]inn)\s/ : 35.00 ms | 209 /["'][^"']{0,30}[?!\.]["']/ : 18.00 ms | 8885 Total Time: 1350.00 ms As you can see the increase in performance is impressive. Credits Many thanks to @Mahdi Safsafi who provided the assembly code for __chkstk and other Delphi developers (@Kas Ob., @David Heffernan, @Remy Lebeau, @Fr0sT.Brutal) that have provided useful advice.

FLRE is the fastest of all! One single pascal file which is good The author is a kind of genius But not as extensively tested as PCRE poorly documented not very well supported the performance gain compared to PCRE+JIT is not that big System.RegularExpressions has a nicer interface All-in-all I would stick to System.RegularExpressions at least if you add JIT.

True. And for the last 3-4 years there was no improvement in Windows svg support. However converting text to path is a reasonable workaround. In any case it is not common to have text in toolbar icons etc.

I am not able to login for some reason.

I would use the TRTLCriticalSection which is a record defined in Windows.pas. TRTLCriticalSection = record DebugInfo: PRTLCriticalSectionDebug; LockCount: Longint; RecursionCount: Longint; OwningThread: THandle; LockSemaphore: THandle; Reserved: ULONG_PTR; end; A record helper is provided in SynchObjs.pas TCriticalSectionHelper = record helper for TRTLCriticalSection procedure Initialize; inline; procedure Destroy; inline; procedure Free; inline; procedure Enter; inline; procedure Leave; inline; function TryEnter: Boolean; inline; end; According to Eric It is much less prone to this issue (I don't have the reference though) and does not incur the overhead of object creation/destruction. The SpinLock is another option. So is the Windows Slim reader writer (SRW) lock. And of course you can use TMonitor which nowadays is comparable in performance to the CriticalSection. (SpinLock and SRW locks are faster). For anyone interested here is a record wrapper to SRW locks. (* Multiple Read Exclusive Write lock based on Windows slim reader/writer (SRW) Locks. Can be also used instead of a critical session. Limitations: non-reentrant, not "fair" *) TSlimMREWSync = record private Lock: TRTLSRWLock; public procedure Create; procedure BeginRead; procedure EndRead; procedure BeginWrite; procedure EndWrite; function TryRead: Boolean; function TryWrite: Boolean; end; { TSMREWSync } procedure TSlimMREWSync.BeginRead; begin AcquireSRWLockShared(Lock); end; procedure TSlimMREWSync.BeginWrite; begin AcquireSRWLockExclusive(Lock); end; procedure TSlimMREWSync.Create; begin InitializeSRWLock(Lock); end; procedure TSlimMREWSync.EndRead; begin ReleaseSRWLockShared(Lock); end; procedure TSlimMREWSync.EndWrite; begin ReleaseSRWLockExclusive(Lock); end; function TSlimMREWSync.TryRead: Boolean; begin Result := TryAcquireSRWLockShared(Lock); end; function TSlimMREWSync.TryWrite: Boolean; begin Result := TryAcquireSRWLockExclusive(Lock); end;

Worked earlier on but now is down again.

I had the same problem. I though I had installed the patch through Getit but it was not actually installed. After reading this thread I installed by the command prompt. It appears that just clicking on the bat file via the File Explorer did not work despite getting the elevation prompt.

lol. It reminded me of the Ecumenical Liberation Army from the Network film. Great film by the way.

You can apply it manually to your own version (it only takes a couple of minutes, because the changes are few) and you only need to do it once). You can then submit a pull request with the patch in your Delphi version.

Here it is not an issue of compiler/interpreter because PERL is using a C library anyway. Perl uses its own regular expression engine not PCRE. There are many other options, Google RE, Intel's Hyperscan to mention a couple. PCRE+JIT is one of the fastest around and most widely used. I should not forget to mention FLRE written in Pascal, which is probably the fastest of all! Sometime ago I had posted in Google+ a benchmark comparing many of the above, But I not sure how to access Google+ material. Also note that the number of matches differs between Delphi and Perl. This is not surprising, because there are subtle differences in how these engines work.

There is an RSP https://quality.embarcadero.com/browse/RSP-21733 that asks for PCRE JIT support in the Delphi RTL. If you want the extra PCRE performance please vote for this. Off-topic. What does RSP stand for?

Not possible I am afraid and not easier. You just need to add the patched System.RegularExpressionsAPI to your project uses clause.

Still unable to login. Who do I need to contact?

I am trying to link an .obj file generated using Visual Studio from C file. The C file makes Windows calls such as VirtualFree; When I try to compile the Delphi source code with {$L} directive in 32 bits Delphi complains about [dcc32 Error] ExtPCRE.pas(889): E2065 Unsatisfied forward or external declaration: '__imp__VirtualFree@12' Is there a way to resolve this without changing the C code? In 64 bits the name comes undecorated as __imp_VirtualFree and I am resolving this as: type TVirtualFree = function(lpAddress: Pointer; dwSize: SIZE_T; dwFreeType: DWORD): BOOL; stdcall; const __imp_VirtualFree: TVirtualFree = WinApi.Windows.VirtualFree; Is this correct? And a related question fpc provides for the following syntax: Var MyVar : MyType; external name ’varname’; The effect of this declaration is twofold: No space is allocated for this variable. The name of the variable used in the assembler code is varname. This is a case sensitive name, so you must be careful. Can you achieve the same somehow in Delphi?

Here it is https://github.com/pyscripter/Pcre-Jit-Delphi. Enjoy

@Mahdi SafsafiThanks!! Now it works in 32-bits as well. 32-bit benchmark Time | Match count ============================================================================== Delphi's own TRegEx: /Twain/ : 6.00 ms | 811 /(?i)Twain/ : 41.00 ms | 965 /[a-z]shing/ : 448.00 ms | 1540 /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ : 528.00 ms | 262 /\b\w+nn\b/ : 634.00 ms | 262 /[a-q][^u-z]{13}x/ : 543.00 ms | 4094 /Tom|Sawyer|Huckleberry|Finn/ : 896.00 ms | 2598 /(?i)Tom|Sawyer|Huckleberry|Finn/ : 1063.00 ms | 4152 /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ : 2980.00 ms | 2598 /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ : 3088.00 ms | 1976 /Tom.{10,25}river|river.{10,25}Tom/ : 528.00 ms | 2 /[a-zA-Z]+ing/ : 918.00 ms | 78423 /\s[a-zA-Z]{0,12}ing\s/ : 548.00 ms | 49659 /([A-Za-z]awyer|[A-Za-z]inn)\s/ : 775.00 ms | 209 /["'][^"']{0,30}[?!\.]["']/ : 312.00 ms | 8885 Total Time: 13321.00 ms ============================================================================== Delphi's own TRegEx with Study: /Twain/ : 6.00 ms | 811 /(?i)Twain/ : 40.00 ms | 965 /[a-z]shing/ : 318.00 ms | 1540 /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ : 24.00 ms | 262 /\b\w+nn\b/ : 619.00 ms | 262 /[a-q][^u-z]{13}x/ : 450.00 ms | 4094 /Tom|Sawyer|Huckleberry|Finn/ : 31.00 ms | 2598 /(?i)Tom|Sawyer|Huckleberry|Finn/ : 256.00 ms | 4152 /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ : 2875.00 ms | 2598 /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ : 2905.00 ms | 1976 /Tom.{10,25}river|river.{10,25}Tom/ : 63.00 ms | 2 /[a-zA-Z]+ing/ : 829.00 ms | 78423 /\s[a-zA-Z]{0,12}ing\s/ : 569.00 ms | 49659 /([A-Za-z]awyer|[A-Za-z]inn)\s/ : 685.00 ms | 209 /["'][^"']{0,30}[?!\.]["']/ : 56.00 ms | 8885 Total Time: 9746.00 ms ============================================================================== Delphi's own TRegEx with JIT: /Twain/ : 12.00 ms | 811 /(?i)Twain/ : 13.00 ms | 965 /[a-z]shing/ : 13.00 ms | 1540 /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ : 3.00 ms | 262 /\b\w+nn\b/ : 189.00 ms | 262 /[a-q][^u-z]{13}x/ : 154.00 ms | 4094 /Tom|Sawyer|Huckleberry|Finn/ : 22.00 ms | 2598 /(?i)Tom|Sawyer|Huckleberry|Finn/ : 64.00 ms | 4152 /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ : 371.00 ms | 2598 /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ : 497.00 ms | 1976 /Tom.{10,25}river|river.{10,25}Tom/ : 12.00 ms | 2 /[a-zA-Z]+ing/ : 105.00 ms | 78423 /\s[a-zA-Z]{0,12}ing\s/ : 197.00 ms | 49659 /([A-Za-z]awyer|[A-Za-z]inn)\s/ : 39.00 ms | 209 /["'][^"']{0,30}[?!\.]["']/ : 19.00 ms | 8885 Total Time: 1730.00 ms I will release it at Github (watch for announcement). Then @Kas Ob.you can try to get rid of __chkstk and see what happens.

@David Heffernan I explained why I am linking C code earlier... @Kas Ob.I would rather use the default options if I can. For example I am not sure of the performance implications of PCRE_NO_RECURSE. Now, if nothing else worked, I will try.

Well yes. I need a 32-bit __chkstk Here is the Visual Studio masm code page ,132 title chkstk - C stack checking routine ;*** ;chkstk.asm - C stack checking routine ; ; Copyright (c) Microsoft Corporation. All rights reserved. ; ;Purpose: ; Provides support for automatic stack checking in C procedures ; when stack checking is enabled. ; ;******************************************************************************* .xlist include vcruntime.inc .list ; size of a page of memory _PAGESIZE_ equ 1000h CODESEG page ;*** ;_chkstk - check stack upon procedure entry ; ;Purpose: ; Provide stack checking on procedure entry. Method is to simply probe ; each page of memory required for the stack in descending order. This ; causes the necessary pages of memory to be allocated via the guard ; page scheme, if possible. In the event of failure, the OS raises the ; _XCPT_UNABLE_TO_GROW_STACK exception. ; ; NOTE: Currently, the (EAX < _PAGESIZE_) code path falls through ; to the "lastpage" label of the (EAX >= _PAGESIZE_) code path. This ; is small; a minor speed optimization would be to special case ; this up top. This would avoid the painful save/restore of ; ecx and would shorten the code path by 4-6 instructions. ; ;Entry: ; EAX = size of local frame ; ;Exit: ; ESP = new stackframe, if successful ; ;Uses: ; EAX ; ;Exceptions: ; _XCPT_GUARD_PAGE_VIOLATION - May be raised on a page probe. NEVER TRAP ; THIS!!!! It is used by the OS to grow the ; stack on demand. ; _XCPT_UNABLE_TO_GROW_STACK - The stack cannot be grown. More precisely, ; the attempt by the OS memory manager to ; allocate another guard page in response ; to a _XCPT_GUARD_PAGE_VIOLATION has ; failed. ; ;******************************************************************************* public _alloca_probe _chkstk proc _alloca_probe = _chkstk push ecx ; Calculate new TOS. lea ecx, [esp] + 8 - 4 ; TOS before entering function + size for ret value sub ecx, eax ; new TOS ; Handle allocation size that results in wraparound. ; Wraparound will result in StackOverflow exception. sbb eax, eax ; 0 if CF==0, ~0 if CF==1 not eax ; ~0 if TOS did not wrapped around, 0 otherwise and ecx, eax ; set to 0 if wraparound mov eax, esp ; current TOS and eax, not ( _PAGESIZE_ - 1) ; Round down to current page boundary cs10: cmp ecx, eax ; Is new TOS bnd jb short cs20 ; in probed page? mov eax, ecx ; yes. pop ecx xchg esp, eax ; update esp mov eax, dword ptr [eax] ; get return address mov dword ptr [esp], eax ; and put it at new TOS bnd ret ; Find next lower page and probe cs20: sub eax, _PAGESIZE_ ; decrease by PAGESIZE test dword ptr [eax],eax ; probe page. jmp short cs10 _chkstk endp end

There are no conversions needed for Win64. For Win 32 I have the issue of windows calls (such as __imp__VirtualFree@12. Fortunately there is just one obj file that makes such calls and I will find a way to work around it (see earlier posts). Why not use bcc32c... I have only access to the free tools that only compile x86 code pcre has a number of configuration steps and options and cmake makes it very easy and convenient to automate everything. It would be very hard to do that manually correctly. cmake has a generator for bcc32c but is rather new and I am not sure how well tested that is. Saddly, I have more trust in the Visual Studio compiler, especially for code that requires stack checking. Does bcc32c do that? When I finish I will release the stuff at github, so people can take a look and make adjustments. Embarcadero can use their own compiler if they wish.

PCRE16. This is what Delphi uses and for a reason. It avoids conversions between delphi strings and PCRE strings.

I would prefer to not mess around with the c files and I think that pcre_jit_compile needs the stack checking anyway. Could anyone help translate the LLMV code to Delphi PLEASE? The prize is that everyone gets a much faster System.RegularExpressions and we may even convince Embarcadero to implement the changes which are really minor.

Use @David Heffernan trick mentioned above. It works fine. Your ld stuff does not appear to work with Visual studio object files (only gnu). I have tried /Gs1000000 but did not help. and yes I am aware of the other stuff you mention and can work around them.

In Visual Studio /GsX is the same as /GcX in other compilers. I am pretty sure: With the current __chkstk a crash occurs when this is called (debugging using the CPU window) Replacing that with linking VSs chkstk.obj works very smoothly

You are confusing /Gs with /GS https://docs.microsoft.com/en-us/cpp/build/reference/gs-control-stack-checking-calls?view=vs-2019https://docs.microsoft.com/en-us/cpp/build/reference/gs-control-stack-checking-calls?view=vs-2019 Already using /GS.