Linking to a C obj file that makes Windows calls

[pyscripter 689]

I am trying to link an .obj file generated using Visual Studio from C file.   The C file makes Windows calls such as VirtualFree;

When I try to compile the Delphi source code with {$L} directive in 32 bits Delphi complains about 

 

[dcc32 Error] ExtPCRE.pas(889): E2065 Unsatisfied forward or external declaration: '__imp__VirtualFree@12'

Is there a way to resolve this without changing the C code?

 

In 64 bits the name comes undecorated as __imp_VirtualFree and I am resolving this as:

 

type
  TVirtualFree = function(lpAddress: Pointer; dwSize: SIZE_T; dwFreeType: DWORD): BOOL; stdcall;

const
  __imp_VirtualFree: TVirtualFree = WinApi.Windows.VirtualFree;

Is this correct?

 

 

And a related question 

 

fpc provides for the following syntax:

 

Var  
  MyVar : MyType; external name ’varname’;

The effect of this declaration is twofold:

1. No space is allocated for this variable.
2. The name of the variable used in the assembler code is varname. This is a case sensitive name, so you must be careful.

 

Can you achieve the same somehow in Delphi? 

 

 

Edited July 22, 2020 by pyscripter

[Remy Lebeau 1396]

43 minutes ago, pyscripter said:

Is there a way to resolve this without changing the C code?

Short answer - no, since you can't have a @ in a variable/function name.

44 minutes ago, pyscripter said:

Is this correct?

You tell us.  Is the C code able to call VirtualFree() correctly when you do that?

45 minutes ago, pyscripter said:

Can you achieve the same somehow in Delphi?

AFAIK, no.  Delphi does not support external variables, only external functions.  In the specific case of variables exported from DLLs, you can use GetProcAddress() directly.  But for linking to variables defined in .obj files, no.

[David Heffernan 2345]

How did you compile the C code? 

[pyscripter 689]

10 minutes ago, David Heffernan said:

How did you compile the C code? 

$addPath = 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin'
$regexAddPath = [regex]::Escape($addPath)
$arrPath = $env:Path -split ';' | Where-Object {$_ -notMatch "^$regexAddPath\\?"}
$env:Path = ($arrPath + $addPath) -join ';'

$pcrePath = 'pcre-8.44'
Remove-Item –path $pcrePath –recurse
Expand-Archive -LiteralPath $pcrePath'.Zip' -DestinationPath .
copy CMakeLists.txt $pcrePath

pushd $pcrePath
cmake . -G "Visual Studio 16 2019" -A Win32 -DBUILD_SHARED_LIBS=OFF -DPCRE_BUILD_PCRE16=ON -DPCRE_BUILD_PCRE8=OFF -DPCRE_SUPPORT_UNICODE_PROPERTIES=ON -DPCRE_SUPPORT_JIT=ON -DPCRE_STATIC_RUNTIME=ON -DCMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH=OFF
cmake --build . --config Release
copy .\pcre16.dir\Release\*.obj ..\Source\obj\pcre\win32\
popd

Using the above Powershell script if that matters.  I have added the /GS- compiler flag to avoid further dependencies.

 

FYI I am trying to compile PCRE with JIT support.   This is what makes the Windows calls.

Edited July 22, 2020 by pyscripter

[pyscripter 689]

10 minutes ago, Remy Lebeau said:

Short answer - no

Is there a long answer?  Is there a way around it?  I am desperate as you see.

 

Is there a way to tell the C compiler not to add the decoration?

Edited July 22, 2020 by pyscripter

[Guest]

I can answer for the __imp__ only as the other is not clear for me, i always bring stuff from FPC to Delphi not the other way around, as for that declaration, Delphi achieve that in DCU format only, and only to be used with CBuilder or Delphi and without renaming, but not sure if you enabled/switched in "Generate - C/C++" something else than "generate DCU's only" to (may be) "Generate C++ .objs, headers, namespaces, export" then the result files will result in such types be exported, never used that.

 

Here an SO question and answers for https://stackoverflow.com/questions/5159353/how-can-i-get-rid-of-the-imp-prefix-in-the-linker-in-vc

most likely the third answer is the solution you are looking for, if didn't work then try other answers.

[pyscripter 689]

5 minutes ago, Kas Ob. said:

Here an SO question and answers for https://stackoverflow.com/questions/5159353/how-can-i-get-rid-of-the-imp-prefix-in-the-linker-in-vc

most likely the third answer is the solution you are looking for, if didn't work then try other answers.

This is only relevant for calls to the C runtime.  I am already doing that 

-DPCRE_STATIC_RUNTIME=ON

Has the same effect (activates the /MT option)

Edited July 22, 2020 by pyscripter

[David Heffernan 2345]

This __imp_ mangling, IIRC, is used by ms tools when linking using a import lib. Delphi links to DLLs in a different way. Not sure if the difference is important.

 

If it were me I'd compile each source file individually using cl and then link those.

 

Although I don't like to do that very much. It's tempting to be able to link objects statically. But it's fraught with difficulties. It's fiddly. And under x64 you have a further problem. The Delphi compiler doesn't respect the meta data in the object file describing the exception tables. So if exceptions are raised in the linked C code, then it's common for the process to be terminated forcefully.

 

These days I always prefer to link to DLLs using an sxs assembly to avoid problems with dll search paths. 

[Mahdi Safsafi 225]

@pyscripter 
 

Quote

type
  TVirtualFree = function(lpAddress: Pointer; dwSize: SIZE_T; dwFreeType: DWORD): BOOL; stdcall;

const
  __imp_VirtualFree: TVirtualFree = WinApi.Windows.VirtualFree;

Is this correct?

Yes this is correct. and it has a lazy form too :
 

const
  __imp_VirtualAlloc: Pointer = @VirtualAlloc;
  __imp_VirtualFree: Pointer = @VirtualFree;

Now to your linking issue. your best choice is to use a dll. If that is not an option, you can compile your c code for x86 using bcc32c and link the output with delphi. I did this several time and it works. EMB offers c/c++ compiler tool chain at no cost (only for x86). https://www.embarcadero.com/free-tools/ccompiler/ 
Use MSVC to compile for x64 and link (probably it already worked for you). 
If bcc32c is not an option, you can manually edit the generated object output(which I don't recommend) : use a hex editor and replace all imported function of your object file : 

// Replace '@' by '_' or better by a char that is not widely used. And make sure that the new name is not used elsewhere.
// you are a scripter man ... you can write a script that do the work for you.
__imp__VirtualAlloc@16 => __imp__VirtualAlloc_16
__imp__VirtualFree@12  => __imp__VirtualFree_12

// ---------------- delphi --------------
const
 __imp__VirtualAlloc_16: Pointer = @VirtualAlloc;
 __imp__VirtualFree_12: Pointer = @VirtualFree;

 

[pyscripter 689]

5 minutes ago, Mahdi Safsafi said:

Now to your linking issue. your best choice is to use a dll. If that is not an option, you can compile your c code for x86 using bcc32c and link the output with delphi. I did this several time and it works. EMB offers c/c++ compiler tool chain at no cost (only for x86). https://www.embarcadero.com/free-tools/ccompiler/ 

Does this EMB compiler not decorate the DLL imports?  If say you add WinApi.Windows to your uses clause does it automatically fix resolve the symbols?

[Mahdi Safsafi 225]

Just now, pyscripter said:

Does this EMB compiler not decorate the DLL imports?  If say you add WinApi.Windows to your uses clause does it automatically fix resolve the symbols?

Yes

[pyscripter 689]

I also found FileBinReplace.  (16K utility).

[Mahdi Safsafi 225]

Just now, pyscripter said:

I also found FileBinReplace.  (16K utility).

WOW ! Most of time I use HxD. Thanks.

There are some other utilities like COFF2OMF. it takes a coff object(msvc) and convert it to omf (delphi). I didn't use them but probably can work for you.

[Guest]

6 hours ago, pyscripter said:

I also found FileBinReplace.  (16K utility).

I get that, this tools is simple search and replace, it will not help here as it might break the checksum of obj file.

 

On other hand as Mahdi said there is many tools to do the convert, but one specific tool is the best of them all

https://www.agner.org/optimize/#objconv to download the binary

https://github.com/gitGNU/objconv for the source

https://www.agner.org/optimize/objconv-instructions.pdf   here you need to read modifying symbols page 7

 

this command might be the solution

Quote

objconv -np:__imp__: pcre16_jit_compile.obj pcre16_jit_compile_NameFix.obj

Quote

dumpbin pcre16_jit_compile.obj  /symbols

...

172 00000000 SECTA9 notype ()    Static       | @sljit_stack_resize@8
173 00000000 UNDEF  notype       External     | __imp__ReleaseMutex@4
174 00000000 UNDEF  notype       External     | __imp__WaitForSingleObject@8
175 00000000 UNDEF  notype       External     | __imp__CreateMutexW@12
176 00000000 UNDEF  notype       External     | __imp__GetSystemInfo@4
177 00000000 UNDEF  notype       External     | __imp__VirtualAlloc@16
178 00000000 UNDEF  notype       External     | __imp__VirtualFree@12
179 00000000 SECT64 notype ()    Static       | @ensure_buf@8
17A 00000000 SECT62 notype ()    Static       | @ensure_abuf@8
17B 00000000 SECT71 notype ()    Static       | @get_cpu_features@0

...

Quote

dumpbin pcre16_jit_compile_namefix.obj  /symbols 

...

72 00000000 SECTA9 notype ()    Static       | @sljit_stack_resize@8
173 00000000 UNDEF  notype       External     | ReleaseMutex@4
174 00000000 UNDEF  notype       External     | WaitForSingleObject@8
175 00000000 UNDEF  notype       External     | CreateMutexW@12
176 00000000 UNDEF  notype       External     | GetSystemInfo@4
177 00000000 UNDEF  notype       External     | VirtualAlloc@16
178 00000000 UNDEF  notype       External     | VirtualFree@12
179 00000000 SECT64 notype ()    Static       | @ensure_buf@8
17A 00000000 SECT62 notype ()    Static       | @ensure_abuf@8
17B 00000000 SECT71 notype ()    Static       | @get_cpu_features@0

Without even an @ or _ , you can tweak it as you want

[pyscripter 689]

On 7/23/2020 at 3:17 AM, Mahdi Safsafi said:

Yes this is correct. and it has a lazy form too :
 

const
  __imp_VirtualAlloc: Pointer = @VirtualAlloc;
  __imp_VirtualFree: Pointer = @VirtualFree;

Firstly I would like to report that the above works fine.  Thanks @Mahdi Safsafi

 

With minimal changes to System.RegularExpressionsAPI only, I got PCRE JIT to work in 64 bits (haven't fully sorted out 32 bits but I now know how to do it thanks to @Kas Ob.).  I had one issue related to this Stackoverflow question  that was resolved using @David Heffernan simple trick (thanks).  Was it worth my while?   Judge for yourselves.  Here are the results of my standard re benchmark:

 

                                                        Time     | Match count
==============================================================================
Delphi's own TRegEx:
                                        /Twain/ :        6.00 ms |         811
                                    /(?i)Twain/ :       45.00 ms |         965
                                   /[a-z]shing/ :      450.00 ms |        1540
                   /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ :      455.00 ms |         262
                                    /\b\w+nn\b/ :      680.00 ms |         262
                             /[a-q][^u-z]{13}x/ :      610.00 ms |        4094
                  /Tom|Sawyer|Huckleberry|Finn/ :      836.00 ms |        2598
              /(?i)Tom|Sawyer|Huckleberry|Finn/ :      991.00 ms |        4152
          /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ :     2755.00 ms |        2598
          /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ :     3021.00 ms |        1976
            /Tom.{10,25}river|river.{10,25}Tom/ :      464.00 ms |           2
                                 /[a-zA-Z]+ing/ :      805.00 ms |       78423
                        /\s[a-zA-Z]{0,12}ing\s/ :      563.00 ms |       49659
                /([A-Za-z]awyer|[A-Za-z]inn)\s/ :      837.00 ms |         209
                    /["'][^"']{0,30}[?!\.]["']/ :      321.00 ms |        8885
Total Time:    12853.00 ms
==============================================================================
Delphi's own TRegEx with Study:
                                        /Twain/ :        6.00 ms |         811
                                    /(?i)Twain/ :       49.00 ms |         965
                                   /[a-z]shing/ :      318.00 ms |        1540
                   /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ :       21.00 ms |         262
                                    /\b\w+nn\b/ :      586.00 ms |         262
                             /[a-q][^u-z]{13}x/ :      417.00 ms |        4094
                  /Tom|Sawyer|Huckleberry|Finn/ :       27.00 ms |        2598
              /(?i)Tom|Sawyer|Huckleberry|Finn/ :      216.00 ms |        4152
          /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ :     2607.00 ms |        2598
          /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ :     2768.00 ms |        1976
            /Tom.{10,25}river|river.{10,25}Tom/ :       49.00 ms |           2
                                 /[a-zA-Z]+ing/ :      758.00 ms |       78423
                        /\s[a-zA-Z]{0,12}ing\s/ :      565.00 ms |       49659
                /([A-Za-z]awyer|[A-Za-z]inn)\s/ :      654.00 ms |         209
                    /["'][^"']{0,30}[?!\.]["']/ :       50.00 ms |        8885
Total Time:     9108.00 ms
==============================================================================
Delphi's own TRegEx with JIT:
                                        /Twain/ :       12.00 ms |         811
                                    /(?i)Twain/ :       13.00 ms |         965
                                   /[a-z]shing/ :       11.00 ms |        1540
                   /Huck[a-zA-Z]+|Saw[a-zA-Z]+/ :        4.00 ms |         262
                                    /\b\w+nn\b/ :      126.00 ms |         262
                             /[a-q][^u-z]{13}x/ :      169.00 ms |        4094
                  /Tom|Sawyer|Huckleberry|Finn/ :       22.00 ms |        2598
              /(?i)Tom|Sawyer|Huckleberry|Finn/ :       63.00 ms |        4152
          /.{0,2}(Tom|Sawyer|Huckleberry|Finn)/ :      273.00 ms |        2598
          /.{2,4}(Tom|Sawyer|Huckleberry|Finn)/ :      334.00 ms |        1976
            /Tom.{10,25}river|river.{10,25}Tom/ :       12.00 ms |           2
                                 /[a-zA-Z]+ing/ :       86.00 ms |       78423
                        /\s[a-zA-Z]{0,12}ing\s/ :      158.00 ms |       49659
                /([A-Za-z]awyer|[A-Za-z]inn)\s/ :       36.00 ms |         209
                    /["'][^"']{0,30}[?!\.]["']/ :       18.00 ms |        8885
Total Time:     1356.00 ms

Almost a 10 fold speedup.

 

Now here is my next question.  System.RegularExpressionsAPI resolves the __chkstk dependency as follows.

procedure __chkstk;
asm
end;

This created all sort of unpredictable crashes.  I am not sure whether the crashes were a result of lack of stack checking or a difference in the calling conventions or both. I had to resolve __chkstk by linking to chkstk.obj provided by Visual Studio, however this is kind of problematic because there are licensing issues and you cannot distribute chkst.obj.

 

I have looked around and saw the the LLMV projects defines __chkstk as follows:

// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include "../assembly.h"
// _chkstk routine
// This routine is windows specific
// http://msdn.microsoft.com/en-us/library/ms648426.aspx
// Notes from r227519
// MSVC x64s __chkstk and cygmings ___chkstk_ms do not adjust %rsp
// themselves. It also does not clobber %rax so we can reuse it when
// adjusting %rsp.
#ifdef __x86_64__
.text
.balign 4
DEFINE_COMPILERRT_FUNCTION(___chkstk_ms)
        push   %rcx
        push   %rax
        cmp    $0x1000,%rax
        lea    24(%rsp),%rcx
        jb     1f
2:
        sub    $0x1000,%rcx
        test   %rcx,(%rcx)
        sub    $0x1000,%rax
        cmp    $0x1000,%rax
        ja     2b
1:
        sub    %rax,%rcx
        test   %rcx,(%rcx)
        pop    %rax
        pop    %rcx
        ret
END_COMPILERRT_FUNCTION(___chkstk_ms)
#endif // __x86_64__

Any help in converting this to a Delphi assembler function would be highly appreciated.

 

Edited July 24, 2020 by pyscripter

[David Heffernan 2345]

Just compile the C++ code with options that suppress stack checking.

 

Seriously though, putting all this in a DLL is a far better approach.

[pyscripter 689]

8 minutes ago, David Heffernan said:

Just compile the C++ code with options that suppress stack checking.

 

Seriously though, putting all this in a DLL is a far better approach.

I have tried with /Gs9999999 which suppresses the stack checking but I still got crashes.   I suspect that pcre needs the stack checking.  Please note __chkstk automatically allocates stack if needed.

 

;_chkstk - check stack upon procedure entry
;
;Purpose:
;       Provide stack checking on procedure entry. Method is to simply probe
;       each page of memory required for the stack in descending order. This
;       causes the necessary pages of memory to be allocated via the guard
;       page scheme, if possible. In the event of failure, the OS raises the
;       _XCPT_UNABLE_TO_GROW_STACK exception.
;

I understand that a DLL is a safer and easier option, but this is meant to be an enhancement of System.RegularExpressions which uses static linking of obj files.

Edited July 24, 2020 by pyscripter

[Guest]

3 minutes ago, pyscripter said:

/Gs9999999

https://docs.microsoft.com/en-us/cpp/build/reference/gs-buffer-security-check?view=vs-2019

[Mahdi Safsafi 225]

Yes disabling stack check will not suppress ___chkstk_ms. Only /GsX will do provided that this option is supported by your compiler and X is higher than probe size. 

Now. are you sure that the crash comes from ___chkstk_ms ? also have you disabled other runtime check ? 

Edited July 24, 2020 by Mahdi Safsafi

[pyscripter 689]

5 minutes ago, Kas Ob. said:

https://docs.microsoft.com/en-us/cpp/build/reference/gs-buffer-security-check?view=vs-2019

You are confusing /Gs with /GS  https://docs.microsoft.com/en-us/cpp/build/reference/gs-control-stack-checking-calls?view=vs-2019https://docs.microsoft.com/en-us/cpp/build/reference/gs-control-stack-checking-calls?view=vs-2019

Already using /GS.

Edited July 24, 2020 by pyscripter

[Fr0sT.Brutal 900]

From my experience on building with C objs.

 

Converting from COFF is not needed, XE2+ can handle COFF.

 

VC:

* build with platform toolset = SDK

* config > C/C++ > Code gen > Runtime = /MD

* config > C/C++ > Code gen > buffer security check = No

* if __RTC_* are required:
'Basic Runtime Checks' are used; to disable them, you can go on the project properties, and set the Configuration Properties>C/C++>Code gen>Basic Runtime Checks option to 'Default'.
https://stackoverflow.com/questions/10158013/unresolved-external-symbols-rtc-in-windows-programming-tutorial

* if __aullshr is required:
turn off "/O1" optimization
Configuration Properties>C/C++>Optimization
https://stackoverflow.com/questions/54719855/unresolved-external-symbol-aullshr-when-optimization-is-turned-off
+ Configuration Properties>C/C++>debug info format C7 (/Z7)

* if chkstk is required
Configuration Properties > C/C++ > Command Line > Additional Options, add /Gs1000000

 

Delphi:

* use Windows, ctrl

* you can declare names as strings if they're mangled

function CheckStream(Data: Pointer; Size: NativeUInt): Byte; cdecl; external name '?CheckStream@@YA_NPBEI@Z';
{$L ZDecoder.obj}

Also: [VC] names are mangled in С++ build mode, try to build in С mode that only adds "_"

 

The main issue I hadn't solved is a specific order of linking obj's when they have wide dependencies on each other. Probably "ld -relocatable .\ReleaseDLL\SomeDir\*.obj -o OneObj.o" would help

Edited July 24, 2020 by Fr0sT.Brutal

[pyscripter 689]

6 minutes ago, Mahdi Safsafi said:

Yes disabling stack check will not suppress ___chkstk_ms. Only /GcX will do provided that this option is supported by your compiler and X is higher than probe size. 

Now. are you sure that the crash comes from ___chkstk_ms ? also have you disabled other runtime check ? 

In Visual Studio /GsX is the same as /GcX in other compilers.

I am pretty sure:

• With the current __chkstk a crash occurs when this is called (debugging using the CPU window)
• Replacing that with linking VSs chkstk.obj works very smoothly

[Guest]

7 minutes ago, pyscripter said:

You are confusing /Gs with /GS

Quote

You can turn stack probes on or off by using the check_stack directive. /Gs and the check_stack pragma have no effect on standard C library routines; they affect only the functions you compile.

Now from this table

https://docs.microsoft.com/en-us/cpp/preprocessor/check-stack?view=vs-2019

if /Gs- enabled then /GS is irrelevant 

and #pragma check_stack(on) will dominate if exist in the code

[pyscripter 689]

5 minutes ago, Fr0sT.Brutal said:

The main issue I hadn't solved is a specific order of linking obj's when they have wide dependencies on each other. Probably "ld -relocatable .\ReleaseDLL\SomeDir\*.obj -o OneObj.o" would help

Use @David Heffernan trick mentioned above.  It works fine.

 

Your ld stuff does not appear to work with Visual studio object files (only gnu).

 

I have tried /Gs1000000 but did not help.

 

and yes I am aware of the other stuff you mention and can work around them.

[pyscripter 689]

12 minutes ago, Kas Ob. said:

ou can turn stack probes on or off by using the check_stack directive. /Gs and the check_stack pragma have no effect on standard C library routines; they affect only the functions you compile.

I would prefer to not mess around with the c files and I think that pcre_jit_compile needs the stack checking anyway.

 

Could anyone help translate the LLMV code to Delphi PLEASE?  The prize is that everyone gets a much faster System.RegularExpressions and we may even convince Embarcadero to implement the changes which are really minor. 

Edited July 24, 2020 by pyscripter