"Gotchas" when calling from a C DLL to a Delphi DLL?

[Dave Nottage 563]

A while ago, I managed to make the whole Citrix Virtual Channel thing work, i.e. I constructed a C DLL much like the examples from the Virtual Channel SDK, and just "re-route" the calls to a DLL written in Delphi (long story short: so far it has been the only way I could make this work).

 

The problem now is that on some machines, the Citrix startup sequence fails, and I'm at a loss as to why. The normal sequence goes like this:

 

1. A user starts an app from within the Citrix environment
2. Citrix checks which virtual channels are registered with the system, which includes ours (the DLL that is "registered" with Citrix is the C-based DLL)
3. Citrix loads the C-based DLL, and calls the DriverOpen function
4. The DriverOpen function in the C-based DLL loads the Delphi DLL and obtains the method address for the corresponding DriverOpen function in the Delphi DLL
5. The C-based DLL calls the DriverOpen function in the Delphi DLL, the Delphi function "does its thing", and returns the value back to the C-based DLL, which passes that result back to Citrix.
6. The next part of the sequence is that Citrix calls 2 other functions, DriverSetInformation and DriverInfo. These are "re-routed" in the same manner as DriverOpen

 

The problem is that Citrix throws an error after step 5 completes - it is known that the Delphi DLL finishes its work with DriverOpen successfully (through logging). As indicated above, this problem happens only on some machines, so I'm wondering whether there's something I might be able to look for that is a "gotcha" that might be causing the problem? Sadly, Citrix's diagnostics (or lack thereof) are particularly woeful - there are no clues given as to what the issue might be, other than a cryptic error message:

 

  Error 1021: Invalid Handle

 

That has no help for it.

 

Keep in mind that this process all works on most machines;  all are Windows 10 (same build). I'm hoping to be able to resolve this without having to divulge any source code.

 

Edit:

 

On thinking a bit more, I figure that it might help to show my conversion from C to Delphi of one of the types being passed to DriverOpen. The function itself is declared like this in C

 

int DriverOpen(PVD pVd, PVDOPEN pVdOpen, PUINT16 puiSize)

 

Note that the pVdOpen parameter is not referenced at all within the function in the Delphi DLL, however I figure if I might have that wrong, it might be causing some problem.  The pVd parameter is used in all functions. These are the relevant types in their C versions and what I've translated them to in Delphi:

 

C
 

#define MODULE_LENGTH 13
#define PWFCAPI TYPEDEF_CDECL TYPEDEF_FAR *

typedef UINT32 (PWFCAPI PLIBPROCEDURE)();
typedef PVOID                 HND;

typedef struct _DLLLINK {
/* NOTE: 1st six elements must be same as MINIDLL */
    USHORT Segment;             /* starting seg of mem allocated for dll (aligned) */
    USHORT DllSize;             /* actual size of dll in paragraphs */
    USHORT ProcCount;           /* number of procedures in call table */
    PVOID pProcedures;          /* pointer to procedure call table */
    PVOID pData;                /* pointer to dll data structure */
    PUCHAR pMemory;             /* pointer to malloced memory (not aligned) */
    BYTE ModuleName[MODULE_LENGTH];        /* client module name (8.3) */
    LPVOID pLibMgrCallTable;    /* Pointer to LibMgrTable (input) */
    USHORT ModuleDate;          /* module date in dos format */
    USHORT ModuleTime;          /* module time in dos format */
    ULONG ModuleSize;           /* module file size in bytes */
    struct _DLLLINK * pNext;    /* pointer to next DLLLINK structure */
    ULONG DllFlags;             /* DLL flags (embed..) */
    /* Everything after here is not included for the ModuleEnum call. */

    HND        LibraryHandle;

} DLLLINK, * PDLLLINK;

typedef struct _VDOPEN
{
    LPVOID pIniSection;
    PDLLLINK pWdLink;
    ULONG ChannelMask;
    PLIBPROCEDURE pfnWFEngPoll;
    PLIBPROCEDURE pfnStatusMsgProc;
    HND hICAEng;
} VDOPEN, FAR * PVDOPEN;

Delphi:

  HND = THandle;
  PLIBPROCEDURE = function: UINT32; cdecl;

  PDLLLINK = ^DLLLINK;
  _DLLLINK = record
    Segment: USHORT;
    DllSize: USHORT;
    ProcCount: USHORT;
    pProcedures: Pointer;
    pData: Pointer;
    pMemory: PUCHAR;
    ModuleName: array[0..12] of Byte;
    pLibMgrCallTable: ^DWORD;
    ModuleDate: USHORT;
    ModuleTime: USHORT;
    ModuleSize: ULONG;
    pNext: PDLLLINK;
    DllFlags: ULONG;
    LibraryHandle: HND
  end;
  DLLLINK = _DLLLINK;

  PVDOPEN = ^VDOPEN;
  __VDOPEN = record
    pIniSection: Pointer;
    pWdLink: PDLLLINK;
    ChannelMask: ULONG;
    pfnWFEngPoll: PLIBPROCEDURE;
    pfnStatusMsgProc: PLIBPROCEDURE;
    hICAEng: HND
  end;
  VDOPEN = __VDOPEN;

 

 

Edited February 18, 2019 by Dave Nottage

[Remy Lebeau 1436]

13 hours ago, Dave Nottage said:

A while ago, I managed to make the whole Citrix Virtual Channel thing work, i.e. I constructed a C DLL much like the examples from the Virtual Channel SDK, and just "re-route" the calls to a DLL written in Delphi (long story short: so far it has been the only way I could make this work).

That would imply that the Delphi DLL is not coded properly for the Citrix driver API, since Delphi can certainly produce C-compatible DLLs.

 

Aside from that, the only thing I see wrong in your Delphi translation is that HND should be a Pointer instead of a THandle, and pLibMgrCallTable should be a Pointer instead of a ^DWORD.

 

However, you did not show your translation of the PVD type, or of the DriverOpen() and other DLL function declarations.  Since Citrix is complaining about an invalid handle, did you debug your C and Delphi DLLs to log the value of the handle that DriverOpen() actually returns, and the value of the handle that the other DLL functions receive?  Where exactly does DriverOpen() return the handle to - the 'int' return value, or in the PVD parameter?

[Dave Nottage 563]

12 hours ago, Remy Lebeau said:

That would imply that the Delphi DLL is not coded properly for the Citrix driver API, since Delphi can certainly produce C-compatible DLLs.

Regarding going down the route of a "pure" Delphi DLL: The DLL needs to link to a .lib file that is provided with the SDK. Since Delphi cannot link to .lib files, I needed to extract the .obj file (there is only one in the .lib) and link to that. There are 9 symbols that need to be linked to: 2 are "direct" static links that look like this:
 

function _Load(pLink: PDLLLINK): Integer; cdecl; external name '_Load';
function VdCallWd(pVd: PVD; ProcIndex: USHORT; pParam: PVOID; puiSize: PUINT16): Integer; stdcall; external name '_VdCallWd@16';

I'm not sure of the correct terminology for it, however the other 7 symbols are the functions that need to be "implemented", including DriverOpen, however because of name mangling, they're exported with symbol names like this (e.g.): _DriverOpen@12. Since it's impossible to name Delphi functions with an "@" symbol, I used ObjConv to rename the symbols, removing the @12 and while I was at it, the underscore. This results in, for example, DriverOpen being declared thus:

 

function DriverOpen(pVd: PVD; pVdOpen: PVDOPEN; puiSize: PUINT16): Integer; stdcall;

I cannot add "external" and "name" for these functions because they need to be "implemented", rather than "direct" static links. Again, not sure of the exact terminology for this distinction.

 

In this "pure" Delphi DLL mode, everything compiles OK and Citrix loads the DLL successfully, however for one reason or another it never calls DriverOpen (at all, in this mode), which apparently (as per my original post) it usually calls first. It does however call the DriverInfo function, and does so successfully. Note that the only function that is exported is the Load function (as per their C examples). The Load function is documented as doing the work of setting up a call table to the other functions.

 

Thanks regarding the HND type and pLibMgrCallTable ; I have now corrected that. As far as the "pure" mode goes, this made no difference, however.

 

Back to the "conduit" mode: I had set up some logging to dump what values are coming into the C DLL, and into the Delphi DLL, and see what goes back from Delphi to C, in case there was something untoward happening, and I've discovered that although the DriverOpen function in the Delphi DLL actually completes, it does not return to the calling point in the C DLL, i.e. there's a problem somewhere in between. I put in an exception handler, but I guess it's not right:

 

char* IntToStr(int Number) {
    char Result[50];
    sprintf(Result, "%d", Number);
    return(Result);
}

int HandleException(unsigned int code, struct _EXCEPTION_POINTERS *ep) {
    char Msg[512];
    strcpy(Msg, "Exception code : ");
    strcat(Msg, IntToStr(code));
    OutputDebugString(Msg);
    return EXCEPTION_EXECUTE_HANDLER;
}

int DriverOpen(PVD pVd, PVDOPEN pVdOpen, PUINT16 puiSize)
{
    OutputDebugString("WSCitrix Conduit DLL DriverOpen called");
    if (LoadDelphiDLL()) {
        __try {
            int result = DelphiDriverOpen(pVd, pVdOpen, puiSize);
            OutputDebugString("WSCitrix Conduit DLL DriverOpen returned from Delphi");
            return(result);
        }
        __except(HandleException(GetExceptionCode(), GetExceptionInformation())) {
        }
    }
}

i.e. it calls DelphiDriverOpen (which completes in the Delphi DLL), but never reaches the subsequent OutputDebugString

My limited C skills are probably shining through 😉

 

Edited February 19, 2019 by Dave Nottage

[Remy Lebeau 1436]

13 hours ago, Dave Nottage said:

I put in an exception handler, but I guess it's not right

No, it is not.  And besides, your IntToStr() has undefined behavior anyway.  You should get rid of that function and just use sprintf() directly in your HandleException().  But that is beside the point.

13 hours ago, Dave Nottage said:

it calls DelphiDriverOpen (which completes in the Delphi DLL), but never reaches the subsequent OutputDebugString

That implies to me that possibly either:

• there is a calling convention mismatch in how DelphiDriverOpen() is declared in both languages.
• the Delphi DLL is corrupting the call stack. 
• an uncaught exception is being thrown.

You will have to use the debugger to verify one way or the other.

Edited February 19, 2019 by Remy Lebeau

[Dave Nottage 563]

6 hours ago, Remy Lebeau said:

No, it is not

What is the correct way?

 

6 hours ago, Remy Lebeau said:

You will have to use the debugger to verify one way or the other.

Unfortunately, that is easier said than done. The process which loads the DLL runs only when an app is run on Citrix, and it makes the calls to DriverOpen etc immediately, and once the process fails it needs to be restarted, which makes it nigh on impossible to attach to the process that is loading the DLL, let alone debug it.

 

6 hours ago, Remy Lebeau said:

• there is a calling convention mismatch in how DelphiDriverOpen() is declared in both languages.
• the Delphi DLL is corrupting the call stack. 
• an uncaught exception is being thrown

For the first point: I'll need to double check that tomorrow, however given that everything works 100% of the time on most machines, I figure it's unlikely.
 

For the second point: in what ways might this happen?

For the third point: That's the reason why I have the exception handler, if I can make it work

Edited February 20, 2019 by Dave Nottage

[Remy Lebeau 1436]

On 2/19/2019 at 4:25 PM, Dave Nottage said:

Unfortunately, that is easier said than done. The process which loads the DLL runs only when an app is run on Citrix, and it makes the calls to DriverOpen etc immediately, and once the process fails it needs to be restarted, which makes it nigh on impossible to attach to the process that is loading the DLL, let alone debug it.

I would suggest writing your own test EXE that simply loads the C DLL directly and calls its DriverOpen() function.  Then you can step into the DLL with the debugger to see how it calls into your DelphiDriverOpen() function, see how the parameter data gets passed back and forth, how the call stack is managed, etc.

On 2/19/2019 at 4:25 PM, Dave Nottage said:

For the first point: I'll need to double check that tomorrow, however given that everything works 100% of the time on most machines, I figure it's unlikely.
 

For the second point: in what ways might this happen?

For the third point: That's the reason why I have the exception handler, if I can make it work

• then it may be an environmental issue rather than a coding mistake.  Maybe the offending machine(s) are simply using a different/newer API than what your DLLs use.  Who knows.
• can't answer that seeing the actual code, so stepping through it with a debugger.
• if the exception handler is not being called, then either there is no exception being thrown at all, or it is not an SEH-based exception.

Edited February 21, 2019 by Remy Lebeau

[Dave Nottage 563]

1 hour ago, Remy Lebeau said:

- if the exception handler is not being called, then either there is no exception being thrown at all, or it is not an SEH-based exception.

Thanks for the other tips. For this one, I still don't know whether or not I have coded the exception handler correctly, or is it just the IntToStr thing that's a problem?

[Remy Lebeau 1436]

26 minutes ago, Dave Nottage said:

I still don't know whether or not I have coded the exception handler correctly, or is it just the IntToStr thing that's a problem?

The exception handler itself is coded fine (once you fix the undefined behavior of your IntToStr() implementation), it just may not catch every possible type of exception that can be thrown.  It can only handle SEH exceptions.  But there is no guarantee that what you are experiencing is even related to exceptions.  There may not even be an exception being thrown at all.

Edited February 20, 2019 by Remy Lebeau

[Dave Nottage 563]

1 hour ago, Remy Lebeau said:

There may not even be an exception being thrown at all

What else might it be doing, if the call to the Delphi DLL does not return "properly"?

[Remy Lebeau 1436]

3 hours ago, Dave Nottage said:

What else might it be doing, if the call to the Delphi DLL does not return "properly"?

There are only 2 other ways that I know of:

• a deadlock inside of the function, so it never even tries to exit.
• a corrupted call stack, ie the return address gets overwritten before the function tries to exit.

Edited February 21, 2019 by Remy Lebeau