Load DLL from resource, bypassing a file?

[Angus Robertson 577]

I'm looking at adding the OpenSSL DLLs as resources to ICS projects, to simplify distribution to one file.

 

But currently it seems a DLL can only be loaded using the LoadLibrary API from a disk file, so the resource has to be written to disk first then LoadLibrary used. 

 

It would be faster to load the resource straight to memory, but there does not seem to be an API that does the magic of LoadLibrary for a memory address, need a handle to the library so that exports can be loaded. 

 

A little Googling suggests some hacking techniques might work, but with the risk that AV software might take a dislike to the application.

 

Any simple solution, or do we have to live with temporary files?

 

Angus

[Remy Lebeau 1436]

29 minutes ago, Angus Robertson said:

But currently it seems a DLL can only be loaded using the LoadLibrary API from a disk file, so the resource has to be written to disk first then LoadLibrary used. 

That is correct.  That is simply how Windows operates.

Quote

It would be faster to load the resource straight to memory, but there does not seem to be an API that does the magic of LoadLibrary for a memory address

Correct, and it would not really be much faster anyway.  You can't just use a DLL as-is in memory, as there are fixups and relocations and such that have to be performed during the load process to make sure the DLL can actually function correctly.  There are 3rd party DLL loaders that can operate on a memory block, but they are dangerous to use, and can't operate on a read-only resource anyway, so you would have to extract the DLL to writable memory first.  So you are better off just using a disk file and letting Windows do its job normally.

Quote

A little Googling suggests some hacking techniques might work, but with the risk that AV software might take a dislike to the application.

That is another reason not to use in-memory DLL loaders.

Quote

Any simple solution, or do we have to live with temporary files?

Do you really need to store the DLLs in your EXE to begin with?  That is just going to bloat the EXE and make more work for Windows when it is loading the EXE.  Why can't you just deploy the DLLs alongside your EXE instead?  If you need to distribute a single file, use an installer or self-extracting zip archive or similar.

Edited October 2, 2023 by Remy Lebeau

[Angus Robertson 577]

Quote

Do you really need to store the DLLs in your EXE to begin with? 

We are providing component libraries for others to built SSL capable applications, and avoiding DLLs does come up regularly, it should be an option that other developers can use or ignore. 

 

The best solution is YuOpenSSL which is OpenSSL compiled into a DCU so no DLLs.  It does increase application size, probably the same as using resources, but generally applications using OpenSSL distribute their favourite version of the DLLs in their application directory, so you end up with numerous similar DLLs all over the place.  But YuOpenSSL is a lot of effort to maintain and is therefore a commercial product not everyone wants to buy. 

 

In theory, the OpenSSL resources only need to be written to files once, but making sure those files are the latest or oldest versions may be challenging.

 

Angus

 

 

 

[David Heffernan 2353]

If you can distribute one file you can distribute more than one. This always feels like a pointless discussion. Solving a problem that doesn't exist. 

[Anders Melander 1815]

17 hours ago, Angus Robertson said:

I'm looking at adding the OpenSSL DLLs as resources to ICS projects, to simplify distribution to one file.

But currently it seems a DLL can only be loaded using the LoadLibrary API from a disk file, so the resource has to be written to disk first then LoadLibrary used. 

I've been loading DLLs from resources for years using Martin Offenwanger's BTMemoryModule: https://github.com/DSPlayer/memorymodule

It's a Pascal port of Joachim Bauch's solution: https://www.joachim-bauch.de/tutorials/loading-a-dll-from-memory/

 

I have a small patch that I have never gotten around to committing to the BTMemoryModule repo. If you elect to go the BTMemoryModule route let me know and I will push a pull request to the repo.

[Fr0sT.Brutal 900]

I maintained the fork of Martin's port of Joahim's lib for some time but abandoned it due to low requirements, instability and extremely frustrating process of porting updates.

https://github.com/Fr0sT-Brutal/Delphi_MemoryModule

Currently I'd recommend either to use original lib via OBJ files or to not use this tech at all. Alas, the process of loading from memory involves too much of black voodoo magic

Edited October 3, 2023 by Fr0sT.Brutal

[Anders Melander 1815]

55 minutes ago, Fr0sT.Brutal said:

https://github.com/Fr0sT-Brutal/Delphi_MemoryModule

Ah, yes. Forgot about that one. It's too bad your changes from Martin Offenwanger's version predate the Git repo. That makes it pretty impossible to identify the changes.

It does look like they have been updated independently since the fork though. FWIW, I can see you have a fix similar to mine in your version.

 

1 hour ago, Fr0sT.Brutal said:

Alas, the process of loading from memory involves too much of black voodoo magic

I do think it's fairly well-documented but I will agree that it's pretty complex and not for the faint of heart (The implementation of it, that is. The use is trivial).

I think my attitude toward it would be: If it works right away use it, otherwise don't bother with it. As I wrote, I've been using it for years without any problems.

[Fr0sT.Brutal 900]

5 minutes ago, Anders Melander said:

Ah, yes. Forgot about that one. It's too bad your changes from Martin Offenwanger's version predate the Git repo. That makes it pretty impossible to identify the changes.

I have all these commits in non-public branch and could send the whole repo to you if you're interested.

 

[Angus Robertson 577]

Thanks all, BTMemoryLoadLibary and BTMemoryGetProcAddress would appear to be the solution, but do add a lot of extra code. 

 

I think I'll implement the simpler save/load resource first, which will be conditional, then look at adding more conditional code to optionally load directly to memory.  Ultimately, how to build applications is up to the developer. 

 

The OBJ solution is best, it's called YuOpenSSL, but is not a simple compile of the OpenSSL C code, lots of macros and other issues. 

 

Angus

 

[sfrazor 3]

I have used both MemoryModuyle from Fr0sT.Brutal and BTMemoryModule from Martin Offenwanger linked above. Both were stable enough for our production code recognizing there are limitations and am adapting code using ICS with YuOpenSSL.  We chose the MemoryModule from Fr0sT.Brutal only because it was the first one we came across and its fairly solid.  If there is an improved branch of either I'd love to have it.  

 

I could go in to a lengthy description of an issue with both if anyone is interested.  The issue is similar between the two.

 

I use both with YuOpenSSL which works very well.  I highly recommend it as a OpenSSL snap-in.  Including using it with ICS and Synapse (I haven't had much luck using it with Indy).  I think Ralf should open a 3rd party section here for his product FWIW.

 

@FroSt.Brutal for an abandoned project, its pretty darn good.  I pulled the dev branch and made a slight mod to remove the Ordinal entry points because it was giving me an error and I just didn't need it.

 

Regardless of the nay-sayers asking why and how come etc...  there is a need.  In my case I have no choice as its in legacy code that I am forced to build upon.

 

The only issue I have with either is figuring out how to debug a DLL loaded with these memory loaders.  I'm learning but thats beyond my skill set.  Can anyone help?

 

@FroSt.Brutal check your PM please.  

Angus, either is an excellent starting point for your memory based resource loader and the need is there.  IMO.

 

Edited October 3, 2023 by sfrazor

[Angus Robertson 577]

My plan is to build the OpenSSL resource file at the same time as signing the DLLs and distribute them together in the zip for each new release.   The latest resource file will also be part of the ICS distribution alongside the latest DLLs.  ICS will only support OpenSSL 3.0 and later, versions will under support. 

 

The resource files will have the three DLLs, licence file, and a version string, used for the path when extracting the DLLs into a temporary directory. 

 

Is there anything else that would be useful to allow other projects to use the same OpenSSL resource files, to avoid building their own?

 

Angus

[sfrazor 3]

7 hours ago, Angus Robertson said:

 

Is there anything else that would be useful to allow other projects to use the same OpenSSL resource files, to avoid building their own?

 

 

You  would have to ask public opinion on this one.  Most would probably say 'why bother doing this at all'.  To a limited few it would be a blessing.  I would use it in my current project depending on how large adding the resource makes the DLL's overall.

 

A  couple of things should be noted:

As just mentioned to me by Fr0sT.Brutal there is no auto-run. DLLMain calls are not supported that I am aware of.  So if you are counting on DLLMain, OpenSSL wont initialize.  Currently when I call SSL_CTX_new it fails ONLY when called from a memoryloaded DLL.  This may be the root of a failure I'm currently experiencing.  I need to test that theory and discuss with Ralf (YuOpenSSL). Someone please correct me if I'm wrong.

 

Next,  and I'm saying this without solid verification, exception handling in memorymodule loaded DLL's doesn't work maybe due to .SAFESEH section of the DLL not being mapped/loaded.   If this is the case Windows won't allow SEH Handlers because it believes they are not safe.  It will crash the app using memorymodule.  This needs to be fixed.  I'm trying to  understand this part now so anyone wanting to correct me please do so.  I am no authority on this.

 

and last, BTMemorymodule is only 32bit last time I looked.  Has that changed?

 

Blackbone an Dark Memory Loader both correct the above issues in C but there is no equivalent in Delphi unless that is stashed away in private repos.

 

I'm just not sure which one would be a viable repo to fork from....  So, Angus, proceed with caution 🙂

 

[Fr0sT.Brutal 900]

You could try original MemoryModule for DLLMain and exception handling.

Looks like Joahim is not developing it anymore too. Exception issue here https://github.com/fancycode/MemoryModule/issues/4 was reported in 2012. It contains some solutions which haven't been implemented though

And the issue with OpenSSL https://github.com/fancycode/MemoryModule/issues/109

Also check this https://www.codeproject.com/tips/430684/loading-win-dlls-manually-without-loadlibrary

 

As you can see, the overall process is still a kind of hack with many issues arising. That's why I dropped this project (and probably this is the case for Joahim too). Alas, the most stable solution is to extract libs to files and then load them normally.

Edited October 5, 2023 by Fr0sT.Brutal

[Angus Robertson 577]

Quote

Alas, the most stable solution is to extract libs to files and then load them normally.

Which is what I'll do for the first version, may experiment later with memory loading, but don't plan to spend much effort on it.

 

Angus

 

[FPiette 385]

2 hours ago, Angus Robertson said:

Quote

Alas, the most stable solution is to extract libs to files and then load them normally.

Which is what I'll do for the first version, may experiment later with memory loading, but don't plan to spend much effort on it.

Care should be taken concerning the directory where to extract the DLL. Usually the application are stored in a directory requiring admin permission to be written (Program Files). The DLL must be extracted to a writable directory. It could be specified in the application INI file (or registry) and/or located in some directory in the user profile (Somewhere in "AppData").

[Angus Robertson 577]

The DLLs will probably end up in a similar place to the existing sample INI files, which works well, unless you want to find them manually...

 

Angus

 

[Fr0sT.Brutal 900]

One more concern: with the global trend to tighten security of user-level OS-es (I mean, all that are not severe Linux), I suspect this kind of lib loading could be likely removed or at least require admin rights.

[Anders Melander 1815]

2 hours ago, Fr0sT.Brutal said:

One more concern: with the global trend to tighten security of user-level OS-es (I mean, all that are not severe Linux), I suspect this kind of lib loading could be likely removed or at least require admin rights.

I can't see how they could prevent it without breaking a lot of applications that are doing similar (but not necessarily the same) things. It's basically just juggling page protection and modifying memory. All done in USER-level.

Also, why should it pose a security threat?

[Fr0sT.Brutal 900]

2 hours ago, Anders Melander said:

It's basically just juggling page protection and modifying memory

Of course and it could be used by malicious software => thus security threat => thus should be removed or at least restricted to admins.

Look at Android which gets more and more restrictions of getting into the system with every new release.

 

[dummzeuch 1517]

17 hours ago, Fr0sT.Brutal said:

Of course and it could be used by malicious software => thus security threat => thus should be removed or at least restricted to admins.

Look at Android which gets more and more restrictions of getting into the system with every new release.

Android <> Windows

Microsoft has a track record of staying backwards compatible for a long time

Edited October 7, 2023 by dummzeuch

[Anders Melander 1815]

7 hours ago, Fr0sT.Brutal said:

it could be used by malicious software

I fail to see how. Can you give an example?

 

Using an embedded DLL, the application executes code that it itself has supplied and linked. If anything, this should be safer than loading an external DLL from disk and executing code in that.

 

7 hours ago, Fr0sT.Brutal said:

Look at Android which gets more and more restrictions of getting into the system with every new release.

This is not "getting into the system". Everything happens in user mode within the application.

[Fr0sT.Brutal 900]

On 10/6/2023 at 4:19 PM, dummzeuch said:

Microsoft has a track record of staying backwards compatible for a long time

UAC, privileges, WinDir & PF write protection...

On 10/6/2023 at 10:28 PM, Anders Melander said:

I fail to see how. Can you give an example?

 

Using an embedded DLL, the application executes code that it itself has supplied and linked. If anything, this should be safer than loading an external DLL from disk and executing code in that.

F.ex., dynamically generated code to bypass AV signature detection.

 

Guys I'm not going to argue, that was just my guess work. Time will show.

[Anders Melander 1815]

54 minutes ago, Fr0sT.Brutal said:

dynamically generated code to bypass AV signature detection.

You are talking about self-modifying code or simply the ability of a process to alter its own page-level protection. While that is an OS feature that is used to implement embedded DLLs it isn't specific to it. Lots of other application techniques use that feature.

 

I thought you meant that embedded DLLs posed a risk to the applications using them. It seems you mean that an application using embedded DLLs potentially poses a risk to the system as a whole.
By extension of that argument, you could just as well argue that compiled code is unsafe and that all applications must be distributed as source and compiled by the user. But since you are not going to argue, let's not go there 🙂

[Fr0sT.Brutal 900]

On 10/10/2023 at 2:15 PM, Anders Melander said:

You are talking about self-modifying code or simply the ability of a process to alter its own page-level protection

The latter; seems AV software could consider it suspicious.

On 10/10/2023 at 2:15 PM, Anders Melander said:

I thought you meant that embedded DLLs posed a risk to the applications using them

BTW, I'm curious about code signature checking of in-memory DLLs. WinVerifyTrust  seems to not support PE BLOBs. So this could be an attack vector.

Anyway, I'd consider this tech if it were working flawlessly. Alas it isn't.