How do I implement a `DllMain`?

Development Windows
1

Or, alternately, how do I get the hInstance so I can create windows and dialogs?

I tried the naive

@MainActor
@_cdecl("DllMain")
public func DllMain(hModule: HINSTANCE, reasonForCall: DWORD, lpReserved: LPVOID) -> WindowsBool {
	hInstance = hModule
	return true
}

The program loading the DLL basically vanished on load. I assume the calling convention is wrong, and it causes the stack to go boom. However, any details about using stdcall in Swift were unclear or missing.

Would really appreciate any workaround, although would prefer a "correct" way to do things.

2

A DllMain implementation is only needed if you are building a DLL and need to change the entry behaviour (i.e. thread construction, destruction, library load or unload). A default implementation is provided by the C runtime for your DLL.

You cannot mark the entry point as @MainActor - this cannot be a main actor thread.

Unfortunately, stdcall is not supported by Swift yet, but it shouldn't matter for non-x86 targets (the C interop is not stable there due to that). For x64 and ARM64 targets __stdcall is ignored. That said, if this were a stdcall function, the name itself would be decorated and most likely be something like __DllMain@12.

Without additional information on what occurred, it is difficult to give any further suggestions. I'd recommend that you run your application under a debugger to determine what is happening.

If you are looking to get the hInstance only, GetCurrentModule(nil) would give you the HMODULE which should be the same as HINSTANCE with modern windows (post-3.11). For a better explanation for working with Windows UI elements, I'd suggest taking a look at GitHub - compnerd/swift-win32: A Windows application framework for Swift

3 Likes
3

If the program only has one thread, it's the main actor thread. At the very least, as far as I could tell, it's the only way to use globals in Swift 6, including constants. If there's a way to tell Swift "this doesn't need to be Sendable because it never changes" or "This only changes once at init before any other method can execute", I'd be glad to use that instead. What I'd definitely not want to do, though, is add locks or atomics to a library that runs single threaded.

You have a point about the decoration, although @12 would be interpreted as the 12th ordinal in a DLL. If it's not stack thrashing causing the crash, though, I have a trouble thinking what it might be. Swift-specific initializers not being executed?

I did look at swift-win32. It's good guidance on how to use callbacks across the C boundary. The use of SetWindowSubclass instead of the usual SetWindowLongPtrW is an interesting touch, although I don't think it would work for dialogs where the dialog proc needs to be supplied on creation.

Speaking of dialogs, there's no demonstration on how to use resources in there. I had to figure (read: implement) that one on my own, including several workarounds for SPM issues.

There's no such thing as GetCurrentModule, but I'm going to try with GetModuleHandleW(nil). I still can't get the dialog to show up, but I'll try to play with it until something happens. It's a tad hard to debug a DLL, but I'll figure something out.

Do you have any lightweight user-friendly debugger you would recommend for such a use-case?

4

I’m actually not sure this statement holds if the program exits its original thread. Besides, a DLL cannot enforce or mandate this condition. And if the process you are loaded in creates a new thread, your DllMain will be called again on that thread, with a reason of DLL_THREAD_ATTACH.

In any event, you shouldn’t be doing any real work in DllMain.

1 Like
5

No, that is not an ordinal specification, but rather part of the name decoration scheme for the stdcall CC (12-bytes on the stack). This is important as the parameters are caller popped.

It absolutely could be stack corruption, but it is difficult to say without any additional details. But, it could also be argument corruption due to the CC mismatch.

You're right, I did mean GetModuleHandleW(nil).

Well, it depends - you should be able to set an unresolved breakpoint on the module entry point, which should trigger upon the module load. Something like bu SampleModule.dll!DllMain might work with WinDBG.

I don't know what exactly would be "lightweight" here. I tend to switch between WinDBG, WinDBGX, LLDB, and CDB as appropriate. I think that WinDBG would be the tool I would reach for here.

6

That’s going to return the HMODULE of the process, not of @SlugFiller’s DLL. The right construction here is to use GetModuleHandleEx() with the GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS flag and some address within the module.

7

Right, but the HINSTANCE for the application should be fro the current process, not the DLL (at least for the UI elements). If you need it to be the DLL itself, withUnsafePointer(to: DllMain) { GetModuleHandleExW(GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT | GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS, $0.baseAddress, &hModule) } or something along those lines would be best. The default behaviour of GetModuleHandleExW is to increment the refcount of the module which would be a leak (or you would need to FreeLibrary the resulting module handle).

8

I would expect the DLL to be loading its own dialog resources rather than the ones bundled in the hosting process.

9

Oh, if the intent is to load Win32 resources (e.g. FindResource), then, yes, you would want the DLL handle and not the process handle.

10

The language doesn’t know anything about the semantics of DllMain relative to the rest of the code in your DLL (or the process), and it would be difficult to even model that since it will be called multiple times as I mentioned above.

You could probably do something along the lines of withUnsafeMutablePointer(to: &myGlobalHInst) { $0.pointee = hInstDLL }.

11

I replaced the use of @MainActor with nonisolated(unsafe) since it seems cleaner.

Using GetModuleHandleExW now. It took jumping through some hoops, since the first argument needs to be cast to DWORD (apparently the constants are interpreted as signed int instead of unsigned), the second needs to be cast to UnsafePointer<WCHAR> (it won't implicitly cast from UnsafeRawPointer), and the 3rd parameter I'm basically hoping I got right.

Still not getting the dialog to open. I need to figure out where it's failing. Maybe I can use message boxes to compensate for the lack of console access from the DLL.

I did notice that for some reason the parent process is failing to terminate, and has to be force closed. I thought it might have been the @MainActor having some unexpected side effect, but it's still happening with nonisolated(unsafe), no DllMain, and even with no attempt to get the module handle or open the dialog.

P.S. Speaking of DLLs, is there anyway to make Swift not export every function or class method in the program, including mangled version of the cdecl exports? I tried creating and using a .def file, with no effect. It's not critical, but would be nice to have.

12

Anything marked as public is exported. Simply make the interface internal (or any other access level) to hide it. Currently, the ABI is tied to the visibility of the interface, which can be a bit of a challenge for package level sharing.

13

From my understanding, internal is the default access, and doesn't need to be explicitly used. Even if it is explicitly used on a class, the related methods are still present in the DLL's exports.

Being able to remove class methods from the export would make things a bit cleaner, but would still leave the mangled clones of the exported methods (they can't be made private), and some exports called (blahblah)Foundation6Bundle(yadayada)module(stuff) that don't seem to correspond with any declared variable or function.

1 Like
14

withUnsafePointer(to: DllMain) can't possibly produce the right pointer, can it?

15

I have a different global variable I'm using. However, there's a caveat. Internally, whether Swift produces the variable's initial value in the code, or creates code that lazily creates the variable's value on first get, is down to the optimization level and the compiler's mood on that day. As a result, there's a possibility of the related pointer actually pointing at the heap, in which case I'm not sure if Windows would recognize it as belonging to the same module.

It would be much easier if I could just use DllMain as intended instead of trying workarounds. But for that I need to figure out why the direct method isn't working.

Actually... I tried it again with nonisolated(unsafe) instead of @MainActor, plus checking if (reasonForCall == DLL_PROCESS_ATTACH), and now it doesn't crash anymore. The dialog still isn't opening, but this feels like progress.

16

It actually should - we don't need a mutable pointer so no need for the inout overload. There is a DllMain per DLL, so that should be a local code address. The address is exported, but even if it were not, it should be within the code segment for the module so it should work just fine.

17

I would expect it to produce a temporary pointer to storage containing a copy of DllMain per Swift's worldview (i.e. a pointer to a function pointer stored on the stack.) Which would then likely be optimized in -O builds, but still the wrong pointer?

18

How could withUnsafePointer(to: DllMain) validly return a pointer to a pointer to DllMain? They don’t even have the same type.

It’s theoretically possible that it would return a pointer to a temporary copy of DllMain’s body, but that’s not actually going to happen, especially on any architecture with W^X.

19

That's… what withUnsafePointer(to:) does: it yields a (generally temporary) pointer to the value passed to it that is valid for the lifetime of the closure that was also passed to it.

I'm not sure what "they" refers to here.

The type of DllMain in Swift is something like @convention(c) (HINSTANCE, DWORD, LPVOID) -> BOOL (ignoring __stdcall which we can't currently represent in Swift.) The value of DllMain, in Swift, is the address of the first instruction in the function, i.e. it's a function pointer. Copying DllMain simply means making another scalar value with the same bit pattern (at least on architectures we generally deal with—technically, function pointers need not be scalar, but that's not interesting right now.)

I think some of the confusion here stems from the fact that, in C and C++, the following two expressions are equivalent:

auto p1 = &DllMain;
auto p2 = DllMain;

The original syntax in C for getting a pointer to a function was p1, and p2 was simply invalid or nonsensical. C++ and later C compilers like GCC/clang accept p2 as equivalent to p1. In other words, &someFunctionName and someFunctionName are equivalent in C.

Swift doesn't let you form a pointer or reference to an immutable value (such as a declared function) using &, but in general withUnsafePointer(to:) and & serve the same purpose. Except that for function types, they don't behave the same way because Swift doesn't have the same magic syntactic sugar that C/C++ have for function pointers.

When you write withUnsafePointer(to: DllMain) { p in ... }, the value of p is some address that points to DllMain, but DllMain is, to Swift, the address of the function's first instruction, not the body of the function itself. So p is a pointer to the function pointer.

@compnerd and I chatted off-forum briefly and confirmed that he's using it something like:

let untypedFunctionPointer = withUnsafePointer(to: DllMain) { p in
  p.withMemoryRebound(to: UnsafeRawPointer.self) { p2 in
    p2.pointee
  }
}

Which is a long way of writing unsafeBitCast(DllMain, to: UnsafeRawPointer.self). :slightly_smiling_face:

2 Likes
20

withUnsafePointer<T>(to: T) returns an UnsafePointer<T>. If we simplify the type of DllMain to () -> Void, we get UnsafePointer<() -> Void>. You are asserting that this function would instead return UnsafePointer<UnsafePointer<() -> Void>>, aka a pointer to a pointer to a function.