Win32 api call via C# fails!

Question

I have a C++ function exported as api like this:

#define WIN322_API __declspec(dllexport)
WIN322_API char* Test(LPSTR str);
WIN322_API char* Test(LPSTR str)
{
 return "hello";
}

the function is exported as API correctly by the .DEF file, cause i can see it in Dependency Walker tool. Now i have a C# tester program:

[DllImport("c:\\win322.dll")]

public static extern string Test([MarshalAs(UnmanagedType.LPStr)] String str);

private void Form1_Load(object sender, EventArgs e)
        {
string _str = "0221";
Test(_str); // runtime error here!

}

on calling the Test() method i get the error:

"A call to PInvoke function 'MyClient!MyClient.Form1::Test' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature."

i tried many other data types and marshalings, but got nothing! plz help me!

Answer 1

It is caused by a mismatch on the calling convention, the default for [DllImport] is Stdcall but the C compiler's default is Cdecl. Use the CallingConvention property in the declaration.

That's not the only problem though, this code will crash on Vista and Win7. Returning a string from a C function is quite troublesome, there's a memory management problem. It isn't clear who is responsible for freeing the string buffer. You are returning a literal now but that's going to stop being useful pretty soon. Next stop is using malloc() for the return string with the intent for the caller to call free(). That's not going to work, the pinvoke marshaller cannot call it since it doesn't know what heap the C code is using.

It will call Marshal.FreeCoTaskMem(). That's wrong, the string wasn't allocated by CoTaskMemAlloc(). That goes unnoticed on XP and earlier, other than the very hard to diagnose memory leak this causes. And goes kaboom on Vista and Win7, they have a much more stricter memory manager.

You need to rewrite the C function like this:

extern "C" __declspec(dllexport) 
void __stdcall Test(const char* input, char* output, int outLen);

Now the caller supplies the buffer, through the output argument, there's no longer a guess who owns the memory. You use StringBuilder in the C# declaration.

[DllImport("foo.dll")]
private static extern void Test(string input, StringBuilder output, int outLen);
...
    var sb = new StringBuilder(666);
    test("bar", sb, sb.Capacity);
    string result = sb.ToString();

Be careful to use the outLen argument in your C code so that you can be sure not to overflow the buffer. That corrupts the garbage collected heap, crashing the app with a Fatal Execution Engine Error.