In c++'s SetWindowsHookEx function

Question

I'm Korean student. Now this is my first question after signing up.

DWORD getProcessId() {
    PROCESSENTRY32 process_infor;
    process_infor.dwSize = sizeof(PROCESSENTRY32);

    HANDLE snap_handle = CreateToolhelp32Snapshot(
        TH32CS_SNAPALL, //스냅 단계
        NULL //스냅할 pid
    );

    if (snap_handle != INVALID_HANDLE_VALUE) {
        Process32First(snap_handle, &process_infor);

        do {
            wchar_t* temp = process_infor.szExeFile;
            wstring ws(temp);
            string name(ws.begin(), ws.end());

            if (name == "notepad.exe") {
                cout << name << " : " << process_infor.th32ProcessID << endl;
                return process_infor.th32ProcessID;
            }

        } while (Process32Next(snap_handle, &process_infor));
    }

    CloseHandle(snap_handle);
    return FALSE;
}

BOOL inject() {
    HMODULE dll_handle;
    HOOKPROC func;
    HHOOK process_hook;

    dll_handle = LoadLibrary(L"hello.dll");
    func = (HOOKPROC) GetProcAddress(dll_handle, "injectSuccess");

    cout << "handle : " << dll_handle << endl;
    cout << "pid : " << getProcessId() << endl;

    process_hook = SetWindowsHookEx(
        WH_KEYBOARD,
        func,
        dll_handle,
        getProcessId()
    );

    cout << "pook : " << process_hook << endl;
    cout << "err : " << GetLastError() << endl;
    FreeLibrary(dll_handle);

    return FALSE;
}

There seems to be a problem with SetWindowsHookEx of the inject function in this case. The dll file loads well, and the injectSuccess function inside is well fetched. (I tried running it, but it worked) And I wondered if the argument values ​​of SetWindowsHookEx were entered incorrectly, so I compared them and checked them continuously, but I couldn't find any difference. So, I tried GetLastError() with the return value of SetWindowsHookEx below, but the return value is 0 and the error code is 87 ("the parameter is incorrect").

So I searched, but I can't speak English well and I'm a beginner, so I'm not sure.

Answer 1

According to the SetWindowsHookExW:

dwThreadId

A handle to the DLL containing the hook procedure pointed to by the lpfn parameter. The hMod parameter must be set to NULL if the dwThreadId parameter specifies a thread created by the current process and if the hook procedure is within the code associated with the current process.

So what SetWindowsHookExW needs is the thread ID, and you pass in the process ID of notepad.exe, so the parameter is wrong.

I creat a sample and test the following code:

BOOL inject() {
HMODULE dll_handle;
HOOKPROC func;
HHOOK process_hook;

dll_handle = LoadLibrary(L"hello.dll");
if (dll_handle) func = (HOOKPROC)GetProcAddress(dll_handle, "injectSuccess");
else return FALSE;
cout << "handle : " << dll_handle << endl;
cout << "pid : " << getProcessId() << endl;
HWND h = FindWindow(L"notepad", NULL);
DWORD pid;
threadID = GetWindowThreadProcessId(h, NULL);
cout << "threadID = " << threadID  << endl;
process_hook = SetWindowsHookEx(
    WH_KEYBOARD,
    func,
    dll_handle,
    threadID
);

cout << "pook : " << process_hook << endl;
cout << "err : " << GetLastError() << endl;
if(dll_handle) FreeLibrary(dll_handle);
return FALSE;
}

This example worked for me, and you can see the difference between pid and threadID:

Edit

According to the document:

An application installs the hook procedure by specifying the WH_KEYBOARD hook type and a pointer to the hook procedure in a call to the SetWindowsHookEx function. This hook may be called in the context of the thread that installed it. The call is made by sending a message to the thread that installed the hook. Therefore, the thread that installed the hook must have a message loop. So if you want to run this func, you need to add a message loop.

You can refer to the following code:

int main()
{
    inject();
    MSG msg;
    while (GetMessageW(&msg, NULL, 0, 0))
    {
        TranslateMessage(&msg);
        DispatchMessageW(&msg);
    }
}

When you press the button, the message box will pop up:

Because there are two messages for each key press and release, func is triggered twice each time.If you only want to trigger every time a button is pressed, you can modify the following code:

if ((0x80000000 & lParam) == 0)//This means that when the key is pressed
{
    MessageBox(NULL, L"Success (dll injection)", L"Window", MB_OK);
}

You can refer to KeyboardProc to view the messages for each value in lParam.