What happens when I double click an executable, technically

Question

tl;dr

I'm trying to understand the difference of running a program directly via double clicking the executable vs running it through a terminal or programatically via CreateProcess in windows 10.

---

long version

Because this is my problem, executing an old game (circa 2003 using D3D8) through double clicking in windows 10 works okay. Executing the game through an seconday executable (also circa 2003) using CreateProcess seems to sometimes work okay.

But executing it through my new golang executable never works. I get a very tiny screen instead. So I want to understand what's the technical difference.

For reference, my golang code goes like: (tiny screen)

cmd := exec.Command(filepath.Join(".", "Game.exe"))
err := cmd.Start()

Disassembling the secondary executable gives me this: (normal screen)

CPU Disasm
Address   Hex dump          Command                                  Comments
004043AF  |.  51            PUSH ECX                                 ; /pProcessInformation = 59C7F521 -> {hProcess=???,hThread=???,ProcessID=???,ThreadID=???}
004043B0  |.  52            PUSH EDX                                 ; |pStartupInfo => OFFSET LOCAL.16
004043B1  |.  68 28E54000   PUSH OFFSET 0040E528                     ; |CurrentDirectory = "."
004043B6  |.  50            PUSH EAX                                 ; |pEnvironment => NULL
004043B7  |.  50            PUSH EAX                                 ; |CreationFlags => 0
004043B8  |.  6A 01         PUSH 1                                   ; |InheritHandles = TRUE
004043BA  |.  50            PUSH EAX                                 ; |pThreadSecurity => NULL
004043BB  |.  50            PUSH EAX                                 ; |pProcessSecurity => NULL
004043BC  |.  68 10E54000   PUSH OFFSET 0040E510                     ; |CommandLine = "Game.exe"
004043C1  |.  50            PUSH EAX                                 ; |ApplicationName => NULL
004043C2  |.  FF15 8CB04000 CALL DWORD PTR DS:[<&KERNEL32.CreateProc ; \KERNEL32.CreateProcessA

When I say the game is a tiny screen it shows up like this:

versus this when it is executed directly with double click: (don't mind it being black its just the normal startup)

---

Additional info: Actually the problem only exists in windows 10. Windows 7 is completely fine.

For windows 10, the only way to make it normal screen is to use this setting:

When that is used, I get the normal screen when using double click or the secondary executable. But its still a tiny screen on my Golang app.

Answer 1

The Windows-API function ShellExecute with "open" as second argument opens a file as if double-clicked. Example:

#include <windows.h>

int main(int, char**)
{
        ShellExecute(NULL, "open", "C:/Windows/System32/notepad.exe", NULL, NULL, SW_SHOWNORMAL);
        return 0;
}

See https://learn.microsoft.com/en-us/windows/win32/api/shellapi/nf-shellapi-shellexecutew

Answer 2

What actually happens when you double-click a file is a red herring but since you specifically asked about that I will answer (TLDR CreateProcess).

When you double-click a file in Explorer, it will

1. Use its IShellFolder instance and PIDL representing the file to call IShellFolder::GetUIObjectOf to get an IContextMenu instance.
2. Call IContextMenu::InvokeCommand on the default item ("Open" in your case).
3. IContextMenu::InvokeCommand will effectively call ShellExecuteEx (without SEE_MASK_INVOKEIDLIST set).
4. ShellExecuteEx will end up calling CreateProcess for files ending in .exe.

Any application can replicate this by calling ShellExecuteEx with the SEE_MASK_INVOKEIDLIST flag. It will then do the IContextMenu dance for you. For .exe files, this is unnecessary and the simpler ShellExecute can be used:

ShellExecuteW(NULL, L"open", L"c:\folder\file.exe", L"/switches /if /you /want", L"c:\folder", SW_SHOW);

The one exception is if the exe file needs UAC elevation (requested in its manifest or you set it in the compatibility tab). Then ShellExecute[Ex] will ask a Windows service to elevate on its behalf instead of calling CreateProcess. The service will call CreateProcessAsUser which is effectively CreateProcess with a different user token.

• When using cmd.exe, executing foo.exe will call CreateProcess and start foo.exe will call ShellExecuteEx.

• exec.go calls os.StartProcess which in turn calls CreateProcess.

---

Without knowing more about the game, it is hard to say if the problem is a bug in the game or a change in Direct 3D.

If you want to force compatibility on for the child process launched by Go, you can append the compatibility environment variable:

cmd := exec.Command(filepath.Join(".", "Game.exe"))
cmd.Env = append(os.Environ(), "__COMPAT_LAYER=16BitColor")
err := cmd.Start()

Also, using "." (the current directory) here is less than ideal. If your Go application is in the same directory as "Game.exe" then you should use the absolute path of your Go application (minus the file name) to join with "Game.exe". You should also set cmd.Dir to this directory.

You could also debug the game process and see which size it passes to USER32::CreateWindowExW.