Why can't I disable MTU discovery on Windows socket? IP_PMTUDISC_DONT fails to set

Question

I'm trying to build uftp on Windows. It's a UDP transfer client/server.

I was eventually able to make static binaries with OpenSSL linked in. However I'm still having a problem with one bit of code. It fails to set the IP_PMTUDISC_DONT flag on the socket.

When I run the code Windows errors:

> uftpd.exe -D C:\xfertest\ -p 1044 -d -T c:\tmp

Error disabling MTU discovery: (10042) An unknown, invalid, or unsupported option or level was specified in a getsockopt or setsockopt call.
Error leaving multicast group: (10038) An operation was attempted on something that is not a socket.

This is the code that is trying to set the flag on the socket in client_init.c:

#ifdef IP_MTU_DISCOVER
{
    int mtuflag = IP_PMTUDISC_DONT;
    if (setsockopt(listener, IPPROTO_IP, IP_MTU_DISCOVER, &mtuflag, sizeof(mtuflag)) == SOCKET_ERROR) {
        sockerror(0, 0, 0, "Error disabling MTU discovery");
        closesocket(listener);
        exit(ERR_SOCKET);
    }
}
#endif

This is built on Windows Server 2016 with VS2017 community edition.

I have tried testing builds that don't include this flag, but they seem to have far worse performance (tons of NAKs), which maybe isn't a surprise if this causes fragmentation.

Can anyone think of why I can't setsockopt() on this? The listener is indeed a socket as created in the same file by listener = socket(family, SOCK_DGRAM, 0). I see other feedback on the internet claiming that some of these APIs may have changed through time. I will see if I can build on earlier VS and OS versions, but I am hoping someone who has hit this before can point out a clear path to a solution faster than I can.

Answer 1

Support for IP_MTU_DISCOVER simply didn't exist yet in Windows Server 2016.

IP_MTU_DISCOVER was implemented in WSL (Windows Subsystem for Linux) for UDP sockets in Windows 10 build 15002 (see WSL issues #69, #170, #717, and #720 on GitHub):

https://learn.microsoft.com/en-us/windows/wsl/release-notes#build-15002

Implemented IP_MTU_DISCOVER INET socket option (GH #720, 717, 170, 69)

IP_MTU_DISCOVER was implemented in WSL for TCP sockets in Windows 10 build 16215 (see WSL issues #1639, #2115, and #2205 on GitHub):

https://learn.microsoft.com/en-us/windows/wsl/release-notes#build-16215

Added support for IP_MTU_DISCOVER for TCP sockets. [GH 1639, 2115, 2205]

Just because the preprocessor has #define'd a value for IP_MTU_DISCOVER doesn't guarantee the target OS that your code ends up running on actually supports IP_MTU_DISCOVER. You need to run the code and ignore the WSAENOPROTOOPT error from setsockopt() if it occurs, eg:

#ifdef IP_MTU_DISCOVER
{
    int mtuflag = IP_PMTUDISC_DONT;
    if (setsockopt(listener, IPPROTO_IP, IP_MTU_DISCOVER, &mtuflag, sizeof(mtuflag)) == SOCKET_ERROR) {
        if (WSAGetLastError() != WSAENOPROTOOPT) {
            sockerror(0, 0, 0, "Error disabling MTU discovery");
            closesocket(listener);
            exit(ERR_SOCKET);
        }
        // IP_MTU_DISCOVER is NOT supported on this OS!
        // Do something else, or just move on...
    }
}
#endif