IPreviewHandler throws uncatchable exception

Question

I've imported the COM interface IPreviewHandler into a WinForms app and am using it to display previews for various types of documents (I look up the GUID of the appropriate preview handler in the registry, then use Activator.CreateInstance(guid) to instantiate the specific COM class.

This works wonderfully for the vast majority of file types - Office formats, PDFs, videos, etc - however, after I instantiate the "Microsoft Windows TXT Preview Handler" {1531d583-8375-4d3f-b5fb-d23bbd169f22}, initialise it with a stream containing an ordinary .txt file, set the bounds of the preview window and then finally call DoPreview(), I get an exception that cannot be caught using try...catch:

try {
    Type comType = Type.GetTypeFromCLSID(guid);
    object handler = Activator.CreateInstance(comType);

    if (handler is IInitializeWithStream) {
        Stream s = File.Open(filename, FileMode.Open);
        // this just passes the System.IO.Stream as the COM type IStream
        ((IInitializeWithStream)handler).Initialize(new StreamWrapper(s), 0);
    }
    else {
        throw new NotSupportedException();
    }

    RECT r = new RECT();
    r.Top = 0;
    r.Left = 0;
    r.Right = hostControl.Width;
    r.Bottom = hostControl.Height;

    ((IPreviewHandler)handler).SetWindow(hostControl.Handle, ref r);
    ((IPreviewHandler)handler).DoPreview();    // <-- crash occurs here
}
catch (Exception) {
    // this will never execute
}

When I step through with the debugger, the Visual Studio Hosting Process crashes. With no debugger, the application crashes without firing the AppDomain.UnHandledException or Application.ThreadException events.

I don't really mind that I can't preview plain text files using this technique (the working preview handlers for Office formats, etc are sufficient for my app's requirements), but I am concerned about having my app crash uncontrollably should the user select a .txt file. Is there any way I can catch this error and handle it gracefully? Better yet, is there some way I can overcome it and get the handler to work?

Answer 1

I couldn't get the GetPreviewHandlerGUID() to recognize a .txt file and had to inject the GUID directly. You can see what goes wrong when you use Project + Properties, Debug, tick Enable unmanaged code debugging.

The debugger will now stop on the problem and display

`STATUS_STACK_BUFFER_OVERRUN encountered

The top of the call stack looks like this:

kernel32.dll!_UnhandledExceptionFilter@4()  + 0x1a368 bytes 
shell32.dll!___report_gsfailure()  + 0xc8 bytes 
shell32.dll!CRTFPreviewHandler::_StreamInCallback()  + 0x74 bytes   
msftedit.dll!CLightDTEngine::ReadPlainText()  + 0xed bytes  
msftedit.dll!CLightDTEngine::LoadFromEs()  + 0x202b3 bytes  
msftedit.dll!CTxtEdit::TxSendMessage()  + 0x1e25f bytes 
msftedit.dll!_RichEditWndProc@16()  + 0x13d bytes   

The problem is located in the StreamInCallback() function. It is called by the RichTextBox that's used to display the preview (msftedit.dll) to load the file. The code in this callback function has a bug, it destroys the 'canary' that's used to detect that the stack frame got corrupted because of a buffer overrun.

This is part of the counter-measures that Microsoft took to prevent viruses from injecting themselves by buffer overruns. The /GS compile option in Visual Studio for the C/C++ languages. Once detected, the CRT very swiftly terminates the program. This happens without an exception getting raised, the stack cannot safely be unwound because it was compromised. Accordingly, the CLR cannot catch the exception.

This bug is specific to the TXT file viewer. There's nothing you can do about it other than not using it. Reporting this bug to connect.microsoft.com probably isn't useful, they'll close it as 'external'. This is otherwise a subtle hint what can happen when you let unmanaged code run inside your program ;)