IDA kernel debugging Error

Question

Environment:

Windows 7 64-bit version

Virtual machine (Vmware 8) + Virtualkd: Windows XP 32-bit version

Trying to debug the kernl with ida 6.1, I get this error:

Windbg: using debugging tools from 'C:\Program Files (x86)\Debugging Tools for Windows (x86)\'
Connecting to the kernel with 'com:port=\\.\pipe\kd_WinXP_kerneldbg'
**Connect failed: Incorrect function.**
Flushing buffers, please wait...ok
Unloading IDP module C:\ida61\procs\pc.w32...

vmmon64 looks ok:

http://s13.postimg.org/qpn50bcdz/Untitled.png

Connection string on IDA:

com:port=\\.\pipe\kd_WinXP_kerneldbg

anyone please

Answer 1

The connection string is you've used is when you have a physical serial port connection. For a virtual machine, you need to use a slightly different string. Try:

com:pipe,port=\\.\pipe\kd_WinXP_kerneldbg

Notice the extra pipe,port section required for virtual machine COM connections. Here are the possible connection strings, referenced from Live Kernel-Mode Debugging Using WinDbg

windbg [-y SymbolPath] -k net:port=PortNumber,key=Key
windbg [-y SymbolPath] -k 1394:channel=1394Channel[,symlink=1394Protocol] 
windbg [-y SymbolPath] -k usb:targetname=USBString
windbg [-y SymbolPath] -k com:port=ComPort,baud=BaudRate
windbg [-y SymbolPath] -k com:pipe,port=\\VMHost\pipe\PipeName[,resets=0][,reconnect]
windbg [-y SymbolPath] -k com:modem
windbg [-y SymbolPath] -kl 
windbg [-y SymbolPath] -k 

In the case where the VM is running on the same machine as the debugger use \\.\ rather than \\VMHost\

Setting Up Kernel-Mode Debugging of a Virtual Machine Manually may also be helpful in general.