Ways to corner a stickiness bug

Question

How to determine exactly what a piece of software is doing when it is stuck, unresponsive to user input and not updating its display?

I have tried oprofile, which records what function is executing, but it's not giving me enough clues. It counts everything that happens during the time it's running, when I need to see what's happening only when the specimen program is stuck.

The problem might involve interrupts, waiting on network sockets, timers, a GUI event handler, or who knows what. How to find out as much as possible about what's going on, not just the execution points of each thread?

The soffware of interest runs on Linux, built using gcc, mostly C++ but may involve other languages including interpreted ones e.g. Python.

The particular case of concern now is Firefox, for which I have checked out source. Firefox pauses all input and screen output at random times, frequently, for about 5-10 seconds each time. Even if someone handed me the solution to this particular problem on a silver platter, sure I'll take it but still be asking. If possible, I'd like to learn general techniques that would apply to any software, especially stuff I'm responsible for.

Answer 1

strace will trace out the system calls. This might give some indication of what is blocking on network sockets and so on.