Debugging a Multi-Threaded server

Question

I was asked this on an interview and now I'm curious because I don't think interviewer was satisfied with my answer. Here's the question:

A Multi-threaded server application stops working and the last log message from the application is:

"Some Server Related Message..."

Code looks like:

CalledFunc ()
{
    Code ...

    Acquiring Thread lock
    Line printing "Some Server Related Message..."
    Func();
    Releasing Thread Lock
}

1. What should the programmer in charge do to debug this?
2. What has happened wrong in the Func() ?
3. If an exception is thrown in the Func() what should be done to fix problem ?

Answer 1

Reason #1: it’s a database problem. This may sound strange, but the main reason an application server hangs is not directly related to the application server itself. The location of the symptom is rarely the location of the root cause. The following scenario is quite common:

The database is bottlenecked, causing queries to run slower than usual. Requests that used to take 1 second, now take 5 seconds to complete. The average number of concurrent requests slowly increases (due to backlog). The server runs out of threads and the application server hangs. If you manage to get a thread dump, you’ll just see a bunch of threads waiting and another group that’s actually running. Another possibility is that the number of waiting threads (or queued threads) will gobble up all available memory and, eventually, lead to an OutOfMemory error.

Reason #2: deadlocks. If it seems that the application server is doing nothing, look for deadlocks. These can be database deadlocks that cause your SQL queries to hang, or seek the update statements. For example, a transaction log that is written to the database for each request may easily hang the entire application if the log table is locked. Also check for shared objects—an operating system file that is written to from multiple threads at once.

Reason #3: run-away thread. In cases where the application server is indeed to blame, you should look for a run-away thread. These are hard to detect because they hardly show up on logs since they are usually only written when the request has completed. A run-away thread will probably not return until it has already affected the entire application. Therefore, the hanging request will not be written to the log. These ‘runaway’ threads typically include infinite loops or code that results in consuming too much heap memory resulting in out of memory. For example, a query that should show results that does not include the option of paging between result pages suddenly needs to display a large number of results. The page takes forever to render and clobbers the application server, eventually causing it to hang.