How to stop a process running in tmux printing thread dumps periodically?

Question

I am running a pyspark shell from a bash session running in tmux, previously we just ran it from bash without a terminal multiplexer but since this runs in a (Jupyter) container, people would lose work (when their login session expired and they logged back in what was on screen would be gone) and I want to use tmux as it's a really powerful tool.

However I have noticed an odd behaviour that I don't understand, but I can reproduce. Every now and then, when the pyspark shell is running inside tmux a thread dump is generated, which fills the screen with hundreds of lines of text, meaning, what was on screen would normally be a long scroll up for the user.

e.g.

>>> 2022-05-10 18:06:43                                                                                              
Full thread dump OpenJDK 64-Bit Server VM (25.332-b08 mixed mode):                                                                   
                                                                                                                                 
"IPC Client (394840392) connection to ip-10-51-112-44/10.51.112.44:8020 from root" #157 daemon prio=5 os_prio=0 tid=0x00007f378802e000 nid=0x693 in Object.wait() [0x00007f38d86f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)                                                                 
        at java.lang.Object.wait(Native Method)                                                                                                 
        at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1034)                                                               
        - locked <0x00007f418c689e48> (a org.apache.hadoop.ipc.Client$Connection)                                                                                               
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1078)                                                                                                               
                                                                                                                                                                       
"SparkUI-156" #156 daemon prio=5 os_prio=0 tid=0x00007f37e0009800 nid=0x676 waiting on condition [0x00007f38ea8bc000]                                                     
   java.lang.Thread.State: TIMED_WAITING (parking)                                                                                                           
        at sun.misc.Unsafe.park(Native Method)                                                                                                                                             
        - parking to wait for  <0x00007f390bb01968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)                                
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)                                                                               
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)                                                      
        at org.sparkproject.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)                                                       
        at org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)  

This continues for around 500 lines or more.

We never had this issue without tmux, and I tested in screen and got a similar behaviour, so I think it's related to the use of a multiplexer.

I am able to generate a similar thread dump if I find the PID of the java process running the spark shell, I can send a kill -3 signal to that PID and I get the same kind of thread dump generated.

I suspect this is an internal mechanism inside the multiplexer to make sure a process is still running, but I don't know for sure.

My question is, is there any way to stop this output from appearing in the terminal window? Can I re-route just this output to /dev/null while keeping the normal interactive session? Can I stop tmux from polling the running application in this way?

Answer 1

If you have access to the runtime command, try adding something like this to the command. It would leave other console output alone (that is, still viewable in the console):

-XX:+LogVMOutput -XX:LogFile=/tmp/jvm-output.log

Another option you asked about – redirecting to /dev/null – is possible, but it would apply to all stdout, including anything you might actually want to see. I would encourage you to find alternatives to this, but it's doable by redirecting stdout ("1") like this (assuming, again, that you have access to the runtime command):

java xyz 1>/dev/null

One final option: redirect stdout to a file (vs. /dev/null) so that you have a chance of seeing other console output if/when the need arises.

java xyz 1>/path/to/file.txt