Nifi 1.8.0 - Thread termination in nifi while using fetchHDFS processor

Question

I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.

When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.

Just wanted to know the working of the terminate option.

1. Does it gracefully shut all the connections with the FS?
2. Since all the threads are killed, do I lose out on the data that was consumed by these threads?
3. Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?

Answer 1

When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.

The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.

In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.