Multithreading curses output in Python

Question

I am trying to implement a simple spinner (using code adapted from this answer) below a progress bar for a long-running function.

[########         ] x%
/ Compressing filename

I have the compression and progress bar running in the main thread of my script and the spinner running in another thread, so it can actually spin while compression takes place. However, I am using curses for both the progress bar and the spinner, and both use curses.refresh()

Sometimes the terminal will randomly output gibberish, and I'm not sure why. I think it is due to the multi-threaded nature of the spinner, as when I disable the spinner the problem goes away.

Here is the pseudocode of the spinner:

def start(self):
  self.busy = True
  global stdscr 
  stdscr = curses.initscr()
  curses.noecho()
  curses.cbreak()
  threading.Thread(target=self.spinner_task).start()

def spinner_task(self):
  while self.busy:
    stdscr.addstr(1, 0, next(self.spinner_generator))
    time.sleep(self.delay)
    stdscr.refresh()

And here is the pseudocode for the progress bar:

progress_bar = "\r[{}] {:.0f}%".format("#" * block + " " * (bar_length - block), round(progress * 100, 0))
progress_file = " {} {}".format(s, filename)
stdscr.clrtoeol()
stdscr.addstr(1, 1, "                                                              ")
stdscr.clrtoeol()
stdscr.addstr(0, 0, progress_bar)
stdscr.addstr(1, 1, progress_file)
stdscr.refresh()

And called from main() like:

spinner.start()
for each file:
  update_progress_bar
  compress(file)
spinner.stop()

Why would the output sometimes become corrupted? Is it because of the separate threads? If so, any suggestions on a better way to design this?

Answer 1

The curses libraries that Python's curses module relies on are not threadsafe.

ncurses has a curs_threads feature, which has apparently been there since 5.7 about a decade ago. But it requires changing the way you do a few API calls, and linking against -lncursest, and it's still not trivial, and… almost nobody ever uses it.

As far as I know, no standard installer or distro package ever builds Python curses to link ncursest—even if the distro includes ncursest in the first place, which they often won't. And even if they did, there are no bindings for the threadsafe functions, so you still wouldn't be able to safely access things like setting the tabsize.

---

In my (possibly out-of-date, and possibly platform-limited) experience, you can nevertheless get away with things, but you need to:

• Obviously only one thread can ever call stuff like getch and getmouse.
• Add a global Lock, then make sure every batch of updates ends with a refresh, and the whole batch is inside the Lock.
• Avoid the Python wrappers around the functionality mentioned in curs_threads—e.g., don't change the escdelay or the tabsize.
• Init (and close) the screen from the main thread, before starting (after exiting) the other threads.
• If at all possible, make sure you also create all of the windows you need in the main thread. (Hopefully you didn't want any dynamic popup subwindows or anything…)

---

But the safe way to do this is to do the same kind of thing you do with tkinter or other GUI libraries that don't understand threads. It's not identical, but the idea is similar. The simplest version is:

• Move your main thread's work to another background thread.
• Add a queue.Queue so that your background threads can ask for curses commands to be run. (You don't need anything complicated to represent a "command", it's just a (func, *args) tuple, because Python.)
• Make the main thread loop around popping commands off that queue and calling them.

If your background threads need to call functions that return a value, obviously you need to make this slightly more complicated. You can look at how multiprocessing.dummy.AsyncResult and concurrent.futures.Future work. Or you can even steal Future for your own purposes. But you probably don't need anything as complicated as either.

If you're looping around input, you'll probably also want your main thread to do that (this means picking a "frame rate" and alternating between waiting on the queue and the input, with a timeout) and dispatch it, even if you're always dispatching to the same thread.

You could even write an mtTkinter-style wrapper that reproduces the curses interface (or even monkeypatches the curses module) but replaces each function with a call to put the function and args on a queue. But I'm not sure this would be worth the effort.

Answer 2

If this is the only place where you're using the curses module, the best solution will be to stop using it.

The only functionality of curses that you're really using here is its ability to clear the screen and move the cursor. This can easily be replicated by outputting the appropriate control sequences directly, e.g:

sys.stdout.write("\x1b[f\x1b[J" + progress_bar + "\n" + progress_file)

The \x1b[f sequence moves the cursor to 1,1, and \x1b[J clears all content from the cursor position to the end of the screen.

No additional calls are needed to refresh the screen, or to reset it when you're done. You can output "\x1b[f\x1b[J" again to clear the screen if you want.

This approach does, admittedly, assume that the user is using a VT100-compatible terminal. However, terminals which do not implement this standard are effectively extinct, so this is probably a safe assumption.