Parallel processing in GNU/Linux command line

Question

Some GNU/Linux command line programs (such "oggenc", to encode audio in ogg vorbis format) use only 1 of all the cores from your CPU. The problem is that, in my case, I have got 4 cores and that programs just use 1 core, making work the CPU at 25% (just one of the cores is working at 100%, but the other 3 are at 0% for that task).

It could be possible to force the using of more cores for a command line program?

It would be great with some kind of tasks like zip, tar, oggenc, etc.

PS: I have found a program called "parallel", but I'm not achieving how to make it work properly... :(

Answer 1

With GNU Parallel you can do:

parallel gzip ::: *
parallel opusenc {} {.}.opus ::: *.wav

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

Installation

For security reasons you should install GNU Parallel with your package manager, but if GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel