Pass each line of stdout as stdin to a new invocation of a tool

Question

I would like to pipe each line of stdout into stdin of a separate invocation of another tool - essentially mapping over lines, in bash on the command line.

I know I can loop, but that's a bit verbose for such a simple task, and also slow:

cat out.ndjson | while read p; do \
echo "$p" | jj key
done

I know I could write a performant program maplines with such an API myself:

cat ndjson | maplines jj key

where maplines reads a line from stdin and pipes it to a new invocation of jj key for each line.

It's important that jj key cannot accept the line as an argument. It needs the input as stdin.

It would be wasteful to reinvent the wheel if there's a tool already. Or one could write a bash function. Surely something like that exists already.

Basically my question is: how do I implement maplines in xargs or parallel. I researched quite a bit but xargs seems to be unable to pass things as stdout.

cat > out.ndjson <<- EOM
{"key": "line1"}
{"key": "line2"}
{"key": "line3"}
EOM

jj is available with brew install tidwall/jj/jj, as binaries or from source. While it shouldn't really matter what the command is, it's useful to test whether an answer works.

Answer 1

The main reason it's slow is that creating new processes is computationally expensive. Since you want to run a new instance of the tool for each line, some of that slowness is inevitable. But it doesn't need to be as slow as your current version, because echo "$p" | jj key creates two processes per line, one to do the echo, and one for jj. In bash, you can replace the echo with a here-string, and similarly replace the cat | with a simple input redirection:

while read p; do
    jj key <<<"$p"
done <out.ndjson

This should cut the overhead (the time it takes above what the jj tool itself takes) in half, meaning it'll run at up to double the speed of the original.