wget capture stdout to variable

Question

I have a shell script that I want to put into a chronjob, and it is supposed to fetch data every day and then write a quick log entry. I would like the log entries to look something like:

2015-12-07 11:16:34: No such file ‘CatchUpPlusActivity_20150801’.
2015-12-07 11:24:20: (14,4 MB/s) - ‘CatchUpPlusActivity_20150801.csv.gz’ saved [52690279]
2015-12-07 11:28:40: File ‘CatchUpPlusActivity_20150801.csv.gz’ already there; not retrieving.

My shell script looks like this:

sPath="ftp://user:[email protected]/public/download/exports/webactivity/PullVODData/"
sFiles="CatchUpPlusActivity_20150801*"

sOut=$(sudo wget ${sPath}${sFiles} -nc)

now="$(date +"%Y-%m-%d %H:%M:%S")"
echo ${now}: ${sOut}  >> /path/to/logs/catchup_plus.log

Where I would like sOut to read the final message from each ftp. However, I'd be happy for now if it just returned the last line of the whole message. As it stands sOut is empty

Answer 1

The [progress] output of wget goes to stderr, so change your command:

OLD: sOut=$(sudo wget ${sPath}${sFiles} -nc)
NEW: sOut=$(sudo wget ${sPath}${sFiles} -nc 2>&1)

This will capture all lines into a single scalar. So, if you want to look at individual lines, you'll need to split sOut into an array variable or some such.

UPDATE:

To answer your followup question, 2>&1 is a shell thing.

In terms of C-like file descriptor numbers, 0=stdin, 1=stdout, 2=stderr. You could have done wget ... 2>wget.log and the file wget.log would have the same output as the command I gave you.

For any command, "foo" (e.g.) you can divert the output of its stdout with foo 1>X 2>Y and X and/or Y can be a file or another stream descriptor [the &Z syntax].

Note foo 1>out is identical to foo >out because "1" is the default stream. The foo 2>&1 >out says divert stream 2 to the same place as stream 1, which here is out. Note that foo >out 2>&1 also works

This also works with appending. foo 2>&1 >>out.

Your sOut syntax said capture the output of stdout [stream 1] for a command. That is, sOut=$(foo). When you do sOut=$(foo 2>&1) you're telling the subshell you spawn to divert it's stderr output to stdout.

The above is just my cursory explanation. There's a more polished one in the bash manpage.