How can I display rsync progress from a log file without the carriage returns?

Question

I have a log file from rsync command with progress on. When running this progress updates the display information on the same line. When I capture the output from this command I get a file that displays normally with cat on a terminal (all the backspaces and re-edits are replayed) but I want to be able to use grep on the file and process it so I see all the backspace edit commands. How can I process the file to remove all the progress updates and just get a file that has the final edits?

Answer 1

If you want to see what your captured file looks like without the progress lines overwriting the previous ones:

sed 's/\r/\n/g' capture-file

which might yield something like this:

created directory /some/dest/dir
filename

       32768   0%    0.00kB/s    0:00:00
     6717440  21%    6.38MB/s    0:00:03
    13139968  41%    6.25MB/s    0:00:02
    19791872  62%    6.28MB/s    0:00:01
    26214400  82%    6.24MB/s    0:00:00
    31784420 100%    6.25MB/s    0:00:04 (xfer#1, to-check=0/1)

sent 31788388 bytes  received 31 bytes  3346149.37 bytes/sec
total size is 31784420  speedup is 1.00

If you want to see just the final step of the progress message and eliminate the previous ones:

sed 's/.*\r/\n/g' capture-file

Which might look like this:

created directory /some/dest/dir
filename

    31784420 100%    6.25MB/s    0:00:04 (xfer#1, to-check=0/1)

sent 31788388 bytes  received 31 bytes  3346149.37 bytes/sec
total size is 31784420  speedup is 1.00

You can run rsync with the --log-file=name option to capture log information in a file. Replace "name" with the name that you want. You can control the information that gets logged using the log-file-format option (see the log format section in man rsyncd.conf for details).

On my system the default rsync log file looks like this:

2009/11/01 17:19:20 [23802] building file list
2009/11/01 17:19:20 [23802] created directory /some/dest/dir
2009/11/01 17:19:25 [23802] <f+++++++++ filename
2009/11/01 17:19:25 [23802] sent 31788388 bytes  received 31 bytes  3346149.37 bytes/sec
2009/11/01 17:19:25 [23802] total size is 31784420  speedup is 1.00