Fun with Linux server migrations, part 2

Seeing progress of a long-running Linux process

During the server migration mentioned in part 1, i wanted to see what a long-running rsync process was doing.  Because we had done several presyncs of the data before the outage window, there was not a lot of progress for rsync to report; it was simply churning through files checking to see which ones had changed.

The usual tool for this is strace, which shows all system calls made by a process.  You can attach it to a running process with strace -p PID, where PID is the numeric process id.  I ran strace briefly to find out what system calls rsync was making, and found that it calls lstat64 for each file.  But because it had to look through so many files, i couldn't very well run strace -p PID 2>&1 | grep lstat641 because even that was too much data.  (I was connected to the system via my home ADSL connection, and with hundreds of thousands of files to copy, it would never have kept up with the trace output.)

So i started looking around for the right tool to sample the data without overwhelming my slow connection.  I considered writing a quick awk script, but it turns out that it's even easier than that: sed has a built-in function for operating on the Nth line of any input file.  The general form is sed -n 'M~Np' file, which prints every Nth line starting with the Mth.  (In my case, i was reading from a pipe from strace, so there was no file.)  I tried a few different combinations and settled on strace -p PID 2>&1 | grep lstat64 | sed -n '1~10000p', which samples one in every ten thousand files that rsync processes.

I need to do this quite often on running processes, so sed -n 'M~Np' is going straight to my pool room for helpful little Linux recipes.


1. The 2>&1 is necessary because strace sends trace output to standard error rather than standard output.