Ethernet adapter clock sync when capturing with tcpdump

Question

Setup and observation

I have a PC equipped with an Intel i350 T2 NIC where I would like to capture on both interfaces simultaneously using tcpdump. Both interfaces are connected to a 100mbit HUB (sic!) which forwards various traffic from an external traffic source to both interfaces at the "same time", so I can measure the difference of the timestamps done by the respective ethernet MACs.

Capturing simultaneously with:

user@rt:~$ sudo tcpdump -j adapter --time-stamp-precision nano -B 1048576 -i eth2 -w test_eth2.pcap
user@rt:~$ sudo tcpdump -j adapter --time-stamp-precision nano -B 1048576 -i eth3 -w test_eth3.pcap

After that I merge the two files together to compare the timestamps:

user@rt:~$ mergecap -F nseclibpcap -w merged.pcap test_eth2.pcap test_eth3.pcap

Wireshark then shows me, that for a few packets I get a timestamp diff of the duplicate frames of around 20-40nsec (which is nice and sufficient for my application!). But there are also lots of frames which show a difference of up to tens of microseconds when comparing the respective duplicates.

Environment

user@rt:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 15.04
Release:        15.04
Codename:       vivid

user@rt:~$ uname -r
3.19.0-28-generic

user@rt:~$ lscpu | grep "Model name"
Model name:            Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz

Questions

1. How/Who does the syncing of the adapter clock to CLOCK_REALTIME (I presume)
2. How often does this syncing occur.
3. Is there some room for tweaking without too much effort?
4. Is it possible to use e.g. phc2sys to sync all adapter clocks to CLOCK_REALTIME?
5. Would (4) interfere with the mechanisms done by (1)?

THX for help or pointing out my mistakes!

Answer 1

From a quick glance at igb_ethtool.c your NIC indeed seems to be capable of hardware timestamping. The jitter you observe (20-40ns) is just in the range of the expected PHY jitter from synchronizing to the Ethernet clock. (For 100Mbit the clock is 25MHz or 40ns.)

So far looking great, thanks to Intel. Not many NICs/drivers have this capability.

Now the bad part: I doubt anything is syncing CLOCK_REALTIME to the NIC adapter clock at the moment. The clocks are probably free-running at a slightly different frequencies. Those oscillators are usually specified at 50ppm, typical drifts will be around 5ppm, which means they will drift apart by ~5us every second, varying with room temperature. Keep that in mind when using the nanosecond precision. If your system uses NTP you may even see NTP drift adjustments happening.

But the good news is that you probably don't need to synchronize them, unless you really want absolute timestamps. The main reason why your NIC supports hardware timestamping at all is probably to support IEEE1588 PTP (precision time protocol). If you need absolute time with sub-microsecond precision, you should look look at this protocol and/or buy a GPS receiver.

If you just need relative timestamps, you could try -j adapter_unsynced instead of -j adapter, or maybe you could try to stop NTP from trying to drift-correct your system clock. If all this fails, you could try to start linuxptp, which may have the capbability to properly sync NIC and system time even if you don't have a PTP network.

And finally... you are using a HUB, which means Ethernet is running in half-duplex mode, which means... collisions. Unless your NIC is absolutely quiet. I guess in theory this shouldn't matter because you observe the same collisions in both NICs, and frames aren't delayed or queued differently depending on which path they take. But since half-duplex is so rare these days it may be that NIC timestamping support wasn't implemented with that in mind. Typical bugs in such implementations are e.g. returning the timestamp of the previous frames instead of the current one.