How to get socket Tx timestamps from python socket.sendmsg

Question

While reading here, here and here I see that on Linux systems you can request timestamps from received and transmitted packets by setting socket options. I am currently able to use SO_TIMESTAMPNS and SO_TIMESTAMPING to get Rx timestamps with recvmsg. With sendmsg I don't know how to properly package the ancillary data for the control message and sendmsg doesn't even have a return that could hold the data I need.

This is how I'm getting the Rx timestamp.

import socket    

SO_TIMESTAMPING = 37
SOF_TIMESTAMPING = 127

server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_sock.setsockopt(socket.SOL_SOCKET, SO_TIMESTAMPING, SO_TIMESTAMPING, SOF_TIMESTAMPING)

server_sock.bind((_svr_ip, _svr_port))
server_sock.listen(_num_clients)
conn, client_addr = server_sock.accept()

msg, ancdata, flags, address = conn.recvmsg(64, 1024)

t_rx = ancdata[0][2]
t_rx_s = int.from_bytes(t_rec[0:7], byteorder="little")
t_rx_ns = int.from_bytes(t_rec[8:], byteorder="little")

Section 1.3.4 mentions enabling timestamps via control messages without having to use socket.setsockopt to reserve overhead. I believe that means you are adding the flag to the control message while calling sendmsg instead of using socket.setsockopt.

Answer 1

As you have seen, there is no field in sendmsg() that would allow return of a timestamp. This wouldn't be practical anyway. sendmsg() only queues the packet into the kernel socket buffer. The tx timestamp is only generated when the packet is actually sent. In the case of hardware timestamps, it is actually generated by the NIC when the packet goes out on the wire.

Instead, what happens for TX timestamps is that the timestamp is added to the socket "error queue". The errors queue is a channel for ancillary data that is asynchronous. This could be a network error, but it's also used for timestamps even though they aren't errors.

You read from a sockets error queue by calling recvmsg with the flag MSG_ERRQUEUE set. This call will not return any actual socket data. Instead, it returns a copy of the transmitted packet, along with anciallary data fields containing the timestamp. You can suppress including the copy of the transmitted packet by using the SOF_TIMESTAMPING_OPT_TSONLY option when you do setsockopt.

If I modify your example to use tx timestamps

import socket    

SO_TIMESTAMPING = 37
SOF_TIMESTAMPING = 127

server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_sock.setsockopt(socket.SOL_SOCKET, SO_TIMESTAMPING, SOF_TIMESTAMPING)

server_sock.bind((_svr_ip, _svr_port))
server_sock.listen(_num_clients)
conn, client_addr = server_sock.accept()

conn.send(b"hello, world")
msg, ancdata, flags, address = conn.recvmsg(64, 1024, socket.MSG_ERRQUEUE)

for layer, opt, data in ancdata:
    if layer == socket.SOL_SOCKET and opt == sol.SO_TIMESTAMPING:
        t_send = data
        t_tx_s = int.from_bytes(t_send[0:7], byteorder="little")
        t_tx_ns = int.from_bytes(t_send[8:], byteorder="little")
        print(f"packet sent time: {t_tx_s}.{t_tx_ns:09}")

It is possible to change the timestamp options for each call to send by using ancillary data. It should look something like this:

SOF_TIMESTAMPING_TX_SOFTWARE = 2
opt_bytes = SOF_TIMESTAMPING_TX_SOFTWARE.to_bytes(4, 'little')
conn.sendmsg([b'hello, world'], [(socket.SOL_SOCKET, SO_TIMESTAMPING, opt_bytes)])
msg, ancdata, flags, address = conn.recvmsg(64, 1024, socket.MSG_ERRQUEUE)

As I understand, you need the timestamping generally enabled with setsockopt, but you can omit the SOF_TIMESTAMPING_TX_SOFTWARE bit, and then apply that with ancillary data as show.

Finally, all of the above is sufficient for software timestamps. Hardware timestamps need NIC support and need to be enabled in the driver system wide. ethtool -T eth0 can show you the supported modes while the hwstamp_ctl utility will allow you to configure the system.