GSoC 2011 project proposal [packet timestamping]

[Dmitry Cherkassov]

Hello everybody!

I am a last year IT student from Saint-Petersburg, Russia.

I want to post my proposal here before pushing it to google-melange,
to receive useful feedback.

Any comments and remarks are welcome & anticipated =)

Just for reference here is my brief cv (html): http://sandycat.info/cv.html

-------------

Google Summer Of Code 2011 project proposal.

"Socket option to timestamp UDP packets in NetBSD kernel"

Rationale

It is often needed for applications that receive data via UDP to store
data
together with its reception time, or account packets arrival time as
precisely
as possible.

Typical use cases include:

    1. Data reproduction in real-time for performance analysis or
quality control;

    2. More precise clock synchronization in NTP/PTP protocol
implementation;

    3. Measuring latency in 10 GB networks (or faster);

    4. Improved precision timing in high frequency trading done by
hedge funds;

    5. Realtime audio/video streaming (RTP/RTCP)

Previously the only way to get the reception time was to call
gettimeofday(2)/clock_gettime(2) upon data reception.

That is not a very good idea because:

    1. It makes one additional system call thereby decreasing maximum
reception rate.

    2. Timing is imprecise, because delay from the time of packet
reception by
       network adapter and time of actual reception by user-land is
non-deterministic
       and isn’t taken into consideration.

Right now NetBSD provides some basic packet timestamping which
needs to be extended and implemented in NIC drivers.

Project objectives

This project’s aim is the further implementation of timestamping in
NetBSD kernel.
The project objectives are:

    1. Implement SO_TIMESTAMPNS option and SO_TIMESTAMPING_* control
messages;

    2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
drivers;

    3. Design and implement an API that [On having enough time]:
        a. selects which clock is used;
        b. provides information about available clocks, their
precision and the position
        where the sampling is performed.

    4. Provide some test programs and documentation.

--------------

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Hubert Feyrer]

Post by Dmitry Cherkassov
Right now NetBSD provides some basic packet timestamping which
needs to be extended and implemented in NIC drivers.

This is probably a stupid question, but:
why in the NIC drivers, and not in the MI UDP/IP code?


- Hubert

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Matt Thomas]

Post by Hubert Feyrer

Post by Dmitry Cherkassov
Right now NetBSD provides some basic packet timestamping which
needs to be extended and implemented in NIC drivers.

why in the NIC drivers, and not in the MI UDP/IP code?

Because that's after netisr has run which can greatly delay
the timestamp.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dmitry Cherkassov]

Great thanks for your attention & time!

I devised the following project deliverables to source tree:

1. Set of patches to src/sys/[kern,netinet]/
2. [Probably] tstamp_clk.c source file to src/sys/kern (?)
(timestamping clock API)

So my questions are:

1. Where (to what files in src/sys) should these changes go?

2. Also, what else should be mentioned in proposal?
(The proposal itself is here:
http://socghop.appspot.com/gsoc/proposal/review/google/gsoc2011/dcherkassov/1
)

3. Is the following short description of proposal is clear?

It is often needed for applications that receive data via UDP to store data together
with its reception time, or account packets arrival time as precisely as possible.
Right now NetBSD provides some basic packet timestamping which needs
to be extended and implemented in NIC drivers.
The idea if this project is to implement timestamping as a capability (like a jumbogram).
If network driver doesn't support it timestamping will be done in if_input (with a small loss of accuracy).

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joerg Sonnenberger]

Post by Dmitry Cherkassov
    2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
drivers;

What speaks against having the timestamping done as part of the normal
ifq operations?

Joerg

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Matt Thomas]

Post by Joerg Sonnenberger

Post by Dmitry Cherkassov
2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
drivers;

What speaks against having the timestamping done as part of the normal
ifq operations?

I think timestamp should be a capability (like jumbogram). If the
driver doesn't support it, it could be generically done in if_input
but with a small loss of accuracy.

Also, what happens on cpus/systems with out a highly accurate clock?

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joerg Sonnenberger]

Post by Matt Thomas
Also, what happens on cpus/systems with out a highly accurate clock?

Worst case is microtime precision or tick resolution. If this is a
kernel option, overhead can be completely skipped if the feature is not
of interest.

Joerg

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Frank Kardel]

Post by Matt Thomas

Post by Joerg Sonnenberger

Post by Dmitry Cherkassov
2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
drivers;

What speaks against having the timestamping done as part of the normal
ifq operations?

I think timestamp should be a capability (like jumbogram). If the
driver doesn't support it, it could be generically done in if_input
but with a small loss of accuracy.
Also, what happens on cpus/systems with out a highly accurate clock?

I agree with the notion that it should be a capability.
The PTP capable interfaces do the time stamping at PHY
level using a local counter of the interface and that
should be transparently supported.

For the ntpd interleave protocol Dave Mills called for
timestamping the start and the end of specific packets.
So this should be controllable/supportable too.
Also modern interfaces use interrupt moderation, thus
software only time stamping will not work too well with
these interfaces.

Frank

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Frank Kardel

Post by Matt Thomas

Post by Joerg Sonnenberger

Post by Dmitry Cherkassov
2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
drivers;

What speaks against having the timestamping done as part of the normal
ifq operations?

I think timestamp should be a capability (like jumbogram). If the
driver doesn't support it, it could be generically done in if_input
but with a small loss of accuracy.
Also, what happens on cpus/systems with out a highly accurate clock?

I agree with the notion that it should be a capability.
The PTP capable interfaces do the time stamping at PHY
level using a local counter of the interface and that
should be transparently supported.

Yes, I've seen those interfaces (though the one's I've seen maintained
the counter in the MAC chip rather than the PHY hardware).

I think the existence of those NICs suggests some arguments about what might
be a good way to implement this functionality. While I have some reservations
about the current system clock code, how it works is basically sound. It
assumes the basic time source is some sort of hardware counter that increments
at a fixed frequency (in the worst case it is a variable in memory which gets
incremented at every clock interrupt). When you want to know what time it is you
sample that counter, then do some math to turn it into a time of day.

When using one of those timestamping NICs you end up doing even more math. The
NIC's counter is normally inappropriate to use for the system clock (it normally
takes a PCI transaction to sample it, and that is way too expensive to use to
find out the time) and, if you have more than one NIC, all those hardware clocks
will be different anyway. The best way to deal with this is to run a sampling
process for each NIC which maintains calibration constants between each NIC's
counter and the counter being used for the system clock, so to convert a NIC
counter to a time of day you first need to do the math to convert that particular
NIC's counter to system count ticks, then do the math to convert system clock ticks
into time of day. If the NIC counter tick-stamp is approximately free then most
of the work of getting a time-of-day stamp is doing that math (as it is when your
system clock tick-stamp is cheap to sample).

This suggests that what should be carried in from the interface with packets is
the rawest form of tick-stamp in whatever form it is available (i.e. system clock
tick-stamps if done in software, or NIC-specific tick-stamps if done in hardware).
Only when it finally arrives at a socket whose listener actually cares to have a
time-of-day stamp should it go to the trouble of doing the math (and figuring out what
math needs to be done in the case of NIC tick-stamps) to produce the time of day. Any
tick-stamps which aren't wanted by the socket the packet arrived at can be fed (in raw
form) to the pool /dev/random feeds from, providing a significant improvement in that
as well.

This would require that the current clock code separate the taking of a tick-stamp
from the math to turn it into a time of day, and that it be capable of turning a
tick-stamp taken "some time ago" (for a reasonable definition of that) into the
same time of day that it would have arrived at had it done the conversion immediately.

Post by Frank Kardel
For the ntpd interleave protocol Dave Mills called for
timestamping the start and the end of specific packets.
So this should be controllable/supportable too.
Also modern interfaces use interrupt moderation, thus
software only time stamping will not work too well with
these interfaces.

Certainly, though it is safe to say that in the absence of NIC
timestamp support taking tick-stamps in the driver software is
still much superior to the alternative of taking them in the user-space
application.

Dennis Ferguson

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Frank Kardel]

Post by Dennis Ferguson

Post by Frank Kardel

Post by Matt Thomas

Post by Joerg Sonnenberger

Post by Dmitry Cherkassov
2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
drivers;

What speaks against having the timestamping done as part of the normal
ifq operations?

I think timestamp should be a capability (like jumbogram). If the
driver doesn't support it, it could be generically done in if_input
but with a small loss of accuracy.
Also, what happens on cpus/systems with out a highly accurate clock?

I agree with the notion that it should be a capability.
The PTP capable interfaces do the time stamping at PHY
level using a local counter of the interface and that
should be transparently supported.

....
This suggests that what should be carried in from the interface with packets is
the rawest form of tick-stamp in whatever form it is available (i.e. system clock
tick-stamps if done in software, or NIC-specific tick-stamps if done in hardware).
Only when it finally arrives at a socket whose listener actually cares to have a
time-of-day stamp should it go to the trouble of doing the math (and figuring out what
math needs to be done in the case of NIC tick-stamps) to produce the time of day. Any
tick-stamps which aren't wanted by the socket the packet arrived at can be fed (in raw
form) to the pool /dev/random feeds from, providing a significant improvement in that
as well.
This would require that the current clock code separate the taking of a tick-stamp
from the math to turn it into a time of day, and that it be capable of turning a
tick-stamp taken "some time ago" (for a reasonable definition of that) into the
same time of day that it would have arrived at had it done the conversion immediately.

Post by Frank Kardel
For the ntpd interleave protocol Dave Mills called for
timestamping the start and the end of specific packets.
So this should be controllable/supportable too.
Also modern interfaces use interrupt moderation, thus
software only time stamping will not work too well with
these interfaces.

Certainly, though it is safe to say that in the absence of NIC
timestamp support taking tick-stamps in the driver software is
still much superior to the alternative of taking them in the user-space
application.
Dennis Ferguson

I agree very much with what you say performancewise. Actually the
timecounter
code is not too far from what you want. tc_delta() is the ticks passed since
the last windup/synchronisation. What we need to maintain is the
relationship
between the cycle delta, the related time scale and the windup effects.
Those counters usually wrap after a short time the seldom last much
longer than
a second - so when picking a cycle counter we may have to copy some state in
order to recreate the time scale long after the cycle time stamp was taken.
Also we need to take into account the effects of the frequency
correction (NTP)
that may be in effect over the time. It may thus not be too easy to recreate
the time scale if the "some time ago" gets too long. How would we handle
that
before the tick stamp becomes worthless?

Frank

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Matthew Mondor]

On Mon, 28 Mar 2011 10:52:04 -0700

Post by Matt Thomas
I think timestamp should be a capability (like jumbogram). If the
driver doesn't support it, it could be generically done in if_input
but with a small loss of accuracy.

I like this idea,

[David Young]

Post by Hubert Feyrer

Post by Dmitry Cherkassov
Right now NetBSD provides some basic packet timestamping which
needs to be extended and implemented in NIC drivers.

why in the NIC drivers, and not in the MI UDP/IP code?

There are NICs that supply stamp received packets with the rx time at
microsecond (or higher) resolution. For example, some WLAN NICs supply
microsecond rx times from the network clock that sychronizes 802.11
beacon transmission and power-saving sleep/wake. It seems that the
Intel 82599 10-gigabit ethernet can capture both packet tx and rx times,
and at higher than microsecond resolution.

Sometimes the NIC documentation or relevant standards describe precisely
when/where in the rx/tx chain the rx/tx time is latched, which helps for
using the timestamps for scientific/esoteric purposes.

Dave

[Dmitry Cherkassov]

Great thanks for your attention & time!

I devised the following project deliverables to source tree:

1. Set of patches to src/sys/[kern,netinet]/
2. [Probably] tstamp_clk.c source file to src/sys/kern (?)
(timestamping clock API)

So my questions are:

1. Where (to what files in src/sys) should these changes go?

2. Also, what else should be mentioned in proposal?
(The proposal itself is here:
http://socghop.appspot.com/gsoc/proposal/review/google/gsoc2011/dcher...
)

3. Is the following short description of proposal is clear?

It is often needed for applications that receive data via UDP to store data together
with its reception time, or account packets arrival time as precisely as possible.
Right now NetBSD provides some basic packet timestamping which needs
to be extended and implemented in NIC drivers.
The idea if this project is to implement timestamping as a capability (like a jumbogram).
If network driver doesn't support it timestamping will be done in if_input (with a small loss of accuracy).

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de