network queue lengths

[David Young]

I am concerned that the lengthy Tx queues/rings in NetBSD lead
to lengthy delays and unfairness in typical home/office gateway
applications. What do you think?

NetBSD's network queues and Tx DMA rings seem rather long. For example,
tlp(4) reserves 1024 Tx descriptors, enough for 64 packets, each with at
most 16 DMA segments. 256 segments is the default maximum length both
for network queues (e.g., ipintrq, ip6intrq) and for interface queues.

If a NetBSD box is the Internet gateway for several 10-, 100-, or
1000-megabit clients, and the clients share a 1-, 2-, or 3-megabit
Internet pipe, it is easy for some outbound stream to fill both the
Tx ring (max 64 packets) and the output queues (max 256 packets) to
capacity with full-size (Ethernet MTU) packets. Once the ring + queue
capacity is reached, every additional packet of outbound traffic that
the LAN offers will linger in the gateway between 1.3 and 3.8 seconds.

Now, suppose that we shorten the interface queue, or else we "shape"
traffic using ALTQ. Outbound traffic nevertheless spends 1/4 to 3/4
second on the Tx ring, which may defeat ALTQ prioritization in some
instances.

This is getting a bit long, so I am going to hastily draw some
conclusions. Please tell me if I am way off base:

1 in order for ALTQ to be really effective at controlling latency for
delay-sensitive traffic, it has to feed a very short Tx ring

2 maximum queue/ring lengths in NetBSD are tuned for very high-speed
networks, now; the maximums should adapt to hold down the expected delay
while absorbing momentary overflows.

Dave

[Thor Lancelot Simon]

Post by David Young
This is getting a bit long, so I am going to hastily draw some

Let me say up front that I don't much care about ALTQ, and I care
more about not harming performance on machines with very fast networks
on both sides (or their only "side") than about reducing latency on
machines with dramatically mismatched networks.

Post by David Young
1 in order for ALTQ to be really effective at controlling latency for
delay-sensitive traffic, it has to feed a very short Tx ring

So, this one I don't much care about, personally.

Post by David Young
2 maximum queue/ring lengths in NetBSD are tuned for very high-speed
networks, now; the maximums should adapt to hold down the expected delay
while absorbing momentary overflows.

Can we trust the "baudrate" reported by an interface? If we can, it should
be possible to algorithmically tune this, but knowledge of the system's
network topology and characteristic packet flows -- which can probably
be inferred at runtime -- may be required in order to get it right.

Constant, parametric tuning like "grow the queue until latency hits _X_"
seems highly likely to interact in very negative ways with TCP.

[Michael Richardson]

Post by David Young
This is getting a bit long, so I am going to hastily draw some

Thor> Let me say up front that I don't much care about ALTQ, and I
Thor> care more about not harming performance on machines with very
Thor> fast networks on both sides (or their only "side") than about
Thor> reducing latency on machines with dramatically mismatched
Thor> networks.

I understand your point of view.
I think this is a question about, if there was a tunable, what would
the default be, and which group of people possess the smarts to tune it
the "other" way.
One important thing might be to treat packets forwarded differently
from those locally generated. This inheritely distinguishes machines
with only one "side"...

Post by David Young
2 maximum queue/ring lengths in NetBSD are tuned for very
high-speed networks, now; the maximums should adapt to hold down
the expected delay while absorbing momentary overflows.

Thor> Can we trust the "baudrate" reported by an interface? If we
Thor> can, it should be possible to algorithmically tune this, but
Thor> knowledge of the system's network topology and characteristic
Thor> packet flows -- which can probably be inferred at runtime --
Thor> may be required in order to get it right.

Thor> Constant, parametric tuning like "grow the queue until latency
Thor> hits _X_" seems highly likely to interact in very negative
Thor> ways with TCP.

As long as the measurement of the latency was done over a period of
many minutes to hours, and was not permitted to change too quickly, I
think that the effects of this would not be important compared to
congestion in the network itself.
The periods involved would ideally be randomized such that a network
of machines with this behaviour do not get into some kind of cycle as a
group. now... this is where the simulations become important.
--
] Y'avait une poule de jammé dans l'muffler!!!!!!!!! | firewalls [
] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[
] ***@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael Richardson]

David> If a NetBSD box is the Internet gateway for several 10-,
David> 100-, or 1000-megabit clients, and the clients share a 1-,
David> 2-, or 3-megabit Internet pipe, it is easy for some outbound
David> stream to fill both the Tx ring (max 64 packets) and the

Hmm. maybe this explains the problems on my home network.... almost
full /25 of (mostly virtual) hosts behind a 4M/800K pipe.

David> Now, suppose that we shorten the interface queue, or else we
David> "shape" traffic using ALTQ. Outbound traffic nevertheless
David> spends 1/4 to 3/4 second on the Tx ring, which may defeat
David> ALTQ prioritization in some instances.

David> This is getting a bit long, so I am going to hastily draw
David> some conclusions. Please tell me if I am way off base:

David> 1 in order for ALTQ to be really effective at controlling
David> latency for delay-sensitive traffic, it has to feed a very
David> short Tx ring

Yes.
Can we do this dynamically in some way? Does WRED help here?

David> 2 maximum queue/ring lengths in NetBSD are tuned for very
David> high-speed networks, now; the maximums should adapt to hold
David> down the expected delay while absorbing momentary overflows.

I would be happy to tell ifconfig some value, since it's not the link
speed that counts...

[Mihai Chelaru]

Hi,

Post by David Young
I am concerned that the lengthy Tx queues/rings in NetBSD lead
to lengthy delays and unfairness in typical home/office gateway
applications. What do you think?

Uff, I was hoping for a moment that you want to write another packet
scheduler to replace the ancient ALTQ that looks mostly unmaintained :)

Post by David Young
If a NetBSD box is the Internet gateway for several 10-, 100-, or
1000-megabit clients, and the clients share a 1-, 2-, or 3-megabit
Internet pipe, it is easy for some outbound stream to fill both the
Tx ring (max 64 packets) and the output queues (max 256 packets) to
capacity with full-size (Ethernet MTU) packets. Once the ring + queue
capacity is reached, every additional packet of outbound traffic that
the LAN offers will linger in the gateway between 1.3 and 3.8 seconds.
Now, suppose that we shorten the interface queue, or else we "shape"
traffic using ALTQ. Outbound traffic nevertheless spends 1/4 to 3/4
second on the Tx ring, which may defeat ALTQ prioritization in some
instances.

It's not very clear to me how did you get this numbers. Transmitting 320
full-sized frames shouldn't take more than 50ms over 100Mbit full-duplex
ethernet link. Are you talking about some other in-kernel delays ?

Post by David Young
This is getting a bit long, so I am going to hastily draw some
1 in order for ALTQ to be really effective at controlling latency for
delay-sensitive traffic, it has to feed a very short Tx ring

64 frames sounds reasonable to me. That's a maximum jitter of about
10ms. Cisco had a default of 40 iirc for output queue. But yes, it's a
good idea to have a global sysctl used to decrease this value if we use
a gateway in order to transport packets for jitter-sensitive
applications. I'm saying global because it's the easiest way to achieve
this and because I'm thinking about MMoIP that's mostly bidirectional.

Post by David Young
2 maximum queue/ring lengths in NetBSD are tuned for very high-speed
networks, now; the maximums should adapt to hold down the expected delay
while absorbing momentary overflows.

No idea for the moment how altq is interacting with ipintrq but a packet
scheduler should be able to re-order it.

[Michael Richardson]

Post by David Young
If a NetBSD box is the Internet gateway for several 10-, 100-, or
1000-megabit clients, and the clients share a 1-, 2-, or
3-megabit Internet pipe, it is easy for some outbound stream to
fill both the Tx ring (max 64 packets) and the output queues (max
256 packets) to capacity with full-size (Ethernet MTU) packets.
Once the ring + queue capacity is reached, every additional
packet of outbound traffic that the LAN offers will linger in the
gateway between 1.3 and 3.8 seconds. Now, suppose that we
shorten the interface queue, or else we "shape" traffic using
ALTQ. Outbound traffic nevertheless spends 1/4 to 3/4 second on
the Tx ring, which may defeat ALTQ prioritization in some
instances.

Mihai> It's not very clear to me how did you get this
Mihai> numbers. Transmitting 320 full-sized frames shouldn't take
Mihai> more than 50ms over 100Mbit full-duplex ethernet link. Are
Mihai> you talking about some other in-kernel delays ?

Imagine you are connected to a 1Mb/s DSL router.
Imagine for a moment that you are connected using GbE, and the DSL
router sends XOFF frames. (I don't think XOFF is permitted at 100Mb/s, but
maybe some boxes do this)

Why have a router with GbE ports connected to 1Mb/s?
Because it's the router between parts of your network, as well as the
uplink. (My NetBSD edge router/firewall has 5 10/100 ports, two of them
are uplinks, 3 connect to three networks at different trust levels, and
I'm aiming to replace it because I want better throughput between
networks)

On plain 10Mb/s links, I would think that you'd just send too many
packets at the DSL router, and it would drop them.
--
] Y'avait une poule de jammé dans l'muffler!!!!!!!!! | firewalls [
] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[
] ***@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Mihai Chelaru
Hi,

Post by David Young
I am concerned that the lengthy Tx queues/rings in NetBSD lead
to lengthy delays and unfairness in typical home/office gateway
applications. What do you think?

Uff, I was hoping for a moment that you want to write another packet
scheduler to replace the ancient ALTQ that looks mostly unmaintained :)

Post by David Young
If a NetBSD box is the Internet gateway for several 10-, 100-, or
1000-megabit clients, and the clients share a 1-, 2-, or 3-megabit
Internet pipe, it is easy for some outbound stream to fill both the
Tx ring (max 64 packets) and the output queues (max 256 packets) to
capacity with full-size (Ethernet MTU) packets. Once the ring + queue
capacity is reached, every additional packet of outbound traffic that
the LAN offers will linger in the gateway between 1.3 and 3.8 seconds.
Now, suppose that we shorten the interface queue, or else we "shape"
traffic using ALTQ. Outbound traffic nevertheless spends 1/4 to 3/4
second on the Tx ring, which may defeat ALTQ prioritization in some
instances.

It's not very clear to me how did you get this numbers. Transmitting 320
full-sized frames shouldn't take more than 50ms over 100Mbit full-duplex
ethernet link. Are you talking about some other in-kernel delays ?

The situation that I am talking about is a LAN "funnelled" into a narrow
WAN pipe. The WAN pipe is 1-, 2-, or 3-megabit; the LAN is 100- or
1000-megabit.

Dave

[Mihai Chelaru]

Post by David Young
The situation that I am talking about is a LAN "funnelled" into a narrow
WAN pipe. The WAN pipe is 1-, 2-, or 3-megabit; the LAN is 100- or
1000-megabit.

I see your point now. ALTQ will not guarantee priority classes in this
setup and combining it with ethernet flow control would be truly disastrous.

I guess the final solution is very dependent on the spot where ALTQ is
doing the classification and admission control. But I think that
shrinking queues is not the definitive solution but the posibility to
insert priority frames/packets into them.

[Rafal Boni]

Post by Michael Richardson

Post by David Young
This is getting a bit long, so I am going to hastily draw some

Thor> Let me say up front that I don't much care about ALTQ, and I
Thor> care more about not harming performance on machines with very
Thor> fast networks on both sides (or their only "side") than about
Thor> reducing latency on machines with dramatically mismatched
Thor> networks.
I understand your point of view.
I think this is a question about, if there was a tunable, what would
the default be, and which group of people possess the smarts to tune it
the "other" way.
One important thing might be to treat packets forwarded differently
from those locally generated. This inheritely distinguishes machines
with only one "side"...

I think this is an interesting concept and might dovetail nicely with use
of multiple rings on the card for different purposes. OTOH, it might still
have a negative effect on routers which straddle two fat / fast networks,
so it would likely still need to be tunable. It also isn't likely to fix
it for the single-dumb-ring-NIC use case.

--rafal