X520-DA2 10Gbps

[Wayne Marsh]

Hello all,

I'm working on getting a NetBSD box working with an X520-DA2 card at 10Gbps.

I've enabled Jumbo Frames up to mtu 8982 and this seemed to give us a gain
of 500Mbps as compared to a 1500 mtu.

The best we can get with this card is 2.9Gbps. using iperf with 8982 mtu
and 2.4Gbps with 1500 mtu.

Any ideas?

Wayne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Wayne Marsh
Hello all,
I'm working on getting a NetBSD box working with an X520-DA2 card at 10Gbps.
I've enabled Jumbo Frames up to mtu 8982 and this seemed to give us a gain
of 500Mbps as compared to a 1500 mtu.
The best we can get with this card is 2.9Gbps. using iperf with 8982 mtu
and 2.4Gbps with 1500 mtu.

What's your test setup look like?

Dave

[Wayne Marsh]

Dell R410 with Intel X520-DA2 -> Force10 switch -> Dell R810 with Intel X520-DA2
R410 has NetBSD 6.0 RC 2
R810 has CentOS Linux 6.4
The Force10 model escapes by but has 48 10Gbps ports + 4x 40Gbps ports
We're trying all benchmarks using iperf.

When I first modified the driver for this card, I enabled Jumbo
Packets on the 410 by replacing the FreeBSD predefined constant with
(1024 * 9), which is what the definition in FreeBSD was. My second
modification to the driver was changing the number of descriptors,
replacing the FreeBSD predefined constant again when setting the
number of descriptors from 1 to 4 - again replacing MJUM9BYTES with
(1024 * 9) - which raised the maximum TCP window size from 128K to
512K. This seems to not affect speed at all.

The Force10 and R810 also have Jumbo Frames set.

Wayne

Post by David Young

Post by Wayne Marsh
Hello all,
I'm working on getting a NetBSD box working with an X520-DA2 card at 10Gbps.
I've enabled Jumbo Frames up to mtu 8982 and this seemed to give us a gain
of 500Mbps as compared to a 1500 mtu.
The best we can get with this card is 2.9Gbps. using iperf with 8982 mtu
and 2.4Gbps with 1500 mtu.

What's your test setup look like?
Dave
--
David Young

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Wayne Marsh
Dell R410 with Intel X520-DA2 -> Force10 switch -> Dell R810 with Intel X520-DA2
R410 has NetBSD 6.0 RC 2
R810 has CentOS Linux 6.4
The Force10 model escapes by but has 48 10Gbps ports + 4x 40Gbps ports
We're trying all benchmarks using iperf.
When I first modified the driver for this card, I enabled Jumbo
Packets on the 410 by replacing the FreeBSD predefined constant with
(1024 * 9), which is what the definition in FreeBSD was. My second
modification to the driver was changing the number of descriptors,
replacing the FreeBSD predefined constant again when setting the
number of descriptors from 1 to 4 - again replacing MJUM9BYTES with
(1024 * 9) - which raised the maximum TCP window size from 128K to
512K. This seems to not affect speed at all.

I don't see the connection between the driver constants that you are
modifying and the TCP window size.

Have you enabled any offload features such as ip4csum? Those make a
difference.

Have you tuned up TCP and the maximum socket buffers? For example,
I use these settings on my *1*-gigabit hosts:

net.inet.tcp.sendspace = 262144
net.inet.tcp.recvspace = 262144
kern.sbmax = 16777216

How have you invoked iperf on each end? Typically you have to tune it
up, too.

Dave