scp troughput

[Edgar Fuß]

[This may belong to tech-userlevel@]

I just observed that scp throughput between my NetBSD machines stalls at some
20 MB/s. Raw troughput (iperf) is up to 800 Mbit/s. CPUs are nowhere near
saturated. The weird thing it always saturates at nearly the exact same
level independent of hardware and file size.

If I scp to a Debian machine, I get 65 MB/s, CPU bound on the receiving side.
Debian->NetBSD is slow, Debian->Debian is fast.

This reminds me of some Nagle-related problem but scp claims to have set
TCP_NODELAY.

Any hints?

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

Post by Edgar Fuß
This reminds me of some Nagle-related problem but scp claims to have
set TCP_NODELAY.

"Claims to have"? In what sense? Did you ktrace it and see that
happening, or is this based on documentation? If it's just
documentation, I would be skeptical until/unless it's confirmed with
something like ktrace.

You also could snoop the traffic and see if that gives any hints. If
it's a TCP-layer issue, it won't matter that the payload is encrypted.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Edgar Fuß]

Post by Mouse
"Claims to have"? In what sense?

It says something like "TCP_NODELAY set on fd 42" in it's debug output.

Post by Mouse
Did you ktrace it and see that happening

Not yet.

The basic question was whether anyone els has observed this behaviour.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by Edgar Fuß
I just observed that scp throughput between my NetBSD machines stalls at some
20 MB/s. Raw troughput (iperf) is up to 800 Mbit/s. CPUs are nowhere near
saturated. The weird thing it always saturates at nearly the exact same
level independent of hardware and file size.
If I scp to a Debian machine, I get 65 MB/s, CPU bound on the receiving side.
Debian->NetBSD is slow, Debian->Debian is fast.

Check your ssh config about whether you are trying to do compression.
Keep in mind that scp may reuse another connection if you have
ControlMaster set up.

Another thing to look at is socket buffer sizes.

I am guessing you have GbE wired, so you have < 1ms round-trip ping
times; I'm seeing 0.330 ms ping times even with a 2014 box and a 2008
laptop.

Normally, I recommend that people capture tcpdump traces and use the
scripts in the xplot package to visualize the TCP behavior. That's a
little involved and they have not kept up with tcpdump format changes.
xplot-devel is the best best.

I am seeing 12 MB/s myself, and looking into it. On the same link, with
the equivalent of ttcp, I am seeing 69 MB/s (553 Mb/s), which seems
perhaps ok for a 2008 notebook with msk(4).


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joerg Sonnenberger]

Post by Greg Troxel
I am guessing you have GbE wired, so you have < 1ms round-trip ping
times; I'm seeing 0.330 ms ping times even with a 2014 box and a 2008
laptop.

In the context of ssh, ping times are somewhat misleading. The real RTT
for the purpose of finding the bandwidth delay product needs to include
the crypto latency, which can be quite significant.

Joerg

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by Joerg Sonnenberger

Post by Greg Troxel
I am guessing you have GbE wired, so you have < 1ms round-trip ping
times; I'm seeing 0.330 ms ping times even with a 2014 box and a 2008
laptop.

In the context of ssh, ping times are somewhat misleading. The real RTT
for the purpose of finding the bandwidth delay product needs to include
the crypto latency, which can be quite significant.

True. I was just trying to establish if we were talking about a
well-functioning local wired GbE.

In my traces, I am seeing bursts that are likely to have about 16kB of
data (in 12 packets), about 700 uS apart. Which is about 23 MB/s.

So far, it looks like the window sizes are much smaller than I'd expect
given the recvspace/sendspace settings (131072).



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joerg Sonnenberger]

Post by Edgar Fuß
I just observed that scp throughput between my NetBSD machines stalls at some
20 MB/s.

Have you changed the default socket buffer limits?

Joerg

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

I just looked at traces of the TCP connections for ssh and a blaster to
discard, from a fast-cpu sender to an apu2 (a bit slow it turns out) and
an old laptop. All of this is wired Gbe with decent switches (netgear,
ubiquiti, 1 each). My "fast" computer is a 2014 dell with re0, so sort
of good stuff but not current and not necessarily the best interface
chip (but perhaps that's not fair).

My tentative conclusions are:

particularly on the apu2, the tcp window (determined by the
recvbuf/sendbuf) really matters in terms of 131072 vs 32768

on the apu2 with discard, the time it takes for the receiver process
getting the data and updating the window matter a lot. Even with
inetd/discard, the window is kept full and the limiting factor seems
to be userland reading the data.

with the apu2 or 2008 laptop and 131072 buffers, the limiting issue is
the sender. It does not appear to be window constrained.

with the 2008 laptop and blaster/discard, it's really not window
limited, and runs at 550 Mb/s.


all rates are in MB/s, with 131072 buffers

ssh: 20.3 apu2, 18.8 2008, 16.1 self
blaster/discard: 31.7 apu2, 69.3 2008, 571.6 self

So I think there is something going on with ssh, perhaps in using kernel
crypto engine.

This is all done with a local build of netbsd-8. I have in sysctl.conf:

net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.inet6.tcp6.sendspace=131072
net.inet6.tcp6.recvspace=131072

net.inet.tcp.recvbuf_auto=0
net.inet.tcp.sendbuf_auto=0
net.inet6.tcp6.recvbuf_auto=0
net.inet6.tcp6.sendbuf_auto=0

In the past I have become suspicious of the auto buf stuff, but I do not
know if that is fair.


I would definitely recommend bumping up your buffer sizes. But I
think there's more going on.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael van Elst]

Post by Greg Troxel
particularly on the apu2, the tcp window (determined by the
recvbuf/sendbuf) really matters in terms of 131072 vs 32768

With

HPNDisabled yes

in sshd_config, scp speed went from 30MByte/s to 78MByte/s.
iperf3 speed between these machines is ~830Mbit/s.

The HPN patches we use seem to be counterproductive nowadays.

N.B. the buffer autoscaling is usually pretty good but not aggressive
enough for short sessions. You need a few seconds to ramp up the
buffer size. Making the sendbuf_inc/recvbuf_inc values larger helps.

[Greg Troxel]

Post by Michael van Elst
HPNDisabled yes
in sshd_config, scp speed went from 30MByte/s to 78MByte/s.
iperf3 speed between these machines is ~830Mbit/s.
The HPN patches we use seem to be counterproductive nowadays.
N.B. the buffer autoscaling is usually pretty good but not aggressive
enough for short sessions. You need a few seconds to ramp up the
buffer size. Making the sendbuf_inc/recvbuf_inc values larger helps.

In -8, I don't see that in sshd_config(8), and disabling it in
ssh_config didn't change anything.

Do our HPN patches patch the docs?

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael van Elst]

Post by Greg Troxel
Do our HPN patches patch the docs?

I fear no.

You can find documentation here:

https://www.psc.edu/hpn-ssh
https://sourceforge.net/p/hpnssh/wiki/Home/

N.B. We don't use the multi-threaded AES code.


Greetings,

[Anders Magnusson]

There is also talk of adding a NONE cipher back, and I hope we didn't do
that (but it seems highly highly unlikely we would have).

We added support for the NONE cipher. That is still secure in that it
guarantees proper authentication and integrity, just confidentiality
gets lost.

"secure" is a complicated word :-) I see your point about authentication
and integrity, but it breaks the property that if you use ssh then you
are sure there is no way to end up without encryption. (I get it that
with TOFU you aren't sure you are sending data to the right place, but I
see that as somewhat separate.)

That is up to the user IMHO.  And if I use a slow machine then I
probably want to avoid using encryption for bulk transfers, otherwise I
have to go back to rcp instead.

-- R

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

There is also talk of adding a NONE cipher back, and I hope we didn't do
that (but it seems highly highly unlikely we would have).

We added support for the NONE cipher. That is still secure in that it
guarantees proper authentication and integrity, just confidentiality
gets lost.

"secure" is a complicated word :-) I see your point about authentication
and integrity, but it breaks the property that if you use ssh then you
are sure there is no way to end up without encryption. (I get it that
with TOFU you aren't sure you are sending data to the right place, but I
see that as somewhat separate.)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael van Elst]

There is also talk of adding a NONE cipher back, and I hope we didn't do
that (but it seems highly highly unlikely we would have).

We added support for the NONE cipher. That is still secure in that it
guarantees proper authentication and integrity, just confidentiality
gets lost.


Greetings,

[Greg Troxel]

So I guess Someone should report a bug upstream; it seems obvious that
the upstream patch should adjust the example config files and
ssh_config/sshd_config to explain the new options. But perhaps they do
and we didn't apply that part.

(sort of note to self)

/etc/ssh/sshd_config is patched.
sshd_config(5) is NOT patched.

/etc/ssh/ssh_config is NOT patched (no hpndisabled commented out).
ssh_config(5) is NOT patched.

ssh(1) and sshd(8) do not say the HPN patches are applied.

ssh -V does not say the HPN patches are applied.

sshd identifies as "OpenSSH_7.6 NetBSD_Secure_Shell-20171007" not
mentioning HPN patches. Not sure what's right here.

Plus, I just tried again with "hpndisabled yes" on sender and
"HPNDisabled yes" on receiver and got the same speed as before.


Could a few others test, on GbE LAN or better, and report experiences?




--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by Michael van Elst

Post by Greg Troxel
Do our HPN patches patch the docs?

I fear no.

So I guess Someone should report a bug upstream; it seems obvious that
the upstream patch should adjust the example config files and
ssh_config/sshd_config to explain the new options. But perhaps they do
and we didn't apply that part.

Post by Michael van Elst
https://www.psc.edu/hpn-ssh
https://sourceforge.net/p/hpnssh/wiki/Home/
N.B. We don't use the multi-threaded AES code.

Thanks - that looks interesting but a little adventurous, and currently
not our big issue.

Hving read their page, I have no idea why performance would be lower and
particularly all the way down to 20 MB/s.

There is also talk of adding a NONE cipher back, and I hope we didn't do
that (but it seems highly highly unlikely we would have).

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Edgar Fuß]

Post by Joerg Sonnenberger
Have you changed the default socket buffer limits?

Not intentionally.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de