scp performance

[comfooc]

Hi,
few weeks ago i made some test using scp.
Systems were out of the box.
Machine 1 : OpenBSD; P3 500Mhz; 192Mram; 10Mbps;
Machine 2 : tripleboot :OpenBSD, Linux, NetBSD; P3 750Mhz, 512Mram, 100Mbps;

Transfers were:
Machine 1 <=average 800KBps=> Machine 2 : Linux
Machine 1 <=average 550KBps=> Machine 2 : NetBSD
Machine 1 <=average 950KBps=> Machine 2 : OpenBSD

What might caused that low transfer? How to improve it?
Cheers...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Steven M. Bellovin]

On Thu, 17 Jan 2008 18:12:46 +0100

Post by comfooc
Hi,
few weeks ago i made some test using scp.
Systems were out of the box.
Machine 1 : OpenBSD; P3 500Mhz; 192Mram; 10Mbps;
Machine 2 : tripleboot :OpenBSD, Linux, NetBSD; P3 750Mhz, 512Mram, 100Mbps;
Machine 1 <=average 800KBps=> Machine 2 : Linux
Machine 1 <=average 550KBps=> Machine 2 : NetBSD
Machine 1 <=average 950KBps=> Machine 2 : OpenBSD
What might caused that low transfer? How to improve it?
Cheers...

What version of NetBSD? What kind of network link connected the
machines? What ciphers did ssh use between the two machines? All of
these affect the answer...


--Steve Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[comfooc]

Post by Steven M. Bellovin
What version of NetBSD?

4.0 and while second day of test current

Post by Steven M. Bellovin
What kind of network link connected the machines?

ethernet twisted-pair wire to D-Link DI-514 router

Post by Steven M. Bellovin
What ciphers did ssh use between the two machines?

i think it was default (3DES) i didn't change it

It is possible that NetBSD ssh client demanded different (more
bandwidth or CPU consuming) cipher from server?

Cheers...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[comfooc]

Post by comfooc

Post by Steven M. Bellovin
What ciphers did ssh use between the two machines?

i think it was default (3DES) i didn't change it

Can you make sure of that? The default is AES with SSH2 which
is the default.

Sorry, You are right they are using AES...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Brad]

Post by comfooc

Post by Steven M. Bellovin
What ciphers did ssh use between the two machines?

i think it was default (3DES) i didn't change it

Can you make sure of that? The default is AES with SSH2 which
is the default.

[comfooc]

But both machines are on the same LAN? I asked because there are some
things that can be done on fast, long-distance connections that don't
matter too much on a LAN, especially if you're not running at gigE
speeds.

They are in the same LAN, two meters of wire afar

Post by comfooc
It is possible that NetBSD ssh client demanded different (more
bandwidth or CPU consuming) cipher from server?

Yes, that's very possible; it's why I asked. Run 'ssh -v' to see what
ciphers are being negotiated, I believe.

I've checked and Linux, OpenBSD, NetBSD are negotiating the same cipher AES128

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Steven M. Bellovin]

On Thu, 17 Jan 2008 20:18:18 +0100

Post by comfooc

But both machines are on the same LAN? I asked because there are
some things that can be done on fast, long-distance connections
that don't matter too much on a LAN, especially if you're not
running at gigE speeds.

They are in the same LAN, two meters of wire afar

Post by comfooc
It is possible that NetBSD ssh client demanded different (more
bandwidth or CPU consuming) cipher from server?

Yes, that's very possible; it's why I asked. Run 'ssh -v' to see
what ciphers are being negotiated, I believe.

I've checked and Linux, OpenBSD, NetBSD are negotiating the same cipher AES128

Hmm... Try installing bench/ttcp; that will let you test the network
piece independent of the crypto. After that, perhaps run

openssl speed

to see if the crypto performance is seriously mismatched.

After that, I'm out of ideas.

--Steve Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[comfooc]

Post by Steven M. Bellovin
After that, perhaps run
openssl speed
to see if the crypto performance is seriously mismatched.

I've made some tests:
Linux:

OpenSSL 0.9.8g 19 Oct 2007
built on: Sun Nov 4 22:05:55 CET 2007
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long)
aes(partial) blowfish(idx)
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS
-D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -O3
-march=i686 -Wa,--noexecstack -g -Wall -DOPENSSL_BN_ASM_PART_WORDS
-DOPENSSL_IA32_SSE2 -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: TIMES TIMEB HZ=100 [sysconf value]
timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md2 655.86k 1412.29k 1989.89k 2206.19k 2283.42k
mdc2 0.00 0.00 0.00 0.00 0.00
md4 5059.02k 18241.45k 57111.73k 120965.36k 179736.83k
md5 4408.26k 15380.70k 44968.74k 86139.02k 117792.77k
hmac(md5) 6156.01k 20009.42k 54049.17k 94867.31k 121435.48k
sha1 4118.37k 13093.27k 32829.66k 52553.31k 63993.17k
rmd160 3821.37k 11405.08k 25952.62k 38543.70k 44593.33k
rc4 79280.68k 89224.58k 96647.48k 98792.79k 97242.58k
des cbc 15648.94k 16048.27k 16279.30k 16394.92k 16351.34k
des ede3 5570.91k 5710.38k 5783.72k 5770.48k 5772.50k
idea cbc 0.00 0.00 0.00 0.00 0.00
seed cbc 0.00 0.00 0.00 0.00 0.00
rc2 cbc 7079.84k 7327.06k 7401.38k 7417.71k 7421.79k
rc5-32/12 cbc 0.00 0.00 0.00 0.00 0.00
blowfish cbc 23706.44k 25177.09k 25942.50k 26115.40k 26214.40k
cast cbc 13247.27k 14046.81k 14341.61k 14414.93k 14477.99k
aes-128 cbc 21271.44k 28059.45k 30584.18k 31360.94k 31692.12k
aes-192 cbc 12041.74k 20540.96k 24835.32k 26303.53k 26766.88k
aes-256 cbc 11279.90k 18372.10k 21828.95k 22815.81k 23158.05k
camellia-128 cbc 0.00 0.00 0.00 0.00
0.00
camellia-192 cbc 0.00 0.00 0.00 0.00
0.00
camellia-256 cbc 0.00 0.00 0.00 0.00
0.00
sha256 3013.38k 6873.09k 12097.06k 14937.13k 16089.09k
sha512 872.42k 3484.71k 5328.03k 7512.78k 8510.43k
aes-128 ige 24473.63k 26976.09k 27636.94k 27876.95k 27889.21k
aes-192 ige 21390.94k 23430.19k 23959.98k 24131.36k 24143.27k
aes-256 ige 19189.68k 20705.34k 21107.58k 21248.17k 21323.78k
sign verify sign/s verify/s
rsa 512 bits 0.002288s 0.000182s 437.1 5509.5
rsa 1024 bits 0.010810s 0.000509s 92.5 1964.4
rsa 2048 bits 0.061534s 0.001653s 16.3 604.8
rsa 4096 bits 0.395385s 0.005684s 2.5 175.9
sign verify sign/s verify/s
dsa 512 bits 0.001821s 0.002110s 549.2 473.9
dsa 1024 bits 0.005000s 0.005793s 200.0 172.6
dsa 2048 bits 0.016207s 0.019437s 61.7 51.4


NetBSD:

OpenSSL 0.9.8e 23 Feb 2007
built on: NetBSD 4.99.45
options:bn(32,32) md2(int) rc4(ptr,int) des(ptr,risc2,16,long)
aes(partial) blowfish(ptr2)
compiler: gcc version 4.1.3 20070620 (prerelease) (NetBSD nb1 20070620)
available timing options: USE_TOD HZ=100 [sysconf value]
timing function used: getrusage
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md2 343.01k 750.87k 1070.51k 1197.80k 1240.93k
mdc2 0.00 0.00 0.00 0.00 0.00
md4 2782.98k 10405.32k 34428.58k 81073.96k 134638.71k
md5 2267.12k 8480.15k 28255.66k 67132.29k 112983.57k
hmac(md5) 5960.67k 19920.40k 54120.23k 94934.39k 120055.53k
sha1 2381.94k 7870.96k 21100.68k 36375.90k 46124.61k
rmd160 2255.79k 7601.37k 20273.15k 34559.73k 43553.84k
rc4 28759.60k 31325.85k 32406.46k 32711.89k 32776.24k
des cbc 7849.09k 8244.36k 8350.75k 8377.68k 8383.01k
des ede3 2886.79k 2947.20k 2966.63k 2971.45k 2972.20k
idea cbc 0.00 0.00 0.00 0.00 0.00
rc2 cbc 6169.94k 6324.21k 6376.67k 6389.97k 6393.49k
rc5-32/12 cbc 0.00 0.00 0.00 0.00 0.00
blowfish cbc 21298.12k 23139.53k 23786.69k 23954.20k 23981.00k
cast cbc 14260.53k 15073.41k 15357.83k 15429.19k 15438.05k
aes-128 cbc 13096.45k 13832.38k 14034.33k 14096.25k 14101.97k
aes-192 cbc 11280.65k 11840.68k 11980.98k 12026.15k 12023.77k
aes-256 cbc 10082.14k 10442.50k 10544.86k 10577.60k 10582.66k
camellia-128 cbc 0.00 0.00 0.00 0.00
0.00
camellia-192 cbc 0.00 0.00 0.00 0.00
0.00
camellia-256 cbc 0.00 0.00 0.00 0.00
0.00
sha256 1844.90k 5076.16k 10497.85k 14314.96k 16019.51k
sha512 622.95k 2492.41k 4305.89k 6366.48k 7393.15k
sign verify sign/s verify/s
rsa 512 bits 0.002332s 0.000216s 428.8 4623.2
rsa 1024 bits 0.010791s 0.000563s 92.7 1775.3
rsa 2048 bits 0.061433s 0.001737s 16.3 575.6
rsa 4096 bits 0.389058s 0.005833s 2.6 171.4
sign verify sign/s verify/s
dsa 512 bits 0.001813s 0.002199s 551.7 454.7
dsa 1024 bits 0.004968s 0.006068s 201.3 164.8
dsa 2048 bits 0.016115s 0.019358s 62.1 51.7


What is most intresting:
L: aes-128 cbc 21271.44k 28059.45k 30584.18k 31360.94k
31692.12k
N: aes-128 cbc 13096.45k 13832.38k 14034.33k 14096.25k
14101.97k

So its not an network issue...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Juan RP]

On Thu, 17 Jan 2008 23:23:03 +0100

Post by comfooc

Post by Steven M. Bellovin
After that, perhaps run
openssl speed
to see if the crypto performance is seriously mismatched.

OpenSSL 0.9.8g 19 Oct 2007
built on: Sun Nov 4 22:05:55 CET 2007
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long)
aes(partial) blowfish(idx)
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS
-D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -O3
-march=i686 -Wa,--noexecstack -g -Wall -DOPENSSL_BN_ASM_PART_WORDS
-DOPENSSL_IA32_SSE2 -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: TIMES TIMEB HZ=100 [sysconf value]

I don't know if openssl on netbsd uses any of these compile time
variables, but I guess that perhaps OPENSSL_IA32_SSE2 would be
one of the reasons (perhaps I'm wrong :-).

Post by comfooc
L: aes-128 cbc 21271.44k 28059.45k 30584.18k 31360.94k
31692.12k
N: aes-128 cbc 13096.45k 13832.38k 14034.33k 14096.25k
14101.97k
So its not an network issue...

I'm curious, could you please compare openbsd and netbsd too?

[Juan RP]

On Thu, 17 Jan 2008 23:30:41 +0100

Post by Juan RP
I don't know if openssl on netbsd uses any of these compile time
variables, but I guess that perhaps OPENSSL_IA32_SSE2 would be
one of the reasons (perhaps I'm wrong :-).

Looks like that on netbsd we don't use the asm versions, that might
be the answer of the difference that you see.

It seems that to build the asm code you need perl, at least that
is my understanding after looking at the openssl distribution code.

[Aaron Turner]

Post by comfooc
Hi,
few weeks ago i made some test using scp.
Systems were out of the box.
Machine 1 : OpenBSD; P3 500Mhz; 192Mram; 10Mbps;
Machine 2 : tripleboot :OpenBSD, Linux, NetBSD; P3 750Mhz, 512Mram, 100Mbps;
Machine 1 <=average 800KBps=> Machine 2 : Linux
Machine 1 <=average 550KBps=> Machine 2 : NetBSD
Machine 1 <=average 950KBps=> Machine 2 : OpenBSD
What might caused that low transfer? How to improve it?
Cheers...

Machine #1 has a 10Mbps NIC? 950KBps is ~ 7.4Mbps. Considering
10Mbps is 1/2 duplex, and considering TCP overhead, etc I can't say
I'm all that shocked your performance sucks.
--
Aaron Turner
http://synfin.net/
http://tcpreplay.synfin.net/ - Pcap editing & replay tools for Unix
They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety. -- Benjamin Franklin

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Brad]

Post by Aaron Turner

Post by comfooc
Hi,
few weeks ago i made some test using scp.
Systems were out of the box.
Machine 1 : OpenBSD; P3 500Mhz; 192Mram; 10Mbps;
Machine 2 : tripleboot :OpenBSD, Linux, NetBSD; P3 750Mhz, 512Mram, 100Mbps;
Machine 1 <=average 800KBps=> Machine 2 : Linux
Machine 1 <=average 550KBps=> Machine 2 : NetBSD
Machine 1 <=average 950KBps=> Machine 2 : OpenBSD
What might caused that low transfer? How to improve it?
Cheers...

Machine #1 has a 10Mbps NIC? 950KBps is ~ 7.4Mbps. Considering
10Mbps is 1/2 duplex, and considering TCP overhead, etc I can't say
I'm all that shocked your performance sucks.

That is an assumption that 10Mbps is half duplex.

He was asking why the system running NetBSD performs quite a bit worse
then when running the same system with OpenBSD or Linux.

[comfooc]

Post by Aaron Turner
Machine #1 has a 10Mbps NIC? 950KBps is ~ 7.4Mbps. Considering
10Mbps is 1/2 duplex, and considering TCP overhead, etc I can't say
I'm all that shocked your performance sucks.

Machine 1 is in this test used mostly as a server and its using full duplex.
The case is that OpenBSD and Linux had a lot better performance than NetBSD.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[comfooc]

On Fri, 18 Jan 2008 00:02:35 +0100

Most interesting that NetBSD (at almost twice better hardware)
perform worse...

there's something wrong with the NetBSD implementation of AES.
Furthermore, since it's probably the same source code, the issue is
likely to be configuration options. It's also likely not the network
itself (though running ttcp would help verify that).
The thread should probably move off of tech-net (after that
verification) and move to, say, tech-security or tech-userlevel, I
suspect.

Send to tech-security and named as "openssl speed".

Cheers and thanks for help...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Steven M. Bellovin]

On Fri, 18 Jan 2008 00:02:35 +0100

Most interesting that NetBSD (at almost twice better hardware)
perform worse...

So this is almost certainly the immediate answer to your question:
there's something wrong with the NetBSD implementation of AES.
Furthermore, since it's probably the same source code, the issue is
likely to be configuration options. It's also likely not the network
itself (though running ttcp would help verify that).

The thread should probably move off of tech-net (after that
verification) and move to, say, tech-security or tech-userlevel, I
suspect.


--Steve Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[comfooc]

This is OpenBSD openssl speed test but it was made at weaker machine
(P3 500Mhz,192MBram) because at this moment I'm unable to run this
test at the previous machine (P3 750Mhz, 512MBram)

OpenSSL 0.9.7j 04 May 2006
built on: date not available
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long)
aes(partial) blowfish(idx)
compiler: information not available
available timing options: USE_TOD HZ=100 [sysconf value]
timing function used: getrusage
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md2 331.42k 710.79k 999.38k 1109.70k 1149.35k
mdc2 0.00 0.00 0.00 0.00 0.00
md4 2876.75k 10333.86k 30843.54k 61399.22k 86241.99k
md5 2353.01k 8506.64k 26275.60k 54396.58k 80879.24k
hmac(md5) 3471.00k 11998.60k 33390.93k 61785.99k 83649.12k
sha1 2418.04k 7663.42k 18308.12k 27930.90k 33145.90k
rmd160 2150.46k 6708.25k 16042.79k 24784.90k 29420.04k
rc4 50226.28k 61680.37k 65024.30k 65584.56k 64439.77k
des cbc 8860.66k 9439.94k 9669.88k 9721.46k 9711.55k
des ede3 3560.12k 3645.52k 3681.34k 3682.24k 3686.32k
idea cbc 0.00 0.00 0.00 0.00 0.00
rc2 cbc 4107.36k 4255.03k 4306.57k 4319.93k 4325.03k
rc5-32/12 cbc 0.00 0.00 0.00 0.00 0.00
blowfish cbc 12978.86k 14570.09k 15268.44k 15437.90k 15394.51k
cast cbc 12728.45k 14322.83k 14882.53k 15082.13k 15115.83k
aes-128 cbc 14453.82k 15232.75k 15760.38k 15924.40k 15926.95k
aes-192 cbc 12819.34k 13449.16k 13842.57k 13928.95k 13999.17k
aes-256 cbc 11519.04k 12000.90k 12332.69k 12432.43k 12428.07k
sign verify sign/s verify/s
rsa 512 bits 0.003409s 0.000413s 293.3 2418.7
rsa 1024 bits 0.015552s 0.000954s 64.3 1047.8
rsa 2048 bits 0.089704s 0.002880s 11.1 347.3
rsa 4096 bits 0.591912s 0.009599s 1.7 104.2
sign verify sign/s verify/s
dsa 512 bits 0.002510s 0.002957s 398.4 338.2
dsa 1024 bits 0.007226s 0.008642s 138.4 115.7
dsa 2048 bits 0.023922s 0.028271s 41.8 35.4


Most interesting that NetBSD (at almost twice better hardware) perform worse...
O : aes-128 cbc 14453.82k 15232.75k 15760.38k 15924.40k
15926.95k
N : aes-128 cbc 13096.45k 13832.38k 14034.33k 14096.25k
14101.97k

Cheers...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de