sluggish network performance in *one* domU

[Petar Bogdanovic]

Hello,

I'm experiencing a rather strange network problem within a Xen guest
system. The problem is bad network performance, it affects all tcp
connections and only appears in one domU.

The hardware is running Xen 3.1.2, NetBSD 4.0 dom0 on i386. There are 3
domUs running: two nearly identical NetBSD copies and one Debian system.

The `problematic' domU is one of those two NetBSD domUs. I checked lots
of stuff: the cpu is idle all the time, there is no disk or network I/O
and I couldn't find anything in the logs.

The layout is simple. Here is a clumsy drawing:

Linux Laptop
+------------------+
|.--------------. |
|| host-eth | | AMD Athlon System
||(host.foo.net)| | +-----------------------+
|`-----.--------' | |.-------. |
+------|-----------+ ,---.|dom0-re| |
| ,-----. / |`---+---' |
\ ,' `. | | \ |
`. / \ / | | |
`-( Internet )-' |.----+----. |
\ / ||dom0-xvif| |
`. ,' |`----`.---' Virtual |
'-----' | \ Machine |
| +----`.------------+|
| | \ ||
| | .-----`.-------. ||
| | | domU-xennet | ||
| | |(domU.bar.net)| ||
| | `--------------' ||
| +------------------+|
+-----------------------+

Based on that I tcpdumped and tcptraced the following ssh session from
my Linux Laptop at home (host.foo.net):

ssh domU.bar.com dd if=/dev/urandom bs=1m count=5 >/dev/null

from four different interfaces:

eth @ host.foo.net
re @ dom0
xvif @ dom0
xennet @ domU.bar.net

And it seems that I'm losing lots of data, since xennet, xvif and re
are all reporting the following:

unique bytes sent: 5254577
actual data pkts: 9346
actual data bytes: 5512613
rexmt data pkts: 369
rexmt data bytes: 258036

(the complete tcptraces are attached)

The xennet interface in domU owns a public IPv4 address which gets
forwarded and filtered by dom0 (+ipfilter).

Now, it would be easy to blame the ISP but it seems strange that the
other domUs are not affected at all. I tested the second NetBSD domU
and there were not one single `rexmt'.

Is there maybe something else I could try?

Thanks,



Petar Bogdanovic

[Steven M. Bellovin]

On Mon, 6 Jul 2009 22:05:31 +0200

Post by Petar Bogdanovic
Hello,
I'm experiencing a rather strange network problem within a Xen guest
system. The problem is bad network performance, it affects all tcp
connections and only appears in one domU.
The hardware is running Xen 3.1.2, NetBSD 4.0 dom0 on i386. There
are 3 domUs running: two nearly identical NetBSD copies and one
Debian system.
The `problematic' domU is one of those two NetBSD domUs. I checked
lots of stuff: the cpu is idle all the time, there is no disk or
network I/O and I couldn't find anything in the logs.

No solutions, but a few things to look at...

First -- try pkgsrc/benchmarks/ttcp rather than dd'ing /dev/urandom; it
avoids confounding network issues with host issues.

Second -- slow TCP generally indicates packet loss; the question is
where the packets are being dropped. pkgsrc/net/mtr might help answer
that.

Third -- just try running a tcpdump on the domUs to see if there's some
background process sending a lot of data on that domU or perhaps on the
dom0.


--Steve Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Petar Bogdanovic]

Post by Petar Bogdanovic
I'm experiencing a rather strange network problem within a Xen guest
system. The problem is bad network performance, it affects all tcp
connections and only appears in one domU.

Definitely not a problem of NetBSD or Xen. It seems that the ISP or
some of his partners have some weird routing problems.

Thanks Steven for the ttcp hint. I'll try that next time I run into
something similar.



Petar Bogdanovic




--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de