virtio net device stuck for UDP burst transmission

[Kenichi Yasukata]

Hi,

When I try UDP burst transmission with vioif driver on KVM,
all network connections stuck.

I'm using kernel source from NetBSD current.
Base system is installed by iso image of NetBSD7.

GUEST < NetBSD >

% uname -a
NetBSD 7.99.29 NetBSD 7.99.29 (MYKERNEL) #6: Tue May 10 10:14:30 JST
2016 root@:/usr/src/sys/arch/amd64/compile/MYKERNEL amd64

** MYKERNEL is just a copy of GENERIC

HOST < Ubuntu >
% uname -a
Linux N-1183 4.4.0-22-generic #39-Ubuntu SMP Thu May 5 16:53:32 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux

VMM < QEMU/KVM >
QEMU version v2.6.0-rc0

( Command )
qemu-system-x86_64
-enable-kvm \
-curses \
-m 4096 \
-drive file=./nbsdvd.img \
-net nic,model=virtio -net
tap,ifname=tap0,script=qemu-ifup,downscript=qemu-ifdown

% cat qemu-ifup
#!/bin/bash
echo "Bringing up $1 for bridged mode..."
ifconfig $1 0.0.0.0 promisc up
echo "Adding $1 to virbr0..."
brctl addif virbr0 $1
exit 0

% cat qemu-ifdown
#!/bin/bash
echo "Removing $1 to virbr0..."
brctl delif virbr0 $1
echo "Shutting down $1..."
ifconfig $1 down
exit 0


Initially, we faced this problem when we run UDP benchmarks with iperf3.

We expect this C program can reproduce the issue in your environments.
This is much simpler and good to clarify the problem.
https://gist.github.com/jmmlmendes/aaea56a8b7044487595ef7a6782fc018

When I run this program, whole network connections stuck.
The error message is "errno 55 - No buffer space available".
(If it works without any problem, could you try increased number of TX
packets "PACKETS_TO_SEND"?)

If I change the driver model to e1000, it works correctly.

Do you have any idea?

Thanks,
Kenichi

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael van Elst]

Post by Kenichi Yasukata
When I run this program, whole network connections stuck.
The error message is "errno 55 - No buffer space available".
(If it works without any problem, could you try increased number of TX
packets "PACKETS_TO_SEND"?)

This could be a pretty normal behaviour, for any interface. But
that state shouldn't last more than a few seconds after you stopped
the UDP "burst".

If it persists, this is probably some kind of mbuf leak in the driver.

[Kenichi Yasukata]

But that state shouldn't last more than a few seconds after you stopped

the UDP "burst".

In my environment, it doesn't recover automatically after this issue happens.

We suspect the problem here is that a deadlock can occur in the virtio driver.

That is, if the sender is too fast, there will be a point where
packets are not sent to the host anymore, buffers are full (hence
ENOBUFS) and they stay this way because of some kind of
synchronization failure.

This behavior is only seen with the virtio driver, not with e1000 for example.

Should this be sent to the bug mailing list?

Thanks,
Kenichi

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael van Elst]

Post by Michael van Elst

But that state shouldn't last more than a few seconds after you stopped

the UDP "burst".
In my environment, it doesn't recover automatically after this issue happens.

[...]

Post by Michael van Elst
Should this be sent to the bug mailing list?

Please send-pr.