TCP/IPv6 stalling

[Jan Danielsson]

Hello,

I'm having some networking issues which aren't completely unheard of;
I googled around a little and found several posts from diffierent people
describing more or less exactly this problem, but couldn't find any
solutions.

The abstract versions is: I can ping Internet from the host, I can
make very small requests ("wget http://www.netbsd.org" works fine), but
if I try to fetch something larger than a few KB, the transfer grinds to
a halt (not entirely true, I get like 50B/s (very unstable) transfer
rate, and that's not a typo, there isn't a "K" missing there, it's in
the magnitude of 50 characters per second).

Any such operations from the router works fine, it's only when
performed from any of the system on the LAN. (All host systems are
affected; NetBSD, Linux, Windows, Mac). IPv4 does not have any issues;
this only occurs with IPv6.

I looked at the traffic on the router in Wireshark while performing:

$ wget http://ftp.gnu.org/pub/gnu/tar/tar-1.29.tar.bz2

.. on the host in a setup which looks like this:

host router
(re0) <--> (wm0) (re0) <--> Modem
Asustek Intel Asustek Cisco

.. and it's immediately apparent that something is wrong. The
capture is riddled with [TCP Retransmission] and [TCP Dup ACK ...].

First entries in the capture are normal:
SYN host -> server
SYN+ACK host <- server
ACK host -> server

This is followed the actual HTTP request being sent from the host to
the server.

After that the oddities begin. The initial SYN is resent; marked in
Wireshark as a [TCP Spurious Retransmission]. The ethernet frame
indicates that the first SYN was sent from the host to the router's wm0
interface. The retransmission is sent from the routers re0 interface to
the modem.

I've attached the summary part of the capture. If anyone is
interested in a more detailed capture, I can email them.

It looks to my highly untrained eyes like the network stack is
suddenly deciding, a while after the initial SYN/ACK handshake has
completed, that the SYN was never ACK'd, so it is resent. The similar
kind of issue occurs over and over again, which causes more or less a
complete stall of the TCP traffic.

The router is running NetBSD/amd64 -7 with npf (currently configured
to just pass everything in both directions).

[Greg Troxel]

Your wireshark capture does not show packets in time order. The
retransmitted SYN is very soon after the first one, too soon for a
sensible timeout.

I suggest running tcpdump (not wireshark, which is too complicated) on
the host (not the router) and saving it to a file with -w, so you can
look later with different options.

Run tcpdump not only on your host, but on both the LAN interface and the
WAN interface (tunnel, if that's how it is) on your router, also saving
to a file. Afterwards, convert them all to text and trace the packet
flow all the way. I am suspicious of firewalls in cases like this.

Think about MTU and PMTU-D. However, bad behavior seems to start before
the far end would send large packets. It is critical that ICMPv6 Packet
Too Big messages from intermediate routers get back to your host, and
also to the far side. There are broken systems out there. You may want
to force MTU down to 1280 (which must be supported) rather than
optimistically trying higher. (I don't think this is your problem, but
you're going to have to understand this eventually.)

Also, run "netstat -p" on both host and router before, and then after,
saving them both so you can run diff to look for changes to counters
that you are not thinking about. Also whatever the firewall stat
program is.