Pathological TCP behavior running ls(1) over SSH

[Andreas Gustafsson]

I'm suffering from annoying 1-second pauses in my SSH connections,
often while waiting during the output from ls(1). This happens when I'm
logged in from one NetBSD-current machine to another NetBSD-current
machine.

The following tcpdump should illustrate the problem. It was taken
from the server side of the connection and starts at a point in time
when I have typed "ls" on the client keyboard but not yet pressed
return, and ends when the "ls" output has been fully displayed
(including a one-second pause in the middle). I'm listing my home
directory which contains 352 files, displayed by ls on 118 lines
totaling 6976 bytes of uncompressed data. The network topology
involved is a simple wireless home network: server - ethernet -
wireless_access_point - wireless - client.

# tcpdump -n -p -i re0 port 64186
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 96 bytes
22:08:46.773481 IP 10.0.1.254.64186 > 91.152.94.125.22: P 3643051533:3643051565(32) ack 1589485459 win 33580
22:08:46.774427 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1:33(32) ack 32 win 33580
22:08:46.781341 IP 91.152.94.125.22 > 10.0.1.254.64186: P 33:65(32) ack 32 win 33580
22:08:46.781483 IP 91.152.94.125.22 > 10.0.1.254.64186: P 65:97(32) ack 32 win 33580
22:08:46.781593 IP 91.152.94.125.22 > 10.0.1.254.64186: P 97:129(32) ack 32 win 33580
22:08:46.781699 IP 91.152.94.125.22 > 10.0.1.254.64186: P 129:161(32) ack 32 win 33580
22:08:46.781799 IP 91.152.94.125.22 > 10.0.1.254.64186: P 161:193(32) ack 32 win 33580
22:08:46.781902 IP 91.152.94.125.22 > 10.0.1.254.64186: P 193:225(32) ack 32 win 33580
22:08:46.782003 IP 91.152.94.125.22 > 10.0.1.254.64186: P 225:257(32) ack 32 win 33580
22:08:46.782106 IP 91.152.94.125.22 > 10.0.1.254.64186: P 257:289(32) ack 32 win 33580
22:08:46.782208 IP 91.152.94.125.22 > 10.0.1.254.64186: P 289:321(32) ack 32 win 33580
22:08:46.782308 IP 91.152.94.125.22 > 10.0.1.254.64186: P 321:353(32) ack 32 win 33580
22:08:46.782413 IP 91.152.94.125.22 > 10.0.1.254.64186: P 353:385(32) ack 32 win 33580
22:08:46.782516 IP 91.152.94.125.22 > 10.0.1.254.64186: P 385:417(32) ack 32 win 33580
22:08:46.782619 IP 91.152.94.125.22 > 10.0.1.254.64186: P 417:449(32) ack 32 win 33580
22:08:46.782722 IP 91.152.94.125.22 > 10.0.1.254.64186: P 449:481(32) ack 32 win 33580
22:08:46.782827 IP 91.152.94.125.22 > 10.0.1.254.64186: P 481:513(32) ack 32 win 33580
22:08:46.782928 IP 91.152.94.125.22 > 10.0.1.254.64186: P 513:545(32) ack 32 win 33580
22:08:46.783029 IP 91.152.94.125.22 > 10.0.1.254.64186: P 545:577(32) ack 32 win 33580
22:08:46.783132 IP 91.152.94.125.22 > 10.0.1.254.64186: P 577:609(32) ack 32 win 33580
22:08:46.783233 IP 91.152.94.125.22 > 10.0.1.254.64186: P 609:641(32) ack 32 win 33580
22:08:46.783333 IP 91.152.94.125.22 > 10.0.1.254.64186: P 641:673(32) ack 32 win 33580
22:08:46.783443 IP 91.152.94.125.22 > 10.0.1.254.64186: P 673:705(32) ack 32 win 33580
22:08:46.783551 IP 91.152.94.125.22 > 10.0.1.254.64186: P 705:737(32) ack 32 win 33580
22:08:46.783656 IP 91.152.94.125.22 > 10.0.1.254.64186: P 737:769(32) ack 32 win 33580
22:08:46.783762 IP 91.152.94.125.22 > 10.0.1.254.64186: P 769:801(32) ack 32 win 33580
22:08:46.783864 IP 91.152.94.125.22 > 10.0.1.254.64186: P 801:833(32) ack 32 win 33580
22:08:46.783991 IP 91.152.94.125.22 > 10.0.1.254.64186: P 833:881(48) ack 32 win 33580
22:08:46.784101 IP 91.152.94.125.22 > 10.0.1.254.64186: P 881:929(48) ack 32 win 33580
22:08:46.784208 IP 91.152.94.125.22 > 10.0.1.254.64186: P 929:977(48) ack 32 win 33580
22:08:46.784316 IP 91.152.94.125.22 > 10.0.1.254.64186: P 977:1025(48) ack 32 win 33580
22:08:46.784422 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1025:1073(48) ack 32 win 33580
22:08:46.784526 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1073:1121(48) ack 32 win 33580
22:08:46.784637 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1121:1169(48) ack 32 win 33580
22:08:46.784744 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1169:1217(48) ack 32 win 33580
22:08:46.784851 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1217:1265(48) ack 32 win 33580
22:08:46.784960 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1265:1313(48) ack 32 win 33580
22:08:46.785066 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1313:1361(48) ack 32 win 33580
22:08:46.785159 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 65 win 33548
22:08:46.785245 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1361:1409(48) ack 32 win 33580
22:08:46.785357 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1409:1457(48) ack 32 win 33580
22:08:46.785465 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1457:1505(48) ack 32 win 33580
22:08:46.785574 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1505:1553(48) ack 32 win 33580
22:08:46.785684 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1553:1601(48) ack 32 win 33580
22:08:46.785786 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1601:1649(48) ack 32 win 33580
22:08:46.785898 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1649:1697(48) ack 32 win 33580
22:08:46.786004 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1697:1745(48) ack 32 win 33580
22:08:46.786112 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1745:1793(48) ack 32 win 33580
22:08:46.786221 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1793:1841(48) ack 32 win 33580
22:08:46.786327 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1841:1889(48) ack 32 win 33580
22:08:46.786437 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1889:1937(48) ack 32 win 33580
22:08:46.786547 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1937:1985(48) ack 32 win 33580
22:08:46.786653 IP 91.152.94.125.22 > 10.0.1.254.64186: P 1985:2033(48) ack 32 win 33580
22:08:46.786764 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2033:2081(48) ack 32 win 33580
22:08:46.786867 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2081:2129(48) ack 32 win 33580
22:08:46.786975 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2129:2177(48) ack 32 win 33580
22:08:46.787079 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2177:2225(48) ack 32 win 33580
22:08:46.787183 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2225:2273(48) ack 32 win 33580
22:08:46.787287 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2273:2321(48) ack 32 win 33580
22:08:46.787397 IP 91.152.94.125.22 > 10.0.1.254.64186: P 2321:2369(48) ack 32 win 33580
22:08:46.789724 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 129 win 33548
22:08:46.792828 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 193 win 33548
22:08:46.797469 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 257 win 33548
22:08:46.800948 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 321 win 33484
22:08:46.804855 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 385 win 33420
22:08:46.809791 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33548 <nop,nop,sack 1 {449:481}>
22:08:46.811619 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 1 {449:513}>
22:08:46.812758 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 1 {449:545}>
22:08:46.815403 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 2 {577:609}{449:545}>
22:08:46.816686 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 2 {577:641}{449:545}>
22:08:46.818773 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {673:705}{577:641}{449:545}>
22:08:46.821499 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {673:737}{577:641}{449:545}>
22:08:46.822713 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {769:801}{673:737}{577:641}>
22:08:46.826083 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {769:833}{673:737}{577:641}>
22:08:46.826741 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {881:929}{769:833}{673:737}>
22:08:46.830789 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {977:1025}{881:929}{769:833}>
22:08:46.834090 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {977:1073}{881:929}{769:833}>
22:08:46.834804 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1121:1169}{977:1073}{881:929}>
22:08:46.838272 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1121:1217}{977:1073}{881:929}>
22:08:46.838883 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1121:1265}{977:1073}{881:929}>
22:08:46.841573 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1361}{1121:1265}{977:1073}>
22:08:46.842755 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1409}{1121:1265}{977:1073}>
22:08:46.845251 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1457}{1121:1265}{977:1073}>
22:08:46.846647 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1505}{1121:1265}{977:1073}>
22:08:46.849678 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1553}{1121:1265}{977:1073}>
22:08:46.853647 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1601}{1121:1265}{977:1073}>
22:08:46.854742 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1649}{1121:1265}{977:1073}>
22:08:46.856660 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1697}{1121:1265}{977:1073}>
22:08:46.858642 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1745}{1121:1265}{977:1073}>
22:08:46.860721 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1313:1793}{1121:1265}{977:1073}>
22:08:46.862679 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1841:1889}{1313:1793}{1121:1265}>
22:08:46.864690 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1841:1937}{1313:1793}{1121:1265}>
22:08:46.866639 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1985:2033}{1841:1937}{1313:1793}>
22:08:46.868653 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {1985:2081}{1841:1937}{1313:1793}>
22:08:46.870658 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2129:2177}{1985:2081}{1841:1937}>
22:08:46.872641 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2129:2225}{1985:2081}{1841:1937}>
22:08:46.874649 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2273:2321}{2129:2225}{1985:2081}>
22:08:46.876534 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2273:2369}{2129:2225}{1985:2081}>

[the one-second pause is here]

22:08:47.800341 IP 91.152.94.125.22 > 10.0.1.254.64186: . 417:1877(1460) ack 32 win 33580
22:08:47.805289 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 1937 win 32060 <nop,nop,sack 3 {2273:2369}{2129:2225}{1985:2081}>
22:08:47.805381 IP 91.152.94.125.22 > 10.0.1.254.64186: . 1937:3397(1460) ack 32 win 33580
22:08:47.805397 IP 91.152.94.125.22 > 10.0.1.254.64186: P 3397:3889(492) ack 32 win 33580
22:08:47.810705 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 3397 win 32120
22:08:48.009779 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 3889 win 33580
^C
104 packets captured
104 packets received by filter
0 packets dropped by kernel
#

This looks similar to what Tomas Svensson reported in 2002 in
<http://mail-index.netbsd.org/tech-net/2002/03/15/0000.html>, but
these tinygrams are even smaller, perhaps because I have enabled
compression in my .ssh/config.

Aside from the question of whether sshd should set TCP_NODELAY or not,
could someone explain why the server waits almost a whole second to
retransmit the segment starting at octet 417; why don't the
38 duplicate ACKs cause a fast retransmit?

[David Young]

Post by Andreas Gustafsson
Aside from the question of whether sshd should set TCP_NODELAY or not,
could someone explain why the server waits almost a whole second to
retransmit the segment starting at octet 417; why don't the
38 duplicate ACKs cause a fast retransmit?

Does the server run a packet filter of any kind? I was astonished to
find that PF was filtering duplicate ACKs unless I told it otherwise with,
e.g., the 'flags A/A' rule, below:

# pfctl -a gateway -s rules
No ALTQ support in kernel
ALTQ related functions disabled
pass out log-all quick on ath0 route-to gre2 from <cuwin> to ! <cuwin> flags A/A
pass out log-all quick on ath0 route-to gre2 from <cuwin> to ! <cuwin> keep state (if-bound)

Dave

[Steven M. Bellovin]

On Sun, 27 Jan 2008 18:06:46 -0600

Post by David Young

Post by Andreas Gustafsson
Aside from the question of whether sshd should set TCP_NODELAY or
not, could someone explain why the server waits almost a whole
second to retransmit the segment starting at octet 417; why don't
the 38 duplicate ACKs cause a fast retransmit?

Does the server run a packet filter of any kind? I was astonished to
find that PF was filtering duplicate ACKs unless I told it otherwise
# pfctl -a gateway -s rules
No ALTQ support in kernel
ALTQ related functions disabled
pass out log-all quick on ath0 route-to gre2 from <cuwin> to !
<cuwin> flags A/A pass out log-all quick on ath0 route-to gre2 from
<cuwin> to ! <cuwin> keep state (if-bound)

Frankly, that strikes me as a bug in PF -- there are TCP semantics for
duplicate ACKs.


--Steve Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Daniel Carosone]

Post by Steven M. Bellovin
Frankly, that strikes me as a bug in PF -- there are TCP semantics for
duplicate ACKs.

And even if there weren't, PF has no way to know that the first ack
was not dropped somewhere further down the line, between it and the
final destination. So, to the receiver, this may not be a dup ack at
all.

--
Dan.

[Andreas Gustafsson]

Post by David Young
Does the server run a packet filter of any kind? I was astonished to
find that PF was filtering duplicate ACKs unless I told it otherwise with,
# pfctl -a gateway -s rules
No ALTQ support in kernel
ALTQ related functions disabled
pass out log-all quick on ath0 route-to gre2 from <cuwin> to ! <cuwin> flags A/A
pass out log-all quick on ath0 route-to gre2 from <cuwin> to ! <cuwin> keep state (if-bound)

I'm running ipf (not pf) and ipnat, but I doubt that's the problem;
the number of duplicate acks reported by "netstat -p tcp" on the
server increased by 48 during a single "ls" command, so it would
appear that the duplicate acks are in fact reaching the TCP stack.

[Darren Reed]

Post by Andreas Gustafsson
I'm suffering from annoying 1-second pauses in my SSH connections,
often while waiting during the output from ls(1). This happens when I'm
logged in from one NetBSD-current machine to another NetBSD-current
machine.

...

Post by Andreas Gustafsson
# tcpdump -n -p -i re0 port 64186
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 96 bytes

...

Post by Andreas Gustafsson
22:08:46.870658 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2129:2177}{1985:2081}{1841:1937}>
22:08:46.872641 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2129:2225}{1985:2081}{1841:1937}>
22:08:46.874649 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2273:2321}{2129:2225}{1985:2081}>
22:08:46.876534 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 417 win 33580 <nop,nop,sack 3 {2273:2369}{2129:2225}{1985:2081}>
[the one-second pause is here]
22:08:47.800341 IP 91.152.94.125.22 > 10.0.1.254.64186: . 417:1877(1460) ack 32 win 33580
22:08:47.805289 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 1937 win 32060 <nop,nop,sack 3 {2273:2369}{2129:2225}{1985:2081}>
22:08:47.805381 IP 91.152.94.125.22 > 10.0.1.254.64186: . 1937:3397(1460) ack 32 win 33580
22:08:47.805397 IP 91.152.94.125.22 > 10.0.1.254.64186: P 3397:3889(492) ack 32 win 33580
22:08:47.810705 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 3397 win 32120
22:08:48.009779 IP 10.0.1.254.64186 > 91.152.94.125.22: . ack 3889 win 33580
^C
104 packets captured
104 packets received by filter
0 packets dropped by kernel
#
This looks similar to what Tomas Svensson reported in 2002 in
<http://mail-index.netbsd.org/tech-net/2002/03/15/0000.html>, but
these tinygrams are even smaller, perhaps because I have enabled
compression in my .ssh/config.

This is a different problem/bug to what you are seeing.

Post by Andreas Gustafsson
Aside from the question of whether sshd should set TCP_NODELAY or not,
could someone explain why the server waits almost a whole second to
retransmit the segment starting at octet 417; why don't the
38 duplicate ACKs cause a fast retransmit?

I'm going to bet that this is somehow tied up with SACK.
All of the retransmitted (duplicate) ACKs include 1 or more
ranges of bytes from SACK.

So it seems worthwhile testing to see if that is a cause here:
sysctl -w net.inet.tcp.sack.enable=0

Darren

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Andreas Gustafsson]

Post by Darren Reed

This looks similar to what Tomas Svensson reported in 2002 [...]

This is a different problem/bug to what you are seeing.

I think the cause of all those tinygrams being sent in the first place
is probably the same, but the lack of fast retransmit when one of them
is lost is a new and separate issue.

Post by Darren Reed
I'm going to bet that this is somehow tied up with SACK.
All of the retransmitted (duplicate) ACKs include 1 or more
ranges of bytes from SACK.
sysctl -w net.inet.tcp.sack.enable=0

I think you're right. If I disable SACK, the pauses disappear, and
tcpdump shows that fast retransmit is now happening:

# tcpdump -n -p -i re0 port 65299
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 96 bytes
19:15:15.397996 IP 10.0.1.254.65299 > 91.152.94.125.22: P 928726461:928726493(32) ack 108106078 win 33580
19:15:15.398954 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1:33(32) ack 32 win 33580
19:15:15.406027 IP 91.152.94.125.22 > 10.0.1.254.65299: P 33:65(32) ack 32 win 33580
19:15:15.406172 IP 91.152.94.125.22 > 10.0.1.254.65299: P 65:97(32) ack 32 win 33580
19:15:15.406284 IP 91.152.94.125.22 > 10.0.1.254.65299: P 97:129(32) ack 32 win 33580
19:15:15.406395 IP 91.152.94.125.22 > 10.0.1.254.65299: P 129:161(32) ack 32 win 33580
19:15:15.406501 IP 91.152.94.125.22 > 10.0.1.254.65299: P 161:193(32) ack 32 win 33580
19:15:15.406607 IP 91.152.94.125.22 > 10.0.1.254.65299: P 193:225(32) ack 32 win 33580
19:15:15.406710 IP 91.152.94.125.22 > 10.0.1.254.65299: P 225:257(32) ack 32 win 33580
19:15:15.406819 IP 91.152.94.125.22 > 10.0.1.254.65299: P 257:289(32) ack 32 win 33580
19:15:15.406923 IP 91.152.94.125.22 > 10.0.1.254.65299: P 289:321(32) ack 32 win 33580
19:15:15.407024 IP 91.152.94.125.22 > 10.0.1.254.65299: P 321:353(32) ack 32 win 33580
19:15:15.407130 IP 91.152.94.125.22 > 10.0.1.254.65299: P 353:385(32) ack 32 win 33580
19:15:15.407238 IP 91.152.94.125.22 > 10.0.1.254.65299: P 385:417(32) ack 32 win 33580
19:15:15.407344 IP 91.152.94.125.22 > 10.0.1.254.65299: P 417:449(32) ack 32 win 33580
19:15:15.407450 IP 91.152.94.125.22 > 10.0.1.254.65299: P 449:481(32) ack 32 win 33580
19:15:15.407557 IP 91.152.94.125.22 > 10.0.1.254.65299: P 481:513(32) ack 32 win 33580
19:15:15.407664 IP 91.152.94.125.22 > 10.0.1.254.65299: P 513:545(32) ack 32 win 33580
19:15:15.407770 IP 91.152.94.125.22 > 10.0.1.254.65299: P 545:577(32) ack 32 win 33580
19:15:15.407877 IP 91.152.94.125.22 > 10.0.1.254.65299: P 577:609(32) ack 32 win 33580
19:15:15.407981 IP 91.152.94.125.22 > 10.0.1.254.65299: P 609:641(32) ack 32 win 33580
19:15:15.408086 IP 91.152.94.125.22 > 10.0.1.254.65299: P 641:673(32) ack 32 win 33580
19:15:15.408196 IP 91.152.94.125.22 > 10.0.1.254.65299: P 673:705(32) ack 32 win 33580
19:15:15.408309 IP 91.152.94.125.22 > 10.0.1.254.65299: P 705:737(32) ack 32 win 33580
19:15:15.408417 IP 91.152.94.125.22 > 10.0.1.254.65299: P 737:769(32) ack 32 win 33580
19:15:15.408529 IP 91.152.94.125.22 > 10.0.1.254.65299: P 769:801(32) ack 32 win 33580
19:15:15.408636 IP 91.152.94.125.22 > 10.0.1.254.65299: P 801:833(32) ack 32 win 33580
19:15:15.408748 IP 91.152.94.125.22 > 10.0.1.254.65299: P 833:865(32) ack 32 win 33580
19:15:15.408851 IP 91.152.94.125.22 > 10.0.1.254.65299: P 865:897(32) ack 32 win 33580
19:15:15.408955 IP 91.152.94.125.22 > 10.0.1.254.65299: P 897:929(32) ack 32 win 33580
19:15:15.409076 IP 91.152.94.125.22 > 10.0.1.254.65299: P 929:961(32) ack 32 win 33580
19:15:15.409179 IP 91.152.94.125.22 > 10.0.1.254.65299: P 961:993(32) ack 32 win 33580
19:15:15.409283 IP 91.152.94.125.22 > 10.0.1.254.65299: P 993:1025(32) ack 32 win 33580
19:15:15.409386 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1025:1057(32) ack 32 win 33580
19:15:15.409496 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1057:1089(32) ack 32 win 33580
19:15:15.409599 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1089:1121(32) ack 32 win 33580
19:15:15.409704 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1121:1153(32) ack 32 win 33580
19:15:15.409810 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1153:1185(32) ack 32 win 33580
19:15:15.409912 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1185:1217(32) ack 32 win 33580
19:15:15.410016 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1217:1249(32) ack 32 win 33580
19:15:15.410129 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1249:1281(32) ack 32 win 33580
19:15:15.410235 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1281:1313(32) ack 32 win 33580
19:15:15.410340 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1313:1345(32) ack 32 win 33580
19:15:15.410442 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1345:1377(32) ack 32 win 33580
19:15:15.410547 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1377:1409(32) ack 32 win 33580
19:15:15.410652 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1409:1441(32) ack 32 win 33580
19:15:15.410760 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1441:1473(32) ack 32 win 33580
19:15:15.410868 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1473:1505(32) ack 32 win 33580
19:15:15.410915 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 65 win 33548
19:15:15.411045 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1505:1537(32) ack 32 win 33580
19:15:15.411151 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1537:1569(32) ack 32 win 33580
19:15:15.411258 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1569:1601(32) ack 32 win 33580
19:15:15.411363 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1601:1633(32) ack 32 win 33580
19:15:15.411472 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1633:1665(32) ack 32 win 33580
19:15:15.411577 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1665:1697(32) ack 32 win 33580
19:15:15.411680 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1697:1729(32) ack 32 win 33580
19:15:15.411787 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1729:1761(32) ack 32 win 33580
19:15:15.411890 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1761:1793(32) ack 32 win 33580
19:15:15.411998 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1793:1825(32) ack 32 win 33580
19:15:15.412103 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1825:1857(32) ack 32 win 33580
19:15:15.412210 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1857:1889(32) ack 32 win 33580
19:15:15.412317 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1889:1921(32) ack 32 win 33580
19:15:15.412422 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1921:1953(32) ack 32 win 33580
19:15:15.412528 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1953:1985(32) ack 32 win 33580
19:15:15.412632 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1985:2017(32) ack 32 win 33580
19:15:15.412736 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2017:2049(32) ack 32 win 33580
19:15:15.412841 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2049:2081(32) ack 32 win 33580
19:15:15.412945 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2081:2113(32) ack 32 win 33580
19:15:15.413049 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2113:2145(32) ack 32 win 33580
19:15:15.413154 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2145:2177(32) ack 32 win 33580
19:15:15.413261 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2177:2209(32) ack 32 win 33580
19:15:15.413363 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2209:2241(32) ack 32 win 33580
19:15:15.413467 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2241:2273(32) ack 32 win 33580
19:15:15.413574 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2273:2305(32) ack 32 win 33580
19:15:15.413676 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2305:2337(32) ack 32 win 33580
19:15:15.413782 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2337:2369(32) ack 32 win 33580
19:15:15.413889 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2369:2401(32) ack 32 win 33580
19:15:15.413999 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2401:2433(32) ack 32 win 33580
19:15:15.414106 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2433:2465(32) ack 32 win 33580
19:15:15.414214 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2465:2497(32) ack 32 win 33580
19:15:15.414321 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2497:2529(32) ack 32 win 33580
19:15:15.414424 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2529:2561(32) ack 32 win 33580
19:15:15.414728 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 129 win 33548
19:15:15.414758 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2561:2657(96) ack 32 win 33580
19:15:15.414808 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2657:2689(32) ack 32 win 33580
19:15:15.418871 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 193 win 33548
19:15:15.423151 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 257 win 33548
19:15:15.423196 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2689:2913(224) ack 32 win 33580
19:15:15.426749 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 321 win 33548
19:15:15.430929 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 385 win 33548
19:15:15.435875 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 449 win 33548
19:15:15.438661 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 513 win 33548
19:15:15.442611 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33548
19:15:15.444684 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.447092 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.449249 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.450701 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.450737 IP 91.152.94.125.22 > 10.0.1.254.65299: . 577:2037(1460) ack 32 win 33580
19:15:15.453314 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.454569 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.456774 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.458572 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.461077 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.462520 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.465030 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.466516 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.468888 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.470495 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.472925 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.474519 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.476489 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.478482 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.481476 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.482523 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.484860 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.486519 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.488804 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.490554 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.492553 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.494643 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.496628 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.498713 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.500539 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.502519 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.504926 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.506460 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.508479 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.510917 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.513033 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.515209 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.516429 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.518457 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.520450 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.522449 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.524437 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.526450 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.528440 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.530446 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.532464 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.534480 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
19:15:15.537238 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 2049 win 32108
19:15:15.537266 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2049:2913(864) ack 32 win 33580
19:15:15.542212 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 2913 win 32716
^C
143 packets captured
145 packets received by filter
0 packets dropped by kernel
#

[Zafer Aydogan]

Post by Andreas Gustafsson

Post by Darren Reed

This looks similar to what Tomas Svensson reported in 2002 [...]

This is a different problem/bug to what you are seeing.

I think the cause of all those tinygrams being sent in the first place
is probably the same, but the lack of fast retransmit when one of them
is lost is a new and separate issue.

Post by Darren Reed
I'm going to bet that this is somehow tied up with SACK.
All of the retransmitted (duplicate) ACKs include 1 or more
ranges of bytes from SACK.
sysctl -w net.inet.tcp.sack.enable=0

I think you're right. If I disable SACK, the pauses disappear, and

Nice to know. Since I turned off sack, my idle ssh connection doesn't
drop after 5 minutes anymore. I always thought it would be my
dsl-routers fault.

Zafer.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joerg Sonnenberger]

Post by Zafer Aydogan
Nice to know. Since I turned off sack, my idle ssh connection doesn't
drop after 5 minutes anymore. I always thought it would be my
dsl-routers fault.

What makes you believe that it doesn't deal incorrectly with SACK?

Joerg

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de