ipfilter randomly dropping (ssh-)connections

Discussion:

(too old to reply)

Petar Bogdanovic

2014-06-11 15:57:22 UTC

Hi,

about a week ago, the automated daily ssh-tunnels to a netbsd-6 box
started to close shortly after they were established (client said
"Connection closed by remote host"; server said: "fatal: Write failed:
Network is unreachable"). A quick tcpdump revealed that the server side
at one point just FINs the connection and then spams the client with a
bunch of TCP resets.

After a while of tcpdump and ktrace, disabling ipfilter (v4.1.34) solved
the problem. Which was very confusing, because its ipf.conf hasn't
changed for years.

The following [1]issue seems similar. I'm also attaching the [2]full
and [3]truncated pcaps of the failed ssh-session, and my [4]ipf.conf.

Maybe someone has some ideas about this.

Thanks,

Petar Bogdanovic

[1] http://sourceforge.net/p/ipfilter/bugs/5/

[2] http://smokva.net/pcap/crane.tgz

[3] tcpdump client (77.X.X.X):
(...)
23:12:30.295355 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5578992:5580440, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.295358 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5580440:5581888, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.295360 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5581888:5583336, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.295361 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5583336:5584784, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.295363 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5584784:5586232, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.295365 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5586232:5587680, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.295374 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5581888, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.295378 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5584784, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.295382 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5587680, win 11303, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.295393 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5587680, win 12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.296126 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5587680:5589128, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.296128 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 5589128:5590576, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.296130 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [FP.], seq 5592024:5592568, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 544
23:12:30.296139 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5590576, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.296145 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5590576, win 12027, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], length 0
23:12:30.296153 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5590576, win 12389, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], length 0
23:12:30.323294 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087322783, win 0, length 0
23:12:30.329066 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087325679, win 0, length 0
23:12:30.329067 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087328575, win 0, length 0
23:12:30.329975 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087330023, win 0, length 0
23:12:30.331049 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087332919, win 0, length 0
23:12:30.333553 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087335815, win 0, length 0
23:12:30.333554 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087338711, win 0, length 0
23:12:30.333555 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087340159, win 0, length 0
23:12:30.333556 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087343055, win 0, length 0
23:12:30.333557 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087345951, win 0, length 0
23:12:30.333558 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087348847, win 0, length 0
23:12:30.333560 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087348847, win 0, length 0
23:12:30.333561 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087351743, win 0, length 0
23:12:30.333562 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087354639, win 0, length 0
23:12:30.333563 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087356087, win 0, length 0
23:12:30.338040 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087361879, win 0, length 0
23:12:30.338041 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087364775, win 0, length 0
23:12:30.338042 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087364775, win 0, length 0
23:12:30.338043 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087367671, win 0, length 0
23:12:30.338044 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087367671, win 0, length 0
23:12:30.338046 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087367671, win 0, length 0
23:12:30.338047 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087358983, win 0, length 0

tcpdump server (85.X.X.X):
(...)
23:12:30.259749 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5492112, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265152 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5495008, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265157 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5492112, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265405 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5578992:5580440, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265418 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5580440:5581888, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265433 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5581888:5583336, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265449 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5583336:5584784, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265460 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5584784:5586232, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265475 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5586232:5587680, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265490 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5587680:5589128, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265505 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 5589128:5590576, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 1448
23:12:30.265806 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5499352, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265831 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5497904, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.266249 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [FP.], seq 5592024:5592568, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], length 544
23:12:30.269507 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5502248, win 12208, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.269516 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5505144, win 11846, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.270483 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5505144, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.275423 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5508040, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.276292 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5510936, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.277578 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5510936, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.285405 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5513832, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.285804 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5516728, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.285809 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5518176, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286450 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5523968, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286463 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5521072, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286468 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5525416, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286476 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5528312, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286494 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5529760, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.292326 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5532656, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.292343 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5535552, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293450 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5535552, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293474 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5538448, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293788 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5541344, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293815 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5542792, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293832 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5545688, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293838 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5545688, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293865 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [P.], seq 6391120:6391168, ack 5545688, win 12389, options [nop,nop,TS val 31 ecr 31], length 48
23:12:30.293902 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087322783, win 0, length 0
23:12:30.299449 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5548584, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.299461 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5551480, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.299473 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087325679, win 0, length 0
23:12:30.299483 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087328575, win 0, length 0
23:12:30.300303 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5552928, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.300314 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087330023, win 0, length 0
23:12:30.301484 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5555824, win 12208, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.301497 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087332919, win 0, length 0
23:12:30.303800 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5558720, win 11846, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303813 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087335815, win 0, length 0
23:12:30.303858 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5561616, win 11484, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303864 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5563064, win 12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303871 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5565960, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303877 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5568856, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303892 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087338711, win 0, length 0
23:12:30.303903 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5571752, win 11303, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303909 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5571752, win 12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303917 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5574648, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303932 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087340159, win 0, length 0
23:12:30.303944 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5577544, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303949 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5578992, win 12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303968 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087343055, win 0, length 0
23:12:30.303978 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087345951, win 0, length 0
23:12:30.303988 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087348847, win 0, length 0
23:12:30.303999 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087348847, win 0, length 0
23:12:30.304009 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087351743, win 0, length 0
23:12:30.304019 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087354639, win 0, length 0
23:12:30.304029 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087356087, win 0, length 0
23:12:30.308639 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5584784, win 11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308646 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5587680, win 11303, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308658 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087361879, win 0, length 0
23:12:30.308668 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087364775, win 0, length 0
23:12:30.308684 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5587680, win 12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308700 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5590576, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308708 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087364775, win 0, length 0
23:12:30.308718 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087367671, win 0, length 0
23:12:30.308735 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5590576, win 12027, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], length 0
23:12:30.308753 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087367671, win 0, length 0
23:12:30.308767 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5590576, win 12389, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], length 0
23:12:30.308780 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087367671, win 0, length 0
23:12:30.308791 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5581888, win 12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308803 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087358983, win 0, length 0

[4] ipf.conf
###
pass in quick on lo0 all
pass out quick on lo0 all

###
block in quick all with mbcast

###
block in quick from 192.168.0.0/16 to any
block in quick from 172.16.0.0/12 to any
block in quick from 10.0.0.0/8 to any
block in quick from 127.0.0.0/8 to any
block in quick from 0.0.0.0/8 to any
block in quick from 169.254.0.0/16 to any
block in quick from 192.0.2.0/24 to any
block in quick from 204.152.64.0/23 to any
block in quick from 224.0.0.0/3 to any

###
block out quick from any to 192.168.0.0/16
block out quick from any to 172.16.0.0/12
block out quick from any to 10.0.0.0/8
block out quick from any to 127.0.0.0/8
block out quick from any to 0.0.0.0/8
block out quick from any to 169.254.0.0/16
block out quick from any to 192.0.2.0/24
block out quick from any to 204.152.64.0/23
block out quick from any to 224.0.0.0/3

###
block in all
block out all

###
pass in proto icmp icmp-type 3
pass out proto icmp icmp-type 3

###
pass in proto icmp icmp-type 8 keep state
pass out proto icmp icmp-type 8 keep state

###
block return-rst in proto tcp
block return-icmp in proto udp

###
pass in proto esp from any to any
pass out proto esp from any to any

###
pass in proto tcp from any to any port = 22000 flags S keep state keep frags
pass in proto tcp from any to any port = ssh flags S keep state keep frags

###
pass in proto udp from any to any port = domain keep state
pass out proto udp from any to any port = domain keep state
pass in proto tcp from any to any port = domain flags S keep state keep frags
pass out proto tcp from any to any port = domain flags S keep state keep frags

###
pass in proto tcp from any to any port = http flags S keep state keep frags

###
pass in proto tcp from any to any port = smtp flags S keep state keep frags
pass out proto tcp from any to any port = smtp flags S keep state keep frags

###
pass in proto tcp from any to any port = submission flags S keep state keep frags

###
pass in proto tcp from any to any port = imap flags S keep state keep frags
pass in proto tcp from any to any port = imaps flags S keep state keep frags

###
pass out proto udp from any to any port = ntp keep state
pass out proto tcp from any to any port = ntp flags S keep state keep frags

###
pass out proto udp from any to any port = 6277 keep state

###
pass out proto tcp from any to any port = 2703 flags S keep state keep frags

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Petar Bogdanovic

2014-06-11 16:06:49 UTC

Permalink

23:12:30.295374 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], (...)
(...)
23:12:30.259749 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], (...)

Note: In order to simplify the pcap output, I replaced the private
address of the client with the public address of the (conventional)
NAT-router. But then I forgot to sync the source ports.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Petar Bogdanovic

2014-06-17 07:25:50 UTC

Permalink

A quick tcpdump revealed that the server side at one point just FINs
the connection and then spams the client with a bunch of TCP resets.

ipmon doesn't seem to register that final FIN. Here are the first and
last few lines of a dropped ssh-session as seen by ipmon. Note how -AF
(or -APF as seen by tcpdump) never happens:

04:00:03.179745 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 64 -S 2750905065 0 32768 K-S K-F IN
04:00:03.179774 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 64 -AS 2937736002 2750905066 32768 K-S K-F OUT
04:00:03.219809 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2750905066 2937736003 4197 K-S K-F IN
04:00:03.232809 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 115 -AP 2937736003 2750905066 4197 K-S K-F OUT
04:00:03.277810 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 115 -AP 2750905066 2937736066 4197 K-S K-F IN
04:00:03.280355 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 852 -AP 2937736066 2750905129 4197 K-S K-F OUT
04:00:03.322407 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 1156 -AP 2750905129 2937736866 4097 K-S K-F IN
04:00:03.515999 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 52 -A 2937736866 2750906233 4197 K-S K-F OUT
04:00:03.555799 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 132 -AP 2750906233 2937736866 4197 K-S K-F IN
04:00:03.561757 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 500 -AP 2937736866 2750906313 4197 K-S K-F OUT
04:00:03.627697 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 68 -AP 2750906313 2937737314 4197 K-S K-F IN
04:00:03.826129 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 52 -A 2937737314 2750906329 4197 K-S K-F OUT
(...)
04:00:27.821939 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392265 2945862410 12027 K-S K-F IN
04:00:27.821948 2x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945955082 2757392265 10341 K-S K-F OUT
04:00:27.821998 3x,re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392265 2945865306 11665 K-S K-F IN
04:00:27.822028 6x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945957978 2757392265 10341 K-S K-F OUT
04:00:27.823735 3x,re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392265 2945871098 12027 K-S K-F IN
04:00:27.823774 4x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945966666 2757392265 10341 K-S K-F OUT
04:00:27.823822 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 100 -AP 2757392265 2945875442 12389 K-S K-F IN
04:00:27.823828 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945878338 12027 K-S K-F IN
04:00:27.823848 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945972458 2757392313 10335 K-S K-F OUT
04:00:27.824013 3x,re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945881234 11665 K-S K-F IN
04:00:27.824069 4x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945973906 2757392313 10341 K-S K-F OUT
04:00:27.824130 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945887026 12027 K-S K-F IN
04:00:27.824140 2x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945979698 2757392313 10341 K-S K-F OUT
04:00:27.824173 3x,re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945889922 11665 K-S K-F IN
04:00:27.824202 4x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945982594 2757392313 10341 K-S K-F OUT
04:00:27.824241 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945895714 12027 K-S K-F IN
04:00:27.824250 2x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945988386 2757392313 10341 K-S K-F OUT
04:00:27.824285 3x,re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945898610 11665 K-S K-F IN
04:00:27.826016 4x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945991282 2757392313 10341 K-S K-F OUT
04:00:27.826078 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945904402 12027 K-S K-F IN
04:00:27.826087 2x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945997074 2757392313 10341 K-S K-F OUT
04:00:27.826120 2x,re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945907298 11665 K-S K-F IN
04:00:27.826143 4x,re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2945999970 2757392313 10341 K-S K-F OUT
04:00:27.826183 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945911642 12027 K-S K-F IN
04:00:27.826192 re0 @0:15 p [85.X.X.X],ssh -> [77.X.X.X],55343 PR tcp len 20 1500 -A 2946005762 2757392313 10341 K-S K-F OUT
04:00:27.826215 re0 @0:15 p [77.X.X.X],55343 -> [85.X.X.X],ssh PR tcp len 20 52 -A 2757392313 2945914538 11665 K-S K-F IN

Is this something for the ipfilter ML?

Full ipmon log: http://pastebin.com/raw.php?i=R3ACgNQa

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Petar Bogdanovic

2014-06-17 06:49:06 UTC

Permalink

A quick tcpdump revealed that the server side at one point just FINs
the connection and then spams the client with a bunch of TCP resets.

Darren Reed

2014-06-17 11:41:41 UTC

Permalink

Post by Petar Bogdanovic
Hi,
about a week ago, the automated daily ssh-tunnels to a netbsd-6 box
started to close shortly after they were established (client said
Network is unreachable"). A quick tcpdump revealed that the server side
at one point just FINs the connection and then spams the client with a
bunch of TCP resets.
After a while of tcpdump and ktrace, disabling ipfilter (v4.1.34) solved
the problem. Which was very confusing, because its ipf.conf hasn't
changed for years.

Did anything change or has it really just started happening?

Darren

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Petar Bogdanovic

2014-06-17 12:30:56 UTC

Permalink

Post by Darren Reed

Did anything change or has it really just started happening?

The software on that machine is basically an image generated from
multiple NetBSD filesets, pkgsrc packages and a git managed tree of
files and diffs. No manual changes are ever supposed to happen, just
indirect changes through regenerated images (which then replaces
everything except the data partition).

Therefore it's probably safe to say that "it just started happening".

Last regen was back in april due to the OpenSSL update. The problems
started appearing about two weeks ago. Of course it's possible that the
network environment has changed in the meantime (the system is hosted in
Germany) but neither I nor the support team there can see any problems
with the packet transport.

It doesn't always fail, though. When it fails, it fails after a very
short time.

Here is a list of the last n daily tunnels. As you can see the success
rate got better in the last few days (T=terminated, f=failed):

(...)
| 2014-06-02 16:00:06 | 66 | 3,459,222 | T |
| 2014-06-02 20:00:06 | 89 | 72,499,475 | T |
| 2014-06-03 00:00:06 | 97 | 3,562,668 | T |
| 2014-06-03 04:00:06 | 87 | 70,702,253 | T |
| 2014-06-03 08:00:07 | 53 | 1,482,232 | T |
| 2014-06-03 12:00:05 | 81 | 3,261,943 | T |
| 2014-06-03 16:00:06 | 110 | 72,548,463 | T |
| 2014-06-03 20:00:06 | 151 | 72,599,796 | T |
| 2014-06-04 00:00:06 | 117 | 4,214,858 | T |
| 2014-06-04 04:00:05 | 114 | 72,296,489 | T |
| 2014-06-04 08:00:06 | 71 | 2,970,687 | T |
| 2014-06-04 12:00:06 | 81 | 3,679,148 | T |
| 2014-06-04 16:00:06 | 79 | 3,824,554 | T |
| 2014-06-04 20:00:06 | 89 | 3,891,503 | T |
| 2014-06-05 00:00:06 | 112 | 6,565,550 | T |
| 2014-06-05 04:00:07 | 0 | 0 | f |
| 2014-06-05 08:00:06 | 116 | 71,642,789 | T |
| 2014-06-05 12:00:08 | 0 | 0 | f |
| 2014-06-05 16:00:05 | 0 | 0 | f |
| 2014-06-05 20:00:06 | 0 | 0 | f |
| 2014-06-06 00:00:06 | 0 | 0 | f |
| 2014-06-06 04:00:06 | 0 | 0 | f |
| 2014-06-06 08:00:05 | 2,261 | 86,091,470 | T |
| 2014-06-06 12:00:06 | 0 | 0 | f |
| 2014-06-06 16:00:06 | 5,009 | 3,666,629 | T |
| 2014-06-06 20:00:06 | 63 | 2,660,740 | T |
| 2014-06-07 00:00:05 | 0 | 0 | f |
| 2014-06-07 04:00:06 | 0 | 0 | f |
| 2014-06-07 08:00:07 | 0 | 0 | f |
| 2014-06-07 10:00:46 | 192 | 72,987,380 | T |
| 2014-06-07 12:00:07 | 0 | 0 | f |
| 2014-06-07 16:00:06 | 0 | 0 | f |
| 2014-06-07 20:00:06 | 0 | 0 | f |
| 2014-06-08 00:00:05 | 0 | 0 | f |
| 2014-06-08 04:00:06 | 0 | 0 | f |
| 2014-06-08 08:00:07 | 0 | 0 | f |
| 2014-06-08 12:00:05 | 0 | 0 | f |
| 2014-06-08 16:00:06 | 136 | 97,835,680 | T |
| 2014-06-08 20:00:06 | 50 | 71,133,165 | T |
| 2014-06-09 00:00:06 | 29 | 1,012,296 | T |
| 2014-06-09 04:00:06 | 0 | 0 | f |
| 2014-06-09 08:00:05 | 0 | 0 | f |
| 2014-06-09 12:00:07 | 44 | 1,314,205 | T |
| 2014-06-09 16:00:05 | 53 | 2,742,456 | T |
| 2014-06-09 20:00:06 | 74 | 2,533,244 | T |
| 2014-06-10 00:00:06 | 0 | 0 | f |
| 2014-06-10 04:00:06 | 0 | 0 | f |
| 2014-06-10 08:00:05 | 143 | 72,823,418 | T |
| 2014-06-10 12:00:06 | 0 | 0 | f |
| 2014-06-10 16:00:05 | 0 | 0 | f |
| 2014-06-10 20:00:06 | 0 | 0 | f |
| 2014-06-11 00:00:06 | 0 | 0 | f |
| 2014-06-11 04:00:05 | 0 | 0 | f |
| 2014-06-11 08:00:06 | 0 | 0 | f |
| 2014-06-11 12:00:05 | 219 | 73,231,862 | T |
| 2014-06-11 16:00:06 | 0 | 0 | f |
| 2014-06-11 20:00:05 | 0 | 0 | f |
| 2014-06-12 00:00:05 | 150 | 72,114,481 | T |
| 2014-06-12 04:00:05 | 98 | 71,960,509 | T |
| 2014-06-12 08:00:06 | 44 | 1,135,453 | T |
| 2014-06-12 12:00:06 | 78 | 3,051,787 | T |
| 2014-06-12 16:00:06 | 70 | 2,483,544 | T |
| 2014-06-12 20:00:06 | 79 | 3,227,264 | T |
| 2014-06-13 00:00:06 | 0 | 0 | f |
| 2014-06-13 12:00:09 | 5,216 | 102,545,019 | T |
| 2014-06-13 16:00:06 | 0 | 0 | f |
| 2014-06-13 20:00:06 | 0 | 0 | f |
| 2014-06-14 00:00:06 | 104 | 2,817,145 | T |
| 2014-06-14 04:00:07 | 0 | 0 | f |
| 2014-06-14 08:00:07 | 88 | 71,905,831 | T |
| 2014-06-14 12:00:05 | 54 | 2,427,955 | T |
| 2014-06-14 16:00:06 | 0 | 0 | f |
| 2014-06-14 20:00:05 | 56 | 2,967,200 | T |
| 2014-06-15 00:00:06 | 67 | 3,066,823 | T |
| 2014-06-15 04:00:06 | 0 | 0 | f |
| 2014-06-15 08:00:07 | 102 | 72,676,624 | T |
| 2014-06-15 12:00:06 | 42 | 1,728,665 | T |
| 2014-06-15 16:00:06 | 68 | 1,890,189 | T |
| 2014-06-15 20:00:06 | 88 | 72,158,931 | T |
| 2014-06-16 00:00:05 | 91 | 2,903,265 | T |
| 2014-06-16 04:00:06 | 73 | 71,246,715 | T |
| 2014-06-16 08:00:05 | 0 | 0 | f |
| 2014-06-16 16:00:06 | 157 | 72,627,958 | T |
| 2014-06-16 17:00:02 | 14 | 615,990 | T |
| 2014-06-16 17:00:43 | 13 | 614,651 | T |
| 2014-06-16 20:00:05 | 94 | 72,513,012 | T |
| 2014-06-17 00:00:05 | 68 | 71,048,464 | T |
| 2014-06-17 04:00:06 | 0 | 0 | f |
| 2014-06-17 08:00:05 | 107 | 74,812,559 | T |
| 2014-06-17 12:00:05 | 92 | 74,303,376 | T |

After seeing this and knowing that nothing has changed, I would be very
convinced that this issue has nothing to do with NetBSD and/or ipfilter
but whenever it happens, I can see that FIN leaving the interface.. :)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de