bad interaction between TCP delayed ack and RSTs

[Joanne M Mikkelson]

Hi,

I've recently been poking at a problem I've been having running a server
on NetBSD. The problem occurs with a Windows (XP) client but I don't
think the problem is restricted to Windows TCP peers.

The client writes a few bytes of data to a TCP connection. No response
is expected from the server. Then the client exits. The server is a
basic select loop reading the data from the client. The problem is:
sometimes, when the client exits, NetBSD does not tear down the
connection. So from my server's point of view, the connection is still
open, and it's happily waiting for more data that's not coming.

This is what is happening:

- The client sends some data.
- Delayed ack causes NetBSD to send no ack for this packet.
- The client exits (or closes the socket).
- Windows sends an RST/ACK to close the TCP connection (it does this a
lot, if not most of the time -- I do not know when it uses a FIN).
- When the RST message arrives, NetBSD responds to the RST with an ACK
and then drops the RST.
- The intention must be that the peer will respond to the ACK with
another RST, at which point all the segments have been acked and the
connection will be properly shut down.
- The *reality* is that the Windows firewall drops the post-RST ack.
(If I disable Windows firewall, a second RST does arrive at the server
side and everything is fine.)

The result is that NetBSD believes the connection is still open, and it
will never believe otherwise unless the connection is set up to time
out. And of course my server never gets an ECONNRESET and cannot act
appropriately as a result.

This seems to be the code that is giving me trouble:

if (tiflags & TH_RST) {
if (th->th_seq != tp->last_ack_sent)
goto dropafterack_ratelim;

(this is tcp_input.c revision 1.291 lines 2225-2227 (from NetBSD 5.0)).

I assume this code is intended as defense against RST spoofing? The log
message (revision 1.194) for this says: "respond to RST by ACK, as
suggested in NISCC recommendation". I can't find any recommendation like
this; advisory 236929 seems likely but it makes no such recommendations
(its primary recommendation seems to be to use IPsec!).

So I'm left assuming that the theory is that if it's an attack RST, the
connection will not be torn down, and if it's a genuine RST, the peer
will respond to the ACK with another RST. But here I have genuine RSTs,
in a situation that is not really all that strange, and the connection
isn't being torn down.

I just don't think it's right for a connection to not close down solely
due to the fact that it's in the middle of doing a delayed ack. (If
there's no delayed ack, everything is fine with the same firewall and
peer.) Does this sound like a bug to anyone else?

I thought about it for a while and it seems like in the case where
there's a delayed ack, the RST should be able to close down the
connection as if the ack hadn't been delayed in the first place. I
tested that out (more below) and it did fix my problem.

But upon further thought I'm not sure if that is even correct. What if
the client sent data and that packet was lost, and then the client
exits? In that case delayed ack isn't the problem, but it would still
leave the connection open on the server... Maybe any RST that arrived
within the window should be allowed to close the connection? That would
reduce the resistance to RST attacks a lot more than just allowing for
the delayed ack case, though.

I am somewhat surprised it's been over 5 years since this code was added
and I haven't been able to find any complaint about this! This problem
leaves my server using up resources indefinitely if nothing else. And it
continues to be annoying even if the server detects the situation by
other means and closes the socket, because the connection still hangs
around for a long time forlornly sending FINs to Windows which drops
them all.

Fortunately I am not trying to run a big server talking to lots of
clients that send RST to close the connection down after sending data
that doesn't expect a response, and apparently neither is anyone
else. :-)


This is the code change I tried out:

if (tiflags & TH_RST) {
if (th->th_seq != tp->last_ack_sent
&& !((tp->t_flags & TF_DELACK)
&& th->th_seq == tp->rcv_nxt))
goto dropafterack_ratelim;

I'm not positive this is exactly correct because I am not familiar with
all the details of this code, but the idea was to drop the packet only
if the sequence number doesn't match the last ack *and* the sequence
number doesn't match what would be the last ack if everything had been
acked.

I wrote simple programs to test this out: a server which selects on and
reads from a socket, and a client to run on Windows which sends 150
bytes of data and then immediately closes the socket. (I can't run the
client on NetBSD because FINs are sent.) With these programs it is
trivial to reproduce the problem. The above code *does* make my server
see the connection reset when it didn't before. (I have the code and
packet traces if they are needed.)


A workaround for me is to set net.inet.tcp.ack_on_push=1 since
empirically I see there is a PSH flag on the packet with the data, but
can we assume that in the general case? If it's agreed that this is a
bug that should be fixed, I can put in a PR if necessary for
it. Hopefully if so it can be fixed by someone who knows the code better
than me; I'm unsure, for example, whether a similar change should be
done at lines 2099-2105...

Joanne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Ignatios Souvatzis]

Post by Joanne M Mikkelson
- The client sends some data.
- Delayed ack causes NetBSD to send no ack for this packet.
- The client exits (or closes the socket).
- Windows sends an RST/ACK to close the TCP connection (it does this a
lot, if not most of the time -- I do not know when it uses a FIN).

Ahem. IMHO, the problem isn't that a RST is lost. It is that it is
sent at all!

Closing down a TCP connection is supposed to happen with
->FIN, <- FIN+ACK, ->ACK if I recall correctly.

This takes care of lost packets, retransmissions, etc. That's why
there are the FIN_WAIT_$N states.

Sending a RST is declaring that the TCP connection means "what ever
you (the receiver) hallucinate about this TCP connection is not
valid at my end, period." for an apparently open (to the receiver)
connection ( or "I don't want to open that connection" for a TCP
SYN).

So it's faster, the RST sending site doesn't have to keep state
about the connection, and it just doesn't work correctly - because
if it arrives at the other end before the unacknowledged data have
been handled, they are lost.

I believe the RST checking foo in our code is for a partial defense
against 3rd-party rogue RSTs. It's not meant, and can't be, to ACK
data-on-the-fly when a RST is receive- because by definition, when
a RST arrives, there is no consensus about any data possibly being
on the fly at all.

So either the application programmers have forgotten to do some
proper TCP flushing/closing call (I don't know the Windows API),
or Windows does a local optimization at the expense of reliability,
or some "firewall" in between does something very fishy.

Maybe one of our TCP gurus can correct be, but I think what I wrote
is in any Introduction To Networking. See, e.g., Tanenbaum, Computer
Networks, 3rd Edition, p. 532.

-is

[Joanne M Mikkelson]

Post by Ignatios Souvatzis
Ahem. IMHO, the problem isn't that a RST is lost. It is that it is
sent at all!

No RSTs were lost in the creation of my problem. :-)

I will not argue with the assertion that the connection *should* be
shut down with a FIN. However, I am reporting what is *actually*
happening.

Post by Ignatios Souvatzis
I believe the RST checking foo in our code is for a partial defense
against 3rd-party rogue RSTs. It's not meant, and can't be, to ACK
data-on-the-fly when a RST is receive- because by definition, when

I have several packet traces showing NetBSD sending an ACK after the
RST. And I can generate more at will. If it's a third party (attack)
RST the ACK is harmless. If it's not a third party RST the ACK will
generate another RST, unless there's a firewall dropping the ACK.

Post by Ignatios Souvatzis
So either the application programmers have forgotten to do some
proper TCP flushing/closing call (I don't know the Windows API),

My test program calls close() on the socket. It demonstrates the
problem with 100% reliability. I believe that if it were written with
the Windows API instead of using POSIX compatibility it would do the
same. If it becomes necessary I will rewrite the test program
tomorrow night to do so.

Post by Ignatios Souvatzis
or Windows does a local optimization at the expense of reliability,
or some "firewall" in between does something very fishy.

By the way, Windows is not the only OS which will send RST to close a
connection. For example, I wrote this a couple years ago while
working with an unrelated problem:

I tested a bit with the FINs and RSTs and apparently on Linux
(2.4.something), the TCP connection is closed with a RST whenever
the user-space process has *not* read all the data on the socket
before the socket is closed. The only way I could get Linux to
send a FIN was to read all the data before closing.

(I went on to say I could not get NetBSD to send an RST due to an
action by an application.) At least in this case the application
could be changed, presumably, to "fix" the RST problem, but the point
is, here is an example of something other than Windows sending
RSTs.

I thought NetBSD does want to work as well as possible with other TCP
implementations.

Given the reality of an application doing the perfectly reasonable
thing of calling close() on an extremely common OS using a firewall
with rather common behavior (dropping packets on closed connections),
I still find it strange that a delayed ack causes my server to behave
poorly when, given the same RST and firewall behavior, it works
correctly when there isn't a delayed ack.

Joanne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Ignatios Souvatzis]

Post by Joanne M Mikkelson

Post by Ignatios Souvatzis
Ahem. IMHO, the problem isn't that a RST is lost. It is that it is
sent at all!

No RSTs were lost in the creation of my problem. :-)
I will not argue with the assertion that the connection *should* be
shut down with a FIN. However, I am reporting what is *actually*
happening.

Post by Ignatios Souvatzis
I believe the RST checking foo in our code is for a partial defense
against 3rd-party rogue RSTs. It's not meant, and can't be, to ACK
data-on-the-fly when a RST is receive- because by definition, when

I have several packet traces showing NetBSD sending an ACK after the
RST. And I can generate more at will. If it's a third party (attack)
RST the ACK is harmless. If it's not a third party RST the ACK will
generate another RST, unless there's a firewall dropping the ACK.

Well... the problem is still the RST.

If the RST wasn't abused, the firewall's behaviour would be
reasonable.

But yes, maybe there should be an easy way to defend against this
behaviour - but weakening the rogue RST defense is a dangerous
thing to do.

Less dangerous seem:

Ack-on-push, if thats enough for you
SO_KEEPALIVE on the server sockets.

Ignatios

[Miles Nordin]

is> If the RST wasn't abused, the firewall's behaviour would be
is> reasonable.

It seems like you're not listening. She says Linux sends an RST the
same way when you call close() on a socket with unread data, and it
seems reasonable to me---why bother acknowledging data that can never
be read?

The firewall and NetBSD are both playing around with the
interpretation of TCP. The problem is, NetBSD's allowed to, and it's
the firewall's job to accomodate NetBSD, not the other way round.
We've allowed this to become inverted, and now TCP design has been
taken over by crazy irresponsible people bouncing off the walls
blocking passing and rewriting whatever they like with a blank check
that says ``securitah,'' a bunch of incomplete hypothetical stories,
no consensus, no testing, no interop pressure. Many TCP improvements
including this one are about security, but TCP designers don't get the
blank check.

Anyway there's no reason the firewall can't send an RST itself. no
good reason---I'm sure they will make one up, involving ``spoofing''
again.

jmm> an extremely common OS using a firewall with rather common
jmm> behavior

uh oh. here we go...

Is there some way to build a NetBSD firewall to notice the
accumulation of stale connections not eliciting RST's trying to DoS
the server, and start blocking the Windows host entirely? Then it can
be your broken firewall vs. my broken firewall.

If you can't beat 'em...

[Ignatios Souvatzis]

Post by Miles Nordin
is> If the RST wasn't abused, the firewall's behaviour would be
is> reasonable.
It seems like you're not listening. She says Linux sends an RST the
same way when you call close() on a socket with unread data, and it
seems reasonable to me---why bother acknowledging data that can never
be read?

Please forgive me. You are both right in this respect; this behaviour
is in the official protocol.

Post by Miles Nordin
The firewall and NetBSD are both playing around with the
interpretation of TCP. The problem is, NetBSD's allowed to, and it's
the firewall's job to accomodate NetBSD, not the other way round.

As I wrote in my other message: we're actually changing the protocol
as in a recommendation then (and still) discussed by the IETF TCP
maintainance working group. So yes, to a certain degree we are to
blame, but what to do about it? After all, packets (e.g. the ACK
that is intended to provoke the 2nd RST, or the 2nd RST) can be
lost for ... legitimate (well, non-firewall-related) reasons.

Maybe we should start a timer that guards against this condition,
and tears down our end of the connection anyway after some reasonable
time, but what is reasonable, and what are the security implications
in this case?

I guess this issue should be discussed with IETF TCPM.

-is

[der Mouse]

Post by Joanne M Mikkelson
- The client sends some data.
- Delayed ack causes NetBSD to send no ack for this packet.
- The client exits (or closes the socket).
- Windows sends an RST/ACK to close the TCP connection (it does this
a lot, if not most of the time -- I do not know when it uses a
FIN).

Problem one; I agree with Ignatios in that respect. However, while
this is liable to lose unacked-and-unread data, that's the only effect
it should have.

Post by Joanne M Mikkelson
- When the RST message arrives, NetBSD responds to the RST with an
ACK and then drops the RST.
- The intention must be that the peer will respond to the ACK with
another RST, [...]

If the first RST was real, yes.

Post by Joanne M Mikkelson
- The *reality* is that the Windows firewall drops the post-RST ack.

It is broken, at least for TCP's use. The RST in response to packets
to nonexistent connections is there for a reason; failing to generate
it breaks TCP, leading to problems such as the one you're seeing.
Either you will need to configure the firewall so endpoints behind it
behave, as seen by the network, the way TCP expects them to behave or
you will have to live with the brokenness that results. Even if NetBSD
didn't have the third-party-RST defense, the same behaviour would
result if the Windows-sent RST got dropped in the network.

Post by Joanne M Mikkelson
This seems to be the code that is giving me trouble: [...]

I would say, rather, that it is the code that's showing up the
bustification in the firewall.

Post by Joanne M Mikkelson
I just don't think it's right for a connection to not close down
solely due to the fact that it's in the middle of doing a delayed
ack.

It's not. It's due to that _plus_ the presence of a misconfigured
(and/or misimplemented) firewall breaking the assumptions TCP is built
upon.

While it's not really my call to make, I think it's a bad idea to
contrive NetBSD in an attempt to compensate for someone else's breaking
the underlying protocol. That's a very slippery slope with lots of
catastrophes waiting to happen. If you really really need to
interoperate with the behaviour exhibited by your
Windows-plus-firewall, you will, first, need to know exactly what
behaviour it exhibits - this is unlikely to be the only respect in
which it breaks TCP - and redesign TCP in light of that, then implement
the resulting not-quite-TCP. Just don't mistake it for TCP.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joanne M Mikkelson]

So, the summary so far appears to be this:

- It is okay for NetBSD to change TCP behaviors but if others change
different behaviors and the two conflict, the other guy is the
broken one.
- Firewalls that drop packets without sending RST are broken.

I am not aware of any TCP standard that says sending ACK in response
to RST and not tearing down the connection is correct. What I see in
RFC 793 says the connection must be closed upon receiving an
acceptable RST in the window. See the end of section 3.4 and "Event
Processing" page 70. I couldn't find the mentioned NISCC
recommendation.

The Linux behavior of sending RST when a socket is close()d without
reading all the data is described as a MAY in RFC 1122, section
4.2.2.13. A perfectly conforming TCP can thus receive data, send
data, close() without reading the data, and send a legitimate RST.

I understand why NetBSD is filtering the RSTs, but in light of these
RFCs, claiming that NetBSD is not the broken one and the TCP sending
the RST is the broken one seems wrong. The thing is, I didn't come
here to have a silly argument about whose TCP stack is most like the
specification. I don't even care much about my application.

I *was* just trying to help NetBSD behave in the best possible way by
pointing out this problem.

I also find it difficult to believe that reducing the number of
sequence numbers a forged RST will fail with from (2^32)-1 to
(2^32)-2 is a noteworthy weakening of resistance to such attacks. And
in my code, it even requires that there's currently a delayed ack
before "accepting" that second sequence number. I did think about
this; in my first message I explicitly mentioned that letting through
RSTs in the window would be a reduction in resistance, because with
window scaling that could actually start to add up.

As for the firewall, my personal observations are that there are an
awful lot of firewalls, "home networking" NAT boxes, whatever, out
there configured to *not* send RSTs when dropping packets, despite
what TCP wants. So, all told, I am no fan of Windows but you will not
convince me this is a "Windows is busted, end of story" situation.


At any rate, my problem appears to be not considered a bug.
Well... that's why I asked. For now I'll just add my solution to
my ever-growing set of private kernel and userland patches.

But this illuminating exchange will definitely inform my decisions if
I ever run a server more important than the current two-bit server
that a few friends and I use. Since I do not presume to "fix" all the
firewalls and Windows machines on the 'net, I would have to find a
different option...

Joanne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Ignatios Souvatzis]

Hi Joanne,

I found time to look up the relevant NetBSD Security Advisory, which

Post by Joanne M Mikkelson
- The client sends some data.
- Delayed ack causes NetBSD to send no ack for this packet.
- The client exits (or closes the socket).
- Windows sends an RST/ACK to close the TCP connection (it does this a
lot, if not most of the time -- I do not know when it uses a FIN).
- When the RST message arrives, NetBSD responds to the RST with an ACK
and then drops the RST.

...

Post by Joanne M Mikkelson
I assume this code is intended as defense against RST spoofing? The log
message (revision 1.194) for this says: "respond to RST by ACK, as
suggested in NISCC recommendation". I can't find any recommendation like
this; advisory 236929 seems likely but it makes no such recommendations
(its primary recommendation seems to be to use IPsec!).

You'll find the full official explanation from the NetBSD side in our
security advisory SA2004-006[1].

As far as I can tell, the behaviour is roughly along the lines of
draft-ietf-tcpm-tcpsecure-11.txt[2] (was -0.txt back then). I'll
have to read more code to tell exactly, especialy as [2] has evolved
and is still not finalized.

So you're right, NetBSD has changed the protocol from RFC 793.

We have done so in a manner which we expected to be official shortly
afterwards. Only it still is in the discussion state...

Regards,
-is

[1] ftp://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2004-006.txt.asc
[2] http://tools.ietf.org/html/draft-ietf-tcpm-tcpsecure-11

[Ignatios Souvatzis]

Hi,

Post by Joanne M Mikkelson
Fortunately I am not trying to run a big server talking to lots of
clients that send RST to close the connection down after sending data
that doesn't expect a response, and apparently neither is anyone
else. :-)
if (tiflags & TH_RST) {
if (th->th_seq != tp->last_ack_sent
&& !((tp->t_flags & TF_DELACK)
&& th->th_seq == tp->rcv_nxt))
goto dropafterack_ratelim;
I'm not positive this is exactly correct because I am not familiar with
all the details of this code, but the idea was to drop the packet only
if the sequence number doesn't match the last ack *and* the sequence
number doesn't match what would be the last ack if everything had been
acked.

Ok, after reading lots of stuff, I reluctantly[1] agree that this
seems desirable.

Just for the record 1: FreeBSD did a similar (in spirit) adjustment
to their RST spoof defence code (which uses a different strategy);
see their tcp_input.c, 1.316 and their log entry.

Just for the record 2: I'm not sure we do all the sequence number
checks for RST that draft-ietf-tcpm-tcpsecure suggests before the
ack-probe is sent (else the RST segement should be silently dropped!).
However, this is a different issue, and should probably only
addressed after draft-ietf-tcpm-tcpsecure is formally accepted as
PROPOSED STANDARD by IETF.

[1] I suspect we might need a generic defense against blackholed RSTs
leading to accumulated waiting server processes; but then again,
maybe the servers in question should have their own timeout handling
outside of the kernel.

Regards,
-is

[der Mouse]

Post by Ignatios Souvatzis
[1] I suspect we might need a generic defense against blackholed RSTs
leading to accumulated waiting server processes; but then again,
maybe the servers in question should have their own timeout handling
outside of the kernel.

They usually should, because a client machine that crashes (or loses
power or connectivity) without warning can lead to the same kind of
half-open connection.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[der Mouse]

Post by Ignatios Souvatzis
[1] I suspect we might need a generic defense against blackholed RSTs
leading to accumulated waiting server processes; but then again,
maybe the servers in question should have their own timeout handling
outside of the kernel.

They usually should, because a client machine that crashes (or loses
power or connectivity) without warning can lead to the same kind of
half-open connection.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joanne M Mikkelson]

Post by Ignatios Souvatzis
You'll find the full official explanation from the NetBSD side in our
security advisory SA2004-006[1].
As far as I can tell, the behaviour is roughly along the lines of
draft-ietf-tcpm-tcpsecure-11.txt[2] (was -0.txt back then). I'll
have to read more code to tell exactly, especialy as [2] has evolved
and is still not finalized.

Thank you for both of these references! I would definitely have been
less puzzled about the behavior and what to do if I had known about
the I-D.

If NetBSD is trying to follow these tcpsecure recommendations, then I
think my problem is clearly the result of a bug. From that draft:

2) If the RST bit is set and the sequence number exactly matches the
next expected sequence number (RCV.NXT), then TCP MUST reset the
connection.

Version 00 said the same thing, although much less clearly.

NetBSD isn't testing against RCV.NXT, it's testing the last ACK sent.
Following the recommendation, I think the code should simply read:
if (tiflags & TH_RST) {
if (th->th_seq != tp->rcv_next) {
goto dropafterack_ratelim;

I've tested this version and it works for my test case. Certainly I
like this change better as it eliminates whether delayed ack is a
factor, which does make a lot more sense. And it's implementing the
proposed spec, if I'm understanding this code correctly.

Now I believe that the same test earlier (lines 2099-2105 of revision
1.291) should be changed as well. I haven't looked at the rest of the
RST-handling requirements in detail against the code either, so there
could be more, but these two would kill the problem I've been seeing.

Post by Ignatios Souvatzis
[1] I suspect we might need a generic defense against blackholed RSTs
leading to accumulated waiting server processes; but then again,
maybe the servers in question should have their own timeout handling
outside of the kernel.

Yes, my server has had this timeout from the beginning. In fact, the
close() after the timeout actually caused the first symptom I
noticed: "Why am I sending all these FINs? The client disconnected
several minutes ago."

Joanne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Joanne M Mikkelson]

Whoops... That code snippet should read:

if (tiflags & TH_RST) {
if (th->th_seq != tp->rcv_nxt)
goto dropafterack_ratelim;

(That's what I get for not copy and pasting.)

Joanne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

jmm> - It is okay for NetBSD to change TCP behaviors but if others
jmm> change different behaviors and the two conflict, the other guy
jmm> is the broken one.

I believe NetBSD's TCP implementation is, like Windows and most
others, quite ``forgiving'' in that it aims to, and largely succeeds,
in bending over backwards to accomodate neighbor's behavior, including
in this case.

That's not true of the combination of ([most any TCP implementation] +
[most firewalls]), which are designed to trip and block everything if
they spot the slightest thing they regard as hand-wavingly odd, and
they make up whatever rules they like to define what's odd without the
hypersensitivity to RFC properness you're trying to grind into NetBSD,
and answer to no one for it.

Ignore this difference, and twist the situation around to satisfy a
practical desire to blame the end which you have some hope of changing
and exhonerate the proprietary inflexible end, at the peril of the
entire fucking internet.

Yeah, in the end, the ``extremely common'' unchangeable bad neighbor
will probably need accomodating just like all the broken drop-all-ICMP
firewalls that fucked everyone over PPPoE had to be accomodated. You
and I both understand that part. But that's no reason to distort
*why* it should be done: this appologetic doublethink crap should
stop.

[Ignatios Souvatzis]

Post by Joanne M Mikkelson
- The client sends some data.
- Delayed ack causes NetBSD to send no ack for this packet.
- The client exits (or closes the socket).
- Windows sends an RST/ACK to close the TCP connection (it does this a
lot, if not most of the time -- I do not know when it uses a FIN).

Ahem. IMHO, the problem isn't that a RST is lost. It is that it is
sent at all!

Closing down a TCP connection is supposed to happen with
->FIN, <- FIN+ACK, ->ACK if I recall correctly.

This takes care of lost packets, retransmissions, etc. That's why
there are the FIN_WAIT_$N states.

Sending a RST is declaring that the TCP connection means "what ever
you (the receiver) hallucinate about this TCP connection is not
valid at my end, period." for an apparently open (to the receiver)
connection ( or "I don't want to open that connection" for a TCP
SYN).

So it's faster, the RST sending site doesn't have to keep state
about the connection, and it just doesn't work correctly - because
if it arrives at the other end before the unacknowledged data have
been handled, they are lost.

I believe the RST checking foo in our code is for a partial defense
against 3rd-party rogue RSTs. It's not meant, and can't be, to ACK
data-on-the-fly when a RST is receive- because by definition, when
a RST arrives, there is no consensus about any data possibly being
on the fly at all.

So either the application programmers have forgotten to do some
proper TCP flushing/closing call (I don't know the Windows API),
or Windows does a local optimization at the expense of reliability,
or some "firewall" in between does something very fishy.

Maybe one of our TCP gurus can correct be, but I think what I wrote
is in any Introduction To Networking. See, e.g., Tanenbaum, Computer
Networks, 3rd Edition, p. 532.

-is

[Ignatios Souvatzis]

Hi Joanne,

I found time to look up the relevant NetBSD Security Advisory, which

Post by Joanne M Mikkelson
- The client sends some data.
- Delayed ack causes NetBSD to send no ack for this packet.
- The client exits (or closes the socket).
- Windows sends an RST/ACK to close the TCP connection (it does this a
lot, if not most of the time -- I do not know when it uses a FIN).
- When the RST message arrives, NetBSD responds to the RST with an ACK
and then drops the RST.

...

Post by Joanne M Mikkelson
I assume this code is intended as defense against RST spoofing? The log
message (revision 1.194) for this says: "respond to RST by ACK, as
suggested in NISCC recommendation". I can't find any recommendation like
this; advisory 236929 seems likely but it makes no such recommendations
(its primary recommendation seems to be to use IPsec!).

You'll find the full official explanation from the NetBSD side in our
security advisory SA2004-006[1].

As far as I can tell, the behaviour is roughly along the lines of
draft-ietf-tcpm-tcpsecure-11.txt[2] (was -0.txt back then). I'll
have to read more code to tell exactly, especialy as [2] has evolved
and is still not finalized.

So you're right, NetBSD has changed the protocol from RFC 793.

We have done so in a manner which we expected to be official shortly
afterwards. Only it still is in the discussion state...

Regards,
-is

[1] ftp://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2004-006.txt.asc
[2] http://tools.ietf.org/html/draft-ietf-tcpm-tcpsecure-11

[Joanne M Mikkelson]

So, the summary so far appears to be this:

- It is okay for NetBSD to change TCP behaviors but if others change
different behaviors and the two conflict, the other guy is the
broken one.
- Firewalls that drop packets without sending RST are broken.

I am not aware of any TCP standard that says sending ACK in response
to RST and not tearing down the connection is correct. What I see in
RFC 793 says the connection must be closed upon receiving an
acceptable RST in the window. See the end of section 3.4 and "Event
Processing" page 70. I couldn't find the mentioned NISCC
recommendation.

The Linux behavior of sending RST when a socket is close()d without
reading all the data is described as a MAY in RFC 1122, section
4.2.2.13. A perfectly conforming TCP can thus receive data, send
data, close() without reading the data, and send a legitimate RST.

I understand why NetBSD is filtering the RSTs, but in light of these
RFCs, claiming that NetBSD is not the broken one and the TCP sending
the RST is the broken one seems wrong. The thing is, I didn't come
here to have a silly argument about whose TCP stack is most like the
specification. I don't even care much about my application.

I *was* just trying to help NetBSD behave in the best possible way by
pointing out this problem.

I also find it difficult to believe that reducing the number of
sequence numbers a forged RST will fail with from (2^32)-1 to
(2^32)-2 is a noteworthy weakening of resistance to such attacks. And
in my code, it even requires that there's currently a delayed ack
before "accepting" that second sequence number. I did think about
this; in my first message I explicitly mentioned that letting through
RSTs in the window would be a reduction in resistance, because with
window scaling that could actually start to add up.

As for the firewall, my personal observations are that there are an
awful lot of firewalls, "home networking" NAT boxes, whatever, out
there configured to *not* send RSTs when dropping packets, despite
what TCP wants. So, all told, I am no fan of Windows but you will not
convince me this is a "Windows is busted, end of story" situation.


At any rate, my problem appears to be not considered a bug.
Well... that's why I asked. For now I'll just add my solution to
my ever-growing set of private kernel and userland patches.

But this illuminating exchange will definitely inform my decisions if
I ever run a server more important than the current two-bit server
that a few friends and I use. Since I do not presume to "fix" all the
firewalls and Windows machines on the 'net, I would have to find a
different option...

Joanne

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de