TCP SYN Cookies for NetBSD

[Paul Goyette]

While at the MeetBSD California un-conference over the weekend, I was
approached by the originator of [1]. Looking through the archives, I
don't see any replies or discussion, so I was wondering (along with
John) if there's any merit to the suggested code/patches? Has anyone
with TCP expertise reviewed them at all?

If the code is "close" (FSV of "close"), is there any chance of using
the Google Code-In as a chance to get clean-up or finishing touched?

[1] http://mail-index.netbsd.org/tech-net/2012/04/28/msg003259.html



-------------------------------------------------------------------------
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com |
| Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net |
| Kernel Developer | | pgoyette at netbsd.org |
-------------------------------------------------------------------------

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[je]

Post by Paul Goyette
While at the MeetBSD California un-conference over the weekend, I was
approached by the originator of [1]. Looking through the archives, I
don't see any replies or discussion, so I was wondering (along with
John) if there's any merit to the suggested code/patches? Has anyone
with TCP expertise reviewed them at all?
If the code is "close" (FSV of "close"), is there any chance of using
the Google Code-In as a chance to get clean-up or finishing touched?
[1] http://mail-index.netbsd.org/tech-net/2012/04/28/msg003259.html
-------------------------------------------------------------------------
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com |
| Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net |
| Kernel Developer | | pgoyette at netbsd.org |
-------------------------------------------------------------------------

Paul,

Thanks for meeting with me at the conference and sending this follow-up.
I'm currently working to insure the code patches cleanly against 6.0 and
am adding the ability to auto-enable cookies based on how full the syn
cache is.

What I would most appreciate is a quick code review and some testing. I
believe everything is functional but I am new to NetBSD and I might not
have done things "the NetBSD way".

Thanks,
John

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

[...SYN cookies...]

Don't they break TCP's retransmission semantics? Certainly SYN cookies
as I understand them do. If the third packet of the three-way
handshake (the pure ACK) is lost, neither end is going to retransmit
ever, the active host because it thinks it has an established
connection and the passive host because it has - this is the whole
point of SYN cookies - no state to retransmit based on.

Thus, we have a half-open connection. If the active peer sends data
without expecting anything from the passive peer first, I'd expect an
RST. If the other way around, the connection is permanently wedged.

I don't consider either consequence acceptable.

It's not obvious to me from the patches - are these SYN cookies
something else?

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[je]

Post by Mouse

[...SYN cookies...]

Don't they break TCP's retransmission semantics? Certainly SYN cookies
as I understand them do. If the third packet of the three-way
handshake (the pure ACK) is lost, neither end is going to retransmit
ever, the active host because it thinks it has an established
connection and the passive host because it has - this is the whole
point of SYN cookies - no state to retransmit based on.
Thus, we have a half-open connection. If the active peer sends data
without expecting anything from the passive peer first, I'd expect an
RST. If the other way around, the connection is permanently wedged.
I don't consider either consequence acceptable.
It's not obvious to me from the patches - are these SYN cookies
something else?
/~\ The ASCII Mouse
\ / Ribbon Campaign
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

Yes, that situation is possible. However I wouldn't go as far as to say
a connection is "permanently" wedged as eventually the client or the
server will attempt to transmit data, a keepalive if the connection was
never intended to transmit data, or the connection will time out due to
inactivity. If an RST is generated the peer will do the right thing and
clean up the connection.

This is obviously not ideal, but as a method to avoid syn cache
exhaustion (or to avoid creating a syn cache at all) I believe it is
acceptable. I also believe leaving syn cookies as an option (compile
time, sysctl, etc) for the user will give reasonable flexibility for
people who don't like this tradeoff.

John

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Robert Elz]

Date: Mon, 5 Nov 2012 21:03:32 -0500 (EST)
From: Mouse <***@Rodents-Montreal.ORG>
Message-ID: <***@Sparkle.Rodents-Montreal.ORG>

| If the third packet of the three-way handshake (the pure ACK) is lost,
| neither end is going to retransmit ever,

That's true, but relatively harmless, in that TCP (or the apps) need
to have some mechanism to recover from this anyway, as it is a state
that TCP can get into even without SYN cookies (a server that uses
SYN cookies is no different from a server that has crashed just after
sending the 2nd packet of the 3 way handshake, causing the 3rd packet,
the ACK, to be lost, along with all state at the server).

With SYN cookies, the server certainly doesn't care when the ACK is lost,
(as you say, it has no state, so never even realises anything happened),
and the client has to be able to recover anyway, so all use of SYN
cookies really does is (possibly) increase the likelihood that the
recovery mechanism (whatever it is) will be exercised more frequently.

If there's no recovery mechanism, then enabling SYN cookie use on
the server is a good thing, as it will highlight the apps that need
remedial work of one for or another (which might be as simple as
a timeout on the wait for the initial message from the server, if
that is the way the protocol is designed).

kre

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

Post by Robert Elz
If the third packet of the three-way handshake (the pure ACK) is
lost, neither end is going to retransmit [...]. Thus, we have a
half-open connection.

Yes, that situation is possible. However I wouldn't go as far as to
say a connection is "permanently" wedged as eventually the client or
the server will attempt to transmit data,

The passive end won't because it doesn't have a connection yet, and
it's thrown away the only state that could have provoked it to
retransmit. The client end might or might not, depending on the
protocol in question; there certainly is no reason to be confident all
protocols will have the active end sending something in the face of
silence from the passive end.

a keepalive if the connection was never intended to transmit data,

Only if the active end has keepalives turned on (something which is out
of the control of the SYN-cookie-using passive end).

or the connection will time out due to inactivity

If you're talking about TCP-level timeouts, this happens only if
someone has data to send. If the active peer has data to send, you'll
get an RST, presumably relatively quickly, without need for any
inactivity timeout; the passive peer can't have data to send yet
because it doesn't have a connection yet. Higher-level timeouts depend
on the protocol in use; there's no reason to assume they will all have
timeouts.

So, if the third packet of the handshake is lost, the best case is that
the connection gets shot down by an RST; the worst case is that it
mysteriously hangs forever. And the things that can keep it from
hanging forever in the latter case are completely out of the control of
the peer with the broken code.

And, of course, both problems get worse as packet loss rates go up -
exactly the worst time to impose mysterious connection troubles.

If an RST is generated the peer will do the right thing and clean up
the connection.

How is that "the right thing"? Letting the active peer think it has a
connection and then RSTing it as soon as it sends any data doesn't
strike me as a very right thing.

This is obviously not ideal,

Well, of course not. Ignoring the TCP spec isn't, not if you expect to
interoperate. (This even fails the "what if everyone did it" test; the
interoperability failures occur even when it's talking to another such
implementation.)

but as a method to avoid syn cache exhaustion (or to avoid creating a
syn cache at all) I believe it is acceptable.

Not to me. I would consider it a crippling bug if I saw such behaviour
from a TCP stack.

I also believe leaving syn cookies as an option (compile time,
sysctl, etc) for the user will give reasonable flexibility for people
who don't like this tradeoff.

We can't force hosts to follow the spec, of course. And we can't keep
people from hacking whatever brokenness they like into the kernel. But
the spec exists to permit interoperability, and just because people can
break their systems is no excuse to hand them pre-broken systems - I
consider it a bug for a TCP stack to be that easy to break this way.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

Post by Robert Elz

Post by Robert Elz
If the third packet of the three-way handshake (the pure ACK) is
lost, neither end is going to retransmit ever,

That's true, but relatively harmless, in that TCP (or the apps) need
to have some mechanism to recover from this anyway, as it is a state
that TCP can get into even without SYN cookies (a server that uses
SYN cookies is no different from a server that has crashed just after
sending the 2nd packet of the 3 way handshake, causing the 3rd
packet, the ACK, to be lost, along with all state at the server).

Except that it's not; it's no different from a server that crashes
immediately after sending _every_ SYN|ACK packet - except it's a weird
sort of crash that manages to retain enough state to handle a third
packet that _doesn't_ get lost. Or, equivalently, crashing whenever
the network path loses the third packet.

There's a (not unreasonable, I think) expectation that a crash loses
all state for all connections to the crashed host, and for a
significant time. The pseudo-crash involved here exhibits neither of
those properties.

I would also think that appearing to have crashed (especially when it
hasn't) would be something you'd want to get rid of, rather than cause.
I never thought I'd see you (apparently) seriously saying that it's a
good thing to add code so that a host can appear to have crashed (and a
very unusual kind of crash, too, not what is normally called a crash)
on a condition as mild as a network path losing a packet.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Robert Elz]

Date: Mon, 5 Nov 2012 22:26:59 -0500 (EST)
From: Mouse <***@Rodents-Montreal.ORG>
Message-ID: <***@Sparkle.Rodents-Montreal.ORG>

| Except that it's not; it's no different from a server that crashes
| immediately after sending _every_ SYN|ACK packet

No, only the ones where the returning ACK is lost by the network.
That is, it is the same as a server that crashes sometimes. That's
every server...

| Or, equivalently, crashing whenever
| the network path loses the third packet.

The point isn't how usual this is at the server end, or even how realistic
- just that at the client end, which is the only place it matters, something
needs to handle this situation anyway, as it can happen with any TCP
implementation at the server.

| There's a (not unreasonable, I think) expectation that a crash loses
| all state for all connections to the crashed host,

No, that is an unreasonable assumption. You're assuming that TCP is
in the kernel, and only a complete system crash loses state. Nothing
in TCP assumes that, though it is the normal environment today. However,
early implementations had most of TCP as part of the application (just
port de-multiplexing common) a simple application bug and crash would kill
a single connection while leaving others alive. It isn't even impossible
for an implementation to have had a bug where a timeout after sending the
SYN+ACK (which should retransmit that in cases where the returning ACK
was lost) to be the very place where the bug exists, causing the app to
crash every time this happens...

| and for a significant time.

Again, not true if you're assuming the a system reboot is required, and
only on systems where things like loading many MB's from slow storage, and
file system checks, etc, are required - other systems reboot from rom, and
have no modifiable data storage, and can reboot complely within microseconds.

Further, other ancient implementations had TCP offloaded in the network
"card" - a simple reset of that device would lose all TCP state, but be
very quick indeed.

| The pseudo-crash involved here exhibits neither of those properties.

And even if that were true, it is irrelevant. It isn't what is visible
at the server (or other clients) that matters to TCP, but only what iss
een, and done, at the remote end of the (each) individual peer connection
affected.

| I would also think that appearing to have crashed (especially when it
| hasn't) would be something you'd want to get rid of, rather than cause.

You know that's not what I was saying. What I said, was that this is
not some horrible violation of the TCP spec that can be expected to cause
all kinds of problems for client implementations (the way you suggested)
and so must be avoided at all costs. Rather it is something that the
clients need to be able to handle anyway. And like all rare events
that don't often get tested (in TCP or anywhere else) forcing testing
and correction of any defects found, isn't a bad thing either.

I agree with apb, and what seems to be je's intent as well, that the
actual implementation would be better if it only drops the syn cache use
when that overflows, rather than removing it completely - but either way,
the clients (including NetBSD TCP clients of course) must be prepared to
deal. And I do think it is better than even when the syn cache overflows,
the server attempt to keep making connections when possible (when the
network doesn't drop the retuirning ACK) rather than simply resetting
all later attempts - that's much better DoS defence.

kre

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

If you have N% packet loss (for reasonable values of N), and if the
losses are evenly distributed per packet, then SYN cookies amplify
the problem from "N% of packets get lost but retransmission causes
most TCP connections to work anyway" to "N% of TCP connections get
into this stuck state".

True...but only if they are always used. If, as kre outlined, SYN
cookies are used only when the TCP is under attack heavy enough to
overload the SYN cache, it's "N% of TCP connections when under attack".
(And don't forget that N will in general be peer-dependent.)

Since the alternatives seem to be crashing outright or dropping
connection attempts on the floor, I don't think this is all that bad a
failure mode. The wedged connections are a serious downside, but I
think they're a less bad downside than the available alternatives.

At least, assuming the SYN cache is blown out because it's under
attack. If it's under real overload, SYN cookies don't help and will
probably hurt (because, in addition to the connections dropped for lack
of resources, there's that N% that will get stuck half-open.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

Post by Robert Elz
And I do think it is better than even when the syn cache overflows,
the server attempt to keep making connections when possible (when the
network doesn't drop the retuirning ACK) rather than simply resetting
all later attempts - that's much better DoS defence.

While I have some issues with the wording (see below), this actually is
a reasonable argument: it's a failure state (in response to an overload
condition) - but it's an attempt to make that failure less drastic than
a crash or silence.

As for the wording, it's not "resetting all later attempts", or at
least if it is I think that too is a bug; it should be "no response to
later attempts". That's what has traditionally happened when
connection attempts are arriving faster than they can be serviced and I
believe it's a far more appropriate response to such an overload than
RSTs.

Also, it's not just "making connections when possible", with an
implicit "and ignoring them when not"; that would be entirely
reasonable. It's "making connections when possible (and creating
half-open connections when not)". I think this is probably outweighed
by the value of the relatively graceful degradation, though.

Most of the rest of this I see no point responding to, because my
responses are basically "you're taking an extremely rare and almost
always highly visible failure case and making it far more common and
less visible" and then arguing that because recovery mechanisms must
exist for the former, they exist for the latter, which is an invalid
argument because recovery mechanisms that work fine for extremely rare
cases are often completely out of the question for relatively common
cases. But, when it's done only as a more-graceful failure mode in
response to an overload condition in the first place, it stays rare.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alan Barrett]

Post by Robert Elz
Date: Mon, 5 Nov 2012 22:26:59 -0500 (EST)
| Except that it's not; it's no different from a server that crashes
| immediately after sending _every_ SYN|ACK packet
No, only the ones where the returning ACK is lost by the network.
That is, it is the same as a server that crashes sometimes. That's
every server...

If you have N% packet loss (for reasonable values of N), and
if the losses are evenly distributed per packet, then SYN
cookies amplify the problem from "N% of packets get lost but
retransmission causes most TCP connections to work anyway" to "N%
of TCP connections get into this stuck state".

I just tried "ping -c100 -i1 www.netbsd.org" twice, and got
3% ping loss once and 5% loss once. Since a successful ping
requires two packets, the packet loss rate is somewhere around
1.5% to 2.5%. Such a packet loss rate is annoying, but if it were
amplified to a 1.5% to 2.5% TCP connection failure rate then it
would be much more annoying.

--apb (Alan Barrett)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alan Barrett]

Post by Paul Goyette
While at the MeetBSD California un-conference over the weekend,
I was approached by the originator of [1]. Looking through
the archives, I don't see any replies or discussion, so I
was wondering (along with John) if there's any merit to the
suggested code/patches? Has anyone with TCP expertise reviewed
them at all?

Are these SYN cookies to be used all the time without a SYN cache,
or will there also be a finite sized SYN cache to allow the host
to avoid violating the TCP protocol (as long as the cache is not
full)?

If they are to be used all the time without a SYN cache, then I
agree with Mouse that they may cause too much harm. If there is
also a SYN cache and the harmful side-effects of SYN cookies arise
only when the SYN cache is full, then I think that is acceptable.
Exhaustion of a properly-sized SYN cache should be so rare as to
occur only when the host is under attack (or something that looks
like an attack), and under such circumstances it's acceptable
for the host's self-defence measures to inconvenience legitimate
clients.

--apb (Alan Barrett)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[je]

Post by Paul Goyette
While at the MeetBSD California un-conference over the weekend, I was
approached by the originator of [1]. Looking through the archives, I
don't see any replies or discussion, so I was wondering (along with
John) if there's any merit to the suggested code/patches? Has anyone
with TCP expertise reviewed them at all?

Are these SYN cookies to be used all the time without a SYN cache, or
will there also be a finite sized SYN cache to allow the host to avoid
violating the TCP protocol (as long as the cache is not full)?
If they are to be used all the time without a SYN cache, then I agree
with Mouse that they may cause too much harm. If there is also a SYN
cache and the harmful side-effects of SYN cookies arise only when the
SYN cache is full, then I think that is acceptable. Exhaustion of a
properly-sized SYN cache should be so rare as to occur only when the
host is under attack (or something that looks like an attack), and under
such circumstances it's acceptable for the host's self-defence measures
to inconvenience legitimate clients.
--apb (Alan Barrett)

The current implementation requires a sysctl to be set to enable syn
cookies. When set the host always responds with syn cookies and when
unset only via the syn cache (though a valid cookie will still be
accepted). This is done for testing purposes only.

My intention for the final version is to do as you stated, that is, to
begin sending cookies when the syn cache becomes full (or nearly so),
also for the reasons you stated. A sysctl could be provided to disable
cookies at runtime to meet feature parity with other operating systems,
but I don't know if this is necessary.

I did see a request that I can't find now to add a syn cookie only build
option, to free up the memory that would be used on the cache, which
should be trivially possible assuming I can find the original request.
The opposite would also be easy to add.

John

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de