patch for duplicate tcp acks

[Charlet, Ricky]

Howdy,

I was suffering duplicate tpc acks. It seems trivial to duplicate.... Any time I receive a non-zero len tcp packet, I ack it twice. Note that I'm using a window scale of 3 and that may or may come into play.

Anyway, did some debugging and some comparing. I found a patch for this problem in freebsd.
https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?r1=216758&r2=220794


Here is how I applied it to netbsd (diff included below). It works for me. But I'm on an embedded system with a funky compiler and modified source. I'm giving this patch to the list in hopes that someone will feel like compiling / testing it in a truer netbsd kernel.




Index: src/sys/netinet/tcp_output.c
===================================================================
RCS file: /cvsroot/src/sys/netinet/tcp_output.c,v
retrieving revision 1.173
diff -r1.173 tcp_output.c
1023,1024c1023,1038
< long adv = min(win, (long)TCP_MAXWIN << tp->rcv_scale) -
< (tp->rcv_adv - tp->rcv_nxt);
---

long adv;
int oldwin;
adv = fp_min(win, (long)FP_TCP_MAXWIN << tp->rcv_scale);
if (SEQ_GT(tp->rcv_adv, tp->rcv_nxt)) {
oldwin = (tp->rcv_adv - tp->rcv_nxt);
adv -= oldwin;
} else
oldwin = 0;
/*
* If the new window size ends up being the same as the old
* size when it is scaled, then don't force a window update.
*/
if (oldwin >> tp->rcv_scale == (adv + oldwin) >> tp->rcv_scale)
goto dontupdate;

1030a1045
--
Ricky Charlet
Software Dev / Routing Dude: Aries team, Roseville CA
***@hp.com<mailto:***@hp.com>
USA: 916.785.2090




--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thor Lancelot Simon]

Post by Charlet, Ricky
Howdy,
I was suffering duplicate tpc acks. It seems trivial to duplicate.... Any time I receive a non-zero len tcp packet, I ack it twice. Note that I'm using a window scale of 3 and that may or may come into play.

Presumably this sent the stack on the far end into fast retransmit at least 50% more often
than it "should". Did you actually see any performance problems associated with this?

I'm pretty sure I've seen this problem in the past, but at the time concluded that it
wasn't worth investigating because it seemed to have only the minor negative impact
of cluttering up my packet traces. Which is not to say it should not be fixed!

Thanks!

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Christos Zoulas]

Post by Charlet, Ricky
Howdy,
I was suffering duplicate tpc acks. It seems trivial to
duplicate.... Any time I receive a non-zero len tcp packet, I ack it
twice. Note that I'm using a window scale of 3 and that may or may come
into play.
Anyway, did some debugging and some comparing. I found a patch
for this problem in freebsd.
https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?r1=216758&r2=220794
Here is how I applied it to netbsd (diff included below). It works for
me. But I'm on an embedded system with a funky compiler and modified
source. I'm giving this patch to the list in hopes that someone will
feel like compiling / testing it in a truer netbsd kernel.

Why didn't you just apply the FreeBSD patch? Are you trying to fix something
else? Can you please file a PR about this, or do you mind if I file one
using this information?

thanks,

christos

Post by Charlet, Ricky
Index: src/sys/netinet/tcp_output.c
===================================================================
RCS file: /cvsroot/src/sys/netinet/tcp_output.c,v
retrieving revision 1.173
diff -r1.173 tcp_output.c
1023,1024c1023,1038
< long adv = min(win, (long)TCP_MAXWIN << tp->rcv_scale) -
< (tp->rcv_adv - tp->rcv_nxt);
---

long adv;
int oldwin;
adv = fp_min(win, (long)FP_TCP_MAXWIN << tp->rcv_scale);
if (SEQ_GT(tp->rcv_adv, tp->rcv_nxt)) {
oldwin = (tp->rcv_adv - tp->rcv_nxt);
adv -= oldwin;
} else
oldwin = 0;
/*
* If the new window size ends up being the same as the old
* size when it is scaled, then don't force a window update.
*/
if (oldwin >> tp->rcv_scale == (adv + oldwin) >> tp->rcv_scale)
goto dontupdate;

1030a1045
--
Ricky Charlet
Software Dev / Routing Dude: Aries team, Roseville CA
USA: 916.785.2090

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Charlet, Ricky]

Hi Christos,
Thanks for your reply. So, about applying patches and starting bugs.... I did not do those things because I'm totally new here having only singed up the day I started this thread. And honestly, I probably won't mature into anything like a regular contributor. Please do start a PR... I don't know how.

What I'm asking for is sanity checking... is this 'the right thing' to do? If it is 'the right thing' then I hope that regular contributors will pick up the notion, file the paperwork and apply patches.

If you want me to contribute by filing the PR, I'll go figure that out (hopefully explained on the netbsd:developers web page).

--
Ricky Charlet


-----Original Message-----
From: tech-net-***@NetBSD.org [mailto:tech-net-***@NetBSD.org] On Behalf Of Christos Zoulas
Sent: Monday, April 27, 2015 8:07 AM
To: tech-***@netbsd.org
Subject: Re: patch for duplicate tcp acks

Post by Charlet, Ricky
Howdy,
I was suffering duplicate tpc acks. It seems trivial to
duplicate.... Any time I receive a non-zero len tcp packet, I ack it
twice. Note that I'm using a window scale of 3 and that may or may
come into play.
Anyway, did some debugging and some comparing. I found a patch
for this problem in freebsd.
https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?r1=216758
&r2=220794
Here is how I applied it to netbsd (diff included below). It works for
me. But I'm on an embedded system with a funky compiler and modified
source. I'm giving this patch to the list in hopes that someone will
feel like compiling / testing it in a truer netbsd kernel.

Why didn't you just apply the FreeBSD patch? Are you trying to fix something else? Can you please file a PR about this, or do you mind if I file one using this information?

thanks,

christos

Post by Charlet, Ricky
Index: src/sys/netinet/tcp_output.c
===================================================================
RCS file: /cvsroot/src/sys/netinet/tcp_output.c,v
retrieving revision 1.173
diff -r1.173 tcp_output.c
1023,1024c1023,1038
< long adv = min(win, (long)TCP_MAXWIN << tp->rcv_scale) -
< (tp->rcv_adv - tp->rcv_nxt);
---

long adv;
int oldwin;
adv = fp_min(win, (long)FP_TCP_MAXWIN << tp->rcv_scale);
if (SEQ_GT(tp->rcv_adv, tp->rcv_nxt)) {
oldwin = (tp->rcv_adv - tp->rcv_nxt);
adv -= oldwin;
} else
oldwin = 0;
/*
* If the new window size ends up being the same as the old
* size when it is scaled, then don't force a window update.
*/
if (oldwin >> tp->rcv_scale == (adv + oldwin) >> tp->rcv_scale)
goto dontupdate;

1030a1045
--
Ricky Charlet
Software Dev / Routing Dude: Aries team, Roseville CA
USA: 916.785.2090

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Charlet, Ricky]

Hi Thor,

It will take me a day to answer your question. At the moment, I have only linux VMs on over taxed VM servers to play with in my lab and I notice no performance problem here (though I am unable to drive any real performance stress). But I'll try and ask some co-workers with 'real' performance test gear to give this a run.

But, in the meantime... Why do you presume this patch would lead the peer into fast retransmit? Note that with this patch, I am observing that I have stopped double acking most data and am now single acking it.

Note: my use case is a slight departure from normal... I'm trying to build a tcp proxy.

[server]-------------------[proxy]---------------------[client]
A B

Where interface A is a proxy client and interface B is a proxy server. But I don't see why my proxy client side would behave differently than a 'normal' client, especially during bulk data flow.

On the other hand, the patch (from freebsd) and it's comment make perfect sense. It's a "don't ack stuff that we have already acked, even in the presence of window scaling" patch.


--
Ricky Charlet

-----Original Message-----
From: Thor Lancelot Simon [mailto:***@panix.com]
Sent: Saturday, April 25, 2015 10:09 AM
To: Charlet, Ricky
Cc: tech-***@NetBSD.org
Subject: Re: patch for duplicate tcp acks

Post by Charlet, Ricky
Howdy,
I was suffering duplicate tpc acks. It seems trivial to duplicate.... Any time I receive a non-zero len tcp packet, I ack it twice. Note that I'm using a window scale of 3 and that may or may come into play.

Presumably this sent the stack on the far end into fast retransmit at least 50% more often than it "should". Did you actually see any performance problems associated with this?

I'm pretty sure I've seen this problem in the past, but at the time concluded that it wasn't worth investigating because it seemed to have only the minor negative impact of cluttering up my packet traces. Which is not to say it should not be fixed!

Thanks!

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Charlet, Ricky]

Got some test results.

The test we ran was to transfer a 10k file repeatedly, sequentially for 2 minutes. The test bed we have has 100Mb ethernet links and achieves 81Mb throughput without my Device Under Test inline and achieves 49Mb throughput with my Device Under Test. I count this as a success for the patch as this fits within my expectations for the proxy app I'm building.

Doing a human scan through the resultant pcap (I scanned very quickly and covered about 50% of the data) shows no dup acks, no window resize events, and no unexpected flow events of any kind.

Wireshark Statistics: TCP stream graph: Window Scaling graph -- show no change for the duration.
Wireshark Statistics: TCP stream graph: round trip time graph -- show variance between .00004 and .00007 seconds (seems ok to me)
Wireshark Statistics: TCP stream graph: throughput graph -- settles in the first 1/4 second to a flat line.
Wireshark Analyze: Expert Infos -- shows 0 errors / 0 warnings.

I can upload the 523MB pcap file somewhere if anyone is curious and would point me to an open repository.



--
Ricky Charlet



-----Original Message-----
From: tech-net-***@NetBSD.org [mailto:tech-net-***@NetBSD.org] On Behalf Of Charlet, Ricky
Sent: Monday, April 27, 2015 9:12 AM
To: Thor Lancelot Simon; tech-***@NetBSD.org
Subject: RE: patch for duplicate tcp acks

Hi Thor,

It will take me a day to answer your question. At the moment, I have only linux VMs on over taxed VM servers to play with in my lab and I notice no performance problem here (though I am unable to drive any real performance stress). But I'll try and ask some co-workers with 'real' performance test gear to give this a run.

But, in the meantime... Why do you presume this patch would lead the peer into fast retransmit? Note that with this patch, I am observing that I have stopped double acking most data and am now single acking it.

Note: my use case is a slight departure from normal... I'm trying to build a tcp proxy.

[server]-------------------[proxy]---------------------[client]
A B

Where interface A is a proxy client and interface B is a proxy server. But I don't see why my proxy client side would behave differently than a 'normal' client, especially during bulk data flow.

On the other hand, the patch (from freebsd) and it's comment make perfect sense. It's a "don't ack stuff that we have already acked, even in the presence of window scaling" patch.


--
Ricky Charlet

-----Original Message-----
From: Thor Lancelot Simon [mailto:***@panix.com]
Sent: Saturday, April 25, 2015 10:09 AM
To: Charlet, Ricky
Cc: tech-***@NetBSD.org
Subject: Re: patch for duplicate tcp acks

Post by Charlet, Ricky
Howdy,
I was suffering duplicate tpc acks. It seems trivial to duplicate.... Any time I receive a non-zero len tcp packet, I ack it twice. Note that I'm using a window scale of 3 and that may or may come into play.

Presumably this sent the stack on the far end into fast retransmit at least 50% more often than it "should". Did you actually see any performance problems associated with this?

I'm pretty sure I've seen this problem in the past, but at the time concluded that it wasn't worth investigating because it seemed to have only the minor negative impact of cluttering up my packet traces. Which is not to say it should not be fixed!

Thanks!

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thor Lancelot Simon]

Post by Charlet, Ricky
But, in the meantime... Why do you presume this patch would lead the peer into fast retransmit? Note that with this patch, I am observing that I have stopped double acking most data and am now single acking it.

I'm asking whether you see any real performance impact *without* the patch.

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Charlet, Ricky]

Oh bummer... now I see the point of your question.

Did not test that. But I could do. Will have to be tomorrow.


--
Ricky Charlet

-----Original Message-----
From: Thor Lancelot Simon [mailto:***@panix.com]
Sent: Monday, April 27, 2015 5:02 PM
To: Charlet, Ricky
Cc: tech-***@NetBSD.org
Subject: Re: patch for duplicate tcp acks

Post by Charlet, Ricky
But, in the meantime... Why do you presume this patch would lead the peer into fast retransmit? Note that with this patch, I am observing that I have stopped double acking most data and am now single acking it.

I'm asking whether you see any real performance impact *without* the patch.

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de