IPV6 router works, but clients fail

[Roy Marples]

OK, I'm almost there with IPv6 now. The router is working fine by itself
with IPv6. My test site is http://www.goscomb.net as it's my ISP and the
transport is pure IPv6

wget -6 http://www.goscomb.net
works fine.

Clients at first appear fine as well, and can connect to
ipv6.google.com, however this fails
wget -6 http://www.goscomb.net

What is really odd is the traceroute6 from ftp.netbsd.org to the client

$ traceroute6 2a01:348:31:2:20e:2eff:fe66:36ec
traceroute6 to 2a01:348:31:2:20e:2eff:fe66:36ec
(2a01:348:31:2:20e:2eff:fe66:36ec) from 2001:4f8:3:7:230:48ff:fe31:43f2,
64 hops max, 12 byte packets
1 2001:4f8:3:7::1 1.363 ms 1.27 ms 0.734 ms
2 int-0-1-0-0-606.r1.sfo2.isc.org 4.614 ms 4.901 ms 5.393 ms
3 int-3-0-0.r1.pao1.isc.org 2.274 ms 1.964 ms 1.913 ms
4 ge-1-11.r03.plalca01.us.bb.gin.ntt.net 2.35 ms 2.049 ms 2.315 ms
5 ae-3.r21.plalca01.us.bb.gin.ntt.net 2.257 ms 2.236 ms 2.157 ms
6 ae-1.r20.snjsca04.us.bb.gin.ntt.net 3.763 ms 3.527 ms 3.804 ms
7 as-1.r21.chcgil09.us.bb.gin.ntt.net 59.208 ms 64.015 ms 64.135 ms
8 ae-0.r20.chcgil09.us.bb.gin.ntt.net 59.49 ms 59.861 ms 64.139 ms
9 as-1.r21.nycmny01.us.bb.gin.ntt.net 79.759 ms 78.668 ms 78.344 ms
10 ae-0.r20.nycmny01.us.bb.gin.ntt.net 83.645 ms 83.981 ms 83.694 ms
11 as-1.r22.londen03.uk.bb.gin.ntt.net 151.626 ms 155.074 ms 155.082 ms
12 po-4.r01.londen03.uk.bb.gin.ntt.net 316.029 ms 244.966 ms 350.231 ms
13 2001:728:0:5000::6e 156.617 ms 280.137 ms 153.643 ms
14 ge-0-0-31.rt1.lon4.ipv6.goscomb.net 152.039 ms 156.782 ms 153.582 ms
15 2a01:348:31:2:209:5bff:fe84:887d 167.777 ms 171.006 ms 165.695 ms
$

Wtf? The last node is not my client - it's the router!
So I think my router isn't passing packets to the client.
However, ftp.netbsd.org can ping6 the client.

I'm not a PF expert by any means so I've attached the pf.conf from my
router.

Anyone got an idea?

Thanks

Roy

[Roy Marples]

Post by Roy Marples
OK, I'm almost there with IPv6 now. The router is working fine by itself
with IPv6. My test site is http://www.goscomb.net as it's my ISP and the
transport is pure IPv6
wget -6 http://www.goscomb.net
works fine.
Clients at first appear fine as well, and can connect to
ipv6.google.com, however this fails
wget -6 http://www.goscomb.net

As usual, my PF configuration to blame. I needed to clamp max-mss in
pppoe0 to 1432. It used to be 1452, which I recall was needed for my
wireless clients. Probably the extra overhead of IPv6.

So my IPv6 is now working very well on server and clients :)

What is nice about this setup is that whilst IPv6 doesn't require nasty
NAT, my PF firewall is a firewall for all my clients too!

Thanks

Roy


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

rm> I needed to clamp max-mss in pppoe0 to 1432. It used to be
rm> 1452, which I recall was needed for my wireless
rm> clients. Probably the extra overhead of IPv6.

The PPPoE MTU problem should not exist on IPv6, and at my site where I
have <1500 links to the internet I didn't have to change mss-scrubbing
smaller for IPv6. It's possible you are creating the PPPoE problem
yourself somewhere by blocking ICMPv6 'too-big' packages.

You should never find you need the mss scrubbing to reach the Internet
period---if you do, you must be blocking too much ICMP on your end.
The symptom of needing smaller mss scrubbing is that a few of other
people's misconfigured sites on the Internet don't work, just a few
not all. I wish you would have a look to your ICMP rules to avoid
publishing bad examples which will infect other sites and spread the
PPPoE problem.

The way it's documented/supposed to work, you must either pass too-big
/ frag-needed ICMP, *or* use 'keep state' TCP rules which pass that
ICMP implicitly. The way it actually works, I'm not so sure.

[Brian A. Seklecki]

As usual, my PF configuration to blame. I needed to clamp max-mss in pppoe0 to
1432. It used to be 1452, which I recall was needed for my wireless clients.

What is the MTU on your PPPoE interface?

Since this thread started -- I've begun to notice a plethora of problems
with TCP v6 sockets.

Someone on #NetBSD @ EFNet recently mentioned that pf(4), V6, and
fragments were getting shredded/dropped.

From the man page pf.conf(5) in -rHEAD:

"Currently, only IPv4 fragments are supported and IPv6 fragments are
blocked unconditionally."


~BAS

Probably the extra overhead of IPv6.
So my IPv6 is now working very well on server and clients :)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Roy Marples]

Post by Brian A. Seklecki

Post by Roy Marples
As usual, my PF configuration to blame. I needed to clamp max-mss in
pppoe0 to 1432. It used to be 1452, which I recall was needed for my
wireless clients.

What is the MTU on your PPPoE interface?

1492
However, you should be aware that the modem is a PPPoA -> PPPoE bridge, which
does affect the MTU.

http://mail-index.netbsd.org/regional-london/2006/09/14/0000.html

So my situation is not that uncommon.

Post by Brian A. Seklecki
Since this thread started -- I've begun to notice a plethora of problems
with TCP v6 sockets.
fragments were getting shredded/dropped.
"Currently, only IPv4 fragments are supported and IPv6 fragments are
blocked unconditionally."

Yes, that's not too great is it.
IPFilter suffers from this also, so there is little choice in BSD land right
now.

Thanks

Roy

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Roy Marples]

Post by Miles Nordin
rm> I needed to clamp max-mss in pppoe0 to 1432. It used to be
rm> 1452, which I recall was needed for my wireless
rm> clients. Probably the extra overhead of IPv6.
The PPPoE MTU problem should not exist on IPv6, and at my site where I
have <1500 links to the internet I didn't have to change mss-scrubbing
smaller for IPv6. It's possible you are creating the PPPoE problem
yourself somewhere by blocking ICMPv6 'too-big' packages.

I've attached my current pf.conf
As Brian pointed out in this thread, PF does not handle IPv6 fragments which
could be the issue as well. However, the router does need a firewall as it's
also my server.

Post by Miles Nordin
You should never find you need the mss scrubbing to reach the Internet
period---if you do, you must be blocking too much ICMP on your end.
The symptom of needing smaller mss scrubbing is that a few of other
people's misconfigured sites on the Internet don't work, just a few
not all. I wish you would have a look to your ICMP rules to avoid
publishing bad examples which will infect other sites and spread the
PPPoE problem.
The way it's documented/supposed to work, you must either pass too-big
/ frag-needed ICMP, *or* use 'keep state' TCP rules which pass that
ICMP implicitly. The way it actually works, I'm not so sure.

Well the router and clients passes all ICMP packets - the clients didn't even
have firewalls installed. Anyway, here's the lines from the attached pf.conf
pass in proto icmp all
pass in proto ipv6-icmp all

It doesn't help that I'm using a PPPoA->PPPoE modem, which apparently affects
things as well.

Interestingly enough, if I drop the MTU on my clients to 1492 then I don't
need the scrub mss line. Anyone have an opinion on which would be better?

Thanks

Roy

[Miles Nordin]

rm> I've attached my current pf.conf As Brian pointed out in this
rm> thread, PF does not handle IPv6 fragments

That's bad but it's not the problem. There will never be any IPv6 TCP
fragments, even with all this nonsense going on. There can be UDP
fragments, though.

rm> if I drop the MTU on my clients to 1492 then I don't need the
rm> scrub mss line. Anyone have an opinion on which would be
rm> better?

the scrubbing is better.

If all hosts on an ethernet do not have the same MTU set, this will
cause a second level of brokenness---now you have two broken things
instead of one. That scenario's likely because you'll forget, or
you'll have test systems or guests or VM's or whatever.

[Roy Marples]

Post by Miles Nordin
rm> I've attached my current pf.conf As Brian pointed out in this
rm> thread, PF does not handle IPv6 fragments
That's bad but it's not the problem. There will never be any IPv6 TCP
fragments, even with all this nonsense going on. There can be UDP
fragments, though.

So what is the problem?

Post by Miles Nordin
rm> if I drop the MTU on my clients to 1492 then I don't need the
rm> scrub mss line. Anyone have an opinion on which would be
rm> better?
the scrubbing is better.
If all hosts on an ethernet do not have the same MTU set, this will
cause a second level of brokenness---now you have two broken things
instead of one. That scenario's likely because you'll forget, or
you'll have test systems or guests or VM's or whatever.

Well, I experimented. It turns out that rtadvd has a nice option to broadcast
the MTU of the IPv6 link. Linux is clever here - this MTU is only applicable
to the IPv6 and does not change the real MTU of the interface. So MTU is 1500
both server and client, but for IPv6 it's 1492. Works like a champ.

Sadly, this is not the case in NetBSD.
Infact I couldn't get ANY MTU values to work for ipv4/ipv6 connectivity past
the router unless I was scrubbing on the router.
What is more, NetBSD ignores (or seems to) the MTU in the IPv6 RA. Is this a
bug or a known issue?

Thanks

Roy

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

rm> So what is the problem?

If you're having the same problem as Matthias, it is that some router
in the to-you direction is accepting packets on a 1500 interface,
forwarding them onto a 1492 interface, and not sending the ICMP
'too-big' message when a packet that won't fit comes along---instead
just dropping it.

Routers never fragment because on IPv6 it is a rule that only
end-systems fragment, and on IPv4 all TCP packets have the DF bit set.
so their correct behavior for TCP is always drop and send ICMP
frag-needed/too-big.

The usual ``PPPoE problem'' is that a router DOES send this message,
but some moron's firewall is configured to block all the ICMP the
sysadmin doesn't understand, which is basically everything.

Matthias was saying in his case, he thinks the guilty party is his DSL
modem translating from PPPoA (1500) to PPPoE (1492), but doing so as
some silly-hack route-bridge thing with no IP address of its own so it
couldn't send an ICMP. (I don't really see why not---it breaks every
other sane behavior, so why not spoof an ICMP from any address it has
handy, yours, the other side's, whatever). but that's the theory, and
why I say, can you replace the modem?

I've never heard of PPPoA to PPPoE translation before though. I've
heard of a PPPoA router. And I have the BroadXtent modem, a PPPoE
route-bridge-thing that routes without having an IP for itself, but it
spits out clear unencapsulated packets.

rm> rtadvd
rm> 1492.
rm> Linux
rm> Works like a champ.

rm> NetBSD ignores (or seems to) the MTU in the IPv6 RA.

There is a line of reasoning:

routing may be asymmetric, so TCP should not assume the MTU is the
same in both directions, and shouldn't give the other end's mss based
on the MTU of a direct-attached interface for the opposite direction,
and instead let the other side figure out the MTU using PMTU-D.

There's a sysctl to change this

net.inet.tcp.mss_ifmtu = 1
net.inet6.tcp6.mss_ifmtu = 1

[Roy Marples]

Post by Miles Nordin
rm> So what is the problem?
If you're having the same problem as Matthias, it is that some router
in the to-you direction is accepting packets on a 1500 interface,
forwarding them onto a 1492 interface, and not sending the ICMP
'too-big' message when a packet that won't fit comes along---instead
just dropping it.
Routers never fragment because on IPv6 it is a rule that only
end-systems fragment, and on IPv4 all TCP packets have the DF bit set.
so their correct behavior for TCP is always drop and send ICMP
frag-needed/too-big.
The usual ``PPPoE problem'' is that a router DOES send this message,
but some moron's firewall is configured to block all the ICMP the
sysadmin doesn't understand, which is basically everything.
Matthias was saying in his case, he thinks the guilty party is his DSL
modem translating from PPPoA (1500) to PPPoE (1492), but doing so as
some silly-hack route-bridge thing with no IP address of its own so it
couldn't send an ICMP. (I don't really see why not---it breaks every
other sane behavior, so why not spoof an ICMP from any address it has
handy, yours, the other side's, whatever). but that's the theory, and
why I say, can you replace the modem?

Thanks for the explanation.

Post by Miles Nordin
I've never heard of PPPoA to PPPoE translation before though. I've
heard of a PPPoA router. And I have the BroadXtent modem, a PPPoE
route-bridge-thing that routes without having an IP for itself, but it
spits out clear unencapsulated packets.
rm> rtadvd
rm> 1492.
rm> Linux
rm> Works like a champ.
rm> NetBSD ignores (or seems to) the MTU in the IPv6 RA.
routing may be asymmetric, so TCP should not assume the MTU is the
same in both directions, and shouldn't give the other end's mss based
on the MTU of a direct-attached interface for the opposite direction,
and instead let the other side figure out the MTU using PMTU-D.
There's a sysctl to change this
net.inet.tcp.mss_ifmtu = 1
net.inet6.tcp6.mss_ifmtu = 1

My understanding is that only works on the router and not the ones
connected by NAT. I think I'll go with the 1492 MTU everywhere as it's
the less "hackish" and would make any box behind my broken modem behave
better to the outside.

Maybe I'll look into allowing NetBSD to use the MTU as advertised by
rtadvd as it currently ignores it.

Thanks

Roy

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

Post by Miles Nordin
net.inet.tcp.mss_ifmtu = 1
net.inet6.tcp6.mss_ifmtu = 1

rm> My understanding is that only works on the router and not the
rm> ones connected by NAT. [...]

if you weren't changing MTU on your ethernet segment that'd be true.

but in your case, no:

rm> Maybe I'll look into allowing NetBSD to use the MTU as
rm> advertised by rtadvd as it currently ignores it.

the above sysctl's are likely to do exactly this.

MTU != mss

MTU is for all protocols and controls what you are allowed to send.

mss is for TCP only, and advises the other end about what you are
prepared to receive.

If you presume asymmetric routing, what you are prepared to receive
doesn't necessarily have anything to do with what you are allowed to
send. so NetBSD may well be obeying the mtu in rtadv's, but even if
it were, this would cause no change in the mss (with the default '0'
setting of those sysctl's) and would not solve your problem.

The sysctl's say, ``go ahead and make the assumption that the receive
path uses the same interface as the send path,'' so NetBSD will
calculate an mss based on IP + TCP overhead and the MTU of the route
for sending TO the peer.

of course there could easily be two problems...

[der Mouse]

Post by Miles Nordin
You should never find you need the mss scrubbing to reach the
Internet period---if you do, you must be blocking too much ICMP on
your end.

That's a nice theory, but it does not agree with (at least my)
experience.

There are _way_ too many places out there who try to do PMTU-D but are
behind firewalls that produce PMTU-D black holes. (Misconfigured
firewalls, to be sure, but still, even if you can reach them somehow,
have you ever tried to convince some semi-competent admin that his[%]
firewall is producing a PMTU-D black hole? The ones who don't MEGO at
the mere mention of PMTU-D are rare - and generally aren't part of the
problem to begin with. The one case I ever was taken seriously, I knew
the remote sysadmin personally already, and even then I don't think we
ever got it fixed.)

[%] In my (admittedly limited) experience, most of them are male. I
don't know whether this is just because most admins are male,
because a woman doing sysadmin _has_ to be good to be taken
seriously :(, or what....

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

rm> http://mail-index.netbsd.org/regional-london/2006/09/14/0000.html

oh you are fucking kidding me.

well, the problem is still on your end, though. It's just in your
modem instead of in your PF rules.

Could you try to get a different modem? I have a BroadXTent modem
that grabs its address over PPPoE, then hands the address it grabs to
the connected host using DHCP with a short lease. If you had a modem
like mine which used PPPoA instead, then in PPPoA mode you could get
the full 1500 mtu, no? Some googling found this:

http://www.wenks.ch/fabian/ADSL-PPPoA.html

Another alternative is to use Cisco CPE like 1721+WIC-1ADSL or 857 and
terminate the IPv6 on the Cisco.

What's the point of this translation to PPPoE that Matthias says his
modem is doing? It seems really odd. I'd think if the modem were
speaking PPPoA it would certainly handle all the passwords and
terminate the PPPoA on the modem, after which it would either act as a
normal router doing its own NAT, or else at the weirdest pass to you
its local PPPoA IP in that nonstandard proxyarp/fakeDHCP way of which
my BroadXtent modem is capable.

Is it for certain the modem is really converting PPPoA<->PPPoE like
this, and not just that the DSLAM in the CO works in a variety of
modes (oA, oE) among which you can choose, and the oE mode is broken
for v6? I hope it's the modem that's broken not the DSLAM since you
can swap the modem out, but I doubt the story because this is the
first I've heard of a modem doing that.

I suppose you'll ignore this and keep using the mss clamping. but
FYI, it is really broken.

[Roy Marples]

Post by Miles Nordin
rm> http://mail-index.netbsd.org/regional-london/2006/09/14/0000.html
oh you are fucking kidding me.
well, the problem is still on your end, though. It's just in your
modem instead of in your PF rules.
Could you try to get a different modem? I have a BroadXTent modem
that grabs its address over PPPoE, then hands the address it grabs to
the connected host using DHCP with a short lease. If you had a modem
like mine which used PPPoA instead, then in PPPoA mode you could get
http://www.wenks.ch/fabian/ADSL-PPPoA.html
Another alternative is to use Cisco CPE like 1721+WIC-1ADSL or 857 and
terminate the IPv6 on the Cisco.
What's the point of this translation to PPPoE that Matthias says his
modem is doing? It seems really odd. I'd think if the modem were
speaking PPPoA it would certainly handle all the passwords and
terminate the PPPoA on the modem, after which it would either act as a
normal router doing its own NAT, or else at the weirdest pass to you
its local PPPoA IP in that nonstandard proxyarp/fakeDHCP way of which
my BroadXtent modem is capable.

This is the modem recommended by my ISP so can I get native IPv6
working. AFAIK it's not possible with my old modem, a DrayTek Vigor 2600
which oddly enough is listed in your link.

At the time of purchase, no ADSL modem in the UK could handle IPv6 that
I could see, except for Cisco routers which are well outside of my price
range by a few hundred pounds. So going for the Vigor100 for a PPPoA
PPPoE bridge was a no brainer as then my server could handle PPPoE + IPv6.

Post by Miles Nordin
Is it for certain the modem is really converting PPPoA<->PPPoE like
this, and not just that the DSLAM in the CO works in a variety of
modes (oA, oE) among which you can choose, and the oE mode is broken
for v6? I hope it's the modem that's broken not the DSLAM since you
can swap the modem out, but I doubt the story because this is the
first I've heard of a modem doing that.

I'm pretty sure the UK is PPPoA only so it would have to be a bridge.
Maybe there are some settings on the vigor I can tweak, or maybe try and
adjust the PPPoE MTU to 1500?

Post by Miles Nordin
I suppose you'll ignore this and keep using the mss clamping. but
FYI, it is really broken.

No, as it really is a hack.
I'll probably end up just enforcing an MTU of 1492 on the network as I
control all the client side equipment. I just hope that my games
consoles obey the MTU DHCP setting :)

Thanks

Roy

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

rm> Cisco routers which are well outside of my price range by a
rm> few hundred pounds.

nonsense. 1721 + wic-1adsl:

http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=190318822531

rm> try and adjust the PPPoE MTU to 1500?

no I don't think they can send minijumbo's.

try to config the modem as a router (if it will route v6? i guess
isp's point is that none of 'em will)

try alternate firmware like http://routertech.org/

search for USB DSL modem support in Unixes (might have to use other
than NetBSD) and run your modem in USB mode where you terminate the
PPPoA on an ATM stack in the unix.

but honestly cisco is almost certain to work and the others are all
longshots.

[Roy Marples]

Post by Miles Nordin
rm> Cisco routers which are well outside of my price range by a
rm> few hundred pounds.
http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=190318822531

I'm on ADSL2+.
That leaves me with the Cisco 877 AFAIK.
There is also a Viking PCI ADSL2+ PPPoA card I could use, but getting
that shipped to the UK is also very expensive.

Post by Miles Nordin
rm> try and adjust the PPPoE MTU to 1500?
no I don't think they can send minijumbo's.
try to config the modem as a router (if it will route v6? i guess
isp's point is that none of 'em will)
try alternate firmware like http://routertech.org/
search for USB DSL modem support in Unixes (might have to use other
than NetBSD) and run your modem in USB mode where you terminate the
PPPoA on an ATM stack in the unix.
but honestly cisco is almost certain to work and the others are all
longshots.

I concurr.
I think I'll just have to live with the modem and save up for a Cisco :/

Thanks

Roy

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Miles Nordin]

Post by Miles Nordin
You should never find you need the mss scrubbing to reach the
Internet period---if you do, you must be blocking too much ICMP
on your end.

dm> That's a nice theory, but it does not agree with (at least my)
dm> experience.

difference is you are saying ``no no really a lot of sites don't work
and you cannot live without the hack'' and OP is saying ``all sites
don't work''. He is not having the usual PPPoE problem which means we
look elsewhere for the answer. This part isn't political.