IPv6 and ND on tap(4)

Hi,

Post by Greg Troxel
/sbin/route add -inet6 2001:608:4:a053::/64 2001:608:4:a053::1:0
There is no reason to do this. Configuring an address automatically
adds a route covering that prefix. It should not have worked (gotten a
route exists error), but either it overwrote or the cloning route was
missing for some reason (which is the thing to fix).

Ah.

I add the route, because it did not automatically show up (which is one
of the weirdest bits of cross-platform code - half of the platforms auto-
add the "connected" route at ifconfig, and the other half doesn't).

Step by step (without an application having the tap open, just ifconfig):

# netstat -rn |grep tap
# ifconfig tap0
ifconfig: SIOCGIFFLAGS tap0: Device not configured
# ifconfig tap0 create
# netstat -rn |grep tap
# ifconfig tap0 inet6 2001:608:4:a053::1:0/64 up
# netstat -rn |grep tap
fe80::%tap0/64 link#9 UC 0 0 - tap0
fe80::f00b:a4ff:fe00:4dd2%tap0 f2:0b:a4:00:4d:d2 UHL 0 0 - lo0
ff01:9::/32 link#9 UC 0 0 - tap0
ff02::%tap0/32 link#9 UC 0 0 - tap0
# ifconfig tap0
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
address: f2:0b:a4:00:4d:d2
media: Ethernet autoselect
inet6 fe80::f00b:a4ff:fe00:4dd2%tap0 prefixlen 64 scopeid 0x9
inet6 2001:608:4:a053::1:0 prefixlen 64

... so where does the route for the /64 prefix disappear to?

# uname -a
NetBSD zeta.medat.de 5.1_STABLE NetBSD 5.1_STABLE (GENERIC) #0: Tue Feb 15 15:46:59 CET 2011 ***@zeta.medat.de:/hilb31/sparc64.compile/obj/hilb31/netbsd/src/sys/arch/sparc64/compile/GENERIC sparc64

As a cross-check, I just tried this on NetBSD 3.1, and indeed, the route
auto-appears on tap0, and ND starts

22:20:00.348908 2001:608:4:a253::1:0 > ff02::1:ff00:2: icmp6: neighbor sol: who has 2001:608:4:a253::2

(different prefix deliberately chosen)

Post by Greg Troxel
2001:608:4:a000::/56 2001:608:4:a053::1 UGS 0 0 - tap0
2001:608:4:a053::/64 2001:608:4:a053::1:0 UGS 1 0 - tap0
This looks wrong, but is probably due to the previous issues. Where
did that /56 come from?

Sorry for being misleading. That's an extra network that is just to
be routed across the tap0 /64, but I didn't want to modify the output
of "netstat -rn" in any way.

Post by Greg Troxel
When doing netstat, do netstat -rn -f inet6 and then filter out what you
are sure is not relevant, rather than assuming grepping for tap0 will
show you everything you should look at.

Yes, sorry for that (usually I grep for ":608:4:", knowing that this
network must only ever appear on tap0, but I wanted to include the fe80
stuff).

Here's "netstat -rn -f inet6" after the ifconfig sequence from above,
excluding everything that has no "tap0" or "2001:608:4:" in it:

2001:608:4:a053::1:0 f2:0b:a4:00:4d:d2 UHL 0 0 - lo0
fe80::%tap0/64 link#9 UC 0 0 - tap0
ff01:9::/32 link#9 UC 0 0 - tap0
ff02::%tap0/32 link#9 UC 0 0 - tap0

Post by Greg Troxel
When you configure the address on tap0, it should get a cloning route,
which will have flags UC. This should be in generic ethernet code.
Note that this apparently happened correctly for v4.

Yes, in v4 this all works.

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany ***@greenie.muc.de
fax: +49-89-35655025 ***@net.informatik.tu-muenchen.de

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Greg Troxel

2011-09-17 00:47:51 UTC

It seems there is a bug in tap.

In systems derived from 4.4BSD, the connected route should appear
automatically when adding an address. I would say it's a bug for
systems that have the concept of cloning route not to add it
automatically when configuring an address.

summary: I tried to replicate on sparc64. Both it and i386 seem to work
correctly.

Post by Gert Doering
# netstat -rn |grep tap
# ifconfig tap0
ifconfig: SIOCGIFFLAGS tap0: Device not configured
# ifconfig tap0 create
# netstat -rn |grep tap
# ifconfig tap0 inet6 2001:608:4:a053::1:0/64 up

up is implied by an address. Probably if you did up bare first, you'd
see the fe80:: routes.

Post by Gert Doering
# netstat -rn |grep tap
fe80::%tap0/64 link#9 UC 0 0 - tap0

This seems right.

Post by Gert Doering
fe80::f00b:a4ff:fe00:4dd2%tap0 f2:0b:a4:00:4d:d2 UHL 0 0 - lo0

This one seems wrong. I have no such entry but instead have an ND cache
entry. I suspect this and the missing cloning route are related.

Post by Gert Doering
ff01:9::/32 link#9 UC 0 0 - tap0
ff02::%tap0/32 link#9 UC 0 0 - tap0

I have those (well, 7 not 9, but that's the ifindex I'm pretty sure).

Post by Gert Doering
# ifconfig tap0
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
address: f2:0b:a4:00:4d:d2
media: Ethernet autoselect
inet6 fe80::f00b:a4ff:fe00:4dd2%tap0 prefixlen 64 scopeid 0x9
inet6 2001:608:4:a053::1:0 prefixlen 64
... so where does the route for the /64 prefix disappear to?

It is perhaps not disappearing, but not getting added.

Run 'route -n monitor' during this whole exercise to see what happens.

Post by Gert Doering
# uname -a
NetBSD zeta.medat.de 5.1_STABLE NetBSD 5.1_STABLE (GENERIC) #0: Tue
Feb 15 15:46:59 CET 2011
sparc64

Note that you are doing this on sparc64, which is LP64, and big endian.
I predict that this will turn out to matter (but mine works).

Post by Gert Doering
As a cross-check, I just tried this on NetBSD 3.1, and indeed, the route
auto-appears on tap0, and ND starts
22:20:00.348908 2001:608:4:a253::1:0 > ff02::1:ff00:2: icmp6: neighbor
sol: who has 2001:608:4:a253::2
(different prefix deliberately chosen)

What hardware?

Sorry for being misleading. That's an extra network that is just to
be routed across the tap0 /64, but I didn't want to modify the output
of "netstat -rn" in any way.

The /56 route doesn't seem wrong, but I asked because it seems
unnecessary to illustrate the problem and I thought perhaps there was
something else that might turn out to matter.

Yes, sorry for that (usually I grep for ":608:4:", knowing that this
network must only ever appear on tap0, but I wanted to include the fe80
stuff).

I didn't mean to complain about what you sent so much as suggest that
looking more broadly would be helpful. Often I figure things out when
something I didn't think to look at jumps out at me. That leads me to
the general strategy of looking at everything and then filtering.

Post by Gert Doering
Here's "netstat -rn -f inet6" after the ifconfig sequence from above,
2001:608:4:a053::1:0 f2:0b:a4:00:4d:d2 UHL 0 0 - lo0
fe80::%tap0/64 link#9 UC 0 0 - tap0
ff01:9::/32 link#9 UC 0 0 - tap0
ff02::%tap0/32 link#9 UC 0 0 - tap0

That UHL route on lo0 also seems wrong, but I suspect that it's a
consequence of the cloning route not getting added.

Yes, in v4 this all works.

I just did this on a sparc64 running 5.1_RC3. (I know that's old, and I
will update to recent netbsd-5 and can try again if you're still having
trouble.). After "ifconfig tap0 inet6 2001:db8::1", diff -u of "netstat
-nr -f inet6" taken before and after shows:

+2001:db8::/64 link#7 UC 0 0 - tap0 =>

and route -n monitor showed (annotated):

got message of size 24 on Fri Sep 16 20:29:41 2011
RTM_IFANNOUNCE: iface arrival/departure: len 24, if# 7, what: arrival

From 'ifconfig tap0 create'

got message of size 108 on Fri Sep 16 20:29:55 2011
RTM_NEWADDR: address being added to iface: len 108, metric 0, flags:
sockaddrs: <NETMASK,IFP,IFA>
ffff:ffff:ffff:ffff:: tap0:f2.b.a4.0.d.3e 2001:db8::1

adding the address (a /64)

got message of size 168 on Fri Sep 16 20:29:55 2011
RTM_ADD: Add Route: len 168, pid 0, seq 0, errno 0, flags: <UP,HOST,LLINFO>
locks: inits:
sockaddrs: <DST,GATEWAY>
2001:db8::1 f2.b.a4.0.d.3e

adding the ND entry for ourselves, which ndp -an shows as:
2001:db8::1 f2:0b:a4:00:0d:3e tap0 permanent R

got message of size 248 on Fri Sep 16 20:29:55 2011
RTM_ADD: Add Route: len 248, pid 0, seq 0, errno 0, flags: <UP,DONE,CLONING>
locks: inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
2001:db8:: ffff:ffff:ffff:ffff:: tap0:f2.b.a4.0.d.3e 2001:db8::1

This is the cloning route above.

got message of size 108 on Fri Sep 16 20:29:55 2011
RTM_NEWADDR: address being added to iface: len 108, metric 0, flags:
sockaddrs: <NETMASK,IFP,IFA>
ffff:ffff:ffff:ffff:: tap0:f2.b.a4.0.d.3e fe80::f00b:a4ff:fe00:d3e%tap0
got message of size 168 on Fri Sep 16 20:29:55 2011

link-local. apparently this is happening because the interface comes up
when I gave it an inet6 address

RTM_ADD: Add Route: len 168, pid 0, seq 0, errno 0, flags: <UP,HOST,LLINFO>
locks: inits:
sockaddrs: <DST,GATEWAY>
fe80::f00b:a4ff:fe00:d3e%tap0 f2.b.a4.0.d.3e
got message of size 248 on Fri Sep 16 20:29:55 2011

ND entry for our link-local address.

RTM_ADD: Add Route: len 248, pid 0, seq 0, errno 0, flags: <UP,DONE,CLONING>
locks: inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
fe80::%tap0 ffff:ffff:ffff:ffff:: tap0:f2.b.a4.0.d.3e 2001:db8::1

link-local cloning route.

I deleted the 2001:db8::1 address, and on re-adding it, I see only
NEWADDR, ADD of ND entry and ADD of cloning route. Which all seems
fine.

So the big mystery is why your machine is not adding the cloning route.

Are you using that prefix on any other interface (you can't)???

Gert Doering

2011-09-17 16:12:43 UTC

Hi,

Post by Greg Troxel
Is the prefix you are using on the tap different than the one on your
real interface?

Yes. hme0 has 2001:608:8003::/64, tap0 has 2001:608:4:a051::/64.

Nowhere close.

Post by Greg Troxel
ip6mode=autohost is supposed to, more or less, configure a system to
accept router advertisements
send router solicitations

... which should be fairly harmless, and "a fairly typical thing for a
workstation".

Post by Greg Troxel
I think there's a general notion in the specs that an autoconfigured
host is only allowed to have one interface.

Which is fine from the point of view "it shouldn't be acting as a router",
but for the generic VPN case, this is too strict.

(Of course, "a *proper* VPN would be IPSEC", and anything else is an
abomination in the first place - but that doesn't really solve
real-world problems either)

Interesting enough, this doesn't pose a problem for OpenVPN "tun" usage,
and most likely not for other ppp-based VPNs (pptp/l2tp) either as there's
no ND involved anyway...

Post by Greg Troxel
But the kernel behavior seems bad.
~/NetBSD-current/src/sys > egrep accept_rtadv */*.c
and maybe this will become clearer.

I'll go digging and thinking... (and testing on Free- and OpenBSD whether
this is a common issue on Kame-based stacks)

Post by Gert Doering
fe80::f00b:a4ff:fe00:4dd2%tap0 f2:0b:a4:00:4d:d2 UHL 0 0 - lo0

This one seems wrong. I have no such entry but instead have an ND cache
entry. I suspect this and the missing cloning route are related.

I see "UHL... lo0" routes for all IPv6 addresses that the system has,
Sorry, I see those too, which makes sense.

OK, good. So it was just a "red herring" muddling the original issue :-)

regards,

gert

Greg Troxel

2011-09-17 20:44:53 UTC

Post by Greg Troxel
I think there's a general notion in the specs that an autoconfigured
host is only allowed to have one interface.

Which is fine from the point of view "it shouldn't be acting as a router",
but for the generic VPN case, this is too strict.

Not sure why you say "too strict". An autoconfigured host sets the
default route from RAs. With >=2 interfaces, the right behavior is
unclear. So you simply configure addresses manually and a default route
(or run ripng in listen-only mode).

It's fine to disagree with this philosophy (and I'm sympathetic in
part), but if that's what the specs say, it seems proper for NetBSD to
implement it. But definitely what you've observed is a bug.

Gert Doering

2011-09-17 20:58:18 UTC

Hi,

(we're now moving to philosophical land "how things should be")

Post by Greg Troxel
I think there's a general notion in the specs that an autoconfigured
host is only allowed to have one interface.

Which is fine from the point of view "it shouldn't be acting as a router",
but for the generic VPN case, this is too strict.

Not sure why you say "too strict".

Well, maybe a terminology issue. I'm perfectly fine with "only a single
interface at time can be autoconfiguring", but I don't think the specs
demand that "if an interface is autoconfiguring, manual configuration on
other interfaces should not be allowed".

If the latter interpretation is used, I think it's more strict that
necessary to achieve well-defined behaviour.

Post by Greg Troxel
An autoconfigured host sets the
default route from RAs. With >=2 interfaces, the right behavior is
unclear.

Install two default routes, decide by whichever algorithm which of the
two default routes to use.

How's that different from "one interface, receiving RAs from two different
routers"?

Post by Greg Troxel
So you simply configure addresses manually and a default route
(or run ripng in listen-only mode).

Well, I'm still thinking of the "workstation, autoconfigured for the
LAN network, but has VPN interface" usage case - there would be one
interface autoconfig'ed, and occasionally others, typically statically
configured. For that case, manual configuration "because I want VPN"
is a bit hard to explain to users.

Post by Greg Troxel
It's fine to disagree with this philosophy (and I'm sympathetic in
part), but if that's what the specs say, it seems proper for NetBSD to
implement it. But definitely what you've observed is a bug.

I'm not exactly sure that this is what the specs say... :-)

Linux (these days) has something called "hybrid router" mode where
RAs are accepted on some interfaces, and *sent* on others, which is what
you *need* if you want to implement a typical DSL router (which would
autoconfigure its WAN interface, and statically configure its LAN ifs,
sending out RAs to clients).

I'll need to go and read RFCs (starting with 6204 and the references in
there) to figure out whether this is sanctioned by the RFCs, or just
implemented as a matter of needed functionality :-)

gert

Gert Doering

2011-09-17 07:46:47 UTC

Hi,

first of all, thanks for your insights. Learned a lot today
("route -n monitor" and the "cloning" route stuff).

Post by Greg Troxel
It seems there is a bug in tap.
In systems derived from 4.4BSD, the connected route should appear
automatically when adding an address. I would say it's a bug for
systems that have the concept of cloning route not to add it
automatically when configuring an address.
summary: I tried to replicate on sparc64. Both it and i386 seem to work
correctly.

Things seem to be a bit more complicated than thought...

I rebooted (to test whether different values of ip6mode have an effect,
and to clear whatever garbage my previous route/ifconfig fumbling might
have left somewhere), and that makes it work "for a while"(!):

# date
Sat Sep 17 09:14:57 CEST 2011
# ifconfig tap0 create
# ifconfig tap0 inet6 2001:608:4:a053::1:0/64
# date
Sat Sep 17 09:15:32 CEST 2011
# netstat -rn -f inet6 |grep :4:
2001:608:4:a053::/64 link#3 UC 0 0 - tap0
2001:608:4:a053::1:0 f2:0b:a4:00:02:7f UHL 0 0 - lo0

(I seem to get these "UHL / lo0" entries for all "local ip address"
values)

... and after a while:

# date
Sat Sep 17 09:19:16 CEST 2011
# netstat -rn -f inet6 |grep :4:
2001:608:4:a053::1:0 f2:0b:a4:00:02:7f UHL 0 0 - lo0

... and when that happens, it will stay that way "no /64 UC" even if
I destroy and recreate the tap0.

"route -n monitor" shows the withdrawal of the /64 UC route:

got message of size 248 on Sat Sep 17 09:19:05 2011
RTM_DELETE: Delete Route: len 248, pid 0, seq 0, errno 0, flags: <DONE,CLONING>
locks: inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
2001:608:4:a053:: ffff:ffff:ffff:ffff:: tap0:f2.b.a4.0.2.7f 2001:608:4:a053::1:0
got message of size 108 on Sat Sep 17 09:19:05 2011
RTM_NEWADDR: address being added to iface: len 108, metric 0, flags:
sockaddrs: <NETMASK,IFP,IFA>
ffff:ffff:ffff:ffff:: hme0:8.0.20.c0.ff.ee 2001:608:8003:0:a00:20ff:fec0:ffee
got message of size 168 on Sat Sep 17 09:19:05 2011
RTM_ADD: Add Route: len 168, pid 0, seq 0, errno 0, flags: <UP,HOST,LLINFO>
locks: inits:
sockaddrs: <DST,GATEWAY>
2001:608:8003:0:a00:20ff:fec0:ffee 8.0.20.c0.ff.ee

... and that makes me suspect it has to do with "ip6mode=autohost", as
the tap0 route disappears the very moment a SLAAC'ed IPv6 address shows
up on hme0.

Trying again with "ip6mode=host"...

$ netstat -rn |grep :4:
2001:608:4:a053::/64 link#3 UC 0 0 - tap0
2001:608:4:a053::1:0 f2:0b:a4:00:04:a6 UHL 0 0 - lo0

... waiting for incoming RA...

09:31:33.962892 IP6 fe80::21d:73ff:feb1:919c > ff02::1: ICMP6, router advertisement, length 80

... route still there!

***@zeta.medat.de:/home/gert$ netstat -rn |grep :4:
2001:608:4:a053::/64 link#3 UC 0 0 - tap0
2001:608:4:a053::1:0 f2:0b:a4:00:04:a6 UHL 0 0 - lo0

... and in this scenario, OpenVPN-on-TAP does what I want (and the extra
"route" statement for the connected /64 properly gets rejected with
"route: writing to routing socket: File exists")

So now this brings up more questions :-)

- what exactly does "ip6mode=autohost" change?
- should it have this effect (UC routes on interface A going away
if a RA is seen on interface B, while the ipv6 address is still there)?

[..]

Post by Gert Doering
fe80::f00b:a4ff:fe00:4dd2%tap0 f2:0b:a4:00:4d:d2 UHL 0 0 - lo0

This one seems wrong. I have no such entry but instead have an ND cache
entry. I suspect this and the missing cloning route are related.

I see "UHL... lo0" routes for all IPv6 addresses that the system has,
and "UHLc" for ND (+arp) cache entries:

$ netstat -rn |grep UHL
10.100.53.1 6e:02:b7:bc:b7:1d UHLc 1 60 - tap0
172.30.1.1 e4:1f:13:8e:80:b0 UHLc 0 32 - hme0
172.30.1.5 e4:1f:13:8e:80:b0 UHLc 4 6179 - hme0
172.30.1.199 00:a0:f9:00:7c:fa UHLc 1 0 - hme0
2001:608:4:a053::1 6e:02:b7:bc:b7:1d UHLc 2 8 - tap0
2001:608:4:a053::1:0 f2:0b:a4:00:0a:32 UHL 1 0 - lo0
2001:608:8003::beef 08:00:20:c0:ff:ee UHL 0 0 - lo0
2001:608:8003::ffff 00:1d:73:b1:91:9c UHLc 1 14 - hme0
fe80::21d:73ff:feb1:919c%hme0 00:1d:73:b1:91:9c UHLc 0 45 - hme0
fe80::a00:20ff:fec0:ffee%hme0 08:00:20:c0:ff:ee UHL 0 0 - lo0
fe80::1%lo0 link#2 UHL 0 0 - lo0
fe80::f00b:a4ff:fe00:a32%tap0 f2:0b:a4:00:0a:32 UHL 0 0 - lo0

... I have no idea whether it should be that way or not, but it's
the same on other NetBSD machines I've checked.

[..]

Post by Greg Troxel
Note that you are doing this on sparc64, which is LP64, and big endian.
I predict that this will turn out to matter (but mine works).

Can't test on anything else, but this morning's test hint at "autohost"
side effects...

[..]

What hardware?

Also Sparc64. Right now, I don't have an i386 installation available.

But that machine has "ip6mode=host" (default setting).

[..]

Post by Greg Troxel
I just did this on a sparc64 running 5.1_RC3. (I know that's old, and I
will update to recent netbsd-5 and can try again if you're still having
trouble.). After "ifconfig tap0 inet6 2001:db8::1", diff -u of "netstat
+2001:db8::/64 link#7 UC 0 0 - tap0 =>

Does this system has IPv6 on another interface, with incoming router
advertisements? How is "ip6mode" set? (Just cross-checking).

thanks again,

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany ***@greenie.muc.de
fax: +49-89-35655025 ***@net.informatik.tu-muenchen.de

Greg Troxel

2011-09-17 13:20:06 UTC

Is the prefix you are using on the tap different than the one on your
real interface?

ip6mode=autohost is supposed to, more or less, configure a system to
accept router advertisements
send router solicitations

I think there's a general notion in the specs that an autoconfigured
host is only allowed to have one interface. But the kernel behavior
seems bad.

You can probably look at the code:
~/NetBSD-current/src/sys > egrep accept_rtadv */*.c

and maybe this will become clearer.