Static interface routes for IP address on different subnet doesn't send an ARP who-has

[Peter Kieser]

Hello,

I have an ISP that requires that I use a subnet mask of 255.255.255.255
(/32) which requires that I add a static interface route for the gateway
IP address because it's on a different subnet
(http://help.ovh.ie/BridgeClient.) In Linux and FreeBSD this works as
intended, and the network stack sends an ARP request for the gateway IP
address when a static route is added.

On NetBSD, the network stack seems to make up a random MAC address and I
don't see an ARP who-has request sent. Below, you can see that the MAC
address of the gateway (198.50.219.62) is correct when the subnet is set
to /27:

watson# arp -a
? (198.50.219.62) at 52:54:00:07:a6:f6 on vioif0
watson# ifconfig vioif0
vioif0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
address: 52:54:00:24:c2:c3
inet 198.50.219.49 netmask 0xffffffe0 broadcast 198.50.219.63
inet6 fe80::5054:ff:fe24:c2c3%vioif0 prefixlen 64 scopeid 0x1

If I set the subnet and broadcast domain how my ISP wants it, and follow
http://www.netbsd.org/docs/network/#nonsubnetgateway the MAC address is
replaced with garbage data:

watson# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255
broadcast 198.50.219.49
watson# route add -host 198.50.219.62 -link -iface vioif0
add host 198.50.219.62: gateway vioif0
watson# ping 198.50.219.62
PING 198.50.219.62 (198.50.219.62): 48 data bytes
^C
----198.50.219.62 PING Statistics----
3 packets transmitted, 0 packets received, 100.0% packet loss
watson# arp -a
? (198.50.219.62) at 76:69:6f:69:66:30:00:0e:00:00:00:00 on vioif0 permanent

Now, if I add the MAC address of the gateway (which isn't a reasonable
solution, since my ISP has HRSP and other failovers setup) everything
works out:

watson# route add -host 198.50.219.62 -link -iface vioif0:52540007a6f6

Does anyone have any idea how to get this configuration to work without
the MAC address of the gateway being specified for the route? I feel
like this is a bug, as with Linux and FreeBSD this configuration (while
I believe it's not particular sane) works just fine, the operating
system does an ARP who-has request and you can access the default gateway.

Thank you,

-Peter

[Greg Troxel]

This does seem like a bug. It's interesting that it works on FreeBSD,
since much of the code has shared heritage.

The basic approach to arp is to have a cloning route ("C" flag) which
causes arp procesing for addresses within the prefix but for which there
isn't an llinfo (L) entry.

route add -host 198.50.219.62 -link -iface vioif0

Why are you giving "-link", with no mac address? That seems wrong, and
it seems like a bug in route not to throw an error. The trick is to
have a routing table entry that will provoke arp.

What command do you run on FreeBSD, and what does the routing table look
like?

I suggest running 'route -n monitor' while doing all of this, to watch
the routing socket.

[Peter Kieser]

Hello,

On FreeBSD 9.1-RELEASE, the network stack sends an ARP who-has request
with the following network settings:

# ifconfig em0 inet 198.50.219.49 netmask 255.255.255.255 broadcast
198.50.219.49
# route add -host 198.50.219.62 -link -iface em0
add host 198.50.219.62: gateway em0
# ping 198.50.219.49
PING 198.50.219.49 (198.50.219.49): 56 data bytes
64 bytes from 198.l50.219.l49: icmp_seq=0 ttl=64 time=0.031 ms
64 bytes from 198.l50.219.l49: icmp_seq=1 ttl=64 time=0.026 ms
^C
--- 198.50.219.49 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.026/0.028/0.031/0.003 ms
# arp -a
? (198.50.219.49) at 52:54:00:51:57:30 on em0 permanent [ethernet]

How would you get a routing table entry that would provoke ARP? Even the
NetBSD FAQ sdoesn't have a MAC address listed:

http://www.netbsd.org/docs/network/#nonsubnetgateway

-Peter

Post by Greg Troxel
This does seem like a bug. It's interesting that it works on FreeBSD,
since much of the code has shared heritage.
The basic approach to arp is to have a cloning route ("C" flag) which
causes arp procesing for addresses within the prefix but for which there
isn't an llinfo (L) entry.
route add -host 198.50.219.62 -link -iface vioif0
Why are you giving "-link", with no mac address? That seems wrong, and
it seems like a bug in route not to throw an error. The trick is to
have a routing table entry that will provoke arp.
What command do you run on FreeBSD, and what does the routing table look
like?
I suggest running 'route -n monitor' while doing all of this, to watch
the routing socket.

[David Young]

Post by Greg Troxel
This does seem like a bug. It's interesting that it works on FreeBSD,
since much of the code has shared heritage.
The basic approach to arp is to have a cloning route ("C" flag) which
causes arp procesing for addresses within the prefix but for which there
isn't an llinfo (L) entry.
route add -host 198.50.219.62 -link -iface vioif0
Why are you giving "-link", with no mac address? That seems wrong, and
it seems like a bug in route not to throw an error. The trick is to
have a routing table entry that will provoke arp.

-link -iface vioif0 makes the kernel create a route with an AF_LINK
sockaddr_dl nexthop whose name is filled w/ vioif0.

I don't think that ARP will operate unless the route is cloning. Add
the -cloning flag?

It's strange that the MAC was filled with some arbitrary bytes, that
sounds like a bug.

Dave

[Greg Troxel]

You should look at the actual routes in the kernel on FreeBSD, with
route -n monitor, and understand all the flags and sockaddrs. Please
also send the result of 'netstat -nr' on FreeBSD.

As David suggests, you may need a /32 cloning route.

Sorry, but you are basically doing something that does not conform to
Internet standards, and that will probably require some code reading. I
think we'd be inclined to take a change that enables doing this
(assuming it doesn't break other things), and it's quite possible that
kernel code changes will be required.

I would also suggest looking through the history of FreeBSD to see if
they made an explicit change to support this multiple-/32-on-a-wire
case.

[John Hawkinson]

It's a pointless waste to blow a /30 on a point-to-point link
if you don't have to. Supporting this is smart.

--***@mit.edu
John Hawkinson

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[John Nemeth]

On Aug 22, 8:22pm, John Hawkinson wrote:
}
} It's a pointless waste to blow a /30 on a point-to-point link
} if you don't have to. Supporting this is smart.

Then use a /31. See RFC 3021

}-- End of excerpt from John Hawkinson

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

Post by John Nemeth

It's a pointless waste to blow a /30 on a point-to-point link if you
don't have to. Supporting this is smart.

Then use a /31. See RFC 3021

Except, as I recall, this isn't about point-to-point links. It's about
a broadcast link where the next-hop address bears no particular
relation to the local interface's address.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[John Nemeth]

On Aug 23, 10:10am, Mouse wrote:
}
} >> It's a pointless waste to blow a /30 on a point-to-point link if you
} >> don't have to. Supporting this is smart.
} > Then use a /31. See RFC 3021
}
} Except, as I recall, this isn't about point-to-point links. It's about
} a broadcast link where the next-hop address bears no particular
} relation to the local interface's address.

True for the original poster. Not so true for the person that
made the /30 comment, to which I responded.

}-- End of excerpt from Mouse

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by John Nemeth
}
} It's a pointless waste to blow a /30 on a point-to-point link
} if you don't have to. Supporting this is smart.
Then use a /31. See RFC 3021

ISPs don't want to use a /31 because it is also wasteful: to establish
point-to-point links between the gateway and N customers, you need 2N
addresses instead of N + 1 addresses.

Dave

[John Nemeth]

On Aug 23, 9:40am, David Young wrote:
} On Fri, Aug 23, 2013 at 12:45:35AM -0700, John Nemeth wrote:
} > On Aug 22, 8:22pm, John Hawkinson wrote:
} > }
} > } It's a pointless waste to blow a /30 on a point-to-point link
} > } if you don't have to. Supporting this is smart.
} >
} > Then use a /31. See RFC 3021
}
} ISPs don't want to use a /31 because it is also wasteful: to establish
} point-to-point links between the gateway and N customers, you need 2N
} addresses instead of N + 1 addresses.

Then don't pretend it is point-to-point when it isn't (most
"broadband" technologies aren't truely point-to-point). And, if
it is truely point-to-point then that's the way it is. And, 2N is
a heck of a lot better then 4N.

}-- End of excerpt from David Young

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Peter Kieser]

I tried the route cloning suggestion to see if it would work for the
/32, on NetBSD 6.1:

# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255 broadcast
198.50.219.49
# route add -host 198.50.219.62 -link -cloning -iface vioif0
add host 198.50.219.62: gateway vioif0
# ping 198.50.219.62
PING 198.50.219.62 (198.50.219.62): 48 data bytes
arplookup: unable to enter address for ***@ff on null (could
not allocate llinfo)
arpresolve: can't allocate llinfo on vioif0 for 198.50.219.62
^C
----198.50.219.62 PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss

As for FreeBSD 9.1, here is the output of "route -n monitor" when I add
the static interface route for the gateway:

# route -n monitor
got message of size 224 on Thu Aug 22 16:55:41 2013
RTM_ADD: Add Route: len 224, pid: 706, seq 1, errno 0,
flags:<UP,HOST,DONE,STATIC>
locks: inits:
sockaddrs: <DST,GATEWAY>
198.50.219.62 em0:52.54.0.40.aa.16

And an output of "netstat -rn" from the host (please excuse any
formatting issues, I am running this under qemu as VNC since I don't
have SSH access to the host when I'm doing this):

# netstat -rn
Routing tables

Internet:

Destination Gateway Flags Refs Use Netif Expire
127.0.0.1 link#3 UH 0 4 lo0
198.50.219.49 link#2 UHS 0 0 lo0 =>
198.50.219.49/32 link#2 U 0 0 em0
198.50.219.62 52:54:00:40:aa:16 UHS 0 4 em0

.. (Output omitted as it's IPv6)

I can also change the MAC address of the Gateway, and FreeBSD updates it
after the ARP timeout period.

-Peter

Post by Greg Troxel
You should look at the actual routes in the kernel on FreeBSD, with
route -n monitor, and understand all the flags and sockaddrs. Please
also send the result of 'netstat -nr' on FreeBSD.
As David suggests, you may need a /32 cloning route.
Sorry, but you are basically doing something that does not conform to
Internet standards, and that will probably require some code reading. I
think we'd be inclined to take a change that enables doing this
(assuming it doesn't break other things), and it's quite possible that
kernel code changes will be required.
I would also suggest looking through the history of FreeBSD to see if
they made an explicit change to support this multiple-/32-on-a-wire
case.

[Peter Kieser]

Using -net and -cloning together does the trick:

# route add -net 198.50.219.62/32 -link -cloning -iface vioif0

Destination Gateway Flags Refs Use Mtu
Interface
198.50.219.49 52:54:00:40:aa:16 UHLc 0 8
- lo0 =>
198.50.219.49/32 link#1 UC 1 0
- vioif0
198.50.219.62 52:54:00:07:a6:f6 UHLc 0 3
- vioif0 =>
198.50.219.62/32 link#1 UCS 1 0
- vioif0

-Peter

Post by David Young

Post by Peter Kieser
I tried the route cloning suggestion to see if it would work for the
# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255
broadcast 198.50.219.49
# route add -host 198.50.219.62 -link -cloning -iface vioif0
add host 198.50.219.62: gateway vioif0

What's the routing table in NetBSD look like at each step?
Hmm, -host doesn't make sense with -cloning. Perhaps you will get a
different result from 'add -net' instead of 'add -host'?
Dave

[Peter Kieser]

So just to wrap up - this configuration did the trick for my ISP, I
think the documentation should be updated (at
http://www.netbsd.org/docs/network/#nonsubnetgateway) this was for my setup:

198.50.219.49 = Failover IP
198.27.66.254 = Dedicated server gateway IP
vioif0 = Ethernet Interface

# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255 broadcast
198.50.219.49
# route add -net 198.27.66.254/32 -cloning -link -iface vioif0
# route add default -ifa 198.50.219.49 198.27.66.254

So if the documentation were to be updated on the NetBSD site (as I'm
sure I'm not the only person that ran into this issue) it would be:

# ifconfig fxp0 inet 10.0.0.1
# route add -net 192.168.0.1/32 -cloning -link fxp0 -iface
# route add default -ifa 10.0.0.1 192.168.0.1

Cheers,

-Peter

Post by Peter Kieser
# route add -net 198.50.219.62/32 -link -cloning -iface vioif0
Destination Gateway Flags Refs Use
Mtu Interface
198.50.219.49 52:54:00:40:aa:16 UHLc 0 8
- lo0 =>
198.50.219.49/32 link#1 UC 1 0
- vioif0
198.50.219.62 52:54:00:07:a6:f6 UHLc 0 3
- vioif0 =>
198.50.219.62/32 link#1 UCS 1 0
- vioif0
-Peter

Post by David Young

Post by Peter Kieser
I tried the route cloning suggestion to see if it would work for the
# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255
broadcast 198.50.219.49
# route add -host 198.50.219.62 -link -cloning -iface vioif0
add host 198.50.219.62: gateway vioif0

What's the routing table in NetBSD look like at each step?
Hmm, -host doesn't make sense with -cloning. Perhaps you will get a
different result from 'add -net' instead of 'add -host'?
Dave

[David Young]

Great!

I knew that I had once made this work, I'm glad it still works,
today.

Dave

[David Young]

Post by Peter Kieser
I tried the route cloning suggestion to see if it would work for the
# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255
broadcast 198.50.219.49
# route add -host 198.50.219.62 -link -cloning -iface vioif0
add host 198.50.219.62: gateway vioif0

What's the routing table in NetBSD look like at each step?

Hmm, -host doesn't make sense with -cloning. Perhaps you will get a
different result from 'add -net' instead of 'add -host'?

Dave

[David Young]

Post by Greg Troxel
Sorry, but you are basically doing something that does not conform to
Internet standards, and that will probably require some code reading.

It's customary to configure a small subnet for host and gateway to
share, and then to configure a route to the gateway, but is that
because an Internet standard says it should be so (and why does it say
so?) or because that is easiest or the only feasible way in the BSD
implementation?

Dave

[John Nemeth]

On Aug 22, 11:54am, David Young wrote:
} On Thu, Aug 22, 2013 at 12:39:04PM -0400, Greg Troxel wrote:
} > Sorry, but you are basically doing something that does not conform to
} > Internet standards, and that will probably require some code reading.
}
} It's customary to configure a small subnet for host and gateway to
} share, and then to configure a route to the gateway, but is that
} because an Internet standard says it should be so (and why does it say

In order to access a node that is "off net" you need to go
through a gateway. That implies that the gateway MUST be "on net",
otherwise you have a catch-22 (i.e. how do you get to the gateway
when it is "off net". This ISPs configuration is just bogus!

}-- End of excerpt from David Young

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by David Young

Post by Greg Troxel
Sorry, but you are basically doing something that does not conform to
Internet standards, and that will probably require some code reading.

It's customary to configure a small subnet for host and gateway to
share, and then to configure a route to the gateway, but is that
because an Internet standard says it should be so (and why does it say
so?) or because that is easiest or the only feasible way in the BSD
implementation?

I'd say that it is irregular to expect a host (or router) to arp for and
send packets to another hour/router that does not fall within the
prefix. It's difficult in BSD precisely because it was irregular (and
unthinkably bizarre) back in the day.

So this is really about "how do you send a packet to another host on
your subet", and there's no real support in standards that I know of for
what to do about sending packets on a broadcast interface to hosts that
are not within that interface's prefix.

So perhaps I'm wrong, and it would be interesting to see
standaards-track RFC text that suggests the disjoint-/32 approach is ok.

[Christos Zoulas]

-=-=-=-=-=-
Hello,
I have an ISP that requires that I use a subnet mask of 255.255.255.255
(/32) which requires that I add a static interface route for the gateway
IP address because it's on a different subnet
(http://help.ovh.ie/BridgeClient.) In Linux and FreeBSD this works as
intended, and the network stack sends an ARP request for the gateway IP
address when a static route is added.
On NetBSD, the network stack seems to make up a random MAC address and I
don't see an ARP who-has request sent. Below, you can see that the MAC
address of the gateway (198.50.219.62) is correct when the subnet is set
watson# arp -a
? (198.50.219.62) at 52:54:00:07:a6:f6 on vioif0
watson# ifconfig vioif0
vioif0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
address: 52:54:00:24:c2:c3
inet 198.50.219.49 netmask 0xffffffe0 broadcast 198.50.219.63
inet6 fe80::5054:ff:fe24:c2c3%vioif0 prefixlen 64 scopeid 0x1
If I set the subnet and broadcast domain how my ISP wants it, and follow
http://www.netbsd.org/docs/network/#nonsubnetgateway the MAC address is
watson# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255
broadcast 198.50.219.49
watson# route add -host 198.50.219.62 -link -iface vioif0
add host 198.50.219.62: gateway vioif0
watson# ping 198.50.219.62
PING 198.50.219.62 (198.50.219.62): 48 data bytes
^C
----198.50.219.62 PING Statistics----
3 packets transmitted, 0 packets received, 100.0% packet loss
watson# arp -a
? (198.50.219.62) at 76:69:6f:69:66:30:00:0e:00:00:00:00 on vioif0 permanent
Now, if I add the MAC address of the gateway (which isn't a reasonable
solution, since my ISP has HRSP and other failovers setup) everything
watson# route add -host 198.50.219.62 -link -iface vioif0:52540007a6f6
Does anyone have any idea how to get this configuration to work without
the MAC address of the gateway being specified for the route? I feel
like this is a bug, as with Linux and FreeBSD this configuration (while
I believe it's not particular sane) works just fine, the operating
system does an ARP who-has request and you can access the default gateway.

Looks like a bug :-) What is the correct MAC for the gateway?

christos


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Christos Zoulas

-=-=-=-=-=-
Hello,
I have an ISP that requires that I use a subnet mask of 255.255.255.255
(/32) which requires that I add a static interface route for the gateway
IP address because it's on a different subnet
(http://help.ovh.ie/BridgeClient.) In Linux and FreeBSD this works as
intended, and the network stack sends an ARP request for the gateway IP
address when a static route is added.
On NetBSD, the network stack seems to make up a random MAC address and I
don't see an ARP who-has request sent. Below, you can see that the MAC
address of the gateway (198.50.219.62) is correct when the subnet is set
watson# arp -a
? (198.50.219.62) at 52:54:00:07:a6:f6 on vioif0
watson# ifconfig vioif0
vioif0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
address: 52:54:00:24:c2:c3
inet 198.50.219.49 netmask 0xffffffe0 broadcast 198.50.219.63
inet6 fe80::5054:ff:fe24:c2c3%vioif0 prefixlen 64 scopeid 0x1
If I set the subnet and broadcast domain how my ISP wants it, and follow
http://www.netbsd.org/docs/network/#nonsubnetgateway the MAC address is
watson# ifconfig vioif0 inet 198.50.219.49 netmask 255.255.255.255
broadcast 198.50.219.49
watson# route add -host 198.50.219.62 -link -iface vioif0
add host 198.50.219.62: gateway vioif0
watson# ping 198.50.219.62
PING 198.50.219.62 (198.50.219.62): 48 data bytes
^C
----198.50.219.62 PING Statistics----
3 packets transmitted, 0 packets received, 100.0% packet loss
watson# arp -a
? (198.50.219.62) at 76:69:6f:69:66:30:00:0e:00:00:00:00 on vioif0 permanent
Now, if I add the MAC address of the gateway (which isn't a reasonable
solution, since my ISP has HRSP and other failovers setup) everything
watson# route add -host 198.50.219.62 -link -iface vioif0:52540007a6f6
Does anyone have any idea how to get this configuration to work without
the MAC address of the gateway being specified for the route? I feel
like this is a bug, as with Linux and FreeBSD this configuration (while
I believe it's not particular sane) works just fine, the operating
system does an ARP who-has request and you can access the default gateway.

Looks like a bug :-) What is the correct MAC for the gateway?

Christos is right. The problem with a netmask of 255.255.255.255 reminds me
of getting a 3G/4G USB modem to work because that was how the telco was
answering the conversation.

And looking at how you went about getting it to work does not seem right.

In this case the vioif is not a broadcast link (subnet mask is
255.255.255.255
so quite clearly nothing else can be on that subnet), it is a point to
point link.

Did you try doing this:

# ifconfig vioif0 inet 198.50.219.49 198.50.219.62

?

Cheers,
Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de