Discussion:
carp and routing
(too old to reply)
Stephen Borrill
2017-03-14 19:15:46 UTC
Permalink
I'm trying to set up redundant firewalls using carp(4) as detailed in
section 28.5 here:
https://www.netbsd.org/docs/guide/en/chap-carp.html

The examples ignore routing, especially setting a default gateway.

Machine 1:
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63

Machine 2:
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63

My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running

The routes looked OK the face of it:

Internet:
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0

But it didn't work:
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss

Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.

I then started up the second machine and looked its routing table:
Internet:
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 80.x.y.20 U - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 192.168.1.88 U - - - carp0

If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts
even though MAC address of the router does appear in the routing table
after a while:

Internet:
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms

----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss

A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Hauke Fath
2017-03-14 20:55:34 UTC
Permalink
While I have many problems with workplace's carp router pair, the
primary machine's default route is not one of them. (The fallback
machine's default route, though, is void, which might give problems
with ntp.)
Post by Stephen Borrill
default 80.x.y.62 UGS - - - carp1
My box says

% netstat -f inet -nr | grep 130.83.a
default 130.83.a.b UGS - - - wm0
130.83.a.b/29 link#2 UC - - - wm0
130.83.a.c 130.83.a.c UH - - -
carp0
130.83.a.d m:n:o:p:q:r UHLc - - - wm0
%

here, and I seem to recall you should generally not use the carpN
interfaces, but stick to the physical ones.

HTH,
hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-21344

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Stephen Borrill
2017-03-15 10:52:34 UTC
Permalink
Post by Hauke Fath
While I have many problems with workplace's carp router pair, the
primary machine's default route is not one of them. (The fallback
machine's default route, though, is void, which might give problems
with ntp.)
Does it not give problems after failover?
Post by Hauke Fath
Post by Stephen Borrill
default 80.x.y.62 UGS - - - carp1
My box says
% netstat -f inet -nr | grep 130.83.a
default 130.83.a.b UGS - - - wm0
130.83.a.b/29 link#2 UC - - - wm0
130.83.a.c 130.83.a.c UH - - -
carp0
130.83.a.d m:n:o:p:q:r UHLc - - - wm0
%
here, and I seem to recall you should generally not use the carpN
interfaces, but stick to the physical ones.
I'm guessing you have an IP address on wm0 and your carp0 address is a
/32. I'm trying to conserve IP addresses.

My workaround for this back in the NetBSD 6 days was to statically add an
arp entry for the router before setting up the default route. I'm sure
this worked, but since upgrading to NetBSD 7 it does not. I get
funky-looking routes like the following on the backup machine:

80.x.y.62 c4.71.fe.65.53.61 UHS - - - carp1
--
Stephen


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Stephen Borrill
2017-05-15 09:30:12 UTC
Permalink
I'm trying to set up redundant firewalls using carp(4) as detailed in section
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a sleep
5. This fixes it with a single machine. The routing table in both cases looks
identical.
Destination Gateway Flags Refs Use Mtu Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 80.x.y.20 U - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 192.168.1.88 U - - - carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down), machine
2 shows its interfaces as MASTER, but again, no route to hosts even though
Destination Gateway Flags Refs Use Mtu Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
I have a workaround which is to watch out for the carrier status using
ifwatchd and then refresh the default route. So:

/etc/kickroute contains:
#!/bin/sh
route change default `cat /etc/mygate`

rc.conf contains:
ifwatchd=YES
ifwatchd_flags="-c /etc/kickroute carp1"

rc.local contains:
sleep 5
/etc/kickroute
A similar problem happens at failback to the master. FreeBSD and OpenBSD have
similar problems reported too, but with no clear answers.
I tried FreeBSD 11 (which is rather different as carp is no longer a
pseudo-device, carp is configured on the hardware interface directly). It
works fine without any of the above hacks.
--
Stephen

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Ryota Ozaki
2017-05-18 06:39:30 UTC
Permalink
Hi,

I fixed an issue of CARP in -current, which is a regression between
-7 and -current, but I'm not sure the fix solves your problem
completely. Could you try the latest source code and report how the
fix changes the situation (or not).

Thanks,
ozaki-r

On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 80.x.y.20 U - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 192.168.1.88 U - - - carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts even
though MAC address of the router does appear in the routing table after a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Stephen Borrill
2017-05-18 11:44:48 UTC
Permalink
Post by Ryota Ozaki
I fixed an issue of CARP in -current, which is a regression between
-7 and -current, but I'm not sure the fix solves your problem
completely. Could you try the latest source code and report how the
fix changes the situation (or not).
Prior to your changes I found that with -current, I could not send any
packets at all even on the LAN (a regression from -7). With your changes I
find:
1) When first booted, machine 1 (Master) can send packets on LAN and via
default route (regression fixed)
2) Upon failover to machine 2, machine 2 can now send packets via default
route without having to do a "route change default" (fixed problem)
3) Upon failback to machine 1, machine 1 can no longer send packets
(regression still present)

I tried to apply your if_arp.c patch to -7. but the code is just too
different.
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 80.x.y.20 U - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 192.168.1.88 U - - - carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts even
though MAC address of the router does appear in the routing table after a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - - carp1
80.x.y.0/26 link#5 UC - - - carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - - carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - - carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Ryota Ozaki
2017-05-18 14:13:55 UTC
Permalink
On Thu, May 18, 2017 at 8:44 PM, Stephen Borrill
Post by Stephen Borrill
Post by Ryota Ozaki
I fixed an issue of CARP in -current, which is a regression between
-7 and -current, but I'm not sure the fix solves your problem
completely. Could you try the latest source code and report how the
fix changes the situation (or not).
Prior to your changes I found that with -current, I could not send any
packets at all even on the LAN (a regression from -7). With your changes I
1) When first booted, machine 1 (Master) can send packets on LAN and via
default route (regression fixed)
2) Upon failover to machine 2, machine 2 can now send packets via default
route without having to do a "route change default" (fixed problem)
Good.
Post by Stephen Borrill
3) Upon failback to machine 1, machine 1 can no longer send packets
(regression still present)
Could you show me the results of the following command on machine 1?
- ifconfig carp0
- ifconfig xennet0
- netstat -nr -f inet
- arp -na
Post by Stephen Borrill
I tried to apply your if_arp.c patch to -7. but the code is just too
different.
The patch can be applicable to -current and is even unnecessary to -7
because it fixes a regression introduced recently.

Thanks,
ozaki-r
Post by Stephen Borrill
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 80.x.y.20 U - - -
carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 192.168.1.88 U - - -
carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts even
though MAC address of the router does appear in the routing table after a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Stephen Borrill
2017-05-18 15:35:46 UTC
Permalink
Post by Ryota Ozaki
On Thu, May 18, 2017 at 8:44 PM, Stephen Borrill
Post by Stephen Borrill
Post by Ryota Ozaki
I fixed an issue of CARP in -current, which is a regression between
-7 and -current, but I'm not sure the fix solves your problem
completely. Could you try the latest source code and report how the
fix changes the situation (or not).
Prior to your changes I found that with -current, I could not send any
packets at all even on the LAN (a regression from -7). With your changes I
1) When first booted, machine 1 (Master) can send packets on LAN and via
default route (regression fixed)
2) Upon failover to machine 2, machine 2 can now send packets via default
route without having to do a "route change default" (fixed problem)
Good.
Post by Stephen Borrill
3) Upon failback to machine 1, machine 1 can no longer send packets
(regression still present)
I noticed that pinging after failback to machine 1 only failed if I had
pinged on machine 2. So I reasoned that the problem was because
switches, etc. hadn't noticed the change. I proved this by using arping
to send a gratuitous arp reply:
arping -c 1 -A -I carp0 192.168.1.88
arping -c 1 -A -I carp1 80.x.y.20

If the above commands are run after the interface becomes a master, then
it works. Are we missing out on sending a gratuitous arp after becomg
master? I notice OpenBSD send two, the second after a small delay:
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/ip_carp.c.diff?r1=1.127&r2=1.128&f=h
Post by Ryota Ozaki
Could you show me the results of the following command on machine 1?
- ifconfig carp0
- ifconfig xennet0
- netstat -nr -f inet
- arp -na
I'll skip these because of the above arp tests.
Post by Ryota Ozaki
Post by Stephen Borrill
I tried to apply your if_arp.c patch to -7. but the code is just too
different.
The patch can be applicable to -current and is even unnecessary to -7
because it fixes a regression introduced recently.
I don't think it is as simple as that. My email below (and elaborated on
here: http://mail-index.netbsd.org/tech-net/2017/05/15/msg006331.html )
describes a problem with -7 here you cannot route via the default gateway
without doing a "route change default" after becoming master. It appears
that with the correct gratuitous arps, -current works OK.
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 80.x.y.20 U - - -
carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 192.168.1.88 U - - -
carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts even
though MAC address of the router does appear in the routing table after a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648 lo0
127.0.0.1 127.0.0.1 UH - - 33648 lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Ryota Ozaki
2017-05-19 02:43:32 UTC
Permalink
On Fri, May 19, 2017 at 12:35 AM, Stephen Borrill
Post by Stephen Borrill
Post by Ryota Ozaki
On Thu, May 18, 2017 at 8:44 PM, Stephen Borrill
Post by Stephen Borrill
Post by Ryota Ozaki
I fixed an issue of CARP in -current, which is a regression between
-7 and -current, but I'm not sure the fix solves your problem
completely. Could you try the latest source code and report how the
fix changes the situation (or not).
Prior to your changes I found that with -current, I could not send any
packets at all even on the LAN (a regression from -7). With your changes I
1) When first booted, machine 1 (Master) can send packets on LAN and via
default route (regression fixed)
2) Upon failover to machine 2, machine 2 can now send packets via default
route without having to do a "route change default" (fixed problem)
Good.
Post by Stephen Borrill
3) Upon failback to machine 1, machine 1 can no longer send packets
(regression still present)
I noticed that pinging after failback to machine 1 only failed if I had
pinged on machine 2. So I reasoned that the problem was because switches,
etc. hadn't noticed the change. I proved this by using arping to send a
arping -c 1 -A -I carp0 192.168.1.88
arping -c 1 -A -I carp1 80.x.y.20
If the above commands are run after the interface becomes a master, then it
works. Are we missing out on sending a gratuitous arp after becomg master? I
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/ip_carp.c.diff?r1=1.127&r2=1.128&f=h
Thank you for the investigation.

I've written a patch to fix the issue in a different way from OpenBSD.
Can you try the patch?
http://www.netbsd.org/~ozaki-r/fix-carp-garp.diff

This is my guess how the regression was introduced:
(1) DAD for IPv4 and IN_IFF_DETACHED flag were introduced (pre -7)
- If IN_IFF_DETACHED flag is on an IP address, any packets won't
be sent via the IP address (including GARP packets)
- The flag is cleared by DAD that is kicked by say an event of
a link state change
(2) The link state change handler was changed to run in softint
(after -7)
(3) CARP was changed to use the handler (after -7)
- This allows CARP to kick DAD and clear IN_IFF_DETACHED flag
*eventually*
- OTOH, by the change, some operations are executed in reverse
- For example, CARP tries to send a GARP packet before the handler
is executed and fails to send it

And my patch allows CARP to execute the handler directly
(not via softint) before sending a GARP packet.
Post by Stephen Borrill
Post by Ryota Ozaki
Could you show me the results of the following command on machine 1?
- ifconfig carp0
- ifconfig xennet0
- netstat -nr -f inet
- arp -na
I'll skip these because of the above arp tests.
Post by Ryota Ozaki
Post by Stephen Borrill
I tried to apply your if_arp.c patch to -7. but the code is just too
different.
The patch can be applicable to -current and is even unnecessary to -7
because it fixes a regression introduced recently.
I don't think it is as simple as that. My email below (and elaborated on
here: http://mail-index.netbsd.org/tech-net/2017/05/15/msg006331.html )
describes a problem with -7 here you cannot route via the default gateway
without doing a "route change default" after becoming master. It appears
that with the correct gratuitous arps, -current works OK.
I guess pulling up the commit ip_carp.c,v 1.88 to -7 would fix the issue.
The commit is (3) in the above list and (2) isn't in -7 so it just fixes
the issue that CARP doesn't send GARP packets, without the regression
introduced by (3) in -current.

ozaki-r
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 80.x.y.20 U - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 192.168.1.88 U - - -
carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts even
though MAC address of the router does appear in the routing table after a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Stephen Borrill
2017-05-19 08:21:31 UTC
Permalink
Post by Ryota Ozaki
Post by Stephen Borrill
I noticed that pinging after failback to machine 1 only failed if I had
pinged on machine 2. So I reasoned that the problem was because switches,
etc. hadn't noticed the change. I proved this by using arping to send a
arping -c 1 -A -I carp0 192.168.1.88
arping -c 1 -A -I carp1 80.x.y.20
If the above commands are run after the interface becomes a master, then it
works. Are we missing out on sending a gratuitous arp after becomg master? I
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/ip_carp.c.diff?r1=1.127&r2=1.128&f=h
Thank you for the investigation.
I've written a patch to fix the issue in a different way from OpenBSD.
Can you try the patch?
http://www.netbsd.org/~ozaki-r/fix-carp-garp.diff
Yay, that works perfectly.
Post by Ryota Ozaki
(1) DAD for IPv4 and IN_IFF_DETACHED flag were introduced (pre -7)
- If IN_IFF_DETACHED flag is on an IP address, any packets won't
be sent via the IP address (including GARP packets)
- The flag is cleared by DAD that is kicked by say an event of
a link state change
(2) The link state change handler was changed to run in softint
(after -7)
(3) CARP was changed to use the handler (after -7)
- This allows CARP to kick DAD and clear IN_IFF_DETACHED flag
*eventually*
- OTOH, by the change, some operations are executed in reverse
- For example, CARP tries to send a GARP packet before the handler
is executed and fails to send it
And my patch allows CARP to execute the handler directly
(not via softint) before sending a GARP packet.
OK.
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
The patch can be applicable to -current and is even unnecessary to -7
because it fixes a regression introduced recently.
I don't think it is as simple as that. My email below (and elaborated on
here: http://mail-index.netbsd.org/tech-net/2017/05/15/msg006331.html )
describes a problem with -7 here you cannot route via the default gateway
without doing a "route change default" after becoming master. It appears
that with the correct gratuitous arps, -current works OK.
I guess pulling up the commit ip_carp.c,v 1.88 to -7 would fix the issue.
The commit is (3) in the above list and (2) isn't in -7 so it just fixes
the issue that CARP doesn't send GARP packets, without the regression
introduced by (3) in -current.
1.88 has already been pulled up to -7 (ticket #1420). It appears to make
things neither better nor worse.

The routing problem still persists on -7, you need to run the following
every time you become the master (even on first boot):

route change default `cat /etc/mygate`

On -current the routing problem has been fixed and your patch fixes the
missing GARPs.
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 80.x.y.20 U - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 192.168.1.88 U - - -
carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts even
though MAC address of the router does appear in the routing table after a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Ryota Ozaki
2017-05-19 09:01:55 UTC
Permalink
On Fri, May 19, 2017 at 5:21 PM, Stephen Borrill
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Stephen Borrill
I noticed that pinging after failback to machine 1 only failed if I had
pinged on machine 2. So I reasoned that the problem was because switches,
etc. hadn't noticed the change. I proved this by using arping to send a
arping -c 1 -A -I carp0 192.168.1.88
arping -c 1 -A -I carp1 80.x.y.20
If the above commands are run after the interface becomes a master, then it
works. Are we missing out on sending a gratuitous arp after becomg master? I
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/ip_carp.c.diff?r1=1.127&r2=1.128&f=h
Thank you for the investigation.
I've written a patch to fix the issue in a different way from OpenBSD.
Can you try the patch?
http://www.netbsd.org/~ozaki-r/fix-carp-garp.diff
Yay, that works perfectly.
Good :)
Post by Stephen Borrill
Post by Ryota Ozaki
(1) DAD for IPv4 and IN_IFF_DETACHED flag were introduced (pre -7)
- If IN_IFF_DETACHED flag is on an IP address, any packets won't
be sent via the IP address (including GARP packets)
- The flag is cleared by DAD that is kicked by say an event of
a link state change
(2) The link state change handler was changed to run in softint
(after -7)
(3) CARP was changed to use the handler (after -7)
- This allows CARP to kick DAD and clear IN_IFF_DETACHED flag
*eventually*
- OTOH, by the change, some operations are executed in reverse
- For example, CARP tries to send a GARP packet before the handler
is executed and fails to send it
And my patch allows CARP to execute the handler directly
(not via softint) before sending a GARP packet.
OK.
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
The patch can be applicable to -current and is even unnecessary to -7
because it fixes a regression introduced recently.
I don't think it is as simple as that. My email below (and elaborated on
here: http://mail-index.netbsd.org/tech-net/2017/05/15/msg006331.html )
describes a problem with -7 here you cannot route via the default gateway
without doing a "route change default" after becoming master. It appears
that with the correct gratuitous arps, -current works OK.
I guess pulling up the commit ip_carp.c,v 1.88 to -7 would fix the issue.
The commit is (3) in the above list and (2) isn't in -7 so it just fixes
the issue that CARP doesn't send GARP packets, without the regression
introduced by (3) in -current.
1.88 has already been pulled up to -7 (ticket #1420). It appears to make
things neither better nor worse.
The routing problem still persists on -7, you need to run the following
route change default `cat /etc/mygate`
On -current the routing problem has been fixed and your patch fixes the
missing GARPs.
Hmm, I have no idea for -7 for now. Roy, do you have any ideas on the issue?

ozaki-r
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 80.x.y.20 U - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 192.168.1.88 U - - -
carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts
even
though MAC address of the router does appear in the routing table
after
a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Ryota Ozaki
2017-05-19 09:12:18 UTC
Permalink
Post by Ryota Ozaki
On Fri, May 19, 2017 at 5:21 PM, Stephen Borrill
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Stephen Borrill
I noticed that pinging after failback to machine 1 only failed if I had
pinged on machine 2. So I reasoned that the problem was because switches,
etc. hadn't noticed the change. I proved this by using arping to send a
arping -c 1 -A -I carp0 192.168.1.88
arping -c 1 -A -I carp1 80.x.y.20
If the above commands are run after the interface becomes a master, then it
works. Are we missing out on sending a gratuitous arp after becomg master? I
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/ip_carp.c.diff?r1=1.127&r2=1.128&f=h
Thank you for the investigation.
I've written a patch to fix the issue in a different way from OpenBSD.
Can you try the patch?
http://www.netbsd.org/~ozaki-r/fix-carp-garp.diff
Yay, that works perfectly.
Good :)
Post by Stephen Borrill
Post by Ryota Ozaki
(1) DAD for IPv4 and IN_IFF_DETACHED flag were introduced (pre -7)
Oops. This was wrong. DAD for IPv4 was implemented after branching -7.
So Roy is perhaps not related to the issue.

ozaki-r
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
- If IN_IFF_DETACHED flag is on an IP address, any packets won't
be sent via the IP address (including GARP packets)
- The flag is cleared by DAD that is kicked by say an event of
a link state change
(2) The link state change handler was changed to run in softint
(after -7)
(3) CARP was changed to use the handler (after -7)
- This allows CARP to kick DAD and clear IN_IFF_DETACHED flag
*eventually*
- OTOH, by the change, some operations are executed in reverse
- For example, CARP tries to send a GARP packet before the handler
is executed and fails to send it
And my patch allows CARP to execute the handler directly
(not via softint) before sending a GARP packet.
OK.
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
The patch can be applicable to -current and is even unnecessary to -7
because it fixes a regression introduced recently.
I don't think it is as simple as that. My email below (and elaborated on
here: http://mail-index.netbsd.org/tech-net/2017/05/15/msg006331.html )
describes a problem with -7 here you cannot route via the default gateway
without doing a "route change default" after becoming master. It appears
that with the correct gratuitous arps, -current works OK.
I guess pulling up the commit ip_carp.c,v 1.88 to -7 would fix the issue.
The commit is (3) in the above list and (2) isn't in -7 so it just fixes
the issue that CARP doesn't send GARP packets, without the regression
introduced by (3) in -current.
1.88 has already been pulled up to -7 (ticket #1420). It appears to make
things neither better nor worse.
The routing problem still persists on -7, you need to run the following
route change default `cat /etc/mygate`
On -current the routing problem has been fixed and your patch fixes the
missing GARPs.
Hmm, I have no idea for -7 for now. Roy, do you have any ideas on the issue?
ozaki-r
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Stephen Borrill
Post by Ryota Ozaki
Post by Ryota Ozaki
On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
Post by Stephen Borrill
I'm trying to set up redundant firewalls using carp(4) as detailed in
https://www.netbsd.org/docs/guide/en/chap-carp.html
The examples ignore routing, especially setting a default gateway.
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
address: 00:00:5e:00:01:01
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
address: 00:00:5e:00:01:01
inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
address: 00:00:5e:00:01:02
inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just
mahcine 1 running
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the
route, I added the "route add default" command to /etc/rc.local after a
sleep 5. This fixes it with a single machine. The routing table in both
cases looks identical.
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 80.x.y.20 U - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 192.168.1.88 U - - -
carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
machine 2 shows its interfaces as MASTER, but again, no route to hosts
even
though MAC address of the router does appear in the routing table
after
a
Destination Gateway Flags Refs Use Mtu
Interface
default 80.x.y.62 UGS - - -
carp1
80.x.y.0/26 link#5 UC - - -
carp1
80.x.y.62 c4:71:fe:65:53:61 UHLc - - -
carp1
127/8 127.0.0.1 UGRS - - 33648
lo0
127.0.0.1 127.0.0.1 UH - - 33648
lo0
192.168.1/24 link#4 UC - - -
carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
A similar problem happens at failback to the master. FreeBSD and OpenBSD
have similar problems reported too, but with no clear answers.
--
Stephen
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Roy Marples
2017-05-19 10:56:04 UTC
Permalink
Post by Ryota Ozaki
Post by Ryota Ozaki
(1) DAD for IPv4 and IN_IFF_DETACHED flag were introduced (pre -7)
Oops. This was wrong. DAD for IPv4 was implemented after branching -7.
So Roy is perhaps not related to the issue.
It's good to know that I'm not related to carp :)

Roy

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Loading...