Bridged ethernet with ipnat redirect to local port - getting ICMP redirects instead

[Thomas Bieg]

Hello,

I am stuck trying to redirect HTTP requests targeted outside to a local httpd
via a bridged and ipf'ed ethernet port.

The bridge machine is running NetBSD 6.1_STABLE as of two weeks ago with a
custom kernel that's basically GENERIC + BRIDGE_IPF enabled.

- re0 is 192.168.1.1, where the httpd is listening.
- re0 is connected to a LAN with 192.168.1.2 as internet gateway (does DHCP and
DNS).

- re1 has no ip.
- re1 is bridged to re0 with ipf enabled.
- re1 is directly connected to the machine (a "smart" TV actually) where the
requests to be redirected are originating from (which succesfully gets its
192.168.1.x IP from 192.168.1.2 over the bridge and can access LAN and
internet just fine if I allow it).

- ipnat.conf has:
rdr re1 1.2.3.4/32 port 80 -> 192.168.1.1 port 80

(IP forwarding is also enabled, but as I understand it, that shouldn't even be
necessary.)

I was expecting/hoping ipnat would silently redirect connections coming in on
re1 and intended for 1.2.3.4 to the local httpd on re0, but instead it's sending
out ICMP redirects on re1.

Shouldn't that work? Or is there something I missed?


Best regards,
Tom

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Thomas Bieg
Hello,
I am stuck trying to redirect HTTP requests targeted outside to a local httpd
via a bridged and ipf'ed ethernet port.
...
I was expecting/hoping ipnat would silently redirect connections coming in on
re1 and intended for 1.2.3.4 to the local httpd on re0, but instead it's sending
out ICMP redirects on re1.
Shouldn't that work? Or is there something I missed?

First thing to try is to have the DHCP server include this line in
dhcpd.conf:
option wpad-url code 252 = text;
and then add this to the dhcp definition for your TV:
option wpad-url "http://192.168.1.1/wpad.dat";
with the appropriate data on your local web server in wpad.dat.

Yes, that's just a proxy definition but I suspect that what you want
to do is capture and control all WWW sessions from your TV andproxy
is another way to do that. Be aware that it may default tousing port
443 and CONNECT if it is doing https.

With respect to the ICMP redirect issue, I suspect that this is down
to poor feature interaction in NetBSD and bridging. Somewhere the code
is treating re0 and re1 as separate interfaces (and thus sending an
ICMP redirect) when in fact they should be treated as one.

This patch might help:

--- sys/netinet/ip_input.c.orig 2012-12-04 07:58:30.000000000 +1100
+++ sys/netinet/ip_input.c 2014-07-06 20:59:33.000000000 +1000
@@ -1452,7 +1452,9 @@
*/
dest.s_addr = 0;
if (!srcrt && V_ipsendredirects &&
- ia != NULL && ia->ia_ifp == m->m_pkthdr.rcvif) {
+ ia != NULL && ia->ia_ifp == m->m_pkthdr.rcvif &&
+ (ia->ia_ifp->if_bridge == NULL ||
+ ia->ia_ifp->if_bridge != m->m_pkthdr.rcvif)) {
struct sockaddr_in *sin;
struct rtentry *rt;


Cheers,
Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thomas Bieg]

Post by Darren Reed
First thing to try is to have the DHCP server include this line in
option wpad-url code 252 = text;
option wpad-url "http://192.168.1.1/wpad.dat";
with the appropriate data on your local web server in wpad.dat.
Yes, that's just a proxy definition but I suspect that what you want
to do is capture and control all WWW sessions from your TV andproxy
is another way to do that. Be aware that it may default tousing port
443 and CONNECT if it is doing https.

DHCP is done by a blackbox device ATM, so I'd have to change that first,
but thanks for the tip!

Post by Darren Reed
With respect to the ICMP redirect issue, I suspect that this is down
to poor feature interaction in NetBSD and bridging. Somewhere the code
is treating re0 and re1 as separate interfaces (and thus sending an
ICMP redirect) when in fact they should be treated as one.
--- sys/netinet/ip_input.c.orig 2012-12-04 07:58:30.000000000 +1100
+++ sys/netinet/ip_input.c 2014-07-06 20:59:33.000000000 +1000
@@ -1452,7 +1452,9 @@
*/
dest.s_addr = 0;
if (!srcrt && V_ipsendredirects &&
- ia != NULL && ia->ia_ifp == m->m_pkthdr.rcvif) {
+ ia != NULL && ia->ia_ifp == m->m_pkthdr.rcvif &&
+ (ia->ia_ifp->if_bridge == NULL ||
+ ia->ia_ifp->if_bridge != m->m_pkthdr.rcvif)) {
struct sockaddr_in *sin;
struct rtentry *rt;

I am a bit puzzled by that patch; the ip_input.c I have here as well as
the one in NetBSD-current seem to be quite different from yours.


Thanks,
Tom


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

This might be a better patch..

Index: ip_input.c
===================================================================
RCS file: /cvsroot/src/sys/netinet/ip_input.c,v
retrieving revision 1.298
diff -u -r1.298 ip_input.c
--- ip_input.c 9 Jan 2012 14:31:22 -0000 1.298
+++ ip_input.c 6 Jul 2014 15:07:32 -0000
@@ -1379,6 +1379,8 @@
* or a route modified by a redirect.
*/
if (rt->rt_ifp == m->m_pkthdr.rcvif &&
+ (rt->rt_ifp->if_bridge == NULL ||
+ rt->rt_ifp->if_bridge != m->m_pkthdr.rcvif) &&
(rt->rt_flags & (RTF_DYNAMIC|RTF_MODIFIED)) == 0 &&
!in_nullhost(satocsin(rt_getkey(rt))->sin_addr) &&
ipsendredirects && !srcrt) {


Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thomas Bieg]

Post by Darren Reed
This might be a better patch..

Indeed, thanks, but unfortunately I don't see any change with it; am still
getting the ICMP redirects (real target IP replaced by 1.2.3.4):

20:33:06.954296 IP (tos 0x0, ttl 64, id 5501, offset 0, flags [none], proto ICMP (1), length 56)
1.2.3.4 > 192.168.111.106: ICMP redirect 1.2.3.4 to host 192.168.111.1, length 36


Best regards,
Tom


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Thomas Bieg

Post by Darren Reed
This might be a better patch..

Indeed, thanks, but unfortunately I don't see any change with it; am still
20:33:06.954296 IP (tos 0x0, ttl 64, id 5501, offset 0, flags [none], proto ICMP (1), length 56)
1.2.3.4 > 192.168.111.106: ICMP redirect 1.2.3.4 to host 192.168.111.1, length 36

The attached patch should sort things out for you.

However in testing I discovered that the interaction between IP and bridging
to be really broken leading me to suspect that there is something more
fundamentally wrong.

For example, without this patch, with a VM running inside ESXi, if I do this:

ifconfig bridge0 create
brconfig bridge0 add wm0

(where wm0 has an IP address assigned to it)

as the first step in bridging wm0 with wm1 then NetBSD starts issuing ICMP
redirects for traffic that it never should. Again this patch shuts down all
of those ICMP redirects but really the problem here is that the IP input code
path is being called for packets that it should not be.

Now it might simply be that the bridge code has been written to only work
when there are no IP addresses assigned to any of the network interfaces
assigned to the bridge. If that's the case then your use (and my use above)
is outside of the scope for supported bridging with NetBSD.

Unfortunately there is nothing in bridge(4) to say that bridging is or isn't
supported when a network interface has an IP address.

kern/48971

Cheers,
Darren

[Thomas Bieg]

Post by Darren Reed
The attached patch should sort things out for you.

Thanks a lot, I will try it later and report back.

Post by Darren Reed
Now it might simply be that the bridge code has been written to only work
when there are no IP addresses assigned to any of the network interfaces
assigned to the bridge. If that's the case then your use (and my use above)
is outside of the scope for supported bridging with NetBSD.
Unfortunately there is nothing in bridge(4) to say that bridging is or isn't
supported when a network interface has an IP address.

Well, at least in
http://www.netbsd.org/docs/guide/en/chap-net-practice.html#chap-net-practice-bridge
the NetBSD Guide says:

"Remember that if you want to give the bridge machine an IP address you can only
allocate an IP address to one of the interfaces which are part of the bridge."


Best regards,
Tom


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thomas Bieg]

Post by Darren Reed
The attached patch should sort things out for you.

Sorry to report it doesn't, still getting those pesky redirects.

Post by Darren Reed
if (rt->rt_ifp == m->m_pkthdr.rcvif &&
+ !bridge_belongsto(rt->rt_ifp->if_bridge, m->m_pkthdr.rcvif) &&

So if the first line checks if the incoming interface is the same as the
outgoing, doesn't the second one just check if that interface is part of
its very own bridge?

But then I probably shouldn't get any redirects at all, so I might be
mistaken.


Best regards,
Tom


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thomas Bieg]

Post by Thomas Bieg

Post by Darren Reed
The attached patch should sort things out for you.

Sorry to report it doesn't, still getting those pesky redirects.

I'd like to bring up this issue one more time.

In the meantime, to be absolutely sure, I commented out the whole block
around "type = ICMP_REDIRECT;" in ip_input.c, did a clean build of the
whole kernel, checked uname that I am actually running that kernel and yet
I am still getting the very same ICMP redirects as before.

This leaves me with no clue at all what I might be doing wrong - or where
else those packets could be created.

Do you have any suggestions? Or do I have to accept that this might just
not be possible with NetBSD? (Which would feel very odd as all the
filtering I tried on the bridge worked perfectly fine, just not that
seemingly simple redirect.)


Thanks and best regards,
Tom


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Darren Reed
With respect to the ICMP redirect issue, I suspect that this is down
to poor feature interaction in NetBSD and bridging. Somewhere the code
is treating re0 and re1 as separate interfaces (and thus sending an
ICMP redirect) when in fact they should be treated as one.

I agree that IP should treat the two ethernets as one interface: re0 and
re1 ought to belong to the same ethernet forwarding domain, and that
forwarding domain should have an IPv4 interface stacked on it.

Post by Darren Reed
--- sys/netinet/ip_input.c.orig 2012-12-04 07:58:30.000000000 +1100
+++ sys/netinet/ip_input.c 2014-07-06 20:59:33.000000000 +1000
@@ -1452,7 +1452,9 @@
*/
dest.s_addr = 0;
if (!srcrt && V_ipsendredirects &&
- ia != NULL && ia->ia_ifp == m->m_pkthdr.rcvif) {
+ ia != NULL && ia->ia_ifp == m->m_pkthdr.rcvif &&
+ (ia->ia_ifp->if_bridge == NULL ||
+ ia->ia_ifp->if_bridge != m->m_pkthdr.rcvif)) {
struct sockaddr_in *sin;
struct rtentry *rt;

The IP stack is already cut through with special cases and layering
violations, but it still is possible to make it worse. :-) I'd hate to
see ip_input.c gain any bridge knowledge.

Dave

[Thomas Bieg]

Post by David Young

Post by Darren Reed
With respect to the ICMP redirect issue, I suspect that this is down
to poor feature interaction in NetBSD and bridging. Somewhere the code
is treating re0 and re1 as separate interfaces (and thus sending an
ICMP redirect) when in fact they should be treated as one.

I agree that IP should treat the two ethernets as one interface: re0 and
re1 ought to belong to the same ethernet forwarding domain, and that
forwarding domain should have an IPv4 interface stacked on it.

I'm a bit out of my depth here, but isn't that how FreeBSD handles this?
From what I gathered, one would assign the IP address to the bridge
interface there (e.g. bridge0) and not one of the member interfaces.

(Which OTOH seems to be problematic if I need to add a bridge to a running
system.)


Best regards,
Tom


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Thomas Bieg

Post by David Young

Post by Darren Reed
With respect to the ICMP redirect issue, I suspect that this is down
to poor feature interaction in NetBSD and bridging. Somewhere the code
is treating re0 and re1 as separate interfaces (and thus sending an
ICMP redirect) when in fact they should be treated as one.

I agree that IP should treat the two ethernets as one interface: re0 and
re1 ought to belong to the same ethernet forwarding domain, and that
forwarding domain should have an IPv4 interface stacked on it.

I'm a bit out of my depth here, but isn't that how FreeBSD handles this?
From what I gathered, one would assign the IP address to the bridge
interface there (e.g. bridge0) and not one of the member interfaces.
(Which OTOH seems to be problematic if I need to add a bridge to a running
system.)

Yep, it is problematic.

I'd rather see a strict separation of L2/L3 responsibilities and a clear
layering, such as what I outline at
<https://mail-index.netbsd.org/tech-net/2011/05/11/msg002614.html>,
than a proliferation of special cases and error conditions.

Dave