Thinking about "branes" for netbsd...

[Darren Reed]

After spending some time thinking about what would be required
to implement branes as part of the SMP networking project,
I've put together the notes below. There are some aspects of
this where I'm not sure yet what the bike shed should look
like (e.g. managing branes) so if you would like to argue
about its colour, please construct it first.

At this point I don't know if it will turn into a formal project
proposal but I thought it would be wise to seek feedback from
others anyway to see if there are aspects of this problem that
I've missed or over designed.

Darren

Assumptions
===========
In the one line brief for branes, there's no stated requirements
about attributes such as their longevity. For instance, should a
brane exist for the duration of a process or longer? This proposal
assumes the former - that a brane is to be created without there
necessarily being any process that is assigned to it.

Phase 1 (Implement default brane)
=================================
* Analyse NetBSD networking stack to determine what data
structures can or cannot be removed from global context
- Include in this analysis what the required scope is for
those that are currently global

* Design and implement structures that describe the external
management interfaces for a network family/protocol.
e.g. create, start, shutdown, destroy
- create may fail
- start never fails
- shutdown/destroy must never fail although cleanup and
action may be delayed if resources are busy
- required for each of inet, inet6, tcp, udp, icmp, etc,
separately.
- support an external source registering its own callbacks
for execution..
+ after create, after start, before shutdown, before
destroy, of the builtins

* Convert the use of global variables in NetBSD's networking
to elements of a local structure that is allocated per
protocol family and network protocol
- use an array of network families for initialisation at
boot/shutdown
- struct protosw is no longer global
- network protocols add themselves to the protosw as part
of initialisation (network families have an array of network
protocols that they initialise)


Phase 2 (Support multiple branes)
=================================
* Design interface to support managing multiple branes
- network interfaces can only belong to one brane
- will require the use of virtual interfaces to share
one physical interface between multiple branes
- a network interface name must be unique across all branes

* pfil updates
- each brane will have its own pfil_head
- it should be possible for a subscriber to either receive
callbacks for all branes or just one

* Design and implement kernel interfaces to allow a process
to determine which brane it is in
* Design and implement system call interface to support this
management from userspace
- allow the brane that a process sees to be changed from
the primary brane (that created when the kernel boots)
to a secondary brane if the correct privilege is held
- do not allow processes to change their brane without
privilege or if it is not the primary brane

* New tool to manage branes is required that does the following:
- create brane
- delete brane
- list branes
- assign a network interface to a brane
- remove a network interface from a brane
- this should potentially be a generic tool, such as nicctl,
or branectl or ... that is dedicated to managing network
interfaces and not IP, etc.
- ifconfig is not an appropriate tool for managing branes
or which brane an interface belongs to because it cannot
see all branes. This means that "ifconfig -a" will not
report all interfaces when branes are in use.

* Implement chroot(3) on top of a new system call that allows
for either filesystem or networking "root" (or both) to be
changed
- modify chroot(8) to use new system call and allow for
setting the brane in addition to or instead of the new
root.
- Comment: whilst it may be desirable to only change the
brane, the most common use will be in addition to the
filesystem root and rather than worry about permissions
and ordering of these operations it would seem better to
support changing both of them together.
- build the system call using propvals to convey data rather
than a special structure.
- of course there are compatibility concerns here that will
be taken care of in due course such as ensuring that old
binaries with syscalls continue to work.

* For a brane to have its own loopback interface, it must be
named uniquely, thus no brane could ever have lo0 ecxcept
the default. To allow each brane to have its own lo0 requires
further work - interface name aliasing.

Interface naming
================
For each brane to have a lo0 accessible within it requires that
each brane have its own network interface name space. This is
almost a separate project in its own right.

The first approach is to describe this name space as a logical
name space that sits above the network device name space. To
implement this requires that each brane would have its own
mapping table of network device names. This would be used by
the system to translate network interface names in ioctls that
work with "struct ifnet". This creates other problems as other
code inside the kernel would need to be aware of this mapping -
such as code that calls ifunit() - or the mapping is enforced
elsewhere as well.

A second alternative to this is to implement a logical network
interface abstraction that sits on top of the device name space.
That is, design and implement a "struct lifnet" (logical ifnet)
for NetBSD. This is a rather large exercise and its presence will
be felt in many applications that issue ioctls for "struct ifnet"
that ultimately are for unkown interfaces.

A third alternative would be for the "struct ifnet" to become the
logical network interface structure that is exposed through IP
and other interfaces. Under "struct ifnet" would be a new data
structure ("struct devnet" for example) that was used by all
network devices in lieu of "struct ifnet". There would be a strict
one to one relationship between "struct ifnet" and "struct devnet".
Initially, all of the data that is currently within "struct ifnet"
would stay there and it would serve only as a tool to support a
namespace that is globally unique (that of network devices) with
one or more "struct ifnet" sets inside branes on top of that.
In the fullness of time it might be appropriate to move some data
elements out of "struct ifnet" and into "struct devnet".

Out of scope
============
Beyond providing support to use the pfil interfaces to support
either global or local brane filtering, modifying any of the
firewalls to work in any capacity beyond filtering for the
default brane is out of scope for this project. The reason for
that is the decision on which behaviour model to implement and
support is something that should be made after considering
which security model to enforce. That type of decision making
is out of scope for this project as it is principally concerned
with supporting the creation and management of branes. Similarly,
discussing which colour to paint that particular bike shed should
wait until branes exist. In the case of per-brane filtering,
there are issues to consider such as how and when a rule set
might become associated with said brane.



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

After spending some time thinking about what would be required to
implement branes as part of the SMP networking project, [...]

What's a brane in this context? The only meaning I'm familiar with for
the term is from particle physics and makes no sense here. I did a
little searching, and, while my Web-fu is admittedly weak, I didn't
find anything the least bit helpful.

I think I've only seen the term used here. It seems to refer to
a multiple routing table, or routing instance, mechanism, which I
understand, but appears to be laden with additional, implied policy
semantics concerning the use of the mechanism, which I don't quite
understand. The term "VRF" is used by router vendors for essentially
the identical mechanism (i.e. more than one routing table), but that
term is loaded with what seem to be an entirely different set of policy
assumptions concerning what the mechanism will be used for, assumptions
also different from, say, NAT, which is another router thing which is
usefully (but not necessarily) structured around a multiple routing
instances.

Assuming this is correct, I'll admit that I much prefer thinking about
the bare underlying mechanism and its associated configuration, trying to
make the latter rich enough that a user can configure his own way to
whatever policy he would like to use the mechanism to implement without
strong a priori assumptions about what that policy will be. I stumble over
the "processes get chroot'd into branes" thing that always seems to arise
in conversations about branes since, for most of the applications of
multiple routing instances that I understand, this is not very useful.

Dennis Ferguson

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

What's a brane in this context?

I think I've only seen the term used here. It seems to refer to a
multiple routing table, or routing instance, mechanism, [...]

Ah. Thanks for explaining.

I stumble over the "processes get chroot'd into branes" thing that
always seems to arise in conversations about branes since, for most
of the applications of multiple routing instances that I understand,
this is not very useful.

I'm not even sure what it could mean, since chroot is about pathname
interpretation and IP packet routing has nothing to do with pathnames
as far as I can see. Perhaps something is being extended
metaphorically, but it's unclear to me what or how that could be.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Gert Doering]

Hi,

Post by Mouse

I stumble over the "processes get chroot'd into branes" thing that
always seems to arise in conversations about branes since, for most
of the applications of multiple routing instances that I understand,
this is not very useful.

I'm not even sure what it could mean, since chroot is about pathname
interpretation and IP packet routing has nothing to do with pathnames
as far as I can see. Perhaps something is being extended
metaphorically, but it's unclear to me what or how that could be.

"Give processes a different view to the network than 'the rest of the
machine' has" - not that different to "give processes a different view to
the file system", no?

In a way similar to jails on FreeBSD, where you can restrict which IP
addresses can be used by the jailed processes.

gert

[Mouse]

Post by Gert Doering

[...] "processes get chroot'd into branes" [...]

I'm not even sure what it could mean, [...]. Perhaps something is
being extended metaphorically, but it's unclear to me what or how
that could be.

"Give processes a different view to the network than 'the rest of the
machine' has" - not that different to "give processes a different
view to the file system", no?

Yeah, but that involves (or, at least, I would expect that to involve)
more than just the routing table. Or do branes cover more than just
routing?

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Gert Doering]

Hi,

Post by Mouse

Post by Gert Doering

[...] "processes get chroot'd into branes" [...]

I'm not even sure what it could mean, [...]. Perhaps something is
being extended metaphorically, but it's unclear to me what or how
that could be.

"Give processes a different view to the network than 'the rest of the
machine' has" - not that different to "give processes a different
view to the file system", no?

Yeah, but that involves (or, at least, I would expect that to involve)
more than just the routing table. Or do branes cover more than just
routing?

I'm not sure how that particular bikeshed is planned, but I'd consider
"IP addresses on Interfaces + Routing + Firewall rules" to be all I
need to define a network environment. If that's a "brane", I could
very well see a process being "chroot()"ed into that.

Sort of like a VRF on a Cisco router, with some of the services on
the box (like "telnet") only operating within a certain routing
instance, but not in others. Or particularily, telnet://192.168.0.1
only working in a particular VRF, and not in another that happens
to use 192.168.0.1 as well [not that this works overly well on Cisco
today, but some bits already do, and the idea is sane].

gert

[David Young]

Post by Mouse

Post by Gert Doering

[...] "processes get chroot'd into branes" [...]

I'm not even sure what it could mean, [...]. Perhaps something is
being extended metaphorically, but it's unclear to me what or how
that could be.

"Give processes a different view to the network than 'the rest of the
machine' has" - not that different to "give processes a different
view to the file system", no?

Yeah, but that involves (or, at least, I would expect that to involve)
more than just the routing table. Or do branes cover more than just
routing?

Matt Thomas describes branes here,
<http://www.netbsd.org/~matt/smpnet.html#branes>.

I should say, before I launch into my own explanation, that I have a
particular understanding of "branes" that may be different from Matt's
and others'.

The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.
Packets cannot cross from one forwarding domain to another except by
going through an interface. We can imagine a virtual interface that has
two "ends," each end in a different forwarding domain, for shuttling
packets from domain to domain. More commonly we will have a hardware
interface that attaches a NetBSD router's forwarding domain to the
forwarding domain of a router/switch that's connected with an ethernet
cable.

ISTM that it will be useful sometimes for a tunnel interface to straddle
two domains, sending/receiving encapsulated packets on one domain and
sending/receiving decap'd packets on the other.

A forwarding domain at layer 2 is commonly called a VLAN. We ought to
replace bridge(4) with ethernet forwarding domains.

ISTM that a process may have (or may not) several privileges that
pertain to forwarding domains. It may have a create/destroy-domain
privilege. It may have privileges on more than one domain for the
purpose of examining/setting both routes and interface configuration.
It will have privileges to socket()/bind()/connect()/send()/recv() on
zero or more domains, and one domain (ordinarily the 0th domain) will be
its default domain.

Dave

[Darren Reed]

Post by Gert Doering
Hi,

Post by Mouse

Post by Gert Doering

[...] "processes get chroot'd into branes" [...]

I'm not even sure what it could mean, [...]. Perhaps something is
being extended metaphorically, but it's unclear to me what or how
that could be.

"Give processes a different view to the network than 'the rest of the
machine' has" - not that different to "give processes a different
view to the file system", no?

Yeah, but that involves (or, at least, I would expect that to involve)
more than just the routing table. Or do branes cover more than just
routing?

I'm not sure how that particular bikeshed is planned, but I'd consider
"IP addresses on Interfaces + Routing + Firewall rules" to be all I
need to define a network environment. If that's a "brane", I could
very well see a process being "chroot()"ed into that.

That is my interpretation as well.

The proposal at this point in time has the following goals:
- make the kernel able to support different networking domains
- make it possible to manage those domains
- make it possible for processes to change domain into a brane

There is additional work required to:
- determine how and when to load the firewall configuration for a brane
- decide how to configure networking inside a brane (is there a
library function using proplists or is there a script to run or...)

Post by Gert Doering
The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.
Packets cannot cross from one forwarding domain to another except by
going through an interface. We can imagine a virtual interface that has
two "ends," each end in a different forwarding domain, for shuttling
packets from domain to domain. More commonly we will have a hardware
interface that attaches a NetBSD router's forwarding domain to the
forwarding domain of a router/switch that's connected with an ethernet
cable.

Except for the notion that a route can belong to one domain,
this is otherwise in agreement with what's proposed. More
than one domain may have a specific route, for example, two
domains may have the same default route.

Post by Gert Doering
ISTM that it will be useful sometimes for a tunnel interface to straddle
two domains, sending/receiving encapsulated packets on one domain and
sending/receiving decap'd packets on the other.

And what if you have three domains?
Do you then need n-1 tunnels in each domain to route between them?
That doesn't scale.

Post by Gert Doering
A forwarding domain at layer 2 is commonly called a VLAN. We ought to
replace bridge(4) with ethernet forwarding domains.

That needs careful thought but is not what I'd consider an important
problem to solve for the current proposal.

Further, if bridge(4) was to support TRILL then it may not make sense
to use VLANs.

... the topic of privileges is something that I'd like to avoid for
now, except to say that changing brane is like changing root - you
can only do it once.

Darren

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

Post by Darren Reed
... the topic of privileges is something that I'd like to avoid for
now, except to say that changing brane is like changing root - you
can only do it once.

Huh? How is changing root something you can do only once? Has chroot
been broken when you're already chrooted? That would strike me as a
bit of a regression....

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Darren Reed

Post by David Young
The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.
Packets cannot cross from one forwarding domain to another except by
going through an interface. We can imagine a virtual interface that has
two "ends," each end in a different forwarding domain, for shuttling
packets from domain to domain. More commonly we will have a hardware
interface that attaches a NetBSD router's forwarding domain to the
forwarding domain of a router/switch that's connected with an ethernet
cable.

Except for the notion that a route can belong to one domain,
this is otherwise in agreement with what's proposed. More
than one domain may have a specific route, for example, two
domains may have the same default route.

How/why do two domains share a route, even the default?

It might help to see a made-up routing table for a couple of domains.

Post by Darren Reed

Post by David Young
ISTM that it will be useful sometimes for a tunnel interface to straddle
two domains, sending/receiving encapsulated packets on one domain and
sending/receiving decap'd packets on the other.

And what if you have three domains?
Do you then need n-1 tunnels in each domain to route between them?
That doesn't scale.

I don't intend for anyone ordinarily to use tunnels---e.g., gre(4) or
gif(4)---to route between domains. That is, the two-ended interfaces I
mentioned are not the same as tunnels.

I'm not sure what problem you are trying to solve by routing each domain
to every other domain, but I suppose that the two-ended interface
posited above could be N-ended or else you could route each domain to a
central domain:

+------+ +------+
| | | |
| |\ /| |
| | +------+ | |
+------+ | | +------+
| |
+------+ | | +------+
| | +------+ | |
| |/ \| |
| | | |
+------+ +------+

Then again, source-routing may be what you need?

Anyway, I'm not too keen on routing domains to each other in the same
box.

Dave

[Gert Doering]

Hi,

Post by David Young

Post by Darren Reed
Except for the notion that a route can belong to one domain,
this is otherwise in agreement with what's proposed. More
than one domain may have a specific route, for example, two
domains may have the same default route.

How/why do two domains share a route, even the default?
It might help to see a made-up routing table for a couple of domains.

In Cisco IOS, you have <n> independent routing tables, but you can
have routes pointing out of a routing instance by specifying a target
interface that belongs to another routing instance. Like:

ip route vrf BLUE 0.0.0.0 0.0.0.0 gige3/6 192.0.2.1

so if "gige3/6" belongs to "vrf RED", the default route for packets
in "vrf BLUE" will make them change the vrf.

What you cannot easily do in IOS is "keep the packets on the same box,
but have them change vrf" - but that's something, for example, Juniper
ScreenOS can do with their "virtual routers" - you can point a route
at another vrouter, to make it jump routing tables

set vrouter "trust-vr"
set route 172.18.0.0/16 vrouter "untrust-vr"

(a "vrouter" is, basically, a collection of routing table entries plus
interfaces - same as a VRF in Cisco-Speak)


So, for example, you could have one network 192.168.100.0/24, which is
reachable from all the virtual routing systems

set vrouter "vr-blue"
set route 192.168.100.0/24 vrouter "vr-management"
set vrouter "vr-green"
set route 192.168.100.0/24 vrouter "vr-management"
set vrouter "vr-yellow"
set route 192.168.100.0/24 vrouter "vr-management"

set vrouter "vr-management"
set route 192.168.100.0/24 interface ethernet0/3 gateway 192.168.99.1

(or so)

gert

[Darren Reed]

Post by David Young

Post by Darren Reed

Post by David Young
The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.
Packets cannot cross from one forwarding domain to another except by
going through an interface. We can imagine a virtual interface that has
two "ends," each end in a different forwarding domain, for shuttling
packets from domain to domain. More commonly we will have a hardware
interface that attaches a NetBSD router's forwarding domain to the
forwarding domain of a router/switch that's connected with an ethernet
cable.

Except for the notion that a route can belong to one domain,
this is otherwise in agreement with what's proposed. More
than one domain may have a specific route, for example, two
domains may have the same default route.

How/why do two domains share a route, even the default?

Sharing a route is not the same as having the same route.

So I do not see routes being shared but I do see more than
one brane needing to have the same route. So perhaps "belong"
is the wrong word to use, but the same route can be present
in more than one brane at any one time.

A very basic instance of this would be different interfaces,
with different IP addresses, each in their own brane. Thus
each would necessarily require the network route that uses
the interface.

Post by David Young

Post by Darren Reed

Post by David Young
ISTM that it will be useful sometimes for a tunnel interface to straddle
two domains, sending/receiving encapsulated packets on one domain and
sending/receiving decap'd packets on the other.

And what if you have three domains?
Do you then need n-1 tunnels in each domain to route between them?
That doesn't scale.

I don't intend for anyone ordinarily to use tunnels---e.g., gre(4) or
gif(4)---to route between domains. That is, the two-ended interfaces I
mentioned are not the same as tunnels.
I'm not sure what problem you are trying to solve by routing each domain
to every other domain, but I suppose that the two-ended interface
posited above could be N-ended or else you could route each domain to a

...

I'm not trying to solve a problem per se, rather I'm testing
the soundness of the proposal that was put forward, which is
to say that it didn't scale linearly (i.e. it is bad.) Using
a central domain is a better idea.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Gert Doering
Hi,

Post by David Young

Post by Darren Reed
Except for the notion that a route can belong to one domain,
this is otherwise in agreement with what's proposed. More
than one domain may have a specific route, for example, two
domains may have the same default route.

How/why do two domains share a route, even the default?
It might help to see a made-up routing table for a couple of domains.

In Cisco IOS, you have <n> independent routing tables, but you can
have routes pointing out of a routing instance by specifying a target
ip route vrf BLUE 0.0.0.0 0.0.0.0 gige3/6 192.0.2.1
so if "gige3/6" belongs to "vrf RED", the default route for packets
in "vrf BLUE" will make them change the vrf.
What you cannot easily do in IOS is "keep the packets on the same box,
but have them change vrf" - but that's something, for example, Juniper
ScreenOS can do with their "virtual routers" - you can point a route
at another vrouter, to make it jump routing tables
set vrouter "trust-vr"
set route 172.18.0.0/16 vrouter "untrust-vr"
(a "vrouter" is, basically, a collection of routing table entries plus
interfaces - same as a VRF in Cisco-Speak)
So, for example, you could have one network 192.168.100.0/24, which is
reachable from all the virtual routing systems
set vrouter "vr-blue"
set route 192.168.100.0/24 vrouter "vr-management"
set vrouter "vr-green"
set route 192.168.100.0/24 vrouter "vr-management"
set vrouter "vr-yellow"
set route 192.168.100.0/24 vrouter "vr-management"
set vrouter "vr-management"
set route 192.168.100.0/24 interface ethernet0/3 gateway 192.168.99.1

Thanks!

The Cisco way is awful. The Juniper way seems much more rational.

I wonder how all of this works under the hood in Cisco & Juniper. A
naive implementation would make one routing-table lookup in vr-blue
(192.168.100.1 -> vrouter "vr-management") and a second lookup in
vr-management (192.168.100.1 -> nexthop), but somehow I think that it
doesn't actually work that way.

Dave

[Dennis Ferguson]

Post by David Young
The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.

Okay, but maybe I can reword what I understand this to say in terms that
(to me) more clearly describe mechanism. It would be possible to configure more
than one route table for a protocol, with routes being added to, changed
deleted from each table independently. This actually requires replication
of several address-bearing structures in addition to the route table (e.g.
tables for PCB lookups), so let me call this group of structures a "routing
instance".

The basic configurables around this are that each interface (or "interface",
for some logical definition of this) in the machine running this network
protocol needs to be told which of these tables it should use to route
incoming packets, with the interface routes generated as a side effect
of address configuration on that interface being stored in the same table.
In addition, each open socket needs to be configured (or somehow come to
understand) which table it should use to route outbound packets and
which "routing instance" its socket binding should be stored in. The
collection of a routing instance, the interfaces configured to forward
packets through and install routes in that routing instance and the
sockets which are using the instance are what I think you are calling
a "domain". Beyond this, what each of these routing instances does that
is "special" is defined by the routes installed in each table, and perhaps
the filters operating around the table.

Post by David Young
Packets cannot cross from one forwarding domain to another except by
going through an interface.

And this is a place where I stumble. If you want to implement a
policy which prevents packets from crossing from one domain to another
except by going through an interface (and if I understand what that
means), why is it not sufficient that you can configure each of these
tables with routes which only route packets from one domain to the other
via an interface? Why is it necessary to go the extra step and prohibit
someone else from installing a route in one table with a next hop which
directly routes out an interface whose inbound packets are forwarded via
another table (i.e. an interface in another domain), if that is what is
required for that someone else to get their work done?

The last question isn't rhetorical, either, since I know there are some
VRF operations which require routes to be installed which work exactly
like that (i.e. the next hop sends packets out an interface which is not
part of the VRF the route is installed in) and I know there are other
VRF operations where the result of one route lookup is to make it
immediately do another route lookup in another table. By not stopping
at "I can configure this policy", but rather going all the way to
"No other policy can possibly be configured" I think you make the
mechanism less generally useful and I'd sort of like to see the
justification for that.

Post by David Young
ISTM that a process may have (or may not) several privileges that
pertain to forwarding domains. It may have a create/destroy-domain
privilege. It may have privileges on more than one domain for the
purpose of examining/setting both routes and interface configuration.
It will have privileges to socket()/bind()/connect()/send()/recv() on
zero or more domains, and one domain (ordinarily the 0th domain) will be
its default domain.

Note that the traditional privilege model for the network stack is that
processes running as the superuser are allowed to configure whatever can
be configured while other processes, with some constraints, are allowed
to look at and use the configuration the superuser has provided. I get
that this might not work well for some arrangements, and there might be
other good ways to slice and dice this, but the traditional privilege
model does still work for a lot of things.

I can see the possible need to constrain certain operations in association
with a non-traditional privilege model, but I don't think it is wise to
do this by placing constraints on what the superuser can do when running
in the superuser-can-do-anything-it-wants traditional model. Starting with
the most unconstrained implementation of multiple routing tables that can
be justified by its implementation costs (i.e. if there are to be constraints
it should be because those things are expensive or otherwise hard to do)
and only paring this back when the privilege policy actually implemented
suggests that certain things are unwise to allow.

Dennis Ferguson
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Darren Reed
Further, if bridge(4) was to support TRILL then it may not make sense
to use VLANs.

How does TRILL help? As I understand it, TRILL is an effort to replace
spanning tree routing with a real link-state routing protocol so you can
use all the links you have configured between switches to carry traffic,
rather than turning off links until there's only one way to reach any
destination a la spanning tree.

Whether spanning tree or IS-IS is used to route a bridged ethernet doesn't
seem to have much effect on what VLANs do, though (I'll also admit I don't
understand how the use of VLANs would eliminate bridge(4) in any case). Is
there something else that TRILL changes?

Dennis Ferguson

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Dennis Ferguson

Post by David Young
The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.

Okay, but maybe I can reword what I understand this to say in terms that
(to me) more clearly describe mechanism. It would be possible to configure more
than one route table for a protocol, with routes being added to, changed
deleted from each table independently. This actually requires replication
of several address-bearing structures in addition to the route table (e.g.
tables for PCB lookups), so let me call this group of structures a "routing
instance".
The basic configurables around this are that each interface (or "interface",
for some logical definition of this) in the machine running this network
protocol needs to be told which of these tables it should use to route
incoming packets, with the interface routes generated as a side effect
of address configuration on that interface being stored in the same table.
In addition, each open socket needs to be configured (or somehow come to
understand) which table it should use to route outbound packets and
which "routing instance" its socket binding should be stored in. The
collection of a routing instance, the interfaces configured to forward
packets through and install routes in that routing instance and the
sockets which are using the instance are what I think you are calling
a "domain".

The collection you describe sounds like a domain to me.

Post by Dennis Ferguson

Post by David Young
Packets cannot cross from one forwarding domain to another except by
going through an interface.

And this is a place where I stumble. If you want to implement a
policy which prevents packets from crossing from one domain to another
except by going through an interface (and if I understand what that
means), why is it not sufficient that you can configure each of these
tables with routes which only route packets from one domain to the other
via an interface? Why is it necessary to go the extra step and prohibit
someone else from installing a route in one table with a next hop which
directly routes out an interface whose inbound packets are forwarded via
another table (i.e. an interface in another domain), if that is what is
required for that someone else to get their work done?
The last question isn't rhetorical, either, since I know there are some
VRF operations which require routes to be installed which work exactly
like that (i.e. the next hop sends packets out an interface which is not
part of the VRF the route is installed in) and I know there are other
VRF operations where the result of one route lookup is to make it
immediately do another route lookup in another table. By not stopping
at "I can configure this policy", but rather going all the way to
"No other policy can possibly be configured" I think you make the
mechanism less generally useful and I'd sort of like to see the
justification for that.

You're right, it isn't desirable to be so restrictive.

Dave

[Mouse]

Post by David Young

By not stopping at "I can configure this policy", but rather going
all the way to "No other policy can possibly be configured" I think
you make the mechanism less generally useful and I'd sort of like to
see the justification for that.

You're right, it isn't desirable to be so restrictive.

I agree. It strikes me as rather restrictive to mandate that which
routing table is used is tied to which process is responsible for the
traffic - especially since sockets can be shared by multiple processes,
meaning that either you have to somehow forbid the same socket from
appearing in the open file tables of processes using different branes,
or you have to tag the data in the socket's send queue with which brane
the writing process was in at the time of writing, or you have to
attach branes to sockets as well as processes, or you have to make the
brane association work only for immediately-sent traffic (eg, UDP but
not TCP), or something of the sort.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Mouse

Post by David Young

By not stopping at "I can configure this policy", but rather going
all the way to "No other policy can possibly be configured" I think
you make the mechanism less generally useful and I'd sort of like to
see the justification for that.

You're right, it isn't desirable to be so restrictive.

I agree. It strikes me as rather restrictive to mandate that which
routing table is used is tied to which process is responsible for the
traffic - especially since sockets can be shared by multiple processes,
meaning that either you have to somehow forbid the same socket from
appearing in the open file tables of processes using different branes,
or you have to tag the data in the socket's send queue with which brane
the writing process was in at the time of writing, or you have to
attach branes to sockets as well as processes, or you have to make the
brane association work only for immediately-sent traffic (eg, UDP but
not TCP), or something of the sort.

It would appear that various people did not fully read what I proposed
or perhaps misunderstood it. What I wrote is that a brane would become
an entire instance of networking and that global structures, etc, would
disappear. If it helps you (or others) to understand it by thinking of
branes in terms of support for virtual routers then I'm happy for you
to do so but it is by no means limited to that.

For the purposes of this and other discussion, I consider the default
brane to be the networking environment that is created at boot.

In that context, if a process belongs to a brane then all of its sockets,
etc, would also belong to the new brane. Given that it is proposed that
a network interface can only ever belong to one brane, there are no issues
with respect to traffic leaking from one to another and the routing table
is fully self contained within the brane.

Given that there are security implications for being able to pass a
socket as a file descriptor from one process to another where the
two processes belong to different branes, that behaviour would need
to be prevented.

What I proposed in terms of a brane is much more than just support for
virtual routing - it takes NetBSD further along the path of being able
to support something like a jail. Whilst what I proposed in terms of a
brane is not enough to support a jail, it does alter the kernel's
networking in such a way as to make moving forward and implementing
something akin to jails for NetBSD easier.

To fully support jails would require further work on IPC, limiting
which processes can signal other processes, etc.

Consider the implementation of a branes for NetBSD being kernel level
virtualisation for networking and a step along the jail path.

Darren

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Darren Reed

Post by Mouse

Post by David Young

By not stopping at "I can configure this policy", but rather going
all the way to "No other policy can possibly be configured" I think
you make the mechanism less generally useful and I'd sort of like to
see the justification for that.

You're right, it isn't desirable to be so restrictive.

I agree. It strikes me as rather restrictive to mandate that which
routing table is used is tied to which process is responsible for the
traffic - especially since sockets can be shared by multiple processes,
meaning that either you have to somehow forbid the same socket from
appearing in the open file tables of processes using different branes,
or you have to tag the data in the socket's send queue with which brane
the writing process was in at the time of writing, or you have to
attach branes to sockets as well as processes, or you have to make the
brane association work only for immediately-sent traffic (eg, UDP but
not TCP), or something of the sort.

It would appear that various people did not fully read what I proposed
or perhaps misunderstood it. What I wrote is that a brane would become
an entire instance of networking and that global structures, etc, would
disappear. If it helps you (or others) to understand it by thinking of
branes in terms of support for virtual routers then I'm happy for you
to do so but it is by no means limited to that.

I think we are all finding out that your ideas and enthusiasm for
branes resonate with others' ideas and enthusiasm about similar but
not-quite-alike overhauls of NetBSD networking. Let's take that
resonance and run with it---not all in different directions, I hope!

There may be less enthusiasm for branes in particular than there is for
multiple routing tables and for a network stack with a more rational
structure.

Dave

[Mouse]

Post by David Young
There may be less enthusiasm for branes in particular than there is
for multiple routing tables and for a network stack with a more
rational structure.

For example, I would like to see multiple routing tables, but if the
design requires that the mechanism which selects which table gets used
is driven off solely which process sent the data, or which process
created the socket...then it's of little value to me. I would prefer
multiple routing tables with less policy attached; I want to select the
routing table based on other things. (The principal one I have in mind
at the moment is, loosely put, ip_src, but I'm fairly confident there
are other alternatives as well; to name three that come to mind
immediately, time-of-day, load average, and anything computable by a
bpf bytecode program.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Ted Lemon]

It might be worth looking at the work that the MIF working group at IETF is doing with respect to provisioning domains when talking about this problem space. I can go into more detail if people are interested, but the essential focus of what they are doing has to do with the problem of a device that has N interfaces, where N is typically greater than 1, and they are connected to different physical networks operated by different operators. It also addresses the case where you have a dual-stack network.

My theory as how this should be made to work sounds a little bit like the brane proposal, and a little bit like what you are saying the brane proposal isn't, Mouse.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Darren Reed
It would appear that various people did not fully read what I proposed
or perhaps misunderstood it. What I wrote is that a brane would become
an entire instance of networking and that global structures, etc, would
disappear. If it helps you (or others) to understand it by thinking of
branes in terms of support for virtual routers then I'm happy for you
to do so but it is by no means limited to that.

The point was not that the mechanism should be thought of in terms of
a VRF implementation, it is only that the mechanism should not have arbitrary
constraints that prevent you from using it for a VRF implementation, or any
other job that the mechanism might be appropriate to use for.

Post by Darren Reed
In that context, if a process belongs to a brane then all of its sockets,
etc, would also belong to the new brane.

But, speaking of a topic that came up a week or two ago, what if the thing
I want to use the multiple tables for is to implement strict host behavior
on a multi-homed host? That is, suppose I am running a server application
on a host with two interfaces, and I just want the reply packets to connections
that came in one or the other of the interfaces to go back out the same
interface (say the interfaces connect to different service providers). Two
routing tables, one per interface, are required for this since the outbound
routing from the host needs to be done through a table that only has routes
pointing out the interface the packet needs to leave on.

For this to work I want my server process to be able to send and receive
packets in either brane (or, at least, using either table) at will. In fact,
an ideal implementation might allow a single listening socket to receive
incoming connections from both tables (or branes, if you will), with only
the connected sockets generated by accept() ending up being tied to one or
the other table, since then the mechanism can be made transparent to the
application, though I guess opening a separate listening socket for each
table wouldn't be a huge burden. None of this seems to work, however, if
the mechanism requires that the process needs to "belong to a brane", and
it is precisely this constraint in your proposal that I object to. It gets
in the way of useful applications for the basic multiple-routing-table
mechanism.

The basic mechanism of multiple routing tables is a forwarding path mechanism.
It effects how and where packets are delivered by the kernel, but needn't
effect anything else. The "process belongs to a brane" thing is an arbitrary,
additional constraint that may be useful for some applications of the
mechanism but which prevents it from being used for others, like a standard
VRF implementation (apparently what you are referring to as "virtual routers"
is not a standard VRF implementation) or the very simple strict host behavior
described above. If you want the "brane" constraint unconditionally applied
in all cases I think you should justify why this constraint is so important
to preclude using the mechanism for other applications (including existing
ones) which are incompatible with the constraint. Pretending such applications
don't exist really doesn't do this.

Dennis Ferguson
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Dennis Ferguson

Post by Darren Reed
In that context, if a process belongs to a brane then all of its sockets,
etc, would also belong to the new brane.

But, speaking of a topic that came up a week or two ago, what if the thing
I want to use the multiple tables for is to implement strict host behavior
on a multi-homed host? That is, suppose I am running a server application
on a host with two interfaces, and I just want the reply packets to connections
that came in one or the other of the interfaces to go back out the same
interface (say the interfaces connect to different service providers). Two
routing tables, one per interface, are required for this since the outbound
routing from the host needs to be done through a table that only has routes
pointing out the interface the packet needs to leave on.

Branes are the wrong solution for that problem.

What is better is to be able to issue an ioctl on the socket
and in that fashion specify which network interface to use
for sending out packets.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Darren Reed

Post by Dennis Ferguson

Post by Darren Reed
In that context, if a process belongs to a brane then all of its sockets,
etc, would also belong to the new brane.

But, speaking of a topic that came up a week or two ago, what if the thing
I want to use the multiple tables for is to implement strict host behavior
on a multi-homed host? That is, suppose I am running a server application
on a host with two interfaces, and I just want the reply packets to connections
that came in one or the other of the interfaces to go back out the same
interface (say the interfaces connect to different service providers). Two
routing tables, one per interface, are required for this since the outbound
routing from the host needs to be done through a table that only has routes
pointing out the interface the packet needs to leave on.

Branes are the wrong solution for that problem.

Probably, but multiple forwarding tables are essential to solve that
problem. This is the problem with "multiple forwarding tables" == "branes".

Post by Darren Reed
What is better is to be able to issue an ioctl on the socket
and in that fashion specify which network interface to use
for sending out packets.

The problem is that, no matter what the ioctl tells the socket
to do, the kernel cannot send a packet out an interface if it
does not have a route to the packet's destination pointing out
that interface. If the destination is being routed with a
default route then there need to be two default routes (if it
is being routed with some other route there need to be two
of those), one for each interface, which means you need two
forwarding tables to store the different routes in. It is a
multiple forwarding table problem even if it isn't one that
a "brane" works for.

Dennis Ferguson


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Dennis Ferguson

Post by Darren Reed

Post by Dennis Ferguson

Post by Darren Reed
In that context, if a process belongs to a brane then all of its sockets,
etc, would also belong to the new brane.

But, speaking of a topic that came up a week or two ago, what if the thing
I want to use the multiple tables for is to implement strict host behavior
on a multi-homed host? That is, suppose I am running a server application
on a host with two interfaces, and I just want the reply packets to connections
that came in one or the other of the interfaces to go back out the same
interface (say the interfaces connect to different service providers). Two
routing tables, one per interface, are required for this since the outbound
routing from the host needs to be done through a table that only has routes
pointing out the interface the packet needs to leave on.

Branes are the wrong solution for that problem.

Probably, but multiple forwarding tables are essential to solve that
problem. This is the problem with "multiple forwarding tables" == "branes".

Post by Darren Reed
What is better is to be able to issue an ioctl on the socket
and in that fashion specify which network interface to use
for sending out packets.

The problem is that, no matter what the ioctl tells the socket
to do, the kernel cannot send a packet out an interface if it
does not have a route to the packet's destination pointing out
that interface. If the destination is being routed with a
default route then there need to be two default routes (if it
is being routed with some other route there need to be two
of those), one for each interface, which means you need two
forwarding tables to store the different routes in. It is a
multiple forwarding table problem even if it isn't one that
a "brane" works for.

For reference, you might like to investigate the following
IP socket options that are available on OpenSolaris:
IP_NEXTHOP
IP_BOUND_IF
The latter of which is similar to Linux's SO_BINDTODEVICE.
I don't know if Linux has an equivalent of IP_NEXTHOP but
as an ioctl, it works in a similar way (for a particular
socket) as does policy routing with ipfilter. I believe
that either one or both of those are a solution to your
problem without implementing virtual routing tables.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Darren Reed

Post by Dennis Ferguson
The problem is that, no matter what the ioctl tells the socket
to do, the kernel cannot send a packet out an interface if it
does not have a route to the packet's destination pointing out
that interface. If the destination is being routed with a
default route then there need to be two default routes (if it
is being routed with some other route there need to be two
of those), one for each interface, which means you need two
forwarding tables to store the different routes in. It is a
multiple forwarding table problem even if it isn't one that
a "brane" works for.

For reference, you might like to investigate the following
IP_NEXTHOP
IP_BOUND_IF
The latter of which is similar to Linux's SO_BINDTODEVICE.
I don't know if Linux has an equivalent of IP_NEXTHOP but
as an ioctl, it works in a similar way (for a particular
socket) as does policy routing with ipfilter. I believe
that either one or both of those are a solution to your
problem without implementing virtual routing tables.

I'm aware of both of those. I think you may be confused about
what IP_BOUND_IF does (hint: last I looked it only effects
where multicasts and broadcasts go). IP_NEXTHOP does help if
you think this is solved by having each application do its
own routing (maybe the application could run DHCP to find out
the next hop for that interface's default route too); if all
applications did this then the kernel could get even simpler
by eliminating all forwarding tables.

Dennis Ferguson


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Dennis Ferguson

Post by Darren Reed

Post by Dennis Ferguson
The problem is that, no matter what the ioctl tells the socket
to do, the kernel cannot send a packet out an interface if it
does not have a route to the packet's destination pointing out
that interface. If the destination is being routed with a
default route then there need to be two default routes (if it
is being routed with some other route there need to be two
of those), one for each interface, which means you need two
forwarding tables to store the different routes in. It is a
multiple forwarding table problem even if it isn't one that
a "brane" works for.

For reference, you might like to investigate the following
IP_NEXTHOP
IP_BOUND_IF
The latter of which is similar to Linux's SO_BINDTODEVICE.
I don't know if Linux has an equivalent of IP_NEXTHOP but
as an ioctl, it works in a similar way (for a particular
socket) as does policy routing with ipfilter. I believe
that either one or both of those are a solution to your
problem without implementing virtual routing tables.

I'm aware of both of those. I think you may be confused about
what IP_BOUND_IF does (hint: last I looked it only effects
where multicasts and broadcasts go). IP_NEXTHOP does help if
you think this is solved by having each application do its
own routing (maybe the application could run DHCP to find out
the next hop for that interface's default route too); if all
applications did this then the kernel could get even simpler
by eliminating all forwarding tables.

You want an application to send a packet back out the same
interface that the packet was received on. That amounts to
the application doing its own routing.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Darren Reed

Post by Dennis Ferguson
I'm aware of both of those. I think you may be confused about
what IP_BOUND_IF does (hint: last I looked it only effects
where multicasts and broadcasts go). IP_NEXTHOP does help if
you think this is solved by having each application do its
own routing (maybe the application could run DHCP to find out
the next hop for that interface's default route too); if all
applications did this then the kernel could get even simpler
by eliminating all forwarding tables.

You want an application to send a packet back out the same
interface that the packet was received on. That amounts to
the application doing its own routing.

See RFC 1122, section 3.3.4.2, the Strong ES model. The routing
operation is well-defined, the application doesn't have to be
aware it is happening, but it needs a routing table per interface
to implement it.

This has reached the point of absurdity, however. I'm done.

Dennis Ferguson


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Dennis Ferguson

Post by Darren Reed

Post by Dennis Ferguson
I'm aware of both of those. I think you may be confused about
what IP_BOUND_IF does (hint: last I looked it only effects
where multicasts and broadcasts go). IP_NEXTHOP does help if
you think this is solved by having each application do its
own routing (maybe the application could run DHCP to find out
the next hop for that interface's default route too); if all
applications did this then the kernel could get even simpler
by eliminating all forwarding tables.

You want an application to send a packet back out the same
interface that the packet was received on. That amounts to
the application doing its own routing.

See RFC 1122, section 3.3.4.2, the Strong ES model. The routing
operation is well-defined, the application doesn't have to be
aware it is happening, but it needs a routing table per interface
to implement it.

No, it does not.

The problem as stated is that the routing decision should
include the source address as part of the key for deciding
where to route a packet.

I'll grant that it is possible to read that as suggesting
that using a routing table per interface is the solution
but it is not the only solution to that problem.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Dennis Ferguson

Post by David Young
The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces. Each route/interface can belong to just one domain.

Okay, but maybe I can reword what I understand this to say in terms that
(to me) more clearly describe mechanism. It would be possible to configure more
than one route table for a protocol, with routes being added to, changed
deleted from each table independently. This actually requires replication
of several address-bearing structures in addition to the route table (e.g.
tables for PCB lookups), so let me call this group of structures a "routing
instance".
The basic configurables around this are that each interface (or "interface",
for some logical definition of this) in the machine running this network
protocol needs to be told which of these tables it should use to route
incoming packets, with the interface routes generated as a side effect
of address configuration on that interface being stored in the same table.
In addition, each open socket needs to be configured (or somehow come to
understand) which table it should use to route outbound packets and
which "routing instance" its socket binding should be stored in. The
collection of a routing instance, the interfaces configured to forward
packets through and install routes in that routing instance and the
sockets which are using the instance are what I think you are calling
a "domain". Beyond this, what each of these routing instances does that
is "special" is defined by the routes installed in each table, and perhaps
the filters operating around the table.

Roughly speaking, more or less yes.

Post by Dennis Ferguson

Post by David Young
Packets cannot cross from one forwarding domain to another except by
going through an interface.

And this is a place where I stumble. If you want to implement a
policy which prevents packets from crossing from one domain to another
except by going through an interface (and if I understand what that
means), why is it not sufficient that you can configure each of these
tables with routes which only route packets from one domain to the other
via an interface? Why is it necessary to go the extra step and prohibit
someone else from installing a route in one table with a next hop which
directly routes out an interface whose inbound packets are forwarded via
another table (i.e. an interface in another domain), if that is what is
required for that someone else to get their work done?

To allow cross references between data structures from one brane to
another or cross talk where packets can move from one brane to another
would be a very dirty implementation. The goal of branes is to support
network virtualisation. One aspect of network virtualisation is the
ability for the kernel to support multiple routing tables as a result.

In terms of implementation, can you imagine the locking hell that would
result if it you tried to directly join both of those domains because
one wanted to use the resources of another?

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mouse]

The goal of branes is to support network virtualisation. One aspect
of network virtualisation is the ability for the kernel to support
multiple routing tables as a result.

Indeed.

But, by choosing a design that has that sort of isolation wired into
it, you end up ensuring that they cannot be used for anything else,
rather than creating a general-purpose mechanism which can, among other
things, be used for virtualization.

As the Jaron File notes (see the entry for `uninteresting'), there is a
strong tendency towards general-purpose tools which solve the original
problem as a special case. I'd much prefer to see this done that way
(though, admittedly, that constitutes armchair quarterbacking).

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Mouse

The goal of branes is to support network virtualisation. One aspect
of network virtualisation is the ability for the kernel to support
multiple routing tables as a result.

Indeed.
But, by choosing a design that has that sort of isolation wired into
it, you end up ensuring that they cannot be used for anything else,
rather than creating a general-purpose mechanism which can, among other
things, be used for virtualization.

I think that the virtualisation approach is superior because it
means that the tools that work with a routing table don't need
to be told that there are different types of routing tables.
In the virtual environment, everything just works without being
aware of which environment it is in.

To my way of thinking, doing something special just for virtual
routing tables means that any tools that work with routing need
to be modified in special ways. If a tool hasn't been modified
then it doesn't work. Thus the solution becomes more frail. As
a quick example of this, what happens to routing socket messages?
Do they get special tags? Do they need to change in a way that
makes them incompatible? And so on. There's hidden complexity
that I believe makes it more trouble than it is worth.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Dennis Ferguson]

Post by Darren Reed

Post by Dennis Ferguson

Post by David Young
Packets cannot cross from one forwarding domain to another except by
going through an interface.

And this is a place where I stumble. If you want to implement a
policy which prevents packets from crossing from one domain to another
except by going through an interface (and if I understand what that
means), why is it not sufficient that you can configure each of these
tables with routes which only route packets from one domain to the other
via an interface? Why is it necessary to go the extra step and prohibit
someone else from installing a route in one table with a next hop which
directly routes out an interface whose inbound packets are forwarded via
another table (i.e. an interface in another domain), if that is what is
required for that someone else to get their work done?

To allow cross references between data structures from one brane to
another or cross talk where packets can move from one brane to another
would be a very dirty implementation. The goal of branes is to support
network virtualisation. One aspect of network virtualisation is the
ability for the kernel to support multiple routing tables as a result.

I learned a long time ago that there is no accounting for taste, and
what looks to me like dirt is often someone else's very excellent and
essential feature. While the goal of branes might be network virtualization
that is only one of many possible applications for multiple forwarding
tables, so it is bothering me a bit that you seem to want to implement
multiple forwarding tables in a way which excludes those other applications
rather than designing a more general underlying implementation of the
multiple forwarding table mechanism, and then figuring out how to specialize
that when network virtualization is the particular application you want to
use it for. A standard VRF implementation needs the above, and also needs
to have a single routing application able to populate multiple forwarding
tables (since the installation of a route in one table often has implications
for others). This might not be to my taste either, yet I see no reason
to implement 98% of the mechanism essential to this application but
deny it the remaining 2% because your application doesn't want to do that.

Post by Darren Reed
In terms of implementation, can you imagine the locking hell that would
result if it you tried to directly join both of those domains because
one wanted to use the resources of another?

Now this part is interesting.

Let me just assert that, while I find talking about "branes" and multiple
forwarding tables in general personally interesting, this is a geeky
feature that the average user of NetBSD is probably almost never going
to use since they'll almost never have one of the problems that this
mechanism solves (and the more you constrain it, the less likely it is
they'll have a problem this is fit for). What those users are far more
likely to be using, however, is multiprocessors, since it is getting
harder all the time to buy CPUs that don't have multiple cores.

This being the case, I think "branes" just can't be the solution for
the NetBSD network stack's SMP problems because the majority, who have
SMP systems but absolutely no problems with networking requiring multiple
forwarding tables, need a network stack which makes good use of an SMP system
even when they only have use for a single forwarding table. The SMP problems
must be solved in a way which does not depend on this, and are a far higher
priority than this. Because of this, I think it is far too early to make
declarations about what operations are or aren't going to be locking problems,
since the big lock has to go regardless of branes.

Dennis Ferguson
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Post by Dennis Ferguson

Post by Darren Reed

Post by Dennis Ferguson

Post by David Young
Packets cannot cross from one forwarding domain to another except by
going through an interface.

And this is a place where I stumble. If you want to implement a
policy which prevents packets from crossing from one domain to another
except by going through an interface (and if I understand what that
means), why is it not sufficient that you can configure each of these
tables with routes which only route packets from one domain to the other
via an interface? Why is it necessary to go the extra step and prohibit
someone else from installing a route in one table with a next hop which
directly routes out an interface whose inbound packets are forwarded via
another table (i.e. an interface in another domain), if that is what is
required for that someone else to get their work done?

To allow cross references between data structures from one brane to
another or cross talk where packets can move from one brane to another
would be a very dirty implementation. The goal of branes is to support
network virtualisation. One aspect of network virtualisation is the
ability for the kernel to support multiple routing tables as a result.

I learned a long time ago that there is no accounting for taste, and
what looks to me like dirt is often someone else's very excellent and
essential feature. While the goal of branes might be network virtualization
that is only one of many possible applications for multiple forwarding
tables, so it is bothering me a bit that you seem to want to implement
multiple forwarding tables in a way which excludes those other applications
rather than designing a more general underlying implementation of the
multiple forwarding table mechanism, and then figuring out how to specialize
that when network virtualization is the particular application you want to
use it for.

I think you misunderstand what I'm proposing. What I'm proposing is
to implement virtual networking stacks for NetBSD - aka branes. It
just so happens that doing so delivers the ability to maintain
independent routing tables. I am not trying to solve the virtual
routing table problem. This goal has been recognised as needing to
be solved for NetBSD. There is nothing saying that you have to use
branes to solve your virtual routing problems but similarly, they
are not defined as being part of the problem that needs solving
for virtual networking stacks in NetBSD and nor should they be.

If you were willing to come up with a virtual routing table
implementation then I'm pretty sure that it would work just
as well with branes as it would without them. The problems that
need to be solved for virtual networking stacks are at a different
level to those of virtual routing tables. The routing table (or
tables) are just one piece of the networking stack that needs to
be made brane aware.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Darren Reed]

Now that folks are paying attention, I would like to
draw people's attention back to this issue. I would
seriosly like feedback from people on their thoughts
on two topics.

Post by Darren Reed
- create brane
- delete brane
- list branes
- assign a network interface to a brane
- remove a network interface from a brane
- this should potentially be a generic tool, such as nicctl,
or branectl or ... that is dedicated to managing network
interfaces and not IP, etc.
- ifconfig is not an appropriate tool for managing branes
or which brane an interface belongs to because it cannot
see all branes. This means that "ifconfig -a" will not
report all interfaces when branes are in use.

Next there is the issue of interface naming.

I will add that it may be that "struct devnet" can
be of help to SMP networking, I don't know...

Post by Darren Reed
...
Interface naming
================
For each brane to have a lo0 accessible within it requires that
each brane have its own network interface name space. This is
almost a separate project in its own right.
The first approach is to describe this name space as a logical
name space that sits above the network device name space. To
implement this requires that each brane would have its own
mapping table of network device names. This would be used by
the system to translate network interface names in ioctls that
work with "struct ifnet". This creates other problems as other
code inside the kernel would need to be aware of this mapping -
such as code that calls ifunit() - or the mapping is enforced
elsewhere as well.
A second alternative to this is to implement a logical network
interface abstraction that sits on top of the device name space.
That is, design and implement a "struct lifnet" (logical ifnet)
for NetBSD. This is a rather large exercise and its presence will
be felt in many applications that issue ioctls for "struct ifnet"
that ultimately are for unkown interfaces.
A third alternative would be for the "struct ifnet" to become the
logical network interface structure that is exposed through IP
and other interfaces. Under "struct ifnet" would be a new data
structure ("struct devnet" for example) that was used by all
network devices in lieu of "struct ifnet". There would be a strict
one to one relationship between "struct ifnet" and "struct devnet".
Initially, all of the data that is currently within "struct ifnet"
would stay there and it would serve only as a tool to support a
namespace that is globally unique (that of network devices) with
one or more "struct ifnet" sets inside branes on top of that.
In the fullness of time it might be appropriate to move some data
elements out of "struct ifnet" and into "struct devnet".

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de