What should be the abstraction for ethernet switches?

[Matt Thomas]

More and more embedded boards come with ethernet switch switches that attach directly to the MII interface of an ethernet controller.

So where do we plug these into autoconf?

How do they interact with the vlan and bridge interfaces?

Do we reveal the other switch ports as interfaces?

ifconfig arswitch0.1 media ....



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Young]

Post by Matt Thomas
More and more embedded boards come with ethernet switch switches that attach directly to the MII interface of an ethernet controller.
So where do we plug these into autoconf?
How do they interact with the vlan and bridge interfaces?
Do we reveal the other switch ports as interfaces?
ifconfig arswitch0.1 media ....

I have been thinking about this. I don't think that virtual ethernet
devices such as agr(4), bridge(4), vlan(4) are tenable for the future.
We need multiple ethernet forwarding domains, and each driver for an
ethernet adapter needs to split into top and bottom halves: logical
ethernet interface on top, hardware driver below, with logical ethernets
assigned to ports and bridge groups using device locators.

The following is drawn from my imagination, especially some of the
drvctl uses and including a little bit of "future autoconf(9) history,"
so you will be disappointed if you grovel in your dmesg for or type into
a command line any of the following.

Hardware parts will have an interface attribute "ether" where logical
ethernet drivers may attach with ether_attach_args adapted from the
content of struct ethercom and some of struct ifnet. One or more
interfaces may attach to each ether attribute. In the case of a
hypothetical 6-port switch chip, at least as many logical ethernets
attach as there are ports:

sw0 at pci0 dev 13 function 0: 6-port ethernet switch
swport0 at sw0 port 0 vlan 0 domain 0
swport0: Ethernet address 00:aa:bb:cc:dd:e0
swport1 at sw0 port 1 vlan 0 domain 1
swport1: Ethernet address 00:aa:bb:cc:dd:e1
swport2 at sw0 port 2 vlan 0 domain 2
swport2: Ethernet address 00:aa:bb:cc:dd:e2
swport3 at sw0 port 3 vlan 0 domain 3
swport3: Ethernet address 00:aa:bb:cc:dd:e3
swport4 at sw0 port 4 vlan 0 domain 4
swport4: Ethernet address 00:aa:bb:cc:dd:e4
swport5 at sw0 port 5 vlan 0 domain 5
swport5: Ethernet address 00:aa:bb:cc:dd:e5

Note a couple of things about the default port locators: each port is
assigned vlan 0 (VLAN ID 0---i.e., no VLAN tagging) and an independent
domain (i.e., there is no bridging from port to port).

Ordinarily, a single-port ethernet adapter will attach one logical
ethernet. Example right below. It's ugly, but for backwards
compatibility I add "hw" to the hardware instances so that I can assign
the logical ethernets their legacy name:

fxphw0 at pci1 dev 8 function 0: Intel PRO/100 VM Network Controller
with 82562ET/EZ PHY, rev 2
fxphw0: interrupting at ioapic0 pin 20
wmhw0: 1-port ethernet
inphy0 at fxphw0 phy 1: i82562ET 10/100 media interface, rev. 0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0 at fxphw0 port 0 vlan 0 domain 6
fxp0: Ethernet address 00:0a:0b:0c:0d:0e
.
.
.
wmhw0 at pci0 dev 12 function 0: Intel i82540EM 1000BASE-T Ethernet,
rev. 2
wmhw0: interrupting at ioapic0 pin 18
wmhw0: 32-bit 33MHz PCI bus
wmhw0: 256 word (8 address bits) MicroWire EEPROM
makphy0 at wmhw0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FD
X, auto
wmhw0: 1-port ethernet
wm0 at wmhw0 port 0 vlan 0 domain 7
wm0: Ethernet address 00:01:02:03:04:05

Let's bridge fxp0 and wm0 to each other. To do that, we put them
in the same ethernet domain. We will move wm0 to fxp0's domain,
domain 6:

# drvctl -d wm0
wm0: detached

(Note: I am anticipating some changes to autoconf(9). wm0 was detached,
but not forgotten! We do not get another instance wm0 in the subsequent
discussion, because we haven't used the same locators as the original
had attached with.)

Now, let's attach wmhw0 port 0 to domain 6:

# drvctl -r -a ether wmhw0 domain 6
wm1 at wmhw0 port 0 vlan 0 domain 6

Now the two NICs are bridged. Say that I want for bridged traffic to be
tagged w/ VLAN ID 3 on the fxp, then:

# drvctl -d fxp0
fxp0: detached

# drvctl -r -a ether fxphw0 vlan 3 domain 6
fxp1 at fxphw0 port 0 vlan 3 domain 6

Finally, I want to bridge fxp1, wm1, and sw0 ports 0..2 together:

# drvctl -d swport0
swport0: detached
# drvctl -d swport1
swport1: detached
# drvctl -d swport2
swport2: detached
# drvctl -r -a ether sw0 port 0 domain 6
swport6 at sw0 port 0 vlan 0 domain 6
swport6: Ethernet address 00:aa:bb:cc:dd:e0
# drvctl -r -a ether sw0 port 1 domain 6
swport7 at sw0 port 1 vlan 0 domain 6
swport7: Ethernet address 00:aa:bb:cc:dd:e1
# drvctl -r -a ether sw0 port 2 domain 6
swport8 at sw0 port 2 vlan 0 domain 6
swport8: Ethernet address 00:aa:bb:cc:dd:e2

Note that sw0 has the opportunity as it detaches/attaches each of
swport0..2 and swport6..8 to update the hardware switch matrix.

Finally, I also want for traffic on sw0 port 3 to be bridged to sw0 port
0 and tagged with VLAN ID 7:

# drvctl -r -a ether sw0 port 0 vlan 7 domain 3
swport9 at sw0 port 0 vlan 7 domain 3
swport9: Ethernet address 00:aa:bb:cc:dd:e0

Note that we now have two logical ethernet instances at sw0 port 0:

swport6 at sw0 port 0 vlan 0 domain 6
swport6: Ethernet address 00:aa:bb:cc:dd:e0
swport9 at sw0 port 0 vlan 7 domain 3
swport9: Ethernet address 00:aa:bb:cc:dd:e0

Now say that we want to bridge two ethernets with a GRE tunnel. Imagine
that gre(4) instances have an interface attribute 'ether', too, and we
have a GRE-in-IP instance set up already. We bridge domain 3 to the
remote site like so:

# drvctl -r -a ether gre0 domain 3
greport0 at gre0 port 0 vlan 0 domain 3

It is too bad that locators are currently so primitive, because I would
like to be able to use MAC addresses as locators so that I can stick
a configuration to a particular sw(4) adapter and/or avoid applying a
configuration to the wrong switch if they renumber:

I added a new sw(4) to the chassis, is sw0 the old one?

# drvctl -r -a ether sw0 macaddr 00:aa:bb:cc:dd:e0 vlan 7 domain 3
drvctl: No such file or directory

No. My old sw(4) has a new unit number:

# drvctl -r -a ether sw1 macaddr 00:aa:bb:cc:dd:e0 vlan 7 domain 3
swport9 at sw1 port 0 vlan 7 domain 3
swport9: Ethernet address 00:aa:bb:cc:dd:e0

I think that agr(4) can fit neatly into this framework, too, but I need
to think more about it.

A nice thing (I think) about wiring together a router/bridge using
autoconf(9) is that you can conceivably write much of the network
configuration for a network appliance in the kernel configuration file.
I think that we can imagine reading a lot of a router's configuration
back out of the kernel in the form of, say, a property list that tells
us the current autoconf(9) state such as parent/child relationships,
locators, etc.

BTW, I think that it may help to take a similar approach all of the way
up the stack by, for example, attaching IPv4 & IPv6 instances to logical
ethernets:

ipvf0 at swport9 domain 0: logical IPv4 interface
ipvs0 at swport9 domain 0: logical IPv6 interface
ipvf0 at swport3 domain 1: logical IPv4 interface

Dave

[Daniel Carosone]

Please look at solaris' dladm infrastructure.

--
Dan.

[Matt Thomas]

Post by Daniel Carosone
Please look at solaris' dladm infrastructure.

While interesting, it doesn't seem to address the issue. It seems
to manage the vlan and agr functionality but not bridging.

A system i have has a gige interface connecting to a VSC7385 5 port switch thus providing 4 external ports. Each of these ports has its own link media attributes and vlan settings and the switch itself can provide multiple bridging domains.

For now, I'm completely ignoring that the switch exists and things just work for a simple LAN. But if I want VLAN tagging, bridging, or aggregation the switch needs to get involved.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Daniel Carosone]

Post by Matt Thomas

Post by Daniel Carosone
Please look at solaris' dladm infrastructure.

While interesting, it doesn't seem to address the issue. It seems
to manage the vlan and agr functionality but not bridging.

There's an equivalent of our bridge(4) as well as a TRILL-capable mode
for it in there. While it's true that this currently is a software
bridge between normal nics, the way the stack is arranged there's no
reason why some other device couldn't provide several of the functions
in hardware - much as the stack allows different hardware capabilities
WRT vlan tagging.

Post by Matt Thomas
A system i have has a gige interface connecting to a VSC7385 5 port
switch thus providing 4 external ports. Each of these ports has its
own link media attributes and vlan settings and the switch itself
can provide multiple bridging domains.

That's a pretty common arrangement for commodity routery-type boxes,
among others. There are also PCI cards that have a similar arrangement.

Post by Matt Thomas
For now, I'm completely ignoring that the switch exists and things
just work for a simple LAN. But if I want VLAN tagging, bridging,
or aggregation the switch needs to get involved.

In such a framework, a driver or config parser could arrange for the
various components (ports and a bridge and maybe an aggr) to be
automatically constructed in the right topology. Depending on the
flexibility of the switch, admin commands to adjust this (such as
creating a second bridging domain) will get mapped to switch config
calls rather than changing the config of the soft switch. Each
function would be provided by a device-specific driver that operates
the same as the software one in the same framework.

--
Dan.

[Lloyd Parkes]

Post by Matt Thomas
How do they interact with the vlan and bridge interfaces?

We may be able to get some ideas from the Cisco Nexus and VMware virtual switch systems. They are supposed to have virtualised the control plane, and we may be able to get useful ideas from their control plane model.

I don't know anything about them other than dark mutterings coming from my co-workers about the accuracy of the documentation.

Cheers,
Lloyd


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Zoltan Arnold NAGY]

On Thu, May 12, 2011 at 9:56 AM, Lloyd Parkes

Post by Lloyd Parkes

Post by Matt Thomas
How do they interact with the vlan and bridge interfaces?

We may be able to get some ideas from the Cisco Nexus and VMware virtual switch systems. They are supposed to have virtualised the control plane, and we may be able to get useful ideas from their control plane model.
I don't know anything about them other than dark mutterings coming from my co-workers about the accuracy of the documentation.

While we're at it, we could have a look at OpenSolaris' network
virtualization code, which I kinda liked very
much.

Zoltan

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Stephen Borrill]

Post by Zoltan Arnold NAGY
On Thu, May 12, 2011 at 9:56 AM, Lloyd Parkes

Post by Lloyd Parkes

Post by Matt Thomas
How do they interact with the vlan and bridge interfaces?

We may be able to get some ideas from the Cisco Nexus and VMware virtual switch systems. They are supposed to have virtualised the control plane, and we may be able to get useful ideas from their control plane model.
I don't know anything about them other than dark mutterings coming from my co-workers about the accuracy of the documentation.

While we're at it, we could have a look at OpenSolaris' network
virtualization code, which I kinda liked very
much.

Or perhaps http://openvswitch.org/

It's even got a useful license.