Mbufs stored in RX queues

[v***@labri.fr]

Hi,

I'm currently rewriting a MAC driver, and was wondering why in many
drivers there is a queue of mbufs allocated during init for RX which
will be in "external storage" mode.
(Of course, if the DMA engine is smart enough to fill mbufs internally,
they have to be in the queue beforehand, but it's not my case.)

The RX interrupt will usually proceed like this:

1. allocate new mbuf; if impossible, drop packet
2. fill "old" mbuf in rxq
3. put new mbuf in rxq
4. pass old mbuf to upper layer (which frees it)

(with bus_dma incantations interspersed)

It looks to me that holding a number of mbufs in the rxq is useless
since anyway we will fail to keep the incoming packet if we can't
allocate a new mbuf when the packet arrives.

Could anyone please help me see when this could be useful? From my
perspective, it looks like passing up a freshly allocated mbuf would
make us spend exactly the same amount of time in the interrupt handler,
waste less memory, and simplify the code.

I could buy that switching between two mbufs would reduce a bit latency
to the upper layer because one operation can be done to prepare the new
mbuf after the old mbuf has been passed up.

You can look for example at nfe_rxeof() in dev/pci/if_nfe.c but I picked
it randomly with a few other drivers and it seemed like a habit.

Regards,
Aymeric

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[vineet garg]

I would understand that initial buffer allocations are there to
prevent buf allocation in data path. So, the packet should be read
into a buffer from pool and returned to pool when upper layer returns
it. In case pool is empty in worst case, then we should allocate a new
buf. but is that a real situation? can't we handle that by optimizing
the size of pool to ensure that chances of running out of buffers in
pool reduces a lot....

      Hi,
I'm currently rewriting a MAC driver, and was wondering why in many
drivers there is a queue of mbufs allocated during init for RX which
will be in "external storage" mode.
(Of course, if the DMA engine is smart enough to fill mbufs internally,
they have to be in the queue beforehand, but it's not my case.)
1. allocate new mbuf; if impossible, drop packet
2. fill "old" mbuf in rxq
3. put new mbuf in rxq
4. pass old mbuf to upper layer (which frees it)
(with bus_dma incantations interspersed)
It looks to me that holding a number of mbufs in the rxq is useless
since anyway we will fail to keep the incoming packet if we can't
allocate a new mbuf when the packet arrives.
Could anyone please help me see when this could be useful? From my
perspective, it looks like passing up a freshly allocated mbuf would
make us spend exactly the same amount of time in the interrupt handler,
waste less memory, and simplify the code.
I could buy that switching between two mbufs would reduce a bit latency
to the upper layer because one operation can be done to prepare the new
mbuf after the old mbuf has been passed up.
You can look for example at nfe_rxeof() in dev/pci/if_nfe.c but I picked
it randomly with a few other drivers and it seemed like a habit.
Regards,
 Aymeric

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Michael van Elst]

Post by v***@labri.fr
It looks to me that holding a number of mbufs in the rxq is useless
since anyway we will fail to keep the incoming packet if we can't
allocate a new mbuf when the packet arrives.

There are systems where multiple packets can arrive even before
the interrupt routine is entered.

Post by v***@labri.fr
Could anyone please help me see when this could be useful? From my
perspective, it looks like passing up a freshly allocated mbuf would
make us spend exactly the same amount of time in the interrupt handler,
waste less memory, and simplify the code.

The mbuf needs to exist before the interrupt is requested. If you want to
pass up a new mbuf you need to copy the data to that new mbuf first. Some
drivers even do this, but only for small packets that fit into a single
header mbuf.

[Michael van Elst]

Post by vineet garg
I would understand that initial buffer allocations are there to
prevent buf allocation in data path. So, the packet should be read
into a buffer from pool and returned to pool when upper layer returns
it. In case pool is empty in worst case, then we should allocate a new
buf. but is that a real situation? can't we handle that by optimizing
the size of pool to ensure that chances of running out of buffers in
pool reduces a lot....

- you first allocate one or more buffers
- then you arm the hardware to use these buffers for incoming packets
- the hardware interrupts you when one or more buffers are filled
- the interrupt routine allocates new buffers as a replacement
- the interrupt routine passes the filled buffers up to the protocol layer
- the interrupt routine arms the hardware to use the new buffers

You need at least one buffer for the hardware in advance. It cannot
receive a packet without such a buffer.

If the allocation of new buffers fails, there is little you can do
but drop the incoming data and reuse the old buffer. For some hardware
you could reduce the number of receive buffers instead until there
is only one buffer left. But if there is really an mbuf shortage,
this will happen so quickly that there is no advantage.

Things are different with network hardware that has its own
private receive buffers. But with modern hardware the 'private
buffer' is mostly just an on-chip FIFO and often smaller than
a single packet. All incoming packets are delivered directly
to the preallocated mbufs.

[Michael van Elst]

Post by Michael van Elst
The mbuf needs to exist before the interrupt is requested. If you want to
pass up a new mbuf you need to copy the data to that new mbuf first. Some
drivers even do this, but only for small packets that fit into a single
header mbuf.

But my case is if the data is stored outside the mbuf (M_EXT). I agree
that the memory which is written by the DMA should be available prior to
the interrupt, and in sufficient quantity to hold enough packets before
the RX interrupt is triggered. My question is specifically about the
mbuf structs in this case. I don't see the point of having a lot of them
lying around.

You are right, that you do not need the mbuf headers for the receive
buffers themselves until you send the data to the protocol stack.

But then you have to write your own buffer and memory management
instead of using the mbuf abstraction that is used everywhere
else. And the gain would be 11% less memory usage for just
the receive buffers (mbuf header is 256 bytes, the mbuf cluster
used for M_EXT is 2048 bytes).


Greetings,

[Aymeric Vincent]

Post by Michael van Elst
The mbuf needs to exist before the interrupt is requested. If you want to
pass up a new mbuf you need to copy the data to that new mbuf first. Some
drivers even do this, but only for small packets that fit into a single
header mbuf.

But my case is if the data is stored outside the mbuf (M_EXT). I agree
that the memory which is written by the DMA should be available prior to
the interrupt, and in sufficient quantity to hold enough packets before
the RX interrupt is triggered. My question is specifically about the
mbuf structs in this case. I don't see the point of having a lot of them
lying around.

Regards,
Aymeric

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de