locking order between softnet_lock and the kernel lock

[Beverly Schwartz]

Here at BBN, we are getting a hang with heavy use of networking code and looking for possible deadlocks. We've made modifications to the kernel. I'm trying to understand the lock order invariants to make sure we have applied it correctly to our code.


I see a number of instances of code that look like this:

mutex_enter(softnet_lock);
KERNEL_LOCK(1, NULL);
<do something>
KERNEL_UNLOCK_ONE(NULL);
<maybe do something else>
mutex_exit(softnet_lock);

Am I correct to infer that the kernel lock should always be taken (if taken at all) AFTER softnet_lock is taken, and released before softnet_lock is released?

In in_pcbdetach (and the equivalent function in ip6), there is the following code:
sofree(so); /* drops the socket's lock */
mutex_enter(softnet_lock); /* reacquire the softnet_lock */

I'm curious if a deadlock could occur between sofree and mutex_enter because the kernel lock may be held when this is called.

What are the rules for locking order with softnet_lock and the kernel lock?

--
Bev Schwartz <***@bbn.com>
BBN Technologies


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Thor Lancelot Simon]

Post by Beverly Schwartz
Here at BBN, we are getting a hang with heavy use of networking code and looking for possible deadlocks. We've made modifications to the kernel. I'm trying to understand the lock order invariants to make sure we have applied it correctly to our code.

Generally speaking, I don't think we require locks to be released
in order - though of course it is good practice.

Someone who knows better will dobutless correct me.

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by Thor Lancelot Simon

Post by Beverly Schwartz
Here at BBN, we are getting a hang with heavy use of networking code
and looking for possible deadlocks. We've made modifications to the
kernel. I'm trying to understand the lock order invariants to make
sure we have applied it correctly to our code.

Generally speaking, I don't think we require locks to be released
in order - though of course it is good practice.

I'd agree with that, but I think Bev is concerned about two locks A and
B where the taking order is A then B and

take A
take B
release A
*
take A

and if someone else grabs A at *, then the other is waiting for B and we
are waiting for B. So out of order releasing seems fine, but once
starting to release I think you have to release all the way.
But I am feeling a bit fuzzy on this.

[Greg Troxel]

Post by Greg Troxel

Post by Thor Lancelot Simon

Post by Beverly Schwartz
Here at BBN, we are getting a hang with heavy use of networking code
and looking for possible deadlocks. We've made modifications to the
kernel. I'm trying to understand the lock order invariants to make
sure we have applied it correctly to our code.

Generally speaking, I don't think we require locks to be released
in order - though of course it is good practice.

I'd agree with that, but I think Bev is concerned about two locks A and
B where the taking order is A then B and
take A
take B
release A
*
take A
and if someone else grabs A at *, then the other is waiting for B and we
are waiting for B. So out of order releasing seems fine, but once
starting to release I think you have to release all the way.
But I am feeling a bit fuzzy on this.

so self-replying in light of yamt@'s answer, I remember now that the
kernel lock is special in that processes that try to take a mutex while
holding the kernel lock will give up the kernel lock while sleeping and
then reacquire it on waking.

[Mouse]

Generally speaking, I don't think we require locks to be released in
order - though of course it is good practice.

If you're going to release them all, then, unless the locks themselves
interact somehow (as someone said was the case with kernel-lock, I
think), then you can release in whatever order you please - _provided_
that you release them all without reacquiring any until you've finished
releasing.

I do, however, agree that proper nesting is usually a good idea. Just
not for correctness reasons.

/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Laight]

Post by Mouse

Generally speaking, I don't think we require locks to be released in
order - though of course it is good practice.

If you're going to release them all, then, unless the locks themselves
interact somehow (as someone said was the case with kernel-lock, I
think), then you can release in whatever order you please - _provided_
that you release them all without reacquiring any until you've finished
releasing.
I do, however, agree that proper nesting is usually a good idea. Just
not for correctness reasons.

There are some very good reasons to allow locks be released in any order.
Consider some list/hash/tree of items, lookup sequence:
1) take global lock
2) find item
3) lock item
4) release global lock

Otherwise you have to perform a dance with a reference count.
(Or keep everything locked for much longer.)

Clearly you can't reacquire the global lock (eg to remove an item)
but that isn't necessarily a problem as items can just be marked
'deleted' and pruned from the global structure at a later date
(eg on insert).

There are issues with correct restoring of SPL levels. I think netbsd
stores the saved spl in the kernel thread (not the mutex) and restores
it when the last mutex is released - this isn't completely optimal!

David

[YAMAMOTO Takashi]

hi,

Post by Beverly Schwartz
Here at BBN, we are getting a hang with heavy use of networking code and looking for possible deadlocks. We've made modifications to the kernel. I'm trying to understand the lock order invariants to make sure we have applied it correctly to our code.
mutex_enter(softnet_lock);
KERNEL_LOCK(1, NULL);
<do something>
KERNEL_UNLOCK_ONE(NULL);
<maybe do something else>
mutex_exit(softnet_lock);
Am I correct to infer that the kernel lock should always be taken (if taken at all) AFTER softnet_lock is taken, and released before softnet_lock is released?
sofree(so); /* drops the socket's lock */
mutex_enter(softnet_lock); /* reacquire the softnet_lock */
I'm curious if a deadlock could occur between sofree and mutex_enter because the kernel lock may be held when this is called.
What are the rules for locking order with softnet_lock and the kernel lock?

an adaptive mutex and kernel lock can be acquired in any order
without deadlock.

YAMAMOTO Takashi

Post by Beverly Schwartz
--
BBN Technologies

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de