Discussion:
Experiments with npf on -current
(too old to reply)
David Brownlee
2011-11-20 20:37:04 UTC
Permalink
I've had a quick test of npf on a dual attached machine which usually ran pf.

# /etc/rc.d/pf stop
# modunload pf
# modload npf
# modstat|grep 'pf '



First tried nat, but could not get it to take effect:

int_if = "bge1"
int_net = "192.168.2.0/24"
ext_if = "bge0"
nat $ext_if from $int_net to any -> $ext_if


# npfctl stats
Packets passed:
570636 default pass
0 ruleset pass
0 session pass

Packets blocked:
0 default block
0 ruleset block

Session and NAT entries:
0 session allocations
0 session destructions
0 NAT entry allocations
0 NAT entry destructions
[...]



Then tried a simple block rule:

group (name "internal", interface $int_if) {
block in quick proto tcp to any port 80
}

/etc/rc.d/npf reload
Reloading NPF ruleset.
npfctl: n-code size got wrong (36 != 72)


/netbsd & /stand/amd64/5.99.56/modules/npf/npf.kmod from the same
build last night.
Could there be anything obvious I'm missing?

Thanks

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Mindaugas Rasiukevicius
2011-11-20 20:48:21 UTC
Permalink
Post by David Brownlee
/etc/rc.d/npf reload
Reloading NPF ruleset.
npfctl: n-code size got wrong (36 != 72)
/netbsd & /stand/amd64/5.99.56/modules/npf/npf.kmod from the same
build last night.
Could there be anything obvious I'm missing?
Thanks
There are multiple regressions after IPv6 merge, which broke IPv4 filtering
as well. I have various fixes in my local tree, which I hope to finish in
upcoming week. Also, separate fixes for TCP state tracking, which are still
under testing.
--
Mindaugas

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
David Brownlee
2011-11-20 22:24:48 UTC
Permalink
Post by Mindaugas Rasiukevicius
Post by David Brownlee
/etc/rc.d/npf reload
Reloading NPF ruleset.
npfctl: n-code size got wrong (36 != 72)
/netbsd & /stand/amd64/5.99.56/modules/npf/npf.kmod  from the same
build last night.
Could there be anything obvious I'm missing?
Thanks
There are multiple regressions after IPv6 merge, which broke IPv4 filtering
as well.  I have various fixes in my local tree, which I hope to finish in
upcoming week.  Also, separate fixes for TCP state tracking, which are still
under testing.
Ah - thanks for the quick response.

Should I just keep an eye on @current-users or @tech-net for an
indication of when it would be worth testing again?

Good luck with the cleanup :)

Thanks

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Darren Reed
2011-11-23 03:34:36 UTC
Permalink
Post by Mindaugas Rasiukevicius
Post by David Brownlee
/etc/rc.d/npf reload
Reloading NPF ruleset.
npfctl: n-code size got wrong (36 != 72)
/netbsd & /stand/amd64/5.99.56/modules/npf/npf.kmod from the same
build last night.
Could there be anything obvious I'm missing?
Thanks
There are multiple regressions after IPv6 merge, which broke IPv4 filtering
as well. I have various fixes in my local tree, which I hope to finish in
upcoming week. Also, separate fixes for TCP state tracking, which are still
under testing.
Well at least there is one firewall solution in NetBSD (ipfilter) where
testing (prior to integration) and security are taken seriously.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Scott Solmonson
2011-11-23 04:10:47 UTC
Permalink
Darren- I think that's entirely uncalled for and is not in the
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
-SS-- NUNQUAM NON PARATUS
Post by Darren Reed
Post by Mindaugas Rasiukevicius
Post by David Brownlee
/etc/rc.d/npf reload
Reloading NPF ruleset.
npfctl: n-code size got wrong (36 != 72)
/netbsd & /stand/amd64/5.99.56/modules/npf/npf.kmod  from the same
build last night.
Could there be anything obvious I'm missing?
Thanks
There are multiple regressions after IPv6 merge, which broke IPv4 filtering
as well.  I have various fixes in my local tree, which I hope to finish in
upcoming week.  Also, separate fixes for TCP state tracking, which are still
under testing.
Well at least there is one firewall solution in NetBSD (ipfilter) where
testing (prior to integration) and security are taken seriously.
Darren
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Jeremy C. Reed
2011-11-23 04:55:09 UTC
Permalink
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)

-current should not have broken code (note that current-users list now
has automated complaints on failures).

We should strive for a higher standard. We should encourage and maybe
better require that we provide unit tests and/or behaviour tests with
commits too. (Was there ever a public core announcement about when code
is added or bug fixed, that the developer should consider adding ATF
tests or regression tests for it?) (I'd like to extend this to include
security audit tests as applicable, documentation requirements, and peer
review requirements too.)

We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Matthew Mondor
2011-11-23 07:56:29 UTC
Permalink
On Tue, 22 Nov 2011 22:55:09 -0600 (CST)
Post by Jeremy C. Reed
We should strive for a higher standard. We should encourage and maybe
better require that we provide unit tests and/or behaviour tests with
commits too. (Was there ever a public core announcement about when code
is added or bug fixed, that the developer should consider adding ATF
tests or regression tests for it?) (I'd like to extend this to include
security audit tests as applicable, documentation requirements, and peer
review requirements too.)
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
While I agree with most of what you said on a technical level,
unfortunately one must also come to the evidence that NetBSD
maintainers are volunteers with limited time and resources :(

So between the ideal and the practice, it's normal if a gap exists...

That said, I find that the NetBSD code base in general is of a high
quality, and the review process which I often see happening on mailing
lists, while sometimes tedious, tends to help a lot.

As for ipfilter vs npf, npf is known to be in development by most of
us, I think; and ipfilter (or sometimes pf) are still being used on
production systems by many where reliability is important and existing
firewall scripts are maintained and relied-upon (I currently use
netbsd-5 and ipfilter myself). This doesn't mean that an alternative
cannot be in development, incomplete or unstable (especially on an OS
also known to be good for research, such as NetBSD)...
--
Matt

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
David Brownlee
2011-11-23 19:36:57 UTC
Permalink
Post by Matthew Mondor
On Tue, 22 Nov 2011 22:55:09 -0600 (CST)
Post by Jeremy C. Reed
We should strive for a higher standard. We should encourage and maybe
better require that we provide unit tests and/or behaviour tests with
commits too.  (Was there ever a public core announcement about when code
is added or bug fixed, that the developer should consider adding ATF
tests or regression tests for it?) (I'd like to extend this to include
security audit tests as applicable, documentation requirements, and peer
review requirements too.)
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
While I agree with most of what you said on a technical level,
unfortunately one must also come to the evidence that NetBSD
maintainers are volunteers with limited time and resources :(
So between the ideal and the practice, it's normal if a gap exists...
That said, I find that the NetBSD code base in general is of a high
quality, and the review process which I often see happening on mailing
lists, while sometimes tedious, tends to help a lot.
As for ipfilter vs npf, npf is known to be in development by most of
us, I think; and ipfilter (or sometimes pf) are still being used on
production systems by many where reliability is important and existing
firewall scripts are maintained and relied-upon (I currently use
netbsd-5 and ipfilter myself).  This doesn't mean that an alternative
cannot be in development, incomplete or unstable (especially on an OS
also known to be good for research, such as NetBSD)...
I think its fine for there to be experimental features in - current -
specifically features which have not yet been in any formal release.

In this case I think a note in the manpage, or a stderr message from
npfctl to alert users might have been helpful...

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Zoltan Arnold NAGY
2011-11-23 23:13:48 UTC
Permalink
Post by Jeremy C. Reed
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)
[...]
Post by Jeremy C. Reed
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
When I committed the code, I did test it with both v4 and v6. Apart from the TCP
state engine bugs, I did not encounter any issues, that's why the commit.

Sorry if it got thru. I'll work with rmind@ on the weekend to fix these.

Best,
Zoltan

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Darren Reed
2011-11-24 04:07:21 UTC
Permalink
Post by Zoltan Arnold NAGY
Post by Jeremy C. Reed
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)
[...]
Post by Jeremy C. Reed
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
When I committed the code, I did test it with both v4 and v6. Apart from the TCP
state engine bugs, I did not encounter any issues, that's why the commit.
Let me summarise the email to which I responded to for the benefit
of yourself and others in a single sentence:

"The IPv6 merge introduced numerous security bugs."

Exactly what testing was done prior to the merge and how was it done?

Did anyone review these changes or the state of the testing?

Were there people behind the scenes telling you to "get it in",
regardless of the condition of the code?

Darren

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Zoltan Arnold NAGY
2011-11-24 08:20:52 UTC
Permalink
Post by Darren Reed
Post by Zoltan Arnold NAGY
Post by Jeremy C. Reed
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)
[...]
Post by Jeremy C. Reed
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
When I committed the code, I did test it with both v4 and v6. Apart from the TCP
state engine bugs, I did not encounter any issues, that's why the commit.
Let me summarise the email to which I responded to for the benefit
"The IPv6 merge introduced numerous security bugs."
Could you list non-NPF specific security bugs that were introduces?
I still yet to see a list.
Post by Darren Reed
Exactly what testing was done prior to the merge and how was it done?
Regular usage scenarios. No automated testing with a packet generator,
if that's what you're suggesting.

If we did introduce security holes even when npf is disabled, I sincerely
apologize; if we did not, then I seriously don't get your tone.
I'm not a half-wit, so I can perfectly follow the conversation so far,
thank you.

Zoltan

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Darren Reed
2011-11-24 14:08:34 UTC
Permalink
Post by Zoltan Arnold NAGY
Post by Darren Reed
Post by Zoltan Arnold NAGY
Post by Jeremy C. Reed
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)
[...]
Post by Jeremy C. Reed
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
When I committed the code, I did test it with both v4 and v6. Apart from the TCP
state engine bugs, I did not encounter any issues, that's why the commit.
Let me summarise the email to which I responded to for the benefit
"The IPv6 merge introduced numerous security bugs."
Could you list non-NPF specific security bugs that were introduces?
I still yet to see a list.
Post by Darren Reed
Exactly what testing was done prior to the merge and how was it done?
Regular usage scenarios. No automated testing with a packet generator,
if that's what you're suggesting.
If we did introduce security holes even when npf is disabled, I sincerely
apologize; if we did not, then I seriously don't get your tone.
Because even if it is disabled by default, there's nothing stopping someone
from downloading -current today, using npf and falling victim to the bugs.

There's more reasoning but I just can't seem to put the thoughts and ideas
into coherant sentences (everything I try just comes out wrong).
My apologies for that.

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
rudolf
2011-11-24 15:47:10 UTC
Permalink
Post by Darren Reed
Post by Zoltan Arnold NAGY
If we did introduce security holes even when npf is disabled, I sincerely
apologize; if we did not, then I seriously don't get your tone.
Because even if it is disabled by default, there's nothing stopping someone
from downloading -current today, using npf and falling victim to the bugs.
There's more reasoning but I just can't seem to put the thoughts and ideas
into coherant sentences (everything I try just comes out wrong).
My apologies for that.
NetBSD provides enough warnings along the way for -current users, e.g. [1]:
This system is running a development snapshot of the NetBSD operating
system, also known as NetBSD-current. It is very possible that it has
serious bugs, regressions, broken features or other problems. Please
bear this in mind and use the system with care.

r.

[1] http://nxr.netbsd.org/xref/src/etc/motd.current

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Zoltan Arnold NAGY
2011-11-24 20:41:00 UTC
Permalink
Post by rudolf
Post by Darren Reed
Post by Zoltan Arnold NAGY
If we did introduce security holes even when npf is disabled, I sincerely
apologize; if we did not, then I seriously don't get your tone.
Because even if it is disabled by default, there's nothing stopping someone
from downloading -current today, using npf and falling victim to the bugs.
There's more reasoning but I just can't seem to put the thoughts and ideas
into coherant sentences (everything I try just comes out wrong).
My apologies for that.
This system is running a development snapshot of the NetBSD operating
system, also known as NetBSD-current.  It is very possible that it has
serious bugs, regressions, broken features or other problems.  Please bear
this in mind and use the system with care.
And by all means, please feel free to open PRs, and feel free to
assign them directly
to me. I'm happy to fix them.
But it's hard to fix something I don't know about :)

Thanks,
Zoltan

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Alistair Crooks
2011-11-25 05:07:48 UTC
Permalink
Post by Jeremy C. Reed
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)
-current should not have broken code (note that current-users list now
has automated complaints on failures).
We should strive for a higher standard. We should encourage and maybe
better require that we provide unit tests and/or behaviour tests with
commits too. (Was there ever a public core announcement about when code
is added or bug fixed, that the developer should consider adding ATF
tests or regression tests for it?) (I'd like to extend this to include
security audit tests as applicable, documentation requirements, and peer
review requirements too.)
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
It is this kind of short-term thinking that depresses me. People do
not (typically) coordinate changes to the repo, and so there is
invariably some fallout, some things need to be fixed up, etc. In
addition to that, I don't know anyone who has every single
architecture, let alone every single platform. So some platforms go
untested.

I think it is completely unrealistic to expect -current to compile at
any one time, and I have been fixing some compilation issues in -current
on and off for a number of years now. Castigating people for checking
things in which are not 100% will have the marvellous effect of encouraging
people not to commit anything, rather than encouraging them to commit 100%
functional work.

Nowadays, we have people running automatic build tests, and the anita
runs are superb (thanks, gson!), along with some very enthusiastic
builders (bch and htodd, to name but two), and havard builds for some
of the more unorthodox architectures. Which leads me to say: I don't
know where you're coming from with this; in fact, I don't remember
your being active in this area, but I may have overlooked something
just recently.

So, yes, laudable aim - completely unworkable in practice.

Regards,
Alistair

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Darren Reed
2011-11-25 09:21:37 UTC
Permalink
Post by Alistair Crooks
Post by Jeremy C. Reed
Post by Scott Solmonson
interest of progress.Remember that this is -CURRENT, where things like
this are *supposed* to happen?
As for me, I was glad Darren pointed this out. (In fact, I was quite
surprised when I read the followup acknowledging known buggy code living
in -current.)
-current should not have broken code (note that current-users list now
has automated complaints on failures).
We should strive for a higher standard. We should encourage and maybe
better require that we provide unit tests and/or behaviour tests with
commits too. (Was there ever a public core announcement about when code
is added or bug fixed, that the developer should consider adding ATF
tests or regression tests for it?) (I'd like to extend this to include
security audit tests as applicable, documentation requirements, and peer
review requirements too.)
We should suggest and even force that code known to be broken to be
reverted. (Well I think this is already true, but not happening?) (It
will be easier when we have a better revision control so many can work
easier on branches.)
It is this kind of short-term thinking that depresses me. People do
not (typically) coordinate changes to the repo, and so there is
invariably some fallout, some things need to be fixed up, etc. In
addition to that, I don't know anyone who has every single
architecture, let alone every single platform. So some platforms go
untested.
I think it is completely unrealistic to expect -current to compile at
any one time, and I have been fixing some compilation issues in -current
on and off for a number of years now. Castigating people for checking
things in which are not 100% will have the marvellous effect of encouraging
people not to commit anything, rather than encouraging them to commit 100%
functional work.
Nowadays, we have people running automatic build tests, and the anita
runs are superb (thanks, gson!), along with some very enthusiastic
builders (bch and htodd, to name but two), and havard builds for some
of the more unorthodox architectures. Which leads me to say: I don't
know where you're coming from with this; in fact, I don't remember
your being active in this area, but I may have overlooked something
just recently.
So, yes, laudable aim - completely unworkable in practice.
Let me outline what my development process is for doing a
merge when importing ipfilter:

1) sync up a local copy of the repo with rsync to netbsd
2) checkout a copy of that repo
3) do a build of that checked out copy over nfs
4) create zfs snapshots for both the local repo and the
local build on the server

This creates my baseline.

1) import local changes into the local repo, update from
vendor branch to head and checkout
2) apply any required patches
3) update the checked out copy
4) run a build (without doing a "cleandir")
5) if there are any issues, add changes to required patches,
rollback zfs snapshots, go back to (1)

Steps 1, 2, 3 and 4 are all scripted.

I should now have something that should let me update netbsd's
cvs without causing anyone grief. If there are any problems,
they're likely to be minor, quickly and easily fixed.

At this point I also have something that is easily tested and
whilst in the past I've relied upon using ipftest as the only
means of regression testing (processing the packets entirely
in a user space application that embodies all of the kernel
functions), I've now expanded that to allow packets to be
tested in the kernel. The next steps along that path are to
use dedicated test NICs (so that I can test the path into
and out of IP) and further to use different hosts.

It goes without saying that the above use of ZFS snapshots
means that my server isn't NetBSD but the additional time
cost from building over NFS is saved by being able to do
snapshot rollbacks.

I haven't tried the above with UFS snapshots from NetBSD
to know if they would work. If someone needs a reason to
keep pushing for ZFS to work well on NetBSD, consider the
above benefits for the commit cycle. For someone doing
test builds, you could do an "update; snapshot; incremental
build" quite easily.

In Solaris, all commits to the internal Hg repo kick off
a similar sequence of events and committers get an automatic
email if their commit resulted in any new build or lint
errors/warnings. ZFS is the primary enabled here.
Occassionally there are commits where an incremental build
is known to be problematic but this is called out in advance
by the committer(s).

Darren


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de
Loading...