IFQ_MAXLEN: How large can it be?

[Christoph Kaegi]

Hello list

Thanks to the help of Manuel, I found the above mentionned
setting which defines the size of the (per-adapter?) IP input
queue.

So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.

- What are the side effects of increasing this value even more?

- Can I safely set it to 8192 or 16384 without getting into
other troubles?

- I guess I could also just use values of 8000, 10000 or 20000
isn't it?

- Also, is this value number of packets or number of bytes
or something else?

I have NMBCLUSTERS=65536 at the moment, btw.

Thanks
Chris

[Manuel Bouyer]

Post by Christoph Kaegi
Hello list
Thanks to the help of Manuel, I found the above mentionned
setting which defines the size of the (per-adapter?) IP input
queue.
So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.
- What are the side effects of increasing this value even more?
- Can I safely set it to 8192 or 16384 without getting into
other troubles?
- I guess I could also just use values of 8000, 10000 or 20000
isn't it?
- Also, is this value number of packets or number of bytes
or something else?
I have NMBCLUSTERS=65536 at the moment, btw.

This is the number of packets per interface. You can count one mbuf cluster
(2k) per packet, so you have to make sure that
- NMBCLUSTERS is larger than IFQ_MAXLEN * num_interfaces (if you know you
have mostly-idle interfaces, maybe you can change this to
IFQ_MAXLEN * num_busy_interfaces + some_more)
- you have enough ram to hold that much NMBCLUSTERS (a mbuf cluser is 2k),
as this memory can't be paged out.

[Steven M. Bellovin]

Post by Christoph Kaegi
Hello list
Thanks to the help of Manuel, I found the above mentionned
setting which defines the size of the (per-adapter?) IP input
queue.
So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.

It's far from clear to me that this is a big help. There's a fair amount
of literature that says that too-large router queues are bad, since they
end up having many retransmissions of the same data. I suggest that you
look at other resources -- CPU and output line rate come to mind -- and
start playing with some of the fancier queueing options on your output
link. (I wonder -- it would be nice to be able to do RED on things like
the IP input queue. Is that possible?)

--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.

I think this is unwise and will just result in more memory stress. If
4096 doesn't help, you're not keeping up often, or there's something
else going on. I'd be nervous putting this above 256 - people usually
don't and then you'll be stressing the mbuf system more than others
have stressed it and fixed it.

Where are the packets being dropped? Normally received packets get
put on the (single, system-wide) IP input queue, and then a soft
interrupt causes them to be processed and placed on output queues.
Can you post your statistics that point at this? "netstat -s" is
very useful if you haven't run that, as is "netstat -i".

[Manuel Bouyer]

Post by Christoph Kaegi
So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.
I think this is unwise and will just result in more memory stress. If
4096 doesn't help, you're not keeping up often, or there's something
else going on. I'd be nervous putting this above 256 - people usually
don't and then you'll be stressing the mbuf system more than others
have stressed it and fixed it.

Well, actually you have to bump it if you have more than one busy interface
with large ring. The wm interface can have a ring of 256 packet, so
you could get in worse case 256 packets in one interrrupt; and if you have
more than one interface you loose.

BTW, Christoph, where do you see these drops ? On the IP input queue, or
interfaces output queues ?

[Christoph Kaegi]

Post by Manuel Bouyer
BTW, Christoph, where do you see these drops ? On the IP input queue, or
interfaces output queues ?

Where can I see the count of dropped packets from the output queues?

If 'netstat -id': they're all 0.

The dropped packets I reported where all on the ipintrq.

(see netstat -q output in my previous reply the list)

Thanks
Chris

[Manuel Bouyer]

Post by Christoph Kaegi

Post by Manuel Bouyer
BTW, Christoph, where do you see these drops ? On the IP input queue, or
interfaces output queues ?

Where can I see the count of dropped packets from the output queues?
If 'netstat -id': they're all 0.

Then there's no dropped paquets at this point

Post by Christoph Kaegi
The dropped packets I reported where all on the ipintrq.

OK, so maybe increasing the input queue size can be of some help to
absorb bursts. how many interfaces with lots of input traffic do you have ?

[Christoph Kaegi]

Post by Manuel Bouyer

Post by Christoph Kaegi
The dropped packets I reported where all on the ipintrq.

OK, so maybe increasing the input queue size can be of some help to
absorb bursts. how many interfaces with lots of input traffic do you have ?

I'd say three. Ca. 1 Minute of nload says:

---------------------------- 8< ----------------------------
Device wm0 [111.222.333.1] (1/4):
========================================================================================I
ncoming: Outgoing:
Curr: 2.72 MBit/s Curr: 5.74 MBit/s
Avg: 3.51 MBit/s Avg: 26.55 MBit/s
Min: 1.23 MBit/s Min: 5.52 MBit/s
Max: 26.61 MBit/s Max: 79.01 MBit/s
Ttl: 1170.86 MByte Ttl: 1726.58 MByte

Device wm1 [111.222.333.2] (2/4):
========================================================================================I
ncoming: Outgoing:
Curr: 5.26 MBit/s Curr: 4.90 MBit/s
Avg: 23.51 MBit/s Avg: 21.65 MBit/s
Min: 3.85 MBit/s Min: 3.65 MBit/s
Max: 72.64 MBit/s Max: 71.25 MBit/s
Ttl: 2420.62 MByte Ttl: 1757.23 MByte

Device wm2 [111.222.333.3] (3/4):
========================================================================================I
ncoming: Outgoing:
Curr: 6.10 MBit/s Curr: 1.97 MBit/s
Avg: 25.31 MBit/s Avg: 3.08 MBit/s
Min: 5.35 MBit/s Min: 1.18 MBit/s
Max: 80.52 MBit/s Max: 9.92 MBit/s
Ttl: 3341.35 MByte Ttl: 2307.23 MByte

Device wm3 [111.222.333.4] (4/4):
========================================================================================I
ncoming: Outgoing:
Curr: 0.05 MBit/s Curr: 1.43 MBit/s
Avg: 0.37 MBit/s Avg: 1.27 MBit/s
Min: 0.00 MBit/s Min: 0.00 MBit/s
Max: 2.80 MBit/s Max: 9.81 MBit/s
Ttl: 2763.97 MByte Ttl: 2053.87 MByte

---------------------------- 8< ----------------------------

[Manuel Bouyer]

Post by Manuel Bouyer

Post by Christoph Kaegi
The dropped packets I reported where all on the ipintrq.

OK, so maybe increasing the input queue size can be of some help to
absorb bursts. how many interfaces with lots of input traffic do you have ?

OK, so 4096 should be more than enough. There are 256 receive descriptors per
wm interface; so even if the ring is full when the hardware interrupt is
processed, it will fill only 768 slots in the IP input queue.

Maybe your problem is lack of CPU power then. What kind of hardware is it ?

[Christoph Kaegi]

Post by Manuel Bouyer

Post by Manuel Bouyer

Post by Christoph Kaegi
The dropped packets I reported where all on the ipintrq.

OK, so maybe increasing the input queue size can be of some help to
absorb bursts. how many interfaces with lots of input traffic do you have ?

OK, so 4096 should be more than enough. There are 256 receive descriptors per
wm interface; so even if the ring is full when the hardware interrupt is
processed, it will fill only 768 slots in the IP input queue.
Maybe your problem is lack of CPU power then. What kind of hardware is it ?

It's a Transtec 1002L (aka Supermicro 6023P-8), dmesg output is below.

vmstat 1 looks like this at the moment:
---------------------------- 8< ----------------------------
intfw (primary)# vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr m0 c0 s0 in sy cs us sy id
0 0 0 268124 593708 74 0 0 0 0 0 0 0 0 4294965249 127 8 0 11 89
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6813 30 7 0 28 72
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7394 30 7 0 13 87
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6687 30 7 0 26 74
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6326 30 7 0 16 84
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6951 30 7 0 19 81
1 0 0 268124 593708 5 0 0 0 0 0 0 0 1 6739 30 12 0 6 94
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7274 35 7 0 7 93
0 0 0 268124 593708 813 0 0 0 0 0 0 0 0 6847 1164 27 0 22 78
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6658 30 8 0 12 88
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6628 30 7 0 36 64
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6784 30 7 0 42 58
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6776 30 7 0 64 36
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7266 30 7 0 14 86
0 0 0 268124 593708 1447 0 0 0 0 0 0 0 0 7115 113 14 0 29 71
0 0 0 268124 593708 8 0 0 0 0 0 0 0 0 7293 48 7 0 14 86
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7530 30 6 0 21 79
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7502 30 7 0 15 85
0 0 0 268124 593708 811 0 0 0 0 0 0 0 1 7744 1170 28 0 16 84
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7401 30 7 0 20 80
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 8295 30 8 0 11 89
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 8049 30 7 0 21 79
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7751 30 7 0 8 92
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6594 36 8 0 9 91
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6511 32 8 0 20 80
---------------------------- 8< ----------------------------

I always wondered, if the system is able to handle the
10'000 to 20'000 interrupts it sometimes gets.
(according to sysstat vm 1)
Sometimes it even gets more than that.

Can I somehow measure if interrupts get lost?

Thanks alot for your attention Manuel!

Chris

---------------------------- 8< ----------------------------
NetBSD 3.1_RC4 (gw) #7: Mon Nov 6 13:07:45 CET 2006
***@gw:/usr/src/NetBSD-3/src/sys/arch/i386/compile/INTFW
total memory = 1023 MB
avail memory = 998 MB
BIOS32 rev. 0 found at 0xfd651
mainbus0 (root)
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel Xeon (686-class), 2666.01 MHz, id 0xf29
cpu0: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 4400<CID,xTPR>
cpu0: "Intel(R) Xeon(TM) CPU 2.66GHz"
cpu0: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu0: L2 cache 512 KB 64B/line 8-way
cpu0: ITLB 4K/4M: 64 entries
cpu0: DTLB 4K/4M: 64 entries
cpu0: running without thermal monitor!
cpu0: 16 page colors
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel E7501 MCH Host (rev. 0x01)
Intel E7500 MCH DRAM Controller (undefined subclass 0x00, revision 0x01) at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 2 function 0: Intel E7500 MCH HI_B vppb 1 (rev. 0x01)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled
wm0 at pci3 dev 2 function 0: Intel i82546EB 1000BASE-T Ethernet, rev. 1
wm0: interrupting at irq 11
wm0: 64-bit 133MHz PCIX bus
wm0: 256 word (8 address bits) MicroWire EEPROM
wm0: Ethernet address 00:30:48:2a:9c:d4
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm1 at pci3 dev 2 function 1: Intel i82546EB 1000BASE-T Ethernet, rev. 1
wm1: interrupting at irq 11
wm1: 64-bit 133MHz PCIX bus
wm1: 256 word (8 address bits) MicroWire EEPROM
wm1: Ethernet address 00:30:48:2a:9c:d5
makphy1 at wm1 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb3 at pci0 dev 3 function 0: Intel E7500 MCH HI_C vppb 1 (rev. 0x01)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci4 dev 28 function 0 not configured
ppb4 at pci4 dev 29 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci5 at ppb4 bus 5
pci5: i/o space, memory space enabled
wm2 at pci5 dev 1 function 0: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm2: interrupting at irq 11
wm2: 64-bit 66MHz PCIX bus
wm2: 256 word (8 address bits) MicroWire EEPROM
wm2: Ethernet address 00:04:23:a7:b9:cc
makphy2 at wm2 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm3 at pci5 dev 1 function 1: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm3: interrupting at irq 11
wm3: 64-bit 66MHz PCIX bus
wm3: 256 word (8 address bits) MicroWire EEPROM
wm3: Ethernet address 00:04:23:a7:b9:cd
makphy3 at wm3 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm4 at pci5 dev 2 function 0: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm4: interrupting at irq 11
wm4: 64-bit 66MHz PCIX bus
wm4: 256 word (8 address bits) MicroWire EEPROM
wm4: Ethernet address 00:04:23:a7:f5:c2
makphy4 at wm4 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy4: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm5 at pci5 dev 2 function 1: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm5: interrupting at irq 11
wm5: 64-bit 66MHz PCIX bus
wm5: 256 word (8 address bits) MicroWire EEPROM
wm5: Ethernet address 00:04:23:a7:f5:c3
makphy5 at wm5 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy5: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci4 dev 30 function 0 not configured
ppb5 at pci4 dev 31 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled
ahd0 at pci6 dev 2 function 0
ahd0: interrupting at irq 11
ahd0: aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
scsibus0 at ahd0: 16 targets, 8 luns per target
ahd1 at pci6 dev 2 function 1
ahd1: interrupting at irq 11
ahd1: aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
scsibus1 at ahd1: 16 targets, 8 luns per target
uhci0 at pci0 dev 29 function 0: Intel 82801CA USB Controller (rev. 0x02)
uhci0: interrupting at irq 11
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801CA USB Controller (rev. 0x02)
uhci1: interrupting at irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2: Intel 82801CA USB Controller (rev. 0x02)
uhci2: interrupting at irq 5
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ppb6 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0x42)
pci7 at ppb6 bus 7
pci7: i/o space, memory space enabled
vga1 at pci7 dev 1 function 0: ATI Technologies Rage XL (rev. 0x27)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801CA LPC Interface (rev. 0x02)
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801CA IDE Controller (ICH3) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at irq 14
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel interrupting at irq 15
atabus1 at piixide0 channel 1
Intel 82801CA SMBus Controller (SMBus serial bus, revision 0x02) at pci0 dev 31 function 3 not configured
isa0 at pcib0
lpt0 at isa0 port 0x378-0x37b irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
lm0 at isa0 port 0x290-0x297: W83627HF
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0-0xff: using exception 16
apm0 at mainbus0: Power Management spec V1.2
Kernelized RAIDframe activated
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 0: <CD-224E, , 1.9A> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(piixide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST336607LC, 0007> disk fixed
sd0: 35003 MB, 49855 cyl, 2 head, 718 sec, 512 bytes/sect x 71687372 sectors
sd0: sync (6.25ns offset 63), 16-bit (320.000MB/s) transfers, tagged queueing
sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST336607LC, 0007> disk fixed
sd1: 35003 MB, 49855 cyl, 2 head, 718 sec, 512 bytes/sect x 71687372 sectors
sd1: sync (6.25ns offset 63), 16-bit (320.000MB/s) transfers, tagged queueing
ses0 at scsibus0 target 6 lun 0: <SUPER, GEM318, 0> processor fixed
ses0: SAF-TE Compliant Device
ses0: async, 8-bit transfers
raid0: RAID Level 1
raid0: Components: /dev/sd0a /dev/sd1a
raid0: Total Sectors: 71687168 (35003 MB)
boot device: raid0
root on raid0a dumps on raid0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
---------------------------- 8< ----------------------------

[Manuel Bouyer]

Post by Christoph Kaegi
[...]
I always wondered, if the system is able to handle the
10'000 to 20'000 interrupts it sometimes gets.
(according to sysstat vm 1)

This shouldn't be an issue. I have less powerfull systems that gets that
much interruts.

Post by Christoph Kaegi
Sometimes it even gets more than that.
Can I somehow measure if interrupts get lost?

You can't really "loose" interrupts; the interrupt in keep pending until
it's serviced. If another comes in while it's pending, the interrupt routine
will just do all the work at once when it's called.

Did you enable ip4, tcp4 and udp4 checksums on your wm adapters ? This
can help a lot.

[Christoph Kaegi]

Post by Thor Lancelot Simon
Of course. You should also enable segmentation offload, if you haven't
already done so.

I'll do so tonight (as we have a maintenance window tonight)

Post by Thor Lancelot Simon
What value of HZ are you using? Perhaps things are backing up because
the softnet processing is not happening often enough?

Never heard of this option before. options(4) doesn't go into
much details what this does. Could you explain its function
to me?

Thanks
Chris

[Thor Lancelot Simon]

Post by Christoph Kaegi

Post by Thor Lancelot Simon
Of course. You should also enable segmentation offload, if you haven't
already done so.

I'll do so tonight (as we have a maintenance window tonight)

It won't break anything; it is safe to turn on while your system is busy.

Post by Christoph Kaegi

Post by Thor Lancelot Simon
What value of HZ are you using? Perhaps things are backing up because
the softnet processing is not happening often enough?

Never heard of this option before. options(4) doesn't go into
much details what this does. Could you explain its function
to me?

It controls how often the system clock "ticks". Much network processing
happens at the next clock tick after the network interrupt that
transfers the data from the network interface, not immediately upon
hardware interrupt, so it is possible that causing the clock to tick
more often may help with your problem. Try a value of 1024.

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jason Thorpe]

Post by Thor Lancelot Simon
It controls how often the system clock "ticks". Much network
processing
happens at the next clock tick after the network interrupt that
transfers the data from the network interface, not immediately upon
hardware interrupt, so it is possible that causing the clock to tick
more often may help with your problem. Try a value of 1024.

This is not true. It happens in the softnet handler often immediately
after the network interface's interrupt handler returns.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jason Thorpe]

Post by Christoph Kaegi

Post by Jason Thorpe

Post by Thor Lancelot Simon
What value of HZ are you using? Perhaps things are backing up because
the softnet processing is not happening often enough?

Softnet processing is not driven by HZ.

Does that mean, increasing HZ probably won't help my problem?

Correct.

Post by Christoph Kaegi
--
----------------------------------------------------------------------
----------------------------------------------------------------------

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Christoph Kaegi]

Post by Manuel Bouyer

Post by Christoph Kaegi
[...]
I always wondered, if the system is able to handle the
10'000 to 20'000 interrupts it sometimes gets.
(according to sysstat vm 1)

This shouldn't be an issue. I have less powerfull systems that gets that
much interruts.

Post by Christoph Kaegi
Sometimes it even gets more than that.
Can I somehow measure if interrupts get lost?

You can't really "loose" interrupts; the interrupt in keep pending until
it's serviced. If another comes in while it's pending, the interrupt routine
will just do all the work at once when it's called.
Did you enable ip4, tcp4 and udp4 checksums on your wm adapters ? This
can help a lot.

No I didn't. Can I do that online while the machine is running
without risking it going belly-up?

If not, I'll enable them tonight.

[Thor Lancelot Simon]

Post by Christoph Kaegi

Post by Manuel Bouyer
Did you enable ip4, tcp4 and udp4 checksums on your wm adapters ? This
can help a lot.

No I didn't. Can I do that online while the machine is running
without risking it going belly-up?

Of course. You should also enable segmentation offload, if you haven't
already done so.

What value of HZ are you using? Perhaps things are backing up because
the softnet processing is not happening often enough?

Thor

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jason Thorpe]

Post by Thor Lancelot Simon
What value of HZ are you using? Perhaps things are backing up because
the softnet processing is not happening often enough?

Softnet processing is not driven by HZ.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Christoph Kaegi]

Post by Jason Thorpe

Post by Thor Lancelot Simon
What value of HZ are you using? Perhaps things are backing up because
the softnet processing is not happening often enough?

Softnet processing is not driven by HZ.

Does that mean, increasing HZ probably won't help my problem?

[Christoph Kaegi]

Post by Steven M. Bellovin

Post by Christoph Kaegi
So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.

It's far from clear to me that this is a big help. There's a fair amount
of literature that says that too-large router queues are bad, since they
end up having many retransmissions of the same data. I suggest that you
look at other resources -- CPU and output line rate come to mind -- and
start playing with some of the fancier queueing options on your output
link. (I wonder -- it would be nice to be able to do RED on things like
the IP input queue. Is that possible?)

What is "RED"? What do you mean bei "output line rate"?
I wasn't aware I had queueing options on my output links.
Did you mean ALTQ? Does that work?

I'll gladly do any measuring and reporting you request if you
help me to interpret the values :-)

I have to say though, that we're somewhat limited playing around
with this system as it is in production. My timeslot for reboots
is some hours every two weeks.

Chris

[Christoph Kaegi]

Post by Christoph Kaegi
So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.
I think this is unwise and will just result in more memory stress. If
4096 doesn't help, you're not keeping up often, or there's something
else going on. I'd be nervous putting this above 256 - people usually
don't and then you'll be stressing the mbuf system more than others
have stressed it and fixed it.
Where are the packets being dropped? Normally received packets get
put on the (single, system-wide) IP input queue, and then a soft
interrupt causes them to be processed and placed on output queues.
Can you post your statistics that point at this? "netstat -s" is
very useful if you haven't run that, as is "netstat -i".

After what I can see, they're dropped from the IP input queue
(see netstat -q output at the end).

# netstat -s
---------------------------- 8< ----------------------------
ip:
2848237254 total packets received
2 bad header checksums
0 with size smaller than minimum
18 with data size < data length
0 with length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped (out of ipqent)
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
395403 packets for this host
0 packets for unknown/unsupported protocol
2801729041 packets forwarded (0 packets fast forwarded)
124599 packets not forwardable
1635 redirects sent
0 packets no matching gif found
2184588 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
99 datagrams with bad address in header
icmp:
1482423 calls to icmp_error
8793 errors not generated because old message was icmp
Output histogram:
echo reply: 320
destination unreachable: 60035
routing redirect: 1635
time exceeded: 1298856
13 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
destination unreachable: 13
echo: 320
320 message responses generated
0 path MTU changes
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
tcp:
743848 packets sent
736473 data packets (87991304 bytes)
1119 data packets (994232 bytes) retransmitted
6181 ack-only packets (11003 delayed)
0 URG only packets
0 window probe packets
0 window update packets
81 control packets
0 send attempts resulted in self-quench
392068 packets received
375345 acks (for 87986634 bytes)
8819 duplicate acks
0 acks for unsent data
14376 packets (805497 bytes) received in-sequence
18 completely duplicate packets (52 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
10 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
503 window update packets
10 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
20 connection requests
43 connection accepts
63 connections established (including accepts)
159080 connections closed (including 7 drops)
0 embryonic connections dropped
0 delayed frees of tcpcb
374663 segments updated rtt (of 91312 attempts)
530 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
103 keepalive timeouts
97 keepalive probes sent
6 connections dropped by keepalive
4490 correct ACK header predictions
6627 correct data packet header predictions
100 PCB hash misses
10 dropped due to no socket
0 connections drained due to memory shortage
0 PMTUD blackholes detected
0 bad connection attempts
43 SYN cache entries added
0 hash collisions
43 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
0 delayed free of SYN cache entries
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
4 SYNs dropped (no route or no space)
0 packets with bad signature
0 packets with good signature
udp:
3029 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
14 dropped due to no socket
81 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
2934 delivered
2449 PCB hash misses
79825 datagrams output
arp:
52026 packets sent
21730 reply packets
30296 request packets
486913 packets received
21310 reply packets
173743 valid request packets
447459 broadcast/multicast packets
0 packets with unknown protocol type
0 packets with bad (short) length
291860 packets with null target IP address
0 packets with null source IP address
0 could not be mapped to an interface
0 packets sourced from a local hardware address
0 packets with a broadcast source hardware address
0 duplicates for a local IP address
0 attempts to overwrite a static entry
0 packets received on wrong interface
0 entrys overwritten
0 changes in hardware address length
23096 packets deferred pending ARP resolution
4809 sent
13537 dropped
0 failures to allocate llinfo
---------------------------- 8< ----------------------------

netstat -i
---------------------------- 8< ----------------------------
(network addresses anonymized)
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls
wm0 1500 <Link> 00:30:48:dd:9c:d4 870327391 0 1099730201 0 0
wm0 1500 120.81.1/24 gw1.zhwin.ch 870327391 0 1099730201 0 0
wm1 1500 <Link> 00:30:48:dd:9c:d5 789681410 0 656853416 0 0
wm1 1500 120.81.3/24 gw2.zhwin.ch 789681410 0 656853416 0 0
wm2 1500 <Link> 00:04:23:dd:b9:cc 1072700285 0 874684304 0 0
wm2 1500 120.81.5/24 gw3.zhwin.c 1072700285 0 874684304 0 0
wm3 1500 <Link> 00:04:23:dd:b9:cd 118809847 0 175760316 0 0
wm3 1500 120.81.8/24 gw4.zhwin.ch 118809847 0 175760316 0 0
wm4* 1500 <Link> 00:04:23:dd:f5:c2 0 0 0 0 0
wm5* 1500 <Link> 00:04:23:dd:f5:c3 0 0 0 0 0
lo0 33192 <Link> 0 0 0 0 0
lo0 33192 loopback/8 localhost 0 0 0 0 0
---------------------------- 8< ----------------------------

# netstat -q
---------------------------- 8< ----------------------------
arpintrq:
queue length: 0
maximum queue length: 50
packets dropped: 4419
ipintrq:
queue length: 0
maximum queue length: 4096
packets dropped: 1102533
---------------------------- 8< ----------------------------

[Steven M. Bellovin]

Post by Christoph Kaegi

Post by Steven M. Bellovin

Post by Christoph Kaegi
So I bumped this number on our quite busy firewall up from 256
to 1024 and later to 4096, but I still get 1'026'678 dropped
packets during 8 days uptime.

It's far from clear to me that this is a big help. There's a fair amount
of literature that says that too-large router queues are bad, since they
end up having many retransmissions of the same data. I suggest that you
look at other resources -- CPU and output line rate come to mind -- and
start playing with some of the fancier queueing options on your output
link. (I wonder -- it would be nice to be able to do RED on things like
the IP input queue. Is that possible?)

What is "RED"? What do you mean bei "output line rate"?
I wasn't aware I had queueing options on my output links.
Did you mean ALTQ? Does that work?

"RED" is an output interface queueing discipline (it stands for, as I
recall, "Random Early Drop", though the altq.conf file says "Random Early
Detection"). Essentially, packets in an output queue are dropped with a
probability once the queue reaches a certain length. Yes, that's right;
it doesnot simply drop newly-arriving packets when the queue has reached
its maximum size. It seems counter-intuitive, but it works better than
the default tail-drop strategy, because it causes sending TCPs to back off
their retransmission timers more quickly and hence avoids duplicate copies
of packets in the queue. (I was talking about ALTQ, but I don't see a way
to apply that to the IP input queue, which is too bad.)

By "output line rate", I was asking how fast the output size runs. From
your later message about 'netstat -q', that's not the relevant question.
I think your CPU is too slow or too overloaded compared to your input line
speed. What's happening is that packets are arriving faster than your IP
stack can process them. The only issue is how long such bursts last;
ultimately, you're going to get packet drops if your CPU can't keep up
with the arrival rate.

Let me repeat what I said earlier: I think that having a 4K queue length
is not only not useful, it's quite possibly counter-productive.

You showed four active wm interfaces in a message. How fast are they
running? I know that some wm interfaces can run at gigabit speeds -- are
you trying to do that? How fast is your CPU? What do 'top' or
'sysstat' show for CPU idle percentage?

--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Christoph Kaegi]

Post by Steven M. Bellovin

Post by Christoph Kaegi
What is "RED"? What do you mean bei "output line rate"?
I wasn't aware I had queueing options on my output links.
Did you mean ALTQ? Does that work?

"RED" is an output interface queueing discipline (it stands for, as I
recall, "Random Early Drop", though the altq.conf file says "Random Early
Detection"). Essentially, packets in an output queue are dropped with a
probability once the queue reaches a certain length. Yes, that's right;
it doesnot simply drop newly-arriving packets when the queue has reached
its maximum size. It seems counter-intuitive, but it works better than
the default tail-drop strategy, because it causes sending TCPs to back off
their retransmission timers more quickly and hence avoids duplicate copies
of packets in the queue. (I was talking about ALTQ, but I don't see a way
to apply that to the IP input queue, which is too bad.)

Thanks for the explanation.

Post by Steven M. Bellovin
You showed four active wm interfaces in a message. How fast are they
running? I know that some wm interfaces can run at gigabit speeds -- are
you trying to do that?

wm0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (1000baseT full-duplex)

wm1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex)

wm2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex)

wm3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex)

wm0 runs gigabit, this is the inwards interface which should
be able to keep up with traffic from internet an dmz, which
both are 100MBit/s

Post by Steven M. Bellovin
How fast is your CPU?

Inter Xeon 2.7GHz

Post by Steven M. Bellovin
What do 'top' or
'sysstat' show for CPU idle percentage?

Usually the cpu shows between 50% and 90% idle.
Though there may be bursts which load the CPU more.i
I never really watched it maxing out.

I'll try switching on the ip4csum,tcp4csum,udp4csum,tso4
options on the interfaces tonight and then I try to
graph the CPU load somehow.

Thanks
Chris

[Jason Thorpe]

Post by Steven M. Bellovin
"RED" is an output interface queueing discipline (it stands for, as I
recall, "Random Early Drop", though the altq.conf file says "Random Early
Detection").

I've always viewed this terminology discrepancy this way:

- From a router's point of view, it's Random Early Drop.

- From an endpoint's point of view, it's Random Early Detection (of
congestion).

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by Jason Thorpe

Post by Steven M. Bellovin
"RED" is an output interface queueing discipline (it stands for, as I
recall, "Random Early Drop", though the altq.conf file says "Random Early
Detection").

- From a router's point of view, it's Random Early Drop.
- From an endpoint's point of view, it's Random Early Detection (of
congestion).

Sally consitently uses "detection", and that should be definitive.

http://www.icir.org/floyd/red.html

[Steven M. Bellovin]

Post by Greg Troxel
Sally consitently uses "detection", and that should be definitive.
http://www.icir.org/floyd/red.html

Indeed. (I was offline when I wrote my note, or I'd have checked. But
I've discussed the topic enough with Sally that it should have stuck in my
brain.)

--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jason Thorpe]

Post by Christoph Kaegi
I'll try switching on the ip4csum,tcp4csum,udp4csum,tso4
options on the interfaces tonight and then I try to
graph the CPU load somehow.

ip4csum should certainly help a router. The others may not have so
much of an impact, but they certainly won't hurt.

-- thorpej


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Steven M. Bellovin]

Post by Christoph Kaegi

Post by Steven M. Bellovin
You showed four active wm interfaces in a message. How fast are they
running? I know that some wm interfaces can run at gigabit speeds -- are
you trying to do that?

wm0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (1000baseT full-duplex)
wm1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex)
wm2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex)
wm3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex)
wm0 runs gigabit, this is the inwards interface which should
be able to keep up with traffic from internet an dmz, which
both are 100MBit/s

The question, then, is which interface the dropped packets are coming
from. They all end up in one IP input queue. And if the CPU is that
idle, I'm quite perplexed. Catching it in the act of dropping packets
might be necessary -- watch for CPU load at the time, and also input
packet counts on those four interfaces.


--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Christoph Kaegi]

Post by Steven M. Bellovin
The question, then, is which interface the dropped packets are coming
from. They all end up in one IP input queue. And if the CPU is that
idle, I'm quite perplexed. Catching it in the act of dropping packets
might be necessary -- watch for CPU load at the time, and also input
packet counts on those four interfaces.

OK, I'll try to graph CPU Load and input/output packets now.
For input/output packets I can use netstat -ni, how would I
go about getting CPU load every second?
How should I interpret sysctl kern.cp_time?

Or can you point me to some useful C functions so I could write and
run a programm which outputs the relevant values every second?