thr3ads.net - freebsd stable - Bogus "igb1: Could not setup receive structures" in 8-STABLE [Oct 2010]

If this information is useful, please help other people find it:
Share via:

Terry Kennedy

2010-Oct-14 23:24 UTC

Bogus "igb1: Could not setup receive structures" in 8-STABLE

I've run across a strange problem with the igb driver in 8-STABLE - when
I try to do anything with the second igb interface, I get one or more
"igb1:
Could not setup receive structures" error messages.

  This can be reproduced as simply as booting in single-user mode with an 
empty /boot/loader.conf and doing:

	ifconfig igb0 up
	ifconfig igb1 up

  I've tried to track this down, and as far as I can see, this is from some
change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE
(igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I
can bring up both interfaces. When I try it with an 8-STABLE kernel, I get
the error and igb1 is missing the "RUNNING" flag:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
        ether 00:25:90:xx:xx:bc
        inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1 
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet 1000baseT <full-duplex>
        status: active
igb1: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 9000
       
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
        ether 00:25:90:xx:xx:bd
        inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid 0x2 
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet 1000baseT <full-duplex>
        status: active

  I see 3 mbuf_jumbo_page allocation failures from:

(1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES

64 Bucket:                536,        0,      263,        3,      263,      106
128 Bucket:              1048,        0,      523,        2,      525,      139
mbuf_jumbo_page:         4096,    12800,    12307,      493,    30343,        3

  which correspond to the 3 "igb1: Could not setup receive structures"
mess-
ages. If I try another "ifconfig igb1 up", I get another console
message and
the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger
value, like 15000, I get the same error message when trying to work with the
igb1 device, so I don't think it is a "real" error but indicates a
problem
in the driver.

  This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board
82576. The dev.igb sysctl's for the two ports (excluding 0 values) are at-
tached. Note that most of the igb1 values are zero:

(0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ": 0"
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1
dev.igb.1.%driver: igb
dev.igb.1.%location: slot=0 function=1
dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9
subdevice=0x0400 class=0x020000
dev.igb.1.%parent: pci1
dev.igb.1.nvm: -1
dev.igb.1.flow_control: 3
dev.igb.1.enable_aim: 1
dev.igb.1.rx_processing_limit: 100
dev.igb.1.link_irq: 1
dev.igb.1.device_control: 13632065
dev.igb.1.extended_int_mask: 2147483648
dev.igb.1.fc_high_water: 47488
dev.igb.1.fc_low_water: 47472

(0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ": 0"
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1
dev.igb.0.%driver: igb
dev.igb.0.%location: slot=0 function=0
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9
subdevice=0x0400 class=0x020000
dev.igb.0.%parent: pci1
dev.igb.0.nvm: -1
dev.igb.0.flow_control: 3
dev.igb.0.enable_aim: 1
dev.igb.0.rx_processing_limit: 100
dev.igb.0.link_irq: 4
dev.igb.0.device_control: 1087373889
dev.igb.0.rx_control: 67338274
dev.igb.0.interrupt_mask: 4
dev.igb.0.extended_int_mask: 2147484671
dev.igb.0.fc_high_water: 47488
dev.igb.0.fc_low_water: 47472
dev.igb.0.queue0.txd_head: 823
dev.igb.0.queue0.txd_tail: 823
dev.igb.0.queue0.tx_packets: 1402
dev.igb.0.queue0.rxd_head: 319
dev.igb.0.queue0.rxd_tail: 318
dev.igb.0.queue0.rx_packets: 319
dev.igb.0.queue0.rx_bytes: 69075
dev.igb.0.queue1.txd_head: 2
dev.igb.0.queue1.txd_tail: 2
dev.igb.0.queue1.tx_packets: 1
dev.igb.0.queue1.rxd_head: 52
dev.igb.0.queue1.rxd_tail: 51
dev.igb.0.queue1.rx_packets: 52
dev.igb.0.queue1.rx_bytes: 6369
dev.igb.0.queue2.txd_head: 27
dev.igb.0.queue2.txd_tail: 27
dev.igb.0.queue2.tx_packets: 13
dev.igb.0.queue2.rxd_head: 72
dev.igb.0.queue2.rxd_tail: 71
dev.igb.0.queue2.rx_packets: 72
dev.igb.0.queue2.rx_bytes: 8789
dev.igb.0.queue3.txd_head: 177
dev.igb.0.queue3.txd_tail: 177
dev.igb.0.queue3.tx_packets: 64
dev.igb.0.queue3.rxd_head: 88
dev.igb.0.queue3.rxd_tail: 87
dev.igb.0.queue3.rx_packets: 88
dev.igb.0.queue3.rx_bytes: 16454
dev.igb.0.queue4.rxd_head: 21
dev.igb.0.queue4.rxd_tail: 20
dev.igb.0.queue4.rx_packets: 21
dev.igb.0.queue4.rx_bytes: 3547
dev.igb.0.queue5.txd_head: 19
dev.igb.0.queue5.txd_tail: 19
dev.igb.0.queue5.tx_packets: 9
dev.igb.0.queue5.rxd_head: 120
dev.igb.0.queue5.rxd_tail: 119
dev.igb.0.queue5.rx_packets: 120
dev.igb.0.queue5.rx_bytes: 35518
dev.igb.0.queue6.txd_head: 21
dev.igb.0.queue6.txd_tail: 21
dev.igb.0.queue6.tx_packets: 10
dev.igb.0.queue6.rxd_head: 21
dev.igb.0.queue6.rxd_tail: 20
dev.igb.0.queue6.rx_packets: 21
dev.igb.0.queue6.rx_bytes: 2876
dev.igb.0.queue7.txd_head: 100
dev.igb.0.queue7.txd_tail: 100
dev.igb.0.queue7.tx_packets: 50
dev.igb.0.queue7.rxd_head: 74
dev.igb.0.queue7.rxd_tail: 73
dev.igb.0.queue7.rx_packets: 1098
dev.igb.0.queue7.rx_bytes: 116918
dev.igb.0.queue8.txd_head: 11
dev.igb.0.queue8.txd_tail: 11
dev.igb.0.queue8.tx_packets: 6
dev.igb.0.queue8.rxd_head: 25
dev.igb.0.queue8.rxd_tail: 24
dev.igb.0.queue8.rx_packets: 25
dev.igb.0.queue8.rx_bytes: 3698
dev.igb.0.mac_stats.total_pkts_recvd: 3138
dev.igb.0.mac_stats.good_pkts_recvd: 1815
dev.igb.0.mac_stats.bcast_pkts_recvd: 383
dev.igb.0.mac_stats.mcast_pkts_recvd: 114
dev.igb.0.mac_stats.rx_frames_64: 217
dev.igb.0.mac_stats.rx_frames_65_127: 673
dev.igb.0.mac_stats.rx_frames_128_255: 779
dev.igb.0.mac_stats.rx_frames_256_511: 61
dev.igb.0.mac_stats.rx_frames_512_1023: 46
dev.igb.0.mac_stats.rx_frames_1024_1522: 39
dev.igb.0.mac_stats.good_octets_recvd: 270378
dev.igb.0.mac_stats.good_octets_txd: 252216
dev.igb.0.mac_stats.total_pkts_txd: 1554
dev.igb.0.mac_stats.good_pkts_txd: 1554
dev.igb.0.mac_stats.bcast_pkts_txd: 36
dev.igb.0.mac_stats.mcast_pkts_txd: 65
dev.igb.0.mac_stats.tx_frames_64: 32
dev.igb.0.mac_stats.tx_frames_65_127: 219
dev.igb.0.mac_stats.tx_frames_128_255: 1222
dev.igb.0.mac_stats.tx_frames_256_511: 55
dev.igb.0.mac_stats.tx_frames_512_1023: 19
dev.igb.0.mac_stats.tx_frames_1024_1522: 7
dev.igb.0.interrupts.asserts: 3350
dev.igb.0.interrupts.rx_pkt_timer: 1815
dev.igb.0.interrupts.tx_abs_timer: 1815
dev.igb.0.interrupts.tx_queue_empty: 1554
dev.igb.0.host.rx_good_bytes: 270378
dev.igb.0.host.tx_good_bytes: 252216

  Both ports are cabled to the same Cisco switch. If I swap the cables at
the FreeBSD end between igb0 and igb1, the problem stays with igb1 so I
don't think it is the switch.

  I have 3 identical systems, all of which exhibit this same issue. Un-
fortunately, I don't have any other hardware with dual igb's.

  I can give a developer root access as well as a web-based remote console
if needed to track this down.

        Terry Kennedy             http://www.tmk.com
        terry@tmk.com             New York, NY USA

Jack Vogel

2010-Oct-14 23:36 UTC

head link

Bogus "igb1: Could not setup receive structures" in 8-STABLE

The problem is mbuf resources, the driver is autoconfiguring the number of
queues based on the number of cores, on newer systems with lots of them
this is outstripping the mbuf resource pool.

I have decided to hard limit the queues to 8, you can fix the number
manually
by searching for num_queues in if_igb.c and setting it to something other
than
0 for now.

I am at work on a number of issues with igb and em right now which is why
there has not been an MFC yet.

Questions to me,

Jack


On Thu, Oct 14, 2010 at 3:26 PM, Terry Kennedy <TERRY@tmk.com> wrote:
>  I've run across a strange problem with the igb driver in 8-STABLE -
when
> I try to do anything with the second igb interface, I get one or more
> "igb1:
> Could not setup receive structures" error messages.
>
>  This can be reproduced as simply as booting in single-user mode with an
> empty /boot/loader.conf and doing:
>
>        ifconfig igb0 up
>        ifconfig igb1 up
>
>  I've tried to track this down, and as far as I can see, this is from
some
> change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE
> (igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I
> can bring up both interfaces. When I try it with an 8-STABLE kernel, I get
> the error and igb1 is missing the "RUNNING" flag:
>
> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
9000
>
> 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
>        ether 00:25:90:xx:xx:bc
>        inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1
>        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
>        media: Ethernet 1000baseT <full-duplex>
>        status: active
> igb1: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 9000
>
> 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
>        ether 00:25:90:xx:xx:bd
>        inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid
> 0x2
>        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
>        media: Ethernet 1000baseT <full-duplex>
>        status: active
>
>  I see 3 mbuf_jumbo_page allocation failures from:
>
> (1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$
> ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
>  FAILURES
>
> 64 Bucket:                536,        0,      263,        3,      263,
>  106
> 128 Bucket:              1048,        0,      523,        2,      525,
>  139
> mbuf_jumbo_page:         4096,    12800,    12307,      493,    30343,
>    3
>
>  which correspond to the 3 "igb1: Could not setup receive
structures" mess-
> ages. If I try another "ifconfig igb1 up", I get another console
message
> and
> the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger
> value, like 15000, I get the same error message when trying to work with
> the
> igb1 device, so I don't think it is a "real" error but
indicates a problem
> in the driver.
>
>  This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board
> 82576. The dev.igb sysctl's for the two ports (excluding 0 values) are
at-
> tached. Note that most of the igb1 values are zero:
>
> (0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ":
0"
> dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1
> dev.igb.1.%driver: igb
> dev.igb.1.%location: slot=0 function=1
> dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9
> subdevice=0x0400 class=0x020000
> dev.igb.1.%parent: pci1
> dev.igb.1.nvm: -1
> dev.igb.1.flow_control: 3
> dev.igb.1.enable_aim: 1
> dev.igb.1.rx_processing_limit: 100
> dev.igb.1.link_irq: 1
> dev.igb.1.device_control: 13632065
> dev.igb.1.extended_int_mask: 2147483648
> dev.igb.1.fc_high_water: 47488
> dev.igb.1.fc_low_water: 47472
>
> (0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ":
0"
> dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1
> dev.igb.0.%driver: igb
> dev.igb.0.%location: slot=0 function=0
> dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9
> subdevice=0x0400 class=0x020000
> dev.igb.0.%parent: pci1
> dev.igb.0.nvm: -1
> dev.igb.0.flow_control: 3
> dev.igb.0.enable_aim: 1
> dev.igb.0.rx_processing_limit: 100
> dev.igb.0.link_irq: 4
> dev.igb.0.device_control: 1087373889
> dev.igb.0.rx_control: 67338274
> dev.igb.0.interrupt_mask: 4
> dev.igb.0.extended_int_mask: 2147484671
> dev.igb.0.fc_high_water: 47488
> dev.igb.0.fc_low_water: 47472
> dev.igb.0.queue0.txd_head: 823
> dev.igb.0.queue0.txd_tail: 823
> dev.igb.0.queue0.tx_packets: 1402
> dev.igb.0.queue0.rxd_head: 319
> dev.igb.0.queue0.rxd_tail: 318
> dev.igb.0.queue0.rx_packets: 319
> dev.igb.0.queue0.rx_bytes: 69075
> dev.igb.0.queue1.txd_head: 2
> dev.igb.0.queue1.txd_tail: 2
> dev.igb.0.queue1.tx_packets: 1
> dev.igb.0.queue1.rxd_head: 52
> dev.igb.0.queue1.rxd_tail: 51
> dev.igb.0.queue1.rx_packets: 52
> dev.igb.0.queue1.rx_bytes: 6369
> dev.igb.0.queue2.txd_head: 27
> dev.igb.0.queue2.txd_tail: 27
> dev.igb.0.queue2.tx_packets: 13
> dev.igb.0.queue2.rxd_head: 72
> dev.igb.0.queue2.rxd_tail: 71
> dev.igb.0.queue2.rx_packets: 72
> dev.igb.0.queue2.rx_bytes: 8789
> dev.igb.0.queue3.txd_head: 177
> dev.igb.0.queue3.txd_tail: 177
> dev.igb.0.queue3.tx_packets: 64
> dev.igb.0.queue3.rxd_head: 88
> dev.igb.0.queue3.rxd_tail: 87
> dev.igb.0.queue3.rx_packets: 88
> dev.igb.0.queue3.rx_bytes: 16454
> dev.igb.0.queue4.rxd_head: 21
> dev.igb.0.queue4.rxd_tail: 20
> dev.igb.0.queue4.rx_packets: 21
> dev.igb.0.queue4.rx_bytes: 3547
> dev.igb.0.queue5.txd_head: 19
> dev.igb.0.queue5.txd_tail: 19
> dev.igb.0.queue5.tx_packets: 9
> dev.igb.0.queue5.rxd_head: 120
> dev.igb.0.queue5.rxd_tail: 119
> dev.igb.0.queue5.rx_packets: 120
> dev.igb.0.queue5.rx_bytes: 35518
> dev.igb.0.queue6.txd_head: 21
> dev.igb.0.queue6.txd_tail: 21
> dev.igb.0.queue6.tx_packets: 10
> dev.igb.0.queue6.rxd_head: 21
> dev.igb.0.queue6.rxd_tail: 20
> dev.igb.0.queue6.rx_packets: 21
> dev.igb.0.queue6.rx_bytes: 2876
> dev.igb.0.queue7.txd_head: 100
> dev.igb.0.queue7.txd_tail: 100
> dev.igb.0.queue7.tx_packets: 50
> dev.igb.0.queue7.rxd_head: 74
> dev.igb.0.queue7.rxd_tail: 73
> dev.igb.0.queue7.rx_packets: 1098
> dev.igb.0.queue7.rx_bytes: 116918
> dev.igb.0.queue8.txd_head: 11
> dev.igb.0.queue8.txd_tail: 11
> dev.igb.0.queue8.tx_packets: 6
> dev.igb.0.queue8.rxd_head: 25
> dev.igb.0.queue8.rxd_tail: 24
> dev.igb.0.queue8.rx_packets: 25
> dev.igb.0.queue8.rx_bytes: 3698
> dev.igb.0.mac_stats.total_pkts_recvd: 3138
> dev.igb.0.mac_stats.good_pkts_recvd: 1815
> dev.igb.0.mac_stats.bcast_pkts_recvd: 383
> dev.igb.0.mac_stats.mcast_pkts_recvd: 114
> dev.igb.0.mac_stats.rx_frames_64: 217
> dev.igb.0.mac_stats.rx_frames_65_127: 673
> dev.igb.0.mac_stats.rx_frames_128_255: 779
> dev.igb.0.mac_stats.rx_frames_256_511: 61
> dev.igb.0.mac_stats.rx_frames_512_1023: 46
> dev.igb.0.mac_stats.rx_frames_1024_1522: 39
> dev.igb.0.mac_stats.good_octets_recvd: 270378
> dev.igb.0.mac_stats.good_octets_txd: 252216
> dev.igb.0.mac_stats.total_pkts_txd: 1554
> dev.igb.0.mac_stats.good_pkts_txd: 1554
> dev.igb.0.mac_stats.bcast_pkts_txd: 36
> dev.igb.0.mac_stats.mcast_pkts_txd: 65
> dev.igb.0.mac_stats.tx_frames_64: 32
> dev.igb.0.mac_stats.tx_frames_65_127: 219
> dev.igb.0.mac_stats.tx_frames_128_255: 1222
> dev.igb.0.mac_stats.tx_frames_256_511: 55
> dev.igb.0.mac_stats.tx_frames_512_1023: 19
> dev.igb.0.mac_stats.tx_frames_1024_1522: 7
> dev.igb.0.interrupts.asserts: 3350
> dev.igb.0.interrupts.rx_pkt_timer: 1815
> dev.igb.0.interrupts.tx_abs_timer: 1815
> dev.igb.0.interrupts.tx_queue_empty: 1554
> dev.igb.0.host.rx_good_bytes: 270378
> dev.igb.0.host.tx_good_bytes: 252216
>
>  Both ports are cabled to the same Cisco switch. If I swap the cables at
> the FreeBSD end between igb0 and igb1, the problem stays with igb1 so I
> don't think it is the switch.
>
>  I have 3 identical systems, all of which exhibit this same issue. Un-
> fortunately, I don't have any other hardware with dual igb's.
>
>  I can give a developer root access as well as a web-based remote console
> if needed to track this down.
>
>        Terry Kennedy             http://www.tmk.com
>        terry@tmk.com             New York, NY USA
>

Terry Kennedy

2010-Oct-15 02:27 UTC

head link

Bogus "igb1: Could not setup receive structures" in 8-STABLE

> The problem is mbuf resources, the driver is autoconfiguring the number of
> queues based on the number of cores, on newer systems with lots of them
> this is outstripping the mbuf resource pool.
  That would make sense, as these systems have 16 cores (dual E5520's).
> I have decided to hard limit the queues to 8, you can fix the number
> manually
> by searching for num_queues in if_igb.c and setting it to something other
> than
> 0 for now.
  I changed it to 8, and saw the same problem. I noted that the igb boot
messages changed from:

Oct 14 18:28:02 rz1m kernel: igb0: Using MSIX interrupts with 10 vectors
Oct 14 18:28:02 rz1m kernel: igb1: Using MSIX interrupts with 10 vectors

  to:

Oct 14 21:53:44 rz1m kernel: igb0: Using MSIX interrupts with 9 vectors
Oct 14 21:53:44 rz1m kernel: igb1: Using MSIX interrupts with 9 vectors

  So I dropped the value to 3 (on the assumption that the system uses one
more than the specified value per interface), and got:

igb0: Using MSIX interrupts with 4 vectors
igb1: Using MSIX interrupts with 4 vectors

  and both igb interfaces came up. I didn't try to find the maximum
number of queues that would work.
> I am at work on a number of issues with igb and em right now which is why
> there has not been an MFC yet.
  Understood. Thanks for the quick response and workaround.

        Terry Kennedy             http://www.tmk.com
        terry@tmk.com             New York, NY USA

freebsd stable - Oct 2010 - Bogus "igb1: Could not setup receive structures" in 8-STABLE

Bogus "igb1: Could not setup receive structures" in 8-STABLE

Bogus "igb1: Could not setup receive structures" in 8-STABLE

Bogus "igb1: Could not setup receive structures" in 8-STABLE