I suspect that the new crossbow architecture might impose additional
resource constraints on vlans -- it makes sense that it would create
additional rings and worker threads for each vlan. Anyone from
crossbow-discuss want to comment?
- Garrett
Saurabh Misra wrote:>
> I''m somewhat sure that there is no leak. Anyway I managed to get
crash
> dump (had to reinstall OS with larger dump device).
>
> Findleak output :-
>
> findleaks: thread ffffff014c477740''s stack swapped out; false
> positives possible
> findleaks: thread ffffff014c4768e0''s stack swapped out; false
> positives possible
> findleaks: thread ffffff014c47e1a0''s stack swapped out; false
> positives possible
> CACHE LEAKED BUFCTL CALLER
> ffffff014622b2e0 1 ffffff01506a8510 impl_acc_hdl_alloc+0x34
> ffffff0146226b20 1 ffffff0150fc6298 impl_acc_hdl_alloc+0x4a
> ffffff014622a020 1 ffffff014eff4e58 impl_acc_hdl_alloc+0x64
> ffffff014ed55020 1 ffffff01510db9c8 rootnex_coredma_allochdl+0x5c
> ffffff014622ab20 1 ffffff015098be70 uhci_polled_create_tw+0x2a
> ------------------------------------------------------------------------
> Total 5 buffers, 3232 bytes
> bash-3.2#
> which does not reveal anything...Another findleak output when freemem
> was 0x1
>
> CACHE LEAKED BUFCTL CALLER
> ffffff01462342e0 1 ffffff01677fb1e8 allocb+0x64
> ffffff014eaeeb20 1 ffffff016a595658 cralloc_flags+0x21
> ffffff0146232b20 1 ffffff01697da048 dblk_constructor+0x3b
> ffffff014622b2e0 1 ffffff01506fe660 impl_acc_hdl_alloc+0x34
> ffffff0146226b20 1 ffffff0150a73050 impl_acc_hdl_alloc+0x4a
> ffffff014622a020 1 ffffff014f02ba48 impl_acc_hdl_alloc+0x64
> ffffff014ed87020 1 ffffff01510b1aa8 rootnex_coredma_allochdl+0x5c
> ffffff014622ab20 1 ffffff0150a2d008 uhci_polled_create_tw+0x2a
> ------------------------------------------------------------------------
> Total 8 buffers, 3676 bytes
>
> bash-3.2# grep "THREAD:" t.1 | wc -l
> 24666 // Total number of threads
>
> bash-3.2# grep "THREAD: mac_srs_worker()" t.1 | wc -l
> 8193 // mac_src_worker threads
> bash-3.2#
>
> PC: _resume_from_idle+0xf1 THREAD: mac_srs_worker()
> stack pointer for thread ffffff0004bf0c60: ffffff0004bf0b80
> [ ffffff0004bf0b80 _resume_from_idle+0xf1() ]
> swtch+0x147()
> cv_wait+0x61()
> mac_srs_worker+0x1cb()
> thread_start+8()
>
>
> bash-3.2# grep "THREAD: mac_soft_ring_worker()" t.1 | wc -l
> 12291 // Total number of mac_soft_ring_worker() threads.
>
> PC: _resume_from_idle+0xf1 THREAD: mac_soft_ring_worker()
> stack pointer for thread ffffff0005e47c60: ffffff0005e47b80
> [ ffffff0005e47b80 _resume_from_idle+0xf1() ]
> swtch+0x147()
> cv_wait+0x61()
> mac_soft_ring_worker+0xb0()
> thread_start+8()
>
> I don''t understand why MAC layer has created so many threads. The
vlan
> test did create 4094 vlan''s.
>
> From ::kmausers output, which reports current medium and large users
> of the kmem allocator
>
> bash-3.2# more kma.1
> 83111936 bytes for 20291 allocations with data size 4096:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> vmem_alloc+0x1bc
> segkmem_xalloc+0x94
> segkmem_alloc_vn+0xcd
> segkmem_alloc+0x24
> vmem_xalloc+0x547
> vmem_alloc+0x161
> kmem_slab_create+0x81
> kmem_slab_alloc+0x5b
> kmem_cache_alloc+0x130
> allocb+0x64
> udp_input+0xeee
> ip_fanout_udp_conn+0x2b2
> 33538048 bytes for 4094 allocations with data size 8192:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> kmem_zalloc+0x6a
> mac_flow_tab_create+0x65
> mac_flow_l2tab_create+0x31
> mac_register+0x56f
> vnic_dev_create+0x42b
> vnic_ioc_create+0x157
> drv_ioctl+0x137
> cdev_ioctl+0x45
> spec_ioctl+0x83
> fop_ioctl+0x7b
> ioctl+0x18e
> 28307456 bytes for 6911 allocations with data size 4096:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> vmem_alloc+0x1bc
> segkmem_xalloc+0x94
> segkmem_alloc_vn+0xcd
> segkmem_alloc+0x24
> vmem_xalloc+0x547
> vmem_alloc+0x161
> kmem_slab_create+0x81
> kmem_slab_alloc+0x5b
> kmem_cache_alloc+0x130
> dblk_constructor+0x3b
> kmem_cache_alloc_debug+0x249
> kmem_cache_alloc+0x164
> 27232000 bytes for 106375 allocations with data size 256:
> kmem_cache_alloc_debug+0x283
> kmem_cache_alloc+0x164
> allocb+0x64
> udp_input+0xeee
> ip_fanout_udp_conn+0x2b2
> ip_fanout_udp+0xc72
> ip_wput_local+0x6ce
> ip_multicast_loopback+0x2cb
> udp_xmit+0x4a9
> udp_send_data+0x3b3
> udp_output_v4+0x9c6
> udp_send_not_connected+0xeb
> udp_send+0x246
> so_sendmsg+0x1c7
> socket_sendmsg+0x61
> 27172864 bytes for 6634 allocations with data size 4096:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> vmem_alloc+0x1bc
> segkmem_xalloc+0x94
> segkmem_alloc_vn+0xcd
> segkmem_alloc+0x24
> vmem_xalloc+0x547
> vmem_alloc+0x161
> kmem_slab_create+0x81
> kmem_slab_alloc+0x5b
> kmem_cache_alloc+0x130
> allocb+0x64
> allocb_tmpl+0x24
> copyb+0x77
> [.]
> 324608 bytes for 128 allocations with data size 2536: // bfe comes
> for 128 buffer allocation
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> rootnex_coredma_allochdl+0x84
> rootnex_dma_allochdl+0x7d
> ddi_dma_allochdl+0x35
> ddi_dma_alloc_handle+0xb8
> bfe_ring_buf_alloc+0x4b
> bfe_ring_desc_alloc+0x149
> bfe_rings_alloc+0xa6
> bfe_attach+0x2d0
> devi_attach+0x80
> attach_node+0x95
> i_ndi_config_node+0xa5
> i_ddi_attachchild+0x40
> [.]
>
> Failures were seen on following caches :
>
> > ::kmastat
> cache buf buf buf memory alloc alloc
> name size in use total in use succeed fail
> ------------------------- ------ ------ ------ ---------- --------- -----
> streams_mblk 64 683051 683130 66621440B 39257487 993
> streams_dblk_16 128 1469 1533 299008B 1324225 0
> streams_dblk_80 192 61535 61568 15761408B 7558613 75
> streams_dblk_144 256 279841 279852 95522816B 4124673 1594
> streams_dblk_208 320 592 600 245760B 279677 0
> streams_dblk_272 384 1032 1044 475136B 2666792 71
>
> And all threads stuck in page throttle because of no memory.
>
> Is there a way to control creating number of vlan''s in NICDRV?
>
> I''m thinking of running NICDRV on e1000g and that can tell whether
> it''s driver''s fault or something in MAC layer.
>
> Looks like findleaks does not do good job in finding leaks related to
> allocb or mblk.
>
> cheers,
>
It seems reducing number of vlan IDs resolves the problem and it has been seen before with NICDRV (VR driver VIA Ethernet chip). Clearly thread reaping by MAC layer isn''t happening when the load reduces. While I''m discussing this, I have a comment to make. While running NICDRV in normal mode, I never saw MAC layer passing multiple mblks to my driver even though all the TX descriptors (256) were used. Something which is worth looking. bfe can program the chip to transmit multiple frames (or packets) at once since we can mark start and end of frame in the descriptor table. Do I have to do anything special in the driver? And I wish MII layer was implemented in upper layer because most of the drivers have common code for MII (PHY) layer. Thanks, -- http://saurabhslr.blogspot.com/ Garrett D''Amore wrote:> I suspect that the new crossbow architecture might impose additional > resource constraints on vlans -- it makes sense that it would create > additional rings and worker threads for each vlan. Anyone from > crossbow-discuss want to comment? > > - Garrett > > Saurabh Misra wrote: >> >> I''m somewhat sure that there is no leak. Anyway I managed to get crash >> dump (had to reinstall OS with larger dump device). >> >> Findleak output :- >> >> findleaks: thread ffffff014c477740''s stack swapped out; false >> positives possible >> findleaks: thread ffffff014c4768e0''s stack swapped out; false >> positives possible >> findleaks: thread ffffff014c47e1a0''s stack swapped out; false >> positives possible >> CACHE LEAKED BUFCTL CALLER >> ffffff014622b2e0 1 ffffff01506a8510 impl_acc_hdl_alloc+0x34 >> ffffff0146226b20 1 ffffff0150fc6298 impl_acc_hdl_alloc+0x4a >> ffffff014622a020 1 ffffff014eff4e58 impl_acc_hdl_alloc+0x64 >> ffffff014ed55020 1 ffffff01510db9c8 rootnex_coredma_allochdl+0x5c >> ffffff014622ab20 1 ffffff015098be70 uhci_polled_create_tw+0x2a >> ------------------------------------------------------------------------ >> Total 5 buffers, 3232 bytes >> bash-3.2# which does not reveal anything...Another findleak output >> when freemem was 0x1 >> >> CACHE LEAKED BUFCTL CALLER >> ffffff01462342e0 1 ffffff01677fb1e8 allocb+0x64 >> ffffff014eaeeb20 1 ffffff016a595658 cralloc_flags+0x21 >> ffffff0146232b20 1 ffffff01697da048 dblk_constructor+0x3b >> ffffff014622b2e0 1 ffffff01506fe660 impl_acc_hdl_alloc+0x34 >> ffffff0146226b20 1 ffffff0150a73050 impl_acc_hdl_alloc+0x4a >> ffffff014622a020 1 ffffff014f02ba48 impl_acc_hdl_alloc+0x64 >> ffffff014ed87020 1 ffffff01510b1aa8 rootnex_coredma_allochdl+0x5c >> ffffff014622ab20 1 ffffff0150a2d008 uhci_polled_create_tw+0x2a >> ------------------------------------------------------------------------ >> Total 8 buffers, 3676 bytes >> >> bash-3.2# grep "THREAD:" t.1 | wc -l >> 24666 // Total number of threads >> >> bash-3.2# grep "THREAD: mac_srs_worker()" t.1 | wc -l >> 8193 // mac_src_worker threads >> bash-3.2# >> >> PC: _resume_from_idle+0xf1 THREAD: mac_srs_worker() >> stack pointer for thread ffffff0004bf0c60: ffffff0004bf0b80 >> [ ffffff0004bf0b80 _resume_from_idle+0xf1() ] >> swtch+0x147() >> cv_wait+0x61() >> mac_srs_worker+0x1cb() >> thread_start+8() >> >> >> bash-3.2# grep "THREAD: mac_soft_ring_worker()" t.1 | wc -l >> 12291 // Total number of mac_soft_ring_worker() threads. >> >> PC: _resume_from_idle+0xf1 THREAD: mac_soft_ring_worker() >> stack pointer for thread ffffff0005e47c60: ffffff0005e47b80 >> [ ffffff0005e47b80 _resume_from_idle+0xf1() ] >> swtch+0x147() >> cv_wait+0x61() >> mac_soft_ring_worker+0xb0() >> thread_start+8() >> >> I don''t understand why MAC layer has created so many threads. The vlan >> test did create 4094 vlan''s. >> >> From ::kmausers output, which reports current medium and large users >> of the kmem allocator >> >> bash-3.2# more kma.1 >> 83111936 bytes for 20291 allocations with data size 4096: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> vmem_alloc+0x1bc >> segkmem_xalloc+0x94 >> segkmem_alloc_vn+0xcd >> segkmem_alloc+0x24 >> vmem_xalloc+0x547 >> vmem_alloc+0x161 >> kmem_slab_create+0x81 >> kmem_slab_alloc+0x5b >> kmem_cache_alloc+0x130 >> allocb+0x64 >> udp_input+0xeee >> ip_fanout_udp_conn+0x2b2 >> 33538048 bytes for 4094 allocations with data size 8192: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> kmem_zalloc+0x6a >> mac_flow_tab_create+0x65 >> mac_flow_l2tab_create+0x31 >> mac_register+0x56f >> vnic_dev_create+0x42b >> vnic_ioc_create+0x157 >> drv_ioctl+0x137 >> cdev_ioctl+0x45 >> spec_ioctl+0x83 >> fop_ioctl+0x7b >> ioctl+0x18e >> 28307456 bytes for 6911 allocations with data size 4096: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> vmem_alloc+0x1bc >> segkmem_xalloc+0x94 >> segkmem_alloc_vn+0xcd >> segkmem_alloc+0x24 >> vmem_xalloc+0x547 >> vmem_alloc+0x161 >> kmem_slab_create+0x81 >> kmem_slab_alloc+0x5b >> kmem_cache_alloc+0x130 >> dblk_constructor+0x3b >> kmem_cache_alloc_debug+0x249 >> kmem_cache_alloc+0x164 >> 27232000 bytes for 106375 allocations with data size 256: >> kmem_cache_alloc_debug+0x283 >> kmem_cache_alloc+0x164 >> allocb+0x64 >> udp_input+0xeee >> ip_fanout_udp_conn+0x2b2 >> ip_fanout_udp+0xc72 >> ip_wput_local+0x6ce >> ip_multicast_loopback+0x2cb >> udp_xmit+0x4a9 >> udp_send_data+0x3b3 >> udp_output_v4+0x9c6 >> udp_send_not_connected+0xeb >> udp_send+0x246 >> so_sendmsg+0x1c7 >> socket_sendmsg+0x61 >> 27172864 bytes for 6634 allocations with data size 4096: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> vmem_alloc+0x1bc >> segkmem_xalloc+0x94 >> segkmem_alloc_vn+0xcd >> segkmem_alloc+0x24 >> vmem_xalloc+0x547 >> vmem_alloc+0x161 >> kmem_slab_create+0x81 >> kmem_slab_alloc+0x5b >> kmem_cache_alloc+0x130 >> allocb+0x64 >> allocb_tmpl+0x24 >> copyb+0x77 >> [.] >> 324608 bytes for 128 allocations with data size 2536: // bfe comes >> for 128 buffer allocation >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> rootnex_coredma_allochdl+0x84 >> rootnex_dma_allochdl+0x7d >> ddi_dma_allochdl+0x35 >> ddi_dma_alloc_handle+0xb8 >> bfe_ring_buf_alloc+0x4b >> bfe_ring_desc_alloc+0x149 >> bfe_rings_alloc+0xa6 >> bfe_attach+0x2d0 >> devi_attach+0x80 >> attach_node+0x95 >> i_ndi_config_node+0xa5 >> i_ddi_attachchild+0x40 >> [.] >> >> Failures were seen on following caches : >> >> > ::kmastat >> cache buf buf buf memory alloc alloc >> name size in use total in use succeed fail >> ------------------------- ------ ------ ------ ---------- --------- ----- >> streams_mblk 64 683051 683130 66621440B 39257487 993 >> streams_dblk_16 128 1469 1533 299008B 1324225 0 >> streams_dblk_80 192 61535 61568 15761408B 7558613 75 >> streams_dblk_144 256 279841 279852 95522816B 4124673 1594 >> streams_dblk_208 320 592 600 245760B 279677 0 >> streams_dblk_272 384 1032 1044 475136B 2666792 71 >> >> And all threads stuck in page throttle because of no memory. >> >> Is there a way to control creating number of vlan''s in NICDRV? >> >> I''m thinking of running NICDRV on e1000g and that can tell whether >> it''s driver''s fault or something in MAC layer. >> >> Looks like findleaks does not do good job in finding leaks related to >> allocb or mblk. >> >> cheers, >> >
On 05/03/09 13:40, Garrett D''Amore wrote:> I suspect that the new crossbow architecture might impose additional > resource constraints on vlans -- it makes sense that it would create > additional rings and worker threads for each vlan. Anyone from > crossbow-discuss want to comment?we hope to have support for hardware rings assigned to vlans. Still, beyond the finite number of rings available on the NIC, the dladm create-vnic -v or dladm create-vlan needs to start failing gracefully before reaching a few thousands of kernel threads. CR# R6776630 (vnics/vlans should limit creating more worker threads after a certain threshold ) is tracking this mis-behavior Kais.> > - Garrett > > Saurabh Misra wrote: >> >> I''m somewhat sure that there is no leak. Anyway I managed to get >> crash dump (had to reinstall OS with larger dump device). >> >> Findleak output :- >> >> findleaks: thread ffffff014c477740''s stack swapped out; false >> positives possible >> findleaks: thread ffffff014c4768e0''s stack swapped out; false >> positives possible >> findleaks: thread ffffff014c47e1a0''s stack swapped out; false >> positives possible >> CACHE LEAKED BUFCTL CALLER >> ffffff014622b2e0 1 ffffff01506a8510 impl_acc_hdl_alloc+0x34 >> ffffff0146226b20 1 ffffff0150fc6298 impl_acc_hdl_alloc+0x4a >> ffffff014622a020 1 ffffff014eff4e58 impl_acc_hdl_alloc+0x64 >> ffffff014ed55020 1 ffffff01510db9c8 rootnex_coredma_allochdl+0x5c >> ffffff014622ab20 1 ffffff015098be70 uhci_polled_create_tw+0x2a >> ------------------------------------------------------------------------ >> Total 5 buffers, 3232 bytes >> bash-3.2# which does not reveal anything...Another findleak output >> when freemem was 0x1 >> >> CACHE LEAKED BUFCTL CALLER >> ffffff01462342e0 1 ffffff01677fb1e8 allocb+0x64 >> ffffff014eaeeb20 1 ffffff016a595658 cralloc_flags+0x21 >> ffffff0146232b20 1 ffffff01697da048 dblk_constructor+0x3b >> ffffff014622b2e0 1 ffffff01506fe660 impl_acc_hdl_alloc+0x34 >> ffffff0146226b20 1 ffffff0150a73050 impl_acc_hdl_alloc+0x4a >> ffffff014622a020 1 ffffff014f02ba48 impl_acc_hdl_alloc+0x64 >> ffffff014ed87020 1 ffffff01510b1aa8 rootnex_coredma_allochdl+0x5c >> ffffff014622ab20 1 ffffff0150a2d008 uhci_polled_create_tw+0x2a >> ------------------------------------------------------------------------ >> Total 8 buffers, 3676 bytes >> >> bash-3.2# grep "THREAD:" t.1 | wc -l >> 24666 // Total number of threads >> >> bash-3.2# grep "THREAD: mac_srs_worker()" t.1 | wc -l >> 8193 // mac_src_worker threads >> bash-3.2# >> >> PC: _resume_from_idle+0xf1 THREAD: mac_srs_worker() >> stack pointer for thread ffffff0004bf0c60: ffffff0004bf0b80 >> [ ffffff0004bf0b80 _resume_from_idle+0xf1() ] >> swtch+0x147() >> cv_wait+0x61() >> mac_srs_worker+0x1cb() >> thread_start+8() >> >> >> bash-3.2# grep "THREAD: mac_soft_ring_worker()" t.1 | wc -l >> 12291 // Total number of mac_soft_ring_worker() threads. >> >> PC: _resume_from_idle+0xf1 THREAD: mac_soft_ring_worker() >> stack pointer for thread ffffff0005e47c60: ffffff0005e47b80 >> [ ffffff0005e47b80 _resume_from_idle+0xf1() ] >> swtch+0x147() >> cv_wait+0x61() >> mac_soft_ring_worker+0xb0() >> thread_start+8() >> >> I don''t understand why MAC layer has created so many threads. The >> vlan test did create 4094 vlan''s. >> >> From ::kmausers output, which reports current medium and large users >> of the kmem allocator >> >> bash-3.2# more kma.1 >> 83111936 bytes for 20291 allocations with data size 4096: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> vmem_alloc+0x1bc >> segkmem_xalloc+0x94 >> segkmem_alloc_vn+0xcd >> segkmem_alloc+0x24 >> vmem_xalloc+0x547 >> vmem_alloc+0x161 >> kmem_slab_create+0x81 >> kmem_slab_alloc+0x5b >> kmem_cache_alloc+0x130 >> allocb+0x64 >> udp_input+0xeee >> ip_fanout_udp_conn+0x2b2 >> 33538048 bytes for 4094 allocations with data size 8192: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> kmem_zalloc+0x6a >> mac_flow_tab_create+0x65 >> mac_flow_l2tab_create+0x31 >> mac_register+0x56f >> vnic_dev_create+0x42b >> vnic_ioc_create+0x157 >> drv_ioctl+0x137 >> cdev_ioctl+0x45 >> spec_ioctl+0x83 >> fop_ioctl+0x7b >> ioctl+0x18e >> 28307456 bytes for 6911 allocations with data size 4096: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> vmem_alloc+0x1bc >> segkmem_xalloc+0x94 >> segkmem_alloc_vn+0xcd >> segkmem_alloc+0x24 >> vmem_xalloc+0x547 >> vmem_alloc+0x161 >> kmem_slab_create+0x81 >> kmem_slab_alloc+0x5b >> kmem_cache_alloc+0x130 >> dblk_constructor+0x3b >> kmem_cache_alloc_debug+0x249 >> kmem_cache_alloc+0x164 >> 27232000 bytes for 106375 allocations with data size 256: >> kmem_cache_alloc_debug+0x283 >> kmem_cache_alloc+0x164 >> allocb+0x64 >> udp_input+0xeee >> ip_fanout_udp_conn+0x2b2 >> ip_fanout_udp+0xc72 >> ip_wput_local+0x6ce >> ip_multicast_loopback+0x2cb >> udp_xmit+0x4a9 >> udp_send_data+0x3b3 >> udp_output_v4+0x9c6 >> udp_send_not_connected+0xeb >> udp_send+0x246 >> so_sendmsg+0x1c7 >> socket_sendmsg+0x61 >> 27172864 bytes for 6634 allocations with data size 4096: >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> vmem_alloc+0x1bc >> segkmem_xalloc+0x94 >> segkmem_alloc_vn+0xcd >> segkmem_alloc+0x24 >> vmem_xalloc+0x547 >> vmem_alloc+0x161 >> kmem_slab_create+0x81 >> kmem_slab_alloc+0x5b >> kmem_cache_alloc+0x130 >> allocb+0x64 >> allocb_tmpl+0x24 >> copyb+0x77 >> [.] >> 324608 bytes for 128 allocations with data size 2536: // bfe comes >> for 128 buffer allocation >> kmem_slab_alloc_impl+0x116 >> kmem_slab_alloc+0xa1 >> kmem_cache_alloc+0x130 >> rootnex_coredma_allochdl+0x84 >> rootnex_dma_allochdl+0x7d >> ddi_dma_allochdl+0x35 >> ddi_dma_alloc_handle+0xb8 >> bfe_ring_buf_alloc+0x4b >> bfe_ring_desc_alloc+0x149 >> bfe_rings_alloc+0xa6 >> bfe_attach+0x2d0 >> devi_attach+0x80 >> attach_node+0x95 >> i_ndi_config_node+0xa5 >> i_ddi_attachchild+0x40 >> [.] >> >> Failures were seen on following caches : >> >> > ::kmastat >> cache buf buf buf memory alloc alloc >> name size in use total in use succeed fail >> ------------------------- ------ ------ ------ ---------- --------- >> ----- >> streams_mblk 64 683051 683130 66621440B 39257487 >> 993 >> streams_dblk_16 128 1469 1533 299008B >> 1324225 0 >> streams_dblk_80 192 61535 61568 15761408B >> 7558613 75 >> streams_dblk_144 256 279841 279852 95522816B 4124673 >> 1594 >> streams_dblk_208 320 592 600 245760B >> 279677 0 >> streams_dblk_272 384 1032 1044 475136B >> 2666792 71 >> >> And all threads stuck in page throttle because of no memory. >> >> Is there a way to control creating number of vlan''s in NICDRV? >> >> I''m thinking of running NICDRV on e1000g and that can tell whether >> it''s driver''s fault or something in MAC layer. >> >> Looks like findleaks does not do good job in finding leaks related to >> allocb or mblk. >> >> cheers, >> > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
Saurabh Misra wrote:> > It seems reducing number of vlan IDs resolves the problem and it has > been seen before with NICDRV (VR driver VIA Ethernet chip). Clearly > thread reaping by MAC layer isn''t happening when the load reduces. > > While I''m discussing this, I have a comment to make. While running > NICDRV in normal mode, I never saw MAC layer passing multiple mblks to > my driver even though all the TX descriptors (256) were used. > Something which is worth looking. bfe can program the chip to transmit > multiple frames (or packets) at once since we can mark start and end > of frame in the descriptor table. Do I have to do anything special in > the driver? > > And I wish MII layer was implemented in upper layer because most of > the drivers have common code for MII (PHY) layer.We''ve been talking about this for some time now. Expect a project to deal with this soon. -- Garrett> > Thanks,
Saurabh Misra wrote:> > It seems reducing number of vlan IDs resolves the problem and it has > been seen before with NICDRV (VR driver VIA Ethernet chip). Clearly > thread reaping by MAC layer isn''t happening when the load reduces. > > While I''m discussing this, I have a comment to make. While running > NICDRV in normal mode, I never saw MAC layer passing multiple mblks to > my driver even though all the TX descriptors (256) were used. Something > which is worth looking. bfe can program the chip to transmit multiple > frames (or packets) at once since we can mark start and end of frame in > the descriptor table. Do I have to do anything special in the driver?Nothing wrong with the driver. It is the MAC layer that is sending one packet at a time. Even if this is fixed in MAC layer, it may not benefit unless upper layers (IP) send chains. -krgopi> > And I wish MII layer was implemented in upper layer because most of the > drivers have common code for MII (PHY) layer. > > Thanks,--
* Rajagopal.Kunhappan at Sun.COM [2009-05-05 18:39:40]> Saurabh Misra wrote: >> >> It seems reducing number of vlan IDs resolves the problem and it has >> been seen before with NICDRV (VR driver VIA Ethernet chip). Clearly >> thread reaping by MAC layer isn''t happening when the load reduces. >> >> While I''m discussing this, I have a comment to make. While running >> NICDRV in normal mode, I never saw MAC layer passing multiple mblks >> to my driver even though all the TX descriptors (256) were >> used. Something which is worth looking. bfe can program the chip to >> transmit multiple frames (or packets) at once since we can mark >> start and end of frame in the descriptor table. Do I have to do >> anything special in the driver? > > Nothing wrong with the driver. It is the MAC layer that is sending one > packet at a time. Even if this is fixed in MAC layer, it may not > benefit unless upper layers (IP) send chains.The Xen network backend driver will pass chains of packets to the downstream MAC client (usually a VNIC). dme. -- David Edmondson, Sun Microsystems, http://dme.org
David Edmondson wrote:> * Rajagopal.Kunhappan at Sun.COM [2009-05-05 18:39:40] >> Saurabh Misra wrote: >>> It seems reducing number of vlan IDs resolves the problem and it has >>> been seen before with NICDRV (VR driver VIA Ethernet chip). Clearly >>> thread reaping by MAC layer isn''t happening when the load reduces. >>> >>> While I''m discussing this, I have a comment to make. While running >>> NICDRV in normal mode, I never saw MAC layer passing multiple mblks >>> to my driver even though all the TX descriptors (256) were >>> used. Something which is worth looking. bfe can program the chip to >>> transmit multiple frames (or packets) at once since we can mark >>> start and end of frame in the descriptor table. Do I have to do >>> anything special in the driver? >> Nothing wrong with the driver. It is the MAC layer that is sending one >> packet at a time. Even if this is fixed in MAC layer, it may not >> benefit unless upper layers (IP) send chains. > > The Xen network backend driver will pass chains of packets to the > downstream MAC client (usually a VNIC).When you have a VNIC, then there is the existence of virtual switch in MAC. Each packet is peeked into to see if the packet has to be sent to another VNIC on the virtual switch. If the packet is not for VNIC, it is sent to the driver. But we can look into building chains before sending packets to VNIC or the driver. -krgopi
* Rajagopal.Kunhappan at Sun.COM [2009-05-05 19:25:30]>> The Xen network backend driver will pass chains of packets to the >> downstream MAC client (usually a VNIC). > > When you have a VNIC, then there is the existence of virtual switch in > MAC. Each packet is peeked into to see if the packet has to be sent to > another VNIC on the virtual switch. If the packet is not for VNIC, it > is sent to the driver. But we can look into building chains before > sending packets to VNIC or the driver.You can run the Xen backend driver directly over a physical NIC as well, but this is not a common configuration. dme. -- David Edmondson, Sun Microsystems, http://dme.org