netfilter buglog - Dec 2011 - [Bug 765] New: Netfilter crash on bridged/TAP device on 2.6.38 & 3.0 kernels

http://bugzilla.netfilter.org/show_bug.cgi?id=765

           Summary: Netfilter crash on bridged/TAP device on 2.6.38 & 3.0
                    kernels
           Product: netfilter/iptables
           Version: linux-2.6.x
          Platform: x86_64
        OS/Version: Ubuntu
            Status: NEW
          Severity: major
          Priority: P2
         Component: bridging
        AssignedTo: netfilter-buglog at lists.netfilter.org
        ReportedBy: shyam at zadarastorage.com
   Estimated Hours: 0.0


Created an attachment (id=371)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=371)
print-screen when the bug happened

Hi,

We have multiple servers running Ubuntu Natty (linux kernel
2.6.38-8-server/2.6.38-13-server) and Ubuntu Oneiric (linux kernel
3.0.0-13-server). We have setup bridges on different network interfaces

bridge name     bridge id               STP enabled     interfaces
br0             8000.782bcb145314       no              eth0
br101           8000.128e6c9ce59d       no              eth101
                                                        tap02163e0e2dc8
                                                        tap02163e496dfb
                                                        tap02163e4f4369
                                                        tap02163e50bee1
br102           8000.001b21a3bf40       no              eth102
br103           8000.001b21a3bf41       no              eth103
virbr0          8000.000000000000       yes


and have TAP devices setup on eth101 like this:
tap02163e0e2dc8: tap vnet_hdr
tap02163e496dfb: tap vnet_hdr
tap02163e4f4369: tap vnet_hdr
tap02163e50bee1: tap vnet_hdr

KVM VM's are connected through TAP devices to external network.

We periodically hit the following crash in netfilter in all the 3 linux kernel
variants.

[ 5445.359446]  [<ffffffffa0395200>] ?
br_nf_pre_routing_finish_bridge+0x60/0xd0 [bridge]
[ 5445.372671]  [<ffffffffa0396808>] br_nf_pre_routing_finish+0x328/0x360
[bridge]
[ 5445.385984]  [<ffffffffa0396b96>] br_nf_pre_routing+0x356/0x370
[bridge]
[ 5445.393255]  [<ffffffff815165d5>] nf_iterate+0x85/0xc0
[ 5445.400827]  [<ffffffffa038ffd0>] ? br_handle_local_finish+0x50/0x50
[bridge]
[ 5445.408643]  [<ffffffff81516686>] nf_hook_slow+0x76/0x130
[ 5445.416438]  [<ffffffffa038ffd0>] ? br_hanlde_local_finish+0x50/0x50
[bridge]
[ 5445.424202]  [<ffffffffa0390424>] br_handle_frame+0x1b4/0x260 [bridge]
[ 5445.431824]  [<ffffffffa0390270>] ? br_hanlde_frame_finish+0x2a0/0x2a0
[bridge]
[ 5445.446876]  [<ffffffff814e9b8d>] __netif_receive_skb+0x17d/0x540
[ 5445.454815]  [<ffffffff814ea191>] process_backlog+0xb1/0x190
[ 5445.462790]  [<ffffffff814eb2f4>] net_rx_action+0x134/0x290
[ 5445.470445]  [<ffffffff81065ea8>] __do_softirq+0xa8/0x210
[ 5445.478364]  [<ffffffff8160831c>] call_softirq+0x1c/0x30
[ 5445.486145]  <EOI>  [<ffffffff8100c295>] ? do_softirq+0x65/0xa0
[ 5445.494076]  [<ffffffff814eb7d8>] netif_rx_ni+0x28/0x30
[ 5445.501902]  [<ffffffff8144c702>] tun_chr_aio_write+0x342/0x4f0
[ 5445.509771]  [<ffffffff8144c3c0>] ? tun_chr_aio_read+0xd0/0xd0
[ 5445.517864]  [<ffffffff81167db3>] do_sync_readv_writev+0xd3/0x110
[ 5445.526011]  [<ffffffff81281ecc>] ? security_file_permission+0x2c/0xb0
[ 5445.534177]  [<ffffffff811674b1>] ? rw_verify_area+0x61/0xf0
[ 5445.542339]  [<ffffffff8116807d>] do_readv_write_v+0xcd/0x1d0
[ 5445.550227]  [<ffffffff811681bc>] vfs_writev+0x3c/0x50
[ 5445.558337]  [<ffffffff8116831a>] sys_writev+0x4a/0xb0
[ 5445.566288]  [<ffffffff81607102>] system_call_fastpath+0x16/0x1b
[ 5445.574267] ---[ end trace f585024e7045bff8 ]---

I am attaching some print-screen outputs as well. The kernel gets locked up
with Fatal error in interrupt context & not able to get more information.

Any help, guidance is highly appreciated.

Thanks
--Shyam


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765





--- Comment #1 from Shyam <shyam at zadarastorage.com>  2011-12-22
13:32:06 ---
Created an attachment (id=372)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=372)
crash-2


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765





--- Comment #2 from Shyam <shyam at zadarastorage.com>  2011-12-22
13:32:33 ---
Created an attachment (id=373)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=373)
oneiric crash


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765


Shyam <shyam at zadarastorage.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Netfilter crash on          |Netfilter crash on
                   |bridged/TAP device on 2.6.38|bridged/TAP device in
                   |& 3.0 kernels              
|br_nf_pre_routing_finish_bri
                   |                            |dge on 2.6.38 & 3.0 kernels




-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765





--- Comment #3 from Alex Lyakas <alex.bolshoy at gmail.com>  2011-12-25
14:41:09 ---
Created an attachment (id=374)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=374)
Additional crash

Additional crash with  2.6.38-8.42


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765





--- Comment #4 from Alex Lyakas <alex.bolshoy at gmail.com>  2011-12-25
14:43:08 ---
Created an attachment (id=375)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=375)
Additional crash

Additional crash on  2.6.38-8.42


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765

--- Comment #5 from Juerg Haefliger <juergh at gmail.com> 2012-06-25
07:56:14 CEST ---
Created attachment 386
  --> http://bugzilla.netfilter.org/attachment.cgi?id=386
[PATCH] tun: reserves space for network in skb

We're seeing a similar crash using an Ubuntu 2.6.38-8-server kernel:

[120550.767696] BUG: unable to handle kernel paging request at ffff8817daa4fffe 
[120550.841825] IP: [<ffffffffa032a5e0>]
br_nf_pre_routing_finish_bridge+0x60/0xd0 [bridge] 
[120550.987644] PGD 1a04063 PUD 17ddaa3063 PMD 17dd881063 PTE 80000017daa4f161 
[120550.987648] Oops: 0003 [#1] SMP 
[120550.987650] last sysfs file: /sys/devices/virtual/dmi/id/chassis_asset_tag 
[120550.987652] CPU 20 
[120550.987653] Modules linked in: xt_comment act_police cls_u32 sch_ingress
cls_fw sch_htb ebt_arp ebt_ip ipmi_devintf ipmi_si ipmi_msghandle 
r xt_recent xt_multiport ebtable_nat ebtables ipt_MASQUERADE iptable_nat
xt_CHECKSUM iptable_mangle bridge kvm_intel kvm 8021q garp nbd stp ib 
_iser rdma_cm ipt_REJECT ib_cm ipt_LOG iw_cm ib_sa ib_mad ib_core xt_limit
ib_addr xt_tcpudp ipt_addrtype iscsi_tcp libiscsi_tcp libiscsi xt_s 
tate scsi_transport_iscsi vesafb ip6table_filter ip6_tables nf_nat_irc
nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_ 
conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables ghes lp hed
parport i7core_edac serio_raw edac_core xfs exportfs usbhid igb ahci 
hid libahci dca hpsa 
[120550.987682] 
[120550.987685] Pid: 14394, comm: kvm Tainted: G M 2.6.38-8-server #42-Ubuntu
HP SE2170s /SE2170s 
[120550.987688] RIP: 0010:[<ffffffffa032a5e0>] [<ffffffffa032a5e0>]
br_nf_pre_routing_finish_bridge+0x60/0xd0 [bridge] 
[120550.987694] RSP: 0018:ffff88183fd03c28 EFLAGS: 00010246 
[120550.987695] RAX: ffff8817db260c00 RBX: ffff8817de661300 RCX:
ffff8816b692a200 
[120550.987697] RDX: 0000000000000000 RSI: ffff8817db260c58 RDI:
ffff8817de661300 
[120550.987698] RBP: ffff88183fd03c28 R08: ffff8817daa5000e R09:
2f1e403e16020000 
[120550.987700] R10: ffffffff81cfeed0 R11: 00000000b95a9a27 R12:
0000000000000000 
[120550.987701] R13: ffff8817daa5000e R14: ffff880bdbc62000 R15:
ffff880b6446a000 
[120550.987703] FS: 00007ffa6c764700(0000) GS:ffff88183fd00000(0000)
knlGS:0000000000000000 
[120550.987705] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 
[120550.987706] CR2: ffff8817daa4fffe CR3: 0000000a948b3000 CR4:
00000000000026e0 
[120550.987707] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000 
[120550.987709] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400 
[120550.987711] Process kvm (pid: 14394, threadinfo ffff8816c0726000, task
ffff8817c172db80) 
[120550.987712] Stack: 
[120550.987713] ffff88183fd03cc8 ffffffffa032ba20 ffff880b00000001
0000000000000000 
[120550.987715] ffff88183fd03cc8 ffffffff8150f833 ffffffffa032b700
ffff880b80000000 
[120550.987717] ffffc90021111000 ffffc900210a2a80 0000000200000001
ffffffff81ac7ba0 
[120550.987719] Call Trace: 
[120550.987720] <IRQ> 
[120550.987724] [<ffffffffa032ba20>] br_nf_pre_routing_finish+0x320/0x360
[bridge] 
[120550.987729] [<ffffffff8150f833>] ? nf_hook_slow+0xb3/0x120 
[120550.987733] [<ffffffffa032b700>] ? br_nf_pre_routing_finish+0x0/0x360
[bridge] 
[120550.987737] [<ffffffffa032bfc6>] br_nf_pre_routing+0x376/0x390
[bridge]
[120550.987739] [<ffffffff8150f764>] nf_iterate+0x84/0xa0 
[120550.987742] [<ffffffffa0325670>] ? br_handle_frame_finish+0x0/0x260
[bridge] 
[120550.987744] [<ffffffff8150f833>] nf_hook_slow+0xb3/0x120 
[120550.987748] [<ffffffffa0325670>] ? br_handle_frame_finish+0x0/0x260
[bridge] 
[120550.987751] [<ffffffffa0325a65>] br_handle_frame+0x195/0x230 [bridge] 
[120550.987757] [<ffffffff814e24f0>] ? ing_filter.clone.57+0xb0/0x100 
[120550.987760] [<ffffffffa03258d0>] ? br_handle_frame+0x0/0x230 [bridge] 
[120550.987762] [<ffffffff814e27ff>] __netif_receive_skb+0x1bf/0x580 
[120550.987765] [<ffffffff814e301b>] process_backlog+0xab/0x1a0 
[120550.987770] [<ffffffff810137e9>] ? read_tsc+0x9/0x20 
[120550.987772] [<ffffffff814e3f88>] net_rx_action+0x128/0x270 
[120550.987776] [<ffffffff8106d398>] __do_softirq+0xa8/0x1c0 
[120550.987778] [<ffffffff8100cedc>] call_softirq+0x1c/0x30 
[120550.987780] <EOI>

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765

--- Comment #6 from Juerg Haefliger <juergh at gmail.com> 2012-06-25
07:57:00 CEST ---
Comment on attachment 386
  --> http://bugzilla.netfilter.org/attachment.cgi?id=386
[PATCH] tun: reserves space for network in skb

This seems to be the patch that fixes the issue.

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=765

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |pablo at netfilter.org
         Resolution|                            |FIXED

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.
You are watching someone on the CC list of the bug.