bugzilla-daemon at bugzilla.netfilter.org
2011-Jun-28 10:59 UTC
[Bug 726] New: Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Summary: Oops in nf_conntrack.
Product: netfilter/iptables
Version: linux-2.6.x
Platform: x86_64
OS/Version: Ubuntu
Status: NEW
Severity: critical
Priority: P5
Component: ip_conntrack
AssignedTo: netfilter-buglog at lists.netfilter.org
ReportedBy: jakahudoklin at gmail.com
Estimated Hours: 0.0
Created an attachment (id=355)
--> (http://bugzilla.netfilter.org/attachment.cgi?id=355)
iptables config
I found bug in nf_conntrack, when cleaning up conenctions. It is highly
reproducaple with following setup:
kernel: Ubuntu 11.04(natty) 2.6.38-8-server
ifconfig: attached below
iptables: -t nat -A POSTROUTING -o eth0 -s 192.168.3.0/24 -j MASQUERADE
ipv4_coontrack turned on
Steps to reproduce(how i was able to reproduce, don't believe it is related
to
lxc, because of kernel crash dump):
1. Create lxc container with template of your choice with ip in a network of
br0(bridge), of course also assign br0 its own ip.
2. Start lxc container with lxc-start -n name_of_container.
3. Connect to lxc container using ssh.
4. Stop lxc container with lxc-stop -n name_of_container while keeping ssh
connection open.
5. Ooops
Kernel crash dump:
[ 619.840155] BUG: unable to handle kernel NULL pointer dereference at
0000000000000274
[ 619.844513] IP: [<ffffffff8150aa99>] netlink_has_listeners+0x9/0x50
[ 619.846648] PGD 0
[ 619.848773] Oops: 0000 [#1] SMP
[ 619.850114] last sysfs file:
/sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2:1.0/ieee80211/phy0/rfkill0/uevent
[ 619.850114] CPU 0
[ 619.850114] Modules linked in: ipt_LOG ipt_MASQUERADE xt_state
iptable_filter nf_nat_amanda nf_nat_h323 nf_nat_proto_udplite nf_nat_irc
nf_nat_tftp nf_nat_snmp_basic nf_nat_ftp nf_nat_proto_sctp libcrc32c
nf_nat_proto_dccp iptable_nat ip_tables nf_nat_pptp nf_nat_proto_gre nf_nat_sip
nf_nat ebt_dnat ebtable_nat ebtables ebt_snat act_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack_sane nf_conntrack_netlink nfnetlink
nf_conntrack_irc nf_conntrack_h323 ts_kmp nf_conntrack_amanda
nf_conntrack_proto_dccp nf_conntrack_proto_udplite nf_conntrack_pptp
nf_conntrack_tftp nf_conntrack_proto_gre nf_conntrack_proto_sctp
nf_conntrack_netbios_ns xt_conntrack x_tables nf_conntrack_ftp nf_conntrack_sip
nf_conntrack binfmt_misc veth vmnet vmblock vsock vmci vmmon nfsd parport_pc
exportfs ppdev nfs joydev bridge lockd stp fscache nfs_acl snd_hda_codec_hdmi
auth_rpcgss snd_hda_codec_realtek arc4 sunrpc snd_hda_intel snd_hda_codec
snd_hwdep snd_pcm rtl8187 snd_seq_midi i915 snd_rawmidi snd_seq_midi_event
snd_seq mac80211 snd_timer snd_seq_device cfg80211 uvcvideo drm_kms_helper snd
drm videodev soundcore vhba v4l2_compat_ioctl32 eeprom_93cx6 snd_page_alloc
psmouse i2c_algo_bit serio_raw sparse_keymap lp video parport r8169
[ 619.850114]
[ 619.850114] Pid: 5, comm: kworker/u:0 Not tainted 2.6.38-8-server #42-Ubuntu
TOSHIBA Satellite L500/KSWAA
[ 619.850114] RIP: 0010:[<ffffffff8150aa99>] [<ffffffff8150aa99>]
netlink_has_listeners+0x9/0x50
[ 619.850114] RSP: 0018:ffff880137907bf0 EFLAGS: 00010246
[ 619.850114] RAX: ffff88009dce0000 RBX: ffff8801075c5000 RCX:
000000000000ffff
[ 619.850114] RDX: 000000000000000e RSI: 0000000000000003 RDI:
0000000000000000
[ 619.850114] RBP: ffff880137907bf0 R08: ffff880137906000 R09:
0000000000000001
[ 619.850114] R10: 0000000000000000 R11: dead000000100100 R12:
ffff880137907cb0
[ 619.850114] R13: ffff8801075c5000 R14: 0000000000000000 R15:
0000000000000004
[ 619.850114] FS: 0000000000000000(0000) GS:ffff8800b5800000(0000)
knlGS:0000000000000000
[ 619.850114] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 619.850114] CR2: 0000000000000274 CR3: 00000000b0603000 CR4:
00000000000406f0
[ 619.850114] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 619.850114] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 619.850114] Process kworker/u:0 (pid: 5, threadinfo ffff880137906000, task
ffff8801378edb80)
[ 619.850114] Stack:
[ 619.850114] ffff880137907c00 ffffffffa0523155 ffff880137907c90
ffffffffa05328eb
[ 619.850114] 0000000000000282 ffffc90011100000 ffffc900110fffff
00000000093ca5c3
[ 619.850114] ffff88009dce0000 00000003ffffffff 0000000000000004
ffff880100000002
[ 619.850114] Call Trace:
[ 619.850114] [<ffffffffa0523155>] nfnetlink_has_listeners+0x15/0x20
[nfnetlink]
[ 619.850114] [<ffffffffa05328eb>] ctnetlink_conntrack_event+0x67b/0x890
[nf_conntrack_netlink]
[ 619.850114] [<ffffffff81038c79>] ? default_spin_lock_flags+0x9/0x10
[ 619.850114] [<ffffffff814dd830>] ? cleanup_net+0x0/0x1d0
[ 619.850114] [<ffffffffa04a6150>] death_by_timeout+0xb0/0x170
[nf_conntrack]
[ 619.850114] [<ffffffffa04a5180>] ? kill_all+0x0/0x10 [nf_conntrack]
[ 619.850114] [<ffffffff814dd830>] ? cleanup_net+0x0/0x1d0
[ 619.850114] [<ffffffffa04a6288>] nf_ct_iterate_cleanup+0x78/0x90
[nf_conntrack]
[ 619.850114] [<ffffffffa04a62d9>] nf_conntrack_cleanup_net+0x39/0x110
[nf_conntrack]
[ 619.850114] [<ffffffffa04a7f37>] nf_conntrack_cleanup+0x27/0x60
[nf_conntrack]
[ 619.850114] [<ffffffffa04a822a>] nf_conntrack_net_exit+0x4a/0x70
[nf_conntrack]
[ 619.850114] [<ffffffff814dd5e5>] ops_exit_list.clone.0+0x35/0x70
[ 619.850114] [<ffffffff814dd942>] cleanup_net+0x112/0x1d0
[ 619.850114] [<ffffffff8108224d>] process_one_work+0x11d/0x420
[ 619.850114] [<ffffffff81082ce9>] worker_thread+0x169/0x360
[ 619.850114] [<ffffffff81082b80>] ? worker_thread+0x0/0x360
[ 619.850114] [<ffffffff810871f6>] kthread+0x96/0xa0
[ 619.850114] [<ffffffff8100cde4>] kernel_thread_helper+0x4/0x10
[ 619.850114] [<ffffffff81087160>] ? kthread+0x0/0xa0
[ 619.850114] [<ffffffff8100cde0>] ? kernel_thread_helper+0x0/0x10
[ 619.850114] Code: 5e ff ff ff eb aa 66 66 66 2e 0f 1f 84 00 00 00 00 00 55
48 89 e5 0f 1f 44 00 00 0f 0b 0f 1f 44 00 00 55 48 89 e5 0f 1f 44 00 00
<f6> 87
74 02 00 00 01 74 30 0f b6 97 21 01 00 00 4c 8b 0d 70 56
[ 619.850114] RIP [<ffffffff8150aa99>] netlink_has_listeners+0x9/0x50
[ 619.850114] RSP <ffff880137907bf0>
[ 619.850114] CR2: 0000000000000274
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jun-28 20:43 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Jaka Hudoklin <jakahudoklin at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |WORKSFORME
--- Comment #1 from Jaka Hudoklin <jakahudoklin at gmail.com> 2011-06-28
22:43:24 ---
This is patch that works for me. Dunno if it is patched correctly.
*** a/net/netfilter/nfnetlink.c 2011-04-19 17:17:35.000000000 +0200
--- b/net/netfilter/nfnetlink.c 2011-06-28 14:07:19.689811219 +0200
***************
*** 99,104 ****
--- 99,106 ----
int nfnetlink_has_listeners(struct net *net, unsigned int group)
{
+ if(net->nfnl==NULL)
+ return 0;
return netlink_has_listeners(net->nfnl, group);
}
EXPORT_SYMBOL_GPL(nfnetlink_has_listeners);
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jun-29 06:18 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Jaka Hudoklin <jakahudoklin at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|WORKSFORME |
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jun-30 18:59 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726 --- Comment #2 from Jaka Hudoklin <jakahudoklin at gmail.com> 2011-06-30 20:59:04 --- Should this be implemented?> nfnetlink_has_listeners() and other funcs need proper > rcu_dereference for net->nfnl under rcu. Also, nfnetlink_send > should free the skb if net->nfnl is NULL.Don't know if i am causing memory leaks with my patch or not, since this was my first time patching any kernel module, and because of lack of knowledge of netfilter. -- Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jul-10 19:22 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Jan Engelhardt <jengelh at medozas.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jengelh at medozas.de
AssignedTo|netfilter- |kaber at trash.net
|buglog at lists.netfilter.org |
Status|REOPENED |NEW
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jul-20 17:30 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Joerg Gollnick <netfilter+bugs at wurzelbenutzer.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |netfilter+bugs at wurzelbenutze
| |r.de
--- Comment #3 from Joerg Gollnick <netfilter+bugs at wurzelbenutzer.de>
2011-07-20 19:30:19 ---
I got nearly the same on ArchLinux with a lxc (git from 20110715), two bridges
connected to the container (one is standalone, one is connected to ethx), using
shorewall on host and on container and kernel 2.6.39.3.
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jul-20 20:16 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726 --- Comment #4 from Joerg Gollnick <netfilter+bugs at wurzelbenutzer.de> 2011-07-20 22:16:32 --- Workaround for me is to load nfnetlink module before all the other related modules. -- Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Jul-21 09:21 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726 --- Comment #5 from Patrick McHardy <kaber at trash.net> 2011-07-21 11:21:38 --- Created an attachment (id=356) --> (http://bugzilla.netfilter.org/attachment.cgi?id=356) IPVS netns exit causes crash in conntrack The attached patch from 3.0.0-rc should fix this. Please test and let me know whether it helps and I'll pass it on to -stable. -- Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Aug-01 02:21 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Alex W <weil1 at gmx.at> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |weil1 at gmx.at
--- Comment #6 from Alex W <weil1 at gmx.at> 2011-08-01 04:21:15 ---
@ Patrick McHardy:
The attached patch (commit 8f4e0a18682d91abfad72ede3d3cb5f3ebdf54b4) does not
solve the problem.
I get the same Ooops.
(Tested with the latest Ubuntu Oneiric kernel.)
# uname -a
Linux kamaji 3.0.0-7-server #9-Ubuntu SMP Fri Jul 29 23:09:08 UTC 2011 x86_64
x86_64 x86_64 GNU/Linux
I can reproduce the bug in Oneiric (3.0.0-7-server) and Natty
(2.6.38-10-server) by following the steps Jaka Hudoklin described.
His patch (comment nr1) works for me too. (Thx btw)
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Sep-17 16:43 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726 --- Comment #7 from Jaka Hudoklin <jakahudoklin at gmail.com> 2011-09-17 18:43:16 --- Why is my patch not included in kernel yet? there are many people using lxc over nat having problems. Make this temporary solution and when you patch somewhere else, remove it. Is it so hard or what, because one thing i know, when i will update my ubuntu, this bug will still be there and i will still be forced to patch kernel, just because you couldn't include my patch that's been working great for me for over half a year. -- Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
bugzilla-daemon at bugzilla.netfilter.org
2011-Dec-14 11:50 UTC
[Bug 726] Oops in nf_conntrack.
http://bugzilla.netfilter.org/show_bug.cgi?id=726
Pablo Neira Ayuso <pablo at netfilter.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |pablo at netfilter.org
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #8 from Pablo Neira Ayuso <pablo at netfilter.org> 2011-12-14
12:50:26 ---
A fix for this has been included in 3.2-rc:
commit 70e9942f17a6193e9172a804e6569a8806633d6b
Author: Pablo Neira Ayuso <pablo at netfilter.org>
Date: Tue Nov 22 00:16:51 2011 +0100
netfilter: nf_conntrack: make event callback registration per-netns
--
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.