From: newtongao <newtongao@tencent.com>
Date: Wed, 19 Jun 2013 14:58:33 +0800
Subject: [PATCH] net bridge: add null pointer check,fix panic
in kernel 3.0, br_port_get_rcu() may return NULL when network interface be
deleting from bridge,
but in function br_handle_frame and br_handle_local_finish, the pointer
didn''t be checked before using,
so all br_port_get_rcu callers must do null check,or there occurs the null
pointer panic.
kernel 3.4 also has this bug,i have verified.
mainline kernel still did not check br_port_get_rcu()''s NULL pointer,
but i have not tested it yet.
method to reproduce null pointer panic:
1. in function del_nbp,amplify the gap between "dev->priv_flags &=
~IFF_BRIDGE_PORT;" and "netdev_rx_handler_unregister(dev);" ,
such as add new line "while(1);".
2. create net bridge testbr and bind some network interface(e.g. testif) on it.
3. send packets to testif frequetly,such as ping testif.
4. brctl delif testbr testif.
backtrace:
[1002130.426411] FS: 00007f2235153700(0000) GS:ffff88007b020000(0000)
knlGS:0000000000000000
[1002130.524656] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[1002130.594931] CR2: 0000000000000021 CR3: 0000000001dc7000 CR4:
0000000000002660
[1002130.681798] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1002130.768654] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[1002130.855525] Process ksoftirqd/1 (pid: 9, threadinfo ffff880062642000, task
ffff880062640800)
[1002130.957868] Stack:
[1002130.983758] 0000000080000000 ffff88007bc06a40 ffff88005e3c6c80
ffffffff81901d10
[1002131.073887] ffff88005efc2000 0000000000000001 ffff880062643bf0
ffffffff818497d9
[1002131.163945] ffff88006147e090 0000000000000020 ffff88007bc06a40
ffff88005e3c6c80
[1002131.254023] Call Trace:
[1002131.285086] [<ffffffff81901d10>] ?
br_handle_frame_finish+0x2b0/0x2b0
[1002131.364677] [<ffffffff818497d9>] __netif_receive_skb+0x109/0x4a0
[1002131.439081] [<ffffffff81849b12>] __netif_receive_skb+0x442/0x4a0
[1002131.513493] [<ffffffff810e3a6e>] ? __kmalloc_node+0x3e/0x50
[1002131.582744] [<ffffffff8184b7d8>] netif_receive_skb+0x78/0x80
[1002131.653053] [<ffffffff8184b928>] napi_skb_finish+0x48/0x60
[1002131.721269] [<ffffffff8184bf1d>] napi_gro_receive+0xfd/0x130
[1002131.791551] [<ffffffff8192b2f6>] vlan_gro_receive+0x16/0x20
[1002131.860796] [<ffffffff816a41d1>] igb_poll+0x6f1/0xdc0
[1002131.923871] [<ffffffff8196c0d5>] ? __schedule+0x2e5/0x810
[1002131.991071] [<ffffffff8196e38b>] ? _raw_spin_lock_irq+0xb/0x30
[1002132.063421] [<ffffffff8184c0c1>] net_rx_action+0xa1/0x1d0
[1002132.130619] [<ffffffff81057449>] __do_softirq+0x99/0x130
[1002132.196803] [<ffffffff8105759a>] run_ksoftirqd+0xba/0x170
[1002132.264000] [<ffffffff810574e0>] ? __do_softirq+0x130/0x130
[1002132.333258] [<ffffffff8106d306>] kthread+0x96/0xa0
[1002132.393224] [<ffffffff8196ffe4>] kernel_thread_helper+0x4/0x10
[1002132.465580] [<ffffffff8196f0f6>] ? int_ret_from_sys_call+0x7/0x1b
[1002132.541016] [<ffffffff8196e8a1>] ? retint_restore_args+0x5/0x6
[1002132.613368] [<ffffffff8196ffe0>] ? gs_change+0x13/0x13
[1002132.677466] Code: 02 f6 81 a5 01 00 00 40 48 8b 81 e0 02 00 00 89 d6 4c 0f
45 f0 48 8b 05 50 76 20 00 66 33 70 04 66 f7 c6 ff f0 0f 84 d0 00 00 00
[1002132.835500] 0f b6 46 21 3c 02 74 50 3c 03 0f 85 60 ff ff ff 48 8b 05 e1
[1002132.920864] RIP [<ffffffff81901e17>] br_handle_frame+0x107/0x260
[1002132.995296] RSP <ffff880062643b50>
[1002133.038735] CR2: 0000000000000021
Signed-off-by: Xiaoming Gao <newtongao@tencent.com>
---
net/bridge/br_input.c | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index f06ee39..c8b365f 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -122,7 +122,8 @@ static int br_handle_local_finish(struct sk_buff *skb)
{
struct net_bridge_port *p = br_port_get_rcu(skb->dev);
- br_fdb_update(p->br, p, eth_hdr(skb)->h_source);
+ if (p)
+ br_fdb_update(p->br, p, eth_hdr(skb)->h_source);
return 0; /* process further */
}
@@ -160,6 +161,8 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
return RX_HANDLER_CONSUMED;
p = br_port_get_rcu(skb->dev);
+ if (!p)
+ goto drop;
if (unlikely(is_link_local(dest))) {
/* Pause frames shouldn''t be passed up by driver anyway */
--
1.7.1
xiaoming gao
2013-Jun-20 06:53 UTC
[Bridge] Fwd: [PATCH] net bridge: add null pointer check, fix panic
HI Eric the problem is as follow: br_del_if()-->del_nbp(): list_del_rcu(&p->list); dev->priv_flags &= ~IFF_BRIDGE_PORT; ------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame() ------>br_port_exists() will return false,so br_get_port_rcu() will return NULL ------>so in br_handle_frame , there will be a null panic. netdev_rx_handler_unregister(dev); synchronize_net(); i have checked commit 00cfec37484761a44, i think it didn't fix this bug.. thanks. On Thu, Jun 20, 2013 at 12:55 PM, Eric Dumazet <eric.dumazet at gmail.com>wrote:> On Thu, 2013-06-20 at 11:08 +0800, xiaoming gao wrote: > > From: newtongao <newtongao at tencent.com> > > Date: Wed, 19 Jun 2013 14:58:33 +0800 > > Subject: [PATCH] net bridge: add null pointer check,fix panic > > > > in kernel 3.0, br_port_get_rcu() may return NULL when network interface > be deleting from bridge, > > but in function br_handle_frame and br_handle_local_finish, the pointer > didn't be checked before using, > > so all br_port_get_rcu callers must do null check,or there occurs the > null pointer panic. > > > > kernel 3.4 also has this bug,i have verified. > > mainline kernel still did not check br_port_get_rcu()'s NULL pointer, > but i have not tested it yet. > > Please check current version before sending a patch. > > This was most probably fixed in commit 00cfec37484761a44 > ("net: add a synchronize_net() in netdev_rx_handler_unregister()") > > Thanks > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linuxfoundation.org/pipermail/bridge/attachments/20130620/be201aa9/attachment-0001.html>
xiaoming gao
2013-Jun-20 07:00 UTC
Re: [PATCH] net bridge: add null pointer check, fix panic
Eric Dumazet said, at 2013-6-20 12:55:> On Thu, 2013-06-20 at 11:08 +0800, xiaoming gao wrote: >> From: newtongao <newtongao@tencent.com> >> Date: Wed, 19 Jun 2013 14:58:33 +0800 >> Subject: [PATCH] net bridge: add null pointer check,fix panic >> >> in kernel 3.0, br_port_get_rcu() may return NULL when network interface be deleting from bridge, >> but in function br_handle_frame and br_handle_local_finish, the pointer didn''t be checked before using, >> so all br_port_get_rcu callers must do null check,or there occurs the null pointer panic. >> >> kernel 3.4 also has this bug,i have verified. >> mainline kernel still did not check br_port_get_rcu()''s NULL pointer, but i have not tested it yet. > > Please check current version before sending a patch. > > This was most probably fixed in commit 00cfec37484761a44 > ("net: add a synchronize_net() in netdev_rx_handler_unregister()") > > Thanks > >HI Eric the problem is as follow: br_del_if()-->del_nbp(): list_del_rcu(&p->list); dev->priv_flags &= ~IFF_BRIDGE_PORT; ------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame() ------>br_port_exists() will return false,so br_get_port_rcu() will return NULL ------>so in br_handle_frame , there will be a null panic. netdev_rx_handler_unregister(dev); synchronize_net(); i have checked commit 00cfec37484761a44, i think it didn''t fix this bug.. thanks.
xiaoming gao
2013-Jun-20 07:47 UTC
Re: [PATCH] net bridge: add null pointer check, fix panic
Eric Dumazet said, at 2013-6-20 15:29:> On Thu, 2013-06-20 at 15:00 +0800, xiaoming gao wrote: > >> HI Eric >> the problem is as follow: >> br_del_if()-->del_nbp(): >> >> list_del_rcu(&p->list); >> dev->priv_flags &= ~IFF_BRIDGE_PORT; >> >> ------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame() >> ------>br_port_exists() will return false,so br_get_port_rcu() will return NULL >> ------>so in br_handle_frame , there will be a null panic. >> >> netdev_rx_handler_unregister(dev); >> synchronize_net(); > > This code is no longer like that in current tree. > Check commit 4a0b5ec12f0ffc3024616e6dc62cf8a04c54edcd > ("bridge: remove a redundant synchronize_net()") > >> >> >> i have checked commit 00cfec37484761a44, i think it didn''t fix this bug.. > > I claim adding NULL tests is not needed in the fast path, it was clearly > stated in the changelog. > > I would change the dismantle path to make sure br_get_port_rcu() does > not return NULL in the fast path, and remove the test on FF_BRIDGE_PORT > > Try something like that : > > diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c > index 1b8b8b8..2edfb80 100644 > --- a/net/bridge/br_input.c > +++ b/net/bridge/br_input.c > @@ -60,7 +60,7 @@ static int br_pass_frame_up(struct sk_buff *skb) > int br_handle_frame_finish(struct sk_buff *skb) > { > const unsigned char *dest = eth_hdr(skb)->h_dest; > - struct net_bridge_port *p = br_port_get_rcu(skb->dev); > + struct net_bridge_port *p = __br_port_get_rcu(skb->dev); > struct net_bridge *br; > struct net_bridge_fdb_entry *dst; > struct net_bridge_mdb_entry *mdst; > @@ -68,7 +68,7 @@ int br_handle_frame_finish(struct sk_buff *skb) > bool unicast = true; > u16 vid = 0; > > - if (!p || p->state == BR_STATE_DISABLED) > + if (p->state == BR_STATE_DISABLED) > goto drop; > > if (!br_allowed_ingress(p->br, nbp_get_vlan_info(p), skb, &vid)) > @@ -142,7 +142,7 @@ drop: > /* note: already called with rcu_read_lock */ > static int br_handle_local_finish(struct sk_buff *skb) > { > - struct net_bridge_port *p = br_port_get_rcu(skb->dev); > + struct net_bridge_port *p = __br_port_get_rcu(skb->dev); > u16 vid = 0; > > br_vlan_get_tag(skb, &vid); > @@ -172,7 +172,7 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb) > if (!skb) > return RX_HANDLER_CONSUMED; > > - p = br_port_get_rcu(skb->dev); > + p = __br_port_get_rcu(skb->dev); > > if (unlikely(is_link_local_ether_addr(dest))) { > /* > diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h > index 3be89b3..9fdd467 100644 > --- a/net/bridge/br_private.h > +++ b/net/bridge/br_private.h > @@ -184,6 +184,11 @@ struct net_bridge_port > > #define br_port_exists(dev) (dev->priv_flags & IFF_BRIDGE_PORT) > > +static inline struct net_bridge_port *__br_port_get_rcu(const struct net_device *dev) > +{ > + return rcu_dereference(dev->rx_handler_data); > +} > + > static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *dev) > { > struct net_bridge_port *port > >if you remove the test on FF_BRIDGE_PORT, and br_port_get_rcu never returns NULL, the problem is fixed. but the codes in mainline is still bugged, am i right?? by the way, kernel-stable 3.0 and 3.4 tree also have this bug, and is very easily to reproduce.
Alexander Y. Fomichev
2013-Nov-11 10:27 UTC
Re: [PATCH] net bridge: add null pointer check, fix panic
On Thu, Jun 20, 2013 at 11:29 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:> On Thu, 2013-06-20 at 15:00 +0800, xiaoming gao wrote: > >> HI Eric >> the problem is as follow: >> br_del_if()-->del_nbp(): >> >> list_del_rcu(&p->list); >> dev->priv_flags &= ~IFF_BRIDGE_PORT; >> >> ------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame() >> ------>br_port_exists() will return false,so br_get_port_rcu() will return NULL >> ------>so in br_handle_frame , there will be a null panic. >> >> netdev_rx_handler_unregister(dev); >> synchronize_net(); > > This code is no longer like that in current tree. > Check commit 4a0b5ec12f0ffc3024616e6dc62cf8a04c54edcd > ("bridge: remove a redundant synchronize_net()") > >> >> >> i have checked commit 00cfec37484761a44, i think it didn''t fix this bug.. > > I claim adding NULL tests is not needed in the fast path, it was clearly > stated in the changelog. > > I would change the dismantle path to make sure br_get_port_rcu() does > not return NULL in the fast path, and remove the test on FF_BRIDGE_PORT > Try something like that : > > diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c > index 1b8b8b8..2edfb80 100644 > --- a/net/bridge/br_input.c > +++ b/net/bridge/br_input.c > @@ -60,7 +60,7 @@ static int br_pass_frame_up(struct sk_buff *skb) > int br_handle_frame_finish(struct sk_buff *skb) > { > const unsigned char *dest = eth_hdr(skb)->h_dest; > - struct net_bridge_port *p = br_port_get_rcu(skb->dev); > + struct net_bridge_port *p = __br_port_get_rcu(skb->dev); > struct net_bridge *br; > struct net_bridge_fdb_entry *dst; > struct net_bridge_mdb_entry *mdst; > @@ -68,7 +68,7 @@ int br_handle_frame_finish(struct sk_buff *skb) > bool unicast = true; > u16 vid = 0; > > - if (!p || p->state == BR_STATE_DISABLED) > + if (p->state == BR_STATE_DISABLED) > goto drop; > > if (!br_allowed_ingress(p->br, nbp_get_vlan_info(p), skb, &vid)) > @@ -142,7 +142,7 @@ drop: > /* note: already called with rcu_read_lock */ > static int br_handle_local_finish(struct sk_buff *skb) > { > - struct net_bridge_port *p = br_port_get_rcu(skb->dev); > + struct net_bridge_port *p = __br_port_get_rcu(skb->dev); > u16 vid = 0; > > br_vlan_get_tag(skb, &vid); > @@ -172,7 +172,7 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb) > if (!skb) > return RX_HANDLER_CONSUMED; > > - p = br_port_get_rcu(skb->dev); > + p = __br_port_get_rcu(skb->dev); > > if (unlikely(is_link_local_ether_addr(dest))) { > /* > diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h > index 3be89b3..9fdd467 100644 > --- a/net/bridge/br_private.h > +++ b/net/bridge/br_private.h > @@ -184,6 +184,11 @@ struct net_bridge_port > > #define br_port_exists(dev) (dev->priv_flags & IFF_BRIDGE_PORT) > > +static inline struct net_bridge_port *__br_port_get_rcu(const struct net_device *dev) > +{ > + return rcu_dereference(dev->rx_handler_data); > +} > + > static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *dev) > { > struct net_bridge_port *port > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/> I claim adding NULL tests is not needed in the fast path, it was clearly > stated in the changelog.Hello, This commit makes trouble for current STP. Two days ago i tried to switch to 3.10.18 and i''ve caught "bad magic" on uninitialized br->lock: ./net/bridge/br_stp_bpdu.c +158 in br_stp_rcv (trace attached): p = br_port_get_rcu(dev); br = p->br; spin_lock(&br->lock); <- here ----------------------------------------------- Bisect pointed to this commit (linux-stable: 960b8e5018a552f62cfbc0dfe94be7b6ba178f13) (mainline 716ec052d2280d511e10e90ad54a86f5b5d4dcc2) As far as i can see this happens when: a) bridge module had been loaded but there was no bridge interface, br->lock had not been initialized. b) interface had been in promiscuous mod (tcpdump) c) stp broadcasts 01:80:c2:00:00:00 coming to this iface (llc_rcv drops PACKET_OTHERHOST to protect us in promiscuous mode but seems like not a broadcasts) d) and finally rx_handler_data had been initialised for this interface ( by macvlan in my case) It seems like STP needs its own IFF_BRIDGE_PORT check. probably an easiest option to check it in br_stp_rcv as before (or probably in llc_rcv)... -- Best regards. Alexander Y. Fomichev <git.user@gmail.com>