Konrad Rzeszutek Wilk
2012-Nov-10 13:59 UTC
domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
Hey Ian, Xen-devel mailingl list, I think the issue of 70% traffic lost was actually introduced in v3.6 or perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of the releases introduced this. The issue we are seeing is that a domU to domU communication breaks - this is with netperf/netserver talking to each other. Anyhow, I think the 3.7 compound page exacerbated the problem and also (at least on some of my test hardware) exposed existing issues with drivers. The issue I have is that the ''skge'' driver has a bug that has been there for ages (I tested way back to 3.0 and still saw it) were it can not work with SWIOTLB. It is probably missing an pci_dma_sync somewhere. Anyhow the compound page got me to look at Xen-SWIOTLB and that looks OK. Even with synthetic driver (the fake one I posted somewhere) it dealt with compound pages properly (with debug or non-debug Xen hypervisor). So was wondering if you had looked at this in more details? Any ideas? Or would it be more prudent to ask that once we know for sure which Linux release introduced the communication failures between guests?
Ian Campbell
2012-Nov-12 09:54 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote:> Hey Ian, Xen-devel mailingl list, > > I think the issue of 70% traffic lost was actually introduced in v3.6 or > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of > the releases introduced this. The issue we are seeing is that a domU > to domU communication breaks - this is with netperf/netserver talking to > each other. > > Anyhow, I think the 3.7 compound page exacerbated the problem and also > (at least on some of my test hardware) exposed existing issues with > drivers. The issue I have is that the ''skge'' driver has a bug that has > been there for ages (I tested way back to 3.0 and still saw it) were it > can not work with SWIOTLB. It is probably missing an pci_dma_sync > somewhere. > > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks > OK. Even with synthetic driver (the fake one I posted somewhere) it > dealt with compound pages properly (with debug or non-debug Xen > hypervisor).The debug build is probably most interesting since it deliberately allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of issues.> So was wondering if you had looked at this in more details? Any > ideas? Or would it be more prudent to ask that once we know for sure > which Linux release introduced the communication failures between > guests?I''ve not looked at it any further I''m afraid. If these changes (be they in 3.5 or later, or earlier) are exposing driver bugs then I suspect the netdev chaps would want to know about it. FWIW I see the issue with tg3. Ian.
Konrad Rzeszutek Wilk
2012-Nov-12 14:28 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote:> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote: > > Hey Ian, Xen-devel mailingl list, > > > > I think the issue of 70% traffic lost was actually introduced in v3.6 or > > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of > > the releases introduced this. The issue we are seeing is that a domU > > to domU communication breaks - this is with netperf/netserver talking to > > each other. > > > > Anyhow, I think the 3.7 compound page exacerbated the problem and also > > (at least on some of my test hardware) exposed existing issues with > > drivers. The issue I have is that the ''skge'' driver has a bug that has > > been there for ages (I tested way back to 3.0 and still saw it) were it > > can not work with SWIOTLB. It is probably missing an pci_dma_sync > > somewhere. > > > > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks > > OK. Even with synthetic driver (the fake one I posted somewhere) it > > dealt with compound pages properly (with debug or non-debug Xen > > hypervisor). > > The debug build is probably most interesting since it deliberately > allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of > issues.Right. My test env runs with that. And so far it only has issues with the skge one.> > > So was wondering if you had looked at this in more details? Any > > ideas? Or would it be more prudent to ask that once we know for sure > > which Linux release introduced the communication failures between > > guests? > > I''ve not looked at it any further I''m afraid. > > If these changes (be they in 3.5 or later, or earlier) are exposing > driver bugs then I suspect the netdev chaps would want to know about it.Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok. And is off checking v3.6. v3.7 is definitly a no go.> > FWIW I see the issue with tg3.I should double-check that but I didn''t get any issues with the tg3 I had. I was running the netperf/netserver from/to to see if I could reproduce the issues (70% loss) you described but did not have much luck.> > Ian.
Ian Campbell
2012-Nov-12 14:41 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
> > FWIW I see the issue with tg3. > > I should double-check that but I didn''t get any issues with the tg3 I had. > > I was running the netperf/netserver from/to to see if I could reproduce the > issues (70% loss) you described but did not have much luck.I was running from offbox to a domU, if that helps. It only happened in one direction but I can''t recall which.
Sander Eikelenboom
2012-Nov-12 14:50 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
Monday, November 12, 2012, 3:28:35 PM, you wrote:> On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote: >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote: >> > Hey Ian, Xen-devel mailingl list, >> > >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of >> > the releases introduced this. The issue we are seeing is that a domU >> > to domU communication breaks - this is with netperf/netserver talking to >> > each other. >> > >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also >> > (at least on some of my test hardware) exposed existing issues with >> > drivers. The issue I have is that the ''skge'' driver has a bug that has >> > been there for ages (I tested way back to 3.0 and still saw it) were it >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync >> > somewhere. >> > >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks >> > OK. Even with synthetic driver (the fake one I posted somewhere) it >> > dealt with compound pages properly (with debug or non-debug Xen >> > hypervisor). >> >> The debug build is probably most interesting since it deliberately >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of >> issues.> Right. My test env runs with that. And so far it only has issues > with the skge one. >> >> > So was wondering if you had looked at this in more details? Any >> > ideas? Or would it be more prudent to ask that once we know for sure >> > which Linux release introduced the communication failures between >> > guests? >> >> I''ve not looked at it any further I''m afraid. >> >> If these changes (be they in 3.5 or later, or earlier) are exposing >> driver bugs then I suspect the netdev chaps would want to know about it.> Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok. > And is off checking v3.6. v3.7 is definitly a no go. >> >> FWIW I see the issue with tg3.After the issues with netback where fixed, I''m seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear. I haven''t seen issues with netfront and netback before this commit. FWIW: the hardware nics in this machine are two r8169 ''s. -- Sander> I should double-check that but I didn''t get any issues with the tg3 I had.> I was running the netperf/netserver from/to to see if I could reproduce the > issues (70% loss) you described but did not have much luck.>> >> Ian.
Konrad Rzeszutek Wilk
2012-Nov-12 16:32 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote:> > Monday, November 12, 2012, 3:28:35 PM, you wrote: > > > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote: > >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote: > >> > Hey Ian, Xen-devel mailingl list, > >> > > >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or > >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of > >> > the releases introduced this. The issue we are seeing is that a domU > >> > to domU communication breaks - this is with netperf/netserver talking to > >> > each other. > >> > > >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also > >> > (at least on some of my test hardware) exposed existing issues with > >> > drivers. The issue I have is that the ''skge'' driver has a bug that has > >> > been there for ages (I tested way back to 3.0 and still saw it) were it > >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync > >> > somewhere. > >> > > >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks > >> > OK. Even with synthetic driver (the fake one I posted somewhere) it > >> > dealt with compound pages properly (with debug or non-debug Xen > >> > hypervisor). > >> > >> The debug build is probably most interesting since it deliberately > >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of > >> issues. > > > Right. My test env runs with that. And so far it only has issues > > with the skge one. > >> > >> > So was wondering if you had looked at this in more details? Any > >> > ideas? Or would it be more prudent to ask that once we know for sure > >> > which Linux release introduced the communication failures between > >> > guests? > >> > >> I''ve not looked at it any further I''m afraid. > >> > >> If these changes (be they in 3.5 or later, or earlier) are exposing > >> driver bugs then I suspect the netdev chaps would want to know about it. > > > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok. > > And is off checking v3.6. v3.7 is definitly a no go. > >> > >> FWIW I see the issue with tg3. > > After the issues with netback where fixed, I''m seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.Were you ever able to trigger the BUG_ON in the patch that Ian posted?> I haven''t seen issues with netfront and netback before this commit. FWIW: the hardware nics in this machine are two r8169 ''s. > > -- > Sander > > > I should double-check that but I didn''t get any issues with the tg3 I had. > > > I was running the netperf/netserver from/to to see if I could reproduce the > > issues (70% loss) you described but did not have much luck. > > >> > >> Ian. > >
Sander Eikelenboom
2012-Nov-12 17:20 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
Monday, November 12, 2012, 5:32:04 PM, you wrote:> On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote: >> >> Monday, November 12, 2012, 3:28:35 PM, you wrote: >> >> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote: >> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote: >> >> > Hey Ian, Xen-devel mailingl list, >> >> > >> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or >> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of >> >> > the releases introduced this. The issue we are seeing is that a domU >> >> > to domU communication breaks - this is with netperf/netserver talking to >> >> > each other. >> >> > >> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also >> >> > (at least on some of my test hardware) exposed existing issues with >> >> > drivers. The issue I have is that the ''skge'' driver has a bug that has >> >> > been there for ages (I tested way back to 3.0 and still saw it) were it >> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync >> >> > somewhere. >> >> > >> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks >> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it >> >> > dealt with compound pages properly (with debug or non-debug Xen >> >> > hypervisor). >> >> >> >> The debug build is probably most interesting since it deliberately >> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of >> >> issues. >> >> > Right. My test env runs with that. And so far it only has issues >> > with the skge one. >> >> >> >> > So was wondering if you had looked at this in more details? Any >> >> > ideas? Or would it be more prudent to ask that once we know for sure >> >> > which Linux release introduced the communication failures between >> >> > guests? >> >> >> >> I''ve not looked at it any further I''m afraid. >> >> >> >> If these changes (be they in 3.5 or later, or earlier) are exposing >> >> driver bugs then I suspect the netdev chaps would want to know about it. >> >> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok. >> > And is off checking v3.6. v3.7 is definitly a no go. >> >> >> >> FWIW I see the issue with tg3. >> >> After the issues with netback where fixed, I''m seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear.> Were you ever able to trigger the BUG_ON in the patch that Ian posted?What exact patch (or any other patch that can help you ) ? (so i can try again to be sure)>> I haven''t seen issues with netfront and netback before this commit. FWIW: the hardware nics in this machine are two r8169 ''s. >> >> -- >> Sander >> >> > I should double-check that but I didn''t get any issues with the tg3 I had. >> >> > I was running the netperf/netserver from/to to see if I could reproduce the >> > issues (70% loss) you described but did not have much luck. >> >> >> >> >> Ian. >> >>
Konrad Rzeszutek Wilk
2012-Nov-14 06:24 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
On Mon, Nov 12, 2012 at 06:20:55PM +0100, Sander Eikelenboom wrote:> > Monday, November 12, 2012, 5:32:04 PM, you wrote: > > > On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote: > >> > >> Monday, November 12, 2012, 3:28:35 PM, you wrote: > >> > >> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote: > >> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote: > >> >> > Hey Ian, Xen-devel mailingl list, > >> >> > > >> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or > >> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of > >> >> > the releases introduced this. The issue we are seeing is that a domU > >> >> > to domU communication breaks - this is with netperf/netserver talking to > >> >> > each other. > >> >> > > >> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also > >> >> > (at least on some of my test hardware) exposed existing issues with > >> >> > drivers. The issue I have is that the ''skge'' driver has a bug that has > >> >> > been there for ages (I tested way back to 3.0 and still saw it) were it > >> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync > >> >> > somewhere. > >> >> > > >> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks > >> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it > >> >> > dealt with compound pages properly (with debug or non-debug Xen > >> >> > hypervisor). > >> >> > >> >> The debug build is probably most interesting since it deliberately > >> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of > >> >> issues. > >> > >> > Right. My test env runs with that. And so far it only has issues > >> > with the skge one. > >> >> > >> >> > So was wondering if you had looked at this in more details? Any > >> >> > ideas? Or would it be more prudent to ask that once we know for sure > >> >> > which Linux release introduced the communication failures between > >> >> > guests? > >> >> > >> >> I''ve not looked at it any further I''m afraid. > >> >> > >> >> If these changes (be they in 3.5 or later, or earlier) are exposing > >> >> driver bugs then I suspect the netdev chaps would want to know about it. > >> > >> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok. > >> > And is off checking v3.6. v3.7 is definitly a no go. > >> >> > >> >> FWIW I see the issue with tg3. > >> > >> After the issues with netback where fixed, I''m seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear. > > > Were you ever able to trigger the BUG_ON in the patch that Ian posted? > > What exact patch (or any other patch that can help you ) ? > (so i can try again to be sure)This one: http://lists.xen.org/archives/html/xen-devel/2012-10/msg00893.html
Sander Eikelenboom
2012-Nov-16 09:46 UTC
Re: domU to domU networking issues in v3.7? (netserver/netperf failing to communicate)
Wednesday, November 14, 2012, 7:24:53 AM, you wrote:> On Mon, Nov 12, 2012 at 06:20:55PM +0100, Sander Eikelenboom wrote: >> >> Monday, November 12, 2012, 5:32:04 PM, you wrote: >> >> > On Mon, Nov 12, 2012 at 03:50:24PM +0100, Sander Eikelenboom wrote: >> >> >> >> Monday, November 12, 2012, 3:28:35 PM, you wrote: >> >> >> >> > On Mon, Nov 12, 2012 at 09:54:44AM +0000, Ian Campbell wrote: >> >> >> On Sat, 2012-11-10 at 13:59 +0000, Konrad Rzeszutek Wilk wrote: >> >> >> > Hey Ian, Xen-devel mailingl list, >> >> >> > >> >> >> > I think the issue of 70% traffic lost was actually introduced in v3.6 or >> >> >> > perhaps v3.5. Annie and Marcos (CC-ed here) are looking to see which of >> >> >> > the releases introduced this. The issue we are seeing is that a domU >> >> >> > to domU communication breaks - this is with netperf/netserver talking to >> >> >> > each other. >> >> >> > >> >> >> > Anyhow, I think the 3.7 compound page exacerbated the problem and also >> >> >> > (at least on some of my test hardware) exposed existing issues with >> >> >> > drivers. The issue I have is that the ''skge'' driver has a bug that has >> >> >> > been there for ages (I tested way back to 3.0 and still saw it) were it >> >> >> > can not work with SWIOTLB. It is probably missing an pci_dma_sync >> >> >> > somewhere. >> >> >> > >> >> >> > Anyhow the compound page got me to look at Xen-SWIOTLB and that looks >> >> >> > OK. Even with synthetic driver (the fake one I posted somewhere) it >> >> >> > dealt with compound pages properly (with debug or non-debug Xen >> >> >> > hypervisor). >> >> >> >> >> >> The debug build is probably most interesting since it deliberately >> >> >> allocates a non 1-1 p-to-m mapping so as to catch exactly these sorts of >> >> >> issues. >> >> >> >> > Right. My test env runs with that. And so far it only has issues >> >> > with the skge one. >> >> >> >> >> >> > So was wondering if you had looked at this in more details? Any >> >> >> > ideas? Or would it be more prudent to ask that once we know for sure >> >> >> > which Linux release introduced the communication failures between >> >> >> > guests? >> >> >> >> >> >> I''ve not looked at it any further I''m afraid. >> >> >> >> >> >> If these changes (be they in 3.5 or later, or earlier) are exposing >> >> >> driver bugs then I suspect the netdev chaps would want to know about it. >> >> >> >> > Right. Annie (CC-ed here) mentioned to me that v3.5 looks to work ok. >> >> > And is off checking v3.6. v3.7 is definitly a no go. >> >> >> >> >> >> FWIW I see the issue with tg3. >> >> >> >> After the issues with netback where fixed, I''m seeing the issues with net_front reverting the single commit 5640f7685831e088fe6c2e1f863a6805962f8e81 (that was pointed out for netback) also makes these disappear. >> >> > Were you ever able to trigger the BUG_ON in the patch that Ian posted? >> >> What exact patch (or any other patch that can help you ) ? >> (so i can try again to be sure)> This one: > http://lists.xen.org/archives/html/xen-devel/2012-10/msg00893.htmlSorry for the late response! I have changed the bug_on in a warn_on and applied it to a kernel with and without commit 5640f7685831e088fe6c2e1f863a6805962f8e81 reverted. - With commit 5640f7685831e088fe6c2e1f863a6805962f8e81 reverted I don''t see any warnings (also after loading domU''s with a few GB of network traffic) - With a vanilla kernel and the patch, i can just ssh to a domU to see the warnings, after which SSH locks up. So it happens immediately, even without giving the domU any serious network load. [ 85.447191] <IRQ> [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0 [ 85.447201] [<ffffffff81067555>] warn_slowpath_null+0x15/0x20 [ 85.447207] [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860 [ 85.447213] [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460 [ 85.447220] [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290 [ 85.447225] [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0 [ 85.447231] [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460 [ 85.447238] [<ffffffff816b0897>] neigh_resolve_output+0x127/0x250 [ 85.447244] [<ffffffff817405ad>] ip_finish_output+0x39d/0x530 [ 85.447250] [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530 [ 85.447256] [<ffffffff81740799>] ip_output+0x59/0xe0 [ 85.447262] [<ffffffff8173f2b8>] ip_local_out+0x28/0x90 [ 85.447267] [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0 [ 85.447273] [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340 [ 85.447282] [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0 [ 85.447288] [<ffffffff816962c9>] ? __skb_clone+0x29/0x120 [ 85.447294] [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0 [ 85.447303] [<ffffffff8113ad55>] ? kmem_cache_free+0x125/0x1c0 [ 85.450359] [<ffffffff8175891a>] tcp_write_xmit+0x21a/0xa50 [ 85.450359] [<ffffffff81698c82>] ? __kfree_skb+0x42/0xa0 [ 85.450359] [<ffffffff817591bd>] __tcp_push_pending_frames+0x2d/0x90 [ 85.450359] [<ffffffff817545c4>] tcp_rcv_established+0x164/0x640 [ 85.450359] [<ffffffff8175cdbf>] ? tcp_v4_rcv+0x6cf/0xb10 [ 85.450359] [<ffffffff8175c3a5>] tcp_v4_do_rcv+0x135/0x480 [ 85.450359] [<ffffffff817d0dc2>] ? _raw_spin_lock_nested+0x42/0x50 [ 85.450359] [<ffffffff8175cdbf>] ? tcp_v4_rcv+0x6cf/0xb10 [ 85.450359] [<ffffffff8175d04d>] tcp_v4_rcv+0x95d/0xb10 [ 85.450359] [<ffffffff810b2158>] ? lock_acquire+0xd8/0x100 [ 85.450359] [<ffffffff81739895>] ? ip_local_deliver_finish+0x45/0x230 [ 85.450359] [<ffffffff8173996a>] ip_local_deliver_finish+0x11a/0x230 [ 85.450359] [<ffffffff81739895>] ? ip_local_deliver_finish+0x45/0x230 [ 85.450359] [<ffffffff81739ab8>] ip_local_deliver+0x38/0x80 [ 85.450359] [<ffffffff8173907a>] ip_rcv_finish+0x15a/0x630 [ 85.450359] [<ffffffff81739768>] ip_rcv+0x218/0x300 [ 85.450359] [<ffffffff816a1a9d>] __netif_receive_skb+0x65d/0x8d0 [ 85.450359] [<ffffffff816a1585>] ? __netif_receive_skb+0x145/0x8d0 [ 85.450359] [<ffffffff810ae3fd>] ? trace_hardirqs_on+0xd/0x10 [ 85.450359] [<ffffffff810faf83>] ? free_hot_cold_page+0x1b3/0x1e0 [ 85.450359] [<ffffffff816a4008>] netif_receive_skb+0x28/0xf0 [ 85.450359] [<ffffffff81699b13>] ? __pskb_pull_tail+0x253/0x340 [ 85.450359] [<ffffffff814b3795>] xennet_poll+0xad5/0xe10 [ 85.450359] [<ffffffff816a4db6>] net_rx_action+0x136/0x260 [ 85.450359] [<ffffffff8106f3a1>] ? __do_softirq+0x71/0x1a0 [ 85.450359] [<ffffffff8106f3f9>] __do_softirq+0xc9/0x1a0 [ 85.450359] [<ffffffff817d393c>] call_softirq+0x1c/0x30 [ 85.450359] [<ffffffff8100fd95>] do_softirq+0x85/0xf0 [ 85.450359] [<ffffffff8106f26e>] irq_exit+0x9e/0xd0 [ 85.450359] [<ffffffff8134659f>] xen_evtchn_do_upcall+0x2f/0x40 [ 85.450359] [<ffffffff817d399e>] xen_do_hypervisor_callback+0x1e/0x30 [ 85.450359] <EOI> [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20 [ 85.450359] [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20 [ 85.450359] [<ffffffff8100880d>] ? xen_force_evtchn_callback+0xd/0x10 [ 85.450359] [<ffffffff810091b2>] ? check_events+0x12/0x20 [ 85.450359] [<ffffffff8100919f>] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 85.450359] [<ffffffff810adfc0>] ? lockdep_trace_alloc+0xa0/0x130 [ 85.450359] [<ffffffff810fc9ad>] ? __alloc_pages_nodemask+0x8d/0x9a0 [ 85.450359] [<ffffffff810b2158>] ? lock_acquire+0xd8/0x100 [ 85.450359] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.450359] [<ffffffff812a332e>] ? check_blkcg_changed+0x8e/0x180 [ 85.450359] [<ffffffff812a32a0>] ? cfq_put_queue+0x220/0x220 [ 85.450359] [<ffffffff810b2158>] ? lock_acquire+0xd8/0x100 [ 85.450359] [<ffffffff81132f21>] ? alloc_pages_vma+0xc1/0x1d0 [ 85.450359] [<ffffffff8110752a>] ? shmem_alloc_page+0x5a/0x70 [ 85.450359] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.450359] [<ffffffff810f40fd>] ? find_get_page+0xad/0x100 [ 85.450359] [<ffffffff810f4050>] ? find_get_pages+0x190/0x190 [ 85.450359] [<ffffffff810b171b>] ? __lock_acquire+0x46b/0xdd0 [ 85.450359] [<ffffffff810f43d1>] ? find_lock_page+0x21/0x80 [ 85.450359] [<ffffffff81107c3f>] ? shmem_getpage_gfp+0x33f/0x720 [ 85.450359] [<ffffffff811080f6>] ? shmem_fault+0x46/0x70 [ 85.450359] [<ffffffff812a2d33>] ? cfq_add_cfqq_rr+0x33/0x100 [ 85.450359] [<ffffffff81269f79>] ? shm_fault+0x19/0x20 [ 85.450359] [<ffffffff81114de7>] ? __do_fault+0x67/0x4f0 [ 85.450359] [<ffffffff8111885d>] ? handle_pte_fault+0xfd/0x820 [ 85.450359] [<ffffffff8100509f>] ? pte_mfn_to_pfn+0x8f/0xf0 [ 85.450359] [<ffffffff81005159>] ? xen_pmd_val+0x9/0x10 [ 85.450359] [<ffffffff81119208>] ? handle_mm_fault+0x288/0x360 [ 85.450359] [<ffffffff8103eafc>] ? __do_page_fault+0x15c/0x510 [ 85.450359] [<ffffffff8128e2f3>] ? blk_finish_plug+0x13/0x50 [ 85.450359] [<ffffffff810ffc99>] ? __do_page_cache_readahead+0x229/0x2b0 [ 85.450359] [<ffffffff810ffb25>] ? __do_page_cache_readahead+0xb5/0x2b0 [ 85.450359] [<ffffffff810ac927>] ? trace_hardirqs_off_caller+0x77/0x150 [ 85.450359] [<ffffffff812b55ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 85.450359] [<ffffffff8103eeb9>] ? do_page_fault+0x9/0x10 [ 85.450359] [<ffffffff817d20b8>] ? page_fault+0x28/0x30 [ 85.450359] [<ffffffff810f39a6>] ? file_read_actor+0x66/0x190 [ 85.450359] [<ffffffff81100230>] ? page_cache_async_readahead+0x80/0xa0 [ 85.450359] [<ffffffff810f4050>] ? find_get_pages+0x190/0x190 [ 85.450359] [<ffffffff810f6050>] ? generic_file_aio_read+0x2e0/0x6f0 [ 85.450359] [<ffffffff81144c82>] ? do_sync_read+0xa2/0xe0 [ 85.450359] [<ffffffff811459f5>] ? vfs_read+0xc5/0x190 [ 85.450359] [<ffffffff81145bca>] ? sys_read+0x5a/0xa0 [ 85.450359] [<ffffffff812b556e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 85.450359] [<ffffffff817d26a9>] ? system_call_fastpath+0x16/0x1b [ 85.450359] ---[ end trace ce03efc8ae5a5e97 ]--- [ 85.650129] ------------[ cut here ]------------ [ 85.650147] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860() [ 85.650154] Modules linked in: [ 85.650161] Pid: 0, comm: swapper/0 Tainted: G W 3.7.0-rc5-20121116-vanilla-warn #1 [ 85.650167] Call Trace: [ 85.650171] <IRQ> [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0 [ 85.650183] [<ffffffff81067555>] warn_slowpath_null+0x15/0x20 [ 85.650189] [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860 [ 85.650196] [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460 [ 85.650203] [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290 [ 85.650208] [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0 [ 85.650214] [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460 [ 85.650221] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.650227] [<ffffffff81740436>] ip_finish_output+0x226/0x530 [ 85.650234] [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530 [ 85.650240] [<ffffffff81740799>] ip_output+0x59/0xe0 [ 85.650245] [<ffffffff8173f2b8>] ip_local_out+0x28/0x90 [ 85.650251] [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0 [ 85.650256] [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340 [ 85.650262] [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0 [ 85.650268] [<ffffffff816962c9>] ? __skb_clone+0x29/0x120 [ 85.650274] [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0 [ 85.650280] [<ffffffff81758026>] tcp_retransmit_skb+0x1c6/0x5a0 [ 85.650290] [<ffffffff8175a300>] ? tcp_write_timer_handler+0x1a0/0x1a0 [ 85.650296] [<ffffffff81759e70>] tcp_retransmit_timer+0x370/0x660 [ 85.650303] [<ffffffff8175a29d>] tcp_write_timer_handler+0x13d/0x1a0 [ 85.650309] [<ffffffff8175a378>] tcp_write_timer+0x78/0x80 [ 85.650315] [<ffffffff81074f9c>] call_timer_fn+0x7c/0x100 [ 85.650321] [<ffffffff81074f20>] ? cascade+0xa0/0xa0 [ 85.650326] [<ffffffff8175a300>] ? tcp_write_timer_handler+0x1a0/0x1a0 [ 85.650332] [<ffffffff8175a300>] ? tcp_write_timer_handler+0x1a0/0x1a0 [ 85.650338] [<ffffffff81075237>] run_timer_softirq+0x217/0x250 [ 85.650344] [<ffffffff8106f3f9>] __do_softirq+0xc9/0x1a0 [ 85.650350] [<ffffffff817d393c>] call_softirq+0x1c/0x30 [ 85.650356] [<ffffffff8100fd95>] do_softirq+0x85/0xf0 [ 85.650362] [<ffffffff8106f26e>] irq_exit+0x9e/0xd0 [ 85.650369] [<ffffffff8134659f>] xen_evtchn_do_upcall+0x2f/0x40 [ 85.650374] [<ffffffff817d399e>] xen_do_hypervisor_callback+0x1e/0x30 [ 85.650379] <EOI> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 [ 85.650389] [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 [ 85.650396] [<ffffffff81008850>] ? xen_safe_halt+0x10/0x20 [ 85.650402] [<ffffffff810170f0>] ? default_idle+0x40/0x90 [ 85.650408] [<ffffffff810174a6>] ? cpu_idle+0x96/0xf0 [ 85.650415] [<ffffffff817a9c8c>] ? rest_init+0xbc/0xd0 [ 85.650420] [<ffffffff817a9bd0>] ? csum_partial_copy_generic+0x170/0x170 [ 85.650428] [<ffffffff81ce4be7>] ? start_kernel+0x390/0x39d [ 85.650433] [<ffffffff81ce4677>] ? repair_env_string+0x5b/0x5b [ 85.650440] [<ffffffff81ce4356>] ? x86_64_start_reservations+0x131/0x136 [ 85.650446] [<ffffffff81ce7915>] ? xen_start_kernel+0x54e/0x550 [ 85.650451] ---[ end trace ce03efc8ae5a5e98 ]--- [ 85.806018] ------------[ cut here ]------------ [ 85.806044] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860() [ 85.806051] Modules linked in: [ 85.806058] Pid: 2019, comm: postgres Tainted: G W 3.7.0-rc5-20121116-vanilla-warn #1 [ 85.806065] Call Trace: [ 85.806073] [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0 [ 85.806080] [<ffffffff81067555>] warn_slowpath_null+0x15/0x20 [ 85.806086] [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860 [ 85.806103] [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460 [ 85.806109] [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290 [ 85.806114] [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0 [ 85.806119] [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460 [ 85.806125] [<ffffffff810abab5>] ? trace_softirqs_off+0x85/0x1b0 [ 85.806132] [<ffffffff81740436>] ip_finish_output+0x226/0x530 [ 85.806138] [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530 [ 85.806143] [<ffffffff81740799>] ip_output+0x59/0xe0 [ 85.806148] [<ffffffff8173f2b8>] ip_local_out+0x28/0x90 [ 85.806153] [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0 [ 85.806157] [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340 [ 85.806163] [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0 [ 85.806168] [<ffffffff816962c9>] ? __skb_clone+0x29/0x120 [ 85.806174] [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0 [ 85.806179] [<ffffffff8175891a>] tcp_write_xmit+0x21a/0xa50 [ 85.806184] [<ffffffff8175917b>] tcp_push_one+0x2b/0x40 [ 85.806190] [<ffffffff81749cdc>] tcp_sendmsg+0x8dc/0xe20 [ 85.806195] [<ffffffff8176fe09>] inet_sendmsg+0xa9/0x100 [ 85.806200] [<ffffffff8176fd60>] ? inet_autobind+0x70/0x70 [ 85.806206] [<ffffffff8168dd12>] sock_sendmsg+0x82/0xb0 [ 85.806211] [<ffffffff8168ddb5>] ? sockfd_lookup_light+0x25/0x80 [ 85.806216] [<ffffffff8168e5e4>] sys_sendto+0x104/0x140 [ 85.806223] [<ffffffff817d26a9>] system_call_fastpath+0x16/0x1b [ 85.806227] ---[ end trace ce03efc8ae5a5e99 ]--- [ 85.806855] ------------[ cut here ]------------ [ 85.806871] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860() [ 85.806879] Modules linked in: [ 85.806884] Pid: 2019, comm: postgres Tainted: G W 3.7.0-rc5-20121116-vanilla-warn #1 [ 85.806891] Call Trace: [ 85.806898] [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0 [ 85.806905] [<ffffffff81067555>] warn_slowpath_null+0x15/0x20 [ 85.806911] [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860 [ 85.806917] [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460 [ 85.806923] [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290 [ 85.806929] [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0 [ 85.806935] [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460 [ 85.806941] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.806947] [<ffffffff81740436>] ip_finish_output+0x226/0x530 [ 85.806953] [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530 [ 85.806959] [<ffffffff81740799>] ip_output+0x59/0xe0 [ 85.806964] [<ffffffff8173f2b8>] ip_local_out+0x28/0x90 [ 85.806970] [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0 [ 85.806975] [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340 [ 85.806982] [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0 [ 85.806987] [<ffffffff816962c9>] ? __skb_clone+0x29/0x120 [ 85.806993] [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0 [ 85.806999] [<ffffffff81758026>] tcp_retransmit_skb+0x1c6/0x5a0 [ 85.807005] [<ffffffff81094806>] ? finish_task_switch+0x46/0xe0 [ 85.807011] [<ffffffff8175859e>] tcp_xmit_retransmit_queue+0x19e/0x300 [ 85.807018] [<ffffffff8175070b>] tcp_fastretrans_alert+0x76b/0xcb0 [ 85.807024] [<ffffffff817515fc>] tcp_ack+0x9ac/0x1150 [ 85.807030] [<ffffffff81752663>] ? tcp_validate_incoming+0x2d3/0x350 [ 85.807036] [<ffffffff81754778>] tcp_rcv_established+0x318/0x640 [ 85.807042] [<ffffffff8106f776>] ? local_bh_enable+0xc6/0x160 [ 85.807048] [<ffffffff8175c3a5>] tcp_v4_do_rcv+0x135/0x480 [ 85.807054] [<ffffffff8106f08f>] ? __local_bh_disable+0x8f/0xe0 [ 85.807061] [<ffffffff817483db>] tcp_prequeue_process+0x6b/0xa0 [ 85.807067] [<ffffffff81748e40>] tcp_recvmsg+0x8f0/0xb70 [ 85.807072] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.807078] [<ffffffff8176ec2a>] inet_recvmsg+0xba/0x100 [ 85.807084] [<ffffffff8176fd60>] ? inet_autobind+0x70/0x70 [ 85.807090] [<ffffffff8168dbf9>] sock_recvmsg+0x89/0xc0 [ 85.807096] [<ffffffff81116b0f>] ? tlb_flush_mmu+0x5f/0xa0 [ 85.807102] [<ffffffff810b171b>] ? __lock_acquire+0x46b/0xdd0 [ 85.807108] [<ffffffff8168ddb5>] ? sockfd_lookup_light+0x25/0x80 [ 85.807114] [<ffffffff8168e45f>] sys_recvfrom+0xbf/0x120 [ 85.807120] [<ffffffff811144db>] ? might_fault+0x3b/0x90 [ 85.807126] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.807133] [<ffffffff817d26a9>] system_call_fastpath+0x16/0x1b [ 85.807138] ---[ end trace ce03efc8ae5a5e9a ]--- [ 85.807159] ------------[ cut here ]------------ [ 85.807165] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860() [ 85.807171] Modules linked in: [ 85.807176] Pid: 2019, comm: postgres Tainted: G W 3.7.0-rc5-20121116-vanilla-warn #1 [ 85.807182] Call Trace: [ 85.807187] [<ffffffff8106750a>] warn_slowpath_common+0x7a/0xb0 [ 85.807193] [<ffffffff81067555>] warn_slowpath_null+0x15/0x20 [ 85.807199] [<ffffffff814b2b6e>] xennet_start_xmit+0x7fe/0x860 [ 85.807205] [<ffffffff816a6169>] dev_hard_start_xmit+0x209/0x460 [ 85.807211] [<ffffffff816c1e56>] sch_direct_xmit+0xf6/0x290 [ 85.807216] [<ffffffff816a6566>] dev_queue_xmit+0x1a6/0x5a0 [ 85.807222] [<ffffffff816a63c0>] ? dev_hard_start_xmit+0x460/0x460 [ 85.807228] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.807233] [<ffffffff81740436>] ip_finish_output+0x226/0x530 [ 85.807239] [<ffffffff817402dd>] ? ip_finish_output+0xcd/0x530 [ 85.807245] [<ffffffff81740799>] ip_output+0x59/0xe0 [ 85.807250] [<ffffffff8173f2b8>] ip_local_out+0x28/0x90 [ 85.807256] [<ffffffff8173f86f>] ip_queue_xmit+0x17f/0x4a0 [ 85.807261] [<ffffffff8173f6f0>] ? ip_send_unicast_reply+0x340/0x340 [ 85.807267] [<ffffffff810a1cf7>] ? getnstimeofday+0x47/0xe0 [ 85.807272] [<ffffffff816962c9>] ? __skb_clone+0x29/0x120 [ 85.807278] [<ffffffff81755940>] tcp_transmit_skb+0x400/0x8d0 [ 85.807283] [<ffffffff81758026>] tcp_retransmit_skb+0x1c6/0x5a0 [ 85.810124] [<ffffffff81094806>] ? finish_task_switch+0x46/0xe0 [ 85.810124] [<ffffffff8175859e>] tcp_xmit_retransmit_queue+0x19e/0x300 [ 85.810124] [<ffffffff8175070b>] tcp_fastretrans_alert+0x76b/0xcb0 [ 85.810124] [<ffffffff817515fc>] tcp_ack+0x9ac/0x1150 [ 85.810124] [<ffffffff81752663>] ? tcp_validate_incoming+0x2d3/0x350 [ 85.810124] [<ffffffff81754778>] tcp_rcv_established+0x318/0x640 [ 85.810124] [<ffffffff8106f776>] ? local_bh_enable+0xc6/0x160 [ 85.810124] [<ffffffff8175c3a5>] tcp_v4_do_rcv+0x135/0x480 [ 85.810124] [<ffffffff8106f08f>] ? __local_bh_disable+0x8f/0xe0 [ 85.810124] [<ffffffff817483db>] tcp_prequeue_process+0x6b/0xa0 [ 85.810124] [<ffffffff81748e40>] tcp_recvmsg+0x8f0/0xb70 [ 85.810124] [<ffffffff810b25e7>] ? lock_release+0x117/0x250 [ 85.810124] [<ffffffff8176ec2a>] inet_recvmsg+0xba/0x100 [ 85.810124] [<ffffffff8176fd60>] ? inet_autobind+0x70/0x70 [ 85.810124] [<ffffffff8168dbf9>] sock_recvmsg+0x89/0xc0 [ 85.810124] [<ffffffff81116b0f>] ? tlb_flush_mmu+0x5f/0xa0 [ 85.8