I''ve been working on getting networking functioning in translated
shadow
mode, and have it to the point where it''s almost working--some packets
get through some of the time before the machine crashes.
In an effort to narrow down the problem, I''ve found that the grant
table
network interface code in xen-unstable.hg seems to have some stability
problems as well. Here''s one of them: this is with a mostly unmodified
checkout of xen-unstable.hg (from yesterday evening), patched to produce
more debugging output and also with IP checksumming optimizations
disabled (since I was seeing some trouble with those). After a very
short while, I get a Dom-0 crash. This transcript is taken from
Domain-0, pinging an unprivileged domain (no shadow modes enabled):
potemkin58:~# ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
(XEN) gnttab_donate: i=0 mfn=000112aa domid=1 gref=000003c1
(XEN) (file=grant_table.c, line=1146) gnttab_prepare_for_transfer rd(1)
ld(0) ref(961).
(XEN) (file=grant_table.c, line=1225) gnttab_notify_transfer rd(1) ld(0)
ref(961).
(XEN) (file=grant_table.c, line=423) Mapping grant ref (706) for domain (1)
with flags (6)
(XEN) (file=grant_table.c, line=119) activate_grant_ref: mapping=0
granting=1
(XEN) (file=grant_table.c, line=315) activate_grant_ref: frame=78807
(XEN) (file=grant_table.c, line=455) map_grant_ref: frame=78807
vaddr=df807000 handle=2
(XEN) gnttab_donate: i=0 mfn=0007689f domid=1 gref=000003c2
(XEN) (file=grant_table.c, line=1146) gnttab_prepare_for_transfer rd(1)
ld(0) ref(962).
(XEN) (file=grant_table.c, line=1225) gnttab_notify_transfer rd(1) ld(0)
ref(962).
(XEN) (file=grant_table.c, line=535) Unmapping grant ref (706) for domain
(1) with handle (2)
(XEN) (file=grant_table.c, line=652) unmap_grant_ref: frame=78807
(XEN) (file=grant_table.c, line=423) Mapping grant ref (706) for domain (1)
with flags (6)
(XEN) (file=grant_table.c, line=119) activate_grant_ref: mapping=0
granting=1
(XEN) (file=grant_table.c, line=315) activate_grant_ref: frame=7b6b0
(XEN) (file=grant_table.c, line=455) map_grant_ref: frame=7b6b0
vaddr=df808000 handle=2
kernel BUG at include/linux/skbuff.h:1148 (kmap_skb_frag)!
[<c03c2a0b>] skb_checksum+0x27b/0x310
[<c040228b>] icmp_rcv+0x16b/0x1a0
[<c03dbebb>] ip_local_deliver+0xdb/0x220
[<c03dc33e>] ip_rcv+0x33e/0x4b0
[<c03dc630>] ip_rcv_finish+0x0/0x250
[<c03c7d64>] netif_receive_skb+0x204/0x270
[<c03c7e89>] process_backlog+0xb9/0x190
[<c03c800d>] net_rx_action+0xad/0x1a0
[<c0123f35>] __do_softirq+0xc5/0xf0
[<c0123feb>] do_softirq+0x8b/0x90
[<c01240b5>] irq_exit+0x35/0x40
[<c010f082>] do_IRQ+0x22/0x30
[<c0106530>] evtchn_do_upcall+0x70/0xa0
[<c010a758>] hypervisor_callback+0x2c/0x34
[<c01064ba>] force_evtchn_callback+0xa/0x10
[<c014b72f>] __pagevec_lru_add+0x15f/0x1c0
[<c013fb46>] add_to_page_cache+0x76/0xf0
[<c018957d>] mpage_readpages+0x18d/0x190
[<c01cd850>] ext3_get_block+0x0/0xc0
[<c0147e74>] read_pages+0x124/0x170
[<c01cd850>] ext3_get_block+0x0/0xc0
[<c0145573>] __alloc_pages+0x2e3/0x430
[<c0147fe0>] __do_page_cache_readahead+0x120/0x230
[<c01415cf>] filemap_nopage+0x2ef/0x410
[<c01528f8>] do_no_page+0xb8/0x3b0
[<c01502a3>] pte_alloc_map+0x93/0x210
[<c0152e36>] handle_mm_fault+0xf6/0x240
[<c01178ec>] do_page_fault+0x19c/0x5f2
[<c0114246>] old_mmap+0xd6/0x110
[<c010a93a>] page_fault+0x2e/0x34
Kernel panic - not syncing: BUG!
(XEN) Domain 0 shutdown: rebooting machine.
The line causing trouble is "BUG_ON(in_irq())". In this example, I
had
tcpdump running in both domains; this seems to trigger the problem more
reliably. I''ve also seen a similar crash with a TCP connection, but it
takes a few packets before this shows up (the handshake completes, and
the crash happens about the time data packets come back from domain-0;
if checksumming optimizations are enabled, it seems the packets are
dropped so I don''t see a crash but I don''t get any data
either).
I''ve been having a difficult time tracking this down, so any help is
appreciated.
--Michael Vrable
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 13 Aug 2005, at 19:59, Michael Vrable wrote:> The line causing trouble is "BUG_ON(in_irq())". In this example, I had > tcpdump running in both domains; this seems to trigger the problem more > reliably. I''ve also seen a similar crash with a TCP connection, but it > takes a few packets before this shows up (the handshake completes, and > the crash happens about the time data packets come back from domain-0; > if checksumming optimizations are enabled, it seems the packets are > dropped so I don''t see a crash but I don''t get any data either).On the stack trace, at irq_exit() you definitely have no hardirqs or softirqs in progress. But somehow, at kmap_skb_frag(), the hardirq section of the preempt mask has become non-zero. You can''t have been preempted to another cpu during any of this because the preempt mask is continuously non-zero throughout original irq handling and subsequent softirq handling. The only code between irq_exit and kmap_skb_frag on the stack trace is unmodified Linux code. Assuming that is all correct (and presumably the same whether we enable grant tables or not) I might guess another interrupt arrives and the handler corrupts things? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Aug 14, 2005 at 09:29:02AM +0100, Keir Fraser wrote:> On 13 Aug 2005, at 19:59, Michael Vrable wrote: > > >The line causing trouble is "BUG_ON(in_irq())". In this example, I had > >tcpdump running in both domains; this seems to trigger the problem more > >reliably. I''ve also seen a similar crash with a TCP connection, but it > >takes a few packets before this shows up (the handshake completes, and > >the crash happens about the time data packets come back from domain-0; > >if checksumming optimizations are enabled, it seems the packets are > >dropped so I don''t see a crash but I don''t get any data either). > > On the stack trace, at irq_exit() you definitely have no hardirqs or > softirqs in progress. But somehow, at kmap_skb_frag(), the hardirq > section of the preempt mask has become non-zero. You can''t have been > preempted to another cpu during any of this because the preempt mask is > continuously non-zero throughout original irq handling and subsequent > softirq handling. > > The only code between irq_exit and kmap_skb_frag on the stack trace is > unmodified Linux code. Assuming that is all correct (and presumably the > same whether we enable grant tables or not) I might guess another > interrupt arrives and the handler corrupts things?I discovered the cause of this and other crashes yesterday: when grant tables are enabled in the netback driver, net_tx_action() and net_tx_action_dealloc() in netback.c each allocate large arrays from the kernel''s stack ("gnttab_map_grant_ref_t map_ops[MAX_PENDING_REQS]" and "gnttab_unmap_grant_ref_t unmap_ops[MAX_PENDING_REQS]"). This results in a stack overflow. This in turn causes a number of very spectacular crashes. One of the more subtle crashes is the in_irq() bug; the preempt count is stored in the current thread info, at the bottom of the stack. Allocating the arrays statically fixes the problem for me. Steve Hand says he''ll likely be committing a fix soon. --Michael Vrable _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote:> Allocating the arrays statically fixes the problem for me. Steve Hand > says he''ll likely be committing a fix soon.Never mind; catching up on the changes to xen-unstable.hg, I see that you''ve already committed a fix. Thanks. --Michael Vrable _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote: > > Allocating the arrays statically fixes the problem for me. > Steve Hand > > says he''ll likely be committing a fix soon. > > Never mind; catching up on the changes to xen-unstable.hg, I > see that you''ve already committed a fix. Thanks.Michael, if this fix is working fine for you, I minded to enable them in the default config... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote: >> > Allocating the arrays statically fixes the problem for me. >> > Steve Hand says he''ll likely be committing a fix soon. >> >> Never mind; catching up on the changes to xen-unstable.hg, I >> see that you''ve already committed a fix. Thanks. > >Michael, if this fix is working fine for you, I minded to enable them in >the default config...Good plan - seems to work fine on driller (the machine which reliably reproduced this bug for me) under limited stress testing. I have a few more bits to check in (using grants for the shmem frames) and can enable net grant in the default config then. cheers, S. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Aug 14, 2005 at 04:59:53PM +0100, Ian Pratt wrote:> > On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote: > > > Allocating the arrays statically fixes the problem for me. > > Steve Hand > > > says he''ll likely be committing a fix soon. > > > > Never mind; catching up on the changes to xen-unstable.hg, I > > see that you''ve already committed a fix. Thanks. > > Michael, if this fix is working fine for you, I minded to enable them in > the default config...I haven''t tested this particular commit, since I''m currently using a testing a version of Xen that is patched to run unprivileged domains in shadow translated mode, with grant-table based networking. However, the equivalent patch in my tree has fixed my networking stability issues. There are still a few robustness issues to work on. For example, the BUG() when grant table donate operations fail in net_rx_action, which can legitimately happen if the unprivileged domain hasn''t set up its grant tables properly. I haven''t seen this come up in normal use, but does need to be fixed before the final release, since it will let an unprivileged domain crash domain-0. (I have seen it in my work, but then I''m not doing normal things with unprivileged domains.) I''d either vote for enabling grant tables now to get some wider testing, but with a warning that some corner cases still need some work, or do a check of the places that netback calls BUG() first. --Michael Vrable _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Aug 14, 2005 at 09:27:12AM -0700, Michael Vrable wrote:> I''d either vote for enabling grant tables now to get some wider testing, > but with a warning that some corner cases still need some work, or do a > check of the places that netback calls BUG() first.To follow up: if enabling grant tables, at least the BUG() in the transmit-to-domU path should be replaced with something less fatal. Full error handling and recovery (including cleaning up resources) could be done in parallel with some wider testing. If Xen 3.0 is being used in some mostly-production environments, we may want to hold off just a small bit longer. --Michael Vrable _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Vrable wrote:> On Sun, Aug 14, 2005 at 09:29:02AM +0100, Keir Fraser wrote: >>On 13 Aug 2005, at 19:59, Michael Vrable wrote: >> >>>The line causing trouble is "BUG_ON(in_irq())". [...] >> >>The only code between irq_exit and kmap_skb_frag on the stack trace is >>unmodified Linux code. Assuming that is all correct (and presumably the >>same whether we enable grant tables or not) I might guess another >>interrupt arrives and the handler corrupts things? > > I discovered the cause of this and other crashes yesterday: when grant > tables are enabled in the netback driver, net_tx_action() and > net_tx_action_dealloc() in netback.c each allocate large arrays from the > kernel''s stack ("gnttab_map_grant_ref_t map_ops[MAX_PENDING_REQS]" and > "gnttab_unmap_grant_ref_t unmap_ops[MAX_PENDING_REQS]"). This results > in a stack overflow.Is there no way to make a kernel stack overflow fail fast with an obvious error, rather than causing other subtle failures? -- David Hopwood <david.nospam.hopwood@blueyonder.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel