xen discuss - Jan 2008 - nfs / diskless opensolaris PV domU hangs, with vcpus=2 ?

Can anyone else reproduce opensolaris PV domU hangs
during domU boot, when the domU is using an root filesystem
on an nfs server and the domU is configured with more than
one vcpu?


Test system 1:

- AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ / 2GB memory
- dom0 running opensolaris snv_83 bits (64-bit)
- PV domU opensolaris snv_83 bits (64-bit), with an nfs root filesystem

Test system 2:

- Core(TM)2 Quad CPU Q6600 / 8GB memory
- dom0 running gentoo 2.6.20 linux (32-bit)
- PV domU opensolaris snv_83 bits (32-bit), with an nfs root filesystem


Config file for the PV domU is something like this:

# cat /etc/xen/solaris-nfs
name            = "solaris"
memory          = 512
vcpus           = 1
localtime       = 1

kernel          =
''/export/root/xen/platform/i86xpv/kernel/amd64/unix''
ramdisk         =
''/export/root/xen/platform/i86pc/amd64/boot_archive''
extra           = ''/platform/i86xpv/kernel/amd64/unix -kv''

nfs_server      = ''192.168.2.20''
nfs_root        = ''moritz:/export/root/xen''

vif             = [ ''mac=0:8:c7:d2:82:63'' ]

on_shutdown     = ''destroy''
on_reboot       = ''destroy''
on_crash        = ''destroy''



With this configuration (note: vcpus=1) the snv_83 domU boots ok.



But with vcpus=2, the domU hangs during the boot, after / while
cpu#1 is initialized:

# xm create solaris-nfs -c vcpus=2
Using config file "/etc/xen/solaris-nfs".
Started domain solaris
module /platform/i86xpv/kernel/amd64/unix: text at [0xfffffffffb800000,
0xfffffffffb9205f3] data at 0xfffffffffbc00000
module /kernel/amd64/genunix: text at [0xfffffffffb920600, 0xfffffffffbb55047]
data at 0xfffffffffbca2000
Loading kmdb...
module /kernel/misc/amd64/kmdbmod: text at [0xfffffffffbb55050,
0xfffffffffbbe42bf] data at 0xfffffffffbd0aa70
module /kernel/misc/amd64/ctf: text at [0xfffffffffbbe42c0, 0xfffffffffbbee1bf]
data at 0xfffffffffbd25d40
v3.0.4-1-xvm chgset ''Mon Oct 29 23:19:32 2007 -0700
13226:b0c4d9d28fee''
SunOS Release 5.11 Version wos_b83 64-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
features: 10e66c6<cpuid,cx16,sse3,nx,sse2,sse,cx8,pae,mmx,cmov,msr,tsc>
mem = 524288K (0x20000000)
root nexus = i86xpv
pseudo0 at root
pseudo0 is /pseudo
scsi_vhci0 at root
scsi_vhci0 is /scsi_vhci
pseudo-device: dld0
dld0 is /pseudo/dld@0
xpvd0 at root
/xpvd/xnf@0 (xnf0) online
Requesting Internet address for 0:8:c7:d2:82:63
Found my IP address: 2802a8c0 (192.168.2.40)
hostname: xen
domainname: gandalf.net
root on moritz:/export/root/xen fstype nfsdyn
/cpus (cpunex0) online
xencons@0, xencons0
xencons0 is /xpvd/xencons@0
cpu0: x86 (AuthenticAMD 40FB2 family 15 model 75 step 2 clock 2210 MHz)
cpu0: AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
cpu1: x86 (AuthenticAMD 40FB2 family 15 model 75 step 2 clock 2210 MHz)
cpu1: AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
NFS server moritz not responding still trying
     <<<<< hangs here / no more progress >>>>>




Workaround: Disable the "cpudrv" module;
now the domU is able to boot with vcpus=2:

extra = ''/platform/i86xpv/kernel/amd64/unix -B disable-cpudrv=true
-kv''
 
 
This message posted from opensolaris.org

On Wed, Jan 30, 2008 at 08:35:13AM -0800, J??rgen Keil
wrote:
> Can anyone else reproduce opensolaris PV domU hangs
> during domU boot, when the domU is using an root filesystem
> on an nfs server and the domU is configured with more than
> one vcpu?
I don''t have this set up at the moment, but it definitely worked ~6
months ago (even 32 way).

David wrote:
> On Wed, Jan 30, 2008 at 08:35:13AM -0800, Jürgen Keil wrote:
> > Can anyone else reproduce opensolaris PV domU hangs
> > during domU boot, when the domU is using an root filesystem
> > on an nfs server and the domU is configured with more than
> > one vcpu?
> 
> I don't have this set up at the moment, but it definitely worked ~6
> months ago (even 32 way).
Hmm, this could be a generic S-x86 mp architecture bug (?).

Under xVM, this happens:

- mp_startup() is called to startup cpu#1

- in mp_startup(),  "(*ap_mlsetup)()" is called,
  which calls xen's xen_psm_post_cpu_start()

- in xen_psm_post_cpu_start() we have this:

        /*
         * Re-distribute interrupts to include the newly added cpu.
         */
        xen_psm_enable_intr(cpun);


   In my setup, this re-binds netfront's interrupt handler
   xnf`xnf_intr() from cpu0 to the new cpu1.

   (This might have changed in snv_77, with the fix for
   6611846 "after boot, all dom0 interrupts are targeting
   CPU 0 in a MP system" - this could explain why it
   did work for you ~6 month ago).

- later on, in mp_setup() it raises the spl for the new cpu1
  to LOCK_LEVEL, and enables interrupts. But at
  spl == LOCK_LEVEL, xnf_intr should be masked.

  add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi)
  is called.  This tries to load & attach the "cpudrv" kernel
  module (while we're still at spl == LOCK_LEVEL on cpu1). 
  It sends packes out of the domU, but the replies from
  the NFS server are never seen by xnf`xnf_intr, which is
  masked.

When the domU is hung, it see this:

[1]> ::cpuinfo -v
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc3fff0  1b    0    0  -1   no    no t-0    ffffff0001005c80
 (idle)
                       |
            RUNNING <--+
              READY
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 ffffff0086199ac0  1b    0   10  60   no    no t-0    ffffff00010cbc80
                       |
            RUNNING <--+
              READY
             EXISTS
             ENABLE
[1]> ::interrupts
IRQ  Vect Evtchn IPL Bus    Trg Type   CPU Share APIC/INT# ISR(s)
256  -    I      15  -      Edg ipi    all -     -         xc_serv
257  -    I      13  -      Edg ipi    all -     -         xc_serv
258  -    I      11  -      Edg ipi    all -     -         poke_cpu
259  -    1      15  -      Edg virq   all -     -         xen_debug_handler
260  -    1      1   -      Edg evtchn 0   -     -         xenbus_intr
261  -    T      14  -      Edg virq   all -     -         cbe_fire
262  -    I      14  -      Edg ipi    all -     -         cbe_fire
263  -    9      6   xpvd   Edg evtchn 1   -     -         xnf`xnf_intr
264  -    2      9   xpvd   Edg evtchn 0   -     -         xencons`xenconsintr
[1]> ::evtchns
Type          Evtchn IRQ IPL CPU Masked Pending ISR(s)
evtchn        1      260 1   0   0      0       xenbus_intr
evtchn        2      264 9   0   0      1       xencons`xenconsintr
ipi           3      256 15  0   1      0       xc_serv
ipi           4      257 13  0   0      0       xc_serv
ipi           5      258 11  0   0      0       poke_cpu
virq:debug    6      259 15  0   0      0       xen_debug_handler
virq:timer    7      261 14  0   1      1       cbe_fire
ipi           8      262 14  0   0      0       cbe_fire
evtchn        9      263 6   1   1      1       xnf`xnf_intr
ipi           10     258 11  1   0      0       poke_cpu
ipi           11     257 13  1   0      0       xc_serv
ipi           12     262 14  1   0      0       cbe_fire
ipi           13     256 15  1   0      0       xc_serv
virq:timer    14     261 14  1   1      1       cbe_fire


A possible fix could be to move the add_cpunode2devtree()
call down a few lines in mp_startup(), after the spl0():

diff -r f6814e9b7def usr/src/uts/i86pc/os/mp_startup.c
--- a/usr/src/uts/i86pc/os/mp_startup.c Wed Jan 30 09:01:17 2008 -0800
+++ b/usr/src/uts/i86pc/os/mp_startup.c Thu Jan 31 01:00:58 2008 +0100
@@ -1518,13 +1518,15 @@ mp_startup(void)
         */
        curthread->t_preempt = 0;

-       add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi);
+       /* add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi); */

        /* The base spl should still be at LOCK LEVEL here */
        ASSERT(cp->cpu_base_spl == ipltospl(LOCK_LEVEL));
        set_base_spl();         /* Restore the spl to its proper value */

        (void) spl0();                          /* enable interrupts */
+
+       add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi);

 #ifndef __xpv
        {
 
 
This message posted from opensolaris.org
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

I wrote:
> David wrote:
> 
> > On Wed, Jan 30, 2008 at 08:35:13AM -0800, Jürgen Keil wrote:
> > > Can anyone else reproduce opensolaris PV domU hangs
> > > during domU boot, when the domU is using an root filesystem
> > > on an nfs server and the domU is configured with more than
> > > one vcpu?
> > 
> > I don't have this set up at the moment, but it definitely worked
~6
> > months ago (even 32 way).
> 
> Hmm, this could be a generic S-x86 mp architecture bug (?).
> 
> Under xVM, this happens:
The same hang occurs on a different MP machine,
on metal, when booting diskless:

- Intel(r) CPU T2500  @ 2.00GHz, dual core cpu
- 2GB memory
- Mainboard: N4L-VM-ASUS-DH

This box is hanging when cpu1 is started.

Using a kernel which has the add_cpunode2devtree()
call moved after the spl0() call in mp_startup(), and
with ucode_check() disabled ("ret" instruction patched
at the start of ucode_check()), the above system is
booting diskless just fine.

See also:
http://www.opensolaris.org/jive/thread.jspa?messageID=198974#198974


It seems the code in mp_startup() should either:

avoid calling functions that load files (e.g. microcode
update files) / kernel modules ("cpudrv") (or could
trigger device interrupts) while a non-boot cpu is started,
and has the spl raised, and already has interrupt
handlers re-assigned to the new cpu

Or:

re-distribute interrupts to the new cpu only after 
the spl is lowered to 0, in mp_startup().
 
 
This message posted from opensolaris.org
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

> Can anyone else reproduce opensolaris PV domU hangs
> during domU boot, when the domU is using an root filesystem
> on an nfs server and the domU is configured with more
> than one vcpu?
I filed a new bug for this one, 6657646.
 
 
This message posted from opensolaris.org