xen discuss - Oct 2009 - after update to last xvm-3.4 gate windows hvm crash on boot

Around midnight (Europe/Berlin), so ~10h back, I re-synced with xvm-3.4
gate and rebuild the packages. This morning around 7:30 I have shutdown
all domains and reinstalled the packages using pkgadd -d . in
packages-nodebug. After rebooting the system the linux pv guest booted
fine. Starting the windows guests (xm start --vncviewer titan). The
domain inits fast and brings me the vncWindow, bootloading works
(windows boot menu and so on). On Windows boot (2008r2 x64) the domain
crashes immediately, also the mentioned boot recovery fails to boot in
the same way.

Florian

Florian Manschwetus wrote:
> Around midnight (Europe/Berlin), so ~10h back, I re-synced with xvm-3.4
> gate and rebuild the packages. This morning around 7:30 I have shutdown
> all domains and reinstalled the packages using pkgadd -d . in
> packages-nodebug. After rebooting the system the linux pv guest booted
> fine. Starting the windows guests (xm start --vncviewer titan). The
> domain inits fast and brings me the vncWindow, bootloading works
> (windows boot menu and so on). On Windows boot (2008r2 x64) the domain
> crashes immediately, also the mentioned boot recovery fails to boot in
> the same way.
> 
I see the following in the log...


    VmError: Disk image does not exist: /export/media_images/win2008r2.iso



MRJ

Mark Johnson wrote:
> 
> 
> Florian Manschwetus wrote:
>> Around midnight (Europe/Berlin), so ~10h back, I re-synced with xvm-3.4
>> gate and rebuild the packages. This morning around 7:30 I have shutdown
>> all domains and reinstalled the packages using pkgadd -d . in
>> packages-nodebug. After rebooting the system the linux pv guest booted
>> fine. Starting the windows guests (xm start --vncviewer titan). The
>> domain inits fast and brings me the vncWindow, bootloading works
>> (windows boot menu and so on). On Windows boot (2008r2 x64) the domain
>> crashes immediately, also the mentioned boot recovery fails to boot in
>> the same way.
>>
> 
> I see the following in the log...
> 
> 
>    VmError: Disk image does not exist: /export/media_images/win2008r2.iso
ah, that''s an earlier attempt...  Yeah, it looks like qemu is
crashing...

   [2009-10-22 08:57:33 648] WARNING (image:501) domain titan: device model
failure: pid 8237: malfunctioning (closed sentinel), killed; see
/var/log/xen/qemu-dm-titan.log
   [2009-10-22 08:57:33 648] WARNING (XendDomainInfo:1905) Domain has crashed:
name=titan id=63.


Can your run the following
  coreadm -e global
  coreadm -e global-setid
  coreadm -g /var/cores/core.%f.%p
  coreadm

Then try staring the guest again...

Then run pstack on the qemu core
   pstack /var/core/core.qemu-dm<>




Thanks,

MRJ

Florian Manschwetus wrote:
> Around midnight (Europe/Berlin), so ~10h back, I re-synced with xvm-3.4
> gate and rebuild the packages. This morning around 7:30 I have shutdown
> all domains and reinstalled the packages using pkgadd -d . in
> packages-nodebug. After rebooting the system the linux pv guest booted
> fine. Starting the windows guests (xm start --vncviewer titan). The
> domain inits fast and brings me the vncWindow, bootloading works
> (windows boot menu and so on). On Windows boot (2008r2 x64) the domain
> crashes immediately, also the mentioned boot recovery fails to boot in
> the same way.
When was the last time you synced up before that?

I pushed the following fix around 11:50PM your time yesterday.
qemu 3.4 has an update device module for uhci which doesn''t quite
work right...


Author: Mark Johnson <Mark.Johnson@Sun.COM>
Repository: /hg/xen-gate/xvm-3.4+qemu-patches.hg
Latest revision: 1c1781683db6cbc472f5c40d1a9bff8a691c64c1
Total changesets: 1
Log message:
revert qemu uhci device. New implementation doesn''t work on Solaris
guests

Files:
	create: revert-uhci
	update: series



It''s possible that reverting to the old model confused windows
on you or caused windows to do something that crashed qemu?

You can try removing USB temporarily from the guest and see
if that helps...

i.e.

    xm list -l <domain> > /tmp/win.sxp
    scp /tmp/win.sxp /win.sxp.backup
    edit /tmp/win.sxp
       remove (usbdevice tablet)
       change (usb 1) to (usb 0)
    xm new -F /tmp/win.sxp

then boot and see if it still crashes. If not,
trying reverting back to the back up and see if
it works now?



MRJ

Am 22.10.2009 13:26, schrieb Mark Johnson:
> ah, that''s an earlier attempt...  Yeah, it looks like qemu is
crashing...
> 
>   [2009-10-22 08:57:33 648] WARNING (image:501) domain titan: device
> model failure: pid 8237: malfunctioning (closed sentinel), killed; see
> /var/log/xen/qemu-dm-titan.log
>   [2009-10-22 08:57:33 648] WARNING (XendDomainInfo:1905) Domain has
> crashed: name=titan id=63.
> 
> 
> Can your run the following
>  coreadm -e global
>  coreadm -e global-setid
>  coreadm -g /var/cores/core.%f.%p
>  coreadm
root@jupiter:~# coreadm -e global
root@jupiter:~# coreadm -e global-setid
root@jupiter:~# coreadm -g /var/cores/core.%f.%p
root@jupiter:~# coreadm
     global core file pattern: /var/cores/core.%f.%p
     global core file content: default
       init core file pattern: core
       init core file content: default
            global core dumps: enabled
       per-process core dumps: enabled
      global setid core dumps: enabled
 per-process setid core dumps: disabled
     global core dump logging: disabled
> 
> Then try staring the guest again...
> 
> Then run pstack on the qemu core
>   pstack /var/core/core.qemu-dm<>
> 
root@jupiter:~# pstack /var/cores/core.qemu-dm.7331
core ''/var/cores/core.qemu-dm.7331'' of 7331:   
/usr/lib/xen/bin/qemu-dm
-d 5 -domain-name titan -videoram 4 -k de -vn
-----------------  lwp# 1 / thread# 1  --------------------
 000000000094a640 ???????? ()
 000000000046d5a2 main_loop_wait () + 2e2
 00000000004ceb6a main_loop () + ba
 000000000046f3db main () + 1bdb
 0000000000469f9c _start () + 6c
-----------------  lwp# 2 / thread# 2  --------------------
 00007fffff23bd1a __read () + a
 00007ffffe1b3306 read_all () + 26
 00007ffffe1b35d8 read_message () + 48
 00007ffffe1b45f8 read_thread () + 18
 00007fffff232b34 _thrp_setup () + bc
 00007fffff232df0 _lwp_start ()
-----------------  lwp# 3 / thread# 3  --------------------
 00007fffff232e37 __lwp_park () + 17
 00007fffff22c1e0 cond_wait_queue () + 68
 00007fffff22c649 cond_wait_common () + 1e1
 00007fffff22c903 __cond_timedwait () + ab
 00007fffff22c94f cond_timedwait () + 27
 00007fffff22c981 pthread_cond_timedwait () + 9
 00000000004782e8 aio_thread () + 178
 00007fffff232b34 _thrp_setup () + bc
 00007fffff232df0 _lwp_start ()
root@jupiter:~#

Domain Config:
name = "titan"

uuid = "cedb6201-82d6-3c96-4da3-e0bfad6a7501"

maxmem = 2048

memory = 2048


#videoram = 32
hpet = 1
guest_os_type = "windows"

vcpus = 2
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "d"
pae = 1
acpi = 1
apic = 1
viridian = 1
localtime = 1
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib/xen/bin/qemu-dm"
usb = 1
usbdevice = "tablet"
vfb = [ "type=vnc,vncunused=1,keymap=de" ]
disk = [
        "phy:/dev/dsk/c0t600A0B800049E902000009294ACD08D6d0p0,hda,w"
        ,"file:/export/media_images/win2008r2_msnaa.iso,hdc:cdrom,r"
         ]
vif = [ "mac=00:16:3e:00:00:01,bridge=e1000g3" ]
parallel = "none"
serial = "pty"

florian

> Author: Mark Johnson <Mark.Johnson@Sun.COM>
> Repository: /hg/xen-gate/xvm-3.4+qemu-patches.hg
> Latest revision: 1c1781683db6cbc472f5c40d1a9bff8a691c64c1
> Total changesets: 1
> Log message:
> revert qemu uhci device. New implementation doesn''t work on
Solaris guests
> 
> Files:
>     create: revert-uhci
>     update: series
> 
> 
> 
> It''s possible that reverting to the old model confused windows
> on you or caused windows to do something that crashed qemu?
> 
> You can try removing USB temporarily from the guest and see
> if that helps...
> 
> i.e.
> 
>    xm list -l <domain> > /tmp/win.sxp
>    scp /tmp/win.sxp /win.sxp.backup
>    edit /tmp/win.sxp
>       remove (usbdevice tablet)
>       change (usb 1) to (usb 0)
>    xm new -F /tmp/win.sxp
> 
> then boot and see if it still crashes. If not,
> trying reverting back to the back up and see if
> it works now?
Tried and failed...

name = "titan"

uuid = "cedb6201-82d6-3c96-4da3-e0bfad6a7501"

maxmem = 2048

memory = 2048


#videoram = 32
hpet = 1
guest_os_type = "windows"

vcpus = 2
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "d"
pae = 1
acpi = 1
apic = 1
viridian = 1
localtime = 1
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib/xen/bin/qemu-dm"
usb = 0
#usbdevice = "tablet"
vfb = [ "type=vnc,vncunused=1,keymap=de" ]
disk = [
        "phy:/dev/dsk/c0t600A0B800049E902000009294ACD08D6d0p0,hda,w"
        ,"file:/export/media_images/win2008r2_msnaa.iso,hdc:cdrom,r"
         ]
vif = [ "mac=00:16:3e:00:00:01,bridge=e1000g3" ]
parallel = "none"
serial = "pty"


pstack /var/cores/core.qemu-dm.8061
core ''/var/cores/core.qemu-dm.8061'' of 8061:   
/usr/lib/xen/bin/qemu-dm
-d 7 -domain-name titan -videoram 4 -k de -vn
-----------------  lwp# 1 / thread# 1  --------------------
 000000000094a640 ???????? ()
 000000000046d5a2 main_loop_wait () + 2e2
 00000000004ceb6a main_loop () + ba
 000000000046f3db main () + 1bdb
 0000000000469f9c _start () + 6c
-----------------  lwp# 2 / thread# 2  --------------------
 00007fffff23bd1a __read () + a
 00007ffffe1b3306 read_all () + 26
 00007ffffe1b35d8 read_message () + 48
 00007ffffe1b45f8 read_thread () + 18
 00007fffff232b34 _thrp_setup () + bc
 00007fffff232df0 _lwp_start ()
-----------------  lwp# 3 / thread# 3  --------------------
 00007fffff232e37 __lwp_park () + 17
 00007fffff22c1e0 cond_wait_queue () + 68
 00007fffff22c649 cond_wait_common () + 1e1
 00007fffff22c903 __cond_timedwait () + ab
 00007fffff22c94f cond_timedwait () + 27
 00007fffff22c981 pthread_cond_timedwait () + 9
 00000000004782e8 aio_thread () + 178
 00007fffff232b34 _thrp_setup () + bc
 00007fffff232df0 _lwp_start ()
> 
> 
> 
> MRJ
>

Florian Manschwetus wrote:
> Am 22.10.2009 13:26, schrieb Mark Johnson:
>> ah, that''s an earlier attempt...  Yeah, it looks like qemu is
crashing...
>>
>>   [2009-10-22 08:57:33 648] WARNING (image:501) domain titan: device
>> model failure: pid 8237: malfunctioning (closed sentinel), killed; see
>> /var/log/xen/qemu-dm-titan.log
>>   [2009-10-22 08:57:33 648] WARNING (XendDomainInfo:1905) Domain has
>> crashed: name=titan id=63.
>>
>>
>> Can your run the following
>>  coreadm -e global
>>  coreadm -e global-setid
>>  coreadm -g /var/cores/core.%f.%p
>>  coreadm
> 
> root@jupiter:~# coreadm -e global
> root@jupiter:~# coreadm -e global-setid
> root@jupiter:~# coreadm -g /var/cores/core.%f.%p
> root@jupiter:~# coreadm
>      global core file pattern: /var/cores/core.%f.%p
>      global core file content: default
>        init core file pattern: core
>        init core file content: default
>             global core dumps: enabled
>        per-process core dumps: enabled
>       global setid core dumps: enabled
>  per-process setid core dumps: disabled
>      global core dump logging: disabled
> 
>> Then try staring the guest again...
>>
>> Then run pstack on the qemu core
>>   pstack /var/core/core.qemu-dm<>
>>
> root@jupiter:~# pstack /var/cores/core.qemu-dm.7331
> core ''/var/cores/core.qemu-dm.7331'' of 7331:   
/usr/lib/xen/bin/qemu-dm
> -d 5 -domain-name titan -videoram 4 -k de -vn
> -----------------  lwp# 1 / thread# 1  --------------------
>  000000000094a640 ???????? ()
>  000000000046d5a2 main_loop_wait () + 2e2
>  00000000004ceb6a main_loop () + ba
>  000000000046f3db main () + 1bdb
>  0000000000469f9c _start () + 6c
> -----------------  lwp# 2 / thread# 2  --------------------
>  00007fffff23bd1a __read () + a
>  00007ffffe1b3306 read_all () + 26
>  00007ffffe1b35d8 read_message () + 48
>  00007ffffe1b45f8 read_thread () + 18
>  00007fffff232b34 _thrp_setup () + bc
>  00007fffff232df0 _lwp_start ()
> -----------------  lwp# 3 / thread# 3  --------------------
>  00007fffff232e37 __lwp_park () + 17
>  00007fffff22c1e0 cond_wait_queue () + 68
>  00007fffff22c649 cond_wait_common () + 1e1
>  00007fffff22c903 __cond_timedwait () + ab
>  00007fffff22c94f cond_timedwait () + 27
>  00007fffff22c981 pthread_cond_timedwait () + 9
>  00000000004782e8 aio_thread () + 178
>  00007fffff232b34 _thrp_setup () + bc
>  00007fffff232df0 _lwp_start ()
> root@jupiter:~#
k, try

   mdb /var/cores/core.qemu-dm.7331
   > $c
...
   > $q


MRJ

Am 22.10.2009 15:05, schrieb Mark Johnson:
> mdb /var/cores/core.qemu-dm.7331
Loading modules: [ libc.so.1 ld.so.1 ]
> $c
0x94a640()
main_loop_wait+0x2e2()
main_loop+0xba()
main+0x1bdb()
_start+0x6c()
> $q