thr3ads.net - Xen users - PV DomU stopped responding, won''t boot, stuck in paused state [Mar 2012]

If this information is useful, please help other people find it:
Share via:

Mike Brancato

2012-Mar-07 21:07 UTC

PV DomU stopped responding, won''t boot, stuck in paused state

Yesterday, I had a PV OpenSolaris DomU stop responding.  I looked on my xen
machine and it was in a paused state.  I ended up destroying it and then doing
xm create to bring it back up.

It won''t boot, and I think PyGrub is hanging.  Not sure how to resolve
this, there aren''t any errors being presented in xen/xm itself.  If I
try ''xm create config.conf'' it just  says "using config
file blah.conf<newline>" and nothing else. Never fails, never boots. 
If I drop out "ctrl-c" and look, it is paused with 0 run time.

A few weeks ago I upgraded from Debian 5.0 to 6.0.  My Xen install was refreshed
with some updated packages, but I was already running 4.0.1 on Debian 5.0.  But
after the upgrade, everything was working fine.  No changes to this machine
since, just noticed it had stopped responding last night, but that DomU was
running earlier in the day.

Hopefully someone can help me figure out what''s wrong, I''m
guessing it has to do with pygrub since it won''t even begin the boot
process.  That still doesn''t explain why it stopped responding unless
there was some other failure that caused a reboot, but this machine has normally
run without incident for almost 3 years.  It *is* and old version of
OpenSolaris.

Here is some info on the Dom0 and DomU.

Debian 6.0 (squeeze)
Xen 4.0.1
My pygrub is named /usr/bin/pygrub-4.1b, I know I had to hack around with this
in the past to get ZFS support.

When I run pygrub-4.1b domu.img, I get an error now:
Traceback (most recent call last):
  File "/usr/bin/pygrub-4.1b", line 20, in <module>
    import xen.lowlevel.xc
ImportError: No module named xen.lowlevel.xc

If I try the provided PyGrub from debian, I get:
farnsworth:/vm/leela# xm create leela.conf 
Using config file "./leela.conf".
Error: Boot loader didn''t return any data!

But the provided pygrub does pull up the grub menu with ''pygrub
leela.img''.

DomU:
I pass through some physical drives to a zpool array, this machine is a storage
server and handles SMB/CIFS.

(domain
    (domid 7)
    (cpu_weight 256)
    (cpu_cap 0)
    (on_crash restart)
    (uuid blah)
    (bootloader_args -q)
    (vcpus 1)
    (name leela)
    (on_poweroff destroy)
    (on_reboot restart)
    (cpus (()))
    (description )
    (bootloader /usr/bin/pygrub-4.1b)
    (maxmem 1024)
    (memory 1024)
    (shadow_memory 0)
    (features )
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (cpu_time 0.0)
    (online_vcpus 0)
    (image
        (linux
            (kernel )
            (superpages 0)
            (tsc_mode 0)
            (videoram 4)
            (pci ())
            (nomigrate 0)
        )
    )
    (status 1)
    (state --p---)
    (device
        (vif
            (bridge eth0)
            (mac blah)
            (uuid blah)
        )
    )
    (device
        (vbd
            (uuid blah)
            (bootable 1)
            (driver paravirtualised)
            (dev xvda)
            (uname file:/vm/leela/leela.img)
            (mode w)
        )
    )
    (device
        (vbd
            (uuid blah)
            (bootable 0)
            (driver paravirtualised)
            (dev xvdb)
            (uname phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah)
            (mode w)
        )
    )
    (device
        (vbd
            (uuid blah)
            (bootable 0)
            (driver paravirtualised)
            (dev xvdc)
            (uname phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah)
            (mode w)
        )
    )
    (device
        (vbd
            (uuid blah)
            (bootable 0)
            (driver paravirtualised)
            (dev xvdd)
            (uname phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah)
            (mode w)
        )
    )
)

xend.log tail:
[2012-03-07 15:38:45 1918] DEBUG (XendDomainInfo:101)
XendDomainInfo.create([''vm'', [''name'',
''leela''], [''memory'', 1024],
[''on_xend_start'', ''ignore''],
[''on_xend_stop'', ''ignore''],
[''vcpus'', 1], [''oos'', 1],
[''bootloader'', ''/usr/bin/pygrub-4.1b''],
[''bootloader_args'', ''-q''],
[''image'', [''linux'',
[''videoram'', 4], [''tsc_mode'', 0],
[''nomigrate'', 0]]], [''s3_integrity'', 1],
[''device'', [''vbd'',
[''uname'', ''file:/vm/leela/leela.img''],
[''dev'', ''xvda''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah''],
[''dev'', ''xvdb''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah''],
[''dev'', ''xvdc''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah''],
[''dev'', ''xvdd''], [''mode'',
''w'']]], [''device'', [''
 vif'', [''bridge'', ''eth0''],
[''mac'', ''blah'']]]])
[2012-03-07 15:38:45 1918] DEBUG (XendDomainInfo:2508)
XendDomainInfo.constructDomain
[2012-03-07 15:38:45 1918] DEBUG (balloon:220) Balloon: 535512 KiB free; need
16384; done.
[2012-03-07 15:38:45 1918] DEBUG (XendDomain:464) Adding Domain: 7
[2012-03-07 15:38:45 1918] DEBUG (XendDomainInfo:2818)
XendDomainInfo.initDomain: 7 256
[2012-03-07 15:38:45 5169] DEBUG (XendBootloader:113) Launching bootloader as
[''/usr/bin/pygrub-4.1b'',
''--output=/var/run/xend/boot/xenbl.20421'',
''-q'', ''/vm/leela/leela.img''].

--
Mike Brancato

Mike Brancato

2012-Mar-08 02:23 UTC

head link

Re: PV DomU stopped responding, won''t boot, stuck in paused state

I managed to fix my issue, it was to simply change pygrub to call
/usr/bin/python2.5 not simply /usr/bin/python.  The default python on Debian 6.0
is 2.6.6 apparently.

It is concerning that the current pygrub does not properly support ZFS.  My ZFS
version is 16 I believe, which is very old, so it isn''t anything new. 
A simple google search yields a number of users who cannot boot ZFS due to the
following error with pygrub when running ''pygrub domu.img'':

Traceback (most recent call last):
  File "/usr/bin/pygrub", line 669, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/bin/pygrub", line 572, in run_grub
    grubcfg["kernel"] = img.kernel[1]
TypeError: ''NoneType'' object is unsubscriptable

--
Mike Brancato


On Mar 7, 2012, at 4:07 PM, Mike Brancato wrote:
> Yesterday, I had a PV OpenSolaris DomU stop responding.  I looked on my xen
machine and it was in a paused state.  I ended up destroying it and then doing
xm create to bring it back up.
> 
> It won''t boot, and I think PyGrub is hanging.  Not sure how to
resolve this, there aren''t any errors being presented in xen/xm itself.
If I try ''xm create config.conf'' it just  says "using
config file blah.conf<newline>" and nothing else. Never fails, never
boots.  If I drop out "ctrl-c" and look, it is paused with 0 run time.
> 
> A few weeks ago I upgraded from Debian 5.0 to 6.0.  My Xen install was
refreshed with some updated packages, but I was already running 4.0.1 on Debian
5.0.  But after the upgrade, everything was working fine.  No changes to this
machine since, just noticed it had stopped responding last night, but that DomU
was running earlier in the day.
> 
> Hopefully someone can help me figure out what''s wrong,
I''m guessing it has to do with pygrub since it won''t even
begin the boot process.  That still doesn''t explain why it stopped
responding unless there was some other failure that caused a reboot, but this
machine has normally run without incident for almost 3 years.  It *is* and old
version of OpenSolaris.
> 
> Here is some info on the Dom0 and DomU.
> 
> Debian 6.0 (squeeze)
> Xen 4.0.1
> My pygrub is named /usr/bin/pygrub-4.1b, I know I had to hack around with
this in the past to get ZFS support.
> 
> When I run pygrub-4.1b domu.img, I get an error now:
> Traceback (most recent call last):
>  File "/usr/bin/pygrub-4.1b", line 20, in <module>
>    import xen.lowlevel.xc
> ImportError: No module named xen.lowlevel.xc
> 
> If I try the provided PyGrub from debian, I get:
> farnsworth:/vm/leela# xm create leela.conf 
> Using config file "./leela.conf".
> Error: Boot loader didn''t return any data!
> 
> But the provided pygrub does pull up the grub menu with ''pygrub
leela.img''.
> 
> DomU:
> I pass through some physical drives to a zpool array, this machine is a
storage server and handles SMB/CIFS.
> 
> (domain
>    (domid 7)
>    (cpu_weight 256)
>    (cpu_cap 0)
>    (on_crash restart)
>    (uuid blah)
>    (bootloader_args -q)
>    (vcpus 1)
>    (name leela)
>    (on_poweroff destroy)
>    (on_reboot restart)
>    (cpus (()))
>    (description )
>    (bootloader /usr/bin/pygrub-4.1b)
>    (maxmem 1024)
>    (memory 1024)
>    (shadow_memory 0)
>    (features )
>    (on_xend_start ignore)
>    (on_xend_stop ignore)
>    (cpu_time 0.0)
>    (online_vcpus 0)
>    (image
>        (linux
>            (kernel )
>            (superpages 0)
>            (tsc_mode 0)
>            (videoram 4)
>            (pci ())
>            (nomigrate 0)
>        )
>    )
>    (status 1)
>    (state --p---)
>    (device
>        (vif
>            (bridge eth0)
>            (mac blah)
>            (uuid blah)
>        )
>    )
>    (device
>        (vbd
>            (uuid blah)
>            (bootable 1)
>            (driver paravirtualised)
>            (dev xvda)
>            (uname file:/vm/leela/leela.img)
>            (mode w)
>        )
>    )
>    (device
>        (vbd
>            (uuid blah)
>            (bootable 0)
>            (driver paravirtualised)
>            (dev xvdb)
>            (uname phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah)
>            (mode w)
>        )
>    )
>    (device
>        (vbd
>            (uuid blah)
>            (bootable 0)
>            (driver paravirtualised)
>            (dev xvdc)
>            (uname phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah)
>            (mode w)
>        )
>    )
>    (device
>        (vbd
>            (uuid blah)
>            (bootable 0)
>            (driver paravirtualised)
>            (dev xvdd)
>            (uname phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah)
>            (mode w)
>        )
>    )
> )
> 
> xend.log tail:
> [2012-03-07 15:38:45 1918] DEBUG (XendDomainInfo:101)
XendDomainInfo.create([''vm'', [''name'',
''leela''], [''memory'', 1024],
[''on_xend_start'', ''ignore''],
[''on_xend_stop'', ''ignore''],
[''vcpus'', 1], [''oos'', 1],
[''bootloader'', ''/usr/bin/pygrub-4.1b''],
[''bootloader_args'', ''-q''],
[''image'', [''linux'',
[''videoram'', 4], [''tsc_mode'', 0],
[''nomigrate'', 0]]], [''s3_integrity'', 1],
[''device'', [''vbd'',
[''uname'', ''file:/vm/leela/leela.img''],
[''dev'', ''xvda''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah''],
[''dev'', ''xvdb''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah''],
[''dev'', ''xvdc''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''phy:/dev/disk/by-id/scsi-SATA_SAMSUNG_blah''],
[''dev'', ''xvdd''], [''mode'',
''w'']]], [''device'', [''vif'', [''bridge'',
''eth0''], [''mac'',
''blah'']]]])> [2012-03-07 15:38:45 1918] DEBUG (XendDomainInfo:2508)
XendDomainInfo.constructDomain
> [2012-03-07 15:38:45 1918] DEBUG (balloon:220) Balloon: 535512 KiB free;
need 16384; done.
> [2012-03-07 15:38:45 1918] DEBUG (XendDomain:464) Adding Domain: 7
> [2012-03-07 15:38:45 1918] DEBUG (XendDomainInfo:2818)
XendDomainInfo.initDomain: 7 256
> [2012-03-07 15:38:45 5169] DEBUG (XendBootloader:113) Launching bootloader
as [''/usr/bin/pygrub-4.1b'',
''--output=/var/run/xend/boot/xenbl.20421'',
''-q'', ''/vm/leela/leela.img''].
> 
> --
> Mike Brancato
> 
>

Mike Brancato

2012-Mar-08 02:25 UTC

head link

Re: PV DomU stopJust ped responding, won''t boot, stuck in paused state

I just realized that Debian is using 4.0 which is not current.  So my
pygrub-4.1b is a version I took from the 4.1 dev tree a long time ago.  So 4.1
probably has support for ZFS.

--
Mike Brancato

Reasonably Related Threads

Search for more seemingly similar threads

Xen users - Mar 2012 - PV DomU stopped responding, won't boot, stuck in paused state

PV DomU stopped responding, won''t boot, stuck in paused state

Re: PV DomU stopped responding, won''t boot, stuck in paused state

Re: PV DomU stopJust ped responding, won''t boot, stuck in paused state

Reasonably Related Threads