Hi, I have a Debian domU that is refusing to boot. Attaching to the console (xm console) it seems to be in the middle of mounting a drive, and then hanging. .... [ 2.387986] EXT3-fs: mounted filesystem with ordered data mode. Begin: Running /scripts/local-bottom ... done. done. Begin: Running /scripts/init-bottom ... done. INIT: version 2.86 booting Starting the hotplug events dispatcher: udevd[ 5.467529] udevd version 125 started . Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...done. Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on /dev/xvda2. Priority:-1 extents:1 across:3145588k [ 89.796051] EXT3 FS on xvda1, internal journal <hang> I know how I might proceed if I were booting a physical machine - try to come up in single user mode, failing that, boot with a live CD and start examining things. But... this is a VM, with more limited "hooks" into the boot process. Any suggestions on how to proceed? Note, I''m running what''s supposed to be a 2-node high-availability configuration (DRBD-mirrored drives, pacemaker). The underlying RAID10 and DRBD arrays are reporting good status, and other VMs are running just fine. This one just refuses to come up on either node in the cluster. Thanks, Miles Fidelman -- In theory, there is no difference between theory and practice. In<fnord> practice, there is. .... Yogi Berra _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
A little more data: ----background-------> I have a Debian domU that is refusing to boot. Attaching to the > console (xm console) it seems to be in the middle of mounting a drive, > and then hanging. > > .... > [ 2.387986] EXT3-fs: mounted filesystem with ordered data mode. > Begin: Running /scripts/local-bottom ... done. > done. > Begin: Running /scripts/init-bottom ... done. > INIT: version 2.86 booting > Starting the hotplug events dispatcher: udevd[ 5.467529] udevd > version 125 started > . > Synthesizing the initial hotplug events...done. > Waiting for /dev to be fully populated...done. > Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on > /dev/xvda2. Priority:-1 extents:1 across:3145588k > [ 89.796051] EXT3 FS on xvda1, internal journal > <hang> > > I know how I might proceed if I were booting a physical machine - try > to come up in single user mode, failing that, boot with a live CD and > start examining things. But... this is a VM, with more limited > "hooks" into the boot process. Any suggestions on how to proceed? > > Note, I''m running what''s supposed to be a 2-node high-availability > configuration (DRBD-mirrored drives, pacemaker). The underlying > RAID10 and DRBD arrays are reporting good status, and other VMs are > running just fine. This one just refuses to come up on either node in > the cluster.------------ 1. looking at another domU that''s working fine, I see that its console log reports: ..... [ 31.088678] EXT3 FS on xvda1, internal journal [ 42.316654] NET: Registered protocol family 10 ---- while my broken machine is hanging between these two steps - suggesting either a disk problem or a networking problem 2. I mounted and fsck''d the volume, and it seems to be fine, so it looks like something is either broken in the network startup, or interfering with it. Which leads to three questions: 1. any suggestions on what might spontaneously go wrong with a previously-functioning network startup? 2. any debugging suggestions? 3. specifically, any suggestions on how to insert myself into a boot process that is normally configured to be driven by pacemaker and xen''s startup hooks - so I can do some step by step debugging of the boot process? Thanks again, Miles Fidelman Miles Fidelman wrote:> Hi, > > I have a Debian domU that is refusing to boot. Attaching to the > console (xm console) it seems to be in the middle of mounting a drive, > and then hanging. > > .... > [ 2.387986] EXT3-fs: mounted filesystem with ordered data mode. > Begin: Running /scripts/local-bottom ... done. > done. > Begin: Running /scripts/init-bottom ... done. > INIT: version 2.86 booting > Starting the hotplug events dispatcher: udevd[ 5.467529] udevd > version 125 started > . > Synthesizing the initial hotplug events...done. > Waiting for /dev to be fully populated...done. > Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on > /dev/xvda2. Priority:-1 extents:1 across:3145588k > [ 89.796051] EXT3 FS on xvda1, internal journal > <hang> > > I know how I might proceed if I were booting a physical machine - try > to come up in single user mode, failing that, boot with a live CD and > start examining things. But... this is a VM, with more limited > "hooks" into the boot process. Any suggestions on how to proceed? > > Note, I''m running what''s supposed to be a 2-node high-availability > configuration (DRBD-mirrored drives, pacemaker). The underlying > RAID10 and DRBD arrays are reporting good status, and other VMs are > running just fine. This one just refuses to come up on either node in > the cluster. > > Thanks, > > Miles Fidelman > > >-- In theory, there is no difference between theory and practice. In<fnord> practice, there is. .... Yogi Berra _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
well - looks like one of my file systems was hosed (not boot, root, or swap) - seems like the Debian boot scripts don''t report stuff going on in the middle of /etc/rcS.d/S35mountall.sh - they log errors, but don''t report to the console (at least with stock configuration). Had to mount the root directory and pour through logs for the DomU to find this running an fsck on it now still begs the question of how to do single-user, or other diagnostic boot of a domU Miles Fidelman wrote:> A little more data: > > ----background------- >> I have a Debian domU that is refusing to boot. Attaching to the >> console (xm console) it seems to be in the middle of mounting a >> drive, and then hanging. >> >> .... >> [ 2.387986] EXT3-fs: mounted filesystem with ordered data mode. >> Begin: Running /scripts/local-bottom ... done. >> done. >> Begin: Running /scripts/init-bottom ... done. >> INIT: version 2.86 booting >> Starting the hotplug events dispatcher: udevd[ 5.467529] udevd >> version 125 started >> . >> Synthesizing the initial hotplug events...done. >> Waiting for /dev to be fully populated...done. >> Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on >> /dev/xvda2. Priority:-1 extents:1 across:3145588k >> [ 89.796051] EXT3 FS on xvda1, internal journal >> <hang> >> >> I know how I might proceed if I were booting a physical machine - try >> to come up in single user mode, failing that, boot with a live CD and >> start examining things. But... this is a VM, with more limited >> "hooks" into the boot process. Any suggestions on how to proceed? >> >> Note, I''m running what''s supposed to be a 2-node high-availability >> configuration (DRBD-mirrored drives, pacemaker). The underlying >> RAID10 and DRBD arrays are reporting good status, and other VMs are >> running just fine. This one just refuses to come up on either node >> in the cluster. > ------------ > > 1. looking at another domU that''s working fine, I see that its console > log reports: > > ..... > [ 31.088678] EXT3 FS on xvda1, internal journal > [ 42.316654] NET: Registered protocol family 10 > ---- > > while my broken machine is hanging between these two steps - > suggesting either a disk problem or a networking problem > > 2. I mounted and fsck''d the volume, and it seems to be fine, so it > looks like something is either broken in the network startup, or > interfering with it. > > Which leads to three questions: > > 1. any suggestions on what might spontaneously go wrong with a > previously-functioning network startup? > 2. any debugging suggestions? > 3. specifically, any suggestions on how to insert myself into a boot > process that is normally configured to be driven by pacemaker and > xen''s startup hooks - so I can do some step by step debugging of the > boot process? > > Thanks again, > > Miles Fidelman > > > Miles Fidelman wrote: >> Hi, >> >> I have a Debian domU that is refusing to boot. Attaching to the >> console (xm console) it seems to be in the middle of mounting a >> drive, and then hanging. >> >> .... >> [ 2.387986] EXT3-fs: mounted filesystem with ordered data mode. >> Begin: Running /scripts/local-bottom ... done. >> done. >> Begin: Running /scripts/init-bottom ... done. >> INIT: version 2.86 booting >> Starting the hotplug events dispatcher: udevd[ 5.467529] udevd >> version 125 started >> . >> Synthesizing the initial hotplug events...done. >> Waiting for /dev to be fully populated...done. >> Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on >> /dev/xvda2. Priority:-1 extents:1 across:3145588k >> [ 89.796051] EXT3 FS on xvda1, internal journal >> <hang> >> >> I know how I might proceed if I were booting a physical machine - try >> to come up in single user mode, failing that, boot with a live CD and >> start examining things. But... this is a VM, with more limited >> "hooks" into the boot process. Any suggestions on how to proceed? >> >> Note, I''m running what''s supposed to be a 2-node high-availability >> configuration (DRBD-mirrored drives, pacemaker). The underlying >> RAID10 and DRBD arrays are reporting good status, and other VMs are >> running just fine. This one just refuses to come up on either node >> in the cluster. >> >> Thanks, >> >> Miles Fidelman >> >> >> > >-- In theory, there is no difference between theory and practice. In<fnord> practice, there is. .... Yogi Berra _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users