CentOS - Oct 2016 - Cannot boot CentOS 7 VM after updating Host CentOS 7 Kernel

Any error in your host logs?

On Sunday, 30 October 2016, Steven Tardy <sjt5atra at gmail.com> wrote:
>
> > On Oct 30, 2016, at 3:27 AM, Paul R. Ganci <ganci at nurdog.com
> <javascript:;>> wrote:
> >
> > grub> set root=(hd0,msdos2)
> > grub> linux (hd0,1)/vmlinuz-3.10.0-327.36.2.el7.x86_64
> root=(hd0,msdos2)/
> > grub> initrd initrd-plymouth.img
> > grub> boot
>
> Try the initrd matching the kernel?
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org <javascript:;>
> https://lists.centos.org/mailman/listinfo/centos
>

On 10/30/2016 07:33 AM, FrancisM wrote:
> Any error in your host logs?
Nothing obvious. I checked /var/log/messages, /var/log/libvirt/qemu, 
/var/log/libvirt/lxc & /var/log/qemu-ga. The /var/log/libvirt/lxc & 
/var/log/qemu-ga were empty. The /var/log/libvirt/qemu directory had a 
log file of interest Outgoing-CentOS-7-VM.log but nothing in it that 
tells me anything obvious.

  2016-10-30 06:40:09.764+0000: shutting down
2016-10-30 06:40:16.926+0000: starting up libvirt version: 1.2.17, 
package: 13.el7_2.5 (CentOS BuildSystem <http://bugs.centos.org>, 
2016-06-23-14:23:27, worker1.bsys.centos.org), qemu version: 1.5.3 
(qemu-kvm-1.5.3-105.el7_2.7)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin 
QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name Outgoing-CentOS-7-VM -S 
-machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn -m 4096 
-realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 
6494b5d9-8adc-4f66-b0cf-4c19a0f6ab66 -no-user-config -nodefaults 
-chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Outgoing-CentOS-7-VM/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet 
-no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 
-boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 
-device 
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 
-device 
ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 
-device 
ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive 
file=/vm-images/centos7.0.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device 
ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev 
tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7b:a5:c2,bus=pci.0,addr=0x3 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -chardev 
spicevmc,id=charchannel0,name=vdagent -device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice 
port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga 
qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 
-global qxl-vga.vgamem_mb=16 -device 
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device 
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev 
spicevmc,id=charredir0,name=usbredir -device 
usb-redir,chardev=charredir0,id=redir0 -chardev 
spicevmc,id=charredir1,name=usbredir -device 
usb-redir,chardev=charredir1,id=redir1 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 0.226000 ms, bitrate 
37925925925 bps (36168.981481 Mbps)
red_dispatcher_set_cursor_peer:
inputs_connect: inputs channel client create
red_peer_receive: Connection reset by peer
red_channel_client_disconnect: rcc=0x7f2b11db6000 
(channel=0x7f2b115b6600 type=2 id=0)
red_peer_receive: Connection reset by peer
red_channel_client_disconnect: rcc=0x7f2b11d71000 
(channel=0x7f2b10faa000 type=3 id=0)
red_channel_client_disconnect: rcc=0x7f2b11d8d000 
(channel=0x7f2b10f8c4e0 type=9 id=0)
red_channel_client_disconnect: rcc=0x7f2b11d88000 
(channel=0x7f2b10f8c680 type=9 id=1)
red_channel_client_disconnect: rcc=0x7f2b11a33000 
(channel=0x7f2b10fa2000 type=1 id=0)
main_channel_client_on_disconnect: rcc=0x7f2b11a33000
red_client_destroy: destroy client 0x7f2b10f2fd00 with #channels=4
red_dispatcher_disconnect_cursor_peer:
red_channel_client_disconnect: rcc=0x7f2b11d6c000 
(channel=0x7f2b11938000 type=4 id=0)
red_dispatcher_disconnect_display_peer

I reboot the host while the guests were running. Is it possible the root 
file system was corrupted during the host reboot? I am thinking of 
putting the CentOS iso out and then booting the VM into it just to poke 
around the file system. Otherwise my other option is to just clone a 
twin VM on another server and then just change the networking 
IPs/hostname. Anybody have any other ideas as to how to debug this problem?
-- 
Paul (ganci at nurdog.com)
Cell: (303)257-5208

You could mount image ja rebuild initrd.

Eero

2016-10-30 20:26 GMT+02:00 Paul R. Ganci <ganci at nurdog.com>:
> On 10/30/2016 07:33 AM, FrancisM wrote:
>
>> Any error in your host logs?
>>
> Nothing obvious. I checked /var/log/messages, /var/log/libvirt/qemu,
> /var/log/libvirt/lxc & /var/log/qemu-ga. The /var/log/libvirt/lxc &
> /var/log/qemu-ga were empty. The /var/log/libvirt/qemu directory had a log
> file of interest Outgoing-CentOS-7-VM.log but nothing in it that tells me
> anything obvious.
>
>  2016-10-30 06:40:09.764+0000: shutting down
> 2016-10-30 06:40:16.926+0000: starting up libvirt version: 1.2.17,
> package: 13.el7_2.5 (CentOS BuildSystem <http://bugs.centos.org>,
> 2016-06-23-14:23:27, worker1.bsys.centos.org), qemu version: 1.5.3
> (qemu-kvm-1.5.3-105.el7_2.7)
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name Outgoing-CentOS-7-VM -S
> -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn -m 4096
> -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
> 6494b5d9-8adc-4f66-b0cf-4c19a0f6ab66 -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Outg
> oing-CentOS-7-VM/monitor.sock,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
>
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
> file=/vm-images/centos7.0.qcow2,if=none,id=drive-virtio-disk0,format=qcow2
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virti
> o-disk0,id=virtio-disk0,bootindex=1 -drive
if=none,id=drive-ide0-0-0,readonly=on,format=raw
> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev
> tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device
>
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7b:a5:c2,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0
> -chardev spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel
> 0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice
> port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl
> -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864
> -global qxl-vga.vgamem_mb=16 -device intel-hda,id=sound0,bus=pci.0,addr=0x4
> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
> spicevmc,id=charredir0,name=usbredir -device
> usb-redir,chardev=charredir0,id=redir0 -chardev
> spicevmc,id=charredir1,name=usbredir -device
> usb-redir,chardev=charredir1,id=redir1 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
> char device redirected to /dev/pts/1 (label charserial0)
> main_channel_link: add main channel client
> main_channel_handle_parsed: net test: latency 0.226000 ms, bitrate
> 37925925925 bps (36168.981481 Mbps)
> red_dispatcher_set_cursor_peer:
> inputs_connect: inputs channel client create
> red_peer_receive: Connection reset by peer
> red_channel_client_disconnect: rcc=0x7f2b11db6000 (channel=0x7f2b115b6600
> type=2 id=0)
> red_peer_receive: Connection reset by peer
> red_channel_client_disconnect: rcc=0x7f2b11d71000 (channel=0x7f2b10faa000
> type=3 id=0)
> red_channel_client_disconnect: rcc=0x7f2b11d8d000 (channel=0x7f2b10f8c4e0
> type=9 id=0)
> red_channel_client_disconnect: rcc=0x7f2b11d88000 (channel=0x7f2b10f8c680
> type=9 id=1)
> red_channel_client_disconnect: rcc=0x7f2b11a33000 (channel=0x7f2b10fa2000
> type=1 id=0)
> main_channel_client_on_disconnect: rcc=0x7f2b11a33000
> red_client_destroy: destroy client 0x7f2b10f2fd00 with #channels=4
> red_dispatcher_disconnect_cursor_peer:
> red_channel_client_disconnect: rcc=0x7f2b11d6c000 (channel=0x7f2b11938000
> type=4 id=0)
> red_dispatcher_disconnect_display_peer
>
> I reboot the host while the guests were running. Is it possible the root
> file system was corrupted during the host reboot? I am thinking of putting
> the CentOS iso out and then booting the VM into it just to poke around the
> file system. Otherwise my other option is to just clone a twin VM on
> another server and then just change the networking IPs/hostname. Anybody
> have any other ideas as to how to debug this problem?
> --
> Paul (ganci at nurdog.com)
> Cell: (303)257-5208
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

On 10/30/2016 12:26 PM, Paul R. Ganci wrote:
> <snip>I am thinking of putting the CentOS iso out and then booting
the
> VM into it just to poke around the file system. Otherwise my other 
> option is to just clone a twin VM on another server and then just 
> change the networking IPs/hostname. Anybody have any other ideas as to 
> how to debug this problem?
So I booted off the CentOS-7-x86_64-DVD-1511.iso and everything looks 
just fine:

 > df
Filesystem                         1K-blocks        Used Available   
Use%   Mounted on
/dev/mapper/live-rw           2030899    949022    1077781     47% /
devtmpfs                           2004040              0 2004040       
0%   /dev
tmpfs                                 2023652              0 
2023652       0%   /dev/shm
tmpfs                                 2023652        8520 2015132       
1%   /run
tmpfs                                 2023652              0 
2023652       0%  /sys/fs/cgroup
/dev/sr1                             4227724  4227724 0   100%  
/run/install/repo
tmpfs                                 2023652          200 2023452       
1%  /tmp
/dev/mapper/centos-root  10799104  3894196    6904908     37% /mnt/sysimage
/dev/vda1                            508588     143516 365072     29%  
/mnt/sysimage/boot
tmpfs                                2023652               0 
2023652       0%  /mnt/sysimage/dev/shm

 > ls /mnt/sysimage
bin   boot   dev   etc   home   lib   lib64   media   misc   mnt net   
opt   proc   root    run    sbin   srv   sys    tmp   usr var

 > ls -l /mnt/sysimage/boot
total 109424
-rw-r--r--.    1 root root       126431  Oct 10 23:18 
config-3.10.0-327.36.2.el7.x86_64
drwxr-xr-x. 2 root root               26  Oct   2  2015  grub
drwx------.  6 root root             104  Oct 13 02:21  grub2
-rw-r--r--.    1 root root   40655493  Apr   3  2015 
initramfs-0-rescue-6494b5d98adc4f66b0cf4c19a0f6ab66.img
-rw-------.    1 root root   29666884 Oct 13  01:25 
initramfs-3.10.0-327.36.2.el7.x86_64.img
-rw-------.    1 root root   18119089  Oct 13 02:20 
initramfs-3.10.0-327.36.2.el7.x86_64kdump.img
-rw-r--r--.    1 root root   10190975  Dec 19  2015 initrd-plymouth.img
-rw-r--r--.    1 root root      252739  Oct  10 23:20 
symvers-3.10.0-327.36.2.el7.x86_64.gz
-rw-------.    1 root root    2965270  Oct  10 23:18 
System.map-3.10.0-327.36.2.el7.x86_64
-rwxr-xr-x.   1 root root    4902656  Apr    3  2015 
vmlinuz0-rescue-6494b5d98adc4f66b0cf4c19a0f6ab66
-rwxr-xr-x.   1 root root    5157936   Oct  10 23:18 
vmlinuz-3.10.0-327.36.2.el7.x86_64

So the CentOS DVD iso in linux rescue mode shows that everything is 
there and can be mounted. I guess that means somehow either grub itself 
is corrupted or one of the boot images. So is there a way for me to 
generate a new initrd while booted in linux resuce mode or will 
re-installing grub help? How would I attempt re-installing grub while 
booted in linux rescue mode?
-- 
Paul (ganci at nurdog.com)
Cell: (303)257-5208