Am attempting to update a RHEL4 domU from a custom created Xen kernel (created
before RHEL5) to a supported kernel (2.6.9-55.0.9.ELxenU) and RHEL5 host without
success. For some reason, the kernel can no longer find xvda. Boots fine on the
old kernel and host, so I''m at a bit of a loss. Host has other domUs
booted off the same SAN, with no security restrictions, so I don''t see
how it could be hardware. Copy of configs and partial debug log below. Ideas?
Conf:
name = "do5"
memory = "1024"
disk = [ ''phy:/dev/mpath/msa-mpathv0p1,xvda,w'', ]
vif = [ ''mac=00:16:3e:55:a5:65, bridge=xenbr2'' ]
vfb = ["type=vnc,vncunused=1"]
bootloader="/usr/bin/pygrub"
vcpus=1
Grub:
root (hd0,0)
kernel /vmlinuz-2.6.9-55.0.9.ELxenU ro root=/dev/xvda2 rhgb quiet
initrd /initrd-2.6.9-55.0.9.ELxenU.img
Debug:
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:190)
XendDomainInfo.create([''vm'', [''name'',
''do5''], [''memory'',
''1024''], [''vcpus'', 1],
[''bootloader'', ''/usr/bin/pygrub''],
[''image'', [''linux'',
[''ramdisk'', ''/var/lib/xen/initrd.MpxKQP''],
[''kernel'', ''/var/lib/xen/vmlinuz.6u2e9w''],
[''args'', ''ro root=/dev/xvda2 rhgb quiet'']]],
[''device'', [''vbd'',
[''uname'', ''phy:/dev/mpath/msa-mpathv0p1''],
[''dev'', ''xvda''], [''mode'',
''w'']]], [''device'', [''vif'',
[''bridge'', ''xenbr2''],
[''mac'', ''00:16:3e:55:a5:65'']]],
[''device'', [''vkbd'']],
[''device'', [''vfb'',
[''vncunused'', ''1''],
[''type'', ''vnc''],
[''display'', ''localhost:10.0''],
[''xauthority'', ''/root/.Xauthority'']]]])
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:296)
parseConfig: config is [''vm'', [''name'',
''do5''], [''memory'',
''1024''], [''vcpus'', 1],
[''bootloader'', ''/usr/bin/pygrub''],
[''image'', [''linux'',
[''ramdisk'', ''/var/lib/xen/initrd.MpxKQP''],
[''kernel'', ''/var/lib/xen/vmlinuz.6u2e9w''],
[''args'', ''ro root=/dev/xvda2 rhgb quiet'']]],
[''device'', [''vbd'',
[''uname'', ''phy:/dev/mpath/msa-mpathv0p1''],
[''dev'', ''xvda''], [''mode'',
''w'']]], [''device'', [''vif'',
[''bridge'', ''xenbr2''],
[''mac'', ''00:16:3e:55:a5:65'']]],
[''device'', [''vkbd'']],
[''device'', [''vfb'',
[''vncunused'', ''1''],
[''type'', ''vnc''],
[''display'', ''localhost:10.0''],
[''xauthority'', ''/root/.Xauthority'']]]]
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:397)
parseConfig: result is {''shadow_memory'': None,
''start_time'': None, ''uuid'': None,
''on_crash'': None, ''on_reboot'': None,
''localtime'': None, ''image'':
[''linux'', [''ramdisk'',
''/var/lib/xen/initrd.MpxKQP''], [''kernel'',
''/var/lib/xen/vmlinuz.6u2e9w''], [''args'',
''ro root=/dev/xvda2 rhgb quiet'']],
''on_poweroff'': None, ''bootloader_args'':
None, ''cpus'': None, ''name'':
''do5'', ''backend'': [],
''vcpus'': 1, ''cpu_weight'': None,
''features'': None, ''vcpu_avail'': None,
''memory'': 1024, ''device'':
[(''vbd'', [''vbd'', [''uname'',
''phy:/dev/mpath/msa-mpathv0p1''], [''dev'',
''xvda''], [''mode'', ''w'']]),
(''vif'', [''vif'', [''bridge'',
''xenbr2''], [''mac'',
''00:16:3e:55:a5:65'']]), (''vkbd'',
[''vkbd'']), (''vfb'', [''vfb'',
[''vncunused'', ''1''],
[''type'', ''vnc''],
[''display'', ''localhost:10.0''],
[''xauthority'', ''/root/.Xauthority'']])],
''bootloader'': ''/usr/bin/pygrub'',
''cpu'': None, ''maxmem'': None}
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:1264)
XendDomainInfo.construct: None
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:1296)
XendDomainInfo.initDomain: 9 1.0
[2007-10-30 12:41:21 xend 4121] DEBUG (balloon:127) Balloon: 1049112 KiB free;
need 1048576; done.
[2007-10-30 12:41:21 xend 4121] INFO (image:136) buildDomain os=linux dom=9
vcpus=1
[2007-10-30 12:41:21 xend 4121] DEBUG (image:199) dom = 9
[2007-10-30 12:41:21 xend 4121] DEBUG (image:200) image =
/var/lib/xen/vmlinuz.6u2e9w
[2007-10-30 12:41:21 xend 4121] DEBUG (image:201) store_evtchn = 1
[2007-10-30 12:41:21 xend 4121] DEBUG (image:202) console_evtchn = 2
[2007-10-30 12:41:21 xend 4121] DEBUG (image:203) cmdline = ro root=/dev/xvda2
rhgb quiet
[2007-10-30 12:41:21 xend 4121] DEBUG (image:204) ramdisk =
/var/lib/xen/initrd.MpxKQP
[2007-10-30 12:41:21 xend 4121] DEBUG (image:205) vcpus = 1
[2007-10-30 12:41:21 xend 4121] DEBUG (image:206) features =
[2007-10-30 12:41:21 xend 4121] DEBUG (blkif:24) exception looking up device
number for xvda: [Errno 2] No such file or directory:
''/dev/xvda''
Jim Klein wrote:> Am attempting to update a RHEL4 domU from a custom created Xen kernel > (created before RHEL5) to a supported kernel (2.6.9-55.0.9.ELxenU) and > RHEL5 host without success. For some reason, the kernel can no longer > find xvda. Boots fine on the old kernel and host, so I''m at a bit of a > loss. Host has other domUs booted off the same SAN, with no security > restrictions, so I don''t see how it could be hardware. Copy of configs > and partial debug log below. Ideas?Hm, everything below, at a first glance, seems to be right. What does the console output look like from the RHEL-4 kernel (make sure to take rhgb quiet off of the command-line)? Chris Lalancette
I have had similar problems, at last i reverted to use sda instead of xvda. Also i am getting errors about /lib/tls even when its been renamed in dom0 and domU. Any ideas on this ? -- regards, Anand Gupta
Interesting: XENBUS: Device with no driver: device/vbd/2048 XENBUS: Device with no driver: device/vif/0 Freeing unused kernel memory: 124k freed device-mapper: 4.5.5-ioctl ( 2006-12-01 ) initialised: dm-devel@redhat.com Kernel panic - not syncing: Attempted to kill init! Not sure where to go from here. ----- Original Message ----- From: "Chris Lalancette" <clalance@redhat.com> To: "Jim Klein" <jklein@saugus.k12.ca.us> Cc: "fedora-xen" <fedora-xen@redhat.com> Sent: Tuesday, October 30, 2007 1:23:08 PM (GMT-0800) America/Los_Angeles Subject: Re: [Fedora-xen] RHEL4 DomU Update Problem Jim Klein wrote:> Am attempting to update a RHEL4 domU from a custom created Xen kernel > (created before RHEL5) to a supported kernel (2.6.9-55.0.9.ELxenU) and > RHEL5 host without success. For some reason, the kernel can no longer > find xvda. Boots fine on the old kernel and host, so I''m at a bit of a > loss. Host has other domUs booted off the same SAN, with no security > restrictions, so I don''t see how it could be hardware. Copy of configs > and partial debug log below. Ideas?Hm, everything below, at a first glance, seems to be right. What does the console output look like from the RHEL-4 kernel (make sure to take rhgb quiet off of the command-line)? Chris Lalancette
Jim Klein wrote:> Interesting: > > XENBUS: Device with no driver: device/vbd/2048 > XENBUS: Device with no driver: device/vif/0 > Freeing unused kernel memory: 124k freed > device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel@redhat.com > Kernel panic - not syncing: Attempted to kill init! > > Not sure where to go from here. >Believe it or not, those XENBUS errors are actually expected; the xenbus driver is loaded before the virtual disk/network drivers, and it sees that it has these nodes without drivers. However, that does lead me to an idea. If I remember correctly, the Xensource kernels have all of the drivers built-in, while we prefer to do things more modular. I''m guessing that you didn''t have the right driver in /etc/modprobe.conf when the kernel was installed, so the initrd doesn''t have the right drivers in it. So what you''ll want to do is: 1) Boot the RHEL-4 domU into the (working) Xen kernel. 2) Edit /etc/modprobe.conf, and add: alias scsi_hostadapter xenblk 3) Remove the RedHat RHEL-4 kernel (rpm -e), and install it again, which should re-generate the initrd with the right stuff in it. Chris Lalancette
Now it''s really interesting. If I just try to boot the system with the disk labeled xvda, I get a huge series of kernel panics (see below.) If I change it to sda (in both the config file and grub) it looks like it might work, but then I get: Freeing unused kernel memory: 124k freed SCSI subsystem initialized Registering block device major 8 register_blkdev: cannot get major 8 for sd xen_blk: can''t get major 8 with name sd vbd vbd-2048: 19 xlvbd_add at /local/domain/0/backend/vbd/27/2048 Registering block device major 8 register_blkdev: cannot get major 8 for sd xen_blk: can''t get major 8 with name sd vbd vbd-2048: 19 xlvbd_add at /local/domain/0/backend/vbd/27/2048 XENBUS: Timeout connecting to device: device/vbd/2048 (state 6) device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel@redhat.com Kernel panic - not syncing: Attempted to kill init! FYI, this wasn''t a Xensource kernel before, it was actually a Fedora 5 kernel (2.6.15-1.2054_FC5xenU) I created a yum repository for RHEL4 with the Fedora kernel, and used it to set up the box. Kernel panics (lots of them, all the same): kernel BUG at arch/i386/mm/pgtable-xen.c:306! invalid operand: 0000 [#1] SMP Modules linked in: ext3 jbd dm_mod xenblk sd_mod scsi_mod CPU: 0 EIP: 0061:[<c011163a>] Not tainted VLI EFLAGS: 00010282 (2.6.9-55.0.9.ELxenU) EIP is at pgd_ctor+0x1d/0x26 eax: fffffff4 ebx: 00000000 ecx: f5392000 edx: 00000000 esi: c19fdd80 edi: eca6aaa0 ebp: 00000001 esp: ecb3cd6c ds: 007b es: 007b ss: 0068 Process 10-udev.hotplug (pid: 398, threadinfo=ecb3c000 task=ecb2a070) Stack: c0141b69 ecb4b000 c19fdd80 00000001 ecb4b000 eca6aaa0 c19fdd80 c19fde40 c0141ceb c19fdd80 eca6aaa0 00000001 c19fdd80 eca6aaa0 ecb4b000 00000010 00000001 000000d0 c1a1b080 0000000c c19fde08 c19fdd80 c0141eda c19fdd80 Call Trace: [<c0141b69>] cache_init_objs+0x35/0x56 [<c0141ceb>] cache_grow+0xfb/0x187 [<c0141eda>] cache_alloc_refill+0x163/0x19c [<c01420f5>] kmem_cache_alloc+0x67/0x97 [<c0111671>] pgd_alloc+0x17/0x336 [<c01199d4>] mm_init+0xd7/0x116 [<c01199e4>] mm_init+0xe7/0x116 [<c0119c8a>] copy_mm+0xbb/0x396 [<c0268f10>] __cond_resched+0x14/0x3c [<c011aa5a>] copy_process+0x6b5/0xb0b [<c011af9d>] do_fork+0x8a/0x16b [<c0107507>] error_code+0x2b/0x30 [<c0105d2c>] sys_clone+0x24/0x28 [<c010737f>] syscall_call+0x7/0xb Code: 74 02 66 a5 a8 01 74 01 a4 5e 5b 5e 5f c3 80 3d 04 07 2f c0 00 75 1c 6a 20 6a 00 ff 74 24 0c e8 ce 37 00 00 83 c4 0c 85 c0 74 08 <0f> 0b 32 01 b6 31 27 c0 c3 80 3d 04 07 2f c0 00 75 0d c7 44 24 <0>Fatal exception: panic in 5 seconds ----- Original Message ----- From: "Chris Lalancette" <clalance@redhat.com> To: "Jim Klein" <jklein@saugus.k12.ca.us> Cc: "fedora-xen" <fedora-xen@redhat.com> Sent: Tuesday, October 30, 2007 3:00:05 PM (GMT-0800) America/Los_Angeles Subject: Re: [Fedora-xen] RHEL4 DomU Update Problem Jim Klein wrote:> Interesting: > > XENBUS: Device with no driver: device/vbd/2048 > XENBUS: Device with no driver: device/vif/0 > Freeing unused kernel memory: 124k freed > device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel@redhat.com > Kernel panic - not syncing: Attempted to kill init! > > Not sure where to go from here. >Believe it or not, those XENBUS errors are actually expected; the xenbus driver is loaded before the virtual disk/network drivers, and it sees that it has these nodes without drivers. However, that does lead me to an idea. If I remember correctly, the Xensource kernels have all of the drivers built-in, while we prefer to do things more modular. I''m guessing that you didn''t have the right driver in /etc/modprobe.conf when the kernel was installed, so the initrd doesn''t have the right drivers in it. So what you''ll want to do is: 1) Boot the RHEL-4 domU into the (working) Xen kernel. 2) Edit /etc/modprobe.conf, and add: alias scsi_hostadapter xenblk 3) Remove the RedHat RHEL-4 kernel (rpm -e), and install it again, which should re-generate the initrd with the right stuff in it. Chris Lalancette
Jim Klein wrote:> > Kernel panics (lots of them, all the same): > kernel BUG at arch/i386/mm/pgtable-xen.c:306! > invalid operand: 0000 [#1] > SMP > Modules linked in: ext3 jbd dm_mod xenblk sd_mod scsi_mod > CPU: 0 > EIP: 0061:[<c011163a>] Not tainted VLI > EFLAGS: 00010282 (2.6.9-55.0.9.ELxenU) > EIP is at pgd_ctor+0x1d/0x26 > eax: fffffff4 ebx: 00000000 ecx: f5392000 edx: 00000000 > esi: c19fdd80 edi: eca6aaa0 ebp: 00000001 esp: ecb3cd6c > ds: 007b es: 007b ss: 0068 > Process 10-udev.hotplug (pid: 398, threadinfo=ecb3c000 task=ecb2a070) > Stack: c0141b69 ecb4b000 c19fdd80 00000001 ecb4b000 eca6aaa0 c19fdd80 > c19fde40 > c0141ceb c19fdd80 eca6aaa0 00000001 c19fdd80 eca6aaa0 ecb4b000 > 00000010 > 00000001 000000d0 c1a1b080 0000000c c19fde08 c19fdd80 c0141eda > c19fdd80 > Call Trace: > [<c0141b69>] cache_init_objs+0x35/0x56 > [<c0141ceb>] cache_grow+0xfb/0x187 > [<c0141eda>] cache_alloc_refill+0x163/0x19c > [<c01420f5>] kmem_cache_alloc+0x67/0x97 > [<c0111671>] pgd_alloc+0x17/0x336 > [<c01199d4>] mm_init+0xd7/0x116 > [<c01199e4>] mm_init+0xe7/0x116 > [<c0119c8a>] copy_mm+0xbb/0x396 > [<c0268f10>] __cond_resched+0x14/0x3c > [<c011aa5a>] copy_process+0x6b5/0xb0b > [<c011af9d>] do_fork+0x8a/0x16b > [<c0107507>] error_code+0x2b/0x30 > [<c0105d2c>] sys_clone+0x24/0x28 > [<c010737f>] syscall_call+0x7/0xb > Code: 74 02 66 a5 a8 01 74 01 a4 5e 5b 5e 5f c3 80 3d 04 07 2f c0 00 75 > 1c 6a 20 6a 00 ff 74 24 0c e8 ce 37 00 00 83 c4 0c 85 c0 74 08 <0f> 0b > 32 01 b6 31 27 c0 c3 80 3d 04 07 2f c0 00 75 0d c7 44 24 > <0>Fatal exception: panic in 5 secondsOK. This crash is a bug in the 4.5 kernel that I''ve now fixed. It should be fixed when 4.6 comes out. In the meantime, you probably should be able to work around this bug by reducing the amount of memory you are assigning to that domain. Chris Lalancette
Thanks for the info - I''ll look for the update. I still get the panic, regardless of memory settings. ----- Original Message ----- From: "Chris Lalancette" <clalance@redhat.com> To: "Jim Klein" <jklein@saugus.k12.ca.us> Cc: "fedora-xen" <fedora-xen@redhat.com> Sent: Wednesday, October 31, 2007 9:25:52 AM (GMT-0800) America/Los_Angeles Subject: Re: [Fedora-xen] RHEL4 DomU Update Problem Jim Klein wrote:> > Kernel panics (lots of them, all the same): > kernel BUG at arch/i386/mm/pgtable-xen.c:306! > invalid operand: 0000 [#1] > SMP > Modules linked in: ext3 jbd dm_mod xenblk sd_mod scsi_mod > CPU: 0 > EIP: 0061:[<c011163a>] Not tainted VLI > EFLAGS: 00010282 (2.6.9-55.0.9.ELxenU) > EIP is at pgd_ctor+0x1d/0x26 > eax: fffffff4 ebx: 00000000 ecx: f5392000 edx: 00000000 > esi: c19fdd80 edi: eca6aaa0 ebp: 00000001 esp: ecb3cd6c > ds: 007b es: 007b ss: 0068 > Process 10-udev.hotplug (pid: 398, threadinfo=ecb3c000 task=ecb2a070) > Stack: c0141b69 ecb4b000 c19fdd80 00000001 ecb4b000 eca6aaa0 c19fdd80 > c19fde40 > c0141ceb c19fdd80 eca6aaa0 00000001 c19fdd80 eca6aaa0 ecb4b000 > 00000010 > 00000001 000000d0 c1a1b080 0000000c c19fde08 c19fdd80 c0141eda > c19fdd80 > Call Trace: > [<c0141b69>] cache_init_objs+0x35/0x56 > [<c0141ceb>] cache_grow+0xfb/0x187 > [<c0141eda>] cache_alloc_refill+0x163/0x19c > [<c01420f5>] kmem_cache_alloc+0x67/0x97 > [<c0111671>] pgd_alloc+0x17/0x336 > [<c01199d4>] mm_init+0xd7/0x116 > [<c01199e4>] mm_init+0xe7/0x116 > [<c0119c8a>] copy_mm+0xbb/0x396 > [<c0268f10>] __cond_resched+0x14/0x3c > [<c011aa5a>] copy_process+0x6b5/0xb0b > [<c011af9d>] do_fork+0x8a/0x16b > [<c0107507>] error_code+0x2b/0x30 > [<c0105d2c>] sys_clone+0x24/0x28 > [<c010737f>] syscall_call+0x7/0xb > Code: 74 02 66 a5 a8 01 74 01 a4 5e 5b 5e 5f c3 80 3d 04 07 2f c0 00 75 > 1c 6a 20 6a 00 ff 74 24 0c e8 ce 37 00 00 83 c4 0c 85 c0 74 08 <0f> 0b > 32 01 b6 31 27 c0 c3 80 3d 04 07 2f c0 00 75 0d c7 44 24 > <0>Fatal exception: panic in 5 secondsOK. This crash is a bug in the 4.5 kernel that I''ve now fixed. It should be fixed when 4.6 comes out. In the meantime, you probably should be able to work around this bug by reducing the amount of memory you are assigning to that domain. Chris Lalancette
Jim Klein wrote:> Thanks for the info - I''ll look for the update. I still get the panic, > regardless of memory settings. >Hm, OK. Can you try one of the kernels from here: http://people.redhat.com/~jbaron/rhel4/ These are the pre-release 4.6 kernels (not officially supported, of course), and should have the fix for your problem. Chris Lalancette