Hi. I''m using Xen on RHEL cluster, and I have strange problems. I gave raw volumes from storage to Xen virtual machines. With windows, I have a problem that nodes don''t see the volume as same one.... for example: clusternode1# clusvcadm -d vm:winxp clusternode1# dd if=/dev/mapper/winxp of=/node1winxp clusternode2# dd if=/dev/mapper/winxp of=/node2winxp clusternode3# dd if=/dev/mapper/winxp of=/node3winxp When I download these files and diff them, they all three differ. Also, sometimes very strange things happen. For example I download some file into winxp, shut it down, then start it on another node, and file is missing?!?!?!?! Should I use maybe CLVM and not raw volumes from storage? Why is this happening? -- | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | ================================================================| start fighting cancer -> http://www.worldcommunitygrid.org/ | _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Feb 27, 2010 at 11:07:41AM +0100, Jakov Sosic wrote:> Hi. > > I''m using Xen on RHEL cluster, and I have strange problems. I gave raw > volumes from storage to Xen virtual machines. With windows, I have a > problem that nodes don''t see the volume as same one.... for example: > > clusternode1# clusvcadm -d vm:winxp > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > When I download these files and diff them, they all three differ. > > Also, sometimes very strange things happen. For example I download some > file into winxp, shut it down, then start it on another node, and file > is missing?!?!?!?! > > Should I use maybe CLVM and not raw volumes from storage? Why is this > happening? >What kind of xen disk backend are you using? phy:? What kind of storage are you using? When sharing block devices in a cluster across nodes you need to make absolutely sure there''s no caching going on in dom0s, so that the disk contents are always in-sync. CLVM only helps with safely managing the LVM metadata, not with the actual usage of the volumes. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I''m using Xen on RHEL cluster, and I have strange problems. I gave raw > volumes from storage to Xen virtual machines. With windows, I have a > problem that nodes don''t see the volume as same one.... for example: > > clusternode1# clusvcadm -d vm:winxp > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > When I download these files and diff them, they all three differ. > > Also, sometimes very strange things happen. For example I download some > file into winxp, shut it down, then start it on another node, and file > is missing?!?!?!?! > > Should I use maybe CLVM and not raw volumes from storage? Why is this > happening?You mean LUN''s from a storage? Do you use the GPLPV drivers inside the Windows HVM ? Then you probably run into the same or a similar trouble which is described in this bug report: https://bugzilla.redhat.com/show_bug.cgi?id=466681 Dropping vm caches helps: echo 1 > /proc/sys/vm/drop_caches Also using clvm helps, but you will loose the snapshot features of the storage, lvm snapshot are not possible in a clustered environment. Sincerly, Klaus _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I had same situation, though I am not using cluster (I am using multipath and raw devices on shared storage). My workaround is issuing: echo 1 > /proc/sys/vm/drop_caches after finishing copy or after an external change (e.g. by action on the shared storage independently from host) on the raw device. It works like a charm, but I am not sure what would be the right solution with cluster and raw devices - GFS is fine with files, CLVM with LVMs, but I am not using LVM either (and CLVM has a drawback - snapshots are not supported). ZP. 2010/2/27 Jakov Sosic <jakov.sosic@srce.hr>> Hi. > > I''m using Xen on RHEL cluster, and I have strange problems. I gave raw > volumes from storage to Xen virtual machines. With windows, I have a > problem that nodes don''t see the volume as same one.... for example: > > clusternode1# clusvcadm -d vm:winxp > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > When I download these files and diff them, they all three differ. > > Also, sometimes very strange things happen. For example I download some > file into winxp, shut it down, then start it on another node, and file > is missing?!?!?!?! > > Should I use maybe CLVM and not raw volumes from storage? Why is this > happening? > > > -- > | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | > ================================================================> | start fighting cancer -> http://www.worldcommunitygrid.org/ | > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I would appreciate for some advice on how to recover a broken DomU image. On an OpenSuse 10.3 (x86-64) server, I am hosting a OpenSuse 10.3 (x86-64) guest, using an image file. The guest os has been running fine for sometime, until recently it suddenly reported IO errors to whatever commands one entered. At that point, the Guest OS would not allow any ssh log in attempts. I stopped the domU by issuing "xm shutdown guest", and restart it with "xm create -c guest". However it fails to start up with an error: Error: Boot loader didn''t return any data! I did try to use lomount: lomount -diskimage image-file -partition 1 mnt-pt but it fails with mount: Structure needs cleaning Google search on the above error suggests it is an xfs file system error and it needs xfs_repair on a usual system with a native disk. However, xfs_repair does not provide an option to select offset for partition 1. Any idea how to recover from this situation? Many thanks. Peter -- Scanned by iCritical. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 02/27/2010 02:07 PM, Pasi Kärkkäinen wrote:> On Sat, Feb 27, 2010 at 11:07:41AM +0100, Jakov Sosic wrote: >> Hi. >> >> I''m using Xen on RHEL cluster, and I have strange problems. I gave raw >> volumes from storage to Xen virtual machines. With windows, I have a >> problem that nodes don''t see the volume as same one.... for example: >> >> clusternode1# clusvcadm -d vm:winxp >> clusternode1# dd if=/dev/mapper/winxp of=/node1winxp >> clusternode2# dd if=/dev/mapper/winxp of=/node2winxp >> clusternode3# dd if=/dev/mapper/winxp of=/node3winxp >> >> When I download these files and diff them, they all three differ. >> >> Also, sometimes very strange things happen. For example I download some >> file into winxp, shut it down, then start it on another node, and file >> is missing?!?!?!?! >> >> Should I use maybe CLVM and not raw volumes from storage? Why is this >> happening? >> > > What kind of xen disk backend are you using? phy:? > What kind of storage are you using? > > When sharing block devices in a cluster across nodes you need > to make absolutely sure there''s no caching going on in dom0s, > so that the disk contents are always in-sync. > > CLVM only helps with safely managing the LVM metadata, > not with the actual usage of the volumes.As far as I''ve managed to test, CLVM actually does resolve this issue. I don''t know how, but starting xen domU on one node, and stopping it, and after that doing dd of the volume on three nodes gives me exactly the same binary images without dropping caches. With purely raw volume, I was not able to achieve this results. -- | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | ================================================================| start fighting cancer -> http://www.worldcommunitygrid.org/ | _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 02/27/2010 06:35 PM, Klaus Steinberger wrote:> >> I''m using Xen on RHEL cluster, and I have strange problems. I gave raw >> volumes from storage to Xen virtual machines. With windows, I have a >> problem that nodes don''t see the volume as same one.... for example: >> >> clusternode1# clusvcadm -d vm:winxp >> clusternode1# dd if=/dev/mapper/winxp of=/node1winxp >> clusternode2# dd if=/dev/mapper/winxp of=/node2winxp >> clusternode3# dd if=/dev/mapper/winxp of=/node3winxp >> >> When I download these files and diff them, they all three differ. >> >> Also, sometimes very strange things happen. For example I download some >> file into winxp, shut it down, then start it on another node, and file >> is missing?!?!?!?! >> >> Should I use maybe CLVM and not raw volumes from storage? Why is this >> happening? > > You mean LUN''s from a storage? Do you use the GPLPV drivers inside the Windows HVM ? > > Then you probably run into the same or a similar trouble which is described in > this bug report: > > https://bugzilla.redhat.com/show_bug.cgi?id=466681 > > > Dropping vm caches helps: > > echo 1 > /proc/sys/vm/drop_caches > > Also using clvm helps, but you will loose the snapshot features of the storage, > lvm snapshot are not possible in a clustered environment.Thank you very much... This forces me to migrate 3TB of data to CLVM volumes, but at least I know that it a working combination... I dropped CLVM because of it''s limits (cluster has to be up for it to map volumes, can''t do snapshots etc), but at least with CLVM everything works. -- | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | ================================================================| start fighting cancer -> http://www.worldcommunitygrid.org/ | _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Feb 27, 2010 at 07:48:50PM +0100, Zoran Popovi? wrote:> I had same situation, though I am not using cluster (I am using multipath > and raw devices on shared storage). My workaround is issuing: > > echo 1 > /proc/sys/vm/drop_caches > > after finishing copy or after an external change (e.g. by action on the > shared storage independently from host) on the raw device. It works like a > charm, but I am not sure what would be the right solution with cluster and > raw devices - GFS is fine with files, CLVM with LVMs, but I am not using > LVM either (and CLVM has a drawback - snapshots are not supported). >Hmm.. I wonder what''s causing that behaviour. Xen phy: disk backend should skip all dom0 caches, so you should not need drop_caches.. Ideas how to track resolve this? -- Pasi> ZP. > > 2010/2/27 Jakov Sosic <[1]jakov.sosic@srce.hr> > > Hi. > > I''m using Xen on RHEL cluster, and I have strange problems. I gave raw > volumes from storage to Xen virtual machines. With windows, I have a > problem that nodes don''t see the volume as same one.... for example: > > clusternode1# clusvcadm -d vm:winxp > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > When I download these files and diff them, they all three differ. > > Also, sometimes very strange things happen. For example I download some > file into winxp, shut it down, then start it on another node, and file > is missing?!?!?!?! > > Should I use maybe CLVM and not raw volumes from storage? Why is this > happening? > > -- > | Jakov Sosic | ICQ: 28410271 | PGP: 0x965CAE2D | > ================================================================> | start fighting cancer -> [2]http://www.worldcommunitygrid.org/ | > > _______________________________________________ > Xen-users mailing list > [3]Xen-users@lists.xensource.com > [4]http://lists.xensource.com/xen-users > > References > > Visible links > 1. mailto:jakov.sosic@srce.hr > 2. http://www.worldcommunitygrid.org/ > 3. mailto:Xen-users@lists.xensource.com > 4. http://lists.xensource.com/xen-users> _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Feb 27, 2010 at 07:48:50PM +0100, Zoran Popovi? wrote:> I had same situation, though I am not using cluster (I am using multipath > and raw devices on shared storage). My workaround is issuing: > > echo 1 > /proc/sys/vm/drop_caches > > after finishing copy or after an external change (e.g. by action on the > shared storage independently from host) on the raw device. It works like a > charm, but I am not sure what would be the right solution with cluster and > raw devices - GFS is fine with files, CLVM with LVMs, but I am not using > LVM either (and CLVM has a drawback - snapshots are not supported). > > ZP. > > 2010/2/27 Jakov Sosic <[1]jakov.sosic@srce.hr> > > Hi. > > I''m using Xen on RHEL cluster, and I have strange problems. I gave raw > volumes from storage to Xen virtual machines. With windows, I have a > problem that nodes don''t see the volume as same one.... for example: > > clusternode1# clusvcadm -d vm:winxp > clusternode1# dd if=/dev/mapper/winxp of=/node1winxp > clusternode2# dd if=/dev/mapper/winxp of=/node2winxp > clusternode3# dd if=/dev/mapper/winxp of=/node3winxp > > When I download these files and diff them, they all three differ. > > Also, sometimes very strange things happen. For example I download some > file into winxp, shut it down, then start it on another node, and file > is missing?!?!?!?! > > Should I use maybe CLVM and not raw volumes from storage? Why is this > happening? >What are these /dev/mapper/winxp devices? multipath devices? Are you using drbd, iSCSI or FC? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> after finishing copy or after an external change (e.g. by action on the > shared storage independently from host) on the raw device. It works like > a charm, but I am not sure what would be the right solution with cluster > and raw devices - GFS is fine with files, CLVM with LVMs, but I am not > using LVM either (and CLVM has a drawback - snapshots are not supported).Yeah, if you''re sharing storage in a cluster, use CLVM. It ensures that things are sync''d between the nodes and that''s.. uh, rather important. CLVM doesn''t support snapshots and pvmoves (others?) but you don''t need them -- you can still snapshot within your domU''s. Why the need to snapshot from dom0? John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> This forces me to migrate 3TB of data to CLVM volumes, but at least I > know that it a working combination... I dropped CLVM because of it''s > limits (cluster has to be up for it to map volumes, can''t do snapshots > etc), but at least with CLVM everything works.You can do it live though, aside from any reboots you need to set up the cluster infrastructure (and you probably should reboot to ensure it all works, it can be tricky). dom0: xm block-attach domU phy:/dev/clvmvg/clvmlv sdN w domU: fdisk /dev/sdN ... domU: pvcreate /dev/sdN1 domU: vgextend /dev/vgnamehere /dev/sdN1 domU: pvmove ... domU: vgreduce ... dom0: xm block-detach domU <id of non-CLVM lv> It''s taken a while, but I''m a fan of the RHCS stuff. There are some caveats (`service clvmd stop; service cman stop` before you reboot a node, other trickiness with getting and keeping everything sync''d up) but it''s well worth the hassle to keep things consistent. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Montag, den 01.03.2010, 10:23 -0500 schrieb John Madden:> > after finishing copy or after an external change (e.g. by action on the > > shared storage independently from host) on the raw device. It works like > > a charm, but I am not sure what would be the right solution with cluster > > and raw devices - GFS is fine with files, CLVM with LVMs, but I am not > > using LVM either (and CLVM has a drawback - snapshots are not supported). > > Yeah, if you''re sharing storage in a cluster, use CLVM. It ensures that > things are sync''d between the nodes and that''s.. uh, rather important. > > CLVM doesn''t support snapshots and pvmoves (others?) but you don''t need > them -- you can still snapshot within your domU''s. Why the need to > snapshot from dom0?For Live-Backups from running domUs ;-) Im doing hot-backups through lvm-snapshots, since there''s no need for backup-agents in dom0, which simplifies administration.> > JohnRegards, Thomas> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>> CLVM doesn''t support snapshots and pvmoves (others?) but you don''t need >> them -- you can still snapshot within your domU''s. Why the need to >> snapshot from dom0? > > For Live-Backups from running domUs ;-)Right, so CLVM is not a problem. The domU''s have no knowledge of clvm (and they don''t need any) -- they can run snapshots just fine. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi John, I think Thomas meant "For Live-Backups from dom0 of running domUs ;-)" Performing snapshot inside dom0 avoid the need to install LVM utilities into domU. And then, if CLVM does not support snapshots, you can not do that. Cheers, JB On 01/03/2010 18:02, John Madden wrote:>>> CLVM doesn''t support snapshots and pvmoves (others?) but you don''t >>> need them -- you can still snapshot within your domU''s. Why the need >>> to snapshot from dom0? >> >> For Live-Backups from running domUs ;-) > > Right, so CLVM is not a problem. The domU''s have no knowledge of clvm > (and they don''t need any) -- they can run snapshots just fine. > > John > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I think Thomas meant "For Live-Backups from dom0 of running domUs ;-)" > Performing snapshot inside dom0 avoid the need to install LVM utilities > into domU. And then, if CLVM does not support snapshots, you can not do > that.I see. I thought he said he does NO backups from dom0. If you ask me, not using LVM inside the guest is silly. Why not run LVM everywhere you possibly can, if for no other reason than to have disk flexibility? ''Seems to me that''d be like running Solaris without ZFS. :) John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I would think running LVM in Dom0 instead of DomU gains you performance by using Dom0''s CPU & disk cache instead of taking it out of the DomU resources. S.W. Xen newb On 3/1/2010 1:20 PM, John Madden wrote:>> I think Thomas meant "For Live-Backups from dom0 of running domUs ;-)" >> Performing snapshot inside dom0 avoid the need to install LVM utilities >> into domU. And then, if CLVM does not support snapshots, you can not do >> that. > > I see. I thought he said he does NO backups from dom0. > > If you ask me, not using LVM inside the guest is silly. Why not run > LVM everywhere you possibly can, if for no other reason than to have > disk flexibility? ''Seems to me that''d be like running Solaris without > ZFS. :) > > John > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steven Wilcoxon wrote:> I would think running LVM in Dom0 instead of DomU gains you performance > by using Dom0''s CPU & disk cache instead of taking it out of the DomU > resources.First off, the overhead of LVM in i/o and such is likely a small percentage of the i/o of Xen itself, which we all know to be very small. In other words, its overhead is negligible no matter where it''s run. Second, I don''t know what you''re referring to here with cpu and disk cache, especially with regard to disk cache, as LVM does little or none as far as I know. I run my dom0''s with a hard-coded 512MB so I don''t have any cache there anyway, but I''m running LVM on top of LVM (phy: mappings of lv''s) so cache is irrelevant). Unless you''re using tap:aio, if you''re going to use cache, it''s in the domU. Third, if there is some resource utilization going on specific to a domU, I''d rather have that happen in the domU where it has minimal impact on other domU''s rather than in dom0 where it affects them all. There are exceptions: you don''t want things that have to cross the hypervisor twice to exist in domU like RAID and multipathing. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
<peter.chiu@stfc.ac.uk>
2010-Mar-24 19:14 UTC
[Xen-users] Full Virtualised Guest not booting with waiting for sda2 to appear error
I would appreciate for some advice on how to resolve a DomU booting problem. On an OpenSuse 11.2 (x86-64) server, I am trying to convert an OpenSuse 10.0 (x86-64) system as a full virtualised guest, using an image file. The image file is produced by running dd if=/dev/sda | ssh xen_server "dd of=/disks/guest_image". This method has worked before on other OpenSuse 10.0 (x86_64) server, but not this time. Basically when the guest os starts up, it seems to start up okay, but later complains "Waiting for /dev/sda2 to appear.... not found - exiting to /bin/sh" and subsequently times out and returns a $ prompt, which does not seem to recognise any commands. I am aware that I need to a. modify the guest config file so that the disk statement is referring to hda, b. the image itself needs to be modified in its /boot/grub/menu.lst so that all references on sda2 are replaced by hda2, and c. all references of sda2 in /etc/fstab are replaced by hda2. This trick seems to work on another host (which incidentally uses sda1 / hda1, as oppose to sda2 / hda2). But this time it does not. Interesting enough, despite of the change of the /boot/grub/menu.lst file, it is still looking for /dev/sda2, rather than /dev/hda2. During the boot up sequence, I can enter into the boot OS selection menu, select the boot entry, and edit the root device from /dev/sda2 to /dev/hda2. Only this time, it fails with a similar error: "Waiting for /dev/hda2 to appear.... not found - exiting to /bin/sh" Any idea how to recover from this situation? Many thanks. Peter -- Scanned by iCritical. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users