I’m running Xen 3.0.3 on a CentOS 5.3 x86 (32-bit) platform. Total system memory is 4GB with 4GB swap. Hard drive is approx 575GB available for root partition. I’ve created 3 Xen virtual machines all also running CentOS 5.3. The images are all using the default CentOS 5.3 LVM partitioning assignment and are allocated 150GB of disk space and max RAM is 1250. The kernel (on both dom0 AND the machines) used reports “2.6.18-128.1.6.el5xen #1 SMP i686 i686 i386 GNU/Linux”. Each virtual machine performs a nightly rsync from another primary server. All drives are formatted as ext3 (dom0 and machines). The disk images are stored under /vm with permissions 755 and ownership root:root and are saved in .img format. No special setup/configuration was taken for the Xen installation. My problem is that 2 of my 3 machines will, at random, remount their root partition as read-only. I am able to console in and reboot the machine. On boot, the machine requires a manual fsck. After the fsck completes, I reboot and the vm is fine for a few days. Then, randomly again, the drive remounts read-only. This happens on the 2nd and 3rd machine I created in dom0. The 1st machine has been fine since install. Has anyone seen this happen before? I came across a thread from 2007 that died out with a suggestion that it may be the hardware of dom0 but no other suggestions (I’d hate for it to be hardware since the server is new). Justin T. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, 30 May 2009 00:37:36 +0300, Justin Tianen <jtianen@gmellc.com> wrote:> My problem is that 2 of my 3 machines will, at random, remount their > root partition as read-only. I am able to console in and reboot the > machine. On boot, the machine requires a manual fsck. After the fsck > completes, I reboot and the vm is fine for a few days. Then, randomly > again, the drive remounts read-only. This happens on the 2nd and 3rd > machine I created in dom0. The 1st machine has been fine since install.Have you checked the dmesg for additional details when it happens? I have seen similar thing before (not with Xen tho) and the cause was read/write timeout due to too high I/O usage since multiple backups were running at the same time and the backend storage couldn''t handle it fast enough. So if you already didn''t, you should set the rsync jobs to perform at different times (also check when some other I/O intensive tasks like updatedb are ran and plan the times so they at least try to avoid running at the same time). You could also try to incrase the timeout values. -Eljas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/5/30 Justin Tianen <jtianen@gmellc.com>:> The disk images are stored under /vm with permissions 755 and > ownership root:root and are saved in .img format.Check your domU config. Does it use file:/ ? For best performance and reliablity you should use phy:/ with LVM/partition for domU storage. Also check dom0''s logs, does it contain any disk or I/O-related error? -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I’m going through the logs right now looking for something more specific (however, I don’t know the time of day the disk remounts outside of usually during the rsync backup). I’ve already gone ahead and staggered the backup launch times to see if that helps at all. The filesystem mounting issue relates only to those being handled as block devices (sda1) which I’m not doing. Here’s what the domU configs all look like (differences are the names, uuid, vif mac, and disk image name). name = "server1" uuid = "52fa8774-020d-d66c-d47c-f1b3ecb66417" maxmem = 1250 memory = 1250 vcpus = 1 bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ ] disk = [ "tap:aio:/vm/ server1.img,xvda,w" ] vif = [ "mac=00:16:3e:12:b6:bf,bridge=xenbr0" ] This is all being ran on a CentOS 5.3 server (and each domU is also CentOS 5.3). All 3 domU were setup via the Python script “virt-install” that is included in the CentOS version of Xen. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Do you get the issue when using LVM volumes? From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Justin Tianen Sent: 02 June 2009 15:44 To: xen-users@lists.xensource.com Subject: Re: [Xen-users] disk image corruption I''m going through the logs right now looking for something more specific (however, I don''t know the time of day the disk remounts outside of usually during the rsync backup). I''ve already gone ahead and staggered the backup launch times to see if that helps at all. The filesystem mounting issue relates only to those being handled as block devices (sda1) which I''m not doing. Here''s what the domU configs all look like (differences are the names, uuid, vif mac, and disk image name). name = "server1" uuid = "52fa8774-020d-d66c-d47c-f1b3ecb66417" maxmem = 1250 memory = 1250 vcpus = 1 bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ ] disk = [ "tap:aio:/vm/ server1.img,xvda,w" ] vif = [ "mac=00:16:3e:12:b6:bf,bridge=xenbr0" ] This is all being ran on a CentOS 5.3 server (and each domU is also CentOS 5.3). All 3 domU were setup via the Python script "virt-install" that is included in the CentOS version of Xen. The SAQ Group Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ SAQ is the trading name of SEMTEC Limited. Registered in England & Wales Company Number: 06481952 http://www.saqnet.co.uk AS29219 SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. ISPA Member Find us in http://www.thebestof.co.uk/petersfield _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Yes. The dom0 is running as a regular disk without LVM (/dev/sda1 = /). All the domU are configured using LVM (default sizes for CentOS/RedHat Enterprise/Fedora). Justin Tianen Classic Services (414) 771-0827 jtianen@gmellc.com> -----Original Message----- > From: Robert Dunkley [mailto:Robert@saq.co.uk] > Sent: Tuesday, June 02, 2009 9:55 AM > To: Justin Tianen; xen-users@lists.xensource.com > Subject: RE: [Xen-users] disk image corruption > > Do you get the issue when using LVM volumes? > > > > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Justin > Tianen > Sent: 02 June 2009 15:44 > To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] disk image corruption > > > > I''m going through the logs right now looking for something more > specific > (however, I don''t know the time of day the disk remounts outside of > usually during the rsync backup). I''ve already gone ahead and staggered > the backup launch times to see if that helps at all. The filesystem > mounting issue relates only to those being handled as block devices > (sda1) which I''m not doing. Here''s what the domU configs all look like > (differences are the names, uuid, vif mac, and disk image name). > > > > name = "server1" > > uuid = "52fa8774-020d-d66c-d47c-f1b3ecb66417" > > maxmem = 1250 > > memory = 1250 > > vcpus = 1 > > bootloader = "/usr/bin/pygrub" > > on_poweroff = "destroy" > > on_reboot = "restart" > > on_crash = "restart" > > vfb = [ ] > > disk = [ "tap:aio:/vm/ server1.img,xvda,w" ] > > vif = [ "mac=00:16:3e:12:b6:bf,bridge=xenbr0" ] > > > > This is all being ran on a CentOS 5.3 server (and each domU is also > CentOS 5.3). All 3 domU were setup via the Python script "virt-install" > that is included in the CentOS version of Xen. > > > The SAQ Group > > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ > SAQ is the trading name of SEMTEC Limited. Registered in England & > Wales > Company Number: 06481952 > > http://www.saqnet.co.uk AS29219 > > SAQ Group Delivers high quality, honestly priced communication and I.T. > services to UK Business. > > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : > Transit : Backups : Managed Networks : Remote Support. > > ISPA Member > > Find us in http://www.thebestof.co.uk/petersfield > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 8.5.339 / Virus Database: 270.12.50/2150 - Release Date: > 06/02/09 06:47:00No virus found in this outgoing message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.50/2150 - Release Date: 06/02/09 06:47:00 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sorry. None of the VM images are actually mounted in dom0. Each .img file
contains a LVM setup as configured by the OS installer. Here''s what the
disk looks like (physical; then VM).
root@vps [/etc/xen]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 611025584 203800484 376186720 36% /
/dev/sda1 256666 20642 222772 9% /boot
tmpfs 1877400 0 1877400 0% /dev/shm
/usr/tmpDSK 495844 10545 459699 3% /tmp
none 1877312 160 1877152 1% /var/lib/xenstored
root@vps [/etc/xen]# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 300 4 r----- 29074.3
server1 22 1249 1 -b---- 15.2
root@vps [/etc/xen]# xm console server1
[root@server1-backup proc]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
150263916 22599896 119907836 16% /
/dev/xvda1 101086 27498 68369 29% /boot
tmpfs 640088 0 640088 0% /dev/shm
Justin Tianen
Classic Services
(414) 771-0827
jtianen@gmellc.com
> -----Original Message-----
> From: Robert Dunkley [mailto:Robert@saq.co.uk]
> Sent: Tuesday, June 02, 2009 10:03 AM
> To: Justin Tianen
> Subject: RE: [Xen-users] disk image corruption
>
> I might be confused here but why are you specifying a filename for the
> disk if the VM has its own LVM partition?
>
> My dedicated LVM partition configs look a bit like this:
> disk = [ ''phy:/dev/lvvolumegroup/lvvolume,hda,w'']
>
>
>
> -----Original Message-----
> From: Justin Tianen [mailto:jtianen@gmellc.com]
> Sent: 02 June 2009 15:58
> To: Robert Dunkley; xen-users@lists.xensource.com
> Subject: RE: [Xen-users] disk image corruption
>
> Yes. The dom0 is running as a regular disk without LVM (/dev/sda1 = /).
> All the domU are configured using LVM (default sizes for CentOS/RedHat
> Enterprise/Fedora).
>
> Justin Tianen
> Classic Services
> (414) 771-0827
> jtianen@gmellc.com
>
>
> > -----Original Message-----
> > From: Robert Dunkley [mailto:Robert@saq.co.uk]
> > Sent: Tuesday, June 02, 2009 9:55 AM
> > To: Justin Tianen; xen-users@lists.xensource.com
> > Subject: RE: [Xen-users] disk image corruption
> >
> > Do you get the issue when using LVM volumes?
> >
> >
> >
> > From: xen-users-bounces@lists.xensource.com
> > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Justin
> > Tianen
> > Sent: 02 June 2009 15:44
> > To: xen-users@lists.xensource.com
> > Subject: Re: [Xen-users] disk image corruption
> >
> >
> >
> > I''m going through the logs right now looking for something
more
> > specific
> > (however, I don''t know the time of day the disk remounts
outside of
> > usually during the rsync backup). I''ve already gone ahead and
> staggered
> > the backup launch times to see if that helps at all. The filesystem
> > mounting issue relates only to those being handled as block devices
> > (sda1) which I''m not doing. Here''s what the domU
configs all look
> like
> > (differences are the names, uuid, vif mac, and disk image name).
> >
> >
> >
> > name = "server1"
> >
> > uuid = "52fa8774-020d-d66c-d47c-f1b3ecb66417"
> >
> > maxmem = 1250
> >
> > memory = 1250
> >
> > vcpus = 1
> >
> > bootloader = "/usr/bin/pygrub"
> >
> > on_poweroff = "destroy"
> >
> > on_reboot = "restart"
> >
> > on_crash = "restart"
> >
> > vfb = [ ]
> >
> > disk = [ "tap:aio:/vm/ server1.img,xvda,w" ]
> >
> > vif = [ "mac=00:16:3e:12:b6:bf,bridge=xenbr0" ]
> >
> >
> >
> > This is all being ran on a CentOS 5.3 server (and each domU is also
> > CentOS 5.3). All 3 domU were setup via the Python script
> "virt-install"
> > that is included in the CentOS version of Xen.
> >
> >
> > The SAQ Group
> >
> > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
> > SAQ is the trading name of SEMTEC Limited. Registered in England &
> > Wales
> > Company Number: 06481952
> >
> > http://www.saqnet.co.uk AS29219
> >
> > SAQ Group Delivers high quality, honestly priced communication and
> I.T.
> > services to UK Business.
> >
> > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks :
> > Transit : Backups : Managed Networks : Remote Support.
> >
> > ISPA Member
> >
> > Find us in http://www.thebestof.co.uk/petersfield
> >
> >
> > No virus found in this incoming message.
> > Checked by AVG - www.avg.com
> > Version: 8.5.339 / Virus Database: 270.12.50/2150 - Release Date:
> > 06/02/09 06:47:00
>
> No virus found in this outgoing message.
> Checked by AVG - www.avg.com
> Version: 8.5.339 / Virus Database: 270.12.50/2150 - Release Date:
> 06/02/09 06:47:00
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.339 / Virus Database: 270.12.50/2150 - Release Date:
> 06/02/09 06:47:00
No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.339 / Virus Database: 270.12.50/2150 - Release Date: 06/02/09
06:47:00
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users