I''m being plagued by repeated corruption of file systems for a few of my DomUs. I originally thought it was a failing hard drive, so I''ve replaced the hard drive but my problems continue. I always end up with the following on the DomU console: [177949.345654] EXT3-fs (xvda2): error in ext3_new_inode: IO failure [177949.345669] Aborting journal on device xvda2. I''m using LVMs to house my file systems. Can anyone help me root out what the cause of this is? Is it reasonable to blame software for this? -Robert
On Mon, Oct 1, 2012 at 3:44 AM, Robert Rust <robert@digitalsilk.net> wrote:> I''m being plagued by repeated corruption of file systems for a few of my > DomUs. I originally thought it was a failing hard drive, so I''ve replaced > the hard drive but my problems continue. I always end up with the following > on the DomU console: > [177949.345654] EXT3-fs (xvda2): error in ext3_new_inode: IO failure > [177949.345669] Aborting journal on device xvda2. > > I''m using LVMs to house my file systems. Can anyone help me root out what > the cause of this is? Is it reasonable to blame software for this?IIRC I had something like that a long time ago, and it was solved by xen update. Can''t remember exactly which version though. Try using the latest version, either from upstream directly, or one that your distro bundled. ... And if you''re using distro-bundled version, make sure to file a bug fix/support request to the distro/package maintainers. -- Fajar
inline ----- Original Message ----- From: Robert Rust <robert@digitalsilk.net> To: xen-users@lists.xen.org Cc: Sent: Sunday, September 30, 2012 1:44 PM Subject: [Xen-users] repeated domU filesystem corruption I''m being plagued by repeated corruption of file systems for a few of my DomUs. I originally thought it was a failing hard drive, so I''ve replaced the hard drive but my problems continue. I always end up with the following on the DomU console: [177949.345654] EXT3-fs (xvda2): error in ext3_new_inode: IO failure [177949.345669] Aborting journal on device xvda2. I''m using LVMs to house my file systems. Can anyone help me root out what the cause of this is? Is it reasonable to blame software for this? -Robert ------------ If its a newer kernel on recent ubuntu, try adding to fstab options (barrier=0) # grep barrier /etc/fstab /dev/xvda / ext4 errors=remount-ro,barrier=0 0 1 -- Mark
I was experiencing the same with Ubuntu domU''s just some months before. The only solution was to migrate from EXT4 to EXT3 - that barrier=0 option didn''t helped me. I am running Xen bundled by Debian - there are some discussions on the internet about issues handling of barrier''s by Xen. BR, -- Peter On 10/01/2012 03:26 AM, Mark Pryor wrote:> inline > > > > ----- Original Message ----- > From: Robert Rust<robert@digitalsilk.net> > To: xen-users@lists.xen.org > Cc: > Sent: Sunday, September 30, 2012 1:44 PM > Subject: [Xen-users] repeated domU filesystem corruption > > I''m being plagued by repeated corruption of file systems for a few of my DomUs. I originally thought it was a failing hard drive, so I''ve replaced the hard drive but my problems continue. I always end up with the following on the DomU console: > [177949.345654] EXT3-fs (xvda2): error in ext3_new_inode: IO failure > [177949.345669] Aborting journal on device xvda2. > > I''m using LVMs to house my file systems. Can anyone help me root out what the cause of this is? Is it reasonable to blame software for this? > > -Robert > ------------ > If its a newer kernel on recent ubuntu, try adding to fstab options (barrier=0) > > # grep barrier /etc/fstab > /dev/xvda / ext4 errors=remount-ro,barrier=0 0 1 >
On Mon, Oct 1, 2012 at 2:18 PM, Peter Viskup <skupko.sk@gmail.com> wrote:> I was experiencing the same with Ubuntu domU''s just some months before. > The only solution was to migrate from EXT4 to EXT3 - that barrier=0 option > didn''t helped me. > I am running Xen bundled by Debian - there are some discussions on the > internet about issues handling of barrier''s by Xen.I though barriers were removed anyway in newer upstream kernel? Anyway, I was using ubuntu lucid as domU (bundled pv-ops 2.6.32 domU kernel), with custom-build xen 4.0.0, custom-built 2.6.34-xenlinux dom0 kernel on RHEL5, and it works just fine with ext4 on domU. So if that old configuration works fine, latest Ubuntu/debian SHOULD work fine as well. If not, then IMHO you need to file a bug report. -- Fajar
On 10/1/12 2:18 AM, Peter Viskup wrote:> I was experiencing the same with Ubuntu domU''s just some months before. > The only solution was to migrate from EXT4 to EXT3 - that barrier=0 > option didn''t helped me. > I am running Xen bundled by Debian - there are some discussions on the > internet about issues handling of barrier''s by Xen.I am already running EXT3 volumes, so I have added that option to see if it helps. Since I forgot to include in original post... distro: Ubuntu 12.0.4.1 LTS 64-bit kernel: 3.2.0-31-generic #50 Xen package: xen-hypervisor-4.1-amd64 4.1.2-2ubuntu2.2 We''ll see if I can go a week without corruption with the barrier=0 option. I looked into filing a bug per Fajar''s suggestion, but I''m not sure how I''d go about gathering the data they want given the nature of Xen (gather data in DomU or Dom0? is gathering the data several hours after the corruption occurs acceptable since it''s not always immediately apparent?). -Robert
On Mon, Oct 1, 2012 at 6:17 PM, Robert Rust <robert@digitalsilk.net> wrote:> On 10/1/12 2:18 AM, Peter Viskup wrote: >> >> I was experiencing the same with Ubuntu domU''s just some months before. >> The only solution was to migrate from EXT4 to EXT3 - that barrier=0 >> option didn''t helped me. >> I am running Xen bundled by Debian - there are some discussions on the >> internet about issues handling of barrier''s by Xen. > > > I am already running EXT3 volumes, so I have added that option to see if it > helps. Since I forgot to include in original post... > distro: Ubuntu 12.0.4.1 LTS 64-bit > kernel: 3.2.0-31-generic #50 > Xen package: xen-hypervisor-4.1-amd64 4.1.2-2ubuntu2.2 > > We''ll see if I can go a week without corruption with the barrier=0 option. > I looked into filing a bug per Fajar''s suggestion, but I''m not sure how I''d > go about gathering the data they want given the nature of Xen (gather data > in DomU or Dom0?If the error is in domU, then domU''s console (and possibly syslog) messages is usually helpful> is gathering the data several hours after the corruption > occurs acceptable since it''s not always immediately apparent?).Usually you''ll be able to speed it up by doing things like: - sync - apt-get update - fio -- Fajar