Xen users - Feb 2009 - How (not) to destroy a PostgreSQL db in domU on powerfail

Now I''m sure that the following configuration can destroy a PostgreSQL 
8.3.5 database:

* Linux host (dom0) with XEN, XFS filesystem with "nobarrier", RAID 
controller with battery backed cache.
* XEN vm (domU) with XFS filesystem with "nobarrier" with postgresql
* my daughter with 3.5 years switching off the power supply of the 
server, just behind the UPS

Seems XEN does lie about fsync, otherwise it shouldn''t have crashed the
database (so badly that I could not even dump it, had to restore).

Is there a way to secure domU''s in a way they survive a powerfail?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Feb 24, 2009 at 4:20 PM, Michael Monnerie
<michael.monnerie@is.it-management.at> wrote:
> * Linux host (dom0) with XEN, XFS filesystem with "nobarrier",
RAID
> controller with battery backed cache.
> * XEN vm (domU) with XFS filesystem with "nobarrier" with
postgresql
> * my daughter with 3.5 years switching off the power supply of the
> server, just behind the UPS
>
> Is there a way to secure domU''s in a way they survive a powerfail?
Are you using file-backed domU, with file:/ in domU config? If yes,
this is a known problem, and you can try:
- changing file:/ to tap:aio:/ , or
- use block-device-backed storage (e.g. LVM, partition, etc.)

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Dienstag 24 Februar 2009 Fajar A. Nugraha wrote:
> Are you using file-backed domU, with file:/ in domU config? If yes,
> this is a known problem, and you can try:
> - changing file:/ to tap:aio:/ , or
> - use block-device-backed storage (e.g. LVM, partition, etc.)
disk=[
        ''tap:aio:/path/root20090113.xvd,xvda,w'',
        ''tap:aio:/path/work20090116.xvd,xvdb,w'',
and so on.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Michael Monnerie wrote:
> Now I''m sure that the following configuration can destroy a
PostgreSQL
> 8.3.5 database:
>
> * Linux host (dom0) with XEN, XFS filesystem with "nobarrier",
RAID
> controller with battery backed cache.
> * XEN vm (domU) with XFS filesystem with "nobarrier" with
postgresql
> * my daughter with 3.5 years switching off the power supply of the
> server, just behind the UPS
>
> Seems XEN does lie about fsync, otherwise it shouldn''t have
crashed the
> database (so badly that I could not even dump it, had to restore).
>
> Is there a way to secure domU''s in a way they survive a powerfail?
if you don''t have highly reliable hardware, with redundant channels and
battery backed caches, etc. stay away from XFS.

it''s a pity, but XFS is only as reliable as the hardware you give it. 
in
other words: you can be really sure that a software issue won''t trash
your XFS
filesystem; but also that a hardware failure definitely will.

ext3, OTOH, might not have such an impressive record of correctness; but
it''s
designed with redundancy in mind.  almost all vital structures are either 
replicated or reconstructable from other hints.  that makes fsck slow but 
effective, and in most cases, the journal analysis done when remounting after 
an unclean shutdown is enough to put it back on track.

i sure wish i could use XFS in more circumstances, but the needed hardware is 
prohibitive.

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Dienstag 24 Februar 2009 Javier Guerra Giraldez
wrote:
> if you don''t have highly reliable hardware, with redundant
channels
> and battery backed caches, etc. stay away from XFS.
That''s what I have! RAID Controller, battery backed cache, disk write 
cache=off, UPS. Well, the UPS didn''t help as my daughter switched off 
the server directly.
> it''s a pity, but XFS is only as reliable as the hardware you give
it.
>  in other words: you can be really sure that a software issue
won''t
> trash your XFS filesystem; but also that a hardware failure
> definitely will.
> ext3, OTOH, might not have such an impressive record of correctness;
> but it''s designed with redundancy in mind.  almost all vital
> structures are either replicated or reconstructable from other hints.
>  that makes fsck slow but effective, and in most cases, the journal
> analysis done when remounting after an unclean shutdown is enough to
> put it back on track.
>
> i sure wish i could use XFS in more circumstances, but the needed
> hardware is prohibitive.
Here the answer from XFS developers:
*************
The difference is just that you actually see the corruption on XFS while 
it''s pretty silent on extN.  If your Hardware (or Hypervisor) is not 
reliable you _will_ lose data.  Either silently or with a spectacular 
blowup if the filesystem actually has consistency checking (which XFS 
has a lot).
*************

IOW: The problem is that XEN doesn''t strictly follow write barriers. Is
there a way I can make it do that?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 4, 2009 at 3:51 AM, Michael Monnerie
<michael.monnerie@is.it-management.at> wrote:
>
> IOW: The problem is that XEN doesn''t strictly follow write
barriers. Is
> there a way I can make it do that?
Is your domU filesystem (/path/root20090113.xvd ?) located on top of LVM?
If yes, then AFAIK LVM does not honor write-barriers.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mittwoch 04 März 2009 Fajar A. Nugraha wrote:
> Is your domU filesystem (/path/root20090113.xvd ?) located on top of
> LVM? If yes, then AFAIK LVM does not honor write-barriers.
Indeed. LVM is bad? I''ve read LVM is suggested in order to have an 
easier backup with snapshots. But if the result is that your XEN domU 
gets destroyed on a power outage - damn LVM.

Can someone confirm that LVM does not honor write-barriers?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mittwoch 04 März 2009 Michael Monnerie wrote:
> On Mittwoch 04 März 2009 Fajar A. Nugraha wrote:
> > Is your domU filesystem (/path/root20090113.xvd ?) located on top
> > of LVM? If yes, then AFAIK LVM does not honor write-barriers.
>
> Indeed. LVM is bad? I''ve read LVM is suggested in order to have an
> easier backup with snapshots. But if the result is that your XEN domU
> gets destroyed on a power outage - damn LVM.
>
> Can someone confirm that LVM does not honor write-barriers?
http://lkml.org/lkml/2008/5/16/390
http://hightechsorcery.com/2008/06/linux-write-barriers-write-caching-
lvm-and-filesystems

If I understand correctly, turning off disk cache write cache should be 
enough to be save, even when using LVM. Is it really XEN that messed the 
disk, or something else? Could the Areca controller driver do something 
wrong? I''d really love to get a stable and data-secure system, as you 
might understand.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/04/2009 05:44 AM, Michael Monnerie wrote:
> On Mittwoch 04 März 2009 Michael Monnerie wrote:
>    
>> On Mittwoch 04 März 2009 Fajar A. Nugraha wrote:
>>      
>>> Is your domU filesystem (/path/root20090113.xvd ?) located on top
>>> of LVM? If yes, then AFAIK LVM does not honor write-barriers.
>>>        
>> Indeed. LVM is bad? I''ve read LVM is suggested in order to
have an
>> easier backup with snapshots. But if the result is that your XEN domU
>> gets destroyed on a power outage - damn LVM.
>>
>> Can someone confirm that LVM does not honor write-barriers?
>>      
>
> http://lkml.org/lkml/2008/5/16/390
> http://hightechsorcery.com/2008/06/linux-write-barriers-write-caching-
> lvm-and-filesystems
>
> If I understand correctly, turning off disk cache write cache should be
> enough to be save, even when using LVM. Is it really XEN that messed the
> disk, or something else? Could the Areca controller driver do something
> wrong? I''d really love to get a stable and data-secure system, as
you
> might understand.
>
> mfg zmi
>    
Mike,
It''s quite strange, I am running xen with xfs and lvm since a couple of
time and I had some server crash (not power cable failure but still).
It''s well known that lvm do not honors barriers (as ext3 as well by 
default until last year: http://lkml.org/lkml/2008/5/16/390) so it means 
that you are not completely sure that metadata are written before the 
real data are modified, they can still be in the cache and if it vanish 
(due to power outage on not battery backed controller) then you''re on 
your own.
But if you have a battery backed cache then it should be ok, that is to 
my understanding (I can be wrong).
It can be that your battery is dead or not working correctly (or your 
controller not using it ...), the other option is that some metadata 
haven''t leaved the os cache (read: the domU has written the information
to the disk but either dom0 or the hypervisor is mainlining is caching 
data for backend device) and in this case it''s normal to face problem.


It would be very interesting to have more information from xfs team.

Matthieu.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mittwoch 04 März 2009 Matthieu Patou wrote:
> Mike,
> It''s quite strange, I am running xen with xfs and lvm since a
couple
> of time and I had some server crash (not power cable failure but
> still). 
A crash is different from a power fail of course, as the disks don''t 
loose power suddenly.
> It''s well known that lvm do not honors barriers 
I thought that is a bug and fixed already? 

Anyway, I''ve even mounted XFS with "nobarrier", as the XFS
FAQ
recommends:
http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabled_with_storage_which_has_a_persistent_write_cache.3F
(BTW: I''ve edited those FAQ after talking to the XFS devs, so
I''m pretty
sure that info is correct).
> (as ext3 as
> well by default until last year: http://lkml.org/lkml/2008/5/16/390)
> so it means that you are not completely sure that metadata are
> written before the real data are modified, they can still be in the
> cache and if it vanish (due to power outage on not battery backed
> controller) then you''re on your own.
But exactly *where* should the data be lost?
1) XFS
2) LVM
3) XEN
4) RAID controller
5) Linux cache
6) Hard disks
7) ???

And how can I come to a secure solution? Should I use reiserfs again? 
Used that for years without a problem, but not with XEN though.
> But if you have a battery backed cache then it should be ok, that is
> to my understanding (I can be wrong).
But you must turn hard disk write cache off, which I have.
> It can be that your battery is dead or not working correctly (or your
> controller not using it ...), 
It''s working - the host itself has had no problem whatsoever.
> the other option is that some metadata
> haven''t leaved the os cache (read: the domU has written the
> information to the disk but either dom0 or the hypervisor is
> mainlining is caching data for backend device) and in this case
it''s
> normal to face problem.
Could be - but then there should be a workaroung.
> It would be very interesting to have more information from xfs team.
I posted there also, no solution until now.

I wonder why there''s no documentation about this problem. There are 
people using XEN in production machines - are they not scared by the 
actual behaviour? Even if I have UPSes and whatever, a crash can always 
occur. I have a customer who wants to use XEN to replace 10 small 
servers by a single one, but currently I''m reluctant to recommend XEN 
because I worry about the data. Imagine you have 10 servers not coming 
up after a problem - it could take hours to get every single server up 
and running again.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 4, 2009 at 11:35 AM, Michael Monnerie
<michael.monnerie@is.it-management.at> wrote:
> I wonder why there''s no documentation about this problem. There
are
> people using XEN in production machines - are they not scared by the
> actual behaviour? Even if I have UPSes and whatever, a crash can always
> occur. I have a customer who wants to use XEN to replace 10 small
> servers by a single one, but currently I''m reluctant to recommend
XEN
> because I worry about the data. Imagine you have 10 servers not coming
> up after a problem - it could take hours to get every single server up
> and running again.
it certainly warrants more investigation.  but i guess very few
production machines are using imagefiles on top of XFS.  the common
scenarios are either block devices or imagefiles on top of
OCFS/NFS/ext3

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/04/2009 08:28 PM, Javier Guerra wrote:
> On Wed, Mar 4, 2009 at 11:35 AM, Michael Monnerie
> <michael.monnerie@is.it-management.at>  wrote:
>
>    
>> I wonder why there''s no documentation about this problem.
There are
>> people using XEN in production machines - are they not scared by the
>> actual behaviour? Even if I have UPSes and whatever, a crash can always
>> occur. I have a customer who wants to use XEN to replace 10 small
>> servers by a single one, but currently I''m reluctant to
recommend XEN
>> because I worry about the data. Imagine you have 10 servers not coming
>> up after a problem - it could take hours to get every single server up
>> and running again.
>>      
>
> it certainly warrants more investigation.  but i guess very few
> production machines are using imagefiles on top of XFS.  the common
> scenarios are either block devices or imagefiles on top of
> OCFS/NFS/ext3
>
>    
But are we sure that it cames from XFS ?
Mike seems to describe that he has everything OK with the requirement of 
XFS. so as I said previously either XFS is doing nasty things when it''s
in XEN or it is LVM that is doing nasty things in XEN or LVM+XFS ...
BTW his first hypothesis was that XEN was lying about fsync, which I 
guess is not that simple but is at the end something approching.

In fact it could be quite interesting (but a bit time consuming) to have 
a normal linux installation with XFS and postgresql and stress it a 
little bit (otherwise I guess that you won''t face the problem) and 
meanwhile unplug it.
And do the same with XFS + LVM.
And one more test with XEN+XFS in domU+postgresql in domU.

I insists on the fact that the postgresql should be stressed otherwise 
data won''t be much modified and no nearly no chance of corruption.

  Matthieu.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Mar 5, 2009 at 6:57 AM, Matthieu Patou
<mat+Informatique.xen@matws.net> wrote:
> But are we sure that it cames from XFS ?
not exactly; in general, the XFS reliability approach is to do
everything right, but if anything else fails, the whole filesystem can
be destroyed with the excuse that "it''s your fault".

in this case it might be LVM or Xen, i have no idea.  remember that
even if XFS over LVM is a well-known case, LVM on a file on XFS isn''t.

but other Xen setups, using ext3, or (better) block devices do work
very reliably on big production setups, so even if Xen doesn''t meet
XFS requirements, it certainly is usable.
> In fact it could be quite interesting (but a bit time consuming) to have a
> normal linux installation with XFS and postgresql and stress it a little
bit
> (otherwise I guess that you won''t face the problem) and meanwhile
unplug it.
> And do the same with XFS + LVM.
> And one more test with XEN+XFS in domU+postgresql in domU.
sounds like a plan, but also repeat the Xen tests with imagefiles on
XFS, ext3, and block devices.


-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users