Xen users - Mar 2007 - Mounting domU filesystem on Dom0. (xm mount)

> I have raised this thing many times in myriad different ways, and to me, 
> Xen still seem to lack the ability to access the domUs in a proper 
> manner so as to allow mass management of the domUs, and one of the 
> important things here is backup. If you have to manage 1000 domUs across 
> 50-60 machines, then it would be essential to be able to take backup of 
> the entire system.
Why not use LVM snapshots? I use snapshot feature to backup domU''s from
dom0 and I am very satisfied.
> This actually can be achieved even on normal Xen system, by running an 
> nfs daemon and exporting the ''/'' to the dom0, which can
then be mounted
> and backedup, like any live system.
And why not use it?

You can also use shared filesystems like GFS or OCFS2.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I have raised this thing many times in myriad different ways, and to me, Xen
still seem to lack the ability to access the domUs in a proper manner so as to
allow mass management of the domUs, and one of the important things here is
backup. If you have to manage 1000 domUs across 50-60 machines, then it would be
essential to be able to take backup of the entire system. This actually can be
achieved even on normal Xen system, by running an nfs daemon and exporting the
''/'' to the dom0, which can then be mounted and backedup, like
any live system.

 Now, my question is: is there some chance that this feature can be built into
the domU kernel itself.

 The command should be something like this:

 xm mount <domuid> /mnt/domU

 What''s the opinion of the people here in the list about a mount
command? This will solve quite a lot of problems that we are facing now
vis-a-vis xen, and I think even UML has some mechnism whereby you can mount
certain directories in the domU to the dom0.



 Thanks.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 14, 2007 at 01:13:15PM +0200, Siim Vahtre
wrote:
> >I have raised this thing many times in myriad different ways, and to
me,
> >Xen still seem to lack the ability to access the domUs in a proper 
> >manner so as to allow mass management of the domUs, and one of the 
> >important things here is backup. If you have to manage 1000 domUs
across
> >50-60 machines, then it would be essential to be able to take backup of
> >the entire system.
> 
> Why not use LVM snapshots? I use snapshot feature to backup domU''s
from
> dom0 and I am very satisfied.
 LVM Snapshots are terrible terrible idea. You are basically mounting an
inconsistent file system that has not yet been synced, and I am sure you have
never tried to restore the backups you have so conscientiously taken. Believe me
one day, if you try to restore the backup, it will completely ruin your system.
The LVM snapshot is the state of a filesystem that''s running, and it is
equivalent to powering off the system. And at the least, you should run e2fsck
on the LVM before you actually tar it up.
> 
> >This actually can be achieved even on normal Xen system, by running an 
> >nfs daemon and exporting the ''/'' to the dom0, which
can then be mounted
> >and backedup, like any live system.
> 
> And why not use it?
 The point is that I can''t enable nfs on 1000 domUs. That''s
another problem with xen. On openvz, I can do for in in `vzlist` ; do vzctl exec
/etc/init.d/nfs restart done. How do I do it on xen? But even if this were
possible,  nfs is too heavy, and it is insanity to have 1000 nfses running on
your system. That is a most callous waste of memory, multiplied a 1000 times.

[quote] 
> 
> You can also use shared filesystems like GFS or OCFS2.
[/quote] 

 I will try that out. It _appears_ to be a possible solution, if only due to the
fact that, it is the only one I haven''t yet tried and found major
problems with.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 14, 2007 at 05:06:37PM +0530, Ligesh wrote:
> > 
> > >This actually can be achieved even on normal Xen system, by
running an
> > >nfs daemon and exporting the ''/'' to the dom0,
which can then be mounted
> > >and backedup, like any live system.
> > 
> > And why not use it?
> 
 The real issue with nfs is not even the load, but its dependency on the
network. You have to first know the ipaddress of the dom0, and make sure that
nfs will export it to dom0 alone and nowhere else, and again, this becomes a
maintanance issue when move domU from one machine to another, where you have to
reconfigure everything. The problem I am trying to solve is mass management, and
it cannot be solved by manually fixing all the domUs. That''s like some
kind of irony or something.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ligesh wrote:
>  I have raised this thing many times in myriad different ways, and to me,
Xen still seem to lack the ability to access the domUs in a proper manner so as
to allow mass management of the domUs, and one of the important things here is
backup. If you have to manage 1000 domUs across 50-60 machines, then it would be
essential to be able to take backup of the entire system. This actually can be
achieved even on normal Xen system, by running an nfs daemon and exporting the
''/'' to the dom0, which can then be mounted and backedup, like
any live system.
>
>  Now, my question is: is there some chance that this feature can be built
into the domU kernel itself.
>
>  The command should be something like this:
>
>  xm mount <domuid> /mnt/domU
>
>  What''s the opinion of the people here in the list about a mount
command? This will solve quite a lot of problems that we are facing now
vis-a-vis xen, and I think even UML has some mechnism whereby you can mount
certain directories in the domU to the dom0.
>   
What should this command do? Just mount the filesystem on the dom0? I 
don''t really get what you mean...

If you want to backup the system have you tried evms[1] and its 
snapshotting?

[1] http://evms.sourceforge.net/

--
Álvaro López

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 14, 2007 at 01:03:17PM +0100, ?lvaro L?pez Garc?a
wrote:
> Ligesh wrote:
> > What''s the opinion of the people here in the list about a
mount command?
> > This will solve quite a lot of problems that we are facing now
vis-a-vis
> > xen, and I think even UML has some mechnism whereby you can mount
certain
> > directories in the domU to the dom0.
> >  
> What should this command do? Just mount the filesystem on the dom0? I 
> don''t really get what you mean...
 Mount it through the domU kernel, so that you can do real time operations on
it. The code has to be implemented in both domU and dom0 Kernels. The
functionality should be equivalent to nfs mount, but it should work without
network, that''s all.


[quote] 
> 
> If you want to backup the system have you tried evms[1] and its 
> snapshotting?
> 
> [1] http://evms.sourceforge.net/
[/quote] 
 
 LVMs are not the solution. You need the domU kernel''s help to do
anything without completely ruining the filesystem. Currently, I think the best
way would be customize a simple nfsd for this purpose.

 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> LVMs are not the solution. You need the domU kernel''s help to do 
> anything without completely ruining the filesystem. Currently, I think 
> the best way would be customize a simple nfsd for this purpose.
Remember that if you backup directly from filesystem (or from NFS export 
or whatever), the filesystem contents can change even while you''re
backing
up! That doesn''t happen with LVM.

To get "really" consistent backup, the backup software must cooperate
with
the application, which can be quite complicated. But if the application is 
written well, it should be able to handle poweroffs, at least as far as 
not completely destroying your data.

LVM is far from perfect, but I wouldn''t say it is "terrible
terrible" for
your task.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 14, 2007 at 02:23:40PM +0200, Siim Vahtre
wrote:
> >LVMs are not the solution. You need the domU kernel''s help to
do
> >anything without completely ruining the filesystem. Currently, I think 
> >the best way would be customize a simple nfsd for this purpose.
> 
> Remember that if you backup directly from filesystem (or from NFS export 
> or whatever), the filesystem contents can change even while you''re
backing
> up! That doesn''t happen with LVM.
> 
> To get "really" consistent backup, the backup software must
cooperate with
> the application, which can be quite complicated. But if the application is 
> written well, it should be able to handle poweroffs, at least as far as 
> not completely destroying your data.
> 
> LVM is far from perfect, but I wouldn''t say it is "terrible
terrible" for
> your task.
 With LVM, you get filesystem (kernel) level corruption. With nfs, you get
application level corruption. The first is obviously graver. There''s
real world proof too. Openvz uses the latter kind of backup, and we have a lot
of customers who are using it for centralized backup of their entire vps farm,
and I havne''t had any real problem with. Unlike the LVM based one which
actually made the entire system unusable twice.

 Thanks.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ligesh wrote:
> On Wed, Mar 14, 2007 at 02:23:40PM +0200, Siim Vahtre wrote:
>   
>> To get "really" consistent backup, the backup software must
cooperate with
>> the application, which can be quite complicated. But if the application
is
>> written well, it should be able to handle poweroffs, at least as far as
>> not completely destroying your data.
>>
>> LVM is far from perfect, but I wouldn''t say it is
"terrible terrible" for
>> your task.
>>     
>
>  With LVM, you get filesystem (kernel) level corruption. With nfs, you get
application level corruption. The first is obviously graver.
I have to disagree.

When you restore something backed up from LVM snapshot, it is as if the
system was powered off at the time of the snapshot. As Siim mentioned,
some applications (e.g : Oracle or MySQL with correct settings) can
handle that nicely and provides a consistent, recovered, usable data.

Nfs backups, on the other hand, has no way guaranting consistent data.
An extreme example, think of a 100-GB MySQL Innodb datafile. If you copy
that file (via nfs or whatever) when MySQL server is running, the first
part and last part will most likely come from different points in time.
Good luck recovering that.

NFS backups SHOULD work out nicely for files that remains unchanged
during backup period (web pages, binaries, etc.).

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Mar 14, 2007 at 07:50:57PM +0700, Fajar A. Nugraha
wrote:
> >
> >  With LVM, you get filesystem (kernel) level corruption. With nfs, you
get application level corruption. The first is obviously graver.
> I have to disagree.
> 
> When you restore something backed up from LVM snapshot, it is as if the
> system was powered off at the time of the snapshot. As Siim mentioned,
> some applications (e.g : Oracle or MySQL with correct settings) can
> handle that nicely and provides a consistent, recovered, usable data.
The point is that the corruption is at the kernel level itself, and if at all
any recovery can be made, it has to be done at the file system level (ext3, or
reiserfs or whatever recovery tools). Applications don''t actually come
into the picture. We have already discussed this earlier. The ideal would be a
''sync-and-save'' command which would actually be equivalent to
a ''kill -9''ing the application, which is something all
applications can recover from. But LVM data corruption happens at the file
system level, and has to be recovered first using e2fsck, before mysql can even
have a go at it fixing the rest of the problems.

 So LVMs have two levels of corruption: File system, and ''kill
-9''ing the application. Mysql can recover the latter. An e2fsck on the
LVM snapshot before tarring up might actually reduce the chances of corruption,
but I am not sure.

[quote] 
> 
> Nfs backups, on the other hand, has no way guaranting consistent data.
> An extreme example, think of a 100-GB MySQL Innodb datafile. If you copy
> that file (via nfs or whatever) when MySQL server is running, the first
> part and last part will most likely come from different points in time.
> Good luck recovering that.
[/quote] 

 This might appear grave, but in practice it is not. Of course, the scenario I
am talking about is 10s of thousands of small domUs running small workloads. As
I have pointed out in the beginning, the purpose is mass management of LARGE
number of light workloads. If you have heavy workload, then individual
management of each domU becomes practical, and then we are talking about
different scenarios altogether.

 Further, in practice, this particular method has been employed for 5 years by
virtuozzo service providers to deliver managed services to their clients. And
again, please note, we are talking about large number of small workloads. If you
have small number of large workloads, then you don''t need the system at
all. You can run a full fledged backup system on each domU.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ligesh wrote:
> But LVM data corruption happens at the file system level, and has to be
recovered first using e2fsck, before mysql can even have a go at it fixing the
rest of the problems.
>
>   
Aah, I think this is the source of our different point of view. I was
refering to journaling filesystems (ext3, reiserfs, etc) where the
kernel will automatically replay last journal, so (on most cases) no
MANUAL fsck is necessary.

Since you''re talking about "10s of thousands of small domUs
running
small workloads" (which implies small, rarely changed files) then NFS
backups should be sufficient.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Mar 15, 2007 at 09:03:19AM +0700, Fajar A. Nugraha
wrote:
> Ligesh wrote:
> > But LVM data corruption happens at the file system level, and has to
be recovered first using e2fsck, before mysql can even have a go at it fixing
the rest of the problems.
> >
> >   
> Aah, I think this is the source of our different point of view. I was
> refering to journaling filesystems (ext3, reiserfs, etc) where the
> kernel will automatically replay last journal, so (on most cases) no
> MANUAL fsck is necessary.
  Are you backing up the entire LVM image as a single file? You are mounting and
tarring the contents right? If you are backing the LVM image as a single entity,
then yes, you can even get perfect backup, if you simultaneously do a xm save,
and keep the ram too. In fact, that''s a nice way to do perfect backup.
Take an lvm snapshot and also save the ram image, and you can restore the state
perfectly.

 But in normal circumstances, what we do is, take an LVM snapshot, then mount
it, and then tar/rsync the contents. That can be hazardous. Anyway, now that I
have given it some consideration, it appears, you can indeed run e2fsck on the
LVM snapshot, and get a consistent file system, albeit without the application
buffers in the ram.

 So I think 

 lvm2 snapshot
 e2fsck -f -y /dev/lvm-snapshot
 mount -o ro /dev/lvm-snapshot /mnt/backup
 rsync /mnt/backup

 Is indeed a solution for a reliable backing up--you will loose the application
buffers, but that''s actually alright for small vpses.

 Any comments?

Thanks.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, 15 Mar 2007, Ligesh wrote:
> In fact, that''s a nice way to do perfect backup. Take an lvm
snapshot
> and also save the ram image, and you can restore the state perfectly.
Don''t forget that you need to get memory and LVM snapshots in the
_exact_
time. But if you "xm pause" just before snapshot+save, it might
actually
work (I really don''t know).

However, you won''t have "perfect" state. You will, for
example, probably
have a nice mess with your network connections if you restore the ram 
image after some connections have already timed out. It _might_ be a 
problem.
> lvm2 snapshot
> e2fsck -f -y /dev/lvm-snapshot
> mount -o ro /dev/lvm-snapshot /mnt/backup
> rsync /mnt/backup
You don''t need forced fsck. Journal reply should be enough. But
occational
full fsck might be useful nevertheless.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Mar 15, 2007 at 11:49:13AM +0200, Siim Vahtre
wrote:
> On Thu, 15 Mar 2007, Ligesh wrote:
> 
> >In fact, that''s a nice way to do perfect backup. Take an lvm
snapshot
> >and also save the ram image, and you can restore the state perfectly.
> 
> Don''t forget that you need to get memory and LVM snapshots in the
_exact_
> time. But if you "xm pause" just before snapshot+save, it might
actually
> work (I really don''t know).
 Save will automatically pause it. 

 xm save <dom> filename
 lvm snapshot
 xm restore filename
 backup filename + snapshot

 
 [quote] 
> 
> However, you won''t have "perfect" state. You will, for
example, probably
> have a nice mess with your network connections if you restore the ram 
> image after some connections have already timed out. It _might_ be a 
> problem.
> 
[/quote] 

 I don''t think that''s an issue, since it is used in live
migration. It would be equivalent to a very delayed live migration. The kernel
as such is a non-real time system, so as far as the kernel is concerned, it
would equivalent to other end forcibly closing the connection. Everything will
always be in a wait state in the kernel--except for the single active thread, so
when the kernel takes the networking part from the wait queue and runs it, it
will just find that the other end has closed in the period it was in wait, and I
don''t think this is too different from what happens in a normal system.



> >lvm2 snapshot
> >e2fsck -f -y /dev/lvm-snapshot
> >mount -o ro /dev/lvm-snapshot /mnt/backup
> >rsync /mnt/backup
> 
> You don''t need forced fsck. Journal reply should be enough. But
occational
> full fsck might be useful nevertheless.
 Will a mount automatically recover the journal?

 Thanks.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ligesh wrote:
> But in normal circumstances, what we do is, take an LVM snapshot,
> then mount it, and then tar/rsync the contents. That can be
> hazardous. Anyway, now that I have given it some consideration, it
> appears, you can indeed run e2fsck on the LVM snapshot, and get a
> consistent file system, albeit without the application buffers in the
> ram.
Yes, it can be hazardous. Most operating systems page out information, and 
only write it to disk when they get around to it. This means that you may 
have mixed in blocks of data in files that haven''t been paged back out
to
disk yet, especially files that get lots of traffic such as databases.
It''s
even worse for databases, because they play various software tricks to 
assure that certain operations go to completion lest the database be 
corrupted, and those techniques do *NOT* typically involve getting 
everything written to disk. So you may backup the db files in the midst of 
what is supposed to be an atomic operation.

Even operations that write to two files in sequence become risky, such as 
adding user accounts and getting the account in /etc/passwd and not in 
/etc/shadow, or not vice versa. Such risks are small in typical operations, 
but they''re real. 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users