Xen users - Mar 2008 - Best practice to backup dom0 and all domU''s

On Tue, 25 Mar 2008, Darren wrote:
> What is the preffered or suggested way to backup a dom0 and all
domU''s
> while system is up and running ? Thanks
Run everything from iSCSI and backup your storage server.


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> On Tue, 25 Mar 2008, Darren wrote:
>
>   
>> What is the preffered or suggested way to backup a dom0 and all
domU''s
>> while system is up and running ? Thanks
>>     
>
> Run everything from iSCSI and backup your storage server.
>
>
> Stefan
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>   
What about a shared hosting environment where iSCSI isn''t available or 
very expensive? i.e one has a dedicated server with a 1 / 2 local
HDD''s,
and FTP backup on another server?

-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stugg


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 25 Mar 2008, Rudi Ahlers wrote:
> What about a shared hosting environment where iSCSI isn''t
available or
> very expensive? i.e one has a dedicated server with a 1 / 2 local
HDD''s,
> and FTP backup on another server?
OpenSolaris is free. There you go. You even get ZFS doing snapshots of
live iscsi mounts.


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> On Tue, 25 Mar 2008, Rudi Ahlers wrote:
>
>   
>> What about a shared hosting environment where iSCSI isn''t
available or
>> very expensive? i.e one has a dedicated server with a 1 / 2 local
HDD''s,
>> and FTP backup on another server?
>>     
>
> OpenSolaris is free. There you go. You even get ZFS doing snapshots of
> live iscsi mounts.
>
>
> Stefan
>
>
>   
What has opensolaris got to do with it?



-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stugg


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 25 Mar 2008, Rudi Ahlers wrote:
> > OpenSolaris is free. There you go. You even get ZFS doing snapshots of
> > live iscsi mounts.
> >
> What has opensolaris got to do with it?
It is a free iSCSI server?


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> On Tue, 25 Mar 2008, Rudi Ahlers wrote:
>
>   
>>> OpenSolaris is free. There you go. You even get ZFS doing snapshots
of
>>> live iscsi mounts.
>>>
>>>       
>> What has opensolaris got to do with it?
>>     
>
> It is a free iSCSI server?
>
>
> Stefan
>
>
>   
Stefan, you may have a spare server in the office for this, but in a 
hosting environment, you have to pay for every server you want, and I 
have to be honest with you, it''s cheaper to buy 200GB FTP space, than 
another server JUST for backups. And I don''t see why any host who 
doesn''t offer / use iSCSI should suddenly start offering it just cause 
OpenSolaris is free. Cause what you say is to have an iSCSI server with 
the VM''s on, and then backup that server, so that means you''d
need 3
physical servers

OpenSolaris isn''t the answer to everything.

-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stugg


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Darren wrote:
> What is the preffered or suggested way to backup a dom0 and all
domU''s
> while system is up and running ? Thanks
>
> Darren
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
I use lvm filesystem for DomU''s and backup via lvm snapshots.


In detail:
I have Dom0 with md (software mirror) and lvm on top.
Nightly I make lvm snapshot for each DomU (make sure that it is large
enough not to overfill while doing backup).
Then I export those lvm snapshots to dedicated backup DomU which boots
on and after backup job shuts down itself.
Then remove snapshots.

I use this in production successfully for a couple of months.

andris


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 25 Mar 2008, Rudi Ahlers wrote:
> Stefan, you may have a spare server in the office for this, but in a
> hosting environment, you have to pay for every server you want, and I
> have to be honest with you, it''s cheaper to buy 200GB FTP space,
than
> another server JUST for backups. And I don''t see why any host who
> doesn''t offer / use iSCSI should suddenly start offering it just
cause
> OpenSolaris is free. Cause what you say is to have an iSCSI server with
> the VM''s on, and then backup that server, so that means
you''d need 3
> physical servers
>
> OpenSolaris isn''t the answer to everything.
I''m a contracter for a company that is in the same situation. And
having a
storage infrastructure is something you get customers with, usually not
loose. But all depends on how big you are. I think 24 Xen servers with one
fileserver is not a bad ratio. 1:1 is...

Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> On Tue, 25 Mar 2008, Rudi Ahlers wrote:
>
>   
>> Stefan, you may have a spare server in the office for this, but in a
>> hosting environment, you have to pay for every server you want, and I
>> have to be honest with you, it''s cheaper to buy 200GB FTP
space, than
>> another server JUST for backups. And I don''t see why any host
who
>> doesn''t offer / use iSCSI should suddenly start offering it
just cause
>> OpenSolaris is free. Cause what you say is to have an iSCSI server with
>> the VM''s on, and then backup that server, so that means
you''d need 3
>> physical servers
>>
>> OpenSolaris isn''t the answer to everything.
>>     
>
> I''m a contracter for a company that is in the same situation. And
having a
> storage infrastructure is something you get customers with, usually not
> loose. But all depends on how big you are. I think 24 Xen servers with one
> fileserver is not a bad ratio. 1:1 is...
>
> Stefan
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>   
Yes, I understand what you''re saying.

But, let''s say you offer the following VPS options:
1. 512MB RAM / 10GB space
2. 1024MB RAM / 20GB space.

Then, on a dual Xeon (4 cores) with 16GB RAM, you could have 15 / 30 
VPS'' per physical server. So, you suggest have a iSCSI server, and back
that up, which means you need 3 physical servers to host the same 20 
VPS'' - which doesn''t make sense. There should be a better way
to backup
the VPS'' directly from the main server, to the FTP server, which is 1/4
the price of a physical server in most cases

-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stugg


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

What is the preffered or suggested way to backup a dom0 and all domU''s 
while system is up and running ? Thanks

Darren

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello Andris

On Tue, 2008-03-25 at 14:20 +0200, Andris wrote:
> I have Dom0 with md (software mirror) and lvm on top.
> Nightly I make lvm snapshot for each DomU (make sure that it is large
> enough not to overfill while doing backup).
Did you get the snapshot with domU running?

> Then I export those lvm snapshots to dedicated backup DomU which boots
> on and after backup job shuts down itself.
> Then remove snapshots.
Sounds like graceful! Can you show your script for us, please?

Thanks
> 
-- 
Tiago Cruz
http://everlinux.com
Linux User #282636



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Rudi,

On Tue, Mar 25, 2008 at 02:33:04PM +0200, Rudi Ahlers
wrote:
> on a dual Xeon (4 cores) with 16GB RAM, you could have 15 / 30
> VPS'' per physical server. So, you suggest have a iSCSI server, and
> back that up, which means you need 3 physical servers to host the
> same 20 VPS'' - which doesn''t make sense.
The dom0 could be the iSCSI server, so in fact all you need is
somewhere else to back up to, which you need anyway.

This topic comes up all the time on this list and a lot of people
seem to be of the frame of mind that backup should be something that
Xen should provide.  Well, it is more complicated than that so I
think it is right that Xen does not attempt to touch it.

I advise you to treat it the same as backing up however many
discrete machines.  The way you do that depends on a lot of factors.
The iSCSI method is one of them.

The other issue is, just snapshotting a filesystem and taking a dump
of it is not enough to get a reliable backup.  You need to consider
the individual applications that are being run, and have ways to
tell them to stop what they are doing while you take the snapshot.
Think database servers for example.  If you can''t or don''t
want to
deal with that, fair enough, you probably do still get a best effort
backup that saves you the majority of the time, but it''s something
to be aware of.
> There should be a better way to backup the VPS'' directly from the
> main server, to the FTP server, which is 1/4 the price of a
> physical server in most cases
And send your customer''s data in clear over an untrusted network to
be stored on someone else''s hardware?

Cheers,
Andy


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tiago Cruz wrote:
> Hello Andris
>
> On Tue, 2008-03-25 at 14:20 +0200, Andris wrote:
>
>   
>> I have Dom0 with md (software mirror) and lvm on top.
>> Nightly I make lvm snapshot for each DomU (make sure that it is large
>> enough not to overfill while doing backup).
>>     
>
> Did you get the snapshot with domU running?
>
>
>   
>> Then I export those lvm snapshots to dedicated backup DomU which boots
>> on and after backup job shuts down itself.
>> Then remove snapshots.
>>     
>
> Sounds like graceful! Can you show your script for us, please?
>
> Thanks
>   
1. Of course, thats for snapshots are supposed for.
2. It is shame for me show my scripting (hard coded paths, etc. not
useful for wide usage, but just for my own configs ), because I am bad
shell script writer, but idea is:
Config file where I put which volumes to backup and desired snapshot
prefix name for each
Then I put manually those snapshot names into backup DomU xen config
file as disk drives.
So there are 3 scripts, one for creating, one for removing lvm snapshots
and one for launching backup server (complicated bacula config then
within backup server itself). And two cron jobs for launching backup
nightly, and removing snapshots then.


This is for creating snapshots, of course it is far from perfect shell
scripting, but idea can be clear:
#################################################################
#!/bin/sh

#script dir
SDIR=/root/scripts/lvm_backup

#test if backup server is already started, backup server DomU name
should be "backup"
TESTBAK=`/usr/sbin/xm list | grep backup | cut -b 1-8`

if [ $TESTBAK ]
then

exit 0

else
        # this is status log to avoid snapshot removing from cron while
script is just started, but not finished
        echo > $SDIR/status.log 1


        VOL=VOL
        NAME=NAME

        backup=_snap

## loop for doing snapshots, for me is enough 22
        for ((  i = 10 ;  i <= 22;  i++  ))
        do

        CUR_VOL=$VOL$i
        CUR_NAME=$NAME$i

        a=`cat  $SDIR/config.volumes | grep $CUR_VOL | cut -b 7-200`
        b=`cat  $SDIR/config.volumes | grep $CUR_NAME | cut -b 8-200`

        if [ $a ]
        then
        ##here we make snapshots
        /usr/sbin/lvcreate -L2G -s -n $b$backup $a
        else
        break
        fi
        sleep 5
        echo > $SDIR/status.log 0
        done


fi
#####################################################################


As considering backup server itself, as I wrote, it is complicated
bacula setup with TLS encryption for remote clients and storage, and no
encryption for a local storage (big sata disk on the same machine(Dom0)
exported as a physical device to backup DomU). This approach allows to
use the same backup server (cloned) for all my clients backups - once
set up, and use for all xen servers.

andris

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Andy Smith wrote:
>
> The other issue is, just snapshotting a filesystem and taking a dump
> of it is not enough to get a reliable backup.  You need to consider
> the individual applications that are being run, and have ways to
> tell them to stop what they are doing while you take the snapshot.
> Think database servers for example.  If you can''t or
don''t want to
> deal with that, fair enough, you probably do still get a best effort
> backup that saves you the majority of the time, but it''s something
> to be aware of.
>
>   
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
Database servers could be backed up via making snapshot.
There can be issues on hard loaded mysql servers for example (see this:
http://mike.kruckenberg.com/archives/2006/05/mysql_backups_u.html), but
for 99% of folks should be enough.
In such high volume enterprise production environments no doubts iSCSI
and similar should be used.


andris

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Andris,

Many thanks for share your solution! :-)

On Tue, 2008-03-25 at 17:29 +0200, Andris wrote:
>         ##here we make snapshots
>         /usr/sbin/lvcreate -L2G -s -n $b$backup $a
2 GB its enough for any machine? ''Cause I have vm''s with 20 GB
until 100
GB...

Thanks!

-- 
Tiago Cruz
http://everlinux.com
Linux User #282636



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tiago Cruz wrote:
> Andris,
> 
> Many thanks for share your solution! :-)
> 
> On Tue, 2008-03-25 at 17:29 +0200, Andris wrote:
>>         ##here we make snapshots
>>         /usr/sbin/lvcreate -L2G -s -n $b$backup $a
> 
> 2 GB its enough for any machine? ''Cause I have vm''s with
20 GB until 100
> GB...
I think you''ll find that for snapshots the size specified is for the 
changes to the filesystem which are expected to occur during the 
lifetime of the snapshot.

So if you anticipate that, _while_ the snapshot is in use, its likely to 
accumulate 20G worth of *change*, the -L2G won''t be enough.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users