libvirt users - Oct 2018 - Re: snapshots with virsh in a pacemaker cluster

Hi,

i have a two node cluster with virtual guests as resources.
I'd like to snapshot the guests once in the night and thought i had a
procedure.
But i realize that things in a cluster are a bit more complicated than expected
:-))

I will shutdown the guests to have a clean snapshot.
I can shutdown the guests via pacemaker.
But then arises the first problem:

When i issue a "virsh snapshot-create-as" libvirt needs the domain
name as a parameter.
but libvirt does not know the domains any longer. When the guests are shutdown a
"virsh list --all"
on both nodes does not show any domain.

A look in the respective resource agent VirtualDomain explains why:
The domain is started with virsh create:

"# The 'create' command guarantees that the domain will be
# undefined on shutdown, ...

OK: Now i could of course define all domains with a virsh define.

But then i have immediately the next problem. Now i'd create the snapshots
with
"virsh snapshot-create-as" and starts the domains afterwards via
cluster.
But let's assume i issue that on node 1 and some guests are started
afterwards via pacemaker on node 2.
I can't predict on which node the guests are starting.

Then i don't get a snapshot, right ?

What to do ?

Bernd



-- 

Bernd Lentes 
Systemadministration 
Institut für Entwicklungsgenetik 
Gebäude 35.34 - Raum 208 
HelmholtzZentrum münchen 
[ mailto:bernd.lentes@helmholtz-muenchen.de | bernd.lentes@helmholtz-muenchen.de
]
phone: +49 89 3187 1241 
fax: +49 89 3187 2294 
[ http://www.helmholtz-muenchen.de/idg | http://www.helmholtz-muenchen.de/idg ] 

wer Fehler macht kann etwas lernen 
wer nichts macht kann auch nichts lernen
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler,
Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

Pacemaker always knows where its resources are running. Query it, stop the
domain, then use the queried location as the host to which to issue the
snapshot?

Cheers,

Peter

On Mon, 15 Oct 2018, 20:36 Lentes, Bernd, <
bernd.lentes@helmholtz-muenchen.de> wrote:
> Hi,
>
> i have a two node cluster with virtual guests as resources.
> I'd like to snapshot the guests once in the night and thought i had a
> procedure.
> But i realize that things in a cluster are a bit more complicated than
> expected :-))
>
> I will shutdown the guests to have a clean snapshot.
> I can shutdown the guests via pacemaker.
> But then arises the first problem:
>
> When i issue a "virsh snapshot-create-as" libvirt needs the
domain name as
> a parameter.
> but libvirt does not know the domains any longer. When the guests are
> shutdown a "virsh list --all"
> on both nodes does not show any domain.
>
> A look in the respective resource agent VirtualDomain explains why:
> The domain is started with virsh create:
>
> "# The 'create' command guarantees that the domain will be
> # undefined on shutdown, ...
>
> OK: Now i could of course define all domains with a virsh define.
>
> But then i have immediately the next problem. Now i'd create the
snapshots
> with
> "virsh snapshot-create-as" and starts the domains afterwards via
cluster.
> But let's assume i issue that on node 1 and some guests are started
> afterwards via pacemaker on node 2.
> I can't predict on which node the guests are starting.
>
> Then i don't get a snapshot, right ?
>
> What to do ?
>
> Bernd
>
>
>
> --
>
> Bernd Lentes
> Systemadministration
> Institut für Entwicklungsgenetik
> Gebäude 35.34 - Raum 208
> HelmholtzZentrum münchen
> [ mailto:bernd.lentes@helmholtz-muenchen.de |
> bernd.lentes@helmholtz-muenchen.de ]
> phone: +49 89 3187 1241
> fax: +49 89 3187 2294
> [ http://www.helmholtz-muenchen.de/idg |
> http://www.helmholtz-muenchen.de/idg ]
>
> wer Fehler macht kann etwas lernen
> wer nichts macht kann auch nichts lernen
>
>
> Helmholtz Zentrum Muenchen
> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
> Ingolstaedter Landstr. 1
> 85764 Neuherberg
> www.helmholtz-muenchen.de
> Aufsichtsratsvorsitzende: NN
> Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
> Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich
> Bassler, Dr. rer. nat. Alfons Enhsen
> Registergericht: Amtsgericht Muenchen HRB 6466
> USt-IdNr: DE 129521671
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users

I snapshot my guests live with qemu-guest-agent. It works pretty much 
flawlessly with qcow2, but even with raw I can almost always boot off of 
a snapshot. I combine this with zfs and hourly snapshots and replication 
and have a really usable backup and DR system that works with my guests, 
both Windows and Linux.

You should test it with qemu-guest-agent and then try to boot off of it 
and see if you run into issues. I've been doing this for 2+ years.


On 10/15/2018 3:29 PM, Lentes, Bernd wrote:
> Hi,
>
> i have a two node cluster with virtual guests as resources.
> I'd like to snapshot the guests once in the night and thought i had a
procedure.
> But i realize that things in a cluster are a bit more complicated than
expected :-))
>
> I will shutdown the guests to have a clean snapshot.
> I can shutdown the guests via pacemaker.
> But then arises the first problem:
>
> When i issue a "virsh snapshot-create-as" libvirt needs the
domain name as a parameter.
> but libvirt does not know the domains any longer. When the guests are
shutdown a "virsh list --all"
> on both nodes does not show any domain.
>
> A look in the respective resource agent VirtualDomain explains why:
> The domain is started with virsh create:
>
> "# The 'create' command guarantees that the domain will be
> # undefined on shutdown, ...
>
> OK: Now i could of course define all domains with a virsh define.
>
> But then i have immediately the next problem. Now i'd create the
snapshots with
> "virsh snapshot-create-as" and starts the domains afterwards via
cluster.
> But let's assume i issue that on node 1 and some guests are started
afterwards via pacemaker on node 2.
> I can't predict on which node the guests are starting.
>
> Then i don't get a snapshot, right ?
>
> What to do ?
>
> Bernd
>
>
>
-- 
Doug Hughes
Keystone NAP
Fairless Hills, PA
1.844.KEYBLOCK (439.2562)

----- Am 15. Okt 2018 um 21:47 schrieb Peter Crowther
peter.crowther@melandra.com:
> Pacemaker always knows where its resources are running. Query it, stop the
> domain, then use the queried location as the host to which to issue the
> snapshot?
But can i be sure that the resource starts on the node it was running before ?
IMHO no.
What is if i start the snapshot on node A but the resource starts afterwards on
node B ?
Then libvirt on node B does not know it should perform a snapshot.
> On Mon, 15 Oct 2018, 20:36 Lentes, Bernd, < [
> mailto:bernd.lentes@helmholtz-muenchen.de |
bernd.lentes@helmholtz-muenchen.de
> ] > wrote:
>> Hi,
>> i have a two node cluster with virtual guests as resources.
>> I'd like to snapshot the guests once in the night and thought i had
a procedure.
>> But i realize that things in a cluster are a bit more complicated than
expected
>> :-))
>> I will shutdown the guests to have a clean snapshot.
>> I can shutdown the guests via pacemaker.
>> But then arises the first problem:
>> When i issue a "virsh snapshot-create-as" libvirt needs the
domain name as a
>> parameter.
>> but libvirt does not know the domains any longer. When the guests are
shutdown a
>> "virsh list --all"
>> on both nodes does not show any domain.
>> A look in the respective resource agent VirtualDomain explains why:
>> The domain is started with virsh create:
>> "# The 'create' command guarantees that the domain will be
>> # undefined on shutdown, ...
>> OK: Now i could of course define all domains with a virsh define.
>> But then i have immediately the next problem. Now i'd create the
snapshots with
>> "virsh snapshot-create-as" and starts the domains afterwards
via cluster.
>> But let's assume i issue that on node 1 and some guests are started
afterwards
>> via pacemaker on node 2.
>> I can't predict on which node the guests are starting.
>> Then i don't get a snapshot, right ?
>> What to do ?
>> Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler,
Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671