libvirt users - Jan 2018 - VM migration upon shutdown in centos 7

Hi,

I have CentOS 7,  two node system which allows live VM migration between
them. Live migration triggered from virsh is happily happening. I am using
GlusterFS for replicating VM disk files.

Now I want to automatically do the live migration at the time of
reboot/shutdown/halt of the host node and for this I have written a systemd
service unit [vPreShutdownHook.service] and placed the live migration
command in a migrate script which is invoked from this service unit. The
migrate script is invoked but migration does not happen.
If someone has some idea please help me to migrate VM upon shutdown.


######## vPreShutdownHook.service ###############

[Unit]
Description=vPreShutdownHook
Requires=network.target
Requires=libvirtd.service
Requires=dbus.service
Requires=glusterd.service
Requires=glusterfsd.service
DefaultDependencies=no
Before=shutdown.target reboot.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true
ExecStop=/root/vm_migrate.sh

[Install]
WantedBy=multi-user.target


########## Command to migrate ############
/usr/bin/virsh migrate --verbose --p2p --tunneled --live  --compressed
--comp-methods "mt"  --comp-mt-level 5 --comp-mt-threads 5
--comp-mt-dthreads 5   MY_VM qemu+ssh://root@$node2/system


Thanks,

----- Original Message -----
> From: "Raman Gupta" <ramangupta16@gmail.com>
> To: libvirt-users@redhat.com
> Sent: Friday, January 5, 2018 7:00:49 PM
> Subject: [libvirt-users] VM migration upon shutdown in centos 7
> 
> Hi,
> 
> I have CentOS 7, two node system which allows live VM migration between
them.
> Live migration triggered from virsh is happily happening. I am using
> GlusterFS for replicating VM disk files.
> 
> Now I want to automatically do the live migration at the time of
> reboot/shutdown/halt of the host node and for this I have written a systemd
> service unit [vPreShutdownHook.service] and placed the live migration
> command in a migrate script which is invoked from this service unit. The
> migrate script is invoked but migration does not happen.
> If someone has some idea please help me to migrate VM upon shutdown.
> 
> 
> ######## vPreShutdownHook.service ###############
> 
> [Unit]
> Description=vPreShutdownHook
> Requires=network.target
> Requires=libvirtd.service
> Requires=dbus.service
> Requires=glusterd.service
> Requires=glusterfsd.service
> DefaultDependencies=no
> Before=shutdown.target reboot.target
> 
> [Service]
> Type=oneshot
> RemainAfterExit=true
> ExecStart=/bin/true
> ExecStop=/root/vm_migrate.sh
> 
> [Install]
> WantedBy=multi-user.target
> 
> 
> ########## Command to migrate ############
> /usr/bin/virsh migrate --verbose --p2p --tunneled --live --compressed
Please modify "--tunneled" to "tunnelled" and try again.
> --comp-methods "mt" --comp-mt-level 5 --comp-mt-threads 5
--comp-mt-dthreads
> 5 MY_VM qemu+ssh://root@$node2/system
> 
> 
> Thanks,
> 
> 
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users

On 01/05/2018 12:00 PM, Raman Gupta wrote:
> Hi,
> 
> I have CentOS 7,  two node system which allows live VM migration between
> them. Live migration triggered from virsh is happily happening. I am using
> GlusterFS for replicating VM disk files.
> 
> Now I want to automatically do the live migration at the time of
> reboot/shutdown/halt of the host node and for this I have written a systemd
> service unit [vPreShutdownHook.service] and placed the live migration
> command in a migrate script which is invoked from this service unit. The
> migrate script is invoked but migration does not happen.
> If someone has some idea please help me to migrate VM upon shutdown.
> 
> 
> ######## vPreShutdownHook.service ###############
> 
> [Unit]
> Description=vPreShutdownHook
> Requires=network.target
> Requires=libvirtd.service
> Requires=dbus.service
> Requires=glusterd.service
> Requires=glusterfsd.service
> DefaultDependencies=no
> Before=shutdown.target reboot.target
> 
> [Service]
> Type=oneshot
> RemainAfterExit=true
> ExecStart=/bin/true
> ExecStop=/root/vm_migrate.sh
> 
> [Install]
> WantedBy=multi-user.target
> 
> 
> ########## Command to migrate ############
> /usr/bin/virsh migrate --verbose --p2p --tunneled --live  --compressed
> --comp-methods "mt"  --comp-mt-level 5 --comp-mt-threads 5
> --comp-mt-dthreads 5   MY_VM qemu+ssh://root@$node2/system
Does this command alone succeed?
BTW: unless really needed --live will only make the migration take longer.

I don't know enough about systemd but maybe it's not waiting for virsh
to finish? Can you try to get any logs to see what is going on actually?

Michal

[Please keed the list CCed]

On 01/10/2018 04:43 PM, Raman Gupta wrote:
>> Does this command alone succeed?
> Yes. I have used this command to migrate VMs successfully, without even
> knowing that spelling has changed.
> 
> 
>> I don't know enough about systemd but maybe it's not waiting
for virsh to
> finish?
> Yes I also think virsh or libvirtd or related service does not wait
> before /root/vm_migrate.sh
> is called and hence live migration fails.
> If I replace Live Migration with a simple ping to peer node in the
> vPreShutdownHook.service,  then ping goes thru successfully thus indicating
> network service was UP at that time.
> 
>> Can you try to get any logs to see what is going on actually?
> Jan  8 15:30:58 desktop4 systemd: Stopping Session c1 of user gdm.
> Jan  8 15:30:58 desktop4 gdm: Freeing conversation
'gdm-launch-environment'
> with active job
> Jan  8 15:30:58 desktop4 systemd: Stopped target Sound Card.
> Jan  8 15:30:58 desktop4 systemd: Stopping Sound Card.
> Jan  8 15:30:58 desktop4 systemd: Stopping LVM2 PV scan on device 8:3...
> Jan  8 15:30:58 desktop4 systemd: Removed slice system-getty.slice.
> Jan  8 15:30:58 desktop4 systemd: Stopping system-getty.slice.
> Jan  8 15:30:58 desktop4 systemd: Stopping Authorization Manager...
> Jan  8 15:30:58 desktop4 systemd: Stopping Virtual Machine and Container
> Registration Service...
> Jan  8 15:30:58 desktop4 systemd: Stopping GlusterFS brick processes
> (stopping only)...
> Jan  8 15:30:58 desktop4 systemd: Removed slice
> system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
> Jan  8 15:30:58 desktop4 systemd: Stopping
> system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
> Jan  8 15:30:58 desktop4 systemd: Stopping Availability of block devices...
> Jan  8 15:30:58 desktop4 systemd: Stopping Virtual Machine
> qemu-1-GsmController.
> Jan  8 15:30:58 desktop4 alsactl[837]: alsactl daemon stopped
> Jan  8 15:30:58 desktop4 systemd: Stopping Session 1 of user root.
> Jan  8 15:30:58 desktop4 systemd: Stopping Manage Sound Card State (restore
> and store)...
I don't see your service being called. Anyway, look at
libvirt-guests.service file. Looks like the following lines cause
systemd to wait for a command to finish:

[Service]
EnvironmentFile=-/etc/sysconfig/libvirt-guests
# Hack just call traditional service until we factor
# out the code
ExecStart=@libexecdir@/libvirt-guests.sh start
ExecStop=@libexecdir@/libvirt-guests.sh stop
Type=oneshot
RemainAfterExit=yes
StandardOutput=journal+console
TimeoutStopSec=0
> 
> 
> 
> On Wed, Jan 10, 2018 at 8:46 PM, Michal Privoznik
<mprivozn@redhat.com>
> wrote:
> 
>> On 01/05/2018 12:00 PM, Raman Gupta wrote:
>>> Hi,
>>>
>>> I have CentOS 7,  two node system which allows live VM migration
between
>>> them. Live migration triggered from virsh is happily happening. I
am
>> using
>>> GlusterFS for replicating VM disk files.
>>>
>>> Now I want to automatically do the live migration at the time of
>>> reboot/shutdown/halt of the host node and for this I have written a
>> systemd
>>> service unit [vPreShutdownHook.service] and placed the live
migration
>>> command in a migrate script which is invoked from this service
unit. The
>>> migrate script is invoked but migration does not happen.
>>> If someone has some idea please help me to migrate VM upon
shutdown.
>>>
>>>
>>> ######## vPreShutdownHook.service ###############
>>>
>>> [Unit]
>>> Description=vPreShutdownHook
>>> Requires=network.target
>>> Requires=libvirtd.service
>>> Requires=dbus.service
>>> Requires=glusterd.service
>>> Requires=glusterfsd.service
>>> DefaultDependencies=no
>>> Before=shutdown.target reboot.target
>>>
>>> [Service]
>>> Type=oneshot
>>> RemainAfterExit=true
>>> ExecStart=/bin/true
>>> ExecStop=/root/vm_migrate.sh
>>>
>>> [Install]
>>> WantedBy=multi-user.target
>>>
>>>
>>> ########## Command to migrate ############
>>> /usr/bin/virsh migrate --verbose --p2p --tunneled --live 
--compressed
>>> --comp-methods "mt"  --comp-mt-level 5 --comp-mt-threads
5
>>> --comp-mt-dthreads 5   MY_VM qemu+ssh://root@$node2/system
>>
>> Does this command alone succeed?
>> BTW: unless really needed --live will only make the migration take
longer.
>>
>> I don't know enough about systemd but maybe it's not waiting
for virsh
>> to finish? Can you try to get any logs to see what is going on
actually?
>>
>> Michal
>>
> 
Michal

Hi,
> I don't see your service being called.
It started when system started, see logs:
Jan 12 14:59:42 desktop4 systemd: systemd 219 running in system mode. (+PAM
+AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
+GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Jan 12 14:59:42 desktop4 systemd: Detected architecture x86-64.
Jan 12 14:59:42 desktop4 systemd: Set hostname to <desktop4>.
Jan 12 14:59:42 desktop4 systemd: Started Create list of required static
device nodes for the current kernel.
Jan 12 14:59:42 desktop4 systemd: Started *vPreShutdownHook*.
Jan 12 14:59:42 desktop4 systemd: Starting Create Static Device Nodes in
/dev...
> Anyway, look at libvirt-guests.service file. Looks like the following
lines cause systemd to wait for a command to finish:
The libvirt-guests.service is NOT running in either of my nodes, even
though VM is working fine.

I tried starting  libvirt-guests.service and added it as Requires target in
vPreShutdownHook.service,  but it did not help. On reboot the VM did not
migrated.

Thanks

On Thu, Jan 11, 2018 at 12:56 PM, Michal Privoznik <mprivozn@redhat.com>
wrote:
> [Please keed the list CCed]
>
> On 01/10/2018 04:43 PM, Raman Gupta wrote:
> >> Does this command alone succeed?
> > Yes. I have used this command to migrate VMs successfully, without
even
> > knowing that spelling has changed.
> >
> >
> >> I don't know enough about systemd but maybe it's not
waiting for virsh
> to
> > finish?
> > Yes I also think virsh or libvirtd or related service does not wait
> > before /root/vm_migrate.sh
> > is called and hence live migration fails.
> > If I replace Live Migration with a simple ping to peer node in the
> > vPreShutdownHook.service,  then ping goes thru successfully thus
> indicating
> > network service was UP at that time.
> >
> >> Can you try to get any logs to see what is going on actually?
> > Jan  8 15:30:58 desktop4 systemd: Stopping Session c1 of user gdm.
> > Jan  8 15:30:58 desktop4 gdm: Freeing conversation
> 'gdm-launch-environment'
> > with active job
> > Jan  8 15:30:58 desktop4 systemd: Stopped target Sound Card.
> > Jan  8 15:30:58 desktop4 systemd: Stopping Sound Card.
> > Jan  8 15:30:58 desktop4 systemd: Stopping LVM2 PV scan on device
8:3...
> > Jan  8 15:30:58 desktop4 systemd: Removed slice system-getty.slice.
> > Jan  8 15:30:58 desktop4 systemd: Stopping system-getty.slice.
> > Jan  8 15:30:58 desktop4 systemd: Stopping Authorization Manager...
> > Jan  8 15:30:58 desktop4 systemd: Stopping Virtual Machine and
Container
> > Registration Service...
> > Jan  8 15:30:58 desktop4 systemd: Stopping GlusterFS brick processes
> > (stopping only)...
> > Jan  8 15:30:58 desktop4 systemd: Removed slice
> > system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
> > Jan  8 15:30:58 desktop4 systemd: Stopping
> > system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
> > Jan  8 15:30:58 desktop4 systemd: Stopping Availability of block
> devices...
> > Jan  8 15:30:58 desktop4 systemd: Stopping Virtual Machine
> > qemu-1-GsmController.
> > Jan  8 15:30:58 desktop4 alsactl[837]: alsactl daemon stopped
> > Jan  8 15:30:58 desktop4 systemd: Stopping Session 1 of user root.
> > Jan  8 15:30:58 desktop4 systemd: Stopping Manage Sound Card State
> (restore
> > and store)...
>
> I don't see your service being called. Anyway, look at
> libvirt-guests.service file. Looks like the following lines cause
> systemd to wait for a command to finish:
>
> [Service]
> EnvironmentFile=-/etc/sysconfig/libvirt-guests
> # Hack just call traditional service until we factor
> # out the code
> ExecStart=@libexecdir@/libvirt-guests.sh start
> ExecStop=@libexecdir@/libvirt-guests.sh stop
> Type=oneshot
> RemainAfterExit=yes
> StandardOutput=journal+console
> TimeoutStopSec=0
>
> >
> >
> >
> > On Wed, Jan 10, 2018 at 8:46 PM, Michal Privoznik
<mprivozn@redhat.com>
> > wrote:
> >
> >> On 01/05/2018 12:00 PM, Raman Gupta wrote:
> >>> Hi,
> >>>
> >>> I have CentOS 7,  two node system which allows live VM
migration
> between
> >>> them. Live migration triggered from virsh is happily
happening. I am
> >> using
> >>> GlusterFS for replicating VM disk files.
> >>>
> >>> Now I want to automatically do the live migration at the time
of
> >>> reboot/shutdown/halt of the host node and for this I have
written a
> >> systemd
> >>> service unit [vPreShutdownHook.service] and placed the live
migration
> >>> command in a migrate script which is invoked from this service
unit.
> The
> >>> migrate script is invoked but migration does not happen.
> >>> If someone has some idea please help me to migrate VM upon
shutdown.
> >>>
> >>>
> >>> ######## vPreShutdownHook.service ###############
> >>>
> >>> [Unit]
> >>> Description=vPreShutdownHook
> >>> Requires=network.target
> >>> Requires=libvirtd.service
> >>> Requires=dbus.service
> >>> Requires=glusterd.service
> >>> Requires=glusterfsd.service
> >>> DefaultDependencies=no
> >>> Before=shutdown.target reboot.target
> >>>
> >>> [Service]
> >>> Type=oneshot
> >>> RemainAfterExit=true
> >>> ExecStart=/bin/true
> >>> ExecStop=/root/vm_migrate.sh
> >>>
> >>> [Install]
> >>> WantedBy=multi-user.target
> >>>
> >>>
> >>> ########## Command to migrate ############
> >>> /usr/bin/virsh migrate --verbose --p2p --tunneled --live 
--compressed
> >>> --comp-methods "mt"  --comp-mt-level 5
--comp-mt-threads 5
> >>> --comp-mt-dthreads 5   MY_VM qemu+ssh://root@$node2/system
> >>
> >> Does this command alone succeed?
> >> BTW: unless really needed --live will only make the migration take
> longer.
> >>
> >> I don't know enough about systemd but maybe it's not
waiting for virsh
> >> to finish? Can you try to get any logs to see what is going on
actually?
> >>
> >> Michal
> >>
> >
>
> Michal
>