libvirt users - Nov 2013 - Re: Calling 'virsh' from within a hook script - avoid deadlock?

I'm trying to create a hook that attaches several virtio-scsi disks to a
starting vm,
and it seems like the hook enters a deadlock while attempting to do so.

Is there any workaround around this? Any better way, forking virsh execution
somehow?

Could anyone explain why this is creating a deadlock in the first place?
Isn't libvirt multithreaded?

Thanks for the help!

Shlomi

### hook script:

#!/bin/bash

enum_scsi() {
    scsi_devices=`cd /dev;ls sd[^a]*`
}

gen_hba_xml() {
    cat > ./hba.xml << EOF
    <controller type='scsi' model='virtio-scsi'/>
EOF
}

passthrough_start() {
    # create virtio-scsi HBA
    gen_hba_xml
    virsh attach-device --persistent $domain_name ./hba.xml
    rm -f ./hba.xml

    # attache scsi devices to guest
    enum_scsi
    for sd in ${scsi_devices[@]}
    do
        virsh attach-disk --persistent $domain_name /dev/${sd} ${sd}
    done
}

domain_name=$1
domain_event=$2

case $domain_event in
    start)
        passthrough_start
        ;;
    *)
                echo "qemu hook called with unexpected options $*"
>&2
        ;;
esac


### starce libvirt:

[root@localhost ~]# strace -f -p 14630
Process 14630 attached with 11 threads - interrupt to quit
[pid 14640] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished
...>
[pid 14638] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished
...>
[pid 14637] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished
...>
[pid 14636] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished
...>
[pid 14635] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished
...>
[pid 14639] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished
...>
[pid 14634] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished
...>
[pid 14631] restart_syscall(<... resuming interrupted call ...>
<unfinished ...>
[pid 14633] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished
...>
[pid 14630] restart_syscall(<... resuming interrupted call ...>
<unfinished ...>
[pid 14632] futex(0xc34e24, FUTEX_WAIT_PRIVATE, 3091, NULL^C <unfinished
...>

...

[pid 14630] poll([{fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=12,
events=POLLIN}, {fd=13, events=POLLIN}, {fd=6, events=POLLIN}, {fd=14,
events=POLLIN}, {fd=16, events=POLLIN}, {fd=18, events=POLLIN|POLLOUT}, {fd=24,
events=POLLIN}], 9, 1927) = 1 ([{fd=18, revents=POLLOUT}])
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] gettid()                    = 14630
[pid 14630] write(18,
"\0\0\0\34keep\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0", 28) = 28

...


### strace bash:

[root@localhost ~]# strace -f -p 14783
Process 14783 attached - interrupt to quit
wait4(-1,

...

On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
> I'm trying to create a hook that attaches several virtio-scsi disks to
a
> starting vm,
> 
> and it seems like the hook enters a deadlock while attempting to do so.
> 
>  
> 
> Is there any workaround around this? Any better way, forking virsh
> execution somehow?
> 
>  
> 
> Could anyone explain why this is creating a deadlock in the first place?
> Isn't libvirt multithreaded?
> 
>  
> 
> Thanks for the help!
> 
>  
> 
> Shlomi
> 
>  
> 
> ### hook script:
> 
>  
> 
> #!/bin/bash
> 
>  
> 
> enum_scsi() {
> 
>     scsi_devices=`cd /dev;ls sd[^a]*`
> 
> }
> 
>  
> 
> gen_hba_xml() {
> 
>     cat > ./hba.xml << EOF
> 
>     <controller type='scsi' model='virtio-scsi'/>
> 
> EOF
> 
> }
> 
>  
> 
> passthrough_start() {
> 
>     # create virtio-scsi HBA
> 
>     gen_hba_xml
> 
>     virsh attach-device --persistent $domain_name ./hba.xml
> 
>     rm -f ./hba.xml
http://libvirt.org/hooks.html#recursive

Do not do this.

Michal

Yes.. I saw that, but I have to :)

The issue was solved by converting the hook to Python and using
'subprocess.Popen'.  deadlock is gone, all looks good.

Regards

Shlomi


-----Original Message-----
From: Michal Privoznik [mailto:mprivozn@redhat.com] 
Sent: Wednesday, November 06, 2013 5:17 PM
To: TSADOK, Shlomi (Shlomi)
Cc: libvirt-users@redhat.com
Subject: Re: [libvirt-users] Calling 'virsh' from within a hook script -
avoid deadlock?

On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
> I'm trying to create a hook that attaches several virtio-scsi disks to 
> a starting vm,
> 
> and it seems like the hook enters a deadlock while attempting to do so.
> 
>  
> 
> Is there any workaround around this? Any better way, forking virsh 
> execution somehow?
> 
>  
> 
> Could anyone explain why this is creating a deadlock in the first place?
> Isn't libvirt multithreaded?
> 
>  
> 
> Thanks for the help!
> 
>  
> 
> Shlomi
> 
>  
> 
> ### hook script:
> 
>  
> 
> #!/bin/bash
> 
>  
> 
> enum_scsi() {
> 
>     scsi_devices=`cd /dev;ls sd[^a]*`
> 
> }
> 
>  
> 
> gen_hba_xml() {
> 
>     cat > ./hba.xml << EOF
> 
>     <controller type='scsi' model='virtio-scsi'/>
> 
> EOF
> 
> }
> 
>  
> 
> passthrough_start() {
> 
>     # create virtio-scsi HBA
> 
>     gen_hba_xml
> 
>     virsh attach-device --persistent $domain_name ./hba.xml
> 
>     rm -f ./hba.xml
http://libvirt.org/hooks.html#recursive

Do not do this.

Michal

On 06.11.2013 16:17, Michal Privoznik wrote:
> On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
>> [...]
> 
> http://libvirt.org/hooks.html#recursive
> 
> Do not do this.
> 
> Michal
> 
Maybe I should explain a bit why this is not supported. It's true that
libvirt is multithreaded. However, not in terms of multiple write/modify
accesses to a single domain. That is, when a domain is being started up
it is a modify access. Meanwhile, at some point during the transaction
the startup hook script is called. If the script decides to start
another modify transaction an deadlock occurs.

Michal