libvirt users - Dec 2011 - Thread-safety issues with vbox driver ?

Hi,

I'm experiencing weird things with the vbox driver when using multiple
threads.
Following is the snippet of code I experience problems with

/*****************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <pthread.h>
#include <libvirt/libvirt.h>

void *create_and_destroy(void *arg)
{
    virDomainPtr dom = (virDomainPtr)arg;

    char buf[VIR_UUID_STRING_BUFLEN];
    virDomainGetUUIDString(dom, buf);
    
    if (virDomainCreate(dom) != 0) {
        printf("failed to start %s\n", buf);
        goto end;
    }
    printf("%s started\n", buf);

    if (virDomainDestroy(dom) != 0) {
        printf("failed to destroy %s\n", buf);
    }
    printf("%s destroyed\n", buf);

end:
    virDomainFree(dom);
    pthread_exit(NULL);
}

int main(int argc, char **argv)
{
    virConnectPtr conn = virConnectOpen("vbox:///session");
    
    int i;
    int count = virConnectNumOfDefinedDomains(conn);
    char **names = malloc(count * sizeof(char *));
    virConnectListDefinedDomains(conn, names, count);
    virDomainPtr *doms = malloc(count * sizeof(virDomainPtr));
    for (i = 0 ; i < count ; ++i) {
        doms[i] = virDomainLookupByName(conn, names[i]);
    }

    pthread_t *threads = malloc(count * sizeof(pthread_t));
    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

    for (i = 0 ; i < count ; ++i) {
        pthread_create(&threads[i], &attr, create_and_destroy, (void
*)doms[i]);
    }

    pthread_attr_destroy(&attr);
    for (i = 0 ; i < count ; ++i) {
        pthread_join(threads[i], NULL);
    }

    virConnectClose(conn);

    pthread_exit(NULL);
}
/************************************************/

Here is the output of the program with 2 defined
domains:
> libvir: VBOX error : operation failed: OpenRemoteSession/LaunchVMProcess
failed, domain can't be started
> failed to start c538c89a-70da-42ab-a88a-5aeb15698c12
> 034cf837-abe7-4c48-8373-0ddcf480d416 started
> 034cf837-abe7-4c48-8373-0ddcf480d416 destroyed
Sometimes the first domain really fails to start, but sometimes it starts
correctly but libvirt reports
an error. Sometimes domains aren't destroyed but libvirt reports no error at
all.
If there is only one domain, no problem occurs at all. I also tried the same
code (ie with multiple domains)
but with only one thread and it works well.

I managed to reproduce these issues with libvirt 0.9.4, 0.9.7, using VirtualBox
4.0 and 4.1.

-- 
Jean-Baptiste ROUAULT
Ing?nieur R&D - diateam : Architectes de l'information
Phone : +33 (0)9 53 16 02 70 Fax : +33 (0)2 98 050 051

2011/12/13 Jean-Baptiste Rouault <jean-baptiste.rouault at
diateam.net>:
> Hi,
>
> I'm experiencing weird things with the vbox driver when using multiple
threads.
> Following is the snippet of code I experience problems with
>
> /*****************************************************/
> #include <stdlib.h>
> #include <stdio.h>
> #include <pthread.h>
> #include <libvirt/libvirt.h>
>
> void *create_and_destroy(void *arg)
> {
> ? ?virDomainPtr dom = (virDomainPtr)arg;
>
> ? ?char buf[VIR_UUID_STRING_BUFLEN];
> ? ?virDomainGetUUIDString(dom, buf);
>
> ? ?if (virDomainCreate(dom) != 0) {
> ? ? ? ?printf("failed to start %s\n", buf);
> ? ? ? ?goto end;
> ? ?}
> ? ?printf("%s started\n", buf);
>
> ? ?if (virDomainDestroy(dom) != 0) {
> ? ? ? ?printf("failed to destroy %s\n", buf);
> ? ?}
> ? ?printf("%s destroyed\n", buf);
>
> end:
> ? ?virDomainFree(dom);
> ? ?pthread_exit(NULL);
> }
>
> int main(int argc, char **argv)
> {
> ? ?virConnectPtr conn = virConnectOpen("vbox:///session");
>
> ? ?int i;
> ? ?int count = virConnectNumOfDefinedDomains(conn);
> ? ?char **names = malloc(count * sizeof(char *));
> ? ?virConnectListDefinedDomains(conn, names, count);
> ? ?virDomainPtr *doms = malloc(count * sizeof(virDomainPtr));
> ? ?for (i = 0 ; i < count ; ++i) {
> ? ? ? ?doms[i] = virDomainLookupByName(conn, names[i]);
> ? ?}
>
> ? ?pthread_t *threads = malloc(count * sizeof(pthread_t));
> ? ?pthread_attr_t attr;
> ? ?pthread_attr_init(&attr);
> ? ?pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
>
> ? ?for (i = 0 ; i < count ; ++i) {
> ? ? ? ?pthread_create(&threads[i], &attr, create_and_destroy, (void
*)doms[i]);
> ? ?}
>
> ? ?pthread_attr_destroy(&attr);
> ? ?for (i = 0 ; i < count ; ++i) {
> ? ? ? ?pthread_join(threads[i], NULL);
> ? ?}
>
> ? ?virConnectClose(conn);
>
> ? ?pthread_exit(NULL);
> }
> /************************************************/
>
> Here is the output of the program with 2 defined domains:
>> libvir: VBOX error : operation failed:
OpenRemoteSession/LaunchVMProcess failed, domain can't be started
>> failed to start c538c89a-70da-42ab-a88a-5aeb15698c12
>> 034cf837-abe7-4c48-8373-0ddcf480d416 started
>> 034cf837-abe7-4c48-8373-0ddcf480d416 destroyed
>
> Sometimes the first domain really fails to start, but sometimes it starts
correctly but libvirt reports
> an error. Sometimes domains aren't destroyed but libvirt reports no
error at all.
> If there is only one domain, no problem occurs at all. I also tried the
same code (ie with multiple domains)
> but with only one thread and it works well.
>
> I managed to reproduce these issues with libvirt 0.9.4, 0.9.7, using
VirtualBox 4.0 and 4.1.
Okay, without looking deeper into this here are some ideas:

The XPCOM API libvirt uses might not be threadsafe, or needs to be
initialized for every thread that wants to use it. Currently its only
initialized for the thread that opens the driver. I know that this is
the case on Windows were VirtualBox uses MSCOM for its API and you
need to call CoInitialize on every thread. This is currently not done
for the MSCOM glue in libvirt, so I know that on Windows the
VirtualBox driver is not threadsafe currently. Also I didn't look into
a solution for this yet. Maybe we need a thread local variable that
holds whether (MS/XP)COM was already initialized for this thread and
add a check to every driver function to initialize it when needed.

Did you try to open a connection for each thread instead of trying to
share one? If that works reliable it might indicate that there is an
VirtualBox API initialization problem.

Did you try to add the actual VirtualBox error code for the
OpenRemoteSession/LaunchVMProcess error in hex to the error message
and look it up in the src/vbox/vbox_CAPI_v4_1.h?

-- 
Matthias Bolte
http://photron.blogspot.com