libvirt users - Oct 2014 - Re: [libvirt] increase number of libvirt threads by starting tansient guest doamin - is it a bug?

Hi Michal,
thank you for your answer.
so if i understand that correctly, no matter if i shutdown or destroy the domain
and no matter if it is a transient or an persistent vm, the thread should
disappear, right?

in my case they still exist, also an hour after i destroy the domain (and don´t
start any new one).

i use libvirt-1.1.35 on fedora core 20, for information.

all the best
max
 
 

Gesendet: Freitag, 26. September 2014 um 11:21 Uhr
Von: "Michal Privoznik" <mprivozn@redhat.com>
An: ustermann78@web.de, libvir-list@redhat.com, libvirt-users@redhat.com
Betreff: Re: [libvirt] increase number of libvirt threads by starting tansient
guest doamin - is it a bug?
On 26.09.2014 09:46, ustermann78@web.de wrote:
> hello,
>
> if i start a transient guest doamin via "virsh create abcd.xml" i
see an additional libvirt thread and also some open files:
>
> pstree -h `pgrep libvirtd`
> libvirtd───11*[{libvirtd}]
>
> libvirtd 3016 root 21w REG 253,0 6044 1052094
/var/log/libvirt/libxl/abcd.log
> libvirtd 3016 root 22r FIFO 0,8 0t0 126124 pipe
> libvirtd 3016 root 23w FIFO 0,8 0t0 126124 pipe
> libvirtd 3016 root 24u REG 0,37 0 4 /proc/xen/privcmd
> libvirtd 3016 root 25u unix 0xffff8807d2c3ad80 0t0 126125 socket
> libvirtd 3016 root 26r FIFO 0,8 0t0 126127 pipe
> libvirtd 3016 root 27w FIFO 0,8 0t0 126127 pipe
> libvirtd 3016 root 28r FIFO 0,8 0t0 124783 pipe
> libvirtd 3016 root 29w FIFO 0,8 0t0 124783 pipe
> libvirtd 3016 root 30r FIFO 0,8 0t0 127140 pipe
> libvirtd 3016 root 31w FIFO 0,8 0t0 127140 pipe
>
> if i destroy these vm via "virsh destroy abcd", i see that the
additional thread still exists and also the list of openfiles is the same.
> if i start the transient guest domain again, i observe an increase in the
number of libvirt threads ans also in the list of openfiles:
>
> [root@localhost libxl]# pstree -h `pgrep libvirtd`
> libvirtd───12*[{libvirtd}]
>
> libvirtd 3016 root 21w REG 253,0 13783 1052094
/var/log/libvirt/libxl/abcd.log
> libvirtd 3016 root 22r FIFO 0,8 0t0 126124 pipe
> libvirtd 3016 root 23w FIFO 0,8 0t0 126124 pipe
> libvirtd 3016 root 24u REG 0,37 0 4 /proc/xen/privcmd
> libvirtd 3016 root 25u unix 0xffff8807d2c3ad80 0t0 126125 socket
> libvirtd 3016 root 26r FIFO 0,8 0t0 126127 pipe
> libvirtd 3016 root 27w FIFO 0,8 0t0 126127 pipe
> libvirtd 3016 root 28r FIFO 0,8 0t0 124783 pipe
> libvirtd 3016 root 29w FIFO 0,8 0t0 124783 pipe
> libvirtd 3016 root 30r FIFO 0,8 0t0 127140 pipe
> libvirtd 3016 root 31w FIFO 0,8 0t0 127140 pipe
> libvirtd 3016 root 32w REG 253,0 13783 1052094
/var/log/libvirt/libxl/abcd.log
> libvirtd 3016 root 33r FIFO 0,8 0t0 129039 pipe
> libvirtd 3016 root 34w FIFO 0,8 0t0 129039 pipe
> libvirtd 3016 root 35u REG 0,37 0 4 /proc/xen/privcmd
> libvirtd 3016 root 36u unix 0xffff8807d398bb80 0t0 129040 socket
> libvirtd 3016 root 37r FIFO 0,8 0t0 129042 pipe
> libvirtd 3016 root 38w FIFO 0,8 0t0 129042 pipe
> libvirtd 3016 root 39r FIFO 0,8 0t0 129043 pipe
> libvirtd 3016 root 40w FIFO 0,8 0t0 129043 pipe
> libvirtd 3016 root 41r FIFO 0,8 0t0 129044 pipe
> libvirtd 3016 root 42w FIFO 0,8 0t0 129044 pipe
>
> if i destroy the doamin again and define them via "virsh define
abcd.xml" and start them then via "virsh start abcd", the number
of libvirtd threads don´t increase again and also the number of open files is
the same.
>
> My question: is it normal that for transient guest domains the created
libvirtd thread sill exists and also the open files after i destroy the doamin?
or is it bug?

The thread is created as a result of this commit:

commit 03b3f8940af1a3179b7d5e19b25f54290bde10e0
Author: Jim Fehlig <jfehlig@suse.com>
AuthorDate: Fri Jan 31 23:06:35 2014 -0700
Commit: Jim Fehlig <jfehlig@suse.com>
CommitDate: Thu Feb 6 10:17:58 2014 -0700

libxl: handle domain shutdown events in a thread

Handling the domain shutdown event within the event handler seems
a bit unfair to libxl's event machinery. Domain "shutdown" could
take considerable time. E.g. if the shutdown reason is reboot,
the domain must be reaped and then started again.

Spawn a shutdown handler thread to do this work, allowing libxl's
event machinery to go about its business.

Signed-off-by: Jim Fehlig <jfehlig@suse.com>



Which was then modified by 0a840e23 (pure code movement). The thing is,
libvirt listens on events from hypervisors. And some actions that are
taken subsequently may take ages to finish. However, the events are
executed single threaded, so they serialize effectively. And for the
long lasting actions (like shutdown, in fact in libxl shutdown only) a
new thread is spawned so the event loop is not burdened for too long.
The thread is created with 'detach-state' attribute so once the shutdown
is handled the thread should just disappear.

Michal

[Disabling cross-post]

On 26.09.2014 12:21, ustermann78@web.de wrote:
> Hi Michal,
> thank you for your answer.
> so if i understand that correctly, no matter if i shutdown or destroy the
domain and no matter if it is a transient or an persistent vm, the thread should
disappear, right?
>
> in my case they still exist, also an hour after i destroy the domain (and
don´t start any new one).
>
> i use libvirt-1.1.35 on fedora core 20, for information.
Well, can you attach gdb and see what is the new thread doing?

Michal

On 09/26/2014 07:22 AM, Michal Privoznik wrote:
> [Disabling cross-post]
> 
> On 26.09.2014 12:21, ustermann78@web.de wrote:
>> Hi Michal,
>> thank you for your answer.
>> so if i understand that correctly, no matter if i shutdown or destroy
>> the domain and no matter if it is a transient or an persistent vm, the
>> thread should disappear, right?
>>
>> in my case they still exist, also an hour after i destroy the domain
>> (and don´t start any new one).
>>
>> i use libvirt-1.1.35 on fedora core 20, for information.
> 
> Well, can you attach gdb and see what is the new thread doing?
Also, we intentionally keep a pool of worker threads, in order to
minimize the overhead of creating and tearing down threads; the pool is
populated on demand.  If there is a free thread, we reuse it; otherwise
we create the new one.  A thread that has nothing to do is not wasting
resources while waiting around for a new reason to be used.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Hello,

sorry for my later answer.

so, after i started libvirtd (no vm´s running) and attach gdb i get the
following threads

(gdb) info thread
 Id Target Id Frame 
 11 Thread 0x7f18fef4e700 (LWP 20695) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7f18fe74d700 (LWP 20696) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7f18fdf4c700 (LWP 20697) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7f18fd74b700 (LWP 20698) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7f18fcf4a700 (LWP 20699) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7f18fc749700 (LWP 20700) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7f18fbf48700 (LWP 20701) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7f18fb747700 (LWP 20702) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7f18faf46700 (LWP 20703) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7f18fa745700 (LWP 20704) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7f190892f840 (LWP 20694) "libvirtd" 0x00007f190677d7cd in
poll () at ../sysdeps/unix/syscall-template.S:81

if i restore an persistent domain, i see the following in gdb:

Detaching after fork from child process 20880.
Detaching after fork from child process 20882.
[New Thread 0x7f190893d700 (LWP 20883)]
Detaching after fork from child process 20890.
Detaching after fork from child process 20906.

(gdb) info thread
 Id Target Id Frame 
 12 Thread 0x7f190893d700 (LWP 20883) "libvirtd" 0x00007f1906e6684d in
read () at ../sysdeps/unix/syscall-template.S:81
 11 Thread 0x7f18fef4e700 (LWP 20695) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7f18fe74d700 (LWP 20696) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7f18fdf4c700 (LWP 20697) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7f18fd74b700 (LWP 20698) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7f18fcf4a700 (LWP 20699) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7f18fc749700 (LWP 20700) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7f18fbf48700 (LWP 20701) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7f18fb747700 (LWP 20702) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7f18faf46700 (LWP 20703) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7f18fa745700 (LWP 20704) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7f190892f840 (LWP 20694) "libvirtd" 0x00007f190677d7cd in
poll () at ../sysdeps/unix/syscall-template.S:81

if i now destroy this vm i get the following:


(gdb) info thread
 Id Target Id Frame 
 12 Thread 0x7f190893d700 (LWP 20883) "libvirtd" 0x00007f1906e6684d in
read () at ../sysdeps/unix/syscall-template.S:81
 11 Thread 0x7f18fef4e700 (LWP 20695) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7f18fe74d700 (LWP 20696) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7f18fdf4c700 (LWP 20697) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7f18fd74b700 (LWP 20698) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7f18fcf4a700 (LWP 20699) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7f18fc749700 (LWP 20700) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7f18fbf48700 (LWP 20701) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7f18fb747700 (LWP 20702) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7f18faf46700 (LWP 20703) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7f18fa745700 (LWP 20704) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7f190892f840 (LWP 20694) "libvirtd" 0x00007f190677d7cd in
poll () at ../sysdeps/unix/syscall-template.S:81

attaching thread Id 12 and bt:
(gdb) thread 12
[Switching to thread 12 (Thread 0x7f190893d700 (LWP 20883))]
#0 0x00007f1906e6684d in read () at ../sysdeps/unix/syscall-template.S:81
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f1906e6684d in read () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f18f8c415be in read (__nbytes=16, __buf=0x7f18d4000c50, __fd=26) at
/usr/include/bits/unistd.h:44
#2 read_all (fd=26, data=data@entry =0x7f18d4000c50, len=len@entry =16,
nonblocking=nonblocking@entry =0) at xs.c:374
#3 0x00007f18f8c41675 in read_message (h=h@entry =0x7f18e8001070,
nonblocking=nonblocking@entry =0) at xs.c:1139
#4 0x00007f18f8c41e90 in read_thread (arg=0x7f18e8001070) at xs.c:1211
#5 0x00007f1906e5ff35 in start_thread (arg=0x7f190893d700) at
pthread_create.c:309
#6 0x00007f1906787c3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111

and then bt full:

(gdb) bt full
#0 0x00007f1906e6684d in read () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f18f8c415be in read (__nbytes=16, __buf=0x7f18d4000c50, __fd=26) at
/usr/include/bits/unistd.h:44
No locals.
#2 read_all (fd=26, data=data@entry =0x7f18d4000c50, len=len@entry =16,
nonblocking=nonblocking@entry =0) at xs.c:374
 done = <optimized out>
#3 0x00007f18f8c41675 in read_message (h=h@entry =0x7f18e8001070,
nonblocking=nonblocking@entry =0) at xs.c:1139
 __clframe = {__cancel_routine = <optimized out>, __cancel_arg =
0x7f18d4000c40, __do_it = 1, __cancel_type = <optimized out>}
 msg = 0x7f18d4000c40
 body = 0x0
 saved_errno = 0
 ret = -1
#4 0x00007f18f8c41e90 in read_thread (arg=0x7f18e8001070) at xs.c:1211
 h = 0x7f18e8001070
 fd = <optimized out>
#5 0x00007f1906e5ff35 in start_thread (arg=0x7f190893d700) at
pthread_create.c:309
 __res = <optimized out>
 pd = 0x7f190893d700
 now = <optimized out>
 unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139745494816512,
9092725460686644452, 1, 0, 139745494817216, 139745494816512,
-9214988784542636828, -9214975709026802460}, mask_was_saved = 0}}, priv = {pad =
{0x0, 0x0, 0x0, 0x0},
 data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
 not_first_call = <optimized out>
 pagesize_m1 = <optimized out>
 sp = <optimized out>
 freesize = <optimized out>
#6 0x00007f1906787c3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.



now the same for a transient domain, also after start of libvirtd (no vm
running):

(gdb) info thread
 Id Target Id Frame 
 11 Thread 0x7fe5cf507700 (LWP 21097) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7fe5ced06700 (LWP 21098) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7fe5ce505700 (LWP 21099) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7fe5cdd04700 (LWP 21100) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7fe5cd503700 (LWP 21101) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7fe5ccd02700 (LWP 21102) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7fe5cc501700 (LWP 21103) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7fe5cbd00700 (LWP 21104) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7fe5cb4ff700 (LWP 21105) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7fe5cacfe700 (LWP 21106) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7fe5d8ee8840 (LWP 21096) "libvirtd" 0x00007fe5d6d367cd in
poll () at ../sysdeps/unix/syscall-template.S:81

now restore the transient domain from a statefile without define them before:

Continuing.
Detaching after fork from child process 21276.
Detaching after fork from child process 21278.
[New Thread 0x7fe5d8ef6700 (LWP 21279)]
Detaching after fork from child process 21285.
Detaching after fork from child process 21302.

(gdb) info thread
 Id Target Id Frame 
 12 Thread 0x7fe5d8ef6700 (LWP 21279) "libvirtd" 0x00007fe5d741f84d in
read () at ../sysdeps/unix/syscall-template.S:81
 11 Thread 0x7fe5cf507700 (LWP 21097) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7fe5ced06700 (LWP 21098) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7fe5ce505700 (LWP 21099) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7fe5cdd04700 (LWP 21100) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7fe5cd503700 (LWP 21101) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7fe5ccd02700 (LWP 21102) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7fe5cc501700 (LWP 21103) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7fe5cbd00700 (LWP 21104) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7fe5cb4ff700 (LWP 21105) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7fe5cacfe700 (LWP 21106) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7fe5d8ee8840 (LWP 21096) "libvirtd" 0x00007fe5d6d367cd in
poll () at ../sysdeps/unix/syscall-template.S:81

destroy these vm:

(gdb) info thread
 Id Target Id Frame 
 12 Thread 0x7fe5d8ef6700 (LWP 21279) "libvirtd" 0x00007fe5d741f84d in
read () at ../sysdeps/unix/syscall-template.S:81
 11 Thread 0x7fe5cf507700 (LWP 21097) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7fe5ced06700 (LWP 21098) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7fe5ce505700 (LWP 21099) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7fe5cdd04700 (LWP 21100) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7fe5cd503700 (LWP 21101) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7fe5ccd02700 (LWP 21102) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7fe5cc501700 (LWP 21103) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7fe5cbd00700 (LWP 21104) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7fe5cb4ff700 (LWP 21105) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7fe5cacfe700 (LWP 21106) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7fe5d8ee8840 (LWP 21096) "libvirtd" 0x00007fe5d6d367cd in
poll () at ../sysdeps/unix/syscall-template.S:81


and again restore vm:

Continuing.
Detaching after fork from child process 21347.
Detaching after fork from child process 21349.
[New Thread 0x7fe5d8dff700 (LWP 21350)]
Detaching after fork from child process 21353.
Detaching after fork from child process 21373.

(gdb) info thread
 Id Target Id Frame 
 13 Thread 0x7fe5d8dff700 (LWP 21350) "libvirtd" 0x00007fe5d741f84d in
read () at ../sysdeps/unix/syscall-template.S:81
 12 Thread 0x7fe5d8ef6700 (LWP 21279) "libvirtd" 0x00007fe5d741f84d in
read () at ../sysdeps/unix/syscall-template.S:81
 11 Thread 0x7fe5cf507700 (LWP 21097) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7fe5ced06700 (LWP 21098) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7fe5ce505700 (LWP 21099) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7fe5cdd04700 (LWP 21100) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7fe5cd503700 (LWP 21101) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7fe5ccd02700 (LWP 21102) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7fe5cc501700 (LWP 21103) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7fe5cbd00700 (LWP 21104) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7fe5cb4ff700 (LWP 21105) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7fe5cacfe700 (LWP 21106) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7fe5d8ee8840 (LWP 21096) "libvirtd" 0x00007fe5d6d367cd in
poll () at ../sysdeps/unix/syscall-template.S:81

and again, destroy vm:

(gdb) info thread
 Id Target Id Frame 
 13 Thread 0x7fe5d8dff700 (LWP 21350) "libvirtd" 0x00007fe5d741f84d in
read () at ../sysdeps/unix/syscall-template.S:81
 12 Thread 0x7fe5d8ef6700 (LWP 21279) "libvirtd" 0x00007fe5d741f84d in
read () at ../sysdeps/unix/syscall-template.S:81
 11 Thread 0x7fe5cf507700 (LWP 21097) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 10 Thread 0x7fe5ced06700 (LWP 21098) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 9 Thread 0x7fe5ce505700 (LWP 21099) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 8 Thread 0x7fe5cdd04700 (LWP 21100) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 7 Thread 0x7fe5cd503700 (LWP 21101) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 6 Thread 0x7fe5ccd02700 (LWP 21102) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 5 Thread 0x7fe5cc501700 (LWP 21103) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 4 Thread 0x7fe5cbd00700 (LWP 21104) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 3 Thread 0x7fe5cb4ff700 (LWP 21105) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 2 Thread 0x7fe5cacfe700 (LWP 21106) "libvirtd"
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7fe5d8ee8840 (LWP 21096) "libvirtd" 0x00007fe5d6d367cd in
poll () at ../sysdeps/unix/syscall-template.S:81

attach thread 13 and bt full:

(gdb) thread 13
[Switching to thread 13 (Thread 0x7fe5d8dff700 (LWP 21350))]
#0 0x00007fe5d741f84d in read () at ../sysdeps/unix/syscall-template.S:81
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt full
#0 0x00007fe5d741f84d in read () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007fe5c91fa5be in read (__nbytes=16, __buf=0x7fe5a4000be0, __fd=37) at
/usr/include/bits/unistd.h:44
No locals.
#2 read_all (fd=37, data=data@entry =0x7fe5a4000be0, len=len@entry =16,
nonblocking=nonblocking@entry =0) at xs.c:374
 done = <optimized out>
#3 0x00007fe5c91fa675 in read_message (h=h@entry =0x7fe5b80019b0,
nonblocking=nonblocking@entry =0) at xs.c:1139
 __clframe = {__cancel_routine = <optimized out>, __cancel_arg =
0x7fe5a4000bd0, __do_it = 1, __cancel_type = <optimized out>}
 msg = 0x7fe5a4000bd0
 body = 0x0
 saved_errno = 0
 ret = -1
#4 0x00007fe5c91fae90 in read_thread (arg=0x7fe5b80019b0) at xs.c:1211
 h = 0x7fe5b80019b0
 fd = <optimized out>
#5 0x00007fe5d7418f35 in start_thread (arg=0x7fe5d8dff700) at
pthread_create.c:309
 __res = <optimized out>
 pd = 0x7fe5d8dff700
 now = <optimized out>
 unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140625162794752,
9202584519348287509, 1, 0, 140625162795456, 140625162794752,
-9188024510920296427, -9188034420488439787}, mask_was_saved = 0}}, priv = {pad =
{0x0, 0x0, 0x0, 0x0},
 data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
 not_first_call = <optimized out>
 pagesize_m1 = <optimized out>
 sp = <optimized out>
 freesize = <optimized out>
#6 0x00007fe5d6d40c3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

i seen nothing that give me a hint while the new threads a still open.

all the best max






 
----------------ursprüngliche Nachricht-----------------
Von: "Michal Privoznik" mprivozn@redhat.com 
An: ustermann78@web.de, libvirt-users@redhat.com 
Datum: Fri, 26 Sep 2014 15:22:01 +0200
-------------------------------------------------
 
 
> [Disabling cross-post]
> 
> On 26.09.2014 12:21, ustermann78@web.de wrote:
>> Hi Michal,
>> thank you for your answer.
>> so if i understand that correctly, no matter if i shutdown or destroy
the
>> domain and no matter if it is a transient or an persistent vm, the
thread should
>> disappear, right?
>>
>> in my case they still exist, also an hour after i destroy the domain
(and don´t
>> start any new one).
>>
>> i use libvirt-1.1.35 on fedora core 20, for information.
> 
> Well, can you attach gdb and see what is the new thread doing?
> 
> Michal
> 
--