Rafal Wojtczuk
2010-Jun-08 10:15 UTC
[Xen-devel] A small issue with multiple xs_daemon_open
Hello, I am experiencing a problem when opening two simultaneous connections (in a single process) to xenstored via xs_daemon_open(). I would think such an operation is supported and should work (no static variables in libxenstore are involved); is it a correct assumption ? If so, the issue should be fixed. The environment is xen-3.4.3, 2.6.32.9-xxx pvops dom0. The order of operations is xs1=xs_daemon_open() xs_watch(xs1,...) xs_read_watch(xs1,...) xs2=xs_daemon_open() xs_read(xs2,...) xs_daemon_close(xs2) xs_read(xs1,...) xs_daemon_close(xs1) There seems to be some sort of a race when calling the second xs_daemon_close() - sometimes it hangs infinitely. Backtrace: (gdb) info threads 2 Thread 0x7fe64697c710 (LWP 4154) 0x00007fe646b8dc44 in __lll_lock_wait() from /lib64/libpthread.so.0 * 1 Thread 0x7fe6482d2700 (LWP 4153) 0x00007fe646b87bfd in pthread_join () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007fe646b87bfd in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fe647776fdc in xs_daemon_close () from /usr/lib64/libxenstore.so.3.0 #2 0x000000000040660a in peer_client_init () #3 0x0000000000405808 in main () (gdb) thread 2 [Switching to thread 2 (Thread 0x7fe64697c710 (LWP 4154))]#0 0x00007fe646b8dc44 in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007fe646b8dc44 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fe646b88f15 in _L_lock_1017 () from /lib64/libpthread.so.0 #2 0x00007fe646b88de7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fe647775db8 in ?? () from /usr/lib64/libxenstore.so.3.0 #4 0x00007fe647775e3c in ?? () from /usr/lib64/libxenstore.so.3.0 #5 0x00007fe646b86a3a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe64729967d in clone () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () (gdb) You can see the code for peer_client_init() at http://qubes-os.org/gitweb/?p=rafal/gui.git;a=blob;f=gui-common/txrx-vchan.c;h=afc691e922edce61683e46203fd9597d029052d3;hb=93f0509b791f8fc433675c58c81f6ea9469d2c91#l168 it calls libvchan_client_init, code at http://qubes-os.org/gitweb/?p=rafal/gui.git;a=blob;f=vchan/vchan/init.c;h=4a3da4f22907445160a6fe3db9ed357f89ebc301;hb=93f0509b791f8fc433675c58c81f6ea9469d2c91#l212 note that libvchan_client_init calls xs_daemon_open/close by itself, too. Naturally it is posible to change peer_client_init() to not call xs_daemon_open twice, but the title issue itself remains. Regards, Rafal Wojtczuk Principal Researcher Invisible Things Lab, Qubes-os project _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Jun-08 10:41 UTC
Re: [Xen-devel] A small issue with multiple xs_daemon_open
Rafal Wojtczuk writes ("[Xen-devel] A small issue with multiple xs_daemon_open"):> I am experiencing a problem when opening two simultaneous connections (in a > single process) to xenstored via xs_daemon_open(). I would think such > an operation is supported and should work (no static variables in > libxenstore are involved); is it a correct assumption ? If so, the issue > should be fixed.This is definitely supposed to work. But as you say there are no static variables in libxenstore (even in 3.4.3). However, some threading bugs were fixed recently in xen-unstable changesets 21353, 21354 and 21374. Do these look like they might be the cause ? Ian. changeset: 21374:9d53864d7be6 user: Keir Fraser <keir.fraser@citrix.com> date: Thu May 13 12:21:16 2010 +0100 files: tools/xenstore/xs.c description: xenstore: Fix cleanup_pop() definition for some (buggy) pthread.h headers. Signed-off-by: Keir Fraser <keir.fraser@citrix.com> changeset: 21354:9de69d816b11 user: Keir Fraser <keir.fraser@citrix.com> date: Wed May 12 08:49:13 2010 +0100 files: tools/xenstore/xs.c description: xs: avoid pthread_join deadlock in xs_daemon_close Doing a pthread_cancel and join on the reader thread while holding all the request/reply/watch mutexes can deadlock if the thread needs to take any of those mutexes to exit. Kill off the reader thread before taking any mutexes (which should be redundant if we''re single-threaded at that point). Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> changeset: 21353:2dd3141b3e3e user: Keir Fraser <keir.fraser@citrix.com> date: Wed May 12 08:48:14 2010 +0100 files: tools/xenstore/xs.c description: xs: make sure mutexes are cleaned up and memory freed if the read thread is cancelled If the read thread is terminated with pthread cancel, it must make sure all memory is freed and mutexes are unlocked. Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jun-08 11:01 UTC
Re: [Xen-devel] A small issue with multiple xs_daemon_open
On 08/06/2010 11:41, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:> Rafal Wojtczuk writes ("[Xen-devel] A small issue with multiple > xs_daemon_open"): >> I am experiencing a problem when opening two simultaneous connections (in a >> single process) to xenstored via xs_daemon_open(). I would think such >> an operation is supported and should work (no static variables in >> libxenstore are involved); is it a correct assumption ? If so, the issue >> should be fixed. > > This is definitely supposed to work. But as you say there are no > static variables in libxenstore (even in 3.4.3). However, some > threading bugs were fixed recently in xen-unstable changesets 21353, > 21354 and 21374. Do these look like they might be the cause ?Yes, I backported these to 4.0.x but not to 3.4.x since noone had reported problems. I''ve now backported the fixes as xen-3.4-testing:19982. They''ll go into Xen 3.4.4. -- Keir> Ian. > > changeset: 21374:9d53864d7be6 > user: Keir Fraser <keir.fraser@citrix.com> > date: Thu May 13 12:21:16 2010 +0100 > files: tools/xenstore/xs.c > description: > xenstore: Fix cleanup_pop() definition for some (buggy) pthread.h headers. > > Signed-off-by: Keir Fraser <keir.fraser@citrix.com> > > > changeset: 21354:9de69d816b11 > user: Keir Fraser <keir.fraser@citrix.com> > date: Wed May 12 08:49:13 2010 +0100 > files: tools/xenstore/xs.c > description: > xs: avoid pthread_join deadlock in xs_daemon_close > > Doing a pthread_cancel and join on the reader thread while holding all > the request/reply/watch mutexes can deadlock if the thread needs to > take any of those mutexes to exit. Kill off the reader thread before > taking any mutexes (which should be redundant if we''re > single-threaded at that point). > > Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> > > > changeset: 21353:2dd3141b3e3e > user: Keir Fraser <keir.fraser@citrix.com> > date: Wed May 12 08:48:14 2010 +0100 > files: tools/xenstore/xs.c > description: > xs: make sure mutexes are cleaned up and memory freed if the read thread is > cancelled > > If the read thread is terminated with pthread cancel, it must make > sure all memory is freed and mutexes are unlocked. > > Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel