pthread_mutex_lock() are not async-signal safe (ref: http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf) but I still see that it is used extensively in xenstored implementation (eg, xs.c). Moreover, pthread_mutex_lock() suffered a higher performance penalty than other synchronization option. For a one-time effort like domain creation this is ok, but Xenstore is used repeatedly to access data, and therefore performance could potentially be enhanced. Does all these sound logical? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote:> pthread_mutex_lock() are not async-signal safe (ref: > http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf) > but I still see that it is used extensively in xenstored > implementation (eg, xs.c).xs is the client part of the xenstore protocol. it''s not used in xenstored (the daemon).> Moreover, pthread_mutex_lock() suffered a higher performance penalty > than other synchronization option. For a one-time effort like domain > creation this is ok, but Xenstore is used repeatedly to access data, > and therefore performance could potentially be enhanced. > > Does all these sound logical?Not really. xenstored is single threaded and has a file-backed-storage anyway, so most of the time you''re going to wait for that. I expect the pthread_mutex_lock to be insignificant compared to the time waiting for xenstored to reply. Also the xs interface is not meant to be use massively in parallel. Cheers, -- Vincent Hanquez _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2007-Sep-13 15:21 UTC
Re: [Xen-devel] pthread_mutex_lock() and Xenstored
On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote:> pthread_mutex_lock() are not async-signal safe (ref: > http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf) > but I still see that it is used extensively in xenstored implementation > (eg, xs.c).So what ? The pthread_mutex_lock calls aren''t used inside any signal handlers, so the question of whether its async-signal safe is irrelevant> Moreover, pthread_mutex_lock() suffered a higher performance penalty > than other synchronization option. For a one-time effort like domain > creation this is ok, but Xenstore is used repeatedly to access data, > and therefore performance could potentially be enhanced.That presentation giving figures comparing pthread_mutex_lock with other primitives is useless. It doesn''t say what OS / version it was tested on, or what those figures are measuring. Linux NPTL (2.6.x) threading has radically different (better) pthread_mutex performance characteristics that the older LinuxThreads impl (2.4.x) for example, and that''s ignoring any other non-Linux OS'' impl of pthreads. The performance characteristics of a contended mutex lock, vs an uncontended lock are also completely different & not considered in those figures. Finally there''s no performance profile data to suggest that mutex locking is even remotely relevant to xenstore performance. There is however data to show that the huge amount of I/O xenstored does hurts performance. By all means consider how to improve the performance of xenstored, but do so based on *measured* facts about where the current problems are, rather than bullet points in a presentation unrelated to Xen. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thank you for your feedback. I am still learning these stuff. On 9/13/07, Daniel P. Berrange <berrange@redhat.com> wrote:> On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote: > > pthread_mutex_lock() are not async-signal safe (ref: > > http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf) > > but I still see that it is used extensively in xenstored implementation > > (eg, xs.c). > > So what ? The pthread_mutex_lock calls aren''t used inside any signal > handlers, so the question of whether its async-signal safe is irrelevant > > > Moreover, pthread_mutex_lock() suffered a higher performance penalty > > than other synchronization option. For a one-time effort like domain > > creation this is ok, but Xenstore is used repeatedly to access data, > > and therefore performance could potentially be enhanced. > > That presentation giving figures comparing pthread_mutex_lock with other > primitives is useless. It doesn''t say what OS / version it was tested on, > or what those figures are measuring. Linux NPTL (2.6.x) threading has radically > different (better) pthread_mutex performance characteristics that the older > LinuxThreads impl (2.4.x) for example, and that''s ignoring any other non-Linux > OS'' impl of pthreads. The performance characteristics of a contended mutex lock, > vs an uncontended lock are also completely different & not considered in those > figures. >Contended locks? I will looked further into these, thanks.> Finally there''s no performance profile data to suggest that mutex locking > is even remotely relevant to xenstore performance. There is however data to > show that the huge amount of I/O xenstored does hurts performance. >And so, is there any particular reasons for Xenstored''s slow performance? Any area needs improvement? I looked into Xenstored''s implementation, and found that it is using TDB (trivial database, which is also used by SAMBA) as its backbone. And inside the tdb.c are a lot of locking statements. What is TDB? Then I read this article: http://www.coda.cs.cmu.edu/maillists/codadev/codadev-2003/0202.html (paying attention to what he said about Dan Bernstein''s CDB) and to refer to what you said - Xenstored is single threaded, so we have a "single reader/single writer situation" here, and so is a database structure like tdb really needed? What I think is we need to redesign these stuff, based on: * identifying the characteristic the relative frequency of read/write access. * identifying concurrent readers, and writer - its different model * does databases need to be shareable across heterogeneous systems by a simple file copy across the network like SAMBA? * Can''t use thread libraries as they conflict with LWP. * identifying the minimal amount of mapping operation needed, and possibly its corresponding hashing structures. * and finally, to come up with the simplest design that meet all the requirements. Can Dan''s CDB be a possible alternative candidate for our database design? Please enlighten me :-).> By all means consider how to improve the performance of xenstored, but do > so based on *measured* facts about where the current problems are, rather > than bullet points in a presentation unrelated to Xen. >Noted, thanks :-). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, Sep 15, 2007 at 02:01:12PM +0800, Peter Teoh wrote:> * does databases need to be shareable across heterogeneous systems > by a simple file copy across the network like SAMBA?Most db files are not interchangable. And the xenstore db is usually useless on another system. You want to define your own transfer format if you want to do this and not rely on the db to support that.> * Can''t use thread libraries as they conflict with LWP."Library for WWW in Perl"? Even wikipedia does not list a better match. pthread is LSB and SUSv3. And with a proper db you can use fork if necessary.> * identifying the minimal amount of mapping operation needed, and > possibly its corresponding hashing structures. > * and finally, to come up with the simplest design that meet all > the requirements. > > Can Dan''s CDB be a possible alternative candidate for our database > design? Please enlighten me :-).Isn''t CDB also read-only, aka needs to be rewritten on each write? xenstored with tdb does this. Usually I would say, bdb is a good choice for this usage. Supports transactions, multiple reader. It is used by many things with large writing frequency. Bastian -- No one wants war. -- Kirk, "Errand of Mercy", stardate 3201.7 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2007-Sep-15 13:50 UTC
Re: [Xen-devel] pthread_mutex_lock() and Xenstored
On Sat, Sep 15, 2007 at 02:01:12PM +0800, Peter Teoh wrote:> Thank you for your feedback. I am still learning these stuff. > > On 9/13/07, Daniel P. Berrange <berrange@redhat.com> wrote: > > On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote: > > > pthread_mutex_lock() are not async-signal safe (ref: > > > http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf) > > > but I still see that it is used extensively in xenstored implementation > > > (eg, xs.c). > > > > So what ? The pthread_mutex_lock calls aren''t used inside any signal > > handlers, so the question of whether its async-signal safe is irrelevant > > > > > Moreover, pthread_mutex_lock() suffered a higher performance penalty > > > than other synchronization option. For a one-time effort like domain > > > creation this is ok, but Xenstore is used repeatedly to access data, > > > and therefore performance could potentially be enhanced. > > > > That presentation giving figures comparing pthread_mutex_lock with other > > primitives is useless. It doesn''t say what OS / version it was tested on, > > or what those figures are measuring. Linux NPTL (2.6.x) threading has radically > > different (better) pthread_mutex performance characteristics that the older > > LinuxThreads impl (2.4.x) for example, and that''s ignoring any other non-Linux > > OS'' impl of pthreads. The performance characteristics of a contended mutex lock, > > vs an uncontended lock are also completely different & not considered in those > > figures. > > > > Contended locks? I will looked further into these, thanks. > > > Finally there''s no performance profile data to suggest that mutex locking > > is even remotely relevant to xenstore performance. There is however data to > > show that the huge amount of I/O xenstored does hurts performance. > > > > And so, is there any particular reasons for Xenstored''s slow > performance? Any area needs improvement?The primary problem is the amount of I/O it does per transaction, and then indirectly, the huge number of transactions XenD makes. http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00487.html I repeated those tests against Xen 3.1 a few months back and the characteristics haven''t changed all that much. Running further OProfile tests on a debug build would confirm this.> (paying attention to what he said about Dan Bernstein''s CDB) and to > refer to what you said - Xenstored is single threaded, so we have a > "single reader/single writer situation" here, and so is a database > structure like tdb really needed? > > What I think is we need to redesign these stuff, based on:Since it is not possible to ever actually restart xenstored, the easy answer is to not use any database on disk at all. Simply keep the entire information set in RAM. None of the data is keeps per VM needs to persist once the VM shuts down - all the important state is maintained by XenD which does persist. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 9/15/07, Bastian Blank <bastian@waldi.eu.org> wrote:> On Sat, Sep 15, 2007 at 02:01:12PM +0800, Peter Teoh wrote: > > * does databases need to be shareable across heterogeneous systems > > by a simple file copy across the network like SAMBA? > > Most db files are not interchangable. And the xenstore db is usually > useless on another system. You want to define your own transfer format > if you want to do this and not rely on the db to support that. > > > * Can''t use thread libraries as they conflict with LWP. > > "Library for WWW in Perl"? Even wikipedia does not list a better match. > pthread is LSB and SUSv3. And with a proper db you can use fork if > necessary. > > > * identifying the minimal amount of mapping operation needed, and > > possibly its corresponding hashing structures. > > * and finally, to come up with the simplest design that meet all > > the requirements. > > > > Can Dan''s CDB be a possible alternative candidate for our database > > design? Please enlighten me :-). > > Isn''t CDB also read-only, aka needs to be rewritten on each write? > xenstored with tdb does this. >Have not the time to look into CDB yet. More important is to understand in depth what actually is the requirements of Xenstored - its basic need for a transaction concept? ACID properties? FIFO queue properties? Can we consider other alternatives like lock-free database? For example: http://en.wikipedia.org/wiki/Lock-free_and_wait-free_algorithms http://www.grame.fr/pub/LockFree.pdf And then there is a concurrency vs performance tradeoffs issue as well (eg, row-level locking in database usually implies higher overheads in processing). So it will be good if lock-free algo are favored. Another possibility is also batching of transactions, just like batching of hypercall in Xen, so as to cut down the overheads of starting and closing the transaction.> Usually I would say, bdb is a good choice for this usage. Supports > transactions, multiple reader. It is used by many things with large > writing frequency. >Yes, I agree, bDB is used in MySQL etc. On 9/15/07, Daniel P. Berrange <berrange@redhat.com> wrote:> I repeated those tests against Xen 3.1 a few months back and the characteristics > haven''t changed all that much. Running further OProfile tests on a debug build > would confirm this. >Any possibility of accessing these data?> Since it is not possible to ever actually restart xenstored, the easy answer > is to not use any database on disk at all. Simply keep the entire information > set in RAM. None of the data is keeps per VM needs to persist once the VM > shuts down - all the important state is maintained by XenD which does persist. >Persistency is traded off here. I thought if traditional filesystem can do without the database concept (eg, rollback concept when non-integrity of data is detected), why not here? Any comments? Apologized if I am wrong in any area. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, Sep 15, 2007 at 02:50:03PM +0100, Daniel P. Berrange wrote:> Since it is not possible to ever actually restart xenstored, the easy answer > is to not use any database on disk at all. Simply keep the entire information > set in RAM. None of the data is keeps per VM needs to persist once the VM > shuts down - all the important state is maintained by XenD which does persist.This has to be fixed at some point though. And part of fixing it will involve persisting the watches (that''s actually the easy bit...) john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel