I''m sure a lot of you know XenSocket (XVMSocket), if not, it''s a Loadable Kernel Module with a socket interface that sends/receives to/from a shared page of memory mapped between two domains. It''s quite a nice idea and brings data throughput close to UNIX sockets. I made some changes and made it compatible with Xen 3.2 and a newer Linux kernel (I''m testing it with 2.6.25.20) however I''m noticing some odd behaviour occasionally. In it, there are atomic_t variables in a descriptor inside a shared memory page. When a domain sends or receives from the socket, it will use one of the atomic_ operations to update the amount of new data available. The problem I''m noticing is that when two domains are concurrently updating this variable, the value becomes inconsistent leading me to believe that the atomic_ operations might not guarantee atomicity between domains. I haven''t been able to repeat any behaviour like this when I limit the two guests to the same physical CPU and eliminate parallelism. Would anyone be able fill me in if I''m missing something? Are the atomic_ operations 100% foolproof for this sort of thing? If not, what would be the best approach to use instead? Kind regards Tim Hayes _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The Linux atomic.h operations will be atomic even across domains. -- Keir On 26/03/2009 00:17, "Timothy Hayes" <hayesti@tcd.ie> wrote:> I''m sure a lot of you know XenSocket (XVMSocket), if not, it''s a Loadable > Kernel Module with a socket interface that sends/receives to/from a shared > page of memory mapped between two domains. It''s quite a nice idea and brings > data throughput close to UNIX sockets. I made some changes and made it > compatible with Xen 3.2 and a newer Linux kernel (I''m testing it with > 2.6.25.20) however I''m noticing some odd behaviour occasionally. In it, there > are atomic_t variables in a descriptor inside a shared memory page. When a > domain sends or receives from the socket, it will use one of the atomic_ > operations to update the amount of new data available. The problem I''m > noticing is that when two domains are concurrently updating this variable, the > value becomes inconsistent leading me to believe that the atomic_ operations > might not guarantee atomicity between domains. I haven''t been able to repeat > any behaviour like this when I limit the two guests to the same physical CPU > and eliminate parallelism. > > Would anyone be able fill me in if I''m missing something? Are the atomic_ > operations 100% foolproof for this sort of thing? If not, what would be the > best approach to use instead?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> The Linux atomic.h operations will be atomic even across domains.... unless CONFIG_SMP was not defined in the domU kernel build. Juergen> On 26/03/2009 00:17, "Timothy Hayes" <hayesti@tcd.ie> wrote: > > I''m sure a lot of you know XenSocket (XVMSocket), if not, it''s a > Loadable Kernel Module with a socket interface that sends/receives > to/from a shared page of memory mapped between two domains. It''s > quite a nice idea and brings data throughput close to UNIX sockets. > I made some changes and made it compatible with Xen 3.2 and a newer > Linux kernel (I''m testing it with 2.6.25.20) however I''m noticing > some odd behaviour occasionally. In it, there are atomic_t variables > in a descriptor inside a shared memory page. When a domain sends or > receives from the socket, it will use one of the atomic_ operations > to update the amount of new data available. The problem I''m noticing > is that when two domains are concurrently updating this variable, > the value becomes inconsistent leading me to believe that the > atomic_ operations might not guarantee atomicity between domains. I > haven''t been able to repeat any behaviour like this when I limit the > two guests to the same physical CPU and eliminate parallelism. > > Would anyone be able fill me in if I''m missing something? Are the > atomic_ operations 100% foolproof for this sort of thing? If not, > what would be the best approach to use instead? > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Juergen Gross Principal Developer IP SW OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/03/2009 07:34, "Juergen Gross" <juergen.gross@fujitsu-siemens.com> wrote:> Keir Fraser wrote: >> The Linux atomic.h operations will be atomic even across domains. > > ... unless CONFIG_SMP was not defined in the domU kernel build.Yes, good point. Actually I assumed the OP was copying atomic.h for his own use, but he probably isn''t actually. And either way the CONFIG_SMP ifdef''ery could bite. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
My domU kernels are configured with: CONFIG_SMP=y CONFIG_X86_FIND_SMP_CONFIG=y CONFIG_X86_MPPARSE=y # CONFIG_X86_PC is not set CONFIG_X86_XEN=y The problem occurs when I limit the VCPUs in each guest to 1, presumably they get scheduled into unique physical CPUs in parallel. Is there anything else I should be aware of? I''m quite optimistic that the XenSocket logic is sound, the only thing I can attribute this to is a lack of atomicity. Tim 2009/3/26 Keir Fraser <keir.fraser@eu.citrix.com>> On 26/03/2009 07:34, "Juergen Gross" <juergen.gross@fujitsu-siemens.com> > wrote: > > > Keir Fraser wrote: > >> The Linux atomic.h operations will be atomic even across domains. > > > > ... unless CONFIG_SMP was not defined in the domU kernel build. > > Yes, good point. Actually I assumed the OP was copying atomic.h for his own > use, but he probably isn''t actually. And either way the CONFIG_SMP > ifdef''ery > could bite. > > -- Keir > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The problem is that we dynamically remove LOCK_PREFIX when the guest has only one vcpu. You need your own version of atomic.h in which you do not use LOCK_PREFIX but use lock¹ directly instead. See include/asm-i386/mach-xen/asm/sync_bitops.h for example, which is our guaranteed inter-domain safe version of bitops.h. -- Keir On 27/03/2009 17:22, "Timothy Hayes" <hayesti@tcd.ie> wrote:> CONFIG_SMP=y > CONFIG_X86_FIND_SMP_CONFIG=y > CONFIG_X86_MPPARSE=y > # CONFIG_X86_PC is not set > CONFIG_X86_XEN=y > > The problem occurs when I limit the VCPUs in each guest to 1, presumably they > get scheduled into unique physical CPUs in parallel. Is there anything else I > should be aware of? I''m quite optimistic that the XenSocket logic is sound, > the only thing I can attribute this to is a lack of atomicity._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ah, that did the trick! I couldn''t really see why that was happening since CONFIG_SMP was defined in my environment so I had no reason to suspect that the LOCK_PREFIX was being removed. In any case, it''s running without hiccups now. Cheers Tim 2009/3/27 Keir Fraser <keir.fraser@eu.citrix.com>> The problem is that we dynamically remove LOCK_PREFIX when the guest has > only one vcpu. You need your own version of atomic.h in which you do not > use > LOCK_PREFIX but use Œlock¹ directly instead. See > include/asm-i386/mach-xen/asm/sync_bitops.h for example, which is our > guaranteed inter-domain safe version of bitops.h. > > -- Keir > > On 27/03/2009 17:22, "Timothy Hayes" <hayesti@tcd.ie> wrote: > > > CONFIG_SMP=y > > CONFIG_X86_FIND_SMP_CONFIG=y > > CONFIG_X86_MPPARSE=y > > # CONFIG_X86_PC is not set > > CONFIG_X86_XEN=y > > > > The problem occurs when I limit the VCPUs in each guest to 1, presumably > they > > get scheduled into unique physical CPUs in parallel. Is there anything > else I > > should be aware of? I''m quite optimistic that the XenSocket logic is > sound, > > the only thing I can attribute this to is a lack of atomicity. > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel