Xen devel - May 2011 - One shared memory read/write issue between 2 different domains

Dear xen-devel team,

 

Recently I encountered one technical issue on Xen platform. I want to request
your help.

 

[Problem Description]

I code one ring buffer by myself for 2 domains to transmit data, one domain is
producer, and the other is consumer. I do not use the ring buffer defined in
“include/xen/interface/io/ring.h”. Just like the ring buffer provided by xen, I
also use 2 pointers for producer and consumer, separately. To ensure memory
access order is consistent, I use memory barrier between memory store/load
operation and pointer update operation.

 

The transmitted data contains 2 parts, one is header, and the other is payload.
Each time, one domain transmits the header first, and then the payload is sent
out.

 

Each time I can get right header at the consumer side, however, sometimes, the
data I get at the consumer side contains part of invalid information.

 

[Develop Environment]

Xen-4.0.1 + Linux-2.6.24-29-xen

 

Could you please give me some suggestion on this issue?

 

[Other Questions]

1.      Is it reasonable to use spin lock to protect shared memory between 2
different domains?

2.      Previously I use one atomic variable to do the synchronization between
producer and consumer, however, sometimes I cannot get correct header at the
consumer side. Is it one known-issue?

 

I am looking forward to your letters. Thanks in advance.

 

Regards,
Chunjie

===========================================
 

阴风催急雨，木叶舞萧萧。

闲看天色变，忧心朋宾来。

 


TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential and
may be subject to copyright or other intellectual property protection. If you
are not the intended recipient, you are not authorized to use or disclose this
information, and we request that you notify us by reply mail or telephone and
delete the original message from your mail system.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

At 17:39 +0800 on 25 May (1306345173), chunjie_zhu@trendmicro.com.cn
wrote:
> I code one ring buffer by myself for 2 domains to transmit data, one
> domain is producer, and the other is consumer. I do not use the ring
> buffer defined in ??include/xen/interface/io/ring.h??.
Why not?  That one has had a lot of testing and debugging.
> Just like the ring buffer provided by xen, I also use 2 pointers for
> producer and consumer, separately. To ensure memory access order is
> consistent, I use memory barrier between memory store/load operation
> and pointer update operation.
> 
> The transmitted data contains 2 parts, one is header, and the other is
> payload. Each time, one domain transmits the header first, and then
> the payload is sent out.
> 
> Each time I can get right header at the consumer side, however,
> sometimes, the data I get at the consumer side contains part of
> invalid information.
> 
> Could you please give me some suggestion on this issue?
Hard to say from just this description, but probably you''re missing
either a hardware memory barrier or a compiler one.   Either that or
something in the rest of your code is corrupting the rings. 

Try comparing your code very carefully to the existing, working ring
buffer code.
> [Other Questions]
> 
> 1.  Is it reasonable to use spin lock to protect shared memory between
> 2 different domains?
It''s probably not a great idea.  You have to worry about what happens
if
one of the domains crashes while holding the lock, for example.
> 2.  Previously I use one atomic variable to do the synchronization
> between producer and consumer, however, sometimes I cannot get correct
> header at the consumer side. Is it one known-issue?
Again, you''re probably missing a compiler barrier.  AIUI, updates to
atomic variables are only guaranteed not to be reordered WRT other write
of the _same_ variable.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim，

Thanks very much for your answer. I will follow your suggestion and try again.

Regards,
Chunjie
===========================================一盘棋不同色子，
一样人各样心肠。

-----Original Message-----
From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
Sent: 2011年6月6日 17:10
To: Chunjie Zhu (RD-CN)
Cc: xen-devel@lists.xensource.com; Jianxin Guo (RD-CN)
Subject: Re: [Xen-devel] One shared memory read/write issue between 2 different
domains

Hi, 

At 17:39 +0800 on 25 May (1306345173), chunjie_zhu@trendmicro.com.cn
wrote:
> I code one ring buffer by myself for 2 domains to transmit data, one 
> domain is producer, and the other is consumer. I do not use the ring 
> buffer defined in ??include/xen/interface/io/ring.h??.
Why not?  That one has had a lot of testing and debugging.
> Just like the ring buffer provided by xen, I also use 2 pointers for 
> producer and consumer, separately. To ensure memory access order is 
> consistent, I use memory barrier between memory store/load operation 
> and pointer update operation.
> 
> The transmitted data contains 2 parts, one is header, and the other is 
> payload. Each time, one domain transmits the header first, and then 
> the payload is sent out.
> 
> Each time I can get right header at the consumer side, however, 
> sometimes, the data I get at the consumer side contains part of 
> invalid information.
> 
> Could you please give me some suggestion on this issue?
Hard to say from just this description, but probably you''re missing
either a hardware memory barrier or a compiler one.   Either that or
something in the rest of your code is corrupting the rings. 

Try comparing your code very carefully to the existing, working ring buffer
code.
> [Other Questions]
> 
> 1.  Is it reasonable to use spin lock to protect shared memory between
> 2 different domains?
It''s probably not a great idea.  You have to worry about what happens
if one of the domains crashes while holding the lock, for example.
> 2.  Previously I use one atomic variable to do the synchronization 
> between producer and consumer, however, sometimes I cannot get correct 
> header at the consumer side. Is it one known-issue?
Again, you''re probably missing a compiler barrier.  AIUI, updates to
atomic variables are only guaranteed not to be reordered WRT other write of the
_same_ variable.

Tim.

--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team Citrix Systems UK Ltd.  (Company
#02937203, SL9 0BG)
TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential and
may be subject to copyright or other intellectual property protection. If you
are not the intended recipient, you are not authorized to use or disclose this
information, and we request that you notify us by reply mail or telephone and
delete the original message from your mail system.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel