Linux Virtualization - Dec 2015 - [PATCH] virtio_ring: use smp_store_mb

On Thu, Dec 17, 2015 at 03:02:12PM +0100, Peter Zijlstra
wrote:
> On Thu, Dec 17, 2015 at 03:26:29PM +0200, Michael S. Tsirkin wrote:
> > > Note that virtio_mb() is weirdly inconsistent with
virtio_[rw]mb() in
> > > that they use dma_* ops for weak_barriers, while virtio_mb() uses
> > > smp_mb().
> > 
> > It's a hack really. I think I'll clean it up a bit to
> > make it more consistent.
> > 
> > To simplify things, you may consider things before
> > the optimization brought in by
> > 	commit 9e1a27ea42691429e31f158cce6fc61bc79bb2e9
> > 	Author: Alexander Duyck <alexander.h.duyck at redhat.com>
> > 	Date:   Mon Apr 13 21:03:49 2015 +0930
> > 
> > 	    virtio_ring: Update weak barriers to use dma_wmb/rmb
> 
> That commit doesn't make any sense. dma_*mb() explicitly does _NOT_
> cover the smp_*mb() part.
> 
> Again, look at the ARM definitions, the smp_*mb() primitives use the
> inner coherence stuff, while the dma_*mb() primitives use the outer
> coherent stuff.
Does outer coherent imply inner coherent?
> the *mb() primitives cover both.

On Thu, Dec 17, 2015 at 04:34:57PM +0200, Michael S. Tsirkin
wrote:
> On Thu, Dec 17, 2015 at 03:02:12PM +0100, Peter Zijlstra wrote:
> > > 	commit 9e1a27ea42691429e31f158cce6fc61bc79bb2e9
> > > 	Author: Alexander Duyck <alexander.h.duyck at redhat.com>
> > > 	Date:   Mon Apr 13 21:03:49 2015 +0930
> > > 
> > > 	    virtio_ring: Update weak barriers to use dma_wmb/rmb
> > 
> > That commit doesn't make any sense. dma_*mb() explicitly does
_NOT_
> > cover the smp_*mb() part.
> > 
> > Again, look at the ARM definitions, the smp_*mb() primitives use the
> > inner coherence stuff, while the dma_*mb() primitives use the outer
> > coherent stuff.
> 
> Does outer coherent imply inner coherent?
> 
> > the *mb() primitives cover both.
I do not think so, but lets add Will, he dreams this stuff.

On Thu, Dec 17, 2015 at 04:09:17PM +0100, Peter Zijlstra
wrote:
> On Thu, Dec 17, 2015 at 04:34:57PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Dec 17, 2015 at 03:02:12PM +0100, Peter Zijlstra wrote:
> 
> > > > 	commit 9e1a27ea42691429e31f158cce6fc61bc79bb2e9
> > > > 	Author: Alexander Duyck <alexander.h.duyck at
redhat.com>
> > > > 	Date:   Mon Apr 13 21:03:49 2015 +0930
> > > > 
> > > > 	    virtio_ring: Update weak barriers to use dma_wmb/rmb
> > > 
> > > That commit doesn't make any sense. dma_*mb() explicitly does
_NOT_
> > > cover the smp_*mb() part.
> > > 
> > > Again, look at the ARM definitions, the smp_*mb() primitives use
the
> > > inner coherence stuff, while the dma_*mb() primitives use the
outer
> > > coherent stuff.
> > 
> > Does outer coherent imply inner coherent?
> > 
> > > the *mb() primitives cover both.
> 
> I do not think so, but lets add Will, he dreams this stuff.
Right, and I don't sleep well these days.

Anyway, the outer-shareable domain (osh) is a superset of the
inner-shareable domain (ish). The inner-shareable domain contains the
CPUs and any peripherals that you and I would call "cache coherent".
The
outer-shareable domain extends this to cover a strange set of "less cache
coherent" devices, which we would just call "not cache coherent"
for the
purposes of Linux. Normal, non-cacheable memory (i.e. the memory type we
use for non-coherent DMA buffers) is outer-shareable.

Since the barrier macros don't know if the device is coherent or not, we
use the stronger semantics of outer-shareable.

I've not been following the thread, but I reckon we could add dma_mb()
(as dmb(osh) on arm), then use that to build dma_load_acquire and
dma_store_release accessors. On arm64, we could probably use the
acquire/release instructions directly, since they inherit the shareability
domain of the address (which has the nice property of being inner-shareable
for coherent devices).

The massive pain with adding new accessors is defining the semantics.
memory-barriers.txt is already lacking for the CPU side, and we're
struggling to express the kind of transitivity guarantees we provide
today, let alone with new primitives :(

Will