Cornelia Huck
2019-May-13 09:41 UTC
[PATCH 06/10] s390/cio: add basic protected virtualization support
On Fri, 26 Apr 2019 20:32:41 +0200 Halil Pasic <pasic at linux.ibm.com> wrote:> As virtio-ccw devices are channel devices, we need to use the dma area > for any communication with the hypervisor. > > This patch addresses the most basic stuff (mostly what is required for > virtio-ccw), and does take care of QDIO or any devices."does not take care of QDIO", surely? (Also, what does "any devices" mean? Do you mean "every arbitrary device", perhaps?)> > An interesting side effect is that virtio structures are now going to > get allocated in 31 bit addressable storage.Hm...> > Signed-off-by: Halil Pasic <pasic at linux.ibm.com> > --- > arch/s390/include/asm/ccwdev.h | 4 +++ > drivers/s390/cio/ccwreq.c | 8 ++--- > drivers/s390/cio/device.c | 65 +++++++++++++++++++++++++++++++++------- > drivers/s390/cio/device_fsm.c | 40 ++++++++++++------------- > drivers/s390/cio/device_id.c | 18 +++++------ > drivers/s390/cio/device_ops.c | 21 +++++++++++-- > drivers/s390/cio/device_pgid.c | 20 ++++++------- > drivers/s390/cio/device_status.c | 24 +++++++-------- > drivers/s390/cio/io_sch.h | 21 +++++++++---- > drivers/s390/virtio/virtio_ccw.c | 10 ------- > 10 files changed, 148 insertions(+), 83 deletions(-)(...)> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c > index 6d989c360f38..bb7a92316fc8 100644 > --- a/drivers/s390/virtio/virtio_ccw.c > +++ b/drivers/s390/virtio/virtio_ccw.c > @@ -66,7 +66,6 @@ struct virtio_ccw_device { > bool device_lost; > unsigned int config_ready; > void *airq_info; > - u64 dma_mask; > }; > > struct vq_info_block_legacy { > @@ -1255,16 +1254,7 @@ static int virtio_ccw_online(struct ccw_device *cdev) > ret = -ENOMEM; > goto out_free; > } > - > vcdev->vdev.dev.parent = &cdev->dev; > - cdev->dev.dma_mask = &vcdev->dma_mask; > - /* we are fine with common virtio infrastructure using 64 bit DMA */ > - ret = dma_set_mask_and_coherent(&cdev->dev, DMA_BIT_MASK(64)); > - if (ret) { > - dev_warn(&cdev->dev, "Failed to enable 64-bit DMA.\n"); > - goto out_free; > - }This means that vring structures now need to fit into 31 bits as well, I think? Is there any way to reserve the 31 bit restriction for channel subsystem structures and keep vring in the full 64 bit range? (Or am I fundamentally misunderstanding something?)> - > vcdev->config_block = kzalloc(sizeof(*vcdev->config_block), > GFP_DMA | GFP_KERNEL); > if (!vcdev->config_block) {
Jason J. Herne
2019-May-14 14:47 UTC
[PATCH 06/10] s390/cio: add basic protected virtualization support
On 5/13/19 5:41 AM, Cornelia Huck wrote:> On Fri, 26 Apr 2019 20:32:41 +0200 > Halil Pasic <pasic at linux.ibm.com> wrote: > >> As virtio-ccw devices are channel devices, we need to use the dma area >> for any communication with the hypervisor. >> >> This patch addresses the most basic stuff (mostly what is required for >> virtio-ccw), and does take care of QDIO or any devices. > > "does not take care of QDIO", surely? (Also, what does "any devices" > mean? Do you mean "every arbitrary device", perhaps?) > >> >> An interesting side effect is that virtio structures are now going to >> get allocated in 31 bit addressable storage. > > Hm... > >> >> Signed-off-by: Halil Pasic <pasic at linux.ibm.com> >> --- >> arch/s390/include/asm/ccwdev.h | 4 +++ >> drivers/s390/cio/ccwreq.c | 8 ++--- >> drivers/s390/cio/device.c | 65 +++++++++++++++++++++++++++++++++------- >> drivers/s390/cio/device_fsm.c | 40 ++++++++++++------------- >> drivers/s390/cio/device_id.c | 18 +++++------ >> drivers/s390/cio/device_ops.c | 21 +++++++++++-- >> drivers/s390/cio/device_pgid.c | 20 ++++++------- >> drivers/s390/cio/device_status.c | 24 +++++++-------- >> drivers/s390/cio/io_sch.h | 21 +++++++++---- >> drivers/s390/virtio/virtio_ccw.c | 10 ------- >> 10 files changed, 148 insertions(+), 83 deletions(-) > > (...) > >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c >> index 6d989c360f38..bb7a92316fc8 100644 >> --- a/drivers/s390/virtio/virtio_ccw.c >> +++ b/drivers/s390/virtio/virtio_ccw.c >> @@ -66,7 +66,6 @@ struct virtio_ccw_device { >> bool device_lost; >> unsigned int config_ready; >> void *airq_info; >> - u64 dma_mask; >> }; >> >> struct vq_info_block_legacy { >> @@ -1255,16 +1254,7 @@ static int virtio_ccw_online(struct ccw_device *cdev) >> ret = -ENOMEM; >> goto out_free; >> } >> - >> vcdev->vdev.dev.parent = &cdev->dev; >> - cdev->dev.dma_mask = &vcdev->dma_mask; >> - /* we are fine with common virtio infrastructure using 64 bit DMA */ >> - ret = dma_set_mask_and_coherent(&cdev->dev, DMA_BIT_MASK(64)); >> - if (ret) { >> - dev_warn(&cdev->dev, "Failed to enable 64-bit DMA.\n"); >> - goto out_free; >> - } > > This means that vring structures now need to fit into 31 bits as well, > I think? Is there any way to reserve the 31 bit restriction for channel > subsystem structures and keep vring in the full 64 bit range? (Or am I > fundamentally misunderstanding something?) >I hope I've understood everything... I'm new to virtio. But from what I'm understanding, the vring structure (a.k.a. the VirtQueue) needs to be accessed and modified by both host and guest. Therefore the page(s) holding that data need to be marked shared if using protected virtualization. This patch set makes use of DMA pages by way of swiotlb (always below 32-bit line right?) for shared memory. Therefore, a side effect is that all shared memory, including VirtQueue data will be in the DMA zone and in 32-bit memory. I don't see any restrictions on sharing pages above the 32-bit line. So it seems possible. I'm not sure how much more work it would be. I wonder if Halil has considered this? Are we worried that virtio data structures are going to be a burden on the 31-bit address space? -- -- Jason J. Herne (jjherne at linux.ibm.com)
Halil Pasic
2019-May-15 20:51 UTC
[PATCH 06/10] s390/cio: add basic protected virtualization support
On Mon, 13 May 2019 11:41:36 +0200 Cornelia Huck <cohuck at redhat.com> wrote:> On Fri, 26 Apr 2019 20:32:41 +0200 > Halil Pasic <pasic at linux.ibm.com> wrote: > > > As virtio-ccw devices are channel devices, we need to use the dma area > > for any communication with the hypervisor. > > > > This patch addresses the most basic stuff (mostly what is required for > > virtio-ccw), and does take care of QDIO or any devices. > > "does not take care of QDIO", surely?I did not bother making the QDIO library code use dma memory for anything that is conceptually dma memory. AFAIK QDIO is out of scope for prot virt for now. If one were to do some emulated qdio with prot virt guests, one wound need to make a bunch of things shared.> (Also, what does "any devices" > mean? Do you mean "every arbitrary device", perhaps?)What I mean is: this patch takes care of the core stuff, but any particular device is likely to have to do more -- that is it ain't all the cio devices support prot virt with this patch. For example virtio-ccw needs to make sure that the ccws constituting the channel programs, as well as the data pointed by the ccws is shared. If one would want to make vfio-ccw DASD pass-through work under prot virt, one would need to make sure, that everything that needs to be shared is shared (data buffers, channel programs). Does is clarify things?> > > > > An interesting side effect is that virtio structures are now going to > > get allocated in 31 bit addressable storage. > > Hm... > > > > > Signed-off-by: Halil Pasic <pasic at linux.ibm.com> > > --- > > arch/s390/include/asm/ccwdev.h | 4 +++ > > drivers/s390/cio/ccwreq.c | 8 ++--- > > drivers/s390/cio/device.c | 65 +++++++++++++++++++++++++++++++++------- > > drivers/s390/cio/device_fsm.c | 40 ++++++++++++------------- > > drivers/s390/cio/device_id.c | 18 +++++------ > > drivers/s390/cio/device_ops.c | 21 +++++++++++-- > > drivers/s390/cio/device_pgid.c | 20 ++++++------- > > drivers/s390/cio/device_status.c | 24 +++++++-------- > > drivers/s390/cio/io_sch.h | 21 +++++++++---- > > drivers/s390/virtio/virtio_ccw.c | 10 ------- > > 10 files changed, 148 insertions(+), 83 deletions(-) > > (...) > > > diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c > > index 6d989c360f38..bb7a92316fc8 100644 > > --- a/drivers/s390/virtio/virtio_ccw.c > > +++ b/drivers/s390/virtio/virtio_ccw.c > > @@ -66,7 +66,6 @@ struct virtio_ccw_device { > > bool device_lost; > > unsigned int config_ready; > > void *airq_info; > > - u64 dma_mask; > > }; > > > > struct vq_info_block_legacy { > > @@ -1255,16 +1254,7 @@ static int virtio_ccw_online(struct ccw_device *cdev) > > ret = -ENOMEM; > > goto out_free; > > } > > - > > vcdev->vdev.dev.parent = &cdev->dev; > > - cdev->dev.dma_mask = &vcdev->dma_mask; > > - /* we are fine with common virtio infrastructure using 64 bit DMA */ > > - ret = dma_set_mask_and_coherent(&cdev->dev, DMA_BIT_MASK(64)); > > - if (ret) { > > - dev_warn(&cdev->dev, "Failed to enable 64-bit DMA.\n"); > > - goto out_free; > > - } > > This means that vring structures now need to fit into 31 bits as well, > I think?Nod.> Is there any way to reserve the 31 bit restriction for channel > subsystem structures and keep vring in the full 64 bit range? (Or am I > fundamentally misunderstanding something?) >At the root of this problem is that the DMA API basically says devices may have addressing limitations expressed by the dma_mask, while our addressing limitations are not coming from the device but from the IO arch: e.g. orb.cpa and ccw.cda are 31 bit addresses. In our case it depends on how and for what is the device going to use the memory (e.g. buffers addressed by MIDA vs IDA vs direct). Virtio uses the DMA properties of the parent, that is in our case the struct device embedded in struct ccw_device. The previous version (RFC) used to allocate all the cio DMA stuff from this global cio_dma_pool using the css0.dev for the DMA API interactions. And we set *css0.dev.dma_mask == DMA_BIT_MASK(31) so e.g. the allocated ccws are 31 bit addressable. But I was asked to change this so that when I allocate DMA memory for a channel program of particular ccw device, a struct device of that ccw device is used as the first argument of dma_alloc_coherent(). Considering void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs) { const struct dma_map_ops *ops = get_dma_ops(dev); void *cpu_addr; WARN_ON_ONCE(dev && !dev->coherent_dma_mask); if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr)) return cpu_addr; /* let the implementation decide on the zone to allocate from: */ flag &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM); that is the GFP flags dropped that implies that we really want cdev->dev restricted to 31 bit addressable memory because we can't tell (with the current common DMA code) hey but this piece of DMA mem you are abot to allocate for me must be 31 bit addressable (using GFP_DMA as we usually do). So, as described in the commit message, the vring stuff being forced into ZONE_DMA is an unfortunate consequence of this all. A side note: making the subchannel device 'own' the DMA stuff of a ccw device (something that was discussed in the RFC thread) is tricky because the ccw device may outlive the subchannel (all that orphan stuff). So the answer is: it is technically possible (e.g. see RFC) but it comes at a price, and I see no obviously brilliant solution. Regards, Halil> > - > > vcdev->config_block = kzalloc(sizeof(*vcdev->config_block), > > GFP_DMA | GFP_KERNEL); > > if (!vcdev->config_block) { >
Halil Pasic
2019-May-15 21:08 UTC
[PATCH 06/10] s390/cio: add basic protected virtualization support
On Tue, 14 May 2019 10:47:34 -0400 "Jason J. Herne" <jjherne at linux.ibm.com> wrote:> On 5/13/19 5:41 AM, Cornelia Huck wrote: > > On Fri, 26 Apr 2019 20:32:41 +0200 > > Halil Pasic <pasic at linux.ibm.com> wrote: > > > >> As virtio-ccw devices are channel devices, we need to use the dma area > >> for any communication with the hypervisor. > >> > >> This patch addresses the most basic stuff (mostly what is required for > >> virtio-ccw), and does take care of QDIO or any devices. > > > > "does not take care of QDIO", surely? (Also, what does "any devices" > > mean? Do you mean "every arbitrary device", perhaps?) > > > >> > >> An interesting side effect is that virtio structures are now going to > >> get allocated in 31 bit addressable storage. > > > > Hm... > > > >> > >> Signed-off-by: Halil Pasic <pasic at linux.ibm.com> > >> --- > >> arch/s390/include/asm/ccwdev.h | 4 +++ > >> drivers/s390/cio/ccwreq.c | 8 ++--- > >> drivers/s390/cio/device.c | 65 +++++++++++++++++++++++++++++++++------- > >> drivers/s390/cio/device_fsm.c | 40 ++++++++++++------------- > >> drivers/s390/cio/device_id.c | 18 +++++------ > >> drivers/s390/cio/device_ops.c | 21 +++++++++++-- > >> drivers/s390/cio/device_pgid.c | 20 ++++++------- > >> drivers/s390/cio/device_status.c | 24 +++++++-------- > >> drivers/s390/cio/io_sch.h | 21 +++++++++---- > >> drivers/s390/virtio/virtio_ccw.c | 10 ------- > >> 10 files changed, 148 insertions(+), 83 deletions(-) > > > > (...) > > > >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c > >> index 6d989c360f38..bb7a92316fc8 100644 > >> --- a/drivers/s390/virtio/virtio_ccw.c > >> +++ b/drivers/s390/virtio/virtio_ccw.c > >> @@ -66,7 +66,6 @@ struct virtio_ccw_device { > >> bool device_lost; > >> unsigned int config_ready; > >> void *airq_info; > >> - u64 dma_mask; > >> }; > >> > >> struct vq_info_block_legacy { > >> @@ -1255,16 +1254,7 @@ static int virtio_ccw_online(struct ccw_device *cdev) > >> ret = -ENOMEM; > >> goto out_free; > >> } > >> - > >> vcdev->vdev.dev.parent = &cdev->dev; > >> - cdev->dev.dma_mask = &vcdev->dma_mask; > >> - /* we are fine with common virtio infrastructure using 64 bit DMA */ > >> - ret = dma_set_mask_and_coherent(&cdev->dev, DMA_BIT_MASK(64)); > >> - if (ret) { > >> - dev_warn(&cdev->dev, "Failed to enable 64-bit DMA.\n"); > >> - goto out_free; > >> - } > > > > This means that vring structures now need to fit into 31 bits as well, > > I think? Is there any way to reserve the 31 bit restriction for channel > > subsystem structures and keep vring in the full 64 bit range? (Or am I > > fundamentally misunderstanding something?) > > > > I hope I've understood everything... I'm new to virtio. But from what I'm understanding, > the vring structure (a.k.a. the VirtQueue) needs to be accessed and modified by both host > and guest. Therefore the page(s) holding that data need to be marked shared if using > protected virtualization. This patch set makes use of DMA pages by way of swiotlb (always > below 32-bit line right?) for shared memory.The last sentence is wrong. You have to differentiate between stuff that is mapped as DMA and that is allocated as DMA. The mapped stuff is handled via swiotlb and bouncing. But that can not work for vring stuff which needs to be allocated as DMA.> Therefore, a side effect is that all shared > memory, including VirtQueue data will be in the DMA zone and in 32-bit memory. >Consequently wrong. The reason I explained in a reply to Connie (see there).> I don't see any restrictions on sharing pages above the 32-bit line. So it seems possible. > I'm not sure how much more work it would be. I wonder if Halil has considered this?I did consider this, the RFC was doing this (again see other mail).> Are we > worried that virtio data structures are going to be a burden on the 31-bit address space? > >That is a good question I can not answer. Since it is currently at least a page per queue (because we use dma direct, right Mimu?), I am concerned about this. Connie, what is your opinion? Regards, Halil
Cornelia Huck
2019-May-16 06:29 UTC
[PATCH 06/10] s390/cio: add basic protected virtualization support
On Wed, 15 May 2019 22:51:58 +0200 Halil Pasic <pasic at linux.ibm.com> wrote:> On Mon, 13 May 2019 11:41:36 +0200 > Cornelia Huck <cohuck at redhat.com> wrote: > > > On Fri, 26 Apr 2019 20:32:41 +0200 > > Halil Pasic <pasic at linux.ibm.com> wrote: > > > > > As virtio-ccw devices are channel devices, we need to use the dma area > > > for any communication with the hypervisor. > > > > > > This patch addresses the most basic stuff (mostly what is required for > > > virtio-ccw), and does take care of QDIO or any devices. > > > > "does not take care of QDIO", surely? > > I did not bother making the QDIO library code use dma memory for > anything that is conceptually dma memory. AFAIK QDIO is out of scope for > prot virt for now. If one were to do some emulated qdio with prot virt > guests, one wound need to make a bunch of things shared.And unless you wanted to support protected virt under z/VM as well, it would be wasted effort :)> > > (Also, what does "any devices" > > mean? Do you mean "every arbitrary device", perhaps?) > > What I mean is: this patch takes care of the core stuff, but any > particular device is likely to have to do more -- that is it ain't all > the cio devices support prot virt with this patch. For example > virtio-ccw needs to make sure that the ccws constituting the channel > programs, as well as the data pointed by the ccws is shared. If one > would want to make vfio-ccw DASD pass-through work under prot virt, one > would need to make sure, that everything that needs to be shared is > shared (data buffers, channel programs). > > Does is clarify things?That's what I thought, but the sentence was confusing :) What about "This patch addresses the most basic stuff (mostly what is required to support virtio-ccw devices). It handles neither QDIO devices, nor arbitrary non-virtio-ccw devices." ?> > > > > > > > > An interesting side effect is that virtio structures are now going to > > > get allocated in 31 bit addressable storage. > > > > Hm... > > > > > > > > Signed-off-by: Halil Pasic <pasic at linux.ibm.com> > > > --- > > > arch/s390/include/asm/ccwdev.h | 4 +++ > > > drivers/s390/cio/ccwreq.c | 8 ++--- > > > drivers/s390/cio/device.c | 65 +++++++++++++++++++++++++++++++++------- > > > drivers/s390/cio/device_fsm.c | 40 ++++++++++++------------- > > > drivers/s390/cio/device_id.c | 18 +++++------ > > > drivers/s390/cio/device_ops.c | 21 +++++++++++-- > > > drivers/s390/cio/device_pgid.c | 20 ++++++------- > > > drivers/s390/cio/device_status.c | 24 +++++++-------- > > > drivers/s390/cio/io_sch.h | 21 +++++++++---- > > > drivers/s390/virtio/virtio_ccw.c | 10 ------- > > > 10 files changed, 148 insertions(+), 83 deletions(-) > > > > (...) > > > > > diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c > > > index 6d989c360f38..bb7a92316fc8 100644 > > > --- a/drivers/s390/virtio/virtio_ccw.c > > > +++ b/drivers/s390/virtio/virtio_ccw.c > > > @@ -66,7 +66,6 @@ struct virtio_ccw_device { > > > bool device_lost; > > > unsigned int config_ready; > > > void *airq_info; > > > - u64 dma_mask; > > > }; > > > > > > struct vq_info_block_legacy { > > > @@ -1255,16 +1254,7 @@ static int virtio_ccw_online(struct ccw_device *cdev) > > > ret = -ENOMEM; > > > goto out_free; > > > } > > > - > > > vcdev->vdev.dev.parent = &cdev->dev; > > > - cdev->dev.dma_mask = &vcdev->dma_mask; > > > - /* we are fine with common virtio infrastructure using 64 bit DMA */ > > > - ret = dma_set_mask_and_coherent(&cdev->dev, DMA_BIT_MASK(64)); > > > - if (ret) { > > > - dev_warn(&cdev->dev, "Failed to enable 64-bit DMA.\n"); > > > - goto out_free; > > > - } > > > > This means that vring structures now need to fit into 31 bits as well, > > I think? > > Nod. > > > Is there any way to reserve the 31 bit restriction for channel > > subsystem structures and keep vring in the full 64 bit range? (Or am I > > fundamentally misunderstanding something?) > > > > At the root of this problem is that the DMA API basically says devices > may have addressing limitations expressed by the dma_mask, while our > addressing limitations are not coming from the device but from the IO > arch: e.g. orb.cpa and ccw.cda are 31 bit addresses. In our case it > depends on how and for what is the device going to use the memory (e.g. > buffers addressed by MIDA vs IDA vs direct). > > Virtio uses the DMA properties of the parent, that is in our case the > struct device embedded in struct ccw_device. > > The previous version (RFC) used to allocate all the cio DMA stuff from > this global cio_dma_pool using the css0.dev for the DMA API > interactions. And we set *css0.dev.dma_mask == DMA_BIT_MASK(31) so > e.g. the allocated ccws are 31 bit addressable. > > But I was asked to change this so that when I allocate DMA memory for a > channel program of particular ccw device, a struct device of that ccw > device is used as the first argument of dma_alloc_coherent(). > > Considering > > void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > gfp_t flag, unsigned long attrs) > { > const struct dma_map_ops *ops = get_dma_ops(dev); > void *cpu_addr; > > WARN_ON_ONCE(dev && !dev->coherent_dma_mask); > > if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr)) > return cpu_addr; > > /* let the implementation decide on the zone to allocate from: */ > flag &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM); > > that is the GFP flags dropped that implies that we really want > cdev->dev restricted to 31 bit addressable memory because we can't tell > (with the current common DMA code) hey but this piece of DMA mem you > are abot to allocate for me must be 31 bit addressable (using GFP_DMA > as we usually do). > > So, as described in the commit message, the vring stuff being forced > into ZONE_DMA is an unfortunate consequence of this all.Yeah. I hope 31 bits are enough for that as well.> > A side note: making the subchannel device 'own' the DMA stuff of a ccw > device (something that was discussed in the RFC thread) is tricky > because the ccw device may outlive the subchannel (all that orphan > stuff).Yes, that's... eww. Not really a problem for virtio-ccw devices (which do not support the disconnected state), but can we make DMA and the subchannel moving play nice with each other at all?> > So the answer is: it is technically possible (e.g. see RFC) but it comes > at a price, and I see no obviously brilliant solution. > > Regards, > Halil > > > > - > > > vcdev->config_block = kzalloc(sizeof(*vcdev->config_block), > > > GFP_DMA | GFP_KERNEL); > > > if (!vcdev->config_block) { > > >
Apparently Analagous Threads
- [PATCH 06/10] s390/cio: add basic protected virtualization support
- [PATCH 06/10] s390/cio: add basic protected virtualization support
- [PATCH 06/10] s390/cio: add basic protected virtualization support
- [PATCH 06/10] s390/cio: add basic protected virtualization support
- [PATCH 06/10] s390/cio: add basic protected virtualization support