Michael S. Tsirkin
2018-Jan-19 12:39 UTC
[PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ
On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:> On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote: > > On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote: > > > > > > > > +{ > > > + struct scatterlist sg; > > > + unsigned int unused; > > > + int err; > > > + > > > + sg_init_one(&sg, addr, sizeof(uint32_t)); > > This passes a guest-endian value to host. This is a problem: > > should always pass LE values. > > I think the endianness is handled when virtqueue_add_outbuf(): > > desc[i].addr = cpu_to_virtio64(_vq->vdev, addr); > > right?No - that handles the address, not the value you pass in.> > > > > + > > > + /* > > > + * This handles the cornercase that the vq happens to be full when > > > + * adding a cmd id. Rarely happen in practice. > > > + */ > > > + while (!vq->num_free) > > > + virtqueue_get_buf(vq, &unused); > > I dislike this busy-waiting. It's a hint after all - > > why not just retry later - hopefully after getting an > > interrupt? > > > > Alternatively, stop adding more entries when we have a single > > ring entry left, making sure we have space for the command. > > I think the second one looks good. Thanks. > > > > + queue_work(system_freezable_wq, > > > + &vb->update_balloon_size_work); > > > + spin_unlock_irqrestore(&vb->stop_update_lock, flags); > > > + } > > > + > > > + virtio_cread(vb->vdev, struct virtio_balloon_config, > > > + free_page_report_cmd_id, &cmd_id); > > You want virtio_cread_feature, don't access the new field > > if the feature has not been negotiated. > > Right. We probably need to put all the following cmd id related things under > the feature check, > > How about > > if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ)) { > virtio_cread(..); > if (cmd_id == VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID) { > .... > } >that's ok too.> > > > > > > + if (cmd_id == VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID) { > > > + WRITE_ONCE(vb->report_free_page, false); > > > + } else if (cmd_id != vb->start_cmd_id) { > > > + /* > > > + * Host requests to start the reporting by sending a new cmd > > > + * id. > > > + */ > > > + WRITE_ONCE(vb->report_free_page, true); > > I don't know why we bother with WRITE_ONCE here. The point of > > report_free_page being used lockless is that that it's not a big deal if > > it's wrong occasionally, right? > > Actually the main reason is that "vb->report_free_page" is a value shared by > two threads: > Written by the config_change here, and read by the worker thread that > reports the free pages.Right but what's wrong if it's read or written twice and not once?> Alternatively, we could let the two sides access to the shared variable with > "volatile" pointers. > > > > > > > > > > > + vb->start_cmd_id = cmd_id; > > > + queue_work(vb->balloon_wq, &vb->report_free_page_work); > > It seems that if a command was already queued (with a different id), > > this will result in new command id being sent to host twice, which will > > likely confuse the host. > > I think that case won't happen, because > - the host sends a cmd id to the guest via the config, while the guest acks > back the received cmd id via the virtqueue; > - the guest ack back a cmd id only when a new cmd id is received from the > host, that is the above check: > > if (cmd_id != vb->start_cmd_id) { --> the driver only queues the > reporting work only when a new cmd id is received > /* > * Host requests to start the reporting by sending a > * new cmd id. > */ > WRITE_ONCE(vb->report_free_page, true); > vb->start_cmd_id = cmd_id; > queue_work(vb->balloon_wq, > &vb->report_free_page_work); > } > > So the same cmd id wouldn't queue the reporting work twice. >Like this: vb->start_cmd_id = cmd_id; queue_work(vb->balloon_wq, &vb->report_free_page_work); command id changes vb->start_cmd_id = cmd_id; work executes queue_work(vb->balloon_wq, &vb->report_free_page_work); work executes again> > > > > > > > > + } > > > +} > > > + > > > static void update_balloon_size(struct virtio_balloon *vb) > > > { > > > u32 actual = vb->num_pages; > > > @@ -417,40 +513,113 @@ static void update_balloon_size_func(struct work_struct *work) > > > static int init_vqs(struct virtio_balloon *vb) > > > { > > > - struct virtqueue *vqs[3]; > > > - vq_callback_t *callbacks[] = { balloon_ack, balloon_ack, stats_request }; > > > - static const char * const names[] = { "inflate", "deflate", "stats" }; > > > - int err, nvqs; > > > + struct virtqueue **vqs; > > > + vq_callback_t **callbacks; > > > + const char **names; > > > + struct scatterlist sg; > > > + int i, nvqs, err = -ENOMEM; > > > + > > > + /* Inflateq and deflateq are used unconditionally */ > > > + nvqs = 2; > > > + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) > > > + nvqs++; > > > + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ)) > > > + nvqs++; > > > + > > > + /* Allocate space for find_vqs parameters */ > > > + vqs = kcalloc(nvqs, sizeof(*vqs), GFP_KERNEL); > > > + if (!vqs) > > > + goto err_vq; > > > + callbacks = kmalloc_array(nvqs, sizeof(*callbacks), GFP_KERNEL); > > > + if (!callbacks) > > > + goto err_callback; > > > + names = kmalloc_array(nvqs, sizeof(*names), GFP_KERNEL); > > > + if (!names) > > > + goto err_names; > > Why not just keep these 3 arrays on stack? they aren't large. > > Sounds good. Here is the new implementation: > > static int init_vqs(struct virtio_balloon *vb) > { > struct virtqueue *vqs[4]; > vq_callback_t *callbacks[4]; > const char *names[4]; > struct scatterlist sg; > int ret; > > > /* > * Inflateq and deflateq are used unconditionally. stats_vq and > * free_page_vq uses names[2] and names[3], respectively. The > names[] > * will be NULL if the related feature is not enabled, which will > * cause no allocation for the corresponding virtqueue in find_vqs. > */ > callbacks[0] = balloon_ack; > names[0] = "inflate"; > callbacks[1] = balloon_ack; > names[1] = "deflate"; > names[2] = NULL; > names[3] = NULL; > > if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) { > names[2] = "stats"; > callbacks[2] = stats_request; > } > if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ)) { > names[3] = "free_page_vq"; > callbacks[3] = NULL; > } > > ret = vb->vdev->config->find_vqs(vb->vdev, 4, vqs, callbacks, names, > NULL, NULL); > if (ret) > return ret; > > vb->inflate_vq = vqs[0]; > vb->deflate_vq = vqs[1]; > > if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) { > vb->stats_vq = vqs[2]; > /* > * Prime this virtqueue with one buffer so the hypervisor > can > * use it to signal us later (it can't be broken yet!). > */ > sg_init_one(&sg, vb->stats, sizeof(vb->stats)); > ret = virtqueue_add_outbuf(vb->stats_vq, &sg, 1, vb, > GFP_KERNEL); > if (ret) { > dev_warn(&vb->vdev->dev, "%s: add stat_vq failed\n", > __func__); > return ret; > } > virtqueue_kick(vb->stats_vq); > } > > if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ)) > vb->free_page_vq = vqs[3]; > > return 0; > } > > > Btw, the QEMU side doesn't have an option to disable STATS_VQ currently, we > may need to add that later. > > Best, > Weiwhy not
Wei Wang
2018-Jan-22 11:25 UTC
[virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ
On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:> On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote: >> On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote: >>> On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote: >>> >>>> + vb->start_cmd_id = cmd_id; >>>> + queue_work(vb->balloon_wq, &vb->report_free_page_work); >>> It seems that if a command was already queued (with a different id), >>> this will result in new command id being sent to host twice, which will >>> likely confuse the host. >> I think that case won't happen, because >> - the host sends a cmd id to the guest via the config, while the guest acks >> back the received cmd id via the virtqueue; >> - the guest ack back a cmd id only when a new cmd id is received from the >> host, that is the above check: >> >> if (cmd_id != vb->start_cmd_id) { --> the driver only queues the >> reporting work only when a new cmd id is received >> /* >> * Host requests to start the reporting by sending a >> * new cmd id. >> */ >> WRITE_ONCE(vb->report_free_page, true); >> vb->start_cmd_id = cmd_id; >> queue_work(vb->balloon_wq, >> &vb->report_free_page_work); >> } >> >> So the same cmd id wouldn't queue the reporting work twice. >> > Like this: > > vb->start_cmd_id = cmd_id; > queue_work(vb->balloon_wq, &vb->report_free_page_work); > > command id changes > > vb->start_cmd_id = cmd_id; > > work executes > > queue_work(vb->balloon_wq, &vb->report_free_page_work); > > work executes again >If we think about the whole working flow, I think this case couldn't happen: 1) device send cmd_id=1 to driver; 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the device via the vq; 3) device revives cmd_id=1; 4) device wants to stop the reporting by sending cmd_id=STOP; 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP to the device via the vq; 6) device sends cmd_id=2 to driver; ... cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in between them (STOP won't queue the work). How about defining the correct device behavior in the spec: The device Should NOT send a second cmd id to the driver until a STOP cmd ack for the previous cmd id has been received from the guest. Best, Wei
Wei Wang
2018-Jan-24 03:18 UTC
[virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ
On 01/22/2018 07:25 PM, Wei Wang wrote:> On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote: >> On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote: >>> On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote: >>>> On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote: >>>> >>>>> + vb->start_cmd_id = cmd_id; >>>>> + queue_work(vb->balloon_wq, &vb->report_free_page_work); >>>> It seems that if a command was already queued (with a different id), >>>> this will result in new command id being sent to host twice, which >>>> will >>>> likely confuse the host. >>> I think that case won't happen, because >>> - the host sends a cmd id to the guest via the config, while the >>> guest acks >>> back the received cmd id via the virtqueue; >>> - the guest ack back a cmd id only when a new cmd id is received >>> from the >>> host, that is the above check: >>> >>> if (cmd_id != vb->start_cmd_id) { --> the driver only queues the >>> reporting work only when a new cmd id is received >>> /* >>> * Host requests to start the reporting by >>> sending a >>> * new cmd id. >>> */ >>> WRITE_ONCE(vb->report_free_page, true); >>> vb->start_cmd_id = cmd_id; >>> queue_work(vb->balloon_wq, >>> &vb->report_free_page_work); >>> } >>> >>> So the same cmd id wouldn't queue the reporting work twice. >>> >> Like this: >> >> vb->start_cmd_id = cmd_id; >> queue_work(vb->balloon_wq, &vb->report_free_page_work); >> >> command id changes >> >> vb->start_cmd_id = cmd_id; >> >> work executes >> >> queue_work(vb->balloon_wq, &vb->report_free_page_work); >> >> work executes again >> > > If we think about the whole working flow, I think this case couldn't > happen: > > 1) device send cmd_id=1 to driver; > 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the > device via the vq; > 3) device revives cmd_id=1; > 4) device wants to stop the reporting by sending cmd_id=STOP; > 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP > to the device via the vq; > 6) device sends cmd_id=2 to driver; > ... > > cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in > between them (STOP won't queue the work). > > How about defining the correct device behavior in the spec: > The device Should NOT send a second cmd id to the driver until a STOP > cmd ack for the previous cmd id has been received from the guest.Thanks for the comments, and I adopted most of them in the new posted v23 patches. The above discussion is the one that I haven't included. If you could still see issues in the above analysis, please let me know. Thanks. Best, Wei
Michael S. Tsirkin
2018-Jan-24 04:29 UTC
[virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ
On Mon, Jan 22, 2018 at 07:25:45PM +0800, Wei Wang wrote:> On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote: > > On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote: > > > On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote: > > > > On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote: > > > > > > > > > + vb->start_cmd_id = cmd_id; > > > > > + queue_work(vb->balloon_wq, &vb->report_free_page_work); > > > > It seems that if a command was already queued (with a different id), > > > > this will result in new command id being sent to host twice, which will > > > > likely confuse the host. > > > I think that case won't happen, because > > > - the host sends a cmd id to the guest via the config, while the guest acks > > > back the received cmd id via the virtqueue; > > > - the guest ack back a cmd id only when a new cmd id is received from the > > > host, that is the above check: > > > > > > if (cmd_id != vb->start_cmd_id) { --> the driver only queues the > > > reporting work only when a new cmd id is received > > > /* > > > * Host requests to start the reporting by sending a > > > * new cmd id. > > > */ > > > WRITE_ONCE(vb->report_free_page, true); > > > vb->start_cmd_id = cmd_id; > > > queue_work(vb->balloon_wq, > > > &vb->report_free_page_work); > > > } > > > > > > So the same cmd id wouldn't queue the reporting work twice. > > > > > Like this: > > > > vb->start_cmd_id = cmd_id; > > queue_work(vb->balloon_wq, &vb->report_free_page_work); > > > > command id changes > > > > vb->start_cmd_id = cmd_id; > > > > work executes > > > > queue_work(vb->balloon_wq, &vb->report_free_page_work); > > > > work executes again > > > > If we think about the whole working flow, I think this case couldn't happen: > > 1) device send cmd_id=1 to driver; > 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the device > via the vq; > 3) device revives cmd_id=1; > 4) device wants to stop the reporting by sending cmd_id=STOP; > 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP to the > device via the vq; > 6) device sends cmd_id=2 to driver; > ... > > cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in between them > (STOP won't queue the work). > > How about defining the correct device behavior in the spec: > The device Should NOT send a second cmd id to the driver until a STOP cmd > ack for the previous cmd id has been received from the guest. > > > Best, > WeiI think we should just fix races in the driver rather than introduce random restrictions in the device. If device wants to start a new sequence, it should be able to do just that without a complicated back and forth with several roundtrips through the driver. -- MST