Linux Virtualization - Jan 2018 - [virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang
wrote:
> On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:
> > On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:
> > 
> 
> > 
> > > +{
> > > +	struct scatterlist sg;
> > > +	unsigned int unused;
> > > +	int err;
> > > +
> > > +	sg_init_one(&sg, addr, sizeof(uint32_t));
> > This passes a guest-endian value to host. This is a problem:
> > should always pass LE values.
> 
> I think the endianness is handled when virtqueue_add_outbuf():
> 
> desc[i].addr = cpu_to_virtio64(_vq->vdev, addr);
> 
> right?

No - that handles the address, not the value you pass in.

> > 
> > > +
> > > +	/*
> > > +	 * This handles the cornercase that the vq happens to be full
when
> > > +	 * adding a cmd id. Rarely happen in practice.
> > > +	 */
> > > +	while (!vq->num_free)
> > > +		virtqueue_get_buf(vq, &unused);
> > I dislike this busy-waiting. It's a hint after all -
> > why not just retry later - hopefully after getting an
> > interrupt?
> > 
> > Alternatively, stop adding more entries when we have a single
> > ring entry left, making sure we have space for the command.
> 
> I think the second one looks good. Thanks.
> 
> > > +			queue_work(system_freezable_wq,
> > > +				   &vb->update_balloon_size_work);
> > > +		spin_unlock_irqrestore(&vb->stop_update_lock, flags);
> > > +	}
> > > +
> > > +	virtio_cread(vb->vdev, struct virtio_balloon_config,
> > > +		     free_page_report_cmd_id, &cmd_id);
> > You want virtio_cread_feature, don't access the new field
> > if the feature has not been negotiated.
> 
> Right. We probably need to put all the following cmd id related things
under
> the feature check,
> 
> How about
> 
> if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ)) {
>     virtio_cread(..);
>     if (cmd_id == VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID) {
>     ....
> }
> 
that's ok too.
> > 
> > 
> > > +	if (cmd_id == VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID) {
> > > +		WRITE_ONCE(vb->report_free_page, false);
> > > +	} else if (cmd_id != vb->start_cmd_id) {
> > > +		/*
> > > +		 * Host requests to start the reporting by sending a new cmd
> > > +		 * id.
> > > +		 */
> > > +		WRITE_ONCE(vb->report_free_page, true);
> > I don't know why we bother with WRITE_ONCE here.  The point of
> > report_free_page being used lockless is that that it's not a big
deal if
> > it's wrong occasionally, right?
> 
> Actually the main reason is that "vb->report_free_page" is a
value shared by
> two threads:
> Written by the config_change here, and read by the worker thread that
> reports the free pages.
Right but what's wrong if it's read or written twice and not once?
> Alternatively, we could let the two sides access to the shared variable
with
> "volatile" pointers.
> 
> 
> > 
> > 
> > 
> > > +		vb->start_cmd_id = cmd_id;
> > > +		queue_work(vb->balloon_wq,
&vb->report_free_page_work);
> > It seems that if a command was already queued (with a different id),
> > this will result in new command id being sent to host twice, which
will
> > likely confuse the host.
> 
> I think that case won't happen, because
> - the host sends a cmd id to the guest via the config, while the guest acks
> back the received cmd id via the virtqueue;
> - the guest ack back a cmd id only when a new cmd id is received from the
> host, that is the above check:
> 
>     if (cmd_id != vb->start_cmd_id) { --> the driver only queues the
> reporting work only when a new cmd id is received
>                         /*
>                          * Host requests to start the reporting by sending
a
>                          * new cmd id.
>                          */
>                         WRITE_ONCE(vb->report_free_page, true);
>                         vb->start_cmd_id = cmd_id;
>                         queue_work(vb->balloon_wq,
> &vb->report_free_page_work);
>     }
> 
> So the same cmd id wouldn't queue the reporting work twice.
> 
Like this:

		vb->start_cmd_id = cmd_id;
		queue_work(vb->balloon_wq, &vb->report_free_page_work);

command id changes

		vb->start_cmd_id = cmd_id;

work executes

		queue_work(vb->balloon_wq, &vb->report_free_page_work);

work executes again

> > 
> > 
> > 
> > > +	}
> > > +}
> > > +
> > >   static void update_balloon_size(struct virtio_balloon *vb)
> > >   {
> > >   	u32 actual = vb->num_pages;
> > > @@ -417,40 +513,113 @@ static void
update_balloon_size_func(struct work_struct *work)
> > >   static int init_vqs(struct virtio_balloon *vb)
> > >   {
> > > -	struct virtqueue *vqs[3];
> > > -	vq_callback_t *callbacks[] = { balloon_ack, balloon_ack,
stats_request };
> > > -	static const char * const names[] = { "inflate",
"deflate", "stats" };
> > > -	int err, nvqs;
> > > +	struct virtqueue **vqs;
> > > +	vq_callback_t **callbacks;
> > > +	const char **names;
> > > +	struct scatterlist sg;
> > > +	int i, nvqs, err = -ENOMEM;
> > > +
> > > +	/* Inflateq and deflateq are used unconditionally */
> > > +	nvqs = 2;
> > > +	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ))
> > > +		nvqs++;
> > > +	if (virtio_has_feature(vb->vdev,
VIRTIO_BALLOON_F_FREE_PAGE_VQ))
> > > +		nvqs++;
> > > +
> > > +	/* Allocate space for find_vqs parameters */
> > > +	vqs = kcalloc(nvqs, sizeof(*vqs), GFP_KERNEL);
> > > +	if (!vqs)
> > > +		goto err_vq;
> > > +	callbacks = kmalloc_array(nvqs, sizeof(*callbacks),
GFP_KERNEL);
> > > +	if (!callbacks)
> > > +		goto err_callback;
> > > +	names = kmalloc_array(nvqs, sizeof(*names), GFP_KERNEL);
> > > +	if (!names)
> > > +		goto err_names;
> > Why not just keep these 3 arrays on stack? they aren't large.
> 
> Sounds good. Here is the new implementation:
> 
> static int init_vqs(struct virtio_balloon *vb)
> {
>         struct virtqueue *vqs[4];
>         vq_callback_t *callbacks[4];
>         const char *names[4];
>         struct scatterlist sg;
>         int ret;
> 
> 
>         /*
>          * Inflateq and deflateq are used unconditionally. stats_vq and
>          * free_page_vq uses names[2] and names[3], respectively. The
> names[]
>          * will be NULL if the related feature is not enabled, which will
>          * cause no allocation for the corresponding virtqueue in find_vqs.
>          */
>         callbacks[0] = balloon_ack;
>         names[0] = "inflate";
>         callbacks[1] = balloon_ack;
>         names[1] = "deflate";
>         names[2] = NULL;
>         names[3] = NULL;
> 
>         if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) {
>                 names[2] = "stats";
>                 callbacks[2] = stats_request;
>         }
>         if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ))
{
>                 names[3] = "free_page_vq";
>                 callbacks[3] = NULL;
>         }
> 
>         ret = vb->vdev->config->find_vqs(vb->vdev, 4, vqs,
callbacks, names,
>                                          NULL, NULL);
>         if (ret)
>                 return ret;
> 
>         vb->inflate_vq = vqs[0];
>         vb->deflate_vq = vqs[1];
> 
>         if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) {
>                 vb->stats_vq = vqs[2];
>                 /*
>                  * Prime this virtqueue with one buffer so the hypervisor
> can
>                  * use it to signal us later (it can't be broken yet!).
>                  */
>                 sg_init_one(&sg, vb->stats, sizeof(vb->stats));
>                 ret = virtqueue_add_outbuf(vb->stats_vq, &sg, 1, vb,
>                                            GFP_KERNEL);
>                 if (ret) {
>                         dev_warn(&vb->vdev->dev, "%s: add
stat_vq failed\n",
>                                  __func__);
>                         return ret;
>                 }
>                 virtqueue_kick(vb->stats_vq);
>         }
> 
>         if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ))
>                 vb->free_page_vq = vqs[3];
> 
>         return 0;
> }
> 
> 
> Btw, the QEMU side doesn't have an option to disable STATS_VQ
currently, we
> may need to add that later.
> 
> Best,
> Wei
why not

On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:
> On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:
>> On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:
>>> On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:
>>>
>>>> +		vb->start_cmd_id = cmd_id;
>>>> +		queue_work(vb->balloon_wq,
&vb->report_free_page_work);
>>> It seems that if a command was already queued (with a different
id),
>>> this will result in new command id being sent to host twice, which
will
>>> likely confuse the host.
>> I think that case won't happen, because
>> - the host sends a cmd id to the guest via the config, while the guest
acks
>> back the received cmd id via the virtqueue;
>> - the guest ack back a cmd id only when a new cmd id is received from
the
>> host, that is the above check:
>>
>>      if (cmd_id != vb->start_cmd_id) { --> the driver only queues
the
>> reporting work only when a new cmd id is received
>>                          /*
>>                           * Host requests to start the reporting by
sending a
>>                           * new cmd id.
>>                           */
>>                          WRITE_ONCE(vb->report_free_page, true);
>>                          vb->start_cmd_id = cmd_id;
>>                          queue_work(vb->balloon_wq,
>> &vb->report_free_page_work);
>>      }
>>
>> So the same cmd id wouldn't queue the reporting work twice.
>>
> Like this:
>
> 		vb->start_cmd_id = cmd_id;
> 		queue_work(vb->balloon_wq, &vb->report_free_page_work);
>
> command id changes
>
> 		vb->start_cmd_id = cmd_id;
>
> work executes
>
> 		queue_work(vb->balloon_wq, &vb->report_free_page_work);
>
> work executes again
>
If we think about the whole working flow, I think this case couldn't happen:

1) device send cmd_id=1 to driver;
2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the 
device via the vq;
3) device revives cmd_id=1;
4) device wants to stop the reporting by sending cmd_id=STOP;
5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP to 
the device via the vq;
6) device sends cmd_id=2 to driver;
...

cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in between 
them (STOP won't queue the work).

How about defining the correct device behavior in the spec:
The device Should NOT send a second cmd id to the driver until a STOP 
cmd ack for the previous cmd id has been received from the guest.


Best,
Wei

On 01/22/2018 07:25 PM, Wei Wang wrote:
> On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:
>> On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:
>>> On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:
>>>> On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:
>>>>
>>>>> +        vb->start_cmd_id = cmd_id;
>>>>> +        queue_work(vb->balloon_wq,
&vb->report_free_page_work);
>>>> It seems that if a command was already queued (with a different
id),
>>>> this will result in new command id being sent to host twice,
which
>>>> will
>>>> likely confuse the host.
>>> I think that case won't happen, because
>>> - the host sends a cmd id to the guest via the config, while the 
>>> guest acks
>>> back the received cmd id via the virtqueue;
>>> - the guest ack back a cmd id only when a new cmd id is received 
>>> from the
>>> host, that is the above check:
>>>
>>>      if (cmd_id != vb->start_cmd_id) { --> the driver only
queues the
>>> reporting work only when a new cmd id is received
>>>                          /*
>>>                           * Host requests to start the reporting by
>>> sending a
>>>                           * new cmd id.
>>>                           */
>>>                          WRITE_ONCE(vb->report_free_page, true);
>>>                          vb->start_cmd_id = cmd_id;
>>>                          queue_work(vb->balloon_wq,
>>> &vb->report_free_page_work);
>>>      }
>>>
>>> So the same cmd id wouldn't queue the reporting work twice.
>>>
>> Like this:
>>
>>         vb->start_cmd_id = cmd_id;
>>         queue_work(vb->balloon_wq,
&vb->report_free_page_work);
>>
>> command id changes
>>
>>         vb->start_cmd_id = cmd_id;
>>
>> work executes
>>
>>         queue_work(vb->balloon_wq,
&vb->report_free_page_work);
>>
>> work executes again
>>
>
> If we think about the whole working flow, I think this case couldn't 
> happen:
>
> 1) device send cmd_id=1 to driver;
> 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the 
> device via the vq;
> 3) device revives cmd_id=1;
> 4) device wants to stop the reporting by sending cmd_id=STOP;
> 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP 
> to the device via the vq;
> 6) device sends cmd_id=2 to driver;
> ...
>
> cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in 
> between them (STOP won't queue the work).
>
> How about defining the correct device behavior in the spec:
> The device Should NOT send a second cmd id to the driver until a STOP 
> cmd ack for the previous cmd id has been received from the guest.

Thanks for the comments, and I adopted most of them in the new posted 
v23 patches. The above discussion is the one that I haven't included. If 
you could still see issues in the above analysis, please let me know. 
Thanks.

Best,
Wei

On Mon, Jan 22, 2018 at 07:25:45PM +0800, Wei Wang
wrote:
> On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:
> > On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:
> > > On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:
> > > > On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:
> > > > 
> > > > > +		vb->start_cmd_id = cmd_id;
> > > > > +		queue_work(vb->balloon_wq,
&vb->report_free_page_work);
> > > > It seems that if a command was already queued (with a
different id),
> > > > this will result in new command id being sent to host twice,
which will
> > > > likely confuse the host.
> > > I think that case won't happen, because
> > > - the host sends a cmd id to the guest via the config, while the
guest acks
> > > back the received cmd id via the virtqueue;
> > > - the guest ack back a cmd id only when a new cmd id is received
from the
> > > host, that is the above check:
> > > 
> > >      if (cmd_id != vb->start_cmd_id) { --> the driver only
queues the
> > > reporting work only when a new cmd id is received
> > >                          /*
> > >                           * Host requests to start the reporting
by sending a
> > >                           * new cmd id.
> > >                           */
> > >                          WRITE_ONCE(vb->report_free_page,
true);
> > >                          vb->start_cmd_id = cmd_id;
> > >                          queue_work(vb->balloon_wq,
> > > &vb->report_free_page_work);
> > >      }
> > > 
> > > So the same cmd id wouldn't queue the reporting work twice.
> > > 
> > Like this:
> > 
> > 		vb->start_cmd_id = cmd_id;
> > 		queue_work(vb->balloon_wq, &vb->report_free_page_work);
> > 
> > command id changes
> > 
> > 		vb->start_cmd_id = cmd_id;
> > 
> > work executes
> > 
> > 		queue_work(vb->balloon_wq, &vb->report_free_page_work);
> > 
> > work executes again
> > 
> 
> If we think about the whole working flow, I think this case couldn't
happen:
> 
> 1) device send cmd_id=1 to driver;
> 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the device
> via the vq;
> 3) device revives cmd_id=1;
> 4) device wants to stop the reporting by sending cmd_id=STOP;
> 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP to the
> device via the vq;
> 6) device sends cmd_id=2 to driver;
> ...
> 
> cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in between
them
> (STOP won't queue the work).
> 
> How about defining the correct device behavior in the spec:
> The device Should NOT send a second cmd id to the driver until a STOP cmd
> ack for the previous cmd id has been received from the guest.
> 
> 
> Best,
> Wei
I think we should just fix races in the driver rather than introduce
random restrictions in the device.

If device wants to start a new sequence, it should be able to
do just that without a complicated back and forth with several
roundtrips through the driver.

-- 
MST