Linux Virtualization - Sep 2014 - [PATCH] virtio_blk: merge S/G list entries by default

Most virtio setups have a fairly limited number of ring entries available.
Enable S/G entry merging by default to fit into less of them.  This restores
the behavior at time of the virtio-blk blk-mq conversion, which was changed
by commit "block: add queue flag for disabling SG merging" which made
the
behavior optional, but didn't update the existing drivers to keep their
previous behavior.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/block/virtio_blk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 0a58140..311b857 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
 	vblk->tag_set.ops = &virtio_mq_ops;
 	vblk->tag_set.queue_depth = virtblk_queue_depth;
 	vblk->tag_set.numa_node = NUMA_NO_NODE;
-	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
+	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
 	vblk->tag_set.cmd_size  		sizeof(struct virtblk_req) +
 		sizeof(struct scatterlist) * sg_elems;
-- 
1.9.1

Il 07/09/2014 01:09, Christoph Hellwig ha scritto:
> Most virtio setups have a fairly limited number of ring entries available.
Are you disabling indirect descriptors?  With indirect descriptors entry
merging doesn't buy you any more space, so perhaps you can key the flag
off the availability of VIRTIO_RING_F_INDIRECT_DESC.

Paolo
> Enable S/G entry merging by default to fit into less of them.  This
restores
> the behavior at time of the virtio-blk blk-mq conversion, which was changed
> by commit "block: add queue flag for disabling SG merging" which
made the
> behavior optional, but didn't update the existing drivers to keep their
> previous behavior.

On Sun, Sep 7, 2014 at 7:09 AM, Christoph Hellwig <hch at lst.de>
wrote:
> Most virtio setups have a fairly limited number of ring entries available.
> Enable S/G entry merging by default to fit into less of them.  This
restores
> the behavior at time of the virtio-blk blk-mq conversion, which was changed
> by commit "block: add queue flag for disabling SG merging" which
made the
> behavior optional, but didn't update the existing drivers to keep their
> previous behavior.
It is a good idea to disable SG merge for vq incapable of indirect because
there are very limited direct descriptors.

For vq capable of indirect, it should be better to not do SG merge at default
because:

- from hypervisor view, no matter how many segments one req has, all are
submitted to host kernel by one syscall, such as readv/io_submit

- host kernel still need to do the same merge again
>From my test(virtio-blk over null_blk), looks enabling SG merge may cause
throughput a little drop(~3%).
>
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  drivers/block/virtio_blk.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 0a58140..311b857 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>         vblk->tag_set.ops = &virtio_mq_ops;
>         vblk->tag_set.queue_depth = virtblk_queue_depth;
>         vblk->tag_set.numa_node = NUMA_NO_NODE;
> -       vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
> +       vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
>         vblk->tag_set.cmd_size >                 sizeof(struct
virtblk_req) +
>                 sizeof(struct scatterlist) * sg_elems;
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-kernel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Thanks,
-- 
Ming Lei

On Sat, Sep 06, 2014 at 04:09:54PM -0700, Christoph Hellwig
wrote:
> Most virtio setups have a fairly limited number of ring entries available.
Seems a bit vague: QEMU at least has pretty large queues.
Which hypervisor do you have in mind?
This could be a gain everywhere if you manage to make descriptors
completely linear, so they fit in a single s/g.
ATM __virtblk_add_req always adds an s/g for the header:
is there a chance linux can pre-allocate a bit of memory
in front of the buffer to stick the header in?
> Enable S/G entry merging by default to fit into less of them.  This
restores
> the behavior at time of the virtio-blk blk-mq conversion, which was changed
> by commit "block: add queue flag for disabling SG merging" which
made the
> behavior optional, but didn't update the existing drivers to keep their
> previous behavior.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
OK so this is an optimization patch right?
What kind of performance gain is observed with it?
> ---
>  drivers/block/virtio_blk.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 0a58140..311b857 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>  	vblk->tag_set.ops = &virtio_mq_ops;
>  	vblk->tag_set.queue_depth = virtblk_queue_depth;
>  	vblk->tag_set.numa_node = NUMA_NO_NODE;
> -	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
> +	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
>  	vblk->tag_set.cmd_size >  		sizeof(struct virtblk_req) +
>  		sizeof(struct scatterlist) * sg_elems;
> -- 
> 1.9.1

On Sun, Sep 07, 2014 at 02:41:53PM +0300, Michael S. Tsirkin
wrote:
> > Signed-off-by: Christoph Hellwig <hch at lst.de>
> 
> OK so this is an optimization patch right?
> What kind of performance gain is observed with it?
None.  I actually wrote it when the block layer had a bug when dm was
used on top of the !merge case, and I decided to send it out as there had been
no discussion about disabling this by default on the existing blk-mq
drivers.

At least for my qemu/kvm setup it doesn't make a difference either way,
although not doing the cheap merge doesn't like the right kind of
optimization
to me.

Il 07/09/2014 13:41, Michael S. Tsirkin ha scritto:
> On Sat, Sep 06, 2014 at 04:09:54PM -0700, Christoph Hellwig wrote:
>> Most virtio setups have a fairly limited number of ring entries
available.
> 
> Seems a bit vague: QEMU at least has pretty large queues.
> Which hypervisor do you have in mind?
> This could be a gain everywhere if you manage to make descriptors
> completely linear, so they fit in a single s/g.
> ATM __virtblk_add_req always adds an s/g for the header:
> is there a chance linux can pre-allocate a bit of memory
> in front of the buffer to stick the header in?
Nope, the buffer usually comes directly from the page cache and will be
page aligned.

Paolo
>> Enable S/G entry merging by default to fit into less of them.  This
restores
>> the behavior at time of the virtio-blk blk-mq conversion, which was
changed
>> by commit "block: add queue flag for disabling SG merging"
which made the
>> behavior optional, but didn't update the existing drivers to keep
their
>> previous behavior.
>>
>> Signed-off-by: Christoph Hellwig <hch at lst.de>
> 
> OK so this is an optimization patch right?
> What kind of performance gain is observed with it?
> 
>> ---
>>  drivers/block/virtio_blk.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index 0a58140..311b857 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device
*vdev)
>>  	vblk->tag_set.ops = &virtio_mq_ops;
>>  	vblk->tag_set.queue_depth = virtblk_queue_depth;
>>  	vblk->tag_set.numa_node = NUMA_NO_NODE;
>> -	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
>> +	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
>>  	vblk->tag_set.cmd_size >>  		sizeof(struct virtblk_req) +
>>  		sizeof(struct scatterlist) * sg_elems;
>> -- 
>> 1.9.1