Xen devel - Mar 2009 - [PATCH 0/2] blkback: Token-based QoS resource limiting for VBD I/O.

Impact: add ability to define I/O usage policies

This enables a nieve token-based QoS system which allows for guests to
get an equal share of I/O requests. Without this patch, a misbehaving or
misconfigured guest can use up any available I/O resources, degrading I/O
performance for other guests on the machine.

The method for specifying these limitations is not implemented in this patch,
because that is an operational detail.

Signed-off-by: William Pitcock <nenolod@dereferenced.org>
---
 drivers/xen/blkback/blkback.c |   22 ++++++++++++++++++++++
 drivers/xen/blkback/common.h  |    6 ++++++
 2 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/blkback/blkback.c b/drivers/xen/blkback/blkback.c
index 8d988f4..996869e 100644
--- a/drivers/xen/blkback/blkback.c
+++ b/drivers/xen/blkback/blkback.c
@@ -203,6 +203,14 @@ static void print_stats(blkif_t *blkif)
 	blkif->st_oo_req = 0;
 }
 
+static void refill_reqcount(blkif_t *blkif)
+{
+	blkif->reqtime = jiffies + msecs_to_jiffies(1000);
+	blkif->reqcount += blkif->reqrate;
+	if (blkif->reqcount > blkif->reqmax)
+		blkif->reqcount = blkif->reqmax;
+}
+
 int blkif_schedule(void *arg)
 {
 	blkif_t *blkif = arg;
@@ -232,6 +240,9 @@ int blkif_schedule(void *arg)
 
 		if (log_stats && time_after(jiffies, blkif->st_print))
 			print_stats(blkif);
+
+		if (time_after(jiffies, blkif->reqtime))
+			refill_reqcount(blkif);
 	}
 
 	if (log_stats)
@@ -313,11 +324,22 @@ static int do_block_io_op(blkif_t *blkif)
 	rp = blk_rings->common.sring->req_prod;
 	rmb(); /* Ensure we see queued requests up to ''rp''. */
 
+	/* if there''s no available request tokens right now, and limiting
+	 * is requested, then don''t bother with going any further. */
+	if (blkif->reqcount <= 0 && blkif->reqmax != 0)
+		return (rc != rp) ? 1 : 0;
+
 	while (rc != rp) {
 
 		if (RING_REQUEST_CONS_OVERFLOW(&blk_rings->common, rc))
 			break;
 
+		/* FIXME: Should we report QoS overages as VBD_OO, or not? */
+		if (--blkif->reqcount <= 0) {
+			more_to_do = 1;
+			break;
+		}
+
 		if (kthread_should_stop()) {
 			more_to_do = 1;
 			break;
diff --git a/drivers/xen/blkback/common.h b/drivers/xen/blkback/common.h
index 57b7825..45af42f 100644
--- a/drivers/xen/blkback/common.h
+++ b/drivers/xen/blkback/common.h
@@ -92,6 +92,12 @@ typedef struct blkif_st {
 
 	grant_handle_t shmem_handle;
 	grant_ref_t    shmem_ref;
+
+	/* qos information */
+	long		    reqtime;
+	int		    reqcount;
+	int		    reqmax;
+	int		    reqfill;
 } blkif_t;
 
 blkif_t *blkif_alloc(domid_t domid);
-- 
1.6.1.3


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

This patch enables the ability to define a QoS policy for a specific guest. As
an example,
consider the following vbd table:

vbd = [
  
''phy:/dev/skylar/guest1-disk,xvda1:tokens-max=500000:tokens-refill=512,w'',
  
''phy:/dev/skylar/guest1-swap,xvda2:tokens-max=140000:tokens-refill=128,w'',
]

This patch would cause the system to give less credit to accesses for xvda2 than
xvda1, meaning that guests heavily into swap will degrade system performance
less.

Since the QoS settings are guest specific, each guest can have it''s own
credit
allotments, which may be considered useful to somebody.

This patch only affects the kernel side, changes to xend will be necessary and
will be covered in another patch against 3.4.

Signed-off-by: William Pitcock <nenolod@dereferenced.org>
---
 drivers/xen/blkback/xenbus.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/blkback/xenbus.c b/drivers/xen/blkback/xenbus.c
index 650f4b3..66357af 100644
--- a/drivers/xen/blkback/xenbus.c
+++ b/drivers/xen/blkback/xenbus.c
@@ -311,6 +311,11 @@ static void backend_changed(struct xenbus_watch *watch,
 		kfree(device_type);
 	}
 
+	/* gather information about QoS policy for this device. */
+	err = xenbus_gather(XBT_NIL, dev->otherend, "tokens-max",
"%d", &be->blkif->reqmax,
+		"tokens-refill", "%d", &be->blkif->reqrate,
NULL);
+	be->blkif->reqtime = jiffies;
+
 	if (be->major == 0 && be->minor == 0) {
 		/* Front end dir is a number, which is used as the handle. */
 
-- 
1.6.1.3


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

This patch series is a completed version of the I/O QoS work that I have
been doing in the blkback driver. The end goal of this work is to enable
constructs in the vm config file to define a ratelimiting policy for I/O
accesses.

This is important because on systems where LVM+AoE are used, I/O performance
is a larger challenge than with local disks. As such, the ability to reduce
the effects of guests performing a lot of I/O operations is desirable.

This patch series was inspired by the UML ''token-limiter''
patch, which was
developed with a similar rationale. However, unlike the token-limiter patch
the QoS limitation is enforced in the backend, which allows any paravirtualized
guest to be run without the ability of getting around the QoS limits.

This patch series provides the host kernel side. Patches to xend will
be necessary, and it may be desirable to see the QoS stats in the guest.
However, the guest side (blkfront) would need to be modified to display
these stats.

William Pitcock (2):
  blkback: Implement VBD QoS mechanics.
  blkback: Read "tokens-refill", and "tokens-max" xenstore
values to
    provide QoS parameters per device.

 drivers/xen/blkback/blkback.c |   22 ++++++++++++++++++++++
 drivers/xen/blkback/common.h  |    6 ++++++
 drivers/xen/blkback/xenbus.c  |    5 +++++
 3 files changed, 33 insertions(+), 0 deletions(-)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel