Ross C Mcilroy
2005-Mar-02 14:39 UTC
[Xen-devel] [PATCH] Credit Limiting Backend Network Interfaces
Hi,
Attached is a patch which allows a domain''s network transmission to be
limited, to a certain credit of bytes every predefined period.
The virtual interface of a domain can be limited using the command
xm vif-limit [domain id] [vif number] [credit in bytes] [period in
uSecs]
e.g. to limit vif2.1 to 2Mb every second:
xm vif-limit 2 1 2000000 1000000
As Keir mentioned, it uses the jiffy timer, so is not very accurate
time-wise. It should be ok as long as you don''t set the period below
about 10 mSecs. Thoughts are welcome as to how this accuracy can be
increase.
In netback.c I moved the incrementing of req_cons below the credit
check, so that it is only incremented if the interface has enough
credit. I wasn''t sure however, if the memory barrier was needed after
this increment, before the request copy, or in both placed. I''ve left
it in both places, which seems to work well enough, but I''m not sure if
I''ve now added an unnecessary memory barrier call.
The patch is made against Xen 2.0.4, and seems to work well enough for
me. Let me know what you think.
Thanks
Ross
diff -Naur xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/common.h
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/common.h
--- xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/common.h
2005-02-04 13:38:30.000000000 +0000
+++ xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/common.h
2005-03-02 12:09:08.439496264 +0000
@@ -76,6 +76,7 @@
void netif_create(netif_be_create_t *create);
void netif_destroy(netif_be_destroy_t *destroy);
+void netif_creditlimit(netif_be_creditlimit_t *creditlimit);
void netif_connect(netif_be_connect_t *connect);
int netif_disconnect(netif_be_disconnect_t *disconnect, u8 rsp_id);
void netif_disconnect_complete(netif_t *netif);
diff -Naur xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/control.c
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/control.c
--- xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/control.c
2005-02-04 13:38:28.000000000 +0000
+++
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/control.c
2005-03-02 12:11:04.247890712 +0000
@@ -21,12 +21,17 @@
if ( msg->length != sizeof(netif_be_destroy_t) )
goto parse_error;
netif_destroy((netif_be_destroy_t *)&msg->msg[0]);
- break;
+ break;
+ case CMSG_NETIF_BE_CREDITLIMIT:
+ if ( msg->length != sizeof(netif_be_creditlimit_t) )
+ goto parse_error;
+ netif_creditlimit((netif_be_creditlimit_t *)&msg->msg[0]);
+ break;
case CMSG_NETIF_BE_CONNECT:
if ( msg->length != sizeof(netif_be_connect_t) )
goto parse_error;
netif_connect((netif_be_connect_t *)&msg->msg[0]);
- break;
+ break;
case CMSG_NETIF_BE_DISCONNECT:
if ( msg->length != sizeof(netif_be_disconnect_t) )
goto parse_error;
diff -Naur
xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/interface.c
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/interface.c
--- xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/interface.c
2005-02-04 13:38:34.000000000 +0000
+++
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/interface.c
2005-03-02 12:12:28.212126208 +0000
@@ -140,7 +140,7 @@
netif->credit_bytes = netif->remaining_credit = ~0UL;
netif->credit_usec = 0UL;
- /*init_ac_timer(&new_vif->credit_timeout);*/
+ init_timer(&netif->credit_timeout);
pnetif = &netif_hash[NETIF_HASH(domid, handle)];
while ( *pnetif != NULL )
@@ -223,6 +223,36 @@
destroy->status = NETIF_BE_STATUS_OKAY;
}
+void netif_creditlimit(netif_be_creditlimit_t *creditlimit)
+{
+ domid_t domid = creditlimit->domid;
+ unsigned int handle = creditlimit->netif_handle;
+ netif_t *netif;
+
+ netif = netif_find_by_handle(domid, handle);
+ if ( unlikely(netif == NULL) )
+ {
+ DPRINTK("netif_creditlimit attempted for non-existent netif"
+ " (%u,%u)\n", creditlimit->domid,
creditlimit->netif_handle);
+ creditlimit->status = NETIF_BE_STATUS_INTERFACE_NOT_FOUND;
+ return;
+ }
+
+ /* set the credit limit (reset remaining credit to new limit) */
+ netif->credit_bytes = netif->remaining_credit
creditlimit->credit_bytes;
+ netif->credit_usec = creditlimit->period_usec;
+
+ if ( netif->status == CONNECTED )
+ {
+ /* schedule work so that any packets waiting under previous
+ credit limit are dealt with (acts like a replenishment
point) */
+ netif->credit_timeout.expires = jiffies;
+ netif_schedule_work(netif);
+ }
+
+ creditlimit->status = NETIF_BE_STATUS_OKAY;
+}
+
void netif_connect(netif_be_connect_t *connect)
{
domid_t domid = connect->domid;
diff -Naur xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/netback.c
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/netback.c
--- xen-2.0/linux-2.6.10-xen-sparse/drivers/xen/netback/netback.c
2005-02-04 13:38:32.000000000 +0000
+++
xen_creditlimit/linux-2.6.10-xen-sparse/drivers/xen/netback/netback.c
2005-03-02 12:14:32.040301464 +0000
@@ -379,14 +379,13 @@
remove_from_net_schedule_list(netif);
}
-#if 0
+
static void tx_credit_callback(unsigned long data)
{
netif_t *netif = (netif_t *)data;
netif->remaining_credit = netif->credit_bytes;
netif_schedule_work(netif);
}
-#endif
static void net_tx_action(unsigned long unused)
{
@@ -470,8 +469,6 @@
continue;
}
- netif->tx->req_cons = ++netif->tx_req_cons;
-
/*
* 1. Ensure that we see the request when we copy it.
* 2. Ensure that frontend sees updated req_cons before we
check
@@ -482,30 +479,36 @@
memcpy(&txreq,
&netif->tx->ring[MASK_NETIF_TX_IDX(i)].req,
sizeof(txreq));
-#if 0
/* Credit-based scheduling. */
- if ( tx.size > netif->remaining_credit )
+ if ( txreq.size > netif->remaining_credit )
{
- s_time_t now = NOW(), next_credit =
- netif->credit_timeout.expires +
MICROSECS(netif->credit_usec);
- if ( next_credit <= now )
+ unsigned long now = jiffies;
+ unsigned long next_credit = netif->credit_timeout.expires
+ + msecs_to_jiffies(netif->credit_usec / 1000);
+ if ( !time_after(next_credit,now) )
{
netif->credit_timeout.expires = now;
netif->remaining_credit = netif->credit_bytes;
}
else
{
- netif->remaining_credit = 0;
- netif->credit_timeout.expires = next_credit;
- netif->credit_timeout.data = (unsigned long)netif;
- netif->credit_timeout.function = tx_credit_callback;
- netif->credit_timeout.cpu = smp_processor_id();
- add_ac_timer(&netif->credit_timeout);
- break;
+ if (!timer_pending(&netif->credit_timeout)) {
+ netif->remaining_credit = 0;
+ netif->credit_timeout.expires = next_credit;
+ netif->credit_timeout.data = (unsigned
long)netif;
+ netif->credit_timeout.function = tx_credit_callback;
+ add_timer_on(&netif->credit_timeout,
smp_processor_id());
+ } /* else already set to replenish credit */
+ break;
}
}
- netif->remaining_credit -= tx.size;
-#endif
+ if (netif->remaining_credit > txreq.size)
+ netif->remaining_credit -= txreq.size;
+ else
+ netif->remaining_credit = 0; /* prevent rollover */
+
+ netif->tx->req_cons = ++netif->tx_req_cons;
+ mb();
netif_schedule_work(netif);
diff -Naur xen-2.0/tools/python/xen/lowlevel/xu/xu.c
xen_creditlimit/tools/python/xen/lowlevel/xu/xu.c
--- xen-2.0/tools/python/xen/lowlevel/xu/xu.c 2005-02-04
13:38:30.000000000 +0000
+++ xen_creditlimit/tools/python/xen/lowlevel/xu/xu.c 2005-02-27
13:22:36.000000000 +0000
@@ -462,6 +462,13 @@
C2P(netif_be_destroy_t, netif_handle, Int, Long);
C2P(netif_be_destroy_t, status, Int, Long);
return dict;
+ case TYPE(CMSG_NETIF_BE, CMSG_NETIF_BE_CREDITLIMIT):
+ C2P(netif_be_creditlimit_t, domid, Int, Long);
+ C2P(netif_be_creditlimit_t, netif_handle, Int, Long);
+ C2P(netif_be_creditlimit_t, credit_bytes, Int, Long);
+ C2P(netif_be_creditlimit_t, period_usec, Int, Long);
+ C2P(netif_be_creditlimit_t, status, Int, Long);
+ return dict;
case TYPE(CMSG_NETIF_BE, CMSG_NETIF_BE_CONNECT):
C2P(netif_be_connect_t, domid, Int, Long);
C2P(netif_be_connect_t, netif_handle, Int, Long);
@@ -628,6 +635,12 @@
P2C(netif_be_destroy_t, domid, u32);
P2C(netif_be_destroy_t, netif_handle, u32);
break;
+ case TYPE(CMSG_NETIF_BE, CMSG_NETIF_BE_CREDITLIMIT):
+ P2C(netif_be_creditlimit_t, domid, u32);
+ P2C(netif_be_creditlimit_t, netif_handle, u32);
+ P2C(netif_be_creditlimit_t, credit_bytes, u32);
+ P2C(netif_be_creditlimit_t, period_usec, u32);
+ break;
case TYPE(CMSG_NETIF_BE, CMSG_NETIF_BE_CONNECT):
P2C(netif_be_connect_t, domid, u32);
P2C(netif_be_connect_t, netif_handle, u32);
diff -Naur xen-2.0/tools/python/xen/xend/server/messages.py
xen_creditlimit/tools/python/xen/xend/server/messages.py
--- xen-2.0/tools/python/xen/xend/server/messages.py 2005-02-04
13:38:31.000000000 +0000
+++ xen_creditlimit/tools/python/xen/xend/server/messages.py
2005-03-02 12:15:43.792393488 +0000
@@ -150,6 +150,7 @@
CMSG_NETIF_BE_DESTROY = 1
CMSG_NETIF_BE_CONNECT = 2
CMSG_NETIF_BE_DISCONNECT = 3
+CMSG_NETIF_BE_CREDITLIMIT = 4
CMSG_NETIF_BE_DRIVER_STATUS = 32
NETIF_INTERFACE_STATUS_CLOSED = 0 #/* Interface doesn''t exist.
*/
@@ -173,6 +174,9 @@
''netif_be_destroy_t'':
(CMSG_NETIF_BE, CMSG_NETIF_BE_DESTROY),
+ ''netif_be_creditlimit_t'':
+ (CMSG_NETIF_BE, CMSG_NETIF_BE_CREDITLIMIT),
+
''netif_be_driver_status_t'':
(CMSG_NETIF_BE, CMSG_NETIF_BE_DRIVER_STATUS),
diff -Naur xen-2.0/tools/python/xen/xend/server/netif.py
xen_creditlimit/tools/python/xen/xend/server/netif.py
--- xen-2.0/tools/python/xen/xend/server/netif.py 2005-02-04
13:38:34.000000000 +0000
+++ xen_creditlimit/tools/python/xen/xend/server/netif.py
2005-02-28 12:45:11.000000000 +0000
@@ -326,7 +326,21 @@
vif = val[''netif_handle'']
self.status = NETIF_INTERFACE_STATUS_CONNECTED
self.reportStatus()
-
+
+ def send_be_creditlimit(self, credit, period):
+ msg = packMsg(''netif_be_creditlimit_t'',
+ { ''domid'' :
self.controller.dom,
+ ''netif_handle'' : self.vif,
+ ''credit_bytes'' : credit,
+ ''period_usec'' : period })
+ d = defer.Deferred()
+ d.addCallback(self.respond_be_creditlimit)
+ self.getBackendInterface().writeRequest(msg, response=d)
+
+ def respond_be_creditlimit(self, msg):
+ val = unpackMsg(''netif_be_creditlimit_t'', msg)
+ return self
+
def reportStatus(self, resp=0):
msg = packMsg(''netif_fe_interface_status_t'',
{ ''handle'' : self.vif,
@@ -410,6 +424,15 @@
d = dev.attach()
return d
+ def limitDevice(self, vif, credit, period):
+ if vif not in self.devices:
+ raise XendError(''device does not exist for credit limit:
vif''
+ + str(self.dom) + ''.'' + str(vif))
+
+ dev = self.devices[vif]
+ d = dev.send_be_creditlimit(credit, period)
+ return d
+
def recv_fe_driver_status(self, msg, req):
if not req: return
print
diff -Naur xen-2.0/tools/python/xen/xend/server/SrvDomain.py
xen_creditlimit/tools/python/xen/xend/server/SrvDomain.py
--- xen-2.0/tools/python/xen/xend/server/SrvDomain.py 2005-02-04
13:38:37.000000000 +0000
+++ xen_creditlimit/tools/python/xen/xend/server/SrvDomain.py
2005-02-27 16:02:46.000000000 +0000
@@ -164,6 +164,15 @@
d = fn(req.args, {''dom'': self.dom.id})
return d
+ def op_vif_credit_limit(self, op, req):
+ fn = FormFn(self.xd.domain_vif_credit_limit,
+ [[''dom'', ''str''],
+ [''vif'', ''int''],
+ [''credit'', ''int''],
+ [''period'', ''int'']])
+ val = fn(req.args, {''dom'': self.dom.id})
+ return val
+
def op_vifs(self, op, req):
devs = self.xd.domain_vif_ls(self.dom.id)
return [ dev.sxpr() for dev in devs ]
diff -Naur xen-2.0/tools/python/xen/xend/XendClient.py
xen_creditlimit/tools/python/xen/xend/XendClient.py
--- xen-2.0/tools/python/xen/xend/XendClient.py 2005-02-04
13:38:28.000000000 +0000
+++ xen_creditlimit/tools/python/xen/xend/XendClient.py 2005-02-28
10:38:24.000000000 +0000
@@ -278,6 +278,13 @@
{ ''op'' :
''maxmem_set'',
''memory'' : memory })
+ def xend_domain_vif_limit(self, id, vif, credit, period):
+ return self.xendPost(self.domainurl(id),
+ { ''op'' :
''vif_credit_limit'',
+ ''vif'' : vif,
+ ''credit'' : credit,
+ ''period'' : period })
+
def xend_domain_vifs(self, id):
return self.xendGet(self.domainurl(id),
{ ''op'' :
''vifs'' })
diff -Naur xen-2.0/tools/python/xen/xend/XendDomainInfo.py
xen_creditlimit/tools/python/xen/xend/XendDomainInfo.py
--- xen-2.0/tools/python/xen/xend/XendDomainInfo.py 2005-02-04
13:38:31.000000000 +0000
+++ xen_creditlimit/tools/python/xen/xend/XendDomainInfo.py
2005-02-28 11:12:38.000000000 +0000
@@ -597,6 +597,18 @@
def get_device_recreate(self, type, index):
return self.get_device_savedinfo(type, index) or self.recreate
+ def limit_vif(self, vif, credit, period):
+ """Limit the rate of a virtual interface
+ @param vif: vif
+ @param credit: vif credit in bytes
+ @param period: vif period in uSec
+ @return: 0 on success
+ """
+
+ ctrl = xend.netif_create(self.dom, recreate=self.recreate)
+ d = ctrl.limitDevice(vif, credit, period)
+ return d
+
def add_config(self, val):
"""Add configuration data to a virtual machine.
@@ -1090,8 +1102,6 @@
vifs = vm.config_devices("vif")
vm.create_domain("plan9", kernel, ramdisk, cmdline)
return vm
-
-
def vm_dev_vif(vm, val, index, change=0):
"""Create a virtual network interface (vif).
diff -Naur xen-2.0/tools/python/xen/xend/XendDomain.py
xen_creditlimit/tools/python/xen/xend/XendDomain.py
--- xen-2.0/tools/python/xen/xend/XendDomain.py 2005-02-04
13:38:31.000000000 +0000
+++ xen_creditlimit/tools/python/xen/xend/XendDomain.py 2005-02-28
11:12:46.000000000 +0000
@@ -723,6 +723,15 @@
dominfo = self.domain_lookup(id)
return dominfo.get_device_by_index(type, idx)
+ def domain_vif_credit_limit(self, id, vif, credit, period):
+ """Limit the vif''s transmission rate
+ """
+ dominfo = self.domain_lookup(id)
+ try:
+ return dominfo.limit_vif(vif, credit, period)
+ except Exception, ex:
+ raise XendError(str(ex))
+
def domain_vif_ls(self, id):
"""Get list of virtual network interface (vif) indexes
for a
domain.
diff -Naur xen-2.0/tools/python/xen/xm/main.py
xen_creditlimit/tools/python/xen/xm/main.py
--- xen-2.0/tools/python/xen/xm/main.py 2005-02-04 13:38:38.000000000
+0000
+++ xen_creditlimit/tools/python/xen/xm/main.py 2005-02-27
15:45:48.000000000 +0000
@@ -717,6 +717,23 @@
xm.prog(ProgLog)
+class ProgVifCreditLimit(Prog):
+ group = ''vif''
+ name= "vif-limit"
+ info = """Limit the transmission rate of a virtual network
interface."""
+
+ def help(self, args):
+ print args[0], "DOMAIN VIF CREDIT_IN_BYTES PERIOD_IN_USECS"
+ print "\nSet the credit limit of a virtual network
interface."
+
+ def main(self, args):
+ if len(args) != 5: self.err("%s: Invalid argument(s)" %
args[0])
+ dom = args[1]
+ v = map(int, args[2:5])
+ server.xend_domain_vif_limit(dom, *v)
+
+xm.prog(ProgVifCreditLimit)
+
class ProgVifList(Prog):
group = ''vif''
name = ''vif-list''
diff -Naur xen-2.0/xen/include/public/io/domain_controller.h
xen_creditlimit/xen/include/public/io/domain_controller.h
--- xen-2.0/xen/include/public/io/domain_controller.h 2005-02-04
13:38:38.000000000 +0000
+++ xen_creditlimit/xen/include/public/io/domain_controller.h
2005-02-27 16:43:38.000000000 +0000
@@ -436,6 +436,7 @@
#define CMSG_NETIF_BE_DESTROY 1 /* Destroy a net-device interface.
*/
#define CMSG_NETIF_BE_CONNECT 2 /* Connect i/f to remote driver.
*/
#define CMSG_NETIF_BE_DISCONNECT 3 /* Disconnect i/f from remote
driver. */
+#define CMSG_NETIF_BE_CREDITLIMIT 4 /* Limit i/f to a given credit
limit. */
/* Messages to domain controller. */
#define CMSG_NETIF_BE_DRIVER_STATUS 32
@@ -498,6 +499,22 @@
} PACKED netif_be_destroy_t; /* 12 bytes */
/*
+ * CMSG_NETIF_BE_CREDITLIMIT:
+ * Limit a virtual interface to "credit_bytes" bytes per
"period_usec"
+ * microseconds.
+ */
+typedef struct {
+ /* IN */
+ domid_t domid; /* 0: Domain attached to new interface.
*/
+ u16 __pad0; /* 2 */
+ u32 netif_handle; /* 4: Domain-specific interface handle.
*/
+ u32 credit_bytes; /* 8: Vifs credit of bytes per period.
*/
+ u32 period_usec; /* 12: Credit replenishment period.
*/
+ /* OUT */
+ u32 status; /* 16 */
+} PACKED netif_be_creditlimit_t; /* 20 bytes */
+
+/*
* CMSG_NETIF_BE_CONNECT:
* When the driver sends a successful response then the interface is
fully
* connected. The controller will send a CONNECTED notification to the
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
Keir Fraser
2005-Mar-02 16:14 UTC
Re: [Xen-devel] [PATCH] Credit Limiting Backend Network Interfaces
On 2 Mar 2005, at 14:39, Ross C Mcilroy wrote:> Attached is a patch which allows a domain''s network transmission to be > limited, to a certain credit of bytes every predefined period.Ross, This patch looks great, but can you port it to our unstable tree? Only bug fixes are now going into the 2.0 series. Also, it would be better if you could send the patch as an attachment: extra line-breaks had got added to the pasted-in diff at some point in the emailing process. :-) If you can send the patch against unstable then we will *definitely* check it in! Cheers, Keir ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ross C Mcilroy
2005-Mar-02 16:55 UTC
RE: [Xen-devel] [PATCH] Credit Limiting Backend Network Interfaces
Keir, As requested, against the unstable tree source tarball. I''ve got to go now, so I''ve not really had time to test it, but it looks as if it should work. Let me know if it doesn''t and I''ll fix it tomorrow. Thanks Ross -----Original Message----- From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] Sent: 02 March 2005 16:15 To: Ross C Mcilroy Cc: xen-devel@lists.sourceforge.net Subject: Re: [Xen-devel] [PATCH] Credit Limiting Backend Network Interfaces On 2 Mar 2005, at 14:39, Ross C Mcilroy wrote:> Attached is a patch which allows a domain''s network transmission to be > limited, to a certain credit of bytes every predefined period.Ross, This patch looks great, but can you port it to our unstable tree? Only bug fixes are now going into the 2.0 series. Also, it would be better if you could send the patch as an attachment: extra line-breaks had got added to the pasted-in diff at some point in the emailing process. :-) If you can send the patch against unstable then we will *definitely* check it in! Cheers, Keir
Nuutti Kotivuori
2005-Mar-03 12:15 UTC
[Xen-devel] Re: [PATCH] Credit Limiting Backend Network Interfaces
Ross C. Mcilroy wrote:> Attached is a patch which allows a domain''s network transmission to > be limited, to a certain credit of bytes every predefined period.I believe this is as it should be, but just checking... This patch limits the amount of bytes accepted by the vif interface at Xen level (or at domain0) and not on the guest side, right? That is, a malicious guest kernel cannot send faster than the limit? The linux implementation of the guest side of this starts and stops the queue on the device as appropriate - so that the normal linux traffic queueing stuff can handle the entire queue as they see fit, just like on a real hardware device that is slower, right? -- Naked ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel