Hi all, I''ve got XCP 1.0 up and running nicely and would like to use it in production. However I''m struggling with the concept of bandwidth management. It seems like such a common problem that everyone must have, but I can''t find any clear direction in which to go. The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth quota which needs to be shared between all the VMs on the XCP. Ideally I would like something to automatically manage the bandwidth such that each VM is capable of using the full 100mbps speed of the connection, but will be throttled back if the throughput is sustained, so we have e.g. 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage each. Alternatively it might be easier to just route all the virtual interfaces though a VM than runs pfsense or use tc on the host to just set some sort of shaping on the physical interface itself, but I really don''t know the best way to go about it. Things I''ve found so far aren''t so good: 1 - Limit the interface using the XenCenter GUI... but that means the VM would never be able to go above about 1mbps, even if it''s sat there and used no bandwidth for the past week and is well within its quota, so that''s not ideal. 2 - Use sFlow in XCP to capture the data. Well this works for looking at how much bandwidth they are using, but I haven''t found any existing tool that will act on that data to do traffic shaping. 3 - Use the XAPI calls to check the bandwidth usage. With methods 2 and 3 I guess I could write something that collects the data and stores it a database table, somehow work out how much the connection needs to be slowed by and then apply it using the XAPI, but that seems rather hacky and difficult and there must be a better way? If anyone could give some tips on how to do this I''d really appreciate it. Basically I just want the quickest and easiest way to make it so that the server as a whole doesn''t go over its bandwidth limit without limiting all the guests to a tiny speed individually. Thanks! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I would start by looking at configuring QOS on the Open VSwitch. That might get you where you want to be. Scott On Fri, May 20, 2011 at 11:38 AM, msgbox450@gmail.com <msgbox450@gmail.com> wrote:> Hi all, > I''ve got XCP 1.0 up and running nicely and would like to use it in > production. However I''m struggling with the concept of bandwidth management. > It seems like such a common problem that everyone must have, but I can''t > find any clear direction in which to go. > The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth > quota which needs to be shared between all the VMs on the XCP. > Ideally I would like something to automatically manage the bandwidth such > that each VM is capable of using the full 100mbps speed of the connection, > but will be throttled back if the throughput is sustained, so we have e.g. > 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage each. > Alternatively it might be easier to just route all the virtual interfaces > though a VM than runs pfsense or use tc on the host to just set some sort of > shaping on the physical interface itself, but I really don''t know the best > way to go about it. > > Things I''ve found so far aren''t so good: > 1 - Limit the interface using the XenCenter GUI... but that means the VM > would never be able to go above about 1mbps, even if it''s sat there and used > no bandwidth for the past week and is well within its quota, so that''s not > ideal. > 2 - Use sFlow in XCP to capture the data. Well this works for looking at how > much bandwidth they are using, but I haven''t found any existing tool that > will act on that data to do traffic shaping. > 3 - Use the XAPI calls to check the bandwidth usage. > > With methods 2 and 3 I guess I could write something that collects the data > and stores it a database table, somehow work out how much the connection > needs to be slowed by and then apply it using the XAPI, but that seems > rather hacky and difficult and there must be a better way? > > If anyone could give some tips on how to do this I''d really appreciate it. > Basically I just want the quickest and easiest way to make it so that the > server as a whole doesn''t go over its bandwidth limit without limiting all > the guests to a tiny speed individually. > Thanks! > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sorry - Meant to include the link - http://openvswitch.org/?page_id=267 On Fri, May 20, 2011 at 11:48 AM, Scott Damron <sdamron@gmail.com> wrote:> I would start by looking at configuring QOS on the Open VSwitch. That > might get you where you want to be. > > Scott > > On Fri, May 20, 2011 at 11:38 AM, msgbox450@gmail.com > <msgbox450@gmail.com> wrote: >> Hi all, >> I''ve got XCP 1.0 up and running nicely and would like to use it in >> production. However I''m struggling with the concept of bandwidth management. >> It seems like such a common problem that everyone must have, but I can''t >> find any clear direction in which to go. >> The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth >> quota which needs to be shared between all the VMs on the XCP. >> Ideally I would like something to automatically manage the bandwidth such >> that each VM is capable of using the full 100mbps speed of the connection, >> but will be throttled back if the throughput is sustained, so we have e.g. >> 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage each. >> Alternatively it might be easier to just route all the virtual interfaces >> though a VM than runs pfsense or use tc on the host to just set some sort of >> shaping on the physical interface itself, but I really don''t know the best >> way to go about it. >> >> Things I''ve found so far aren''t so good: >> 1 - Limit the interface using the XenCenter GUI... but that means the VM >> would never be able to go above about 1mbps, even if it''s sat there and used >> no bandwidth for the past week and is well within its quota, so that''s not >> ideal. >> 2 - Use sFlow in XCP to capture the data. Well this works for looking at how >> much bandwidth they are using, but I haven''t found any existing tool that >> will act on that data to do traffic shaping. >> 3 - Use the XAPI calls to check the bandwidth usage. >> >> With methods 2 and 3 I guess I could write something that collects the data >> and stores it a database table, somehow work out how much the connection >> needs to be slowed by and then apply it using the XAPI, but that seems >> rather hacky and difficult and there must be a better way? >> >> If anyone could give some tips on how to do this I''d really appreciate it. >> Basically I just want the quickest and easiest way to make it so that the >> server as a whole doesn''t go over its bandwidth limit without limiting all >> the guests to a tiny speed individually. >> Thanks! >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
You can limit the outbound bandwidth rate using the Network Properties QoS option on a per VM basis. That would prevent of lot of potential abuse. You would not need to limit each one to 1Mbps, though. You could set it higher than that. Alternatively, you could write an application that checked bandwidth stats using the API and then updated the QoS limit using the API. That would permit you a lot of flexibility when writing business rules to manage it. -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of msgbox450@gmail.com Sent: Friday, May 20, 2011 11:39 AM To: xen-users@lists.xensource.com Subject: [Xen-users] XCP bandwidth management Hi all, I''ve got XCP 1.0 up and running nicely and would like to use it in production. However I''m struggling with the concept of bandwidth management. It seems like such a common problem that everyone must have, but I can''t find any clear direction in which to go. The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth quota which needs to be shared between all the VMs on the XCP. Ideally I would like something to automatically manage the bandwidth such that each VM is capable of using the full 100mbps speed of the connection, but will be throttled back if the throughput is sustained, so we have e.g. 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage each. Alternatively it might be easier to just route all the virtual interfaces though a VM than runs pfsense or use tc on the host to just set some sort of shaping on the physical interface itself, but I really don''t know the best way to go about it. Things I''ve found so far aren''t so good: 1 - Limit the interface using the XenCenter GUI... but that means the VM would never be able to go above about 1mbps, even if it''s sat there and used no bandwidth for the past week and is well within its quota, so that''s not ideal. 2 - Use sFlow in XCP to capture the data. Well this works for looking at how much bandwidth they are using, but I haven''t found any existing tool that will act on that data to do traffic shaping. 3 - Use the XAPI calls to check the bandwidth usage. With methods 2 and 3 I guess I could write something that collects the data and stores it a database table, somehow work out how much the connection needs to be slowed by and then apply it using the XAPI, but that seems rather hacky and difficult and there must be a better way? If anyone could give some tips on how to do this I''d really appreciate it. Basically I just want the quickest and easiest way to make it so that the server as a whole doesn''t go over its bandwidth limit without limiting all the guests to a tiny speed individually. Thanks! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks both. Rate limiting the individual VMs though would stop them ever being able to go beyond that rate though right? In practice I want them to be able to get 100mbit in sustained bursts (i.e. download a few big files per day) but ensure the average is around 1mbps-2mbps across the month (i.e. not constantly downloading). Am I right in saying that the vSwitch QoS can''t do that? The idea of checking the bandwidth stats with XAPI and then updating the limit accordingly seems like the option I might have to go for, but I have no idea what the complicated algorithm I''d need to use is and it sounds painful... was really hoping there''s an easier way :S On Fri, May 20, 2011 at 6:00 PM, <admin@xenhive.com> wrote:> You can limit the outbound bandwidth rate using the Network Properties > QoS option on a per VM basis. That would prevent of lot of potential > abuse. You would not need to limit each one to 1Mbps, though. You could > set it higher than that. > > > > Alternatively, you could write an application that checked bandwidth stats > using the API and then updated the QoS limit using the API. That would > permit you a lot of flexibility when writing business rules to manage it. > > > > -----Original Message----- > *From:* xen-users-bounces@lists.xensource.com [mailto: > xen-users-bounces@lists.xensource.com] *On Behalf Of *msgbox450@gmail.com > *Sent:* Friday, May 20, 2011 11:39 AM > *To:* xen-users@lists.xensource.com > *Subject:* [Xen-users] XCP bandwidth management > > > > Hi all, > > > > I''ve got XCP 1.0 up and running nicely and would like to use it in > production. However I''m struggling with the concept of bandwidth management. > It seems like such a common problem that everyone must have, but I can''t > find any clear direction in which to go. > > The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth > quota which needs to be shared between all the VMs on the XCP. > > > Ideally I would like something to automatically manage the bandwidth such > that each VM is capable of using the full 100mbps speed of the connection, > but will be throttled back if the throughput is sustained, so we have e.g. > 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage each. > > Alternatively it might be easier to just route all the virtual interfaces > though a VM than runs pfsense or use tc on the host to just set some sort of > shaping on the physical interface itself, but I really don''t know the best > way to go about it. > > > > > > Things I''ve found so far aren''t so good: > > 1 - Limit the interface using the XenCenter GUI... but that means the VM > would never be able to go above about 1mbps, even if it''s sat there and used > no bandwidth for the past week and is well within its quota, so that''s not > ideal. > > > > 2 - Use sFlow in XCP to capture the data. Well this works for looking at > how much bandwidth they are using, but I haven''t found any existing tool > that will act on that data to do traffic shaping. > > > > 3 - Use the XAPI calls to check the bandwidth usage. > > > > > > With methods 2 and 3 I guess I could write something that collects the data > and stores it a database table, somehow work out how much the connection > needs to be slowed by and then apply it using the XAPI, but that seems > rather hacky and difficult and there must be a better way? > > > > > > If anyone could give some tips on how to do this I''d really appreciate it. > Basically I just want the quickest and easiest way to make it so that the > server as a whole doesn''t go over its bandwidth limit without limiting all > the guests to a tiny speed individually. > > > > Thanks! > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, May 20, 2011 at 9:38 AM, msgbox450@gmail.com <msgbox450@gmail.com> wrote:> Hi all, > I''ve got XCP 1.0 up and running nicely and would like to use it in > production. However I''m struggling with the concept of bandwidth management. > It seems like such a common problem that everyone must have, but I can''t > find any clear direction in which to go. > The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth > quota which needs to be shared between all the VMs on the XCP. > Ideally I would like something to automatically manage the bandwidth such > that each VM is capable of using the full 100mbps speed of the connection, > but will be throttled back if the throughput is sustained, so we have e.g. > 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage each. > Alternatively it might be easier to just route all the virtual interfaces > though a VM than runs pfsense or use tc on the host to just set some sort of > shaping on the physical interface itself, but I really don''t know the best > way to go about it. > > Things I''ve found so far aren''t so good: > 1 - Limit the interface using the XenCenter GUI... but that means the VM > would never be able to go above about 1mbps, even if it''s sat there and used > no bandwidth for the past week and is well within its quota, so that''s not > ideal. > 2 - Use sFlow in XCP to capture the data. Well this works for looking at how > much bandwidth they are using, but I haven''t found any existing tool that > will act on that data to do traffic shaping. > 3 - Use the XAPI calls to check the bandwidth usage. > > With methods 2 and 3 I guess I could write something that collects the data > and stores it a database table, somehow work out how much the connection > needs to be slowed by and then apply it using the XAPI, but that seems > rather hacky and difficult and there must be a better way? > > If anyone could give some tips on how to do this I''d really appreciate it. > Basically I just want the quickest and easiest way to make it so that the > server as a whole doesn''t go over its bandwidth limit without limiting all > the guests to a tiny speed individually. > Thanks! > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >I don''t know of any shrink wrap solutions that would meet your requirements today, but XCP does contain the APIs needed to develop a bandwidth manager. http://blog.sflow.com/2011/05/openflow-and-sflow.html It is important to distinguish between the amount of local traffic that a VM generates (inter-VM traffic, backup etc) and non-local traffic that counts against the 5TB quota. XAPI calls just give the traffic totals. sFlow monitoring in the vSwitch easily distinguishes between the local and non-local traffic. On the control side, Open Flow allows the controller to create separate policies for forwarding local and non-local traffic. Combining the two, allows for adaptive management. http://blog.sflow.com/2009/10/network-edge.html OpenFlow has only recently started to be available in production environments so the management tools are still lagging. There are many open source and commercial OpenFlow controllers in development and I expect that there will be a number of solutions available for managing XCP in the near future. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks Peter, This is very interesting... It looks like OpenFlow would be the best solution, although it seems very complicated and scarcely documented at the moment so I doubt I''d have much chance at implementing anything in a few days work. It is perhaps more something that I could tag on later in a year or two when the tools and APIs start appearing. We''ve already had a go at implementing an sFlow collector, but seem to get multiple readings for the same interface which is a bit confusing :P It always seems like I''m trying to play with things that aren''t ready yet, but as by the time I find something that''s ready, its all old news and I''m the last one using it :) On Sat, May 21, 2011 at 8:05 PM, Peter Phaal <peter.phaal@gmail.com> wrote:> On Fri, May 20, 2011 at 9:38 AM, msgbox450@gmail.com > <msgbox450@gmail.com> wrote: > > Hi all, > > I''ve got XCP 1.0 up and running nicely and would like to use it in > > production. However I''m struggling with the concept of bandwidth > management. > > It seems like such a common problem that everyone must have, but I can''t > > find any clear direction in which to go. > > The dedicated host I am using (Hetzner) gives me a 5TB monthly bandwidth > > quota which needs to be shared between all the VMs on the XCP. > > Ideally I would like something to automatically manage the bandwidth such > > that each VM is capable of using the full 100mbps speed of the > connection, > > but will be throttled back if the throughput is sustained, so we have > e.g. > > 24 x 1GB VMs on the host with average of 213GB/month bandwidth usage > each. > > Alternatively it might be easier to just route all the virtual interfaces > > though a VM than runs pfsense or use tc on the host to just set some sort > of > > shaping on the physical interface itself, but I really don''t know the > best > > way to go about it. > > > > Things I''ve found so far aren''t so good: > > 1 - Limit the interface using the XenCenter GUI... but that means the VM > > would never be able to go above about 1mbps, even if it''s sat there and > used > > no bandwidth for the past week and is well within its quota, so that''s > not > > ideal. > > 2 - Use sFlow in XCP to capture the data. Well this works for looking at > how > > much bandwidth they are using, but I haven''t found any existing tool that > > will act on that data to do traffic shaping. > > 3 - Use the XAPI calls to check the bandwidth usage. > > > > With methods 2 and 3 I guess I could write something that collects the > data > > and stores it a database table, somehow work out how much the connection > > needs to be slowed by and then apply it using the XAPI, but that seems > > rather hacky and difficult and there must be a better way? > > > > If anyone could give some tips on how to do this I''d really appreciate > it. > > Basically I just want the quickest and easiest way to make it so that the > > server as a whole doesn''t go over its bandwidth limit without limiting > all > > the guests to a tiny speed individually. > > Thanks! > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > I don''t know of any shrink wrap solutions that would meet your > requirements today, but XCP does contain the APIs needed to develop a > bandwidth manager. > > http://blog.sflow.com/2011/05/openflow-and-sflow.html > > It is important to distinguish between the amount of local traffic > that a VM generates (inter-VM traffic, backup etc) and non-local > traffic that counts against the 5TB quota. XAPI calls just give the > traffic totals. sFlow monitoring in the vSwitch easily distinguishes > between the local and non-local traffic. > > On the control side, Open Flow allows the controller to create > separate policies for forwarding local and non-local traffic. > Combining the two, allows for adaptive management. > > http://blog.sflow.com/2009/10/network-edge.html > > OpenFlow has only recently started to be available in production > environments so the management tools are still lagging. There are many > open source and commercial OpenFlow controllers in development and I > expect that there will be a number of solutions available for managing > XCP in the near future. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi,> 1 - Limit the interface using the XenCenter GUI..I believe in free version of XenServer bandwidth limiting does not work (it silently does nothing), I doubt this feature is enabled in XCP, have you tried? I have patched /etc/xensource/scripts/vif script to (among other things) call ovs-vsctl to set ingress policy rate. I imagine commercial edition of XenServer do pretty much the same. Downside is that this approach only limits traffic from VM. If you are billed for traffic your VMs download from the Internet (like I am), this is not enough. I''ve tried to play with ''tc'' to limit traffic in another direction, but it never worked properly -- I want to cap the rate to 10Mbps, but it always gave me below 1Mbps whatever I do.> 2 - Use sFlow in XCP to capture the data. Well this works for looking > at how much bandwidth they are using, but I haven''t found any existing > tool that > will act on that data to do traffic shaping.I have tried this path too, but didn''t go very far. What sFlow collector are you using? Best regards, Alex Bezroutchko http://www.gremwell.com/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ah, nice to know I am not the only one with this problem. I didn''t actually test the XCP bandwidth limit either...I just assumed it worked because it didn''t give an error, perhaps you are right about it silently doing nothing. I guess that is a feature request for the XCP devs then if it doesn''t? Do you know of any better documentation for the ovs-vsctl? Maybe there is an answer there but I can''t seem to find a full syntax explanation :( For the sFlow collector, it''s one we wrote ourselves...not really too sure it''s working right though because we seem to get multiple readings for the same interface. Also, not sure how to stop it being easily floodable (sFlow would be in a domU so if another domU floods it with these sFlow packets it would screw up the results I guess). Even if it works, not sure how useful it is... it only tells us what''s been used, we''d still need the rate limiter thing to work. I guess the ideal solution for me would be to have a VM guest that logs all the traffic, does some calculations. I want to check if the VM is averaging too fast across the month to meet its monthly target, and if it is,slow it down until the target is met (using 95 percentile perhaps to ignore spikes of activity). Really don''t know how to achieve that with what''s available :( On Sun, May 22, 2011 at 9:39 PM, Alexandre Bezroutchko <abb@scanit.be>wrote:> Hi, > > 1 - Limit the interface using the XenCenter GUI.. >> > I believe in free version of XenServer bandwidth limiting does not work (it > silently does nothing), I doubt this feature is enabled in XCP, have you > tried? > > I have patched /etc/xensource/scripts/vif script to (among other things) > call ovs-vsctl to set ingress policy rate. I imagine commercial edition of > XenServer do pretty much the same. Downside is that this approach only > limits traffic from VM. If you are billed for traffic your VMs download from > the Internet (like I am), this is not enough. > > I''ve tried to play with ''tc'' to limit traffic in another direction, but it > never worked properly -- I want to cap the rate to 10Mbps, but it always > gave me below 1Mbps whatever I do. > > > 2 - Use sFlow in XCP to capture the data. Well this works for looking at >> how much bandwidth they are using, but I haven''t found any existing tool >> that >> will act on that data to do traffic shaping. >> > I have tried this path too, but didn''t go very far. What sFlow collector > are you using? > > Best regards, > Alex Bezroutchko > http://www.gremwell.com/ > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Do you know of any better documentation for the ovs-vsctl? Maybe there > is an answer there but I can''t seem to find a full syntax explanation :(I have found a lot of useful info here http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf (also see http://openvswitch.org/?page_id=14). I don''t expect you will find a full solution there.> For the sFlow collector, it''s one we wrote ourselves...not really too > sure it''s working right though because we seem to get multiple > readings for the same interface. Also, not sure how to stop it being > easily floodable (sFlow would be in a domU so if another domU floods > it with these sFlow packets it would screw up the results I guess).Would creating a dedicated network for sFlow-collector domU solve the problem? Normally your management network should be isolated from untrusted domUs already, so you could use it for sFlow-collector domU too.> I guess the ideal solution for me would be to have a VM guest that > logs all the traffic, does some calculations. I want to check if the > VM is averaging too fast across the month to meet its monthly target, > and if it is,slow it down until the target is met (using 95 percentile > perhaps to ignore spikes of activity). > > Really don''t know how to achieve that with what''s available :(As far as I can see, there are three pieces in this puzzle: 1) traffic usage metering; 2) control algorithm to decide when it is time to apply a quota; 3) policing mechanism to cap the bandwidth of the offending VMs Problem #1 is pretty much solved. It can be done with sFlow collector or like this http://www.falsyana.com/2011/loeniks/xenserver-customization-bandwidth/. I currently use approach similar to the latter, but I save interface counters in RRD files, for the history. Exact solution for problem #2 will probably vary greatly from installation to installation. Effectively we are talking about developing a system which takes set of timestamped values (vm_uud; vif_uud; rx_bytes; tx_bytes) as input and produces (vif_uuid; ingress_poliing_rate; //ingress_poliing_burst) values as output. Regarding problem #3 -- it is easy to limit traffic from VM, I trust it is possible to limit it in other direction too, but I didn''t have enough time to make it work. One catch with limiting egress traffic is that here we actually have no control over how much stuff Internet hosts send to the IP address of VM. By limiting we are talking about dropping the excess of the traffic and hoping TCP will throttle down accordingly, which normally it will do. Another aspect of problem #3 is how to transfer the policy from sFlow domU to dom0. I would do it by publishing desired policy on sFlow domU. It can be a simple as putting a plain text file on HTTP server, or maybe having LDAP server on sFlow domU, if it is justified. Then a cron job in dom0 would poll the desired policy from sFlow domU and reconfigure vswitch accordingly. Alternatively, one can push the config from sFlow domU into dom0 over native vSwitch API, but I feel it is less preferable from security point of view. I would prefer to keep my dom0 shut, even from trusted systems I own. Wanna trade ;) your sFlow collector for vif script patch? The patch does bandwidth limiting (hardcoded value) and MAC/IP spoofing and ARP poisoning protection (allowed IP taken from VM other-config parameter). Actually I was going to publish the it anyway, if there is any interest... Cheers, Alex _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have tested the bandwidth limiting in XCP and it does work, but only on outbound traffic. -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Alexandre Bezroutchko Sent: Sunday, May 22, 2011 3:40 PM To: xen-users@lists.xensource.com Subject: Re: [Xen-users] XCP bandwidth management Hi,> 1 - Limit the interface using the XenCenter GUI..I believe in free version of XenServer bandwidth limiting does not work (it silently does nothing), I doubt this feature is enabled in XCP, have you tried? I have patched /etc/xensource/scripts/vif script to (among other things) call ovs-vsctl to set ingress policy rate. I imagine commercial edition of XenServer do pretty much the same. Downside is that this approach only limits traffic from VM. If you are billed for traffic your VMs download from the Internet (like I am), this is not enough. I''ve tried to play with ''tc'' to limit traffic in another direction, but it never worked properly -- I want to cap the rate to 10Mbps, but it always gave me below 1Mbps whatever I do.> 2 - Use sFlow in XCP to capture the data. Well this works for looking > at how much bandwidth they are using, but I haven''t found any existing > tool that > will act on that data to do traffic shaping.I have tried this path too, but didn''t go very far. What sFlow collector are you using? Best regards, Alex Bezroutchko http://www.gremwell.com/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''ve tested the internal netback capabilities for shaping - it not very accurate, but working. It counts only outgoing traffic from VM. But it do not allow any burst capabilities or traffic accounting. To enable it: xe vif-param-set uuid= qos_algorithm_type=ratelimit xe vif-paraxe vif-param-set uuid= qos_algorithm_params:kbps=2500 If you wish to count VM traffic you will need to dig beyond XCP into xen and dom0 linux (hint: in /sys in dom0 there is xen-backend tree with statistics counters - but this is definitely not ''XCP'', it''s pure hacking into Xen). On 20.05.2011 20:38, msgbox450@gmail.com wrote:> Hi all, > > I''ve got XCP 1.0 up and running nicely and would like to use it in > production. However I''m struggling with the concept of bandwidth > management. It seems like such a common problem that everyone must > have, but I can''t find any clear direction in which to go. > The dedicated host I am using (Hetzner) gives me a 5TB monthly > bandwidth quota which needs to be shared between all the VMs on the XCP. > > Ideally I would like something to automatically manage the bandwidth > such that each VM is capable of using the full 100mbps speed of the > connection, but will be throttled back if the throughput is sustained, > so we have e.g. 24 x 1GB VMs on the host with average of 213GB/month > bandwidth usage each. > Alternatively it might be easier to just route all the virtual > interfaces though a VM than runs pfsense or use tc on the host to just > set some sort of shaping on the physical interface itself, but I > really don''t know the best way to go about it. > > > Things I''ve found so far aren''t so good: > 1 - Limit the interface using the XenCenter GUI... but that means the > VM would never be able to go above about 1mbps, even if it''s sat there > and used no bandwidth for the past week and is well within its quota, > so that''s not ideal. > > 2 - Use sFlow in XCP to capture the data. Well this works for looking > at how much bandwidth they are using, but I haven''t found any existing > tool that will act on that data to do traffic shaping. > > 3 - Use the XAPI calls to check the bandwidth usage. > > > With methods 2 and 3 I guess I could write something that collects the > data and stores it a database table, somehow work out how much the > connection needs to be slowed by and then apply it using the XAPI, but > that seems rather hacky and difficult and there must be a better way? > > > If anyone could give some tips on how to do this I''d really appreciate > it. Basically I just want the quickest and easiest way to make it so > that the server as a whole doesn''t go over its bandwidth limit without > limiting all the guests to a tiny speed individually. > > Thanks! > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, May 23, 2011 at 4:57 AM, Alexandre Bezroutchko <abb@scanit.be> wrote:> Another aspect of problem #3 is how to transfer the policy from sFlow domU > to dom0. I would do it by publishing desired policy on sFlow domU. It can be > a simple as putting a plain text file on HTTP server, or maybe having LDAP > server on sFlow domU, if it is justified. Then a cron job in dom0 would poll > the desired policy from sFlow domU and reconfigure vswitch accordingly. > Alternatively, one can push the config from sFlow domU into dom0 over native > vSwitch API, but I feel it is less preferable from security point of view. I > would prefer to keep my dom0 shut, even from trusted systems I own.The sFlow instrumentation is in dom0 (it''s part of the the Open vSwitch). Traffic to every virtual machine is visible from dom0 so there is no need to install sFlow agents in the virtual machines. The quota controller should be on the management network so it can receive sFlow from dom0 and implement vSwitch configuration changes in dom0. To maintain security, entire system should be part of the control plane and avoid direct interactions with the domUs that are being managed. It might also be interesting to look at managing domU CPU quotas as well as bandwidth quotas: http://blog.sflow.com/2011/06/resource-allocation.html sFlow can be installed in the domUs, but this is only necessary if you are operating in a public cloud where you don''t have access to dom0: http://blog.sflow.com/2010/12/visibility-in-cloud.html http://blog.sflow.com/2011/01/rackspace-cloudservers.html As a cloud customer the controls would be a bit different, you might use the data to decide how many VMs to run etc. Cheers, Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Perhaps you have misunderstood what I have said. I was suggesting to install sFlow _collector_ in domU, not agents. Thanks for the references to sFlow resource management. I''m looking forward to see this approach to become usable in practice. Regards, Alex On 06/05/2011 08:30 PM, Peter Phaal wrote:> On Mon, May 23, 2011 at 4:57 AM, Alexandre Bezroutchko<abb@scanit.be> wrote: >> Another aspect of problem #3 is how to transfer the policy from sFlow domU >> to dom0. I would do it by publishing desired policy on sFlow domU. It can be >> a simple as putting a plain text file on HTTP server, or maybe having LDAP >> server on sFlow domU, if it is justified. Then a cron job in dom0 would poll >> the desired policy from sFlow domU and reconfigure vswitch accordingly. >> Alternatively, one can push the config from sFlow domU into dom0 over native >> vSwitch API, but I feel it is less preferable from security point of view. I >> would prefer to keep my dom0 shut, even from trusted systems I own. > The sFlow instrumentation is in dom0 (it''s part of the the Open > vSwitch). Traffic to every virtual machine is visible from dom0 so > there is no need to install sFlow agents in the virtual machines. The > quota controller should be on the management network so it can receive > sFlow from dom0 and implement vSwitch configuration changes in dom0. > To maintain security, entire system should be part of the control > plane and avoid direct interactions with the domUs that are being > managed. > > It might also be interesting to look at managing domU CPU quotas as > well as bandwidth quotas: > http://blog.sflow.com/2011/06/resource-allocation.html > > sFlow can be installed in the domUs, but this is only necessary if you > are operating in a public cloud where you don''t have access to dom0: > http://blog.sflow.com/2010/12/visibility-in-cloud.html > http://blog.sflow.com/2011/01/rackspace-cloudservers.html > > As a cloud customer the controls would be a bit different, you might > use the data to decide how many VMs to run etc. > > Cheers, > Peter_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users