Hi, with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it worked very well and accurate with the HTB class: Limit: 1mbit / actual rate: 116.2KB/s (given by scp) Now i looked at 3.0.3rc1 and noticed that traffic shaping using the same setup as before does not work as accurate as before (same hardware): Limit: 1mbit / actual rate: 449.0KB/s (given by scp) Interstingly shaping of the incoming transfer rate at the vif*.* interfaces works correct. The problem is reproducible on my side and for me it seems to be a problem with the new Xen-kernel. Here is a stripped down version of my tc setup for you to verify the problem: ----<snip>---- tc qdisc add dev peth0 root handle 1:0 htb tc class add dev peth0 parent 1:0 classid 1:1 htb rate 1mbit tc qdisc add dev peth0 parent 1:1 handle 11: prio tc filter add dev peth0 parent 1:0 protocol ip prio 1 u32 match ip src \ 172.17.0.100/32 flowid 1:1 ----<snap>---- 172.17.0.100/32 is the IP of my Domain-0. Greetings, -timo -- Timo Benk - B1 Systems GmbH (http://www.b1-systems.de) Jabber ID: fry@jabber.org - ICQ ID: #241877854 PGP Public Key: http://m28s01.vlinux.de/b1_gpg_key.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Matt Ayres
2006-Oct-08 16:17 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
Timo Benk wrote:> Hi, > > with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it > worked very well and accurate with the HTB class: >Why not just use Xen''s own rate limiting system that works directly on the vif device line? Regards, Matt _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-08 17:07 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 8/10/06 5:17 pm, "Matt Ayres" <matta@tektonic.net> wrote:>> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it >> worked very well and accurate with the HTB class: > > Why not just use Xen''s own rate limiting system that works directly on > the vif device line?Yes, that''s the best way. It''s odd that TC behaviour has changed though. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Timo Benk
2006-Oct-08 18:30 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
Keir Fraser wrote:> On 8/10/06 5:17 pm, "Matt Ayres" <matta@tektonic.net> wrote: > >> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it >> worked very well and accurate with the HTB class: >> Why not just use Xen''s own rate limiting system that works directly on >> the vif device line? > > Yes, that''s the best way. It''s odd that TC behaviour has changed though.AFAIK Xens rate limiting feature is limited to outgoing traffic and i implement a rather complex (complex at least with tc) scenario: - each domain will get a guaranteed minimum bandwidth - if one domain is not using all of the guaranteed bandwidth, the not used bandwidth can be reused by other domains in addition to the guaranteed bandwidth they own - in- and outcoming traffic will be limited That is perfectly possible with tc and the HTB queue, but imho not with Xens rate limiting feature. The other reason was that in my experience rate limiting in 3.0.2 was buggy and leaving undestroyable zombie domains. BTW a quick test on 3.0.3rc2 showed the following behaviour of Xen rate limiting on my host: ---<snip>--- vif = [ ''rate=1MB/s'' ] ---<snap>--- xendom0:~ # xm shutdown xendom2 # rate limited domain [... wait until xendom2 shutdown finished ...] xendom0:~ # xm list Error: Device 0 not connected Usage: xm list [options] [Domain, ...] List information about all/some domains. -l, --long Output all VM details in SXP --label Include security labels However, besides my personal problem with my setup that is not working anymore i think tc is a important feature in QoS environments and should work as solid on a Xen kernel as it runs on a legacy Linux kernel. Greetings, -timo -- Timo Benk - B1 Systems GmbH (http://www.b1-systems.de) Jabber ID: fry@jabber.org - ICQ ID: #241877854 PGP Public Key: http://m28s01.vlinux.de/b1_gpg_key.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-09 12:38 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 8/10/06 19:30, "Timo Benk" <Timo.Benk@gmx.de> wrote:> However, besides my personal problem with my setup that is not working > anymore i think tc is a important feature in QoS environments and should > work as solid on a Xen kernel as it runs on a legacy Linux kernel.It sounds like maybe something has changed in the kernel config meaning that your packets are no longer passing through the traffic classifier. Apart from that, nothing has changed that should cause generic tc code to not work properly any more. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-09 15:40 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 7/10/06 20:21, "Timo Benk" <Timo.Benk@gmx.de> wrote:> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it > worked very well and accurate with the HTB class: > Limit: 1mbit / actual rate: 116.2KB/s (given by scp) > > Now i looked at 3.0.3rc1 and noticed that traffic shaping using the same > setup as before does not work as accurate as before (same hardware): > Limit: 1mbit / actual rate: 449.0KB/s (given by scp)This is not a Xen-specific bug. The problem is that now we support TSO, packets are larger than the maximum that TC expects and so we overflow the pre-computed rate table (and our 8kB packets get treated as 1.6kB packets). To see this, run ''tc -s class show dev peth0''. You''ll see a ''giants'' field that is non-zero -- those are all packets that were too big to handle properly. The fix is simple. Add an explicit mtu parameter to the htb class: tc class add dev peth0 parent 1:0 classid 1:1 htb rate 1mbit mtu 16000 Regards, Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-09 16:18 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 8/10/06 19:30, "Timo Benk" <Timo.Benk@gmx.de> wrote:> BTW a quick test on 3.0.3rc2 showed the following behaviour of Xen rate > limiting on my host: > ---<snip>--- > vif = [ ''rate=1MB/s'' ] > ---<snap>--- > > xendom0:~ # xm shutdown xendom2 # rate limited domain > [... wait until xendom2 shutdown finished ...] > > xendom0:~ # xm list > Error: Device 0 not connected > Usage: xm list [options] [Domain, ...]This is now fixed (the domain doesn''t hang around as a zombie any more), but the built-in rate limiting still seems broken (guest loses network connectivity after a while). I''ll have to investigate further. As for save/restore of domains with non-dom0 backends. That''s pushing the envelope somewhat and might require some changes to our tools. But at the very least it shouldn''t leave zombies hanging around. I''ll give your setup a test and see what can be done. -- keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-10 10:16 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 9/10/06 17:18, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:> As for save/restore of domains with non-dom0 backends. That''s pushing the > envelope somewhat and might require some changes to our tools. But at the > very least it shouldn''t leave zombies hanging around. I''ll give your setup a > test and see what can be done.Actually, you cannot save/restore backend domains at all, even when they are not actually configured to access real hardware. We should give a better error message about that, of course. ;-) The zombie xendom2 will exist until xendom1 is destroyed, for sure. This is because xendom1 holds onto some xendom2 resources. *But* does xendom2 continue to exist as a zombie even after you forcibly ''xm destroy'' xendom1? That would definitely be a bug we want to fix. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Timo Benk
2006-Oct-10 10:50 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
Keir Fraser wrote:> On 9/10/06 17:18, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote: > >> As for save/restore of domains with non-dom0 backends. That''s pushing the >> envelope somewhat and might require some changes to our tools. But at the >> very least it shouldn''t leave zombies hanging around. I''ll give your setup a >> test and see what can be done. > > Actually, you cannot save/restore backend domains at all, even when they are > not actually configured to access real hardware. We should give a better > error message about that, of course. ;-)If it is not possible to save/restore a backend domain it is ok. The reason why stumbled across this problem was that the xendomains script tried to save the domains on reboot of Domain-0. That failed of course. Then xendomains tries to shutdown these remaining domains. That fails too and the host did not reboot for a long time. IMHO that is the main problem. The xen scripts that lie around in the system should be aware of the fact that certain domains cannot be saved, and handle this situation gracefully.> The zombie xendom2 will exist until xendom1 is destroyed, for sure. This is > because xendom1 holds onto some xendom2 resources. *But* does xendom2 > continue to exist as a zombie even after you forcibly ''xm destroy'' xendom1?You are right. After destroying xendom1 the zombie domain went away too. Greetings, -timo -- Timo Benk - Jabber ID: fry@jabber.org - ICQ ID: #241877854 PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-10 11:36 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 10/10/06 11:50, "Timo Benk" <timo.benk@gmx.de> wrote:> The xen scripts that lie around in the system should be aware of the > fact that certain domains cannot be saved, and handle this situation > gracefully.I think we''ll add a feature flag that guest kernel can write to xenstore to indicate whether they support save/restore. Alternatively the tools could remember that they set up a guest as a backend of another. They probably do that already in fact. Anyway, this is definitely fixable for 3.0.4. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Oct-10 12:50 UTC
Re: [Xen-devel] Linux Traffic Shaping broken in 3.0.3rc1 ?
On 9/10/06 17:18, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:> This is now fixed (the domain doesn''t hang around as a zombie any more), but > the built-in rate limiting still seems broken (guest loses network > connectivity after a while). I''ll have to investigate further.This is now fixed. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anand Gupta wrote:> Would you mind sharing your tc rules / scripts and how you have used > them to > shape traffic for domU''s ? Will appreciate the help. > > On 1/7/07, Timo Benk <timo.benk@gmx.de> wrote: >> >> why choppy? It works on my side. BTW, no ebtables is needed to achieve >> traffic shaping. You can stick your tc rules inside Dom0 at the >> vif-Interfaces of the gateway domain. It is nothing more than tc-magic;-)You can use the IFB-Device for shaping of the incoming traffic: (The following tc commands are not tested, but they should give you a good starting point) #### ## IFB Rules # modprobe ifb # tc qdisc add dev eth0 ingress # tc filter add dev eth0 parent ffff: \ protocol ip prio 10 u32 \ match ip dst 0/0 flowid 1:1 \ action mirred egress redirect dev ifb0 #### ## HTB QDiscs # tc qdisc add dev ifb0 root handle 1:0 htb # tc class add dev ifb0 parent 1:0 \ classid 1:1 htb \ rate 1mbit mtu 16000 quantum 16000 # tc class add dev ifb0 parent 1:0 \ classid 1:2 htb \ rate 10mbit mtu 16000 quantum 16000 #### ## Filter Definitions # tc filter add dev ifb0 parent 1:0 \ protocol ip prio 1 u32 \ match ip dst 172.17.0.101/32 flowid 1:1 # tc filter add dev ifb0 parent 1:0 \ protocol ip prio 1 u32 \ match ip dst 172.17.0.102/32 flowid 1:2 Greetings, -timo -- Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 PGP Public Key: http://www.m28s01.vlinux.de/timo_benk_gpg_key.asc _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks for the rules and quick response. On 4/28/07, Timo Benk <timo.benk@gmx.de> wrote:> > Anand Gupta wrote: > > Would you mind sharing your tc rules / scripts and how you have used > > them to > > shape traffic for domU''s ? Will appreciate the help. > > > > On 1/7/07, Timo Benk <timo.benk@gmx.de> wrote: > >> > >> why choppy? It works on my side. BTW, no ebtables is needed to achieve > >> traffic shaping. You can stick your tc rules inside Dom0 at the > >> vif-Interfaces of the gateway domain. It is nothing more than > tc-magic;-) > > You can use the IFB-Device for shaping of the incoming traffic: > > (The following tc commands are not tested, but they should give you a > good starting point) > > #### > ## IFB Rules > > # modprobe ifb > # tc qdisc add dev eth0 ingress > # tc filter add dev eth0 parent ffff: \ > protocol ip prio 10 u32 \ > match ip dst 0/0 flowid 1:1 \ > action mirred egress redirect dev ifb0 > > #### > ## HTB QDiscs > > # tc qdisc add dev ifb0 root handle 1:0 htb > # tc class add dev ifb0 parent 1:0 \ > classid 1:1 htb \ > rate 1mbit mtu 16000 quantum 16000 > > # tc class add dev ifb0 parent 1:0 \ > classid 1:2 htb \ > rate 10mbit mtu 16000 quantum 16000 > > #### > ## Filter Definitions > > # tc filter add dev ifb0 parent 1:0 \ > protocol ip prio 1 u32 \ > match ip dst 172.17.0.101/32 flowid 1:1 > > # tc filter add dev ifb0 parent 1:0 \ > protocol ip prio 1 u32 \ > match ip dst 172.17.0.102/32 flowid 1:2 > > Greetings, > -timo > -- > Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 > PGP Public Key: http://www.m28s01.vlinux.de/timo_benk_gpg_key.asc > >-- regards, Anand Gupta _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users