Hi, I''ve been experimenting with Xen on and off for the past couple months and it''s come a long way and the roadmap looks great. What really made it a possibility for me was the loopback support for vbd''s. I''m getting pretty far with it, but I have a few questions that hopefully I can get answered. 1) IP Stealing - I want to be able to assign IP''s to a domain and for it to be limited to them only. I tried setting the anti-spoof option to on in the xend config file and it seemed to add a rule to the FORWARD table that matched base on the physdev eth0, but it just blocked any domain from accessing the network. I was setting the IP using the "ip" config parameter in the domains config file. Is there also a way to assign multiple IP''s, perhaps multiple ip config options or comma deliminted? If the anti-spoof isn''t what I think is it, would using something like ebtables be usable to provide what I want to do? My problem is it looks like the bridge interface is based on the domain id and domain id''s are still dynamically assigned (poor IMHO). What is the status on static domain id''s? It''d be good for many things, another problem is the telnet console is based off of the domain id... if I handed a Xen server off to a customer i''d have to basically write an interface that they''d have to log in to to find out what port they should use since the last server reboot, it''d be easier to just say "telnet to this host on port 9605 to gain access". 2) CPU QoS - This looks good, even though this is for selling a service where each customer should receive their guaranteed amount it looks like the atropos scheduler is broken and it''s better to use BVT for now. What i''m confused on is how to break down the numbers based on what kind of priority each domain should get. ie. If I want the base accounts to get a priority of 1 out of 4, next 2 out of 4 (total units). That doesn''t even make complete sense to many of you as i''m really that confused as to what to set these variables to. Are there any more specific documents or example configs for using the CPU schedulers? 3) HyperThreading - Good or bad with Xen? I''d be using the 2.6 kernel for domain0 which I believe has full SMT support so I would assume good. Also, since each domain is assign 1 CPU this would make the possibilities out of 4 instead of 2 so i''m not sure if that benefits over the SMT scheduler or if domains are even threaded so they benefit from having the extra logical CPU''s and it just makes the load un-even and increases scheduling latency. 4) I/O QoS - This is just round robin for now? I see it on the roadmap. I have already envisioned the scenario with someone purchasing a server with 64MB physical ram and adding a 512MB swap file inside their server and just completely thrashing the disks. Is the current system good enough to handle this (assuming some good SCSI disks are being used) ? Any answers to these questions would be great. I''m also all geared up for some discussion/arguing :) ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Mark A. Williamson
2004-Nov-12 01:01 UTC
Re: [Xen-devel] ip stealing, cpu qos, and hyperthreading
> If the anti-spoof isn''t what I think is it, would using something like > ebtables be usable to provide what I want to do?I don''t know about the antispoof stuff but ebtables should work fine. Anything that''ll work with a normal Linux ethernet device should Just Work (TM) for a virtual interface.> My problem is it looks > like the bridge interface is based on the domain id and domain id''s are > still dynamically assigned (poor IMHO). What is the status on static > domain id''s?The /etc/xen/scripts/vif-bridge script is used to configure firewalling, etc for each VIF that''s created, so you don''t need to statically specify interface names rules. You can make a derivative of this script and tell xend-config.sxp where to find it. I agree that it might be nice to just set up a static set of rules but with appropriate script magic, using dynamic configuration can probably be made just as easy.> It''d be good for many things, another problem is the > telnet console is based off of the domain id... if I handed a Xen server > off to a customer i''d have to basically write an interface that they''d > have to log in to to find out what port they should use since the last > server reboot, it''d be easier to just say "telnet to this host on port > 9605 to gain access".You can specify a console port using a console= entry in your domain configuration file. This is missing from the docs at the moment... *note to self...* Whilst you mention it, telnet is not as good as xencons for this: with telnet, console resizing (and some other stuff) don''t work properly. With xencons, it''s just like being there ;-)> 2) CPU QoS - This looks good, even though this is for selling a service > where each customer should receive their guaranteed amount it looks like > the atropos scheduler is broken and it''s better to use BVT for now.Yeah, sorry. This ought to be working by the 2.1 release (it''s on my "fix this on a slow weekend") list.> What i''m confused on is how to break down the numbers based on what kind > of priority each domain should get. ie. If I want the base accounts to > get a priority of 1 out of 4, next 2 out of 4 (total units). That > doesn''t even make complete sense to many of you as i''m really that > confused as to what to set these variables to. Are there any more > specific documents or example configs for using the CPU schedulers?> 3) HyperThreading - Good or bad with Xen? I''d be using the 2.6 kernelHmmmm. Depends what you''re doing with it. We found an improvement in networking performance when running domains on separate HyperThreads on a UP box, compared to disabling HT. Not everyone has found this, however - it''ll depend on your processor model and workload.> for domain0 which I believe has full SMT support so I would assume > good. Also, since each domain is assign 1 CPU this would make the > possibilities out of 4 instead of 2 so i''m not sure if that benefits > over the SMT scheduler or if domains are even threaded so they benefit > from having the extra logical CPU''s and it just makes the load un-even > and increases scheduling latency.Xen will allow you to run a uniprocessor domain on each HyperThread. The domains won''t see an HT processor. I''m skeptical over the benefits of this, since the domains on each core will compete for cache space, etc. This hasn''t been benchmarked so YMMV. You can use the hyperthreads to provide performance isolation - e.g. if a domain is blocking lots and ends up letting by a CPU intensive domain hog the processor, a quick fix is to just give them a hyperthread each.> 4) I/O QoS - This is just round robin for now? I see it on the > roadmap. I have already envisioned the scenario with someone purchasing > a server with 64MB physical ram and adding a 512MB swap file inside > their server and just completely thrashing the disks. Is the current > system good enough to handle this (assuming some good SCSI disks are > being used) ?For network, you should be able to use any standard Linux QoS tools you want. For block IO, the requests are batched and serviced in a round robin fashion. This (obviously) doesn''t provide service differentiation, or accounting. I think this is still on the roadmap and would certainly be cool to have. HTH, Mark> Any answers to these questions would be great. I''m also all geared up > for some discussion/arguing :) > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: > Sybase ASE Linux Express Edition - download now for FREE > LinuxWorld Reader''s Choice Award Winner for best database on Linux. > http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Thu, 2004-11-11 at 20:01, Mark A. Williamson wrote:> > If the anti-spoof isn''t what I think is it, would using something like > > ebtables be usable to provide what I want to do? > > I don''t know about the antispoof stuff but ebtables should work fine. > Anything that''ll work with a normal Linux ethernet device should Just Work > (TM) for a virtual interface. >Ok, ebtables is it then.> > My problem is it looks > > like the bridge interface is based on the domain id and domain id''s are > > still dynamically assigned (poor IMHO). What is the status on static > > domain id''s? > > The /etc/xen/scripts/vif-bridge script is used to configure firewalling, etc > for each VIF that''s created, so you don''t need to statically specify > interface names rules. You can make a derivative of this script and tell > xend-config.sxp where to find it. > > I agree that it might be nice to just set up a static set of rules but with > appropriate script magic, using dynamic configuration can probably be made > just as easy. >I have only looked at this real quick, but it looks like this should do it: iptables ${iptcmd} FORWARD -m physdev --physdev-in ${vif} -s ${addr} -j ACCEPT if not, I guess i''d just have to add a ebtables line right below it. I''ll have to experiment some more.> > It''d be good for many things, another problem is the > > telnet console is based off of the domain id... if I handed a Xen server > > off to a customer i''d have to basically write an interface that they''d > > have to log in to to find out what port they should use since the last > > server reboot, it''d be easier to just say "telnet to this host on port > > 9605 to gain access". > > You can specify a console port using a console= entry in your domain > configuration file. This is missing from the docs at the moment... *note to > self...* > > Whilst you mention it, telnet is not as good as xencons for this: with telnet, > console resizing (and some other stuff) don''t work properly. With xencons, > it''s just like being there ;-)Ah... I didn''t even know xencons existed. Yes, it is much better.. I had many problems with telnet dependent on the terminal emulation. Does anyone see a problem with creating a user account for each domain in domain0 whose shell runs xencons connecting to localhost with their console port? Since it''s technically the login shell it shouldn''t pose a security problem? I''d prefer that anyhow since it would be over ssh.> > > 2) CPU QoS - This looks good, even though this is for selling a service > > where each customer should receive their guaranteed amount it looks like > > the atropos scheduler is broken and it''s better to use BVT for now. > > Yeah, sorry. This ought to be working by the 2.1 release (it''s on my "fix > this on a slow weekend") list.Ok... that''s great, but any input as far as BVT values/meaning/docs/examples?> > > What i''m confused on is how to break down the numbers based on what kind > > of priority each domain should get. ie. If I want the base accounts to > > get a priority of 1 out of 4, next 2 out of 4 (total units). That > > doesn''t even make complete sense to many of you as i''m really that > > confused as to what to set these variables to. Are there any more > > specific documents or example configs for using the CPU schedulers? > > > 3) HyperThreading - Good or bad with Xen? I''d be using the 2.6 kernel > > Hmmmm. Depends what you''re doing with it. We found an improvement in > networking performance when running domains on separate HyperThreads on a UP > box, compared to disabling HT. Not everyone has found this, however - it''ll > depend on your processor model and workload.Yes.. I read that thread. These are Xeon''s (only 512k) cache, I believe the consensus was HT on a regular P4 was poor, but it helped on Xeon''s.> > > for domain0 which I believe has full SMT support so I would assume > > good. Also, since each domain is assign 1 CPU this would make the > > possibilities out of 4 instead of 2 so i''m not sure if that benefits > > over the SMT scheduler or if domains are even threaded so they benefit > > from having the extra logical CPU''s and it just makes the load un-even > > and increases scheduling latency.> Xen will allow you to run a uniprocessor domain on each HyperThread. The > domains won''t see an HT processor. I''m skeptical over the benefits of this, > since the domains on each core will compete for cache space, etc. This > hasn''t been benchmarked so YMMV. > > You can use the hyperthreads to provide performance isolation - e.g. if a > domain is blocking lots and ends up letting by a CPU intensive domain hog the > processor, a quick fix is to just give them a hyperthread each.This is exactly what I was thinking, regarding the cache coherency and CPU intensive domains. It seems it is both a pro and a con depending on the workload.> > > 4) I/O QoS - This is just round robin for now? I see it on the > > roadmap. I have already envisioned the scenario with someone purchasing > > a server with 64MB physical ram and adding a 512MB swap file inside > > their server and just completely thrashing the disks. Is the current > > system good enough to handle this (assuming some good SCSI disks are > > being used) ? > > For network, you should be able to use any standard Linux QoS tools you want. > For block IO, the requests are batched and serviced in a round robin fashion. > This (obviously) doesn''t provide service differentiation, or accounting. I > think this is still on the roadmap and would certainly be cool to have. >That''s what I thought also... I know for a fact the domain swapping a lot will happen so there should definitely be a way to penalize for it rather than having it slow down the other domains. At least there is _something_ in place now though. ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel