During my test machine''s idle time I''m running the Linux Test Project on multiple unprivileged Xen domains (i.e., more domains than the number of real processors on the machine), for general stress testing & to see whether any problems crop up. While running this workload I noticed occasional horrendously slow interactive performance in Domain-0 (in which I was not running the LTP). Although I haven''t yet looked in depth at the source of the slowdown, my hypothesis is that Domain-0 blocks handling the high I/O load generated by the unprivileged domains, leading to slow keyboard responses. This brings up several questions: 1. What is the model for allocating processor time to Domain-0? Based on my read of the Xen docs to date, I would expect it to [at least be intended to] have an unbounded priority share of the total processing resources, with some attempt at allocating unprivileged-domain-specific processing (e.g., handling I/O or memory allocation requests) to the requesting unprivileged domain. Along these lines, should there be a parameterizable configuration file for Domain-0? 2. Have there been discussions about allowing multiple simultaneous privileged domains, among which the physical resources are split? Or perhaps "semiprivileged" domains -- for example, a domain that handles all the I/O requests to a particular storage device, or alternatively handles all the I/O requests for a particular class/subset of unprivileged domains? I envision a desire for a master control partition (with priority resource allocation) that forms the root of a hierarchical domain structure, under which one or more I/O partitions execute. (I recall reading about this sort of design in one of the older VMM papers, or perhaps a recent Denali paper?) 3. I don''t seem to be able to create more than 8 VBDs. [I am using xen-2.0.1-src.tgz.] While trying to start 5 domains, each of which had two "scsi" disks (/ and swap), I discovered that the 5th domain wouldn''t start until I removed the swap disks from both the 4th and 5th domains'' configuration files -- i.e., not exceeding 8 VBDs total. A cursory search through the code didn''t reveal any relevant #define''s, and I didn''t see anything about this in the Xen docs, so before I look further I thought I''d ask the list if this is a known limit. (If so, I am surprised the VBD structures aren''t allocated dynamically to prevent this?) A search of the mailing list revealed a message from Ian on 2004-03-10 stating "Mark wrote a pretty good readme on VBDs and put it in the tree", but I couldn''t find the readme; is it still part of the tree? 4. On a loosely related note, what regression tests are used by the Xen developers? (I.e., what should be run before generating patches?) -- Dr. John Linwood Griffin Research Staff Member, Secure Systems Department IBM T.J. Watson Research Center, Hawthorne, New York, USA JLG at us.ibm.com, http://www.research.ibm.com/people/j/jlg/ ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> 1. What is the model for allocating processor time to Domain-0? Based on > my read of the Xen docs to date, I would expect it to [at least be > intended to] have an unbounded priority share of the total processing > resources, with some attempt at allocating unprivileged-domain-specific > processing (e.g., handling I/O or memory allocation requests) to the > requesting unprivileged domain. Along these lines, should there be a > parameterizable configuration file for Domain-0?At the current time, there''s no attempt to ''recharge'' CPU time domain 0 spends doing I/O on behalf of other domains back to the domains. It would actually be pretty tricky to do this accounting within Linux. Fortunately, if you''ve got decent DMA-capable hardware with interrupt coalescing then domain0 typically doesn''t use too much CPU anyhow. Obviously if you''re using loop files for VBDs or other higher-level processing the situation might be different.> 2. Have there been discussions about allowing multiple simultaneous > privileged domains, among which the physical resources are split? Or > perhaps "semiprivileged" domains -- for example, a domain that handles all > the I/O requests to a particular storage device, or alternatively handles > all the I/O requests for a particular class/subset of unprivileged > domains? I envision a desire for a master control partition (with > priority resource allocation) that forms the root of a hierarchical domain > structure, under which one or more I/O partitions execute. (I recall > reading about this sort of design in one of the older VMM papers, or > perhaps a recent Denali paper?)It''s quite possible to have multiple privileged domains owning different bits of hardware -- see the paper on I/O on the website. This can even be used to enable the system to recover from crashes due to buggy device drivers without a reboot. The control tools haven''t always made this easy to set up as its not something that many people do.> 3. I don''t seem to be able to create more than 8 VBDs. [I am using > xen-2.0.1-src.tgz.] While trying to start 5 domains, each of which had > two "scsi" disks (/ and swap), I discovered that the 5th domain wouldn''t > start until I removed the swap disks from both the 4th and 5th domains'' > configuration files -- i.e., not exceeding 8 VBDs total. A cursory search > through the code didn''t reveal any relevant #define''s, and I didn''t see > anything about this in the Xen docs, so before I look further I thought > I''d ask the list if this is a known limit. (If so, I am surprised the VBD > structures aren''t allocated dynamically to prevent this?) A search of the > mailing list revealed a message from Ian on 2004-03-10 stating "Mark wrote > a pretty good readme on VBDs and put it in the tree", but I couldn''t find > the readme; is it still part of the tree?Are you using loop files? If so, Linux has a statically compiled allocation, set to 8 by default. Fixing this and submitting the patch to lkml would a good thing... I don''t believe the VBD code has any such limitations. (We tend to use LVM rather than loop files).> 4. On a loosely related note, what regression tests are used by the Xen > developers? (I.e., what should be run before generating patches?)ltp, ttcp and lmbench are useful. We run osdb/postgresql, postmark and specweb/apache from time to time too. We have a nightly regression test procedure, but its rather old now, being based on RH7.2. As such, it misses some of the excitement caused by NPTL etc. It would be very useful to get an automated regression test setup on something more modern, with as many apps as possible. Ian> -- > Dr. John Linwood Griffin > Research Staff Member, Secure Systems Department > IBM T.J. Watson Research Center, Hawthorne, New York, USA > JLG at us.ibm.com, http://www.research.ibm.com/people/j/jlg/ > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Mark Williamson wrote: l.> > Obviously, dom0 will need control of some physical devices inorder to have > something to boot from (unless ofcourse you use a *really big* initrd :-) ). >hi, In my setup, you do not need a very large initrd to do this. Essentially, the only thing running in dom0 is a simple ICMP/ping server. You send a specially formatted and signed ping payload to dom0, and in response to that it fires up a new VM with a UIP TCP/IP stack inside. You then connect to that TCP/IP stack, and self-migrate or (eventually, not implemented yet) bootstrap directly into the unprivileged guest. Dom0 does not need to have a TCP/IP stack, and at present the ping server main loop is only 50 lines of code. With pre-NGIO version of Xen 1.3 that I am currently using, all you need in your dom0-initrd is the ping server and an ELF image from which to create UIP guest VMs. I am hoping that in the future the parts of xend that have to do with setting up network interfaces to domains will split out into a small C executable, as that would allow me to provide similar functionality for Xen 2.0 or 2.1. Source and binaries are available from http://www.diku.dk/~jacobg/self-migration/ Jacob PS: I do apologize to the long-time subscribers who are fed with me spamming this list with ads for self-migration stuff every time a new user who could find this relevant shows up on the list. Perhaps I can convince the Xen webpage maintainers to link to my stuff as a related project? ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt wrote:>>3. I don''t seem to be able to create more than 8 VBDs. [I am using >>xen-2.0.1-src.tgz.] While trying to start 5 domains, each of which had >>two "scsi" disks (/ and swap), I discovered that the 5th domain wouldn''t >>start until I removed the swap disks from both the 4th and 5th domains'' >>configuration files -- i.e., not exceeding 8 VBDs total. A cursory search >>through the code didn''t reveal any relevant #define''s, and I didn''t see >>anything about this in the Xen docs, so before I look further I thought >>I''d ask the list if this is a known limit. (If so, I am surprised the VBD >>structures aren''t allocated dynamically to prevent this?) A search of the >>mailing list revealed a message from Ian on 2004-03-10 stating "Mark wrote >>a pretty good readme on VBDs and put it in the tree", but I couldn''t find >>the readme; is it still part of the tree? > > > Are you using loop files? If so, Linux has a statically compiled > allocation, set to 8 by default. Fixing this and submitting the > patch to lkml would a good thing... > > I don''t believe the VBD code has any such limitations. (We tend > to use LVM rather than loop files).If you use files as backend for VBDs in domain0, you use loop device. If the CONFIG_BLK_DEV_LOOP in dom0 kernel is compiled as module, you can use max_loop parameter while doing modprobe/insmod. And if it''s compiled directly into the kernel, use max_loop=n boot option. jkt -- cd /local/pub && more beer > /dev/mouth ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Neugebauer, Rolf wrote:> Yes, but in 1.3 device drivers are still in Xen. In 2.0 device drivers > are run in a guest OS (which was what the original poster was looking > for). In order to boot dom0 in this model without physical device > drivers it would need a *really big* initrd (as Mark said) because it > would need to create other VMs from it to provide dom0 with virtual > devices. I''m not sure at all if xend supports this sort of setting.Yep, it seems I misunderstood the original poster''s intention. I guess that in an ideal world every device would provide a Xen VM driver in its firmware, similar to how Open Firmware provides Forth drivers, if I understand that part correctly. But then there is the issue of how to upgrade drivers etc. Btw. I remember some discussion about wrapping NDIS drivers as Xen driver domains, did anyone every try doing that? Jacob ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sun, 19 Dec 2004 00:52:54 -0500, John L Griffin <jlg@us.ibm.com> wrote:> 1. What is the model for allocating processor time to Domain-0? Based on > my read of the Xen docs to date, I would expect it to [at least be > intended to] have an unbounded priority share of the total processing > resources, with some attempt at allocating unprivileged-domain-specific > processing (e.g., handling I/O or memory allocation requests) to the > requesting unprivileged domain. Along these lines, should there be a > parameterizable configuration file for Domain-0?Current schedulers in Xen don''t treat Dom0 differently. In current model, Dom0 plays two major roles: provides a control console for user-interaction; provides I/O handling for other domains. These two roles have different or even conflicting scheduling requirement: the former is response time; the latter is I/O throughput and latency. Ideally, we should put the two parts in two privileged domains and let the scheduler treat them differently. This leads to your next question about multiple privileged domains.> 2. Have there been discussions about allowing multiple simultaneous > privileged domains, among which the physical resources are split? Or > perhaps "semiprivileged" domains -- for example, a domain that handles all > the I/O requests to a particular storage device, or alternatively handles > all the I/O requests for a particular class/subset of unprivileged > domains? I envision a desire for a master control partition (with > priority resource allocation) that forms the root of a hierarchical domain > structure, under which one or more I/O partitions execute. (I recall > reading about this sort of design in one of the older VMM papers, or > perhaps a recent Denali paper?)I agree that multiple privleged domains make resource management both conceptually and structurally clearer. So far, very vey few people actually do that, for several reasons: (1) It requires non-trivial changes to Xend, which people either don''t bother to hack or can''t. (2) With multiple privileged domains, the domain context switch overhead can seriously decrease performance. It makes little sense on current uni- or dual-processors. (3) Managing multiple privilged domains is surely less easy than managing one big Dom0. Surprisingly enough, it''s my phd topic at cambridge to do all above. I''ve offloaded the entire tcp/ip stack to a privilged domain that''s shared by all others. My next step is to introduce the Makefile changes, Xend changes to support multiple privilged domains (i.e. put Xend, NIC device drivers, disk device drivers, tcp/ip stack in different privileged domains). Then I''ll investigate how to schedule all the domains on multi-core chips. Mark Williamson has experiences in putting NIC and disk device drivers into privliged domains. I''ll consult him when questions arise. - Bin ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> 2. Have there been discussions about allowing multiple simultaneous > privileged domains, among which the physical resources are split? Or > perhaps "semiprivileged" domains -- for example, a domain that handles all > the I/O requests to a particular storage device, or alternatively handles > all the I/O requests for a particular class/subset of unprivileged > domains?This has been done - it''s possible to give domans direct control of aa particular device, whilst still limiting their ability to interact with other devices in the system. It should also be possible to have unprivileged domains use different backend domains for different devices. I think there''s some support for this in Xend.> I envision a desire for a master control partition (with > priority resource allocation) that forms the root of a hierarchical domain > structure, under which one or more I/O partitions execute. (I recall > reading about this sort of design in one of the older VMM papers, or > perhaps a recent Denali paper?)You can do that sort of thing if you want, although it''s not a particularly common setup so the control tools don''t support it that well. Obviously, dom0 will need control of some physical devices inorder to have something to boot from (unless ofcourse you use a *really big* initrd :-) ). We''re planning to maintain the flexibility to configure driver domains and possibly extend it some more - it''s rather useful for various things we''ll be looking at in the future. HTH, Mark ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> -----Original Message----- > From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel- > admin@lists.sourceforge.net] On Behalf Of Jacob Gorm Hansen > Sent: 19 December 2004 11:52 > To: Mark Williamson > Cc: xen-devel@lists.sourceforge.net; John L Griffin > Subject: Re: [Xen-devel] Multiple priviliged domains > > Mark Williamson wrote: > l. > > > > Obviously, dom0 will need control of some physical devices inorderto> have > > something to boot from (unless ofcourse you use a *really big*initrd> :-) ). > > > > hi, > > In my setup, you do not need a very large initrd to do this. > Essentially, the only thing running in dom0 is a simple ICMP/ping > server. You send a specially formatted and signed ping payload todom0,> and in response to that it fires up a new VM with a UIP TCP/IP stack > inside. You then connect to that TCP/IP stack, and self-migrate or > (eventually, not implemented yet) bootstrap directly into the > unprivileged guest. Dom0 does not need to have a TCP/IP stack, and at > present the ping server main loop is only 50 lines of code. > > With pre-NGIO version of Xen 1.3 that I am currently using, all youneed> in your dom0-initrd is the ping server and an ELF image from which to > create UIP guest VMs.Yes, but in 1.3 device drivers are still in Xen. In 2.0 device drivers are run in a guest OS (which was what the original poster was looking for). In order to boot dom0 in this model without physical device drivers it would need a *really big* initrd (as Mark said) because it would need to create other VMs from it to provide dom0 with virtual devices. I''m not sure at all if xend supports this sort of setting. If you have a decent server class machine with multiple nics and scsi controllers you could use some of these to get dom0 booting and then fire off other VMs to provide isolated device services to other VMs. That "should" be supported by xend... rolf> I am hoping that in the future the parts of xend > that have to do with setting up network interfaces to domains willsplit> out into a small C executable, as that would allow me to providesimilar> functionality for Xen 2.0 or 2.1. > > Source and binaries are available from > http://www.diku.dk/~jacobg/self-migration/ > > Jacob > > PS: I do apologize to the long-time subscribers who are fed with me > spamming this list with ads for self-migration stuff every time a new > user who could find this relevant shows up on the list. Perhaps I can > convince the Xen webpage maintainers to link to my stuff as a related > project? > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from realusers.> Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Yes, but in 1.3 device drivers are still in Xen. In 2.0 device drivers > are run in a guest OS (which was what the original poster was looking > for). In order to boot dom0 in this model without physical device > drivers it would need a *really big* initrd (as Mark said) because it > would need to create other VMs from it to provide dom0 with virtual > devices. I''m not sure at all if xend supports this sort of setting. > > If you have a decent server class machine with multiple nics and scsi > controllers you could use some of these to get dom0 booting and then > fire off other VMs to provide isolated device services to other VMs. > That "should" be supported by xend...One way we could do this would be to have Grub package up all the driver domains as modules loaded ahead of dom0. Each driver domain would have a very simple initrd containing one file that enables it listen on the control interface and receive configuration instructions. Dom0 would then issue configuration instructions to the driver domains and then bind its virtual drivers. This is another argument for splitting this kind of functionality out of xend into a set of standalone libraries with a daemon to give a network facing interface to it. Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel