I am trying to write a script to handle setup of domains using virtual block devices for their root FS and have been unable to get the virtual devices to work with any sort of consistency. This is using xen-1.1.bk on Redhat 7.3 (I had to rebuild all the tool binaries to run on 7.3). Anyway, I would first like to make sure I am using the vd/vbd stuff correctly. Here is an example of the problem I see. I have a partition to back the vds: xenctl partitions add -phdb2 -f hdb2 is of course the second partition on the second disk. I am assuming that the partition doesn''t need to have a filesystem on it, i.e., it just uses the raw blocks for storage? I create three vds: xenctl vd create -nD1 -s128M xenctl vd create -nD2 -s128M xenctl vd create -nD3 -s128M Then I use the returned keys to create virtual block devices: xenctl vbd create -n0 -w -v0 -k3827077824 xenctl vbd create -n0 -w -v1 -k8055618233 xenctl vbd create -n0 -w -v2 -k6945314927 Write access is given to domain0 (-n0) so I can initialize it. My assumption here is that the <vdb_num> given to the -v option translates into /dev/xvda for -v0, /dev/xvdb for -v1, etc. Is that correct? BTW, do I need to create a distinct virtual device which grants access to the domain whose kernel is going to use the virtual disk as its root? Currently I do not, I just set root=/dev/xvdN in xi_build where xvdN is the device I create/use here for dom0 initialization. Do I need to do a "xenctl physical grant" for either the virtual block device or the partition on which the virtual disks reside? Moving on: xen_refresh_dev /dev/xvda xen_refresh_dev /dev/xvdb xen_refresh_dev /dev/xvdc I read in the mailing list archive that this refresh is needed... Now I run "fdisk -l" on each of them and get: Disk /dev/xvda: 255 heads, 63 sectors, 16 cylinders Disk /dev/xvdb: 255 heads, 63 sectors, 32 cylinders Disk /dev/xvdc: 255 heads, 63 sectors, 48 cylinders Note the ever increasing number of cylinders. This makes mkfs think that xvdb and xvdc are larger than they really are. mkfs does succeed, but you get a lot of: DOM0: Bad return from blkdev data request: 1 on the console. And if you try to fsck it or actually use it, you get filesystem errors. If I explicitly tell mkfs how big the partition is, everything works fine. So someone is just reporting the geometry wrong, probably forgetting to subtract off a non-zero starting location. ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, 2003-12-19 at 18:23, Mike Hibler wrote:> I am trying to write a script to handle setup of domains using virtual > block devices for their root FS and have been unable to get the virtual > devices to work with any sort of consistency. This is using xen-1.1.bk > on Redhat 7.3 (I had to rebuild all the tool binaries to run on 7.3).<-- snip --> Everything looks good so far but my memory is foggy...> Write access is given to domain0 (-n0) so I can initialize it. My assumption > here is that the <vdb_num> given to the -v option translates into /dev/xvda > for -v0, /dev/xvdb for -v1, etc. Is that correct?Yes that is correct.> > BTW, do I need to create a distinct virtual device which grants access to > the domain whose kernel is going to use the virtual disk as its root? > Currently I do not, I just set root=/dev/xvdN in xi_build where xvdN is the > device I create/use here for dom0 initialization. Do I need to do a > "xenctl physical grant" for either the virtual block device or the > partition on which the virtual disks reside?See below.> Moving on: > > xen_refresh_dev /dev/xvda > xen_refresh_dev /dev/xvdb > xen_refresh_dev /dev/xvdc > > I read in the mailing list archive that this refresh is needed... > Now I run "fdisk -l" on each of them and get: > > Disk /dev/xvda: 255 heads, 63 sectors, 16 cylinders > Disk /dev/xvdb: 255 heads, 63 sectors, 32 cylinders > Disk /dev/xvdc: 255 heads, 63 sectors, 48 cylinders > > Note the ever increasing number of cylinders. This makes mkfs think that > xvdb and xvdc are larger than they really are. mkfs does succeed, but you > get a lot of: >Ahh yes, this looks like a 1.0 problem I had. I haven''t tried the 1.1 myself but Ian tells me there are people using the 1.1 VD/VBD stuff. I can confirm the behavior you are seeing: /dev/xvdb''s start offset begins at ''0 instead of +sizeof(xvda).'' Some additional information about my experiences in 1.0/1.1: If you set root=/dev/sda5 and then do a physical grant sda5 you''ll get some interesting results when you write to the filesystem. What I believe happens here is the writes are executed twice in the 1.0 tree. This doesn''t seem to happen in the 1.1 tree. To answer your question about granting physical access, the way it is suppose to work (I believe) is that root=/dev/XXXX implies a physical grant but that is not the case in the 1.0 tree. In the 1.1 tree, that seems to work but only on the first disk. I could never get the XenoLinux domains to see root=/dev/sdb* unless I did a physical grant in their startup script. I suppose I should test under -unstable but I have my own unstable BK tree that I experiment on. I hope all that is accurate, a lot of my devel time happened during 11PM - 7AM GMT -5 so its a little fuzzy unless I''m sitting at that workstation. Stephen Evanchik ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Now I run "fdisk -l" on each of them and get: > > Disk /dev/xvda: 255 heads, 63 sectors, 16 cylinders > Disk /dev/xvdb: 255 heads, 63 sectors, 32 cylinders > Disk /dev/xvdc: 255 heads, 63 sectors, 48 cylinders > > Note the ever increasing number of cylinders. This makes mkfs think that > xvdb and xvdc are larger than they really are. mkfs does succeed, but you > get a lot of:Weird -- We''ve had some success reports with VDs working under 1.1, but you and Stephen have hit the same problem. This area of code has been entirely rewritten for 1.2, so its hard to muster much enthusiasm for fixing it in 1.1, but I can take a look. Please can you the output of "sfdisk -d" on the VD''s. Thanks, Ian ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
>I read in the mailing list archive that this refresh is needed... >Now I run "fdisk -l" on each of them and get: > > Disk /dev/xvda: 255 heads, 63 sectors, 16 cylinders > Disk /dev/xvdb: 255 heads, 63 sectors, 32 cylinders > Disk /dev/xvdc: 255 heads, 63 sectors, 48 cylinders > >Note the ever increasing number of cylinders. This makes mkfs think that >xvdb and xvdc are larger than they really are. mkfs does succeed, but you >get a lot of: > > DOM0: Bad return from blkdev data request: 1 > >on the console. And if you try to fsck it or actually use it, you get >filesystem errors. If I explicitly tell mkfs how big the partition is, >everything works fine. So someone is just reporting the geometry wrong, >probably forgetting to subtract off a non-zero starting location.Hmm -- the ''geometry'' is [supposed to be] hardwired in 1.1 -- see the ioctl code in xenolinux-2.4.23-sparse/arch/drivers/block/xl_block.c Can you "strace fdisk -l" to see how fdisk is computing the values? As Ian mentioned previously, this code has all been rewritten for 1.2; might be worth giving that a try (although of course 1.2. is still awaiting the high-level VD tools...) S.
> Note that fdisk is claiming to have 48 cylinders. Presumably it is the > BLKGETSIZE (which purports to return the "device size") that is at fault, > though there is the fstat call there as well. Recall this is Redhat 7.3, > not Redhat 9 in case that matters. > > Anyway, by explicitly setting the size when doing the partition table > initialization, all is well.Ah, I see the problem. I''ll check a fix into the 1.1 tree which should sort things out for you. 1.2 does not have this problem as teh code is substantially different. -- Keir
> I didn''t mention that I was not initializing a partition table on those, > didn''t realize I had to until I got further with booting Linux off of one.As you''ve probably figured out, Linux is pretty good at spotting whether a ''disk'' has a partition table on it or not -- you can quite happily partition virtual disks, or use them raw without a partition table. Given that they''re virtual anyway, I suppose there''s a fair argument for doing the latter and just having multiple VDs. Glad to hear the VD size bug is now fixed. Cheers, Ian (signing off ''till the New Year)
Over a month ago, I mentioned I was working to configure Xen as a set of virtual machines for Grid software development. The outcome is here: http://www.arsc.edu/~newby/grid/xen-for-grid.txt Yes, it works! Since the Xen design (for xenctl and friends) has changed since I started, my steps are somewhat outdated. Most of the problems I describe in the file above were with Globus, not Xen, but it did take awhile to get Xen built correctly, both the basic kernel build and boot, and then the configuration for additional domains. There is still a lot of Grid development to me do, but once it''s relatively stable and Xen 1.2 is out, I''ll upgrade Xen. The main items I''m desiring from Xen: - large memory support (up to 4GB per virtual domain) - SMP support (it looks like my dual processor system never uses the second processor at all, even with multiple Xen domains) My only general comment for Xen developers is to update this page: http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ Currently, it only mentions the 1.0 CD, not how to get and build 1.1 or 1.2rc , and it doesn''t include a link to the HOWTO (which moved since the last time I looked at it). The BitKeeper instructions on the Download page should be updated for 1.2rc, too. Otherwise, you''ll have any new user trying to get 1.0 installed, which doesn''t make much sense. Thanks for a great program! This is really helpful for setting up simulated clusters and Grid environments, and I''ll be sharing my results with the Grid developers I know. -- Greg Dr. Gregory B. Newby, Research Faculty, Arctic Region Supercomputing Center University of Alaska Fairbanks. PO Box 756020, Fairbanks, AK 99775 e: newby AT arsc.edu v: 907-474-7160 f: 907-474-5494 w: www.arsc.edu/~newby ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> There is still a lot of Grid development to me do, but once it''s > relatively stable and Xen 1.2 is out, I''ll upgrade Xen. The main > items I''m desiring from Xen: > > - large memory support (up to 4GB per virtual domain)This shouldn''t be too hard, and I expect it will ''just happen'' when we do the 2.6 port. We''re going to wait for 2.6 to settle down a bit first, though.> - SMP support (it looks like my dual processor system never uses > the second processor at all, even with multiple Xen domains)SMP guests is on the todo list, but Xen is already itself SMP capable. All of our test machines are dual Xeon boxes (effectively four CPUs), so we regularly exercise it in SMP mode. Check the boot messages to see how many CPUs Xen finds. Also, if you''ve started multiple domains, do a ''domain list'' and see which CPUs they''re assigned to. Alternatively, hit ''d'' on the serial line (or scroll lock -h on the keyboard) to get a dump of the run queue on each CPU.> My only general comment for Xen developers is to update this page: > http://www.cl.cam.ac.uk/Research/SRG/netos/xen/Yep, sorry about that. Our documentation and web site are generally poor. I promise to update things a bit when 1.2 is released, but volunteers to write a FAQ or do other documentation would be very helpful.> Thanks for a great program! This is really helpful for setting > up simulated clusters and Grid environments, and I''ll be sharing > my results with the Grid developers I know.Thanks for the vote of confidence! Cheers, Ian ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Tue, Dec 23, 2003 at 08:45:44PM +0000, Ian Pratt wrote:> > > There is still a lot of Grid development to me do, but once it''s > > relatively stable and Xen 1.2 is out, I''ll upgrade Xen. The main > > items I''m desiring from Xen: > > > > - large memory support (up to 4GB per virtual domain) > > This shouldn''t be too hard, and I expect it will ''just happen'' > when we do the 2.6 port. We''re going to wait for 2.6 to settle > down a bit first, though.Good to know - I will probably wait for 2.6 before upgrading my Xen.> > - SMP support (it looks like my dual processor system never uses > > the second processor at all, even with multiple Xen domains) > > SMP guests is on the todo list, but Xen is already itself SMP > capable. All of our test machines are dual Xeon boxes > (effectively four CPUs), so we regularly exercise it in SMP mode. > > Check the boot messages to see how many CPUs Xen finds. Also, if > you''ve started multiple domains, do a ''domain list'' and see which > CPUs they''re assigned to. Alternatively, hit ''d'' on the serial > line (or scroll lock -h on the keyboard) to get a dump of the > run queue on each CPU.Hmmm, it only finds one. I thought this was all it supported, from prior messages to the list. Note, I''m using a pre-1.1 Xen, so if it was added later that would explain.>From dmesg:Kernel command line: /xenolinux.gz root=/dev/sda2 ro console=xencons0 DOMID=0 Initializing CPU#0 Xen reported: 1694.911 MHz processor. Just the one CPU...from "xenctl domain list", all domains are on CPU0. My grub boot args specify "noht", but I''m pretty sure the kernel is SMP (it''s the one from you folks, with only ReiserFS added). Maybe "noht" is the problem (the 1.7Ghz Xeon processors don''t support hyperthreading, so I presumed I needed noht). I know the second CPU works, from when I boot to the usual RH9 kernel. Any ideas are welcome, otherwise I won''t worry about it until I upgrade to Xen 1.2 (or whatever) and kernel 2.6. Have a happy! -- Greg ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > Check the boot messages to see how many CPUs Xen finds. Also, if > > you''ve started multiple domains, do a ''domain list'' and see which > > CPUs they''re assigned to. Alternatively, hit ''d'' on the serial > > line (or scroll lock -h on the keyboard) to get a dump of the > > run queue on each CPU. > > Hmmm, it only finds one. I thought this was all it supported, > from prior messages to the list. Note, I''m using a pre-1.1 Xen, > so if it was added later that would explain.Xen has been SMP since the very beginning. Each domain will only see 1 CPU, but Xen should see all of them.> From dmesg: > Kernel command line: /xenolinux.gz root=/dev/sda2 ro console=xencons0 DOMID=0 > Initializing CPU#0 > Xen reported: 1694.911 MHz processor.This is what I''d expect -- each domain, including dom0 only sees one CPU.> Just the one CPU...from "xenctl domain list", all domains are > on CPU0.Odd. I suspect its having trouble parsing the BIOS tables. Please can you try connecting a serial line and sending us the Xen boot output. (On newer Xen versions you can read this boot output from a ring buffer using domain 0, but I don''t think we had support in 1.1) [Alternatively, you could try boot 1.2-rc using the pre-compiled binaries I posted a link to yesterday]> My grub boot args specify "noht", but I''m pretty sure the kernel > is SMP (it''s the one from you folks, with only ReiserFS added). > Maybe "noht" is the problem (the 1.7Ghz Xeon processors don''t > support hyperthreading, so I presumed I needed noht).I doubt ''noht'' is causing any harm, but it might be worth removing it. As long as you don''t have ''nosmp''... Cheers, Ian ------------------------------------------------------- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM''s Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel