Xen devel - Oct 2003 - Virtual block devices (vbd) still not working... :-(

Ian wrote:
>> I haven''t been able to try the "virtual disk" stuff
as I haven''t a
>> spare partition on my test machine. I''ll make one free if
you''re
>> still having problems ;-)
>With the 3 lines hacked out of xenctl, I''ve successfully
>"formatted" a partition to contain virtual disks, created a
>virtual disk, attached it as a "virtual block device" to domain0,
>made a file system on it, and even mounted it.
>
>Amazingly, the code hasn''t rotted! (other than the java goop)
>
>The log is attached, with comments. The UI for the tools isn''t
exactly
i>ntuitive. The web-based interface was pretty cool until
it
>rotted.
>
>Cheers,
>Ian
>
>
>iap10 > mkdir /var/lib/xen
>
>#we need a directory in which the vdstate.xml file will be
>#created. This keeps track of which chunks of disk space are used
>#by which virtual disks.
>
>iap10 > xenctl partitions add -psda15 -f
>warning: state file not found [/var/lib/xen/vdstate.xml]
>Added partition sda15
....

I''ve tried Ian''s guide in the post to the mailing list to
create
vds and vbds and had no success.

xenctl always reported that it added the partition, created the vd
and created the vbd for domain 0 and also the vd show and vbd show
gave the correct display of the created vds and vbds.

But, as I tried to mkfs on /dev/xvda , mkfs showed an error
message that the filesystem reported to have zero size ?

Then I tried fdisk /dev/xvda  and it complained about not having
any partition label on it and that I have to set the number of
cylinders.
Entering the ''p'' option in fdisk showed an empty partition
table
and 255 heads, 63 sectors, 0 cylinders ??

Trying the same on /dev/xvdb showed the same result with mkfs
but fdisk now reported "unable to read /dev/xvdb".
(I added the local ide partition /dev/hda7 with xenctl partition add, 
and #0 as the number of the vbd).

Something seems to be broken here....

I also recognized another strange thing:
Everybody is showing examples with vbd create and using the -w flag in order
to enable write access to the vbd. 
But I have to use "-rw" with xenctl vbd create, otherwise I end up
only with
read
access.
Also, with a lot of xenctl commands there is a x<subst> option.
What''s the
meaning of that ?
Are you using a newer version of xen than me ? ;-)
Mine was copied from the xen-1.0 demo disk.
I also used the source tree from this disk and compiled xeno and xenolinux
from
this tree.

Anybody else has these strange problems with virtual block devices and
virtual 
disks ? Any solutions ?

Regards

Sven







-- 
NEU FÜR ALLE - GMX MediaCenter - für Fotos, Musik, Dateien...
Fotoalbum, File Sharing, MMS, Multimedia-Gruß, GMX FotoService

Jetzt kostenlos anmelden unter http://www.gmx.net

+++ GMX - die erste Adresse für Mail, Message, More! +++



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Then I tried fdisk /dev/xvda  and it complained about not having
> any partition label on it and that I have to set the number of
> cylinders.
> Entering the ''p'' option in fdisk showed an empty
partition table
> and 255 heads, 63 sectors, 0 cylinders ??
Sounds like we''re not handling a geometry ioctl properly. I can
reproduce this problem so I''ll look into it. I don''t know how
it could
have worked for Ian!
 
> Trying the same on /dev/xvdb showed the same result with mkfs
> but fdisk now reported "unable to read /dev/xvdb".
> (I added the local ide partition /dev/hda7 with xenctl partition add, 
> and #0 as the number of the vbd).
The vbd must be created with ''-v1'' to appear as xvdb.
> I also recognized another strange thing:
> Everybody is showing examples with vbd create and using the -w flag in
order
> to enable write access to the vbd. 
> But I have to use "-rw" with xenctl vbd create, otherwise I end
up only with
> read access.
I don''t see this. ''-w'' works for me.
> Also, with a lot of xenctl commands there is a x<subst> option.
What''s the
> meaning of that ?
There''s no need to use this unless you specify a partition name as a
domain-id-specific pattern (e.g. sda7+). I wouldn''t worry about it :-)
> Are you using a newer version of xen than me ? ;-)
> Mine was copied from the xen-1.0 demo disk.
> I also used the source tree from this disk and compiled xeno and xenolinux
> from this tree.
We are more up to date now. There is a xeno-unstable.bk repository on
xen.bkbits.net that includes various fixes and improvements. You''ll
probably want to clone that repository when I fix teh geometry ioctl.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> xenctl always reported that it added the partition, created the vd
> and created the vbd for domain 0 and also the vd show and vbd show
> gave the correct display of the created vds and vbds.
This is an odd one. Can you send me the output of "vd show".
 
> But, as I tried to mkfs on /dev/xvda , mkfs showed an error
> message that the filesystem reported to have zero size ?
Please can you show me the output of "cat /proc/xeno/vhd"
> Then I tried fdisk /dev/xvda  and it complained about not having
> any partition label on it and that I have to set the number of
> cylinders.
> Entering the ''p'' option in fdisk showed an empty
partition table
> and 255 heads, 63 sectors, 0 cylinders ??
There wouldn''t have been a valid partition table on the virtual
disk, so fdisk would have just read rubbish and got confused. You
could just delete all the bogus partitions, but using "dd
if=/dev/zero of=/dev/vbda count=5" is probably safest.

In normal circumstances, fdisk should then be happy.
> Trying the same on /dev/xvdb showed the same result with mkfs
> but fdisk now reported "unable to read /dev/xvdb".
Have you done a "vbd create -v1 -n0 -w" for xvdb, or are you
getting this error on non existent vbd''s?
> I also recognized another strange thing:
> Everybody is showing examples with vbd create and using the -w flag in
order
> to enable write access to the vbd. 
> But I have to use "-rw" with xenctl vbd create, otherwise I end
up only with
> read
> access.
That really is odd. Write access seems to work for me with the
"-w" flag. I don''t believe there is a "-rw" flag as
read access
is the default.
> Also, with a lot of xenctl commands there is a x<subst> option.
What''s the
> meaning of that ?
The -x flag comes into play if you''re using the ''+''
postfix to
any of the entries in your /etc/xenctl.xml (e.g. for the new
domain''s IP address or disk partition). By default, xenctl will
add the new domains domainID to the contents all fields
postfixed by a ''+'' (i.e. /dev/sda7+ for domain 2 becomes
/dev/sda9). Rather than using the domainID, you can force xenctl
to use a particular value in the calculation with the "-x" option.

[Note to self: We desperately need some documentation for xenctl.]
> Are you using a newer version of xen than me ? ;-)
I''m using the xeno-unstable.bk tree, which is 30 changesets ahead
of the 1.0 tree, but none of them have gone anywhere near the vbd
stuff.

I think Keir kindly volunteered to do a code review of that
section of code over the weekend, so hopefully he''ll spot
something.

I''m confident we''ll get things working for you eventually...

Best,
Ian



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > Then I tried fdisk /dev/xvda  and it complained about not having
> > any partition label on it and that I have to set the number of
> > cylinders.
> > Entering the ''p'' option in fdisk showed an empty
partition table
> > and 255 heads, 63 sectors, 0 cylinders ??
> 
> There wouldn''t have been a valid partition table on the virtual
> disk, so fdisk would have just read rubbish and got confused. You
> could just delete all the bogus partitions, but using "dd
> if=/dev/zero of=/dev/vbda count=5" is probably safest.
This doesn''t work for me. The problem is that vbd''s are
registered
with the Linux blkdev layer at boot time, but we have no mechanism to
properly attach new vbd''s that are created after the domain has
already booted.

I really can''t see how it could possibly have worked for you --- DOM0
would have booted with no vbd''s and there''s no way to register
them
later!
> That really is odd. Write access seems to work for me with the
> "-w" flag. I don''t believe there is a "-rw"
flag as read access
> is the default.
Something here is broke. ''vbd show'' will show that the vbd is
ro if
you specify -w, but rw if you specify -rw. I''ll have a look... 

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

There are now fixes to the vbd problems in the xen-unstable.bk
repository at xen.bkbits.net.

Please clone and build this repsoitory if you''re having problems.
> This doesn''t work for me. The problem is that vbd''s are
registered
> with the Linux blkdev layer at boot time, but we have no mechanism to
> properly attach new vbd''s that are created after the domain has
> already booted.
> 
> I really can''t see how it could possibly have worked for you ---
DOM0
> would have booted with no vbd''s and there''s no way to
register them
> later!
I''ve fixed this now. If you attach a VBD to an already-running domain,
you should run tools/misc/xen_refresh_dev. 
e.g. ''xen_refresh_dev /dev/xvda''.

This refreshes Linux''s view of its partition table, and also polls Xen
to find out the current parameters of the VBD. Without doing this,
Linux will never see the correct capacity for the VBD (it will
probably continue to think that it has zero size!).

A future TODO is probably to get Xen to automatically inform a domain
when one of its VBDs changes so that the refresh doesn''t need to be
done manually.
 
> Something here is broke. ''vbd show'' will show that the
vbd is ro if
> you specify -w, but rw if you specify -rw. I''ll have a look... 
This is fixed. -w is definitely the write way to get write privileges
on a new VBD.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> This refreshes Linux''s view of its partition table, and also polls
Xen
> to find out the current parameters of the VBD. Without doing this,
> Linux will never see the correct capacity for the VBD (it will
> probably continue to think that it has zero size!).
Whenever Alex or I have used VDs they''ve just worked: There''s
no way to populate a vd with a file system unless you can access
it from domain0, so its obviously been working for some people.

I wander if it''s got something to do with whether the newly
created virtual disk just happens to have something approximating
a valid partition table or not?

Having the refresh tool seems like a good solution until we
figure out how to make Linux do it automatically. I can''t think
of any real block devices that have their size change dynamically
that we can borrow the mechanism from. I guess we could use
''media change'' events, but it would be good to be able to 
extend a virtual disk that''s already mounted.


Ian


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> 
> > This refreshes Linux''s view of its partition table, and also
polls Xen
> > to find out the current parameters of the VBD. Without doing this,
> > Linux will never see the correct capacity for the VBD (it will
> > probably continue to think that it has zero size!).
> 
> Whenever Alex or I have used VDs they''ve just worked:
There''s
> no way to populate a vd with a file system unless you can access
> it from domain0, so its obviously been working for some people.
> 
> I wander if it''s got something to do with whether the newly
> created virtual disk just happens to have something approximating
> a valid partition table or not?
No, it doesn''t. Linux definitely thinks the vbd has zero size so never
even gets to try to read a partition table. 

I followed what you did exactly, and without an explicit poll of Xen
it definitely didn''t work. I''d like to see how your machine
setup is
different -- for example, are you sure the /dev/vbd* nodes have the
right device numbers? (Should be major = 125, minor (vbd_id*16)+partition).

I''m going to have some more of a look at the blkdev stuff now, but it
would be nice to know how come it''s been "working" for you.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Whenever Alex or I have used VDs they''ve just worked:
There''s
> no way to populate a vd with a file system unless you can access
> it from domain0, so its obviously been working for some people.
> 
> I wander if it''s got something to do with whether the newly
> created virtual disk just happens to have something approximating
> a valid partition table or not?
I''ve now fully worked outwhat was going on. The reason it 
worked for you is pretty vile, so I''ll describe it here :-)

After doing ''vbd create'' you must have executed
''mkfs'' as
non-root. The test machine was incorrectly set up so that users had
read and write access directly via the /dev/xvd* files (NB. I''ve
checked in a fixed xen-mkdevnodes that ensures this cannot happen).

However, although the device file gives you write access to make a
filesystem, the first thing mkfs does is an ioctl to find the device
size. We have a check for CAP_SYS_ADMIN in our ioctl function
(incorrectly, since permission checking is done when opening the /dev
file --- I''ll check in a fix to remove that, and another, bogus
check).  

The ioctl returns -1 (errno==EPERM), but mkfs is broken and doesn''t
check the return value. It therefore thinks it''s dealing with an
enormous blkdev of size ~0UL :-)

It proceeds to write the inode and superblock info over the disk. Even
though access checks are probably failing, enough are successfully
written at the start of the vbd that mount can still work.

It''s unfortunate that you get no errors for the access violations by
mkfs (it must surely be writing off the end of the 128M vbd). Xen
returns an error condition, and this is propagated by the xenolinux
driver, but it appears to be silently dropped by the generic blkdev
layer :-( :-(

I did have the xenolinux driver make a noise about errors before, but
it got rather messy at boot time (generic blkdev layer would pass down
requests that failed for legitimate reasons) so I removed it.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> From: xen-devel-admin@lists.sourceforge.net
> [mailto:xen-devel-admin@lists.sourceforge.net]On Behalf Of Ian Pratt
> Sent: Saturday, October 11, 2003 5:22 AM
>
> Having the refresh tool seems like a good solution until we
> figure out how to make Linux do it automatically. I can''t think
> of any real block devices that have their size change dynamically
> that we can borrow the mechanism from. I guess we could use
> ''media change'' events, but it would be good to be able to
> extend a virtual disk that''s already mounted.
EVMS does this type of thing, but I have no idea what the mechanism is as I
have never looked.


paul



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel