Xen devel - Sep 2006 - Help/problems with blktap on xen-unstable

Hi

I''ve been trying to play with the new blktap support in xen-unstable.

First, an observation - blktapctrl is pinned at 100% CPU:
-----------------------------------------------------------------------------------------------------------------

top - 17:29:48 up 4 days,  2:02,  3 users,  load average: 0.90, 1.01, 1.00
Tasks:  83 total,   3 running,  80 sleeping,   0 stopped,   0 zombie
Cpu(s): 15.2% us, 34.9% sy,  0.0% ni, 49.9% id,  0.0% wa,  0.0% hi,  
0.0% si
Mem:    262356k total,   240936k used,    21420k free,     4832k buffers
Swap:   917496k total,    96976k used,   820520k free,    67180k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
3807 root      25   0 11040  508  400 R 99.9  0.2   0:17.56 blktapctrl
   1 root      16   0  1744  124   92 S  0.0  0.0   0:00.87 init
------------------------------------------------------------------------------------------------------------------


Then, I am trying to follow the README to get an image file mounted via 
tap:aio in dom0 (as a precursor, or course, to using it for rootfs in a 
domU).

So:

[root@dom0-0-50-45-5d-59-aa tmp]# xm block-attach 0 
tap:aio:/tmp/sparse.img /dev/xvda1 w 0
...
[root@dom0-0-50-45-5d-59-aa tmp]# xm block-list 0
Vdev  BE handle state evt-ch ring-ref BE-path
51713    0    0     3      15     8     /local/domain/0/backend/tap/0/51713

But
[root@dom0-0-50-45-5d-59-aa tmp]# mount /dev/xvda1 /mnt/tapaio
mount: you must specify the filesystem type
[root@dom0-0-50-45-5d-59-aa tmp]# mount -t ext3 /dev/xvda1 /mnt/tapaio
mount: special device /dev/xvda1 does not exist

And, indeed, there''s no /dev/xvda1.

Help!
Roland


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am Mittwoch, den 20.09.2006, 08:37 +0200 schrieb Roland Paterson-Jones:
> I''ve been trying to play with the new blktap support in
xen-unstable.
> 
> First, an observation - blktapctrl is pinned at 100% CPU:
Do you have a 3Ware controller or something? On my machine, the
controller management interface (/dev/twe*) gets assigned the major 254,
and XEN has the same major hardcoded, and the registration therefore
fails. blktapctrl then creates its device nodes /dev/xen/tapctrl* with
major 254 and tries to talk to it. The twe driver doesn''t understand
its
ioctl''s and blktapctrl tries to resend its commands in an infinite
loop.

The plan (according to blktap.c) is to make the registration of the
device node major number dynamic:

#define BLKTAP_DEV_MAJOR 254         /* TODO: Make major number dynamic  *
                                      * and create devices in the kernel *
                                      */

Nobody has done so yet.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christophe Saout wrote:
>Am Mittwoch, den 20.09.2006, 08:37 +0200 schrieb Roland Paterson-Jones:
>
>  
>
>>I''ve been trying to play with the new blktap support in
xen-unstable.
>>
>>First, an observation - blktapctrl is pinned at 100% CPU:
>>    
>>
>
>Do you have a 3Ware controller or something? On my machine, the
>controller management interface (/dev/twe*) gets assigned the major 254,
>  
>
Thanks for the tip, but it doesn''t (necessarily) seem to be the case 
with our box.

I do have /dev/xen with:

[root@dom0-0-50-45-5d-59-aa ~]# ls -als /dev/xen/
total 0
0 drwxr-xr-x   2 root root       80 Sep 19 17:43 .
0 drwxr-xr-x  13 root root    14100 Sep 19 17:37 ..
0 crw-------   1 root root 254,   0 Sep 19 17:43 blktap0
0 crw-------   1 root root  10, 201 Sep 19 17:36 evtchn

But there''s nothing in /dev with major 254.

Regards
Roland


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Roland Paterson-Jones
> Sent: 20 September 2006 13:06
> To: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Help/problems with blktap on xen-unstable
> 
> Christophe Saout wrote:
> 
> >Am Mittwoch, den 20.09.2006, 08:37 +0200 schrieb Roland 
> Paterson-Jones:
> >
> >  
> >
> >>I''ve been trying to play with the new blktap support in 
> xen-unstable.
> >>
> >>First, an observation - blktapctrl is pinned at 100% CPU:
Have you tried to run something like strace or gdb on blktapctrl? It''s
not necessarily going to show much, but until you try, it''s impossible
to say. 

--
Mats



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am Mittwoch, den 20.09.2006, 14:06 +0200 schrieb Roland Paterson-Jones:
> >Do you have a 3Ware controller or something? On my machine, the
> >controller management interface (/dev/twe*) gets assigned the major
254,
> >
> Thanks for the tip, but it doesn''t (necessarily) seem to be the
case
> with our box.
> 
> But there''s nothing in /dev with major 254.
What das "cat /proc/devices" tell about assigned characted device
major
numbers (in the upper half)?



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christophe Saout wrote:
>What das "cat /proc/devices" tell about assigned characted device
major
>numbers (in the upper half)?
>  
>
Indeed:

Block devices:
...
253 device-mapper
254 mdp

I''m presuming mdp is not code for blktap. So, I guess I need to 
recompile with an alternate device major number.

Many thanks
Roland

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christophe Saout wrote:
>What das "cat /proc/devices" tell about assigned characted device
major
>numbers (in the upper half)?
>  
>
Sorry, ignore previous e-mail - it''s a character device, and yes:

Character devices:
...
254 twa

Thanks again
Roland


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Roland Paterson-Jones wrote:
>  So, I guess I need to recompile with an alternate device major number.
I can find the following refs to 254 in the source (some are dups):

./linux-2.6.16.13-xen0/drivers/xen/blktap/blktap.c:              
major=254, minor numbers begin at 0            */
./linux-2.6.16.13-xen0/drivers/xen/blktap/blktap.c:#define 
BLKTAP_DEV_MAJOR 254         /* TODO: Make major number dynamic  *
./linux-2.6-xen-sparse/drivers/xen/blktap/blktap.c:              
major=254, minor numbers begin at 0            */
./linux-2.6-xen-sparse/drivers/xen/blktap/blktap.c:#define 
BLKTAP_DEV_MAJOR 254         /* TODO: Make major number dynamic  *
./tools/blktap/lib/blktaplib.h:#define BLKTAP_DEV_MAJOR 254
./tools/blktap/drivers/blktapctrl.c:    make_blktap_dev(devname,254,0);
./tools/blktap_user/aiotools/blktapctrl.c:    
make_blktap_dev(devname,254,0);
./tools/blktap_user/blktaplib.h:#define BLKTAP_DEV_MAJOR 254

I''m guessing it would be good to have one definition, but I''m
not too
familiar with the code structure. Can a .h file span kernel and user 
source? Also, any idea what is meant by making major number
''dynamic'' -
are they proposing walking the major numbers in code until successful, 
or just a config option somewhere (where?).

Regards
Roland

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Roland Paterson-Jones
> Sent: 20 September 2006 14:44
> To: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Help/problems with blktap on xen-unstable
> 
> Roland Paterson-Jones wrote:
> 
> >  So, I guess I need to recompile with an alternate device 
> major number.
> 
> I can find the following refs to 254 in the source (some are dups):
> 
> ./linux-2.6.16.13-xen0/drivers/xen/blktap/blktap.c:              
> major=254, minor numbers begin at 0            */
> ./linux-2.6.16.13-xen0/drivers/xen/blktap/blktap.c:#define 
> BLKTAP_DEV_MAJOR 254         /* TODO: Make major number dynamic  *
> ./linux-2.6-xen-sparse/drivers/xen/blktap/blktap.c:              
> major=254, minor numbers begin at 0            */
> ./linux-2.6-xen-sparse/drivers/xen/blktap/blktap.c:#define 
> BLKTAP_DEV_MAJOR 254         /* TODO: Make major number dynamic  *
> ./tools/blktap/lib/blktaplib.h:#define BLKTAP_DEV_MAJOR 254
> ./tools/blktap/drivers/blktapctrl.c:    
> make_blktap_dev(devname,254,0);
> ./tools/blktap_user/aiotools/blktapctrl.c:    
> make_blktap_dev(devname,254,0);
> ./tools/blktap_user/blktaplib.h:#define BLKTAP_DEV_MAJOR 254
> 
> I''m guessing it would be good to have one definition, but
I''m not too
> familiar with the code structure. Can a .h file span kernel and user 
> source? 
There''s nothing preventing the same file being used in both kernel and
user-mode compiles - in fact in this case it would probably be a good
idea to ADD such a header-file - I''m not sure if there''s
anything ELSE
that should go into such a file [presumably there is some shared data
structures too that could go in there]. The above use of the same
constant in three places + using the same as a magic number is
definitely b0rkened... 

--
Mats



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christophe Saout wrote:
> Am Mittwoch, den 20.09.2006, 08:37 +0200 schrieb Roland Paterson-Jones:
> 
>> I''ve been trying to play with the new blktap support in
xen-unstable.
>>
>> First, an observation - blktapctrl is pinned at 100% CPU:
> 
> Do you have a 3Ware controller or something? On my machine, the
> controller management interface (/dev/twe*) gets assigned the major 254,
> and XEN has the same major hardcoded, and the registration therefore
> fails. blktapctrl then creates its device nodes /dev/xen/tapctrl* with
> major 254 and tries to talk to it. The twe driver doesn''t
understand its
> ioctl''s and blktapctrl tries to resend its commands in an infinite
loop.
> 
> The plan (according to blktap.c) is to make the registration of the
> device node major number dynamic:
> 
> #define BLKTAP_DEV_MAJOR 254         /* TODO: Make major number dynamic  *
>                                       * and create devices in the kernel *
>                                       */
> 
> Nobody has done so yet.
> 
Any thought about making blktap a miscellaneous device and have blkctrl 
just read /proc/misc to find the node?  I could easily write up a patch, 
if others think this is appropriate.

-- Steve


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Roland Paterson-Jones wrote:
> Steven Rostedt wrote:
>> Any thought about making blktap a miscellaneous device and have 
>> blkctrl just read /proc/misc to find the node?  I could easily write 
>> up a patch, if others think this is appropriate.
> 
> I don''t understand the technical details, but this sounds like it
would
> avoid both hardcoded or configured major number, so if that''s the
case
> I''d be very keen for the patch.
> 
There''s a few issues.

As a miscellaneous device, it no longer has a major number.  All 
versions would become a miscellaneous device. So instead of having a 
blktap major, and the minor pointing to each of the blktap descriptors 
in the array, you would record the minor in the descriptor, and among 
opening the device, you need to search the descriptors to find the minor 
that matches the opened device.

So you might get something like:

# cat /proc/misc
35 blktap0
39 blktap1
40 blktap2
84 blktap3

Where the index no longer matches the inode minor increment.  It 
probably wont be this bad, and usually would increment, but you don''t 
know if another device has jumped in.

But the search is only on opening the device and not an the operations, 
because once you find the descriptor in blktap_open you save that in the 
private date field of the file descriptor.


Another solution is to just make it a dynamic major, and then put the 
result somewhere that a tool can find it.  Most acceptable now, I guess, 
is in /sys.  Even though I find working in the /proc is easier.

Or, have a blktapctl as a miscellaneous device, and this can tell you 
where the major number was. Finding the blktapctl would be easy, since 
it would be in /proc/misc.

So there''s several solutions to this.  It''s just a matter of
which is
most acceptable.

-- Steve


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The current device allocation code is pretty crap, and I completely
agree that the fixed assignments suck.  I think my inclination would
be towards everything being dynamically assigned, and that the tapdisk
devices be allocated on demand.  This would clean up a bunch of the
kernel code -- get rid of the fixed array of tap structs/device nodes
and let us scale up to larger numbers of active devices in the future.

If you''re offering to cook a patch, we''d be delighted to see
it.  I
could also be argued into an interim solution to fix the current
conflict problems in advance of a more substantial effort. ;)

Thanks for the comments Steve, sorry for the slowish reply.

a.

On 9/21/06, Steven Rostedt <srostedt@redhat.com>
wrote:
> Roland Paterson-Jones wrote:
> > Steven Rostedt wrote:
>
> >> Any thought about making blktap a miscellaneous device and have
> >> blkctrl just read /proc/misc to find the node?  I could easily
write
> >> up a patch, if others think this is appropriate.
> >
> > I don''t understand the technical details, but this sounds
like it would
> > avoid both hardcoded or configured major number, so if that''s
the case
> > I''d be very keen for the patch.
> >
>
> There''s a few issues.
>
> As a miscellaneous device, it no longer has a major number.  All
> versions would become a miscellaneous device. So instead of having a
> blktap major, and the minor pointing to each of the blktap descriptors
> in the array, you would record the minor in the descriptor, and among
> opening the device, you need to search the descriptors to find the minor
> that matches the opened device.
>
> So you might get something like:
>
> # cat /proc/misc
> 35 blktap0
> 39 blktap1
> 40 blktap2
> 84 blktap3
>
> Where the index no longer matches the inode minor increment.  It
> probably wont be this bad, and usually would increment, but you
don''t
> know if another device has jumped in.
>
> But the search is only on opening the device and not an the operations,
> because once you find the descriptor in blktap_open you save that in the
> private date field of the file descriptor.
>
>
> Another solution is to just make it a dynamic major, and then put the
> result somewhere that a tool can find it.  Most acceptable now, I guess,
> is in /sys.  Even though I find working in the /proc is easier.
>
> Or, have a blktapctl as a miscellaneous device, and this can tell you
> where the major number was. Finding the blktapctl would be easy, since
> it would be in /proc/misc.
>
> So there''s several solutions to this.  It''s just a matter
of which is
> most acceptable.
>
> -- Steve
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Thu, 2006-09-21 at 07:58 -0400, Steven Rostedt wrote:
> As a miscellaneous device, it no longer has a major number.  All 
> versions would become a miscellaneous device. So instead of having a 
> blktap major, and the minor pointing to each of the blktap descriptors 
> in the array, you would record the minor in the descriptor, and among 
> opening the device, you need to search the descriptors to find the minor 
> that matches the opened device.
There''s an easy way to do both, I think.  We already have an ioctl to
return the blktap major, so simply registering the control device as a
misc device and then allocating the major for the full set of domain
devices dynamically should work.

--Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel