In xen 4.0.X I was able to write custom blktap2 drivers as a nice way to
intercept VM disk traffic. I''m now trying to take a step up to xen
4.1.2
and find that blktap2 doesn''t seem to be supported anymore, or at least
it
requires a kernel module which I''m not sure where to find. Is blktap2
still in use, or is there an entirely different way I should be approaching
this?
Previously I could run commands like:
tapdisk2 -n tap:aio:/home/twood/vms/testdisk.img
the new tapdisk2 doesn''t support that interface anymore, but it seems
like
the correct approach is now:
xm block-attach 0 tap2:aio:/path/disk.img xvda w 0
Error: (''create'', ''-aaio:/path/disk.img'')
failed (55808 blktap kernel
module not installed )
What is the "proper" way of getting the blktap kernel module
installed? I
found this:
http://packages.debian.org/sid/blktap-dkms
but couldn''t get it to actually install. In any case, it seems
unlikely
that is the best way to go.
Also, I will admit that I am running 4.1.2 (from source) rather than
unstable---looking through the threads here doesn''t make it sound like
upgrading to that will fix anything, but I can do so if need be.
thanks for any suggestions
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
On Fri, 2012-04-06 at 22:24 +0100, Tim Wood wrote:> In xen 4.0.X I was able to write custom blktap2 drivers as a nice way > to intercept VM disk traffic. I''m now trying to take a step up to xen > 4.1.2 and find that blktap2 doesn''t seem to be supported anymore, or > at least it requires a kernel module which I''m not sure where to > find. Is blktap2 still in use, or is there an entirely different way > I should be approaching this? > > > Previously I could run commands like: > > > tapdisk2 -n tap:aio:/home/twood/vms/testdisk.img > > > the new tapdisk2 doesn''t support that interface anymore,What do you mean? I don''t think the tap interface has changed (but I''m not sure). In any case I''m pretty sure the functionality should be there even if the command line has changed.> but it seems like the correct approach is now: > > > xm block-attach 0 tap2:aio:/path/disk.img xvda w 0 > Error: (''create'', ''-aaio:/path/disk.img'') failed (55808 blktap kernel > module not installed )I think this ultimately turns into the same sort of command as the one above.> What is the "proper" way of getting the blktap kernel module > installed? I found this: > http://packages.debian.org/sid/blktap-dkmsUnfortunately the blktap2 module is not something which can be upstreamed. The DKMS package is probably a good bet for now. The other alternative is to switch to a slightly older kernel which has the blktap2 driver in it, for example the 2.6.32 based xen.git kernel tree or one of the classic Xen forward ports. You mentioned writing your own blktap modules so the qemu-supplied qdisk backend is probably not much use to you. This is used by the "xl" toolstack by default when blktap is not present, but isn''t supported by xm and doesn''t allow for custom modules . I''m actually quite interested in the fact that you are writing custom blktap modules -- are you able to share the details?> but couldn''t get it to actually install.Please share the details so we can try and figure out why.> In any case, it seems unlikely that is the best way to go.Sadly it is, for now. Long term someone is working on a "blktap3" which is fully userspace and so doesn''t require a kernel module. We hope to see this for 4.3. For 4.2 it looks like we''ll be sticking with the qdisk backend. Ian.
On Sat, Apr 7, 2012 at 3:37 AM, Ian Campbell <Ian.Campbell@citrix.com>wrote:> On Fri, 2012-04-06 at 22:24 +0100, Tim Wood wrote: > > In xen 4.0.X I was able to write custom blktap2 drivers as a nice way > > to intercept VM disk traffic. I''m now trying to take a step up to xen > > 4.1.2 and find that blktap2 doesn''t seem to be supported anymore, or > > at least it requires a kernel module which I''m not sure where to > > find. Is blktap2 still in use, or is there an entirely different way > > I should be approaching this? > > > > > > Previously I could run commands like: > > > > > > tapdisk2 -n tap:aio:/home/twood/vms/testdisk.img > > > > > > the new tapdisk2 doesn''t support that interface anymore, > > What do you mean? I don''t think the tap interface has changed (but I''m > not sure). In any case I''m pretty sure the functionality should be there > even if the command line has changed. >Sorry, I wasn''t clear--the functionality is still there but the command line interface has changed. The xm-block attach command below seems to be equivalent.> > > but it seems like the correct approach is now: > > > > > > xm block-attach 0 tap2:aio:/path/disk.img xvda w 0 > > Error: (''create'', ''-aaio:/path/disk.img'') failed (55808 blktap kernel > > module not installed ) > > I think this ultimately turns into the same sort of command as the one > above. > > > What is the "proper" way of getting the blktap kernel module > > installed? I found this: > > http://packages.debian.org/sid/blktap-dkms > > Unfortunately the blktap2 module is not something which can be > upstreamed. > > The DKMS package is probably a good bet for now. The other alternative > is to switch to a slightly older kernel which has the blktap2 driver in > it, for example the 2.6.32 based xen.git kernel tree or one of the > classic Xen forward ports. >OK, I will fiddle with DKMS a bit more or switch to 2.6.32.> > You mentioned writing your own blktap modules so the qemu-supplied qdisk > backend is probably not much use to you. This is used by the "xl" > toolstack by default when blktap is not present, but isn''t supported by > xm and doesn''t allow for custom modules . > > I''m actually quite interested in the fact that you are writing custom > blktap modules -- are you able to share the details? > >Sure, I''m a CS professor and do research on what cool stuff you can add in the virtualization layer. What I did before was a module similar to remus''s, but designed for WAN replication and synchronizing across groups of VMs. The paper can be a bit tough to get through, but it has some of the details: http://faculty.cs.gwu.edu/~timwood/papers/SOCC11-PipeCloud.pdf Blktap is a very handy way for us to prototype these types of features without having to muck through the core xen code.> > but couldn''t get it to actually install. > > Please share the details so we can try and figure out why. > > > In any case, it seems unlikely that is the best way to go. > > Sadly it is, for now. > > Long term someone is working on a "blktap3" which is fully userspace and > so doesn''t require a kernel module. We hope to see this for 4.3. For 4.2 > it looks like we''ll be sticking with the qdisk backend. > > Sounds great, I look forward to it._______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Apr 9, 8:13am, Tim Wood wrote: } Subject: Re: [Xen-devel] where to find blktap2 kernel module Hi, hope the day is going well for efveryone.> --485b397dd775dd0da404bd3df0b5 > Content-Type: text/plain; charset=ISO-8859-1 > > On Sat, Apr 7, 2012 at 3:37 AM, Ian Campbell <Ian.Campbell@citrix.com>wrote: > > > On Fri, 2012-04-06 at 22:24 +0100, Tim Wood wrote: > > > In xen 4.0.X I was able to write custom blktap2 drivers as a nice way > > > to intercept VM disk traffic. I''m now trying to take a step up to xen > > > 4.1.2 and find that blktap2 doesn''t seem to be supported anymore, or > > > at least it requires a kernel module which I''m not sure where to > > > find. Is blktap2 still in use, or is there an entirely different way > > > I should be approaching this? > > > > > > > > > Previously I could run commands like: > > > > > > > > > tapdisk2 -n tap:aio:/home/twood/vms/testdisk.img > > > > > > > > > the new tapdisk2 doesn''t support that interface anymore, > > > > What do you mean? I don''t think the tap interface has changed (but I''m > > not sure). In any case I''m pretty sure the functionality should be there > > even if the command line has changed. > > > > Sorry, I wasn''t clear--the functionality is still there but the command > line interface has changed. The xm-block attach command below seems to be > equivalent. > > > > > > > but it seems like the correct approach is now: > > > > > > > > > xm block-attach 0 tap2:aio:/path/disk.img xvda w 0 > > > Error: (''create'', ''-aaio:/path/disk.img'') failed (55808 blktap kernel > > > module not installed ) > > > > I think this ultimately turns into the same sort of command as the one > > above.The documentation in the Xen sources is out of date with respect to userspace interfaces to blktap2. The ''tap-ctl'' command is the simplest way to manage dom0 instances of block devices. Its actually a pretty well defined interface. Here is a brief recipe: 1.) Allocate a blktap minor device: tap-ctl allocate 2.) Spawn a userspace driver: tap-ctl spawn The ''tap-ctl list'' command is useful at this point in order to get the status of available devices and process numbers. In the following commands substitue PPP with the PID of the userspace driver and MMM with the minor number of the device you are working with. 3.) Bind the userspace driver to a minor instance: tap-ctl attach -p PPP -m MMM 4.) Open a filesystem image with an appropriate driver. In this case it is aio, if you are rolling your own subsitute the driver prefix (aio:) with the name of your driver. tap-ctl open -p PPP -m MMM aio:FNAME If this command succeeds you will see a hotplug/udev inventing indicating that a new block device was created. The ''official'' device node locations should be: /dev/xen/blktap-2/tapdevMMM Where MMM is replaced with the minor number of the block device. 5.) Close the filesystem image: tap-ctl close -p PPP -m MMM 6.) Detach the process from the allocated minor number: tap-ctl detach -p PPP -m MMM 7.) Free the minor number: tap-ctl free -m 0> > > What is the "proper" way of getting the blktap kernel module > > > installed? I found this: > > > http://packages.debian.org/sid/blktap-dkms > > > > Unfortunately the blktap2 module is not something which can be > > upstreamed. > > > > The DKMS package is probably a good bet for now. The other alternative > > is to switch to a slightly older kernel which has the blktap2 driver in > > it, for example the 2.6.32 based xen.git kernel tree or one of the > > classic Xen forward ports.> OK, I will fiddle with DKMS a bit more or switch to 2.6.32.We have a vested interest in all so in the interests of advancing this technology we have clean kernel patches available at the following locations: ftp://ftp.enjellic.com/pub/xen/blktap2-3.0.patch ftp://ftp.enjellic.com/pub/xen/blktap2-3.2.patch These are basically cleanups from the last available GIT tree which Dan Stodden had made available. There are minimal dependencies so they should drop into other kernels without too much effort. We are focusing on 3.0 and 3.2 since they seem to be odds on favorites for ''stable'' kernels.> > You mentioned writing your own blktap modules so the qemu-supplied qdisk > > backend is probably not much use to you. This is used by the "xl" > > toolstack by default when blktap is not present, but isn''t supported by > > xm and doesn''t allow for custom modules . > > > > I''m actually quite interested in the fact that you are writing custom > > blktap modules -- are you able to share the details?> Sure, I''m a CS professor and do research on what cool stuff you can > add in the virtualization layer. What I did before was a module > similar to remus''s, but designed for WAN replication and > synchronizing across groups of VMs. The paper can be a bit tough to > get through, but it has some of the details: > > http://faculty.cs.gwu.edu/~timwood/papers/SOCC11-PipeCloud.pdfThanks for forwarding the reference.> Blktap is a very handy way for us to prototype these types of > features without having to muck through the core xen code.Indeed, we are looking at it with respect to NPIV support. QEMU is difficult to deal with as it is. It seems to be an architectural advantage to have an independent block support architecture given what people want to do with block devices these days. blktap2 is a very useful architecture, it seems to suffer mainly from a lack of attention. Hopefully this can be improved a bit.> > Long term someone is working on a "blktap3" which is fully userspace and > > so doesn''t require a kernel module. We hope to see this for 4.3. For 4.2 > > it looks like we''ll be sticking with the qdisk backend. > > > > Sounds great, I look forward to it.A pure userspace implementation would be obviously an advantage. That would seem to be a ways out however and if our observations are useful it seems that moving user Xen implementations forward are suffering a bit secondary to the number of things which change a bit from release to release and make maintenance troublesome. Hope the above is useful. Good luck with your efforts. Greg }-- End of excerpt from Tim Wood As always, Dr. G.W. Wettstein, Ph.D. Enjellic Systems Development, LLC. 4206 N. 19th Ave. Specializing in information infra-structure Fargo, ND 58102 development. PH: 701-281-1686 FAX: 701-281-3949 EMAIL: greg@enjellic.com ------------------------------------------------------------------------------ "In the future, company names will be a 32-character hex string." -- Bruce Schneier