jpranevich@kniggit.net
2008-Aug-22 21:35 UTC
[Xen-devel] Almost working iSCSI booting, need advice
Hello, I''ve adapted the patch I sent yesterday to Xen 3.3.0. (Recent changes to blkif.py and DevController.py had to be worked around. Mostly, just change some over-zealous checks to let the code pass for now. Needs looked at later.) Unfortunately, now I can only start one iSCSI mount. The second fails with "device already in use" because /dev/null (my dummy device) was used by the previous mount. (See the diff I sent yesterday for an explanation.) I can comment out that check and get it to work, but that''s not really a good idea overall. I''m committed to getting this to work and contributing to the community whatever fixes are required. But, I need some help understanding how this *should* work, so that I make changes in a way which are compatible with the vision of the primary maintainers. Can anyone here recommend who I should email about this? I''m new to this project and the lists and don''t know all the right names. My big issue involves the creation of the dummy device. It''s a chicken and egg problem with create_vbd() that I''m not sure how to get around. (The hotplug scripts have to run to get the device, but need the device to run the hotplug script.) I may be able to fix that by simply writing the correct information to the xenstore in the right places after I learn them, but I don''t know what places need updating for the different parts of Xen. (I also haven''t tested this with Xen-API or migrations, so I may discover more problems if I''m not structuring this in the right way.) Thanks in advance for any advice you can provide. Joe _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan de Konink
2008-Aug-22 21:41 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> My big issue involves the creation of the dummy device. It''s a > chicken and egg problem with create_vbd() that I''m not sure how to > get around. (The hotplug scripts have to run to get the device, but > need the device to run the hotplug script.) I may be able to fix that > by simply writing the correct information to the xenstore in the > right places after I learn them, but I don''t know what places need > updating for the different parts of Xen. (I also haven''t tested this > with Xen-API or migrations, so I may discover more problems if I''m > not structuring this in the right way.)Can you show me your code of create_vbd? Because in my code I never specify any device name. And if you look at file:/// syntax, do you ever specify /dev/loop0 as filename? Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
jpranevich@kniggit.net
2008-Aug-22 22:01 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
Hello,
I think the relevant code from XendDomainInfo.py _configureBootloader() for
file: mounts looks like this:
mounted = devtype == ''tap'' and taptype !=
''aio'' and taptype != ''sync'' and not
os.stat(fn).st_rdev
if mounted:
# This is a file, not a device. pygrub can cope with a
# file if it''s raw, but if it''s QCOW or other
such formats
# used through blktap, then we need to mount it first.
log.info("Mounting %s on %s." %
(fn, BOOTLOADER_LOOPBACK_DEVICE))
vbd = {
''mode'': ''RO'',
''device'': BOOTLOADER_LOOPBACK_DEVICE,
}
from xen.xend import XendDomain
dom0 = XendDomain.instance().privilegedDomain()
dom0._waitForDeviceUUID(dom0.create_vbd(vbd, disk))
fn = BOOTLOADER_LOOPBACK_DEVICE
This is the code that I''ve mimiced. Here,
"BOOTLOADER_LOOPBACK_DEVICE" is always "/dev/xvdp". But, in
the case of ''tap'' devices, it gets destroyed later. (I
don''t want to destroy it later since the iSCSI mount needs to be kept
around.)
finally:
if mounted:
log.info("Unmounting %s from %s." %
(fn, BOOTLOADER_LOOPBACK_DEVICE))
dom0.destroyDevice(''tap'',
BOOTLOADER_LOOPBACK_DEVICE)
My parallel code is very similar in steps to the tap code, except I need to read
from xenstore after the iSCSI mount is created to find out what the real device
name it was given is. (I can search by its uuid.)
if (access_type == ''iscsi''):
log.info("Mounting %s using %s." % (access_location,
access_type))
from xen.xend import XendDomain
dom0 = XendDomain.instance().privilegedDomain()
# We need a bogus VBD at this point...
vbd = {
''mode'': ''RO'',
''device'': ''/dev/null''
}
vbd_uuid = dom0.create_vbd(vbd, disk)
# Now, we need to find out what the REAL device node is, to pass
to the
bootloader
from xen.xend.XendDomain import DOM0_ID
from xen.xend.xenstore.xsutil import GetDomainPath
dom_path = GetDomainPath(DOM0_ID)
real_device = 0
vbds = xstransact.List("%s/backend/vbd/0" % dom_path)
for x in vbds:
uuid = xstransact.Read("%s/backend/vbd/0/%s/uuid"
% (dom_path, x))
if (uuid == vbd_uuid):
found = 1
real_device =
xstransact.Read("%s/backend/vbd/0/%s/node" % (dom
_path, x))
# Swap out the /dev/null now
xstransact.Store("%s/backend/vbd/0/%s" %
(dom_path, x), (''dev'',
real_device))
break
else:
msg = "Unable to find VBD: %s" % vbd_uuid
log.error(msg)
raise VmError(msg)
msg = "ISCSI Device Detected: %s on %s" % (disk,
real_device)
fn = real_device
log.info(msg)
This actually works very well... for one iSCSI boot, because of the dummy device
problem. I may have misunderstood how file: devices work and where that is
handled, so I''ll dig further. (And it doesn''t seem right that
I''m doing this in the bootloader section, by pygrub needs it that
early.) I''m worried that while this WORKS, that this isn''t the
right place to be triggering the device creation. Perhaps, it needs to happen
before the call to _configureBootloader(), either _initDomain() or
_constructDomain()...
Thanks,
Joe
----- Original Message -----
From: "Stefan de Konink" <skinkie@xs4all.nl>
To: jpranevich@kniggit.net
Cc: "xen-devel" <xen-devel@lists.xensource.com>, "Stephan
Seitz" <s.seitz@netz-haut.de>
Sent: Friday, August 22, 2008 5:41:16 PM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> My big issue involves the creation of the dummy device. It''s a
> chicken and egg problem with create_vbd() that I''m not sure how to
> get around. (The hotplug scripts have to run to get the device, but
> need the device to run the hotplug script.) I may be able to fix that
> by simply writing the correct information to the xenstore in the
> right places after I learn them, but I don''t know what places need
> updating for the different parts of Xen. (I also haven''t tested
this
> with Xen-API or migrations, so I may discover more problems if I''m
> not structuring this in the right way.)
Can you show me your code of create_vbd? Because in my code I never
specify any device name. And if you look at file:/// syntax, do you ever
specify /dev/loop0 as filename?
Stefan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Stefan de Konink
2008-Aug-22 22:16 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> This is the code that I''ve mimiced. Here, > "BOOTLOADER_LOOPBACK_DEVICE" is always "/dev/xvdp". But, in the case > of ''tap'' devices, it gets destroyed later. (I don''t want to destroy > it later since the iSCSI mount needs to be kept around.)You are working at the wrong layer... there are block-scripts... don''t edit the xend python stuff! And you understand that /dev/xvdp is for INSIDE the host? But there is some other news... with Xen 3.3 pvSCSI is possible. Thus it might be better to pass along a scsi generic device instead of a disk. Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
jpranevich@kniggit.net
2008-Aug-22 22:53 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
Hello,
Without these changes, or at least similar changes, I was never getting the
hotplug scripts called in my config. Maybe it''s because I''m
using pygrub. Maybe it''s some other reason. I traced the offending code
to bootloader:
./tools/python/xen/xend/XendBootloader.py:
def bootloader(blexec, disk, dom, quiet = False, blargs = '''',
kernel = '''',
ramdisk = '''', kernel_args =
''''):
...
if not os.access(blexec, os.X_OK):
msg = "Bootloader isn''t executable"
log.error(msg)
raise VmError(msg)
if not os.access(disk, os.R_OK):
msg = "Disk isn''t accessible"
log.error(msg)
raise VmError(msg)
In this case, "disk" was set to the IQN and not a real file, so the
second check always failed. I could also show that at this point, the block-*
scripts hadn''t been called yet, so no real device existed to give it.
I backtracked further into the code and looked at how the SuSE patches worked.
In that case, the SuSE version of the code also made changes to
_configureBootloader() to create the VBD early so that it could pass a valid
disk name in the bootloader() call. It also added a bunch of other things which
they found it necessary to add for domUloader support, which I don''t
believe is relevant to what I''m doing now. I admit to not understanding
everything they were doing.
If you tell me where in the python code the hotplug scripts are supposed to get
called from, I''ll add copious debugging around that point in stock Xen
to see what it''s doing in my configuration and report back or provide a
fix.
There are no problems with your block-iscsi scripts. Those are perfectly fine.
They need tweaked a bit because the kernel maintainers keep changing the format
and layout of /sys between versions, but it should be a simple matter to select
the right file location. (It seems to be one of two.) I was going to send you a
patch to do this once I got further along.
Joe
----- Original Message -----
From: "Stefan de Konink" <skinkie@xs4all.nl>
To: jpranevich@kniggit.net
Cc: "Stephan Seitz" <s.seitz@netz-haut.de>,
"xen-devel" <xen-devel@lists.xensource.com>
Sent: Friday, August 22, 2008 6:16:07 PM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> This is the code that I''ve mimiced. Here,
> "BOOTLOADER_LOOPBACK_DEVICE" is always "/dev/xvdp".
But, in the case
> of ''tap'' devices, it gets destroyed later. (I
don''t want to destroy
> it later since the iSCSI mount needs to be kept around.)
You are working at the wrong layer... there are block-scripts... don''t
edit the xend python stuff!
And you understand that /dev/xvdp is for INSIDE the host?
But there is some other news... with Xen 3.3 pvSCSI is possible. Thus it
might be better to pass along a scsi generic device instead of a disk.
Stefan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Stefan de Konink
2008-Aug-22 23:05 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> Without these changes, or at least similar changes, I was never > getting the hotplug scripts called in my config. Maybe it''s because > I''m using pygrub. Maybe it''s some other reason. I traced the > offending code to bootloader:I guess here is where the problem is... I never used pygrub before it. So if you are hacking in PyGrub you might be in the right place ;)> If you tell me where in the python code the hotplug scripts are > supposed to get called from, I''ll add copious debugging around that > point in stock Xen to see what it''s doing in my configuration and > report back or provide a fix.You ask smart questions... but I don''t know ''how'' PyGrub does this. And actually I have no clue were in the actual booting process PyGrub steps in.> There are no problems with your block-iscsi scripts. Those are > perfectly fine. They need tweaked a bit because the kernel > maintainers keep changing the format and layout of /sys between > versions, but it should be a simple matter to select the right file > location. (It seems to be one of two.) I was going to send you a > patch to do this once I got further along.;) Yes... I Know ;) The Libvirt guys know it too... What I asked to the open-iscsi.org guys was to copy there utility functions to seperate library something like libopen-iscsi.so and create a program that does the /sys operations based on their code. (It could be interesting to make a block script in plain C, I have investigated this route for libvirt storage pools as well... but that one has an extra ''issue'', it should be able to do the ''tap'' interface too...) Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
jpranevich@kniggit.net
2008-Aug-23 01:54 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
----- "Stefan de Konink" <skinkie@xs4all.nl> wrote:> > I guess here is where the problem is... I never used pygrub before it. > > So if you are hacking in PyGrub you might be in the right place ;) >Can you tell me what bootloader you are using and whether you have any patches on top of stock Xen, particularly in XendDomainInfo.py or XendBootloader.py? I''ll try and reproduce your setup and trace the code path for what you are using. There are some weird conditional statements in the Python for what bootloader you are using (especially in the SuSE patches), and there could be something in there. As far as I can tell _configureBootloader(), where my changes are, sets up the environment for the bootloader. It could be that the hotplug code happens there or at some other place. I''ve had significant difficulties understanding it, which is why I was hoping to get a response from an "expert" on the list in this area. I understand how it works once /etc/xen/scripts/block is executed, but not all the assumptions leading up to that point.> > You ask smart questions... but I don''t know ''how'' PyGrub does this. > And > actually I have no clue were in the actual booting process PyGrub > steps in.Something like: _init_domain() _configure_bootloader() bootloader() I''ll look deeper into PyGrub, the new bootloader, and whatever you are using on Monday and try and figure out where my issue is from there. (As an aside, Xen really does seem intent on thinking that you need a real block device to boot from in a few other areas. I was playing around and was able to get RedHat AS 5.2''s newly-added root-on-iSCSI support working under Xen without any code modifications using the SuSE domUloader and RedHat''s initrd and kernel, but I still had to specify a block device, even though it''s ignored by the initrd which has its own mini-open-iscsi built in and mounts root from that. The RedHat support has too many limitations to be practical for Xen just yet, but it''s an interesting different approach. Still, having Xen manage the iSCSI connections appears to be the best way.) Joe _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan de Konink
2008-Aug-23 02:47 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> ----- "Stefan de Konink" <skinkie@xs4all.nl> wrote: > >> I guess here is where the problem is... I never used pygrub before >> it. >> >> So if you are hacking in PyGrub you might be in the right place ;) >> > > Can you tell me what bootloader you are using and whether you have > any patches on top of stock Xen, particularly in XendDomainInfo.py or > XendBootloader.py?I hope you don''t cry now, but I''m not using any bootloader :) I really can''t see the use for it. Actually I have *two* kernels now, but the only thing my users see is an linux-2.6 as kernel in their config.> (As an aside, Xen really does seem intent on thinking that you need a > real block device to boot from in a few other areas. I was playing > around and was able to get RedHat AS 5.2''s newly-added root-on-iSCSI > support working under Xen without any code modifications using the > SuSE domUloader and RedHat''s initrd and kernel, but I still had to > specify a block device, even though it''s ignored by the initrd which > has its own mini-open-iscsi built in and mounts root from that. The > RedHat support has too many limitations to be practical for Xen just > yet, but it''s an interesting different approach. Still, having Xen > manage the iSCSI connections appears to be the best way.)Oh they are doing: kernel initrd-hey-here-are-the-iscsi-tools-lets-switch-rootfs Instead the nfsway: kernel-lets-find-a-nice-iscsi-machine Not only the best way, the only way you can *safely* hide your storage from the domU''s. Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
jpranevich@kniggit.net
2008-Aug-23 15:04 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
----- "Stefan de Konink" <skinkie@xs4all.nl> wrote:> > I hope you don''t cry now, but I''m not using any bootloader :) I really > > can''t see the use for it. Actually I have *two* kernels now, but the > only thing my users see is an linux-2.6 as kernel in their config. >So I can reproduce it, how do you have it configured then? I''m not familiar with a sans-bootloader configuration.> > Not only the best way, the only way you can *safely* hide your storage > from the domU''s. >My specific application doesn''t need the security. I''m using this for legacy applications, not untrusted users. I probably wouldn''t be using paravirt if I wanted that level of security, but I''m not sure how the big ISPs do it. Joe _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan de Konink
2008-Aug-23 15:24 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> ----- "Stefan de Konink" <skinkie@xs4all.nl> wrote: > >> I hope you don''t cry now, but I''m not using any bootloader :) I really >> >> can''t see the use for it. Actually I have *two* kernels now, but the >> only thing my users see is an linux-2.6 as kernel in their config. >> > > So I can reproduce it, how do you have it configured then? I''m not familiar with a sans-bootloader configuration.kernel = "/home/skinkie/xen/boot/gentoo";>> Not only the best way, the only way you can *safely* hide your storage >> from the domU''s. >> > > My specific application doesn''t need the security. I''m using this for legacy applications, not untrusted users. I probably wouldn''t be using paravirt if I wanted that level of security, but I''m not sure how the big ISPs do it.Ok. Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
jpranevich@kniggit.net
2008-Aug-25 16:53 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
----- "Stefan de Konink" <skinkie@xs4all.nl> wrote:> > So I can reproduce it, how do you have it configured then? I''m not > familiar with a sans-bootloader configuration. > > kernel = "/home/skinkie/xen/boot/gentoo"; >Bingo! I''ve found that in your case, _configureBootloader() exits out almost immediately, and instead the root disk and hotplug scripts are run later-- probably in _createDevices()-- but I don''t completely understand that code yet. My issue appears to be here someplace. Since the bootloader gets called before the devices are created, we need to create the one-off VBD early and get the kernel/ramdisk from it. At that point, we can either destroy the VBD and let the subsequent call to _createDevices() recreate it (which is slow) or carry it forward. I''m not sure what the best mechanism would be. (Someone on this list probably understands this logic better and may have a strong opinion which is the correct way.)) I think there is also a second "bug" in Xen 3.3.0 on blkif.py, line 59. Are you using 3.3.0 yet? I''d be curious to see if you come to the same conclusion. try: (typ, params) = string.split(uname, '':'', 1) if typ not in (''phy'', ''file'', ''tap''): raise VmError( ''Block device must have "phy", "file" or "tap" '' ''specified to type'') That code appears to explicitly block any of the hotplug-type devices from working. I can get around this by adding "iscsi" to the supported types, and it works properly. I''m sure there''s a better fix. (Perhaps, we scan the scripts directory to look for block-* and add exceptions for all that match. Perhaps, there is a better way.) Anyway, I''m going to work up a revised patch now that I have a better inkling what is going on. My goal will be to get hotplug devices work with the bootloaders. (But, at least I have something that appears to work without needing that support.) Also, you mentioned pvgrub. Can you tell me how to use this? I did a "grep -R -i pvgrub xen-3.3.0" and can''t find it in the Xen 3.3.0 release. The only reference I can find to it is in the release notes. Can you give me a pointer where this lives and how it works, so that I can see if it suffers from the same problem? Thanks, Joe _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefan de Konink
2008-Aug-25 17:06 UTC
Re: [Xen-devel] Almost working iSCSI booting, need advice
jpranevich@kniggit.net schreef:> I think there is also a second "bug" in Xen 3.3.0 on blkif.py, line > 59. Are you using 3.3.0 yet? I''d be curious to see if you come to the > same conclusion.I have tested the latest Xen some weeks ago... but only with tapdisk.> Anyway, I''m going to work up a revised patch now that I have a better > inkling what is going on. My goal will be to get hotplug devices work > with the bootloaders. (But, at least I have something that appears to > work without needing that support.)Work is appreciated!> Also, you mentioned pvgrub. Can you tell me how to use this? I did a > "grep -R -i pvgrub xen-3.3.0" and can''t find it in the Xen 3.3.0 > release. The only reference I can find to it is in the release notes. > Can you give me a pointer where this lives and how it works, so that > I can see if it suffers from the same problem?That was another Steven :) and I think it goes about having an actual grub on the filesystem. Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel