James Dingwall
2013-Jan-19 15:46 UTC
pygrub/hvm boot with alternate script= for block devices
Hi, I am doing some experimentation with xen and Ceph and have a problem booting my guest when my disk = [] uses an alternate block script. Installation from a .iso was ok since the boot device was a file but now trying to boot from the rbd neither the hvmbuilder or pygrub can start as they treat the first value after target= as the /dev node to try and use. My disk parameter looks like: disk = [ ''format=raw, script=block-rbd, vdev=xvda, access=w, target=image=ubuntu-test'' ] In the pygrub log: OSError: [Errno 2] No such file or directory: ''image=ubuntu-test'' and there is a similar error trying an HVM boot. My block-rbd script parses the value passed after target= to dynamically rbd map the image and then call the write_dev function from block-common.sh to save the corresponding /dev name in xenstore. According to the logging that I have in my block-rbd script this isn''t even called before pygrub is executed. Is there a good reason that the block devices aren''t connected before pygrub is called so that it can then be invoked with the appropriate /dev device that has been written in xenstore? Thanks, James
Roger Pau Monné
2013-Jan-21 09:56 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 19/01/13 16:46, James Dingwall wrote:> > Hi, > > I am doing some experimentation with xen and Ceph and have a problem > booting my guest when my disk = [] uses an alternate block script. > Installation from a .iso was ok since the boot device was a file but > now > trying to boot from the rbd neither the hvmbuilder or pygrub can start > as they treat the first value after target= as the /dev node to try and > use.Yes, this is known problem which I''m trying to solve with this series: http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html> > My disk parameter looks like: > disk = [ ''format=raw, script=block-rbd, vdev=xvda, access=w, > target=image=ubuntu-test'' ] > > In the pygrub log: > OSError: [Errno 2] No such file or directory: ''image=ubuntu-test'' > > and there is a similar error trying an HVM boot. > > My block-rbd script parses the value passed after target= to > dynamically rbd map the image and then call the write_dev function from > block-common.sh to save the corresponding /dev name in xenstore. > According to the logging that I have in my block-rbd script this isn''t > even called before pygrub is executed.The only solution right now will be to use PV guests and boot directly with the kernel (no pygrub)> Is there a good reason that the block devices aren''t connected before > pygrub is called so that it can then be invoked with the appropriate > /dev device that has been written in xenstore?
Ian Campbell
2013-Jan-21 10:01 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote:> On 19/01/13 16:46, James Dingwall wrote: > > > > Hi, > > > > I am doing some experimentation with xen and Ceph and have a problem > > booting my guest when my disk = [] uses an alternate block script. > > Installation from a .iso was ok since the boot device was a file but > > now > > trying to boot from the rbd neither the hvmbuilder or pygrub can start > > as they treat the first value after target= as the /dev node to try and > > use. > > Yes, this is known problem which I'm trying to solve with this series: > > http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html > > > > > My disk parameter looks like: > > disk = [ 'format=raw, script=block-rbd, vdev=xvda, access=w, > > target=image=ubuntu-test' ] > > > > In the pygrub log: > > OSError: [Errno 2] No such file or directory: 'image=ubuntu-test' > > > > and there is a similar error trying an HVM boot. > > > > My block-rbd script parses the value passed after target= to > > dynamically rbd map the image and then call the write_dev function from > > block-common.sh to save the corresponding /dev name in xenstore. > > According to the logging that I have in my block-rbd script this isn't > > even called before pygrub is executed. > > The only solution right now will be to use PV guests and boot directly > with the kernel (no pygrub)Does commenting out the stat & check in the short term allow things to work well enough for James' purposes? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
James Dingwall
2013-Jan-21 10:14 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 2013-01-21 09:56, Roger Pau Monné wrote:> On 19/01/13 16:46, James Dingwall wrote: >> I am doing some experimentation with xen and Ceph and have a problem >> booting my guest when my disk = [] uses an alternate block script. >> Installation from a .iso was ok since the boot device was a file but >> now >> trying to boot from the rbd neither the hvmbuilder or pygrub can >> start >> as they treat the first value after target= as the /dev node to try >> and >> use. > > Yes, this is known problem which I'm trying to solve with this > series: > > http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.htmlIf these will apply on 4.2.1 then would you like any help testing these patches? I can also try writing a block-rbd script with a similar structure to the iscsi one that you propose in 10/10. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Roger Pau Monné
2013-Jan-21 10:19 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 21/01/13 11:01, Ian Campbell wrote:> On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote: >> On 19/01/13 16:46, James Dingwall wrote: >>> >>> Hi, >>> >>> I am doing some experimentation with xen and Ceph and have a problem >>> booting my guest when my disk = [] uses an alternate block script. >>> Installation from a .iso was ok since the boot device was a file but >>> now >>> trying to boot from the rbd neither the hvmbuilder or pygrub can start >>> as they treat the first value after target= as the /dev node to try and >>> use. >> >> Yes, this is known problem which I'm trying to solve with this series: >> >> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html >> >>> >>> My disk parameter looks like: >>> disk = [ 'format=raw, script=block-rbd, vdev=xvda, access=w, >>> target=image=ubuntu-test' ] >>> >>> In the pygrub log: >>> OSError: [Errno 2] No such file or directory: 'image=ubuntu-test' >>> >>> and there is a similar error trying an HVM boot. >>> >>> My block-rbd script parses the value passed after target= to >>> dynamically rbd map the image and then call the write_dev function from >>> block-common.sh to save the corresponding /dev name in xenstore. >>> According to the logging that I have in my block-rbd script this isn't >>> even called before pygrub is executed. >> >> The only solution right now will be to use PV guests and boot directly >> with the kernel (no pygrub) > > Does commenting out the stat & check in the short term allow things to > work well enough for James' purposes?Current hotplug scripts don't have any need to write the dev in xenstore, they only need to write the device major:minor in the physical-device xenstore backend entry. The only way to solve this (with current hotplug scripts) will be to call libxl_device_disk_add earlier on (before pygrub), fetch "physical-device" from xenstore and translate that back into the physical device it points to, and overwrite pdev_path in libxl_device_disk struct with this value. I would rather wait for the new interface to be finished rather than adding this "hack" to libxl. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Roger Pau Monné
2013-Jan-21 10:26 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 21/01/13 11:14, James Dingwall wrote:> On 2013-01-21 09:56, Roger Pau Monné wrote: >> On 19/01/13 16:46, James Dingwall wrote: >>> I am doing some experimentation with xen and Ceph and have a problem >>> booting my guest when my disk = [] uses an alternate block script. >>> Installation from a .iso was ok since the boot device was a file but >>> now >>> trying to boot from the rbd neither the hvmbuilder or pygrub can >>> start >>> as they treat the first value after target= as the /dev node to try >>> and >>> use. >> >> Yes, this is known problem which I'm trying to solve with this >> series: >> >> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html > > If these will apply on 4.2.1 then would you like any help testing these > patches? I can also try writing a block-rbd script with a similar > structure to the iscsi one that you propose in 10/10.Haven't tried applying them to 4.2.1 but there hasn't been changes in the device handling code since the release, so I would say that you should be able to apply them without much problem. If you prefer I can also publish the patches as a branch in my local git repo, so you can fetch them easily (that will be based on top of -unstable) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Ian Campbell
2013-Jan-21 10:48 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On Mon, 2013-01-21 at 10:19 +0000, Roger Pau Monne wrote:> On 21/01/13 11:01, Ian Campbell wrote: > > On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote: > >> On 19/01/13 16:46, James Dingwall wrote: > >>> > >>> Hi, > >>> > >>> I am doing some experimentation with xen and Ceph and have a problem > >>> booting my guest when my disk = [] uses an alternate block script. > >>> Installation from a .iso was ok since the boot device was a file but > >>> now > >>> trying to boot from the rbd neither the hvmbuilder or pygrub can start > >>> as they treat the first value after target= as the /dev node to try and > >>> use. > >> > >> Yes, this is known problem which I'm trying to solve with this series: > >> > >> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html > >> > >>> > >>> My disk parameter looks like: > >>> disk = [ 'format=raw, script=block-rbd, vdev=xvda, access=w, > >>> target=image=ubuntu-test' ] > >>> > >>> In the pygrub log: > >>> OSError: [Errno 2] No such file or directory: 'image=ubuntu-test' > >>> > >>> and there is a similar error trying an HVM boot. > >>> > >>> My block-rbd script parses the value passed after target= to > >>> dynamically rbd map the image and then call the write_dev function from > >>> block-common.sh to save the corresponding /dev name in xenstore. > >>> According to the logging that I have in my block-rbd script this isn't > >>> even called before pygrub is executed. > >> > >> The only solution right now will be to use PV guests and boot directly > >> with the kernel (no pygrub) > > > > Does commenting out the stat & check in the short term allow things to > > work well enough for James' purposes? > > Current hotplug scripts don't have any need to write the dev in > xenstore, they only need to write the device major:minor in the > physical-device xenstore backend entry. The only way to solve this (with > current hotplug scripts) will be to call libxl_device_disk_add earlier > on (before pygrub), fetch "physical-device" from xenstore and translate > that back into the physical device it points to, and overwrite pdev_path > in libxl_device_disk struct with this value.OK, I think I follow. Presumably a similar hack would be needed to make the local_attach interface do something useful here too.> I would rather wait for the new interface to be finished rather than > adding this "hack" to libxl.Fair enough. In the meantime I suppose James could work around this by logging into rdb manually (or in a script) before starting the domain and specifying /dev/rdbN in the guest config? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Roger Pau Monné
2013-Jan-21 12:03 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 21/01/13 11:48, Ian Campbell wrote:> On Mon, 2013-01-21 at 10:19 +0000, Roger Pau Monne wrote: >> On 21/01/13 11:01, Ian Campbell wrote: >>> On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote: >>>> On 19/01/13 16:46, James Dingwall wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am doing some experimentation with xen and Ceph and have a problem >>>>> booting my guest when my disk = [] uses an alternate block script. >>>>> Installation from a .iso was ok since the boot device was a file but >>>>> now >>>>> trying to boot from the rbd neither the hvmbuilder or pygrub can start >>>>> as they treat the first value after target= as the /dev node to try and >>>>> use. >>>> >>>> Yes, this is known problem which I'm trying to solve with this series: >>>> >>>> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html >>>> >>>>> >>>>> My disk parameter looks like: >>>>> disk = [ 'format=raw, script=block-rbd, vdev=xvda, access=w, >>>>> target=image=ubuntu-test' ] >>>>> >>>>> In the pygrub log: >>>>> OSError: [Errno 2] No such file or directory: 'image=ubuntu-test' >>>>> >>>>> and there is a similar error trying an HVM boot. >>>>> >>>>> My block-rbd script parses the value passed after target= to >>>>> dynamically rbd map the image and then call the write_dev function from >>>>> block-common.sh to save the corresponding /dev name in xenstore. >>>>> According to the logging that I have in my block-rbd script this isn't >>>>> even called before pygrub is executed. >>>> >>>> The only solution right now will be to use PV guests and boot directly >>>> with the kernel (no pygrub) >>> >>> Does commenting out the stat & check in the short term allow things to >>> work well enough for James' purposes? >> >> Current hotplug scripts don't have any need to write the dev in >> xenstore, they only need to write the device major:minor in the >> physical-device xenstore backend entry. The only way to solve this (with >> current hotplug scripts) will be to call libxl_device_disk_add earlier >> on (before pygrub), fetch "physical-device" from xenstore and translate >> that back into the physical device it points to, and overwrite pdev_path >> in libxl_device_disk struct with this value. > > OK, I think I follow. Presumably a similar hack would be needed to make > the local_attach interface do something useful here too. > >> I would rather wait for the new interface to be finished rather than >> adding this "hack" to libxl. > > Fair enough. > > In the meantime I suppose James could work around this by logging into > rdb manually (or in a script) before starting the domain and > specifying /dev/rdbN in the guest config?Yes, I guess that's what most high-level toolstacks (like libvirt or XAPI) do for supporting this kind of backends. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
James Dingwall
2013-Jan-22 08:50 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 2013-01-21 10:26, Roger Pau Monné wrote:> On 21/01/13 11:14, James Dingwall wrote: >> On 2013-01-21 09:56, Roger Pau Monné wrote: >>> On 19/01/13 16:46, James Dingwall wrote: >>>> I am doing some experimentation with xen and Ceph and have a >>>> problem >>>> booting my guest when my disk = [] uses an alternate block script. >>>> Installation from a .iso was ok since the boot device was a file >>>> but >>>> now >>>> trying to boot from the rbd neither the hvmbuilder or pygrub can >>>> start >>>> as they treat the first value after target= as the /dev node to >>>> try >>>> and >>>> use. >>> >>> Yes, this is known problem which I'm trying to solve with this >>> series: >>> >>> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html >> >> If these will apply on 4.2.1 then would you like any help testing >> these >> patches? I can also try writing a block-rbd script with a similar >> structure to the iscsi one that you propose in 10/10. > > Haven't tried applying them to 4.2.1 but there hasn't been changes in > the device handling code since the release, so I would say that you > should be able to apply them without much problem. > > If you prefer I can also publish the patches as a branch in my local > git > repo, so you can fetch them easily (that will be based on top of > -unstable)If this isn't too much effort for you then I can work (out how to work) with a git branch. Regards, James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Roger Pau Monné
2013-Jan-22 09:05 UTC
Re: pygrub/hvm boot with alternate script= for block devices
On 22/01/13 09:50, James Dingwall wrote:> If this isn''t too much effort for you then I can work (out how to work) > with a git branch.Hello James, I''m in the process of rebasing + updating, will publish the new revision on a public git branch. Thanks, Roger.