Xen devel - Jan 2013 - pygrub/hvm boot with alternate script= for block devices

Hi,

I am doing some experimentation with xen and Ceph and have a problem
booting my guest when my disk = [] uses an alternate block script.
Installation from a .iso was ok since the boot device was a file but 
now
trying to boot from the rbd neither the hvmbuilder or pygrub can start
as they treat the first value after target= as the /dev node to try and
use.

My disk parameter looks like:
disk = [ ''format=raw, script=block-rbd, vdev=xvda, access=w,
target=image=ubuntu-test'' ]

In the pygrub log:
OSError: [Errno 2] No such file or directory:
''image=ubuntu-test''

and there is a similar error trying an HVM boot.

My block-rbd script parses the value passed after target= to
dynamically rbd map the image and then call the write_dev function from
block-common.sh to save the corresponding /dev name in xenstore.
According to the logging that I have in my block-rbd script this isn''t
even called before pygrub is executed.

Is there a good reason that the block devices aren''t connected before
pygrub is called so that it can then be invoked with the appropriate
/dev device that has been written in xenstore?

Thanks,
James

On 19/01/13 16:46, James Dingwall wrote:
> 
> Hi,
> 
> I am doing some experimentation with xen and Ceph and have a problem
> booting my guest when my disk = [] uses an alternate block script.
> Installation from a .iso was ok since the boot device was a file but 
> now
> trying to boot from the rbd neither the hvmbuilder or pygrub can start
> as they treat the first value after target= as the /dev node to try and
> use.
Yes, this is known problem which I''m trying to solve with this series:

http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
> 
> My disk parameter looks like:
> disk = [ ''format=raw, script=block-rbd, vdev=xvda, access=w,
> target=image=ubuntu-test'' ]
> 
> In the pygrub log:
> OSError: [Errno 2] No such file or directory:
''image=ubuntu-test''
> 
> and there is a similar error trying an HVM boot.
> 
> My block-rbd script parses the value passed after target= to
> dynamically rbd map the image and then call the write_dev function from
> block-common.sh to save the corresponding /dev name in xenstore.
> According to the logging that I have in my block-rbd script this
isn''t
> even called before pygrub is executed.
The only solution right now will be to use PV guests and boot directly
with the kernel (no pygrub)
> Is there a good reason that the block devices aren''t connected
before
> pygrub is called so that it can then be invoked with the appropriate
> /dev device that has been written in xenstore?

On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné
wrote:
> On 19/01/13 16:46, James Dingwall wrote:
> > 
> > Hi,
> > 
> > I am doing some experimentation with xen and Ceph and have a problem
> > booting my guest when my disk = [] uses an alternate block script.
> > Installation from a .iso was ok since the boot device was a file but 
> > now
> > trying to boot from the rbd neither the hvmbuilder or pygrub can start
> > as they treat the first value after target= as the /dev node to try
and
> > use.
> 
> Yes, this is known problem which I'm trying to solve with this series:
> 
> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
> 
> > 
> > My disk parameter looks like:
> > disk = [ 'format=raw, script=block-rbd, vdev=xvda, access=w,
> > target=image=ubuntu-test' ]
> > 
> > In the pygrub log:
> > OSError: [Errno 2] No such file or directory:
'image=ubuntu-test'
> > 
> > and there is a similar error trying an HVM boot.
> > 
> > My block-rbd script parses the value passed after target= to
> > dynamically rbd map the image and then call the write_dev function
from
> > block-common.sh to save the corresponding /dev name in xenstore.
> > According to the logging that I have in my block-rbd script this
isn't
> > even called before pygrub is executed.
> 
> The only solution right now will be to use PV guests and boot directly
> with the kernel (no pygrub)
Does commenting out the stat & check in the short term allow things to
work well enough for James' purposes?

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 2013-01-21 09:56, Roger Pau Monné wrote:
> On 19/01/13 16:46, James Dingwall wrote:
>> I am doing some experimentation with xen and Ceph and have a problem
>> booting my guest when my disk = [] uses an alternate block script.
>> Installation from a .iso was ok since the boot device was a file but
>> now
>> trying to boot from the rbd neither the hvmbuilder or pygrub can 
>> start
>> as they treat the first value after target= as the /dev node to try 
>> and
>> use.
>
> Yes, this is known problem which I'm trying to solve with this 
> series:
>
> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
If these will apply on 4.2.1 then would you like any help testing these 
patches? I can also try writing a block-rbd script with a similar 
structure to the iscsi one that you propose in 10/10.

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 21/01/13 11:01, Ian Campbell wrote:
> On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote:
>> On 19/01/13 16:46, James Dingwall wrote:
>>>
>>> Hi,
>>>
>>> I am doing some experimentation with xen and Ceph and have a
problem
>>> booting my guest when my disk = [] uses an alternate block script.
>>> Installation from a .iso was ok since the boot device was a file
but
>>> now
>>> trying to boot from the rbd neither the hvmbuilder or pygrub can
start
>>> as they treat the first value after target= as the /dev node to try
and
>>> use.
>>
>> Yes, this is known problem which I'm trying to solve with this
series:
>>
>> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
>>
>>>
>>> My disk parameter looks like:
>>> disk = [ 'format=raw, script=block-rbd, vdev=xvda, access=w,
>>> target=image=ubuntu-test' ]
>>>
>>> In the pygrub log:
>>> OSError: [Errno 2] No such file or directory:
'image=ubuntu-test'
>>>
>>> and there is a similar error trying an HVM boot.
>>>
>>> My block-rbd script parses the value passed after target= to
>>> dynamically rbd map the image and then call the write_dev function
from
>>> block-common.sh to save the corresponding /dev name in xenstore.
>>> According to the logging that I have in my block-rbd script this
isn't
>>> even called before pygrub is executed.
>>
>> The only solution right now will be to use PV guests and boot directly
>> with the kernel (no pygrub)
> 
> Does commenting out the stat & check in the short term allow things to
> work well enough for James' purposes?
Current hotplug scripts don't have any need to write the dev in
xenstore, they only need to write the device major:minor in the
physical-device xenstore backend entry. The only way to solve this (with
current hotplug scripts) will be to call libxl_device_disk_add earlier
on (before pygrub), fetch "physical-device" from xenstore and
translate
that back into the physical device it points to, and overwrite pdev_path
in libxl_device_disk struct with this value.

I would rather wait for the new interface to be finished rather than
adding this "hack" to libxl.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 21/01/13 11:14, James Dingwall wrote:
> On 2013-01-21 09:56, Roger Pau Monné wrote:
>> On 19/01/13 16:46, James Dingwall wrote:
>>> I am doing some experimentation with xen and Ceph and have a
problem
>>> booting my guest when my disk = [] uses an alternate block script.
>>> Installation from a .iso was ok since the boot device was a file
but
>>> now
>>> trying to boot from the rbd neither the hvmbuilder or pygrub can 
>>> start
>>> as they treat the first value after target= as the /dev node to try
>>> and
>>> use.
>>
>> Yes, this is known problem which I'm trying to solve with this 
>> series:
>>
>> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
> 
> If these will apply on 4.2.1 then would you like any help testing these 
> patches? I can also try writing a block-rbd script with a similar 
> structure to the iscsi one that you propose in 10/10.
Haven't tried applying them to 4.2.1 but there hasn't been changes in
the device handling code since the release, so I would say that you
should be able to apply them without much problem.

If you prefer I can also publish the patches as a branch in my local git
repo, so you can fetch them easily (that will be based on top of -unstable)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Mon, 2013-01-21 at 10:19 +0000, Roger Pau Monne
wrote:
> On 21/01/13 11:01, Ian Campbell wrote:
> > On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote:
> >> On 19/01/13 16:46, James Dingwall wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am doing some experimentation with xen and Ceph and have a
problem
> >>> booting my guest when my disk = [] uses an alternate block
script.
> >>> Installation from a .iso was ok since the boot device was a
file but
> >>> now
> >>> trying to boot from the rbd neither the hvmbuilder or pygrub
can start
> >>> as they treat the first value after target= as the /dev node
to try and
> >>> use.
> >>
> >> Yes, this is known problem which I'm trying to solve with this
series:
> >>
> >> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
> >>
> >>>
> >>> My disk parameter looks like:
> >>> disk = [ 'format=raw, script=block-rbd, vdev=xvda,
access=w,
> >>> target=image=ubuntu-test' ]
> >>>
> >>> In the pygrub log:
> >>> OSError: [Errno 2] No such file or directory:
'image=ubuntu-test'
> >>>
> >>> and there is a similar error trying an HVM boot.
> >>>
> >>> My block-rbd script parses the value passed after target= to
> >>> dynamically rbd map the image and then call the write_dev
function from
> >>> block-common.sh to save the corresponding /dev name in
xenstore.
> >>> According to the logging that I have in my block-rbd script
this isn't
> >>> even called before pygrub is executed.
> >>
> >> The only solution right now will be to use PV guests and boot
directly
> >> with the kernel (no pygrub)
> > 
> > Does commenting out the stat & check in the short term allow
things to
> > work well enough for James' purposes?
> 
> Current hotplug scripts don't have any need to write the dev in
> xenstore, they only need to write the device major:minor in the
> physical-device xenstore backend entry. The only way to solve this (with
> current hotplug scripts) will be to call libxl_device_disk_add earlier
> on (before pygrub), fetch "physical-device" from xenstore and
translate
> that back into the physical device it points to, and overwrite pdev_path
> in libxl_device_disk struct with this value.
OK, I think I follow. Presumably a similar hack would be needed to make
the local_attach interface do something useful here too.
> I would rather wait for the new interface to be finished rather than
> adding this "hack" to libxl.
Fair enough.

In the meantime I suppose James could work around this by logging into
rdb manually (or in a script) before starting the domain and
specifying /dev/rdbN in the guest config?

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 21/01/13 11:48, Ian Campbell wrote:
> On Mon, 2013-01-21 at 10:19 +0000, Roger Pau Monne wrote:
>> On 21/01/13 11:01, Ian Campbell wrote:
>>> On Mon, 2013-01-21 at 09:56 +0000, Roger Pau Monné wrote:
>>>> On 19/01/13 16:46, James Dingwall wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am doing some experimentation with xen and Ceph and have
a problem
>>>>> booting my guest when my disk = [] uses an alternate block
script.
>>>>> Installation from a .iso was ok since the boot device was a
file but
>>>>> now
>>>>> trying to boot from the rbd neither the hvmbuilder or
pygrub can start
>>>>> as they treat the first value after target= as the /dev
node to try and
>>>>> use.
>>>>
>>>> Yes, this is known problem which I'm trying to solve with
this series:
>>>>
>>>>
http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
>>>>
>>>>>
>>>>> My disk parameter looks like:
>>>>> disk = [ 'format=raw, script=block-rbd, vdev=xvda,
access=w,
>>>>> target=image=ubuntu-test' ]
>>>>>
>>>>> In the pygrub log:
>>>>> OSError: [Errno 2] No such file or directory:
'image=ubuntu-test'
>>>>>
>>>>> and there is a similar error trying an HVM boot.
>>>>>
>>>>> My block-rbd script parses the value passed after target=
to
>>>>> dynamically rbd map the image and then call the write_dev
function from
>>>>> block-common.sh to save the corresponding /dev name in
xenstore.
>>>>> According to the logging that I have in my block-rbd script
this isn't
>>>>> even called before pygrub is executed.
>>>>
>>>> The only solution right now will be to use PV guests and boot
directly
>>>> with the kernel (no pygrub)
>>>
>>> Does commenting out the stat & check in the short term allow
things to
>>> work well enough for James' purposes?
>>
>> Current hotplug scripts don't have any need to write the dev in
>> xenstore, they only need to write the device major:minor in the
>> physical-device xenstore backend entry. The only way to solve this
(with
>> current hotplug scripts) will be to call libxl_device_disk_add earlier
>> on (before pygrub), fetch "physical-device" from xenstore and
translate
>> that back into the physical device it points to, and overwrite
pdev_path
>> in libxl_device_disk struct with this value.
> 
> OK, I think I follow. Presumably a similar hack would be needed to make
> the local_attach interface do something useful here too.
> 
>> I would rather wait for the new interface to be finished rather than
>> adding this "hack" to libxl.
> 
> Fair enough.
> 
> In the meantime I suppose James could work around this by logging into
> rdb manually (or in a script) before starting the domain and
> specifying /dev/rdbN in the guest config?
Yes, I guess that's what most high-level toolstacks (like libvirt or
XAPI) do for supporting this kind of backends.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 2013-01-21 10:26, Roger Pau Monné wrote:
> On 21/01/13 11:14, James Dingwall wrote:
>> On 2013-01-21 09:56, Roger Pau Monné wrote:
>>> On 19/01/13 16:46, James Dingwall wrote:
>>>> I am doing some experimentation with xen and Ceph and have a 
>>>> problem
>>>> booting my guest when my disk = [] uses an alternate block
script.
>>>> Installation from a .iso was ok since the boot device was a
file
>>>> but
>>>> now
>>>> trying to boot from the rbd neither the hvmbuilder or pygrub
can
>>>> start
>>>> as they treat the first value after target= as the /dev node to
>>>> try
>>>> and
>>>> use.
>>>
>>> Yes, this is known problem which I'm trying to solve with this
>>> series:
>>>
>>> http://lists.xen.org/archives/html/xen-devel/2012-12/msg01753.html
>>
>> If these will apply on 4.2.1 then would you like any help testing 
>> these
>> patches? I can also try writing a block-rbd script with a similar
>> structure to the iscsi one that you propose in 10/10.
>
> Haven't tried applying them to 4.2.1 but there hasn't been changes
in
> the device handling code since the release, so I would say that you
> should be able to apply them without much problem.
>
> If you prefer I can also publish the patches as a branch in my local 
> git
> repo, so you can fetch them easily (that will be based on top of 
> -unstable)
If this isn't too much effort for you then I can work (out how to work) 
with a git branch.

Regards,
James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 22/01/13 09:50, James Dingwall wrote:
> If this isn''t too much effort for you then I can work (out how to
work)
> with a git branch.
Hello James,

I''m in the process of rebasing + updating, will publish the new
revision
on a public git branch.

Thanks, Roger.