Xen devel - Jul 2013 - RFC: Automatically making a PCI device assignable in the config file

I''ve been doing some work to try to make driver domains easier to set
up and use.  At the moment, in order to pass a device through to a
guest, you first need to assign it to pciback.  This involves doing
one of three things:
* Running xl pci-assignable-add for the device
* Specifying the device to be grabbed on the dom0 Linux command-line
* Doing some hackery in /etc/modules.d

None of these are very satisfying.  What I think would be better is if
there was a way to specify in the guest config file, "If device X is
not assignable, try to make it assignable".  That way you can have a
driver domain grab the appropriate device just by running "xl create
domnet"; and once we have the xendomains script up and running with
xl, you can simply configure your domnet appropriately, and then put
it in /etc/xen/auto, to be started automatically on boot.

My initial idea was to add a parameter to the pci argument in the
config file; for example:

pci = [''08:04.1,permissive=1,seize=1'']

The ''seize=1'' would indicate that if bdf 08:04.1 is not
already
assignable, that xl should try to make is assignable.

The problem here is that this would need to be parsed by
xlu_pci_parse_bdf(), which only takes an argumen tof type
libxl_device_pci.

Now it seems to me that the right place to do this "seizing" is in xl,
not inside libxl -- the functions for doing assignment exist already,
and are simple and straightforward.  But doing it in xl, but as a
parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
pass something else back, which begins to get awkward.

So it seems to me we have a couple of options:
1. Create a new argument, "pci_seize" or something like that, which
would be processed separately from pci
2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
arguments directed at xl rather than libxl
3. Add "seize" to libxl_device_pci, but have it only used by xl
4. Add "seize" to libxl_device_pci, and have libxl do the seizing.

Any preference -- or any other ideas?

 -George

On 05/07/13 12:01, George Dunlap wrote:
> I''ve been doing some work to try to make driver domains easier to
set
> up and use.  At the moment, in order to pass a device through to a
> guest, you first need to assign it to pciback.  This involves doing
> one of three things:
> * Running xl pci-assignable-add for the device
> * Specifying the device to be grabbed on the dom0 Linux command-line
> * Doing some hackery in /etc/modules.d
>
> None of these are very satisfying.  What I think would be better is if
> there was a way to specify in the guest config file, "If device X is
> not assignable, try to make it assignable".  That way you can have a
> driver domain grab the appropriate device just by running "xl create
> domnet"; and once we have the xendomains script up and running with
> xl, you can simply configure your domnet appropriately, and then put
> it in /etc/xen/auto, to be started automatically on boot.
>
> My initial idea was to add a parameter to the pci argument in the
> config file; for example:
>
> pci = [''08:04.1,permissive=1,seize=1'']
>
> The ''seize=1'' would indicate that if bdf 08:04.1 is not
already
> assignable, that xl should try to make is assignable.
>
> The problem here is that this would need to be parsed by
> xlu_pci_parse_bdf(), which only takes an argumen tof type
> libxl_device_pci.
>
> Now it seems to me that the right place to do this "seizing" is
in xl,
> not inside libxl -- the functions for doing assignment exist already,
> and are simple and straightforward.  But doing it in xl, but as a
> parameter of the "pci" setting, means changing
xlu_pci_parse_bdf() to
> pass something else back, which begins to get awkward.
>
> So it seems to me we have a couple of options:
> 1. Create a new argument, "pci_seize" or something like that,
which
> would be processed separately from pci
> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
> arguments directed at xl rather than libxl
> 3. Add "seize" to libxl_device_pci, but have it only used by xl
> 4. Add "seize" to libxl_device_pci, and have libxl do the
seizing.
>
> Any preference -- or any other ideas?
>
>  -George
How about a setting in xl.conf of "auto-seize pci devices" ?  That way
the seizing is entirely part of xl

~Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On 05/07/13 14:39, Andrew Cooper wrote:
> On 05/07/13 12:01, George Dunlap wrote:
>> I''ve been doing some work to try to make driver domains easier
to set
>> up and use.  At the moment, in order to pass a device through to a
>> guest, you first need to assign it to pciback.  This involves doing
>> one of three things:
>> * Running xl pci-assignable-add for the device
>> * Specifying the device to be grabbed on the dom0 Linux command-line
>> * Doing some hackery in /etc/modules.d
>>
>> None of these are very satisfying.  What I think would be better is if
>> there was a way to specify in the guest config file, "If device X
is
>> not assignable, try to make it assignable".  That way you can have
a
>> driver domain grab the appropriate device just by running "xl
create
>> domnet"; and once we have the xendomains script up and running
with
>> xl, you can simply configure your domnet appropriately, and then put
>> it in /etc/xen/auto, to be started automatically on boot.
>>
>> My initial idea was to add a parameter to the pci argument in the
>> config file; for example:
>>
>> pci = [''08:04.1,permissive=1,seize=1'']
>>
>> The ''seize=1'' would indicate that if bdf 08:04.1 is
not already
>> assignable, that xl should try to make is assignable.
>>
>> The problem here is that this would need to be parsed by
>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>> libxl_device_pci.
>>
>> Now it seems to me that the right place to do this "seizing"
is in xl,
>> not inside libxl -- the functions for doing assignment exist already,
>> and are simple and straightforward.  But doing it in xl, but as a
>> parameter of the "pci" setting, means changing
xlu_pci_parse_bdf() to
>> pass something else back, which begins to get awkward.
>>
>> So it seems to me we have a couple of options:
>> 1. Create a new argument, "pci_seize" or something like that,
which
>> would be processed separately from pci
>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>> arguments directed at xl rather than libxl
>> 3. Add "seize" to libxl_device_pci, but have it only used by
xl
>> 4. Add "seize" to libxl_device_pci, and have libxl do the
seizing.
>>
>> Any preference -- or any other ideas?
>>
>>   -George
> How about a setting in xl.conf of "auto-seize pci devices" ? 
That way
> the seizing is entirely part of xl
Auto-seizing is fairly dangerous; you could easily accidentally yank out 
the ethernet card, or even the disk that dom0 is using.  I really think 
it should have to be enabled on a device-by-device basis.

I suppose another option would be to be able to set, in xl.conf, a list 
of auto-seizeable devices.  I don''t really like that option as well, 
though.  I''d rather be able to keep all the configuration in one place.

  -George

On 05/07/13 14:45, George Dunlap wrote:
> On 05/07/13 14:39, Andrew Cooper wrote:
>> On 05/07/13 12:01, George Dunlap wrote:
>>> I''ve been doing some work to try to make driver domains
easier to set
>>> up and use.  At the moment, in order to pass a device through to a
>>> guest, you first need to assign it to pciback.  This involves doing
>>> one of three things:
>>> * Running xl pci-assignable-add for the device
>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>> * Doing some hackery in /etc/modules.d
>>>
>>> None of these are very satisfying.  What I think would be better is
if
>>> there was a way to specify in the guest config file, "If
device X is
>>> not assignable, try to make it assignable".  That way you can
have a
>>> driver domain grab the appropriate device just by running "xl
create
>>> domnet"; and once we have the xendomains script up and running
with
>>> xl, you can simply configure your domnet appropriately, and then
put
>>> it in /etc/xen/auto, to be started automatically on boot.
>>>
>>> My initial idea was to add a parameter to the pci argument in the
>>> config file; for example:
>>>
>>> pci = [''08:04.1,permissive=1,seize=1'']
>>>
>>> The ''seize=1'' would indicate that if bdf 08:04.1
is not already
>>> assignable, that xl should try to make is assignable.
>>>
>>> The problem here is that this would need to be parsed by
>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>> libxl_device_pci.
>>>
>>> Now it seems to me that the right place to do this
"seizing" is in xl,
>>> not inside libxl -- the functions for doing assignment exist
already,
>>> and are simple and straightforward.  But doing it in xl, but as a
>>> parameter of the "pci" setting, means changing
xlu_pci_parse_bdf() to
>>> pass something else back, which begins to get awkward.
>>>
>>> So it seems to me we have a couple of options:
>>> 1. Create a new argument, "pci_seize" or something like
that, which
>>> would be processed separately from pci
>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct,
for
>>> arguments directed at xl rather than libxl
>>> 3. Add "seize" to libxl_device_pci, but have it only used
by xl
>>> 4. Add "seize" to libxl_device_pci, and have libxl do the
seizing.
>>>
>>> Any preference -- or any other ideas?
>>>
>>>   -George
>> How about a setting in xl.conf of "auto-seize pci devices" ? 
That way
>> the seizing is entirely part of xl
>
> Auto-seizing is fairly dangerous; you could easily accidentally yank
> out the ethernet card, or even the disk that dom0 is using.  I really
> think it should have to be enabled on a device-by-device basis.
>
> I suppose another option would be to be able to set, in xl.conf, a
> list of auto-seizeable devices.  I don''t really like that option
as
> well, though.  I''d rather be able to keep all the configuration in
one
> place.
>
>  -George
Or a slight less extreme version.

If xl sees that it would need seize a device, it could ask "You are
trying to create a domain with device $FOO.  Would you like to seize it
from dom0 ?"

I do think that libxl is not the correct place to have any logic like this.

~Andrew

On 05/07/13 14:48, Andrew Cooper wrote:
> On 05/07/13 14:45, George Dunlap wrote:
>> On 05/07/13 14:39, Andrew Cooper wrote:
>>> On 05/07/13 12:01, George Dunlap wrote:
>>>> I''ve been doing some work to try to make driver
domains easier to set
>>>> up and use.  At the moment, in order to pass a device through
to a
>>>> guest, you first need to assign it to pciback.  This involves
doing
>>>> one of three things:
>>>> * Running xl pci-assignable-add for the device
>>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>>> * Doing some hackery in /etc/modules.d
>>>>
>>>> None of these are very satisfying.  What I think would be
better is if
>>>> there was a way to specify in the guest config file, "If
device X is
>>>> not assignable, try to make it assignable".  That way you
can have a
>>>> driver domain grab the appropriate device just by running
"xl create
>>>> domnet"; and once we have the xendomains script up and
running with
>>>> xl, you can simply configure your domnet appropriately, and
then put
>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>
>>>> My initial idea was to add a parameter to the pci argument in
the
>>>> config file; for example:
>>>>
>>>> pci = [''08:04.1,permissive=1,seize=1'']
>>>>
>>>> The ''seize=1'' would indicate that if bdf
08:04.1 is not already
>>>> assignable, that xl should try to make is assignable.
>>>>
>>>> The problem here is that this would need to be parsed by
>>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>>> libxl_device_pci.
>>>>
>>>> Now it seems to me that the right place to do this
"seizing" is in xl,
>>>> not inside libxl -- the functions for doing assignment exist
already,
>>>> and are simple and straightforward.  But doing it in xl, but as
a
>>>> parameter of the "pci" setting, means changing
xlu_pci_parse_bdf() to
>>>> pass something else back, which begins to get awkward.
>>>>
>>>> So it seems to me we have a couple of options:
>>>> 1. Create a new argument, "pci_seize" or something
like that, which
>>>> would be processed separately from pci
>>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra
struct, for
>>>> arguments directed at xl rather than libxl
>>>> 3. Add "seize" to libxl_device_pci, but have it only
used by xl
>>>> 4. Add "seize" to libxl_device_pci, and have libxl do
the seizing.
>>>>
>>>> Any preference -- or any other ideas?
>>>>
>>>>    -George
>>> How about a setting in xl.conf of "auto-seize pci
devices" ?  That way
>>> the seizing is entirely part of xl
>> Auto-seizing is fairly dangerous; you could easily accidentally yank
>> out the ethernet card, or even the disk that dom0 is using.  I really
>> think it should have to be enabled on a device-by-device basis.
>>
>> I suppose another option would be to be able to set, in xl.conf, a
>> list of auto-seizeable devices.  I don''t really like that
option as
>> well, though.  I''d rather be able to keep all the
configuration in one
>> place.
>>
>>   -George
> Or a slight less extreme version.
>
> If xl sees that it would need seize a device, it could ask "You are
> trying to create a domain with device $FOO.  Would you like to seize it
> from dom0 ?"
That won''t work for driver domains, as we want it all to happen 
automatically when the host is booting. :-)

  -George

On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap
wrote:
> On 05/07/13 14:48, Andrew Cooper wrote:
> >On 05/07/13 14:45, George Dunlap wrote:
> >>On 05/07/13 14:39, Andrew Cooper wrote:
> >>>On 05/07/13 12:01, George Dunlap wrote:
> >>>>I''ve been doing some work to try to make driver
domains easier to set
> >>>>up and use.  At the moment, in order to pass a device
through to a
> >>>>guest, you first need to assign it to pciback.  This
involves doing
> >>>>one of three things:
> >>>>* Running xl pci-assignable-add for the device
> >>>>* Specifying the device to be grabbed on the dom0 Linux
command-line
> >>>>* Doing some hackery in /etc/modules.d
> >>>>
> >>>>None of these are very satisfying.  What I think would be
better is if
> >>>>there was a way to specify in the guest config file,
"If device X is
> >>>>not assignable, try to make it assignable".  That way
you can have a
> >>>>driver domain grab the appropriate device just by running
"xl create
> >>>>domnet"; and once we have the xendomains script up and
running with
> >>>>xl, you can simply configure your domnet appropriately, and
then put
> >>>>it in /etc/xen/auto, to be started automatically on boot.
> >>>>
> >>>>My initial idea was to add a parameter to the pci argument
in the
> >>>>config file; for example:
> >>>>
> >>>>pci = [''08:04.1,permissive=1,seize=1'']
> >>>>
> >>>>The ''seize=1'' would indicate that if bdf
08:04.1 is not already
> >>>>assignable, that xl should try to make is assignable.
> >>>>
> >>>>The problem here is that this would need to be parsed by
> >>>>xlu_pci_parse_bdf(), which only takes an argumen tof type
> >>>>libxl_device_pci.
> >>>>
> >>>>Now it seems to me that the right place to do this
"seizing" is in xl,
> >>>>not inside libxl -- the functions for doing assignment
exist already,
> >>>>and are simple and straightforward.  But doing it in xl,
but as a
> >>>>parameter of the "pci" setting, means changing
xlu_pci_parse_bdf() to
> >>>>pass something else back, which begins to get awkward.
> >>>>
> >>>>So it seems to me we have a couple of options:
> >>>>1. Create a new argument, "pci_seize" or
something like that, which
> >>>>would be processed separately from pci
> >>>>2. Change xlu_pci_parse_bdf to take a pointer to an extra
struct, for
> >>>>arguments directed at xl rather than libxl
> >>>>3. Add "seize" to libxl_device_pci, but have it
only used by xl
> >>>>4. Add "seize" to libxl_device_pci, and have
libxl do the seizing.
> >>>>
> >>>>Any preference -- or any other ideas?
> >>>>
> >>>>   -George
> >>>How about a setting in xl.conf of "auto-seize pci
devices" ?  That way
> >>>the seizing is entirely part of xl
> >>Auto-seizing is fairly dangerous; you could easily accidentally
yank
> >>out the ethernet card, or even the disk that dom0 is using.  I
really
> >>think it should have to be enabled on a device-by-device basis.
> >>
> >>I suppose another option would be to be able to set, in xl.conf, a
> >>list of auto-seizeable devices.  I don''t really like that
option as
> >>well, though.  I''d rather be able to keep all the
configuration in one
> >>place.
> >>
> >>  -George
> >Or a slight less extreme version.
> >
> >If xl sees that it would need seize a device, it could ask "You
are
> >trying to create a domain with device $FOO.  Would you like to seize it
> >from dom0 ?"
> 
> That won''t work for driver domains, as we want it all to happen
> automatically when the host is booting. :-)
The high-level goal is that we want to put the network devices with a
network backend and storage devices with storage backend. Ignorning
that for network devices you might want seperate backends for each
device (say one backend for Wireless, one for Ethernet, etc).

Perhaps the logic ought to do grouping - so you say:
 a) "backends:all-network" (which would created one backend with all
of the
   wireless, ethernet, etc PCI devices), or
 b) "backends:all-network,seperate-storage", which  create one backend
with
  all of the wireless, ethernet in one backend; and one backend domain for each
  storage device?

Naturally the user gets to chose which grouping they would
like?
> 
>  -George
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
>> On 05/07/13 14:48, Andrew Cooper wrote:
>>> On 05/07/13 14:45, George Dunlap wrote:
>>>> On 05/07/13 14:39, Andrew Cooper wrote:
>>>>> On 05/07/13 12:01, George Dunlap wrote:
>>>>>> I''ve been doing some work to try to make
driver domains easier to set
>>>>>> up and use.  At the moment, in order to pass a device
through to a
>>>>>> guest, you first need to assign it to pciback.  This
involves doing
>>>>>> one of three things:
>>>>>> * Running xl pci-assignable-add for the device
>>>>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>
>>>>>> None of these are very satisfying.  What I think would
be better is if
>>>>>> there was a way to specify in the guest config file,
"If device X is
>>>>>> not assignable, try to make it assignable".  That
way you can have a
>>>>>> driver domain grab the appropriate device just by
running "xl create
>>>>>> domnet"; and once we have the xendomains script up
and running with
>>>>>> xl, you can simply configure your domnet appropriately,
and then put
>>>>>> it in /etc/xen/auto, to be started automatically on
boot.
>>>>>>
>>>>>> My initial idea was to add a parameter to the pci
argument in the
>>>>>> config file; for example:
>>>>>>
>>>>>> pci =
[''08:04.1,permissive=1,seize=1'']
>>>>>>
>>>>>> The ''seize=1'' would indicate that if
bdf 08:04.1 is not already
>>>>>> assignable, that xl should try to make is assignable.
>>>>>>
>>>>>> The problem here is that this would need to be parsed
by
>>>>>> xlu_pci_parse_bdf(), which only takes an argumen tof
type
>>>>>> libxl_device_pci.
>>>>>>
>>>>>> Now it seems to me that the right place to do this
"seizing" is in xl,
>>>>>> not inside libxl -- the functions for doing assignment
exist already,
>>>>>> and are simple and straightforward.  But doing it in
xl, but as a
>>>>>> parameter of the "pci" setting, means
changing xlu_pci_parse_bdf() to
>>>>>> pass something else back, which begins to get awkward.
>>>>>>
>>>>>> So it seems to me we have a couple of options:
>>>>>> 1. Create a new argument, "pci_seize" or
something like that, which
>>>>>> would be processed separately from pci
>>>>>> 2. Change xlu_pci_parse_bdf to take a pointer to an
extra struct, for
>>>>>> arguments directed at xl rather than libxl
>>>>>> 3. Add "seize" to libxl_device_pci, but have
it only used by xl
>>>>>> 4. Add "seize" to libxl_device_pci, and have
libxl do the seizing.
>>>>>>
>>>>>> Any preference -- or any other ideas?
>>>>>>
>>>>>>    -George
>>>>> How about a setting in xl.conf of "auto-seize pci
devices" ?  That way
>>>>> the seizing is entirely part of xl
>>>> Auto-seizing is fairly dangerous; you could easily accidentally
yank
>>>> out the ethernet card, or even the disk that dom0 is using.  I
really
>>>> think it should have to be enabled on a device-by-device basis.
>>>>
>>>> I suppose another option would be to be able to set, in
xl.conf, a
>>>> list of auto-seizeable devices.  I don''t really like
that option as
>>>> well, though.  I''d rather be able to keep all the
configuration in one
>>>> place.
>>>>
>>>>   -George
>>> Or a slight less extreme version.
>>>
>>> If xl sees that it would need seize a device, it could ask
"You are
>>> trying to create a domain with device $FOO.  Would you like to
seize it
>> >from dom0 ?"
>>
>> That won''t work for driver domains, as we want it all to
happen
>> automatically when the host is booting. :-)
>
> The high-level goal is that we want to put the network devices with a
> network backend and storage devices with storage backend. Ignorning
> that for network devices you might want seperate backends for each
> device (say one backend for Wireless, one for Ethernet, etc).
>
> Perhaps the logic ought to do grouping - so you say:
>   a) "backends:all-network" (which would created one backend with
all of the
>     wireless, ethernet, etc PCI devices), or
>   b) "backends:all-network,seperate-storage", which  create one
backend with
>    all of the wireless, ethernet in one backend; and one backend domain for
each
>    storage device?
>
> Naturally the user gets to chose which grouping they would like?
We seem to be talking about different things.  You seem to be talking 
about automatically starting some pre-made VMs and assigning devices and 
backends to them?  But I''m not really sure.

I was assuming that the user was going to be installing and configuring 
their own driver domains.  The user already has to specify 
"pci=[''$BDF'']" in their config file to get specific
devices passed
through -- this would just be making it easy to have the device assigned 
to pciback as well.

I suspect that a lot of people will want to have one network card 
assigned to domain 0 as a "management network", and only have other 
devices assigned to driver domains.  I think that having one device per 
domain is probably the best recommendation; although we obviously want 
to support someone who wants a single "manage all the devices" domain,
we should assume that people are going to have one device per driver domain.

  -George

On Tue, Jul 09, 2013 at 01:52:38PM +0100, George Dunlap
wrote:
> On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> >On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
> >>On 05/07/13 14:48, Andrew Cooper wrote:
> >>>On 05/07/13 14:45, George Dunlap wrote:
> >>>>On 05/07/13 14:39, Andrew Cooper wrote:
> >>>>>On 05/07/13 12:01, George Dunlap wrote:
> >>>>>>I''ve been doing some work to try to make
driver domains easier to set
> >>>>>>up and use.  At the moment, in order to pass a
device through to a
> >>>>>>guest, you first need to assign it to pciback. 
This involves doing
> >>>>>>one of three things:
> >>>>>>* Running xl pci-assignable-add for the device
> >>>>>>* Specifying the device to be grabbed on the dom0
Linux command-line
> >>>>>>* Doing some hackery in /etc/modules.d
> >>>>>>
> >>>>>>None of these are very satisfying.  What I think
would be better is if
> >>>>>>there was a way to specify in the guest config
file, "If device X is
> >>>>>>not assignable, try to make it assignable". 
That way you can have a
> >>>>>>driver domain grab the appropriate device just by
running "xl create
> >>>>>>domnet"; and once we have the xendomains
script up and running with
> >>>>>>xl, you can simply configure your domnet
appropriately, and then put
> >>>>>>it in /etc/xen/auto, to be started automatically on
boot.
> >>>>>>
> >>>>>>My initial idea was to add a parameter to the pci
argument in the
> >>>>>>config file; for example:
> >>>>>>
> >>>>>>pci =
[''08:04.1,permissive=1,seize=1'']
> >>>>>>
> >>>>>>The ''seize=1'' would indicate that
if bdf 08:04.1 is not already
> >>>>>>assignable, that xl should try to make is
assignable.
> >>>>>>
> >>>>>>The problem here is that this would need to be
parsed by
> >>>>>>xlu_pci_parse_bdf(), which only takes an argumen
tof type
> >>>>>>libxl_device_pci.
> >>>>>>
> >>>>>>Now it seems to me that the right place to do this
"seizing" is in xl,
> >>>>>>not inside libxl -- the functions for doing
assignment exist already,
> >>>>>>and are simple and straightforward.  But doing it
in xl, but as a
> >>>>>>parameter of the "pci" setting, means
changing xlu_pci_parse_bdf() to
> >>>>>>pass something else back, which begins to get
awkward.
> >>>>>>
> >>>>>>So it seems to me we have a couple of options:
> >>>>>>1. Create a new argument, "pci_seize" or
something like that, which
> >>>>>>would be processed separately from pci
> >>>>>>2. Change xlu_pci_parse_bdf to take a pointer to an
extra struct, for
> >>>>>>arguments directed at xl rather than libxl
> >>>>>>3. Add "seize" to libxl_device_pci, but
have it only used by xl
> >>>>>>4. Add "seize" to libxl_device_pci, and
have libxl do the seizing.
> >>>>>>
> >>>>>>Any preference -- or any other ideas?
> >>>>>>
> >>>>>>   -George
> >>>>>How about a setting in xl.conf of "auto-seize pci
devices" ?  That way
> >>>>>the seizing is entirely part of xl
> >>>>Auto-seizing is fairly dangerous; you could easily
accidentally yank
> >>>>out the ethernet card, or even the disk that dom0 is using.
I really
> >>>>think it should have to be enabled on a device-by-device
basis.
> >>>>
> >>>>I suppose another option would be to be able to set, in
xl.conf, a
> >>>>list of auto-seizeable devices.  I don''t really
like that option as
> >>>>well, though.  I''d rather be able to keep all the
configuration in one
> >>>>place.
> >>>>
> >>>>  -George
> >>>Or a slight less extreme version.
> >>>
> >>>If xl sees that it would need seize a device, it could ask
"You are
> >>>trying to create a domain with device $FOO.  Would you like to
seize it
> >>>from dom0 ?"
> >>
> >>That won''t work for driver domains, as we want it all to
happen
> >>automatically when the host is booting. :-)
> >
> >The high-level goal is that we want to put the network devices with a
> >network backend and storage devices with storage backend. Ignorning
> >that for network devices you might want seperate backends for each
> >device (say one backend for Wireless, one for Ethernet, etc).
> >
> >Perhaps the logic ought to do grouping - so you say:
> >  a) "backends:all-network" (which would created one backend
with all of the
> >    wireless, ethernet, etc PCI devices), or
> >  b) "backends:all-network,seperate-storage", which  create
one backend with
> >   all of the wireless, ethernet in one backend; and one backend domain
for each
> >   storage device?
> >
> >Naturally the user gets to chose which grouping they would like?
> 
> We seem to be talking about different things.  You seem to be
> talking about automatically starting some pre-made VMs and assigning
> devices and backends to them?  But I''m not really sure.
I am trying to look at it from a high perspective to see whether we can
make this automated for 99% of people out of the box. Hence the
idea of grouping. And yes to ''..assigning devices and backends to
them''.
> 
> I was assuming that the user was going to be installing and
> configuring their own driver domains.  The user already has to
> specify "pci=[''$BDF'']" in their config file to
get specific devices
> passed through -- this would just be making it easy to have the
> device assigned to pciback as well.
I think the technical bits what libxl is doing and yanking devices
around is driven either by the admin or a policy. If the policy
is this idea of grouping (that is a terrible name now that I think
of it), then perhaps we should think how to make that work and then
the details (such as this automatic yanking of devices to pci-back)
can be filled in.

> 
> I suspect that a lot of people will want to have one network card
> assigned to domain 0 as a "management network", and only have
other
> devices assigned to driver domains.  I think that having one device
> per domain is probably the best recommendation; although we
> obviously want to support someone who wants a single "manage all the
> devices" domain, we should assume that people are going to have one
> device per driver domain.
I don''t know. My feeble idea was that we would have at minimum _two_
guests on bootup. One is a control one that has no devices - but is
the one that launches the guests.

Then there is the dom1 which would have all (or some) of the storage
and network devices plugged in along with the backends. Then a dom2
which would be the old-style-dom0 - so it would have the graphic card
and the rest of the PCI devices.

In other words, when I boot I would have two tiny domains launch
right before "old-style-dom0" is started. But I am getting in
specifics
here.

Perhaps you could explain to me how you envisioned how the device
driver domains idea would work? How would you want it to work on your
laptop?

Or are we right now just thinking of the small pieces of making the
code be able to yank the devices around and assign them?

Thanks.
> 
>  -George

On 07/09/2013 03:25 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Jul 09, 2013 at 01:52:38PM +0100, George Dunlap wrote:
>> On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
>>>> On 05/07/13 14:48, Andrew Cooper wrote:
>>>>> On 05/07/13 14:45, George Dunlap wrote:
>>>>>> On 05/07/13 14:39, Andrew Cooper wrote:
>>>>>>> On 05/07/13 12:01, George Dunlap wrote:
>>>>>>>> I''ve been doing some work to try to
make driver domains easier to set
>>>>>>>> up and use.  At the moment, in order to pass a
device through to a
>>>>>>>> guest, you first need to assign it to pciback. 
This involves doing
>>>>>>>> one of three things:
>>>>>>>> * Running xl pci-assignable-add for the device
>>>>>>>> * Specifying the device to be grabbed on the
dom0 Linux command-line
>>>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>>>
>>>>>>>> None of these are very satisfying.  What I
think would be better is if
>>>>>>>> there was a way to specify in the guest config
file, "If device X is
>>>>>>>> not assignable, try to make it
assignable".  That way you can have a
>>>>>>>> driver domain grab the appropriate device just
by running "xl create
>>>>>>>> domnet"; and once we have the xendomains
script up and running with
>>>>>>>> xl, you can simply configure your domnet
appropriately, and then put
>>>>>>>> it in /etc/xen/auto, to be started
automatically on boot.
>>>>>>>>
>>>>>>>> My initial idea was to add a parameter to the
pci argument in the
>>>>>>>> config file; for example:
>>>>>>>>
>>>>>>>> pci =
[''08:04.1,permissive=1,seize=1'']
>>>>>>>>
>>>>>>>> The ''seize=1'' would indicate
that if bdf 08:04.1 is not already
>>>>>>>> assignable, that xl should try to make is
assignable.
>>>>>>>>
>>>>>>>> The problem here is that this would need to be
parsed by
>>>>>>>> xlu_pci_parse_bdf(), which only takes an
argumen tof type
>>>>>>>> libxl_device_pci.
>>>>>>>>
>>>>>>>> Now it seems to me that the right place to do
this "seizing" is in xl,
>>>>>>>> not inside libxl -- the functions for doing
assignment exist already,
>>>>>>>> and are simple and straightforward.  But doing
it in xl, but as a
>>>>>>>> parameter of the "pci" setting, means
changing xlu_pci_parse_bdf() to
>>>>>>>> pass something else back, which begins to get
awkward.
>>>>>>>>
>>>>>>>> So it seems to me we have a couple of options:
>>>>>>>> 1. Create a new argument, "pci_seize"
or something like that, which
>>>>>>>> would be processed separately from pci
>>>>>>>> 2. Change xlu_pci_parse_bdf to take a pointer
to an extra struct, for
>>>>>>>> arguments directed at xl rather than libxl
>>>>>>>> 3. Add "seize" to libxl_device_pci,
but have it only used by xl
>>>>>>>> 4. Add "seize" to libxl_device_pci,
and have libxl do the seizing.
>>>>>>>>
>>>>>>>> Any preference -- or any other ideas?
>>>>>>>>
>>>>>>>>    -George
>>>>>>> How about a setting in xl.conf of "auto-seize
pci devices" ?  That way
>>>>>>> the seizing is entirely part of xl
>>>>>> Auto-seizing is fairly dangerous; you could easily
accidentally yank
>>>>>> out the ethernet card, or even the disk that dom0 is
using.  I really
>>>>>> think it should have to be enabled on a
device-by-device basis.
>>>>>>
>>>>>> I suppose another option would be to be able to set, in
xl.conf, a
>>>>>> list of auto-seizeable devices.  I don''t
really like that option as
>>>>>> well, though.  I''d rather be able to keep all
the configuration in one
>>>>>> place.
>>>>>>
>>>>>>   -George
>>>>> Or a slight less extreme version.
>>>>>
>>>>> If xl sees that it would need seize a device, it could ask
"You are
>>>>> trying to create a domain with device $FOO.  Would you like
to seize it
>>>> >from dom0 ?"
>>>>
>>>> That won''t work for driver domains, as we want it all
to happen
>>>> automatically when the host is booting. :-)
>>>
>>> The high-level goal is that we want to put the network devices with
a
>>> network backend and storage devices with storage backend. Ignorning
>>> that for network devices you might want seperate backends for each
>>> device (say one backend for Wireless, one for Ethernet, etc).
>>>
>>> Perhaps the logic ought to do grouping - so you say:
>>>   a) "backends:all-network" (which would created one
backend with all of the
>>>     wireless, ethernet, etc PCI devices), or
>>>   b) "backends:all-network,seperate-storage", which 
create one backend with
>>>    all of the wireless, ethernet in one backend; and one backend
domain for each
>>>    storage device?
>>>
>>> Naturally the user gets to chose which grouping they would like?
>>
>> We seem to be talking about different things.  You seem to be
>> talking about automatically starting some pre-made VMs and assigning
>> devices and backends to them?  But I''m not really sure.
>
> I am trying to look at it from a high perspective to see whether we can
> make this automated for 99% of people out of the box. Hence the
> idea of grouping. And yes to ''..assigning devices and backends to
them''.
>>
>> I was assuming that the user was going to be installing and
>> configuring their own driver domains.  The user already has to
>> specify "pci=[''$BDF'']" in their config file
to get specific devices
>> passed through -- this would just be making it easy to have the
>> device assigned to pciback as well.
>
> I think the technical bits what libxl is doing and yanking devices
> around is driven either by the admin or a policy. If the policy
> is this idea of grouping (that is a terrible name now that I think
> of it), then perhaps we should think how to make that work and then
> the details (such as this automatic yanking of devices to pci-back)
> can be filled in.
>
>
>>
>> I suspect that a lot of people will want to have one network card
>> assigned to domain 0 as a "management network", and only have
other
>> devices assigned to driver domains.  I think that having one device
>> per domain is probably the best recommendation; although we
>> obviously want to support someone who wants a single "manage all
the
>> devices" domain, we should assume that people are going to have
one
>> device per driver domain.
>
> I don''t know. My feeble idea was that we would have at minimum
_two_
> guests on bootup. One is a control one that has no devices - but is
> the one that launches the guests.
>
> Then there is the dom1 which would have all (or some) of the storage
> and network devices plugged in along with the backends. Then a dom2
> which would be the old-style-dom0 - so it would have the graphic card
> and the rest of the PCI devices.
>
> In other words, when I boot I would have two tiny domains launch
> right before "old-style-dom0" is started. But I am getting in
specifics
> here.
>
> Perhaps you could explain to me how you envisioned how the device
> driver domains idea would work? How would you want it to work on your
> laptop?
>
> Or are we right now just thinking of the small pieces of making the
> code be able to yank the devices around and assign them?
I was thinking for now just making the "manually configure it" case 
easier.  I decided to switch one of my test boxen to using a network 
driver domain by default, and although the core is there, there are a 
bunch of things that are unnecessarily crufty.

I do agree that long term it would be nice to make it easy to make 
driver domains the default, but that''s not what I had in mind for this 
conversation. :-)

The hard part for making it really automated, it seems to me, comes from 
two things.  O

One, you have to make sure your driver domain has the appropriate 
hardware drivers for your system as well.  We don''t want to be in the 
business of maintaining a distro; most people will probably want the 
driver domain to be from the same distro they''re using for dom0, which 
means that setting up such a domain will need to be done differently on 
a distro-by-distro basis.

Two, you have the configuration problem.  In Debian, for instance, if 
you wanted to switch a device from being owned by dom0 to being in a 
driver domain, you''d have to:
* Copy over the udev rules recognizing the mac address, so it got the 
same ethN
* copy over the eth and bridge info from dom0''s /etc/network/interfaces
into the guest /etc/network/interfaces

I''m not sure exactly what you have to do in Fedora, but I bet
it''s
something similar.

It might be nice to work with distros to make the process of making 
driver domains / stub domains easier, and to make it easy to configure 
driver domain networking options from the distro''s network scripts; but
that''s kind of another level of functionality.

I think first things first: make manually-set-up driver domains actually 
easy to use.

  -George

On Tue, 9 Jul 2013, George Dunlap wrote:
> On 07/09/2013 03:25 PM, Konrad Rzeszutek Wilk wrote:
> > On Tue, Jul 09, 2013 at 01:52:38PM +0100, George Dunlap wrote:
> > > On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap
wrote:
> > > > > On 05/07/13 14:48, Andrew Cooper wrote:
> > > > > > On 05/07/13 14:45, George Dunlap wrote:
> > > > > > > On 05/07/13 14:39, Andrew Cooper wrote:
> > > > > > > > On 05/07/13 12:01, George Dunlap wrote:
> > > > > > > > > I''ve been doing some work
to try to make driver domains easier
> > > > > > > > > to set
> > > > > > > > > up and use.  At the moment, in
order to pass a device through
> > > > > > > > > to a
> > > > > > > > > guest, you first need to assign it
to pciback.  This involves
> > > > > > > > > doing
> > > > > > > > > one of three things:
> > > > > > > > > * Running xl pci-assignable-add for
the device
> > > > > > > > > * Specifying the device to be
grabbed on the dom0 Linux
> > > > > > > > > command-line
> > > > > > > > > * Doing some hackery in
/etc/modules.d
> > > > > > > > > 
> > > > > > > > > None of these are very satisfying. 
What I think would be
> > > > > > > > > better is if
> > > > > > > > > there was a way to specify in the
guest config file, "If
> > > > > > > > > device X is
> > > > > > > > > not assignable, try to make it
assignable".  That way you can
> > > > > > > > > have a
> > > > > > > > > driver domain grab the appropriate
device just by running "xl
> > > > > > > > > create
> > > > > > > > > domnet"; and once we have the
xendomains script up and running
> > > > > > > > > with
> > > > > > > > > xl, you can simply configure your
domnet appropriately, and
> > > > > > > > > then put
> > > > > > > > > it in /etc/xen/auto, to be started
automatically on boot.
> > > > > > > > > 
> > > > > > > > > My initial idea was to add a
parameter to the pci argument in
> > > > > > > > > the
> > > > > > > > > config file; for example:
> > > > > > > > > 
> > > > > > > > > pci =
[''08:04.1,permissive=1,seize=1'']
> > > > > > > > > 
> > > > > > > > > The ''seize=1''
would indicate that if bdf 08:04.1 is not
> > > > > > > > > already
> > > > > > > > > assignable, that xl should try to
make is assignable.
> > > > > > > > > 
> > > > > > > > > The problem here is that this would
need to be parsed by
> > > > > > > > > xlu_pci_parse_bdf(), which only
takes an argumen tof type
> > > > > > > > > libxl_device_pci.
> > > > > > > > > 
> > > > > > > > > Now it seems to me that the right
place to do this "seizing"
> > > > > > > > > is in xl,
> > > > > > > > > not inside libxl -- the functions
for doing assignment exist
> > > > > > > > > already,
> > > > > > > > > and are simple and straightforward.
But doing it in xl, but
> > > > > > > > > as a
> > > > > > > > > parameter of the "pci"
setting, means changing
> > > > > > > > > xlu_pci_parse_bdf() to
> > > > > > > > > pass something else back, which
begins to get awkward.
> > > > > > > > > 
> > > > > > > > > So it seems to me we have a couple
of options:
> > > > > > > > > 1. Create a new argument,
"pci_seize" or something like that,
> > > > > > > > > which
> > > > > > > > > would be processed separately from
pci
> > > > > > > > > 2. Change xlu_pci_parse_bdf to take
a pointer to an extra
> > > > > > > > > struct, for
> > > > > > > > > arguments directed at xl rather
than libxl
> > > > > > > > > 3. Add "seize" to
libxl_device_pci, but have it only used by
> > > > > > > > > xl
> > > > > > > > > 4. Add "seize" to
libxl_device_pci, and have libxl do the
> > > > > > > > > seizing.
> > > > > > > > > 
> > > > > > > > > Any preference -- or any other
ideas?
> > > > > > > > > 
> > > > > > > > >    -George
> > > > > > > > How about a setting in xl.conf of
"auto-seize pci devices" ?
> > > > > > > > That way
> > > > > > > > the seizing is entirely part of xl
> > > > > > > Auto-seizing is fairly dangerous; you could
easily accidentally
> > > > > > > yank
> > > > > > > out the ethernet card, or even the disk that
dom0 is using.  I
> > > > > > > really
> > > > > > > think it should have to be enabled on a
device-by-device basis.
> > > > > > > 
> > > > > > > I suppose another option would be to be able
to set, in xl.conf, a
> > > > > > > list of auto-seizeable devices.  I
don''t really like that option
> > > > > > > as
> > > > > > > well, though.  I''d rather be able to
keep all the configuration in
> > > > > > > one
> > > > > > > place.
> > > > > > > 
> > > > > > >   -George
> > > > > > Or a slight less extreme version.
> > > > > > 
> > > > > > If xl sees that it would need seize a device, it
could ask "You are
> > > > > > trying to create a domain with device $FOO.  Would
you like to seize
> > > > > > it
> > > > > >from dom0 ?"
> > > > > 
> > > > > That won''t work for driver domains, as we want
it all to happen
> > > > > automatically when the host is booting. :-)
> > > > 
> > > > The high-level goal is that we want to put the network
devices with a
> > > > network backend and storage devices with storage backend.
Ignorning
> > > > that for network devices you might want seperate backends
for each
> > > > device (say one backend for Wireless, one for Ethernet,
etc).
> > > > 
> > > > Perhaps the logic ought to do grouping - so you say:
> > > >   a) "backends:all-network" (which would created
one backend with all of
> > > > the
> > > >     wireless, ethernet, etc PCI devices), or
> > > >   b) "backends:all-network,seperate-storage",
which  create one backend
> > > > with
> > > >    all of the wireless, ethernet in one backend; and one
backend domain
> > > > for each
> > > >    storage device?
> > > > 
> > > > Naturally the user gets to chose which grouping they would
like?
> > > 
> > > We seem to be talking about different things.  You seem to be
> > > talking about automatically starting some pre-made VMs and
assigning
> > > devices and backends to them?  But I''m not really sure.
> > 
> > I am trying to look at it from a high perspective to see whether we
can
> > make this automated for 99% of people out of the box. Hence the
> > idea of grouping. And yes to ''..assigning devices and
backends to them''.
> > > 
> > > I was assuming that the user was going to be installing and
> > > configuring their own driver domains.  The user already has to
> > > specify "pci=[''$BDF'']" in their
config file to get specific devices
> > > passed through -- this would just be making it easy to have the
> > > device assigned to pciback as well.
> > 
> > I think the technical bits what libxl is doing and yanking devices
> > around is driven either by the admin or a policy. If the policy
> > is this idea of grouping (that is a terrible name now that I think
> > of it), then perhaps we should think how to make that work and then
> > the details (such as this automatic yanking of devices to pci-back)
> > can be filled in.
> > 
> > 
> > > 
> > > I suspect that a lot of people will want to have one network card
> > > assigned to domain 0 as a "management network", and
only have other
> > > devices assigned to driver domains.  I think that having one
device
> > > per domain is probably the best recommendation; although we
> > > obviously want to support someone who wants a single "manage
all the
> > > devices" domain, we should assume that people are going to
have one
> > > device per driver domain.
> > 
> > I don''t know. My feeble idea was that we would have at
minimum _two_
> > guests on bootup. One is a control one that has no devices - but is
> > the one that launches the guests.
> > 
> > Then there is the dom1 which would have all (or some) of the storage
> > and network devices plugged in along with the backends. Then a dom2
> > which would be the old-style-dom0 - so it would have the graphic card
> > and the rest of the PCI devices.
> > 
> > In other words, when I boot I would have two tiny domains launch
> > right before "old-style-dom0" is started. But I am getting
in specifics
> > here.
> > 
> > Perhaps you could explain to me how you envisioned how the device
> > driver domains idea would work? How would you want it to work on your
> > laptop?
> > 
> > Or are we right now just thinking of the small pieces of making the
> > code be able to yank the devices around and assign them?
> 
> I was thinking for now just making the "manually configure it"
case easier.  I
> decided to switch one of my test boxen to using a network driver domain by
> default, and although the core is there, there are a bunch of things that
are
> unnecessarily crufty.
> 
> I do agree that long term it would be nice to make it easy to make driver
> domains the default, but that''s not what I had in mind for this
conversation.
> :-)
> 
> The hard part for making it really automated, it seems to me, comes from
two
> things.  O
> 
> One, you have to make sure your driver domain has the appropriate hardware
> drivers for your system as well.  We don''t want to be in the
business of
> maintaining a distro; most people will probably want the driver domain to
be
> from the same distro they''re using for dom0, which means that
setting up such
> a domain will need to be done differently on a distro-by-distro basis.
Can we just use the same kernel used in dom0 as a starting point? That
way we would be sure that has everything we need.

> Two, you have the configuration problem.  In Debian, for instance, if you
> wanted to switch a device from being owned by dom0 to being in a driver
> domain, you''d have to:
> * Copy over the udev rules recognizing the mac address, so it got the same
> ethN
> * copy over the eth and bridge info from dom0''s
/etc/network/interfaces into
> the guest /etc/network/interfaces
> 
> I''m not sure exactly what you have to do in Fedora, but I bet
it''s something
> similar.
> 
> It might be nice to work with distros to make the process of making driver
> domains / stub domains easier, and to make it easy to configure driver
domain
> networking options from the distro''s network scripts; but
that''s kind of
> another level of functionality.
We could host some example per-distro scripts on the xen-unstable
repository to show distros how to setup driver domains by default.  Then
we can go to Fedora, Ubuntu, etc and tell them: "look, if you start this
at boot you get driver domains!".

> I think first things first: make manually-set-up driver domains actually
easy
> to use.
Right

On Tue, 9 Jul 2013, Konrad Rzeszutek Wilk wrote:
> > I suspect that a lot of people will want to have one network card
> > assigned to domain 0 as a "management network", and only
have other
> > devices assigned to driver domains.  I think that having one device
> > per domain is probably the best recommendation; although we
> > obviously want to support someone who wants a single "manage all
the
> > devices" domain, we should assume that people are going to have
one
> > device per driver domain.
> 
> I don''t know. My feeble idea was that we would have at minimum
_two_
> guests on bootup. One is a control one that has no devices - but is
> the one that launches the guests.
So this would be "dom0".

> Then there is the dom1 which would have all (or some) of the storage
> and network devices plugged in along with the backends. Then a dom2
> which would be the old-style-dom0 - so it would have the graphic card
> and the rest of the PCI devices.
If "dom2" has the graphic card, then it''s the domain the user
is
interacting with. Therefore the user would be starting domains by using
a tool in "dom2" then it would need to communicate with
"dom0" to do
the actual domain creation.

I am not arguing against this solution, but this sounds very much like a
"stage 2" thing to have.
I would keep "dom0" and "dom2" collapsed in the same domain
for the
moment.

George Dunlap writes ("RFC: Automatically making a PCI device assignable in
the config file"):
> I''ve been doing some work to try to make driver domains easier to
set
> up and use.  At the moment, in order to pass a device through to a
> guest, you first need to assign it to pciback.  This involves doing
> one of three things:
> * Running xl pci-assignable-add for the device
> * Specifying the device to be grabbed on the dom0 Linux command-line
> * Doing some hackery in /etc/modules.d
>
> None of these are very satisfying.  What I think would be better is if
> there was a way to specify in the guest config file, "If device X is
> not assignable, try to make it assignable".  That way you can have a
> driver domain grab the appropriate device just by running "xl create
> domnet"; and once we have the xendomains script up and running with
> xl, you can simply configure your domnet appropriately, and then put
> it in /etc/xen/auto, to be started automatically on boot.
> 
> My initial idea was to add a parameter to the pci argument in the
> config file; for example:
> 
> pci = [''08:04.1,permissive=1,seize=1'']
> 
> The ''seize=1'' would indicate that if bdf 08:04.1 is not
already
> assignable, that xl should try to make is assignable.
I think it''s a design error that this isn''t done automatically
by
default.

It would be nice if there was a safety check that the device isn''t
currently in use by dom0, but I don''t think it''s essential. 
We could
just put a note in the docs saying "if you specify your dom0 nic it
will go away, duh" or something.

And I think this should be done by libxl, under the control of the
usual kinds of xl configuration.

Ian.

George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI
device assignable in the config file"):
> Auto-seizing is fairly dangerous; you could easily accidentally yank out 
> the ethernet card, or even the disk that dom0 is using.  I really think 
> it should have to be enabled on a device-by-device basis.
I don''t think this makes any sense.

In practice you are saying that in order to avoid mistakes, the local
admin must provide the BDF of the device to be passed through in two
separate places.

But that doesn''t actually help.  If this is all properly documented
and so forth (ie, if the admin has a smooth experience and doesn''t
trip over having dailed to "seize" the device), they will just
cut-and-paste the same value into both places in the config.

They will also mutter under their breath to ask why this is
necessary...

Ian.

On 10/07/13 14:55, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI
device assignable in the config file"):
>> Auto-seizing is fairly dangerous; you could easily accidentally yank
out
>> the ethernet card, or even the disk that dom0 is using.  I really think
>> it should have to be enabled on a device-by-device basis.
> I don''t think this makes any sense.
>
> In practice you are saying that in order to avoid mistakes, the local
> admin must provide the BDF of the device to be passed through in two
> separate places.
That''s not what I had in mind; what I had in mind was something like
this:

pci = [ ''08:04.1,seize=1'' ]

Or alternately:

xl pci-attach h0 08:04.1,seize=1

One could also imagine having something in xl.conf like the following:

pci.autoseize = [ ''08:04.1'',''01:00.0'' ]

In this case you wouldn''t be simply copy and pasting, because
you''d
probably have different domains handling each device.  But in any case, 
this was just exploring the alternatives -- I don''t think
that''s the
best thing to do.
> But that doesn''t actually help.  If this is all properly
documented
> and so forth (ie, if the admin has a smooth experience and doesn''t
> trip over having dailed to "seize" the device), they will just
> cut-and-paste the same value into both places in the config.
>
> They will also mutter under their breath to ask why this is
> necessary...
If someone can accidentally type "xl pci-attach 08:04.0" instead of
"xl
pci-attach 08.04.1" and suddenly completely lose their network 
connectivity (or yank their hard drive out), then I think they''ll 
appreciate it.

In any case, at the moment it''s much worse: you have to either 
scriptwise run "xl pci-assignable-add" a device, or add an exclusion
on
the Linux commandline in grub.  So far I''m the only person I know who 
has complained about it. :-)

  -George

On 10/07/13 14:53, Ian Jackson wrote:
> George Dunlap writes ("RFC: Automatically making a PCI device
assignable in the config file"):
>> I''ve been doing some work to try to make driver domains easier
to set
>> up and use.  At the moment, in order to pass a device through to a
>> guest, you first need to assign it to pciback.  This involves doing
>> one of three things:
>> * Running xl pci-assignable-add for the device
>> * Specifying the device to be grabbed on the dom0 Linux command-line
>> * Doing some hackery in /etc/modules.d
>>
>> None of these are very satisfying.  What I think would be better is if
>> there was a way to specify in the guest config file, "If device X
is
>> not assignable, try to make it assignable".  That way you can have
a
>> driver domain grab the appropriate device just by running "xl
create
>> domnet"; and once we have the xendomains script up and running
with
>> xl, you can simply configure your domnet appropriately, and then put
>> it in /etc/xen/auto, to be started automatically on boot.
>>
>> My initial idea was to add a parameter to the pci argument in the
>> config file; for example:
>>
>> pci = [''08:04.1,permissive=1,seize=1'']
>>
>> The ''seize=1'' would indicate that if bdf 08:04.1 is
not already
>> assignable, that xl should try to make is assignable.
> I think it''s a design error that this isn''t done
automatically by
> default.
>
> It would be nice if there was a safety check that the device isn''t
> currently in use by dom0, but I don''t think it''s
essential.  We could
> just put a note in the docs saying "if you specify your dom0 nic it
> will go away, duh" or something.
I think it''s a really bad interface decision if a simple typo might 
result in you yanking out your disk.

But in any case, if you think libxl_pci_attach() should automatically 
grab the device, that makes things a lot easier.  I''ll write up a patch
to add a "seize" flag and that will be that.

  -George

On Wed, 10 Jul 2013 15:45:49 +0100, George Dunlap 
 <george.dunlap@eu.citrix.com> wrote:
> On 10/07/13 14:55, Ian Jackson wrote:
>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a
>> PCI device assignable in the config file"):
>>> Auto-seizing is fairly dangerous; you could easily accidentally 
>>> yank out
>>> the ethernet card, or even the disk that dom0 is using.  I really 
>>> think
>>> it should have to be enabled on a device-by-device basis.
>> I don''t think this makes any sense.
>>
>> In practice you are saying that in order to avoid mistakes, the 
>> local
>> admin must provide the BDF of the device to be passed through in two
>> separate places.
>
> That''s not what I had in mind; what I had in mind was something
like
> this:
>
> pci = [ ''08:04.1,seize=1'' ]
>
> Or alternately:
>
> xl pci-attach h0 08:04.1,seize=1
>
> One could also imagine having something in xl.conf like the 
> following:
>
> pci.autoseize = [ ''08:04.1'',''01:00.0'' ]
>
> In this case you wouldn''t be simply copy and pasting, because
you''d
> probably have different domains handling each device.  But in any
> case, this was just exploring the alternatives -- I don''t think 
> that''s
> the best thing to do.
>
>> But that doesn''t actually help.  If this is all properly
documented
>> and so forth (ie, if the admin has a smooth experience and
doesn''t
>> trip over having dailed to "seize" the device), they will
just
>> cut-and-paste the same value into both places in the config.
>>
>> They will also mutter under their breath to ask why this is
>> necessary...
>
> If someone can accidentally type "xl pci-attach 08:04.0" instead
of
> "xl pci-attach 08.04.1" and suddenly completely lose their
network
> connectivity (or yank their hard drive out), then I think they''ll
> appreciate it.
>
> In any case, at the moment it''s much worse: you have to either
> scriptwise run "xl pci-assignable-add" a device, or add an
exclusion
> on the Linux commandline in grub.  So far I''m the only person I
know
> who has complained about it. :-)
 This would be an awesome feature, but success can be mixed. My
 experience is that Nvidia drivers (as one example, I''m sure there
 are others) don''t handle device detaching very gracefully. ACS
 might help but I have no means of testing
 that at the moment. So ultimately, you still have to either
 exclude at least some devices from grub (if
 xen-pciback is built in) or in modprobe.d (attach device to
 pciback in driver pre-install). I''m not sure what can be
 done about this considering we are talking about 3rd
 party binary drivers. :(

 Gordan

On 10/07/13 16:12, Gordan Bobic wrote:
> On Wed, 10 Jul 2013 15:45:49 +0100, George Dunlap 
> <george.dunlap@eu.citrix.com> wrote:
>> On 10/07/13 14:55, Ian Jackson wrote:
>>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically
making a
>>> PCI device assignable in the config file"):
>>>> Auto-seizing is fairly dangerous; you could easily accidentally
>>>> yank out
>>>> the ethernet card, or even the disk that dom0 is using.  I
really
>>>> think
>>>> it should have to be enabled on a device-by-device basis.
>>> I don''t think this makes any sense.
>>>
>>> In practice you are saying that in order to avoid mistakes, the
local
>>> admin must provide the BDF of the device to be passed through in
two
>>> separate places.
>>
>> That''s not what I had in mind; what I had in mind was
something like
>> this:
>>
>> pci = [ ''08:04.1,seize=1'' ]
>>
>> Or alternately:
>>
>> xl pci-attach h0 08:04.1,seize=1
>>
>> One could also imagine having something in xl.conf like the following:
>>
>> pci.autoseize = [
''08:04.1'',''01:00.0'' ]
>>
>> In this case you wouldn''t be simply copy and pasting, because
you''d
>> probably have different domains handling each device.  But in any
>> case, this was just exploring the alternatives -- I don''t
think that''s
>> the best thing to do.
>>
>>> But that doesn''t actually help.  If this is all properly
documented
>>> and so forth (ie, if the admin has a smooth experience and
doesn''t
>>> trip over having dailed to "seize" the device), they will
just
>>> cut-and-paste the same value into both places in the config.
>>>
>>> They will also mutter under their breath to ask why this is
>>> necessary...
>>
>> If someone can accidentally type "xl pci-attach 08:04.0"
instead of
>> "xl pci-attach 08.04.1" and suddenly completely lose their
network
>> connectivity (or yank their hard drive out), then I think
they''ll
>> appreciate it.
>>
>> In any case, at the moment it''s much worse: you have to either
>> scriptwise run "xl pci-assignable-add" a device, or add an
exclusion
>> on the Linux commandline in grub.  So far I''m the only person
I know
>> who has complained about it. :-)
>
> This would be an awesome feature, but success can be mixed. My
> experience is that Nvidia drivers (as one example, I''m sure there
> are others) don''t handle device detaching very gracefully. ACS
> might help but I have no means of testing
> that at the moment. So ultimately, you still have to either
> exclude at least some devices from grub (if
> xen-pciback is built in) or in modprobe.d (attach device to
> pciback in driver pre-install). I''m not sure what can be
> done about this considering we are talking about 3rd
> party binary drivers. :(
Yes, unfortunately we''ll always have to have the fallback for devices 
that don''t play well with being re-assigned.  But I my very limited 
experience with network cards gives me some hope that at least for many 
devices for which driver domains might be in use, the auto-reassign 
thing will be useful in a number of cases.

  -George

On Wed, 10 Jul 2013 16:29:33 +0100, George Dunlap 
 <george.dunlap@eu.citrix.com> wrote:
> On 10/07/13 16:12, Gordan Bobic wrote:
>> On Wed, 10 Jul 2013 15:45:49 +0100, George Dunlap 
>> <george.dunlap@eu.citrix.com> wrote:
>>> On 10/07/13 14:55, Ian Jackson wrote:
>>>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically
making a
>>>> PCI device assignable in the config file"):
>>>>> Auto-seizing is fairly dangerous; you could easily
accidentally
>>>>> yank out
>>>>> the ethernet card, or even the disk that dom0 is using.  I
really
>>>>> think
>>>>> it should have to be enabled on a device-by-device basis.
>>>> I don''t think this makes any sense.
>>>>
>>>> In practice you are saying that in order to avoid mistakes, the
>>>> local
>>>> admin must provide the BDF of the device to be passed through
in
>>>> two
>>>> separate places.
>>>
>>> That''s not what I had in mind; what I had in mind was
something
>>> like this:
>>>
>>> pci = [ ''08:04.1,seize=1'' ]
>>>
>>> Or alternately:
>>>
>>> xl pci-attach h0 08:04.1,seize=1
>>>
>>> One could also imagine having something in xl.conf like the 
>>> following:
>>>
>>> pci.autoseize = [
''08:04.1'',''01:00.0'' ]
>>>
>>> In this case you wouldn''t be simply copy and pasting,
because you''d
>>> probably have different domains handling each device.  But in any
>>> case, this was just exploring the alternatives -- I don''t
think
>>> that''s
>>> the best thing to do.
>>>
>>>> But that doesn''t actually help.  If this is all
properly
>>>> documented
>>>> and so forth (ie, if the admin has a smooth experience and
doesn''t
>>>> trip over having dailed to "seize" the device), they
will just
>>>> cut-and-paste the same value into both places in the config.
>>>>
>>>> They will also mutter under their breath to ask why this is
>>>> necessary...
>>>
>>> If someone can accidentally type "xl pci-attach 08:04.0"
instead of
>>> "xl pci-attach 08.04.1" and suddenly completely lose
their network
>>> connectivity (or yank their hard drive out), then I think
they''ll
>>> appreciate it.
>>>
>>> In any case, at the moment it''s much worse: you have to
either
>>> scriptwise run "xl pci-assignable-add" a device, or add
an
>>> exclusion
>>> on the Linux commandline in grub.  So far I''m the only
person I
>>> know
>>> who has complained about it. :-)
>>
>> This would be an awesome feature, but success can be mixed. My
>> experience is that Nvidia drivers (as one example, I''m sure
there
>> are others) don''t handle device detaching very gracefully. ACS
>> might help but I have no means of testing
>> that at the moment. So ultimately, you still have to either
>> exclude at least some devices from grub (if
>> xen-pciback is built in) or in modprobe.d (attach device to
>> pciback in driver pre-install). I''m not sure what can be
>> done about this considering we are talking about 3rd
>> party binary drivers. :(
>
> Yes, unfortunately we''ll always have to have the fallback for
devices
> that don''t play well with being re-assigned.  But I my very
limited
> experience with network cards gives me some hope that at least for
> many devices for which driver domains might be in use, the
> auto-reassign thing will be useful in a number of cases.
 What about having a feature that upon parsing the config file modifies
 the grub.conf or modprobe.d/xen-pciback.conf (depending on whether
 xen-pciback is module or built in) and adds the required entries
 for the device(s)?

 Gordan

On 10/07/13 15:48, George Dunlap wrote:
> On 10/07/13 14:53, Ian Jackson wrote:
>> George Dunlap writes ("RFC: Automatically making a PCI device
>> assignable in the config file"):
>>> I''ve been doing some work to try to make driver domains
easier to set
>>> up and use.  At the moment, in order to pass a device through to a
>>> guest, you first need to assign it to pciback.  This involves doing
>>> one of three things:
>>> * Running xl pci-assignable-add for the device
>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>> * Doing some hackery in /etc/modules.d
>>>
>>> None of these are very satisfying.  What I think would be better is
if
>>> there was a way to specify in the guest config file, "If
device X is
>>> not assignable, try to make it assignable".  That way you can
have a
>>> driver domain grab the appropriate device just by running "xl
create
>>> domnet"; and once we have the xendomains script up and running
with
>>> xl, you can simply configure your domnet appropriately, and then
put
>>> it in /etc/xen/auto, to be started automatically on boot.
>>>
>>> My initial idea was to add a parameter to the pci argument in the
>>> config file; for example:
>>>
>>> pci = [''08:04.1,permissive=1,seize=1'']
>>>
>>> The ''seize=1'' would indicate that if bdf 08:04.1
is not already
>>> assignable, that xl should try to make is assignable.
>> I think it''s a design error that this isn''t done
automatically by
>> default.
>>
>> It would be nice if there was a safety check that the device
isn''t
>> currently in use by dom0, but I don''t think it''s
essential.  We could
>> just put a note in the docs saying "if you specify your dom0 nic
it
>> will go away, duh" or something.
> 
> I think it''s a really bad interface decision if a simple typo
might
> result in you yanking out your disk.
I don''t think this proposal really helps with avoiding this.  I think
most people will end up always adding ''seize=1'' because to
avoid having
to do so means altering config files elsewhere and rebooting.

David

On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com>
wrote:
> On 10/07/13 15:48, George Dunlap wrote:
>> On 10/07/13 14:53, Ian Jackson wrote:
>>> George Dunlap writes ("RFC: Automatically making a PCI device
>>> assignable in the config file"):
>>>> I''ve been doing some work to try to make driver
domains easier to set
>>>> up and use.  At the moment, in order to pass a device through
to a
>>>> guest, you first need to assign it to pciback.  This involves
doing
>>>> one of three things:
>>>> * Running xl pci-assignable-add for the device
>>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>>> * Doing some hackery in /etc/modules.d
>>>>
>>>> None of these are very satisfying.  What I think would be
better is if
>>>> there was a way to specify in the guest config file, "If
device X is
>>>> not assignable, try to make it assignable".  That way you
can have a
>>>> driver domain grab the appropriate device just by running
"xl create
>>>> domnet"; and once we have the xendomains script up and
running with
>>>> xl, you can simply configure your domnet appropriately, and
then put
>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>
>>>> My initial idea was to add a parameter to the pci argument in
the
>>>> config file; for example:
>>>>
>>>> pci = [''08:04.1,permissive=1,seize=1'']
>>>>
>>>> The ''seize=1'' would indicate that if bdf
08:04.1 is not already
>>>> assignable, that xl should try to make is assignable.
>>> I think it''s a design error that this isn''t done
automatically by
>>> default.
>>>
>>> It would be nice if there was a safety check that the device
isn''t
>>> currently in use by dom0, but I don''t think it''s
essential.  We could
>>> just put a note in the docs saying "if you specify your dom0
nic it
>>> will go away, duh" or something.
>>
>> I think it''s a really bad interface decision if a simple typo
might
>> result in you yanking out your disk.
>
> I don''t think this proposal really helps with avoiding this.  I
think
> most people will end up always adding ''seize=1'' because
to avoid having
> to do so means altering config files elsewhere and rebooting.
I guess what I''m worried about is the fact that we would be changing
things that are now "safe" to things that are not safe.  At the
moment, "xl pci-assignable-add" might yank out a system device if you
make a typo; but it was introduced that way, so people always had to
be careful.  But currently, "pci=[]" and "xl pci-attach" do
*not*
behave that way; you have to make the device assignable first.  So you
don''t need to be particularly careful.  Adding "seize" at
least should
flag up to people that they need to double-check.

If people really object to the extra flag, I can write up a patch
without it, but I''d prefer to have something...

 -George

On 12/07/13 10:36, George Dunlap wrote:
> On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel
<david.vrabel@citrix.com> wrote:
>> On 10/07/13 15:48, George Dunlap wrote:
>>> On 10/07/13 14:53, Ian Jackson wrote:
>>>> George Dunlap writes ("RFC: Automatically making a PCI
device
>>>> assignable in the config file"):
>>>>> I''ve been doing some work to try to make driver
domains easier to set
>>>>> up and use.  At the moment, in order to pass a device
through to a
>>>>> guest, you first need to assign it to pciback.  This
involves doing
>>>>> one of three things:
>>>>> * Running xl pci-assignable-add for the device
>>>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>>>> * Doing some hackery in /etc/modules.d
>>>>>
>>>>> None of these are very satisfying.  What I think would be
better is if
>>>>> there was a way to specify in the guest config file,
"If device X is
>>>>> not assignable, try to make it assignable".  That way
you can have a
>>>>> driver domain grab the appropriate device just by running
"xl create
>>>>> domnet"; and once we have the xendomains script up and
running with
>>>>> xl, you can simply configure your domnet appropriately, and
then put
>>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>>
>>>>> My initial idea was to add a parameter to the pci argument
in the
>>>>> config file; for example:
>>>>>
>>>>> pci = [''08:04.1,permissive=1,seize=1'']
>>>>>
>>>>> The ''seize=1'' would indicate that if bdf
08:04.1 is not already
>>>>> assignable, that xl should try to make is assignable.
>>>> I think it''s a design error that this isn''t
done automatically by
>>>> default.
>>>>
>>>> It would be nice if there was a safety check that the device
isn''t
>>>> currently in use by dom0, but I don''t think
it''s essential.  We could
>>>> just put a note in the docs saying "if you specify your
dom0 nic it
>>>> will go away, duh" or something.
>>>
>>> I think it''s a really bad interface decision if a simple
typo might
>>> result in you yanking out your disk.
>>
>> I don''t think this proposal really helps with avoiding this. 
I think
>> most people will end up always adding ''seize=1''
because to avoid having
>> to do so means altering config files elsewhere and rebooting.
> 
> I guess what I''m worried about is the fact that we would be
changing
> things that are now "safe" to things that are not safe.  At the
> moment, "xl pci-assignable-add" might yank out a system device if
you
> make a typo; but it was introduced that way, so people always had to
> be careful.  But currently, "pci=[]" and "xl
pci-attach" do *not*
> behave that way; you have to make the device assignable first.  So you
> don''t need to be particularly careful.  Adding "seize"
at least should
> flag up to people that they need to double-check.
You could see if you can determine if a PCI device provides either a
block device or a network device and then check whether the block device
is mounted or the network device has a IP address configured.

These checks should probably be handled by a script that xl calls.

David

On 12/07/13 10:55, David Vrabel wrote:
> On 12/07/13 10:36, George Dunlap wrote:
>> On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel
<david.vrabel@citrix.com> wrote:
>>> On 10/07/13 15:48, George Dunlap wrote:
>>>> On 10/07/13 14:53, Ian Jackson wrote:
>>>>> George Dunlap writes ("RFC: Automatically making a PCI
device
>>>>> assignable in the config file"):
>>>>>> I''ve been doing some work to try to make
driver domains easier to set
>>>>>> up and use.  At the moment, in order to pass a device
through to a
>>>>>> guest, you first need to assign it to pciback.  This
involves doing
>>>>>> one of three things:
>>>>>> * Running xl pci-assignable-add for the device
>>>>>> * Specifying the device to be grabbed on the dom0 Linux
command-line
>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>
>>>>>> None of these are very satisfying.  What I think would
be better is if
>>>>>> there was a way to specify in the guest config file,
"If device X is
>>>>>> not assignable, try to make it assignable".  That
way you can have a
>>>>>> driver domain grab the appropriate device just by
running "xl create
>>>>>> domnet"; and once we have the xendomains script up
and running with
>>>>>> xl, you can simply configure your domnet appropriately,
and then put
>>>>>> it in /etc/xen/auto, to be started automatically on
boot.
>>>>>>
>>>>>> My initial idea was to add a parameter to the pci
argument in the
>>>>>> config file; for example:
>>>>>>
>>>>>> pci =
[''08:04.1,permissive=1,seize=1'']
>>>>>>
>>>>>> The ''seize=1'' would indicate that if
bdf 08:04.1 is not already
>>>>>> assignable, that xl should try to make is assignable.
>>>>> I think it''s a design error that this
isn''t done automatically by
>>>>> default.
>>>>>
>>>>> It would be nice if there was a safety check that the
device isn''t
>>>>> currently in use by dom0, but I don''t think
it''s essential.  We could
>>>>> just put a note in the docs saying "if you specify
your dom0 nic it
>>>>> will go away, duh" or something.
>>>> I think it''s a really bad interface decision if a
simple typo might
>>>> result in you yanking out your disk.
>>> I don''t think this proposal really helps with avoiding
this.  I think
>>> most people will end up always adding ''seize=1''
because to avoid having
>>> to do so means altering config files elsewhere and rebooting.
>> I guess what I''m worried about is the fact that we would be
changing
>> things that are now "safe" to things that are not safe.  At
the
>> moment, "xl pci-assignable-add" might yank out a system
device if you
>> make a typo; but it was introduced that way, so people always had to
>> be careful.  But currently, "pci=[]" and "xl
pci-attach" do *not*
>> behave that way; you have to make the device assignable first.  So you
>> don''t need to be particularly careful.  Adding
"seize" at least should
>> flag up to people that they need to double-check.
> You could see if you can determine if a PCI device provides either a
> block device or a network device and then check whether the block device
> is mounted or the network device has a IP address configured.
>
> These checks should probably be handled by a script that xl calls.
I think we''d be opening a can of worms.  For example, if you do
ifconfig
on an eth device that''s being used by a bridge, it comes up empty; so 
the script would have to be smart enough to find the associated bridge 
as well.  That seems to be the case both for Linux bridging and 
openvswitch.  This might become arbitrarily complicated for various 
devices: what about a USB root switch with an external hub that has a 
mounted disk?

I think the only maintainable option is to have it just done with no 
checks; the only question is if we should require an option to enable 
the auto-assigning or having it happen by default.

  -George

George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI
device assignable in the config file"):
> On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel
<david.vrabel@citrix.com> wrot> > I don''t think this
proposal really helps with avoiding this.  I think
> > most people will end up always adding ''seize=1''
because to avoid having
> > to do so means altering config files elsewhere and rebooting.
> 
> I guess what I''m worried about is the fact that we would be
changing
> things that are now "safe" to things that are not safe.  At the
> moment, "xl pci-assignable-add" might yank out a system device if
you
> make a typo; but it was introduced that way, so people always had to
> be careful.  But currently, "pci=[]" and "xl
pci-attach" do *not*
> behave that way; you have to make the device assignable first.  So you
> don''t need to be particularly careful.  Adding "seize"
at least should
> flag up to people that they need to double-check.
> 
> If people really object to the extra flag, I can write up a patch
> without it, but I''d prefer to have something...
I think at the very least there should be the ability to have it as a
global config option.

Ian.

On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson
wrote:
> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI
device assignable in the config file"):
> > On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel
<david.vrabel@citrix.com> wrot> > I don''t think this
proposal really helps with avoiding this.  I think
> > > most people will end up always adding ''seize=1''
because to avoid having
> > > to do so means altering config files elsewhere and rebooting.
> > 
> > I guess what I''m worried about is the fact that we would be
changing
> > things that are now "safe" to things that are not safe.  At
the
> > moment, "xl pci-assignable-add" might yank out a system
device if you
> > make a typo; but it was introduced that way, so people always had to
> > be careful.  But currently, "pci=[]" and "xl
pci-attach" do *not*
> > behave that way; you have to make the device assignable first.  So you
> > don''t need to be particularly careful.  Adding
"seize" at least should
> > flag up to people that they need to double-check.
> > 
> > If people really object to the extra flag, I can write up a patch
> > without it, but I''d prefer to have something...
> 
> I think at the very least there should be the ability to have it as a
> global config option.
Like the seatbelt option or perhaps a better name ''expert'' in
the /etc/xl.conf?
> 
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically making a
PCI device assignable in the config file"):
> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> > I think at the very least there should be the ability to have it as a
> > global config option.
> 
> Like the seatbelt option or perhaps a better name
''expert'' in the /etc/xl.conf?
I was thinking "default.pci.seize" or something.

Ian.

Friday, July 12, 2013, 3:48:34 PM, you wrote:
> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a
PCI device assignable in the config file"):
>> > On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel
<david.vrabel@citrix.com> wrot> > I don''t think this
proposal really helps with avoiding this.  I think
>> > > most people will end up always adding
''seize=1'' because to avoid having
>> > > to do so means altering config files elsewhere and rebooting.
>> > 
>> > I guess what I''m worried about is the fact that we would
be changing
>> > things that are now "safe" to things that are not safe. 
At the
>> > moment, "xl pci-assignable-add" might yank out a system
device if you
>> > make a typo; but it was introduced that way, so people always had
to
>> > be careful.  But currently, "pci=[]" and "xl
pci-attach" do *not*
>> > behave that way; you have to make the device assignable first.  So
you
>> > don''t need to be particularly careful.  Adding
"seize" at least should
>> > flag up to people that they need to double-check.
>> > 
>> > If people really object to the extra flag, I can write up a patch
>> > without it, but I''d prefer to have something...
>> 
>> I think at the very least there should be the ability to have it as a
>> global config option.
> Like the seatbelt option or perhaps a better name
''expert'' in the /etc/xl.conf?
Another way could be to make it possible to run a script onbefore/onafter any
hotplug event.
That would be flexible as well as inline with what currently is done for
networking/vifs.

--
Sander
>> 
>> Ian.
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson
wrote:
> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically
making a PCI device assignable in the config file"):
> > On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> > > I think at the very least there should be the ability to have it
as a
> > > global config option.
> > 
> > Like the seatbelt option or perhaps a better name
''expert'' in the /etc/xl.conf?
> 
> I was thinking "default.pci.seize" or something.
Why not combine this with the other - the one where you can call
''xl vpcu-set <guest> <huge number of VCPUs>'' where
the number of
VCPUs is greater than PCPUs. And if the ''expert'' option is
turned
on it will let you. If it is not, it will notify you about this
being a bad idea.

Which reminds me, I need to send a patch for that..
> 
> Ian.

On 12/07/13 16:01, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
>> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically
making a PCI device assignable in the config file"):
>>> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
>>>> I think at the very least there should be the ability to have
it as a
>>>> global config option.
>>> Like the seatbelt option or perhaps a better name
''expert'' in the /etc/xl.conf?
>> I was thinking "default.pci.seize" or something.
> Why not combine this with the other - the one where you can call
> ''xl vpcu-set <guest> <huge number of VCPUs>''
where the number of
> VCPUs is greater than PCPUs. And if the ''expert'' option
is turned
> on it will let you. If it is not, it will notify you about this
> being a bad idea.
...because you may want to have one and not the other?

  -George

On Fri, Jul 12, 2013 at 04:09:29PM +0100, George Dunlap
wrote:
> On 12/07/13 16:01, Konrad Rzeszutek Wilk wrote:
> >On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
> >>Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC:
Automatically making a PCI device assignable in the config file"):
> >>>On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> >>>>I think at the very least there should be the ability to
have it as a
> >>>>global config option.
> >>>Like the seatbelt option or perhaps a better name
''expert'' in the /etc/xl.conf?
> >>I was thinking "default.pci.seize" or something.
> >Why not combine this with the other - the one where you can call
> >''xl vpcu-set <guest> <huge number of
VCPUs>'' where the number of
> >VCPUs is greater than PCPUs. And if the ''expert''
option is turned
> >on it will let you. If it is not, it will notify you about this
> >being a bad idea.
> 
> ...because you may want to have one and not the other?
Seatbelts are seatbelts. They have many parts - but nonethless
at the end of the day you describe them by just one word.

I am not a big fan of many knobs - even thought I did work at some
point for IBM which was known for its knob-for-everything products.
> 
>  -George

On 12/07/13 17:02, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 12, 2013 at 04:09:29PM +0100, George Dunlap wrote:
>> On 12/07/13 16:01, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
>>>> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC:
Automatically making a PCI device assignable in the config file"):
>>>>> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson
wrote:
>>>>>> I think at the very least there should be the ability
to have it as a
>>>>>> global config option.
>>>>> Like the seatbelt option or perhaps a better name
''expert'' in the /etc/xl.conf?
>>>> I was thinking "default.pci.seize" or something.
>>> Why not combine this with the other - the one where you can call
>>> ''xl vpcu-set <guest> <huge number of
VCPUs>'' where the number of
>>> VCPUs is greater than PCPUs. And if the ''expert''
option is turned
>>> on it will let you. If it is not, it will notify you about this
>>> being a bad idea.
>> ...because you may want to have one and not the other?
> Seatbelts are seatbelts. They have many parts - but nonethless
> at the end of the day you describe them by just one word.
>
> I am not a big fan of many knobs - even thought I did work at some
> point for IBM which was known for its knob-for-everything products.
Well remember that my main concern here is with the *changing* of 
behavior, not the *existence* of behavior.  (i.e., xl pci-assignable-add 
already behaves in the way in which xl pci-attach would behave with the 
seatbelt off).  If we had already had a single "expert" option in 
xl.conf, then we would be in nearly the same situation -- add xl 
pci-attach to the "expert" option and have it change from safe to
unsafe
without any warning, or add yet another parameter.

  -George