I am running Xen 4.2 with the XL toolstack on Arch Linux. I have a set of PCI devices that I am trying to hide. From the command line I can hide the devices with # xl pci-assignable-add 01:00.0 # xl pci-assignable-add 01:00.1 # xl pci-assignable-add 02:00.0 # xl pci-assignable-add 03:00.0 and confirm they are hidden with # xl pci-assignable-list I wanted to automate this so I created a file called /etc/modprobe.d/pcihide.conf and included options xen-pciback hide=(01:00.0)(01.00.1)(02:00.0)(03:00.0) When I reboot only some of the devices are hidden. Looking in dmesg and xl dmesg I can see entries for the devices that are successfully hidden, but no obvious entries for the devices that are not hidden. If I remove the /etc/modprobe.d/pcihide.conf file and reboot, no devices are hidden, so the file is getting loaded and I am able to hide some devices with this method. I am unable to hide a built in SATA controller, two built in USB 2.0 controllers, two built in USB 3.0 controllers, and an add on SATA card with the modprobe.d file. I am able to hide two add on video cards(with HDMI audio) and a TV tuner card with the modprobe.d file. I can hide all the devices with xl pci-assignable-add. Is using a modprobe.d file the right way to hide PCI devices on boot? If so, I know this isn''t a lot to work with, but where should I be looking for information about what is going wrong? This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Mon, 2013-06-17 at 11:25 +0100, Daniel Shub wrote:> When I reboot only some of the devices are hidden. Looking in dmesg > and xl dmesg I can see entries for the devices that are successfully > hidden, but no obvious entries for the devices that are not hidden. If > I remove the /etc/modprobe.d/pcihide.conf file and reboot, no devices > are hidden, so the file is getting loaded and I am able to hide some > devices with this method.My guess would be that it is todo with the module load order. Those which are loaded before pciback do not get hidden, those which are after do. It''s not often an option with distro kernels but building pciback in statically (and putting the options on the hypervisor command line) seems like it might help. If you are passing through every device of a given type then you could also consider blacklisting the driver in dom0. Obviously this doesn''t help if you have two and want to pass one through and keep the other for dom0. Other than that I''m not sure what to suggest, perhaps others on the list have advice. Ian.
On Mon, 17 Jun 2013 11:25:01 +0100, Daniel Shub <Daniel.Shub@nottingham.ac.uk> wrote:> I am running Xen 4.2 with the XL toolstack on Arch Linux. I have a > set > of PCI devices that I am trying to hide. From the command line I can > hide the devices with > > # xl pci-assignable-add 01:00.0 > > # xl pci-assignable-add 01:00.1 > > # xl pci-assignable-add 02:00.0 > > # xl pci-assignable-add 03:00.0 > > and confirm they are hidden with > > # xl pci-assignable-list > > I wanted to automate this so I created a file called > /etc/modprobe.d/pcihide.conf and included > > options xen-pciback hide=(01:00.0)(01.00.1)(02:00.0)(03:00.0) > > When I reboot only some of the devices are hidden. Looking in dmesg > and xl dmesg I can see entries for the devices that are successfully > hidden, but no obvious entries for the devices that are not hidden. > If > I remove the /etc/modprobe.d/pcihide.conf file and reboot, no devices > are hidden, so the file is getting loaded and I am able to hide some > devices with this method. > > I am unable to hide a built in SATA controller, two built in USB 2.0 > controllers, two built in USB 3.0 controllers, and an add on SATA > card > with the modprobe.d file. I am able to hide two add on video > cards(with HDMI audio) and a TV tuner card with the modprobe.d file. > I > can hide all the devices with xl pci-assignable-add. > > Is using a modprobe.d file the right way to hide PCI devices on boot? > If so, I know this isn''t a lot to work with, but where should I be > looking for information about what is going wrong?You''ll find the answer and an example here: https://lists.wireless.org.au/pipermail/kernel-xen/2013-May/000241.html You will need to modify it for your setup (virsh vs. xm vs. xl), but it should give you the right idea about what to do, and how to make sure the drivers are unbound before you need to use them. In a nutshell - xen-pciback will not unbind drivers that are already bound to hardware - it will only bind itself to the devices you listed if nothing else has claimed them already. So you need to create an install script for each of the driver modules that intercepts the load and does the required unbinding work. If a particular kernel driver module handles only the hardware you want to pass through, you can just blacklist that module and skip the intercept for it. You only need the intercept if you have multiple pieces of similar hardware handled by the same driver and you need to some of it in the dom0. Otherwise you might as well just blacklist the driver. Gordan
> -----Original Message----- > From: Ian Campbell [mailto:Ian.Campbell@citrix.com] > > My guess would be that it is todo with the module load order. Those > which are loaded before pciback do not get hidden, those which are > after do.That seems like a reasonable explanation. I think I can modify my modprobe.d conf to unbind the bound modules> > It''s not often an option with distro kernels but building pciback in > statically (and putting the options on the hypervisor command line) > seems like it might help. >I was hoping to avoid this. I think in the worse case I could always create a dummy system service which uses xl to hide the PCI devices and launch my domUs, but it doesn''t quite seem right.
> -----Original Message----- > From: Gordan Bobic [mailto:gordan@bobich.net] > > You''ll find the answer and an example here: > > https://lists.wireless.org.au/pipermail/kernel-xen/2013- > May/000241.html >Super helpful, thank you. Instead of creating configurations for each problematic module, could I instead add a script to my pcihide.conf that unbinds the devices? Something like a pciback.conf of: install xen-pciback /usr/local/sbin/detach-pci.sh; insmod /path/to/the/xen-pciback.ko (I am not near my dom0 and don''t know the path) and a detach-pci.sh file of: echo -n "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind Thanks for the help.
On Mon, 17 Jun 2013 12:58:34 +0100, Daniel Shub <Daniel.Shub@nottingham.ac.uk> wrote:>> -----Original Message----- >> From: Gordan Bobic [mailto:gordan@bobich.net] >> >> You''ll find the answer and an example here: >> >> https://lists.wireless.org.au/pipermail/kernel-xen/2013- >> May/000241.html >> > > > Super helpful, thank you. Instead of creating configurations for each > problematic module, could I instead add a script to my pcihide.conf > that unbinds the devices? Something like a pciback.conf of: > > install xen-pciback /usr/local/sbin/detach-pci.sh; insmod > /path/to/the/xen-pciback.ko > > (I am not near my dom0 and don''t know the path) and a detach-pci.sh > file of: > > echo -n "0000:01:00.0" > > /sys/bus/pci/devices/0000:01:00.0/driver/unbindIndeed, that should work. That is pretty much exactly the same thing that the other methods for detaching or making a device assignable do. Note: 1) You (mostly) don''t need this if you (can) blacklist the driver. 2) If you create a modprobe.d conf file for each of the affected drivers and you make the script modprobe xen-pciback, there is a good chance that you won''t need any manual unbinding at all - when the first driver load is triggered it will load xen-pciback first, and since you listed the devices for it to hide (i.e. bind to), when the driver tries to load it won''t find any unused hardware to bind to and thus won''t need to be unbound. Gordan
Daniel, On Mon, Jun 17, 2013 at 5:25 AM, Daniel Shub <Daniel.Shub@nottingham.ac.uk>wrote:> I am running Xen 4.2 with the XL toolstack on Arch Linux. I have a set of > PCI devices that I am trying to hide. From the command line I can hide the > devices with**** > > ** ** > > # xl pci-assignable-add 01:00.0**** > > # xl pci-assignable-add 01:00.1**** > > # xl pci-assignable-add 02:00.0**** > > # xl pci-assignable-add 03:00.0**** > > ** ** > > and confirm they are hidden with**** > > ** ** > > # xl pci-assignable-list**** > > ** ** > > I wanted to automate this so I created a file called > /etc/modprobe.d/pcihide.conf and included**** > > ** ** > > options xen-pciback hide=(01:00.0)(01.00.1)(02:00.0)(03:00.0)**** > > ** ** > > When I reboot only some of the devices are hidden. Looking in dmesg and xl > dmesg I can see entries for the devices that are successfully hidden, but > no obvious entries for the devices that are not hidden. If I remove the > /etc/modprobe.d/pcihide.conf file and reboot, no devices are hidden, so the > file is getting loaded and I am able to hide some devices with this method. > **** > > ** ** > > I am unable to hide a built in SATA controller, two built in USB 2.0 > controllers, two built in USB 3.0 controllers, and an add on SATA card with > the modprobe.d file. I am able to hide two add on video cards(with HDMI > audio) and a TV tuner card with the modprobe.d file. I can hide all the > devices with xl pci-assignable-add.**** > > ** ** > > Is using a modprobe.d file the right way to hide PCI devices on boot? If > so, I know this isn’t a lot to work with, but where should I be looking for > information about what is going wrong? >I''m also using Arch Linux, with xen-4.2.2. The way I tackled this problem is moved the loading of xen-pciback into the initrd, so that it would load before any of the other modules. In order to do this, edit your /etc/mkinitcpio.conf and make the following changes: * Add ''xen-pciback'' into the MODULES array. This will make sure that the module is included in the initrd. * Add ''modconf'' into the ''HOOKS'' array . This will make sure that all the files in /etc/modprobe.d/ get included in the initrd. The only other thing I did was use the same name as the module for my modprobe.d file: $ cat /etc/modprobe.d/xen-pciback.conf options xen-pciback hide=(0000:06:00.1)(0000:06:00.0)(0000:00:12.2)(0000:00:12.0) Regards, David _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Mon, 17 Jun 2013 08:59:53 -0500, David Sutton <kantras@gmail.com> wrote:> Daniel, > > On Mon, Jun 17, 2013 at 5:25 AM, Daniel Shub wrote: > > I am running Xen 4.2 with the XL toolstack on Arch Linux. I have a > set > of PCI devices that I am trying to hide. From the command line I can > hide the devices with > > > > # xl pci-assignable-add 01:00.0 > > # xl pci-assignable-add 01:00.1 > > # xl pci-assignable-add 02:00.0 > > # xl pci-assignable-add 03:00.0 > > > > and confirm they are hidden with > > > > # xl pci-assignable-list > > > > I wanted to automate this so I created a file called > /etc/modprobe.d/pcihide.conf and included > > > > options xen-pciback hide=(01:00.0)(01.00.1)(02:00.0)(03:00.0) > > > > When I reboot only some of the devices are hidden. Looking in dmesg > and xl dmesg I can see entries for the devices that are successfully > hidden, but no obvious entries for the devices that are not hidden. > If > I remove the /etc/modprobe.d/pcihide.conf file and reboot, no devices > are hidden, so the file is getting loaded and I am able to hide some > devices with this method. > > > > I am unable to hide a built in SATA controller, two built in USB 2.0 > controllers, two built in USB 3.0 controllers, and an add on SATA > card > with the modprobe.d file. I am able to hide two add on video > cards(with HDMI audio) and a TV tuner card with the modprobe.d file. > I > can hide all the devices with xl pci-assignable-add. > > > > Is using a modprobe.d file the right way to hide PCI devices on > boot? If so, I know this isn’t a lot to work with, but where should > I be looking for information about what is going wrong? > > I'm also using Arch Linux, with xen-4.2.2. The way I tackled this > problem is moved the loading of xen-pciback into the initrd, so that > it would load before any of the other modules. In order to do this, > edit your /etc/mkinitcpio.conf and make the following changes: > > * Add 'xen-pciback' into the MODULES array. This will make sure that > the module is included in the initrd. > > * Add 'modconf' into the 'HOOKS' array . This will make sure that all > the files in /etc/modprobe.d/ get included in the initrd. > > The only other thing I did was use the same name as the module for my > modprobe.d file: > > $ cat /etc/modprobe.d/xen-pciback.conf > options xen-pciback > hide=(0000:06:00.1)(0000:06:00.0)(0000:00:12.2)(0000:00:12.0)That is a reasonably neat solution, but it does come with the drawback that you have to rebuild your initrd every time you need to change the devices you need to pass through. Granted, though, that probably won't happen much outside of a testing environment. Gordan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
I have now tried a number of different solutions for hiding the PCI devices, but nothing is working completely. The simplest conceptually is David''s suggestion to load the xen-pciback module in the initrd. This guarantees that the xen-pciback module gets loaded before any other modules have a chance to bind the devices. This works nearly flawlessly, except for an issue with my Intel integrated sound card that I am trying to hide. I am having the identical problem as was discussed at http://lists.xen.org/archives/html/xen-users/2013-05/msg00283.html where I need to allow the dom0 to bind the sound card and load the kernel modules before hiding it, otherwise I get noise and choppy playback in the domU. This means Ian''s suggestion of recompiling the kernel and and hiding the devices with grub will likely not be any better. I then tried to hide the devices later since this would allow the dom0 to bind the sound card. This is was basically a modification of Gordan''s suggestion where I unbind devices with a modprobe.d conf file since. My conf file is install xen-pciback /root/pcidetach.sh; insmod /lib/modules/$(/bin/uname -r)/kernel/drivers/xen/xen-pciback/xen-pciback.ko.gz where I tried a number of different things in pcidetach.sh. The problem I kept running into is that I am running pcidetach.sh before loading the xen-pciback module which means I cannot hide anything. i should say that Gordan''s actual suggestion probably would not run into this problem since he suggested providing the configuration to prevent other modules from binding the devices. I finally tried hiding the devices even later by creating a systemd udev service [Unit] Description=PCI hide [Service] Type=oneshot ExecStart=/bin/bash /root/pcihide.sh [Install] WantedBy=multi-user.target where pcihide.sh uses the xl toolstack to hide the devices. This works, except it runs after the domUs in /etc/xen/auto get started. I could obviously add the start up of the domUs into pcihide.sh, but seems like a real hack. My plan is to go back to hiding the devices in the initrd where I load explicitly load snd-intel-hda and then xen-pciback. I would need to create a modprobe.d conf file that can handle unbinding the sound card similar to Gordan''s orginal suggestion. It is a little harder since I need to let it get bound and then unbound before loading xen-pciback. I am still open to any suggestions. This of course would be a lot easier if the sound card issue (http://lists.xen.org/archives/html/xen-users/2013-05/msg00283.html) has been resolved. Have other people experienced the sound card problem and/or is it being worked on? Thanks for all the help so far This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.
Daniel, On Mon, Jun 17, 2013 at 4:04 PM, Daniel Shub <Daniel.Shub@nottingham.ac.uk>wrote: <snip>> > I finally tried hiding the devices even later by creating a systemd udev > service > > [Unit] > Description=PCI hide > > [Service] > Type=oneshot > ExecStart=/bin/bash /root/pcihide.sh > > [Install] > WantedBy=multi-user.target >> where pcihide.sh uses the xl toolstack to hide the devices. This works, > except it runs after the domUs in /etc/xen/auto get started. I could > obviously add the start up of the domUs into pcihide.sh, but seems like a > real hack. > >Just a quick thought, why not specify a ''Before='' so that it would load before xendomains? Regards, David _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 17/06/2013 22:04, Daniel Shub wrote:> I have now tried a number of different solutions for hiding the PCI devices, but nothing is working completely. The simplest conceptually is David''s suggestion to load the xen-pciback module in the initrd. This guarantees that the xen-pciback module gets loaded before any other modules have a chance to bind the devices. This works nearly flawlessly, except for an issue with my Intel integrated sound card that I am trying to hide. I am having the identical problem as was discussed at http://lists.xen.org/archives/html/xen-users/2013-05/msg00283.html where I need to allow the dom0 to bind the sound card and load the kernel modules before hiding it, otherwise I get noise and choppy playback in the domU. This means Ian''s suggestion of recompiling the kernel and and hiding the devices with grub will likely not be any better.> > I then tried to hide the devices later since this would allow the dom0 to bind the sound card. This is was basically a modification of Gordan''s suggestion where I unbind devices with a modprobe.d conf file since. My conf file is > > install xen-pciback /root/pcidetach.sh; insmod /lib/modules/$(/bin/uname -r)/kernel/drivers/xen/xen-pciback/xen-pciback.ko.gz > > where I tried a number of different things in pcidetach.sh. The problem I kept running into is that I am running pcidetach.sh before loading the xen-pciback module which means I cannot hide anything. i should say that Gordan''s actual suggestion probably would not run into this problem since he suggested providing the configuration to prevent other modules from binding the devices.Did you notice (or did I omit it...) that the first thing pcidetach.sh does is modprobe xen-pciback?> I finally tried hiding the devices even later by creating a systemd udev service > > [Unit] > Description=PCI hide > > [Service] > Type=oneshot > ExecStart=/bin/bash /root/pcihide.sh > > [Install] > WantedBy=multi-user.target > > where pcihide.sh uses the xl toolstack to hide the devices. This works, except it runs after the domUs in /etc/xen/auto get started. I could obviously add the start up of the domUs into pcihide.sh, but seems like a real hack. > > My plan is to go back to hiding the devices in the initrd where I load explicitly load snd-intel-hda and then xen-pciback. I would need to create a modprobe.d conf file that can handle unbinding the sound card similar to Gordan''s orginal suggestion. It is a little harder since I need to let it get bound and then unbound before loading xen-pciback. I am still open to any suggestions.I guess the reason it works for me (I also use Intel ICH audio) is because I don''t auto-start my VMs. I always start mine manually. Gordan
_______________________________________ From: David Sutton [kantras@gmail.com] Just a quick thought, why not specify a ''Before='' so that it would load before xendomains? That works. I have now written my first systemd service. Thanks for the help.This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.
> -----Original Message----- > From: Gordan Bobic [mailto:gordan@bobich.net] > > This is was basically a modification of Gordan''s suggestion where I unbind devices with > > a modprobe.d conf file > > install xen-pciback /root/pcidetach.sh; insmod /lib/modules/$(/bin/uname > >-r)/kernel/drivers/xen/xen-pciback/xen-pciback.ko.gz> Did you notice (or did I omit it...) that the first thing pcidetach.sh > does is modprobe xen-pciback?Yes, but your solution was to run pcidetach.sh when loading the nvidia module. I didn''t think I could load the xen-pciback module while loading the xen-pciback module, which is the flaw in my "modification" Dan This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.