Hi,
We have a VM with several USB devices attached. Everything works well, but
sometimes, after a reboot of the host, some usb device get a different
bus/device number and that prevent the reboot of the VM :
-------
error: Failed to start domain xxx
error: internal error: Did not find USB device 04b9:0300 bus:1 device:5
-------
I guess this is because we have multiple usb devices with the same
vendor/product ID :
-------
<hostdev mode='subsystem' type='usb'
managed='yes'>
<source>
<vendor id='0x04b9'/>
<product id='0x0300'/>
<address bus='2' device='4'/>
</source>
<alias name='hostdev0'/>
<address type='usb' bus='0' port='2'/>
</hostdev>
<hostdev mode='subsystem' type='usb'
managed='yes'>
<source>
<vendor id='0x0529'/>
<product id='0x0001'/>
<address bus='1' device='7'/>
</source>
<alias name='hostdev1'/>
<address type='usb' bus='0' port='1.2'/>
</hostdev>
<hostdev mode='subsystem' type='usb'
managed='yes'>
<source>
<vendor id='0x0529'/>
<product id='0x0001'/>
<address bus='2' device='3'/>
</source>
<alias name='hostdev2'/>
<address type='usb' bus='0' port='1.4'/>
</hostdev>
<hostdev mode='subsystem' type='usb'
managed='yes'>
<source>
<vendor id='0x04b9'/>
<product id='0x0300'/>
<address bus='1' device='6'/>
</source>
<alias name='hostdev3'/>
<address type='usb' bus='0' port='1.1'/>
</hostdev>
-------
Any idea on how to handle this case ?
We were thinking about using udev, but it seems that libvirt only supports
vendor/id (not unique in our case) and bus/device (not predictable) to identify
usb devices.
Thanks,
Maxime Accadia
P.S. We use libvirt 4.0.0 on Ubuntu 18.04
On Mon, Feb 03, 2020 at 02:20:47PM +0100, Maxime Accadia wrote:>Hi, > >We have a VM with several USB devices attached. Everything works well, but sometimes, after a reboot of the host, some usb device get a different bus/device number and that prevent the reboot of the VM : > >------- >error: Failed to start domain xxx >error: internal error: Did not find USB device 04b9:0300 bus:1 device:5 >------- > >I guess this is because we have multiple usb devices with the same vendor/product ID : > >------- > <hostdev mode='subsystem' type='usb' managed='yes'> > <source> > <vendor id='0x04b9'/> > <product id='0x0300'/> > <address bus='2' device='4'/> > </source> > <alias name='hostdev0'/> > <address type='usb' bus='0' port='2'/> > </hostdev> > <hostdev mode='subsystem' type='usb' managed='yes'> > <source> > <vendor id='0x0529'/> > <product id='0x0001'/> > <address bus='1' device='7'/> > </source> > <alias name='hostdev1'/> > <address type='usb' bus='0' port='1.2'/> > </hostdev> > <hostdev mode='subsystem' type='usb' managed='yes'> > <source> > <vendor id='0x0529'/> > <product id='0x0001'/> > <address bus='2' device='3'/> > </source> > <alias name='hostdev2'/> > <address type='usb' bus='0' port='1.4'/> > </hostdev> > <hostdev mode='subsystem' type='usb' managed='yes'> > <source> > <vendor id='0x04b9'/> > <product id='0x0300'/> > <address bus='1' device='6'/> > </source> > <alias name='hostdev3'/> > <address type='usb' bus='0' port='1.1'/> > </hostdev> >------- > >Any idea on how to handle this case ? > >We were thinking about using udev, but it seems that libvirt only supports vendor/id (not unique in our case) and bus/device (not predictable) to identify usb devices.There has been a proposal (or two?) for addressing USB hostdevs by bus/port https://bugzilla.redhat.com/show_bug.cgi?id=1497984 Sadly it never landed upstream. Other than writing a custom script that fills in the <address bus='1' device='6'/> XML with the current values, I can't think of any solution. Jano> >Thanks, > >Maxime Accadia > >P.S. We use libvirt 4.0.0 on Ubuntu 18.04 > >
Hi Maxime, Quoting Maxime Accadia (2020-02-03 14:20:47)> Hi, > > We have a VM with several USB devices attached. Everything works well, > but sometimes, after a reboot of the host, some usb device get a > different bus/device number and that prevent the reboot of the VM : > ... > Any idea on how to handle this case ? > > We were thinking about using udev, but it seems that libvirt only > supports vendor/id (not unique in our case) and bus/device (not > predictable) to identify usb devices.I had a similar issue where I have to distribute some identical USB devices across three VMs. On my setup, the key to the fix is that the *PCI* topology of the USB buses doesn't change, the port in which the device is attached doesn't change, and udev supports globbing. Here's my udev rules. Note I've left in the non-globbed old rules, so that you can see how a typical bus distribution can look in my OS under /sys/devices. /etc/udev/rules.d/99-libvirt-wireless-dongles.rules: # This doesn't work for removal since ATTR{devpath} isn't available at # that point. #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ATTR{busnum}=="8", ATTR{devpath}=="6", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh gromit" SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="0cf3", ENV{ID_MODEL_ID}=="9271", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1d.7/usb?/?-5", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh gromit" #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="0cf3", ENV{ID_MODEL_ID}=="9271", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1d.7/usb8/8-5", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh gromit" SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb?/?-2", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh odie" #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb7/7-2", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh odie" SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb?/?-6", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh snoopy" #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb7/7-6", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh snoopy" == Now, due to the globs, this *might* break down if you have two identical devices on two USB buses that are attached to the same PCI address, but I *think* every bus actually has a dedicated PCI sub-address that doesn't change even though the bus number itself might (the .7 in 1d.7 and 1a.7 in the rules). So YMMV, but I'm highly confident it will just work. The script being called, usb-libvirt-hotlug.sh <vmname>, can be found at <https://github.com/olavmrk/usb-libvirt-hotplug/blob/master/usb-libvirt-hotplug.sh> However, I made a minor edit so that it uses attach --live, so that they're immediately forwarded even if the machine is already up. So line 101-102 becomes: echo "Running virsh ${COMMAND} ${DOMAIN} --live for USB vendor=0x${ID_VENDOR_ID} product=0x${ID_MODEL_ID} bus=${BUSNUM} device=${DEVNUM}:" >&2 virsh "${COMMAND}" "${DOMAIN}" /dev/stdin --live <<END == I'm including the entire script below for posterity's sake, it's MIT-licensed. And yes, I'm including the license as well because that's what you're supposed to do. The MIT License (MIT) Copyright (c) 2016 Olav Morken Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ==/usr/local/bin/usb-libvirt-hotplug.sh: !/bin/bash # # usb-libvirt-hotplug.sh # # This script can be used to hotplug USB devices to libvirt virtual # machines from udev rules. # # This can be used to attach devices when they are plugged into a # specific port on the host machine. # # See: https://github.com/olavmrk/usb-libvirt-hotplug # # Abort script execution on errors set -e PROG="$(basename "$0")" if [ ! -t 1 ]; then # stdout is not a tty. Send all output to syslog. coproc logger --tag "${PROG}" exec >&${COPROC[1]} 2>&1 fi DOMAIN="$1" if [ -z "${DOMAIN}" ]; then echo "Missing libvirt domain parameter for ${PROG}." >&2 exit 1 fi # # Do some sanity checking of the udev environment variables. # if [ -z "${SUBSYSTEM}" ]; then echo "Missing udev SUBSYSTEM environment variable." >&2 exit 1 fi if [ "${SUBSYSTEM}" != "usb" ]; then echo "Invalid udev SUBSYSTEM: ${SUBSYSTEM}" >&2 echo "You should probably add a SUBSYSTEM=\"USB\" match to your udev rule." >&2 exit 1 fi if [ -z "${DEVTYPE}" ]; then echo "Missing udev DEVTYPE environment variable." >&2 exit 1 fi if [ "${DEVTYPE}" == "usb_interface" ]; then # This is normal -- sometimes the udev rule will match # usb_interface events as well. exit 0 fi if [ "${DEVTYPE}" != "usb_device" ]; then echo "Invalid udev DEVTYPE: ${DEVTYPE}" >&2 exit 1 fi if [ -z "${ACTION}" ]; then echo "Missing udev ACTION environment variable." >&2 exit 1 fi if [ "${ACTION}" == 'add' ]; then COMMAND='attach-device' elif [ "${ACTION}" == 'remove' ]; then COMMAND='detach-device' else echo "Invalid udev ACTION: ${ACTION}" >&2 exit 1 fi if [ -z "${BUSNUM}" ]; then echo "Missing udev BUSNUM environment variable." >&2 exit 1 fi if [ -z "${DEVNUM}" ]; then echo "Missing udev DEVNUM environment variable." >&2 exit 1 fi # # This is a bit ugly. udev passes us the USB bus number and # device number with leading zeroes. E.g.: # BUSNUM=001 DEVNUM=022 # This causes libvirt to assume that the numbers are octal. # To work around this, we need to strip the leading zeroes. # The easiest way is to ask bash to convert the numbers from # base 10: # BUSNUM=$((10#$BUSNUM)) DEVNUM=$((10#$DEVNUM)) # # Now we have all the information we need to update the VM. # Run the appropriate virsh-command, and ask it to read the # update XML from stdin. # echo "Running virsh ${COMMAND} ${DOMAIN} --live for USB vendor=0x${ID_VENDOR_ID} product=0x${ID_MODEL_ID} bus=${BUSNUM} device=${DEVNUM}:" >&2 virsh "${COMMAND}" "${DOMAIN}" /dev/stdin --live <<END <hostdev mode='subsystem' type='usb' managed='yes'> <source startupPolicy='optional'> <vendor id='0x${ID_VENDOR_ID}' /> <product id='0x${ID_MODEL_ID}' /> <address bus='${BUSNUM}' device='${DEVNUM}' /> </source> </hostdev> END ===
Thank you for your answers and pointing to the relevant feature request. Hopefully, we will see some progress with the new assignee. We will try the script solution until then. Maxime ----- Mail original ----- De: "Pol Van Aubel" <libvirt@qwfp.nl> À: "MAXIME ACCADIA" <maxime.accadia@univ-grenoble-alpes.fr>, "libvirt-users" <libvirt-users@redhat.com> Cc: "ovidiu gacea" <ovidiu.gacea@univ-grenoble-alpes.fr> Envoyé: Lundi 3 Février 2020 14:59:09 Objet: Re: USB devices with same vendor:product id Hi Maxime, Quoting Maxime Accadia (2020-02-03 14:20:47)> Hi, > > We have a VM with several USB devices attached. Everything works well, > but sometimes, after a reboot of the host, some usb device get a > different bus/device number and that prevent the reboot of the VM : > ... > Any idea on how to handle this case ? > > We were thinking about using udev, but it seems that libvirt only > supports vendor/id (not unique in our case) and bus/device (not > predictable) to identify usb devices.I had a similar issue where I have to distribute some identical USB devices across three VMs. On my setup, the key to the fix is that the *PCI* topology of the USB buses doesn't change, the port in which the device is attached doesn't change, and udev supports globbing. Here's my udev rules. Note I've left in the non-globbed old rules, so that you can see how a typical bus distribution can look in my OS under /sys/devices. /etc/udev/rules.d/99-libvirt-wireless-dongles.rules: # This doesn't work for removal since ATTR{devpath} isn't available at # that point. #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ATTR{busnum}=="8", ATTR{devpath}=="6", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh gromit" SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="0cf3", ENV{ID_MODEL_ID}=="9271", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1d.7/usb?/?-5", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh gromit" #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="0cf3", ENV{ID_MODEL_ID}=="9271", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1d.7/usb8/8-5", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh gromit" SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb?/?-2", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh odie" #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb7/7-2", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh odie" SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb?/?-6", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh snoopy" #SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="148f", ENV{ID_MODEL_ID}=="5370", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:1a.7/usb7/7-6", RUN+="/usr/local/bin/usb-libvirt-hotplug.sh snoopy" == Now, due to the globs, this *might* break down if you have two identical devices on two USB buses that are attached to the same PCI address, but I *think* every bus actually has a dedicated PCI sub-address that doesn't change even though the bus number itself might (the .7 in 1d.7 and 1a.7 in the rules). So YMMV, but I'm highly confident it will just work. The script being called, usb-libvirt-hotlug.sh <vmname>, can be found at <https://github.com/olavmrk/usb-libvirt-hotplug/blob/master/usb-libvirt-hotplug.sh> However, I made a minor edit so that it uses attach --live, so that they're immediately forwarded even if the machine is already up. So line 101-102 becomes: echo "Running virsh ${COMMAND} ${DOMAIN} --live for USB vendor=0x${ID_VENDOR_ID} product=0x${ID_MODEL_ID} bus=${BUSNUM} device=${DEVNUM}:" >&2 virsh "${COMMAND}" "${DOMAIN}" /dev/stdin --live <<END == I'm including the entire script below for posterity's sake, it's MIT-licensed. And yes, I'm including the license as well because that's what you're supposed to do. The MIT License (MIT) Copyright (c) 2016 Olav Morken Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ==/usr/local/bin/usb-libvirt-hotplug.sh: !/bin/bash # # usb-libvirt-hotplug.sh # # This script can be used to hotplug USB devices to libvirt virtual # machines from udev rules. # # This can be used to attach devices when they are plugged into a # specific port on the host machine. # # See: https://github.com/olavmrk/usb-libvirt-hotplug # # Abort script execution on errors set -e PROG="$(basename "$0")" if [ ! -t 1 ]; then # stdout is not a tty. Send all output to syslog. coproc logger --tag "${PROG}" exec >&${COPROC[1]} 2>&1 fi DOMAIN="$1" if [ -z "${DOMAIN}" ]; then echo "Missing libvirt domain parameter for ${PROG}." >&2 exit 1 fi # # Do some sanity checking of the udev environment variables. # if [ -z "${SUBSYSTEM}" ]; then echo "Missing udev SUBSYSTEM environment variable." >&2 exit 1 fi if [ "${SUBSYSTEM}" != "usb" ]; then echo "Invalid udev SUBSYSTEM: ${SUBSYSTEM}" >&2 echo "You should probably add a SUBSYSTEM=\"USB\" match to your udev rule." >&2 exit 1 fi if [ -z "${DEVTYPE}" ]; then echo "Missing udev DEVTYPE environment variable." >&2 exit 1 fi if [ "${DEVTYPE}" == "usb_interface" ]; then # This is normal -- sometimes the udev rule will match # usb_interface events as well. exit 0 fi if [ "${DEVTYPE}" != "usb_device" ]; then echo "Invalid udev DEVTYPE: ${DEVTYPE}" >&2 exit 1 fi if [ -z "${ACTION}" ]; then echo "Missing udev ACTION environment variable." >&2 exit 1 fi if [ "${ACTION}" == 'add' ]; then COMMAND='attach-device' elif [ "${ACTION}" == 'remove' ]; then COMMAND='detach-device' else echo "Invalid udev ACTION: ${ACTION}" >&2 exit 1 fi if [ -z "${BUSNUM}" ]; then echo "Missing udev BUSNUM environment variable." >&2 exit 1 fi if [ -z "${DEVNUM}" ]; then echo "Missing udev DEVNUM environment variable." >&2 exit 1 fi # # This is a bit ugly. udev passes us the USB bus number and # device number with leading zeroes. E.g.: # BUSNUM=001 DEVNUM=022 # This causes libvirt to assume that the numbers are octal. # To work around this, we need to strip the leading zeroes. # The easiest way is to ask bash to convert the numbers from # base 10: # BUSNUM=$((10#$BUSNUM)) DEVNUM=$((10#$DEVNUM)) # # Now we have all the information we need to update the VM. # Run the appropriate virsh-command, and ask it to read the # update XML from stdin. # echo "Running virsh ${COMMAND} ${DOMAIN} --live for USB vendor=0x${ID_VENDOR_ID} product=0x${ID_MODEL_ID} bus=${BUSNUM} device=${DEVNUM}:" >&2 virsh "${COMMAND}" "${DOMAIN}" /dev/stdin --live <<END <hostdev mode='subsystem' type='usb' managed='yes'> <source startupPolicy='optional'> <vendor id='0x${ID_VENDOR_ID}' /> <product id='0x${ID_MODEL_ID}' /> <address bus='${BUSNUM}' device='${DEVNUM}' /> </source> </hostdev> END ===
Quoting Ján Tomko (2020-02-03 14:45:38)> There has been a proposal (or two?) for addressing USB hostdevs by bus/port > https://bugzilla.redhat.com/show_bug.cgi?id=1497984 > Sadly it never landed upstream.Regarding that patch, as I mentioned in another reply, *USB* bus numbers on Linux aren't stable -- see <https://github.com/torvalds/linux/blob/master/drivers/usb/core/usb.c#L629-L633> ("Each device's dev->devpath will be stable until USB is re-cabled, and hubs are often labeled with these port numbers. The name isn't as stable: bus->busnum changes easily from modprobe order, cardbus or pci hotplugging, and so on.") So unless it uses PCI devpaths, which from a quick glance at the code I don't think it does, that doesn't actually solve the problem of changing *bus* numbers -- it only solves the problem of changing *device* numbers under an assumed-to-be-but-not-actually stable bus. -- Pol