Filip Maj
2014-Apr-12 00:32 UTC
[libvirt-users] LXC + USB passthrough = Operation not permitted
Hi! First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D I'm running ubuntu 13.10 with LXC 1.0.1 and tried both libvirt 1.1.1 and 1.2.2 (backported from ubuntu-trusty), but with either version of libvirt am getting issues as soon as I try to get access to USB devices inside the container. Relevant versions of stuff: $ dpkg -l | grep -i lxc ii liblxc1 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools (library) ii lxc 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools ii lxc-templates 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools (templates) ii python3-lxc 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools (Python 3.x bindings) $ dpkg -l | grep libvirt ii libvirt-bin 1.2.2-0ubuntu12 i386 programs for the libvirt library ii libvirt0 1.2.2-0ubuntu12 i386 library for interfacing with different virtualization systems ii python-libvirt 1.2.2-0ubuntu1 i386 libvirt Python bindings Here's my entire domain definition: <domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain> Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device connected to the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log: 2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted I get the same output above when I use libvirt 1.1.1. Tracing down through the libvirt code it looks like it fails when calling `mknod`. It's clear that libvirt converts the product/vendor ids I specified in the domain definition file to the appropriate USB bus (/dev/bus/usb/002/003, verified by comparing against output of `usb-devices`). In a parallel effort, a colleague of mine set up a vanilla LXContainer on a new install of the same version of ubuntu, and successfully was able to communicate with the Android device from a container. The only configuration tweaks he made was make sure to whitelist the cgroup with the appropriate major/minor device number (associated with the Android device) inside the LXC configuration file. From there the container was able to appropriately create the device file as you attach/detach the Android device. Hopefully someone can shed some light, and appreciate your patience with me as I am learning a lot of this stuff as I go :) Thanks in advance for any help! Fil
Daniel P. Berrange
2014-Apr-15 08:33 UTC
Re: [libvirt-users] LXC + USB passthrough = Operation not permitted
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:> Hi! > > First post, kind of a noobie. I've been working with LXC and libvirt for a > few months now. Trying to do some interesting things with containers and > Android devices :D > Here's my entire domain definition: > > <domain type='lxc'> > <name>oshi32134</name> > <uuid>xxxxx</uuid> > <memory unit='KiB'>3145728</memory> > <currentMemory unit='KiB'>3145728</currentMemory> > <vcpu placement='static'>1</vcpu> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='i686'>exe</type> > <init>/sbin/init</init> > </os> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/some/valid/filesystem/location'/> > <target dir='/'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/another/valid/filesystem/location'/> > <target dir='/mnt/android'/> > </filesystem> > <interface type='bridge'> > <mac address='xx:xx:xx:xx:xx:xx'/> > <source bridge='br1'/> > </interface> > <console type='pty'> > <target type='lxc' port='0'/> > </console> > <hostdev mode='capabilities' type='misc'> > <source> > <char>/dev/kvm</char> > </source> > </hostdev> > <hostdev mode='subsystem' type='usb' managed='yes'> > <source> > <vendor id='0x04e8'/> > <product id='0x6860'/> > </source> > </hostdev> > </devices> > </domain>Your config looks fine here.> > Everything worked fine until I added the USB <hostdev> element. I'm > essentially trying to get access to a physical Android device connected to > the host from inside a container. When I go to start the container, I get > an error about Operation not permitted. Here's the relevant bits from > /var/log/libvirt/lxc/machine.log: > > 2014-04-11 22:46:40.491+0000: starting up > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc > --name oshi32134 --console 24 --security=none --handshake 27 --background > --veth vnet1 > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 > 2014-04-11 22:46:40.597+0000: 685: error : > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > permitted > Unable to create device > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > permittedDo you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Filip Maj
2014-Apr-16 22:25 UTC
Re: [libvirt-users] LXC + USB passthrough = Operation not permitted
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode: $ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626) And still get issues. From libvirtd.log: 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted 2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices Thanks in advance for any help, Daniel! Cheers, Fil On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote:> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote: > > Hi! > > > > First post, kind of a noobie. I've been working with LXC and libvirt for > a > > few months now. Trying to do some interesting things with containers and > > Android devices :D > > Here's my entire domain definition: > > > > <domain type='lxc'> > > <name>oshi32134</name> > > <uuid>xxxxx</uuid> > > <memory unit='KiB'>3145728</memory> > > <currentMemory unit='KiB'>3145728</currentMemory> > > <vcpu placement='static'>1</vcpu> > > <resource> > > <partition>/machine</partition> > > </resource> > > <os> > > <type arch='i686'>exe</type> > > <init>/sbin/init</init> > > </os> > > <clock offset='utc'/> > > <on_poweroff>destroy</on_poweroff> > > <on_reboot>restart</on_reboot> > > <on_crash>destroy</on_crash> > > <devices> > > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> > > <filesystem type='mount' accessmode='passthrough'> > > <source dir='/some/valid/filesystem/location'/> > > <target dir='/'/> > > </filesystem> > > <filesystem type='mount' accessmode='passthrough'> > > <source dir='/another/valid/filesystem/location'/> > > <target dir='/mnt/android'/> > > </filesystem> > > <interface type='bridge'> > > <mac address='xx:xx:xx:xx:xx:xx'/> > > <source bridge='br1'/> > > </interface> > > <console type='pty'> > > <target type='lxc' port='0'/> > > </console> > > <hostdev mode='capabilities' type='misc'> > > <source> > > <char>/dev/kvm</char> > > </source> > > </hostdev> > > <hostdev mode='subsystem' type='usb' managed='yes'> > > <source> > > <vendor id='0x04e8'/> > > <product id='0x6860'/> > > </source> > > </hostdev> > > </devices> > > </domain> > > Your config looks fine here. > > > > > Everything worked fine until I added the USB <hostdev> element. I'm > > essentially trying to get access to a physical Android device connected > to > > the host from inside a container. When I go to start the container, I get > > an error about Operation not permitted. Here's the relevant bits from > > /var/log/libvirt/lxc/machine.log: > > > > 2014-04-11 22:46:40.491+0000: starting up > > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc > > --name oshi32134 --console 24 --security=none --handshake 27 --background > > --veth vnet1 > > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 > > 2014-04-11 22:46:40.597+0000: 685: error : > > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device > > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > > permitted > > Unable to create device > > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > > permitted > > Do you have AppArmour enabled on the machine. That seems like the > most likely thing that would result in libvirt getting that permission > error. > > Regards, > Daniel > -- > |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| > |: http://libvirt.org -o- http://virt-manager.org:| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:| >