Filip Maj
2014-Apr-16 22:25 UTC
Re: [libvirt-users] LXC + USB passthrough = Operation not permitted
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode: $ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626) And still get issues. From libvirtd.log: 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted 2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices Thanks in advance for any help, Daniel! Cheers, Fil On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote:> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote: > > Hi! > > > > First post, kind of a noobie. I've been working with LXC and libvirt for > a > > few months now. Trying to do some interesting things with containers and > > Android devices :D > > Here's my entire domain definition: > > > > <domain type='lxc'> > > <name>oshi32134</name> > > <uuid>xxxxx</uuid> > > <memory unit='KiB'>3145728</memory> > > <currentMemory unit='KiB'>3145728</currentMemory> > > <vcpu placement='static'>1</vcpu> > > <resource> > > <partition>/machine</partition> > > </resource> > > <os> > > <type arch='i686'>exe</type> > > <init>/sbin/init</init> > > </os> > > <clock offset='utc'/> > > <on_poweroff>destroy</on_poweroff> > > <on_reboot>restart</on_reboot> > > <on_crash>destroy</on_crash> > > <devices> > > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> > > <filesystem type='mount' accessmode='passthrough'> > > <source dir='/some/valid/filesystem/location'/> > > <target dir='/'/> > > </filesystem> > > <filesystem type='mount' accessmode='passthrough'> > > <source dir='/another/valid/filesystem/location'/> > > <target dir='/mnt/android'/> > > </filesystem> > > <interface type='bridge'> > > <mac address='xx:xx:xx:xx:xx:xx'/> > > <source bridge='br1'/> > > </interface> > > <console type='pty'> > > <target type='lxc' port='0'/> > > </console> > > <hostdev mode='capabilities' type='misc'> > > <source> > > <char>/dev/kvm</char> > > </source> > > </hostdev> > > <hostdev mode='subsystem' type='usb' managed='yes'> > > <source> > > <vendor id='0x04e8'/> > > <product id='0x6860'/> > > </source> > > </hostdev> > > </devices> > > </domain> > > Your config looks fine here. > > > > > Everything worked fine until I added the USB <hostdev> element. I'm > > essentially trying to get access to a physical Android device connected > to > > the host from inside a container. When I go to start the container, I get > > an error about Operation not permitted. Here's the relevant bits from > > /var/log/libvirt/lxc/machine.log: > > > > 2014-04-11 22:46:40.491+0000: starting up > > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc > > --name oshi32134 --console 24 --security=none --handshake 27 --background > > --veth vnet1 > > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 > > 2014-04-11 22:46:40.597+0000: 685: error : > > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device > > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > > permitted > > Unable to create device > > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > > permitted > > Do you have AppArmour enabled on the machine. That seems like the > most likely thing that would result in libvirt getting that permission > error. > > Regards, > Daniel > -- > |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| > |: http://libvirt.org -o- http://virt-manager.org:| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:| >
Filip Maj
2014-Apr-16 22:58 UTC
Re: [libvirt-users] LXC + USB passthrough = Operation not permitted
To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point. On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote:> Yeah, AppArmor is enabled, but I put everything (that I could find) into > complain mode: > > $ sudo apparmor_status > apparmor module is loaded. > 12 profiles are loaded. > 3 profiles are in enforce mode. > lxc-container-default > lxc-container-default-with-mounting > lxc-container-default-with-nesting > 9 profiles are in complain mode. > /sbin/dhclient > /usr/bin/lxc-start > /usr/lib/NetworkManager/nm-dhcp-client.action > /usr/lib/connman/scripts/dhclient-script > /usr/lib/libvirt/virt-aa-helper > /usr/sbin/libvirtd > /usr/sbin/ntpd > /usr/sbin/rsyslogd > /usr/sbin/tcpdump > 3 processes have profiles defined. > 0 processes are in enforce mode. > 2 processes are in complain mode. > /usr/sbin/libvirtd (30419) > /usr/sbin/ntpd (3418) > 1 processes are unconfined but have a profile defined. > /usr/sbin/rsyslogd (626) > > And still get issues. From libvirtd.log: > > 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 > 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : > Cannot recv data: Connection reset by peer > 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : > internal error: guest failed to start: Unable to create device > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > permitted > > 2014-04-16 22:19:10.964+0000: 30420: warning : > virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in > list of active USB devices > > Thanks in advance for any help, Daniel! > > Cheers, > Fil > > > On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote: > >> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote: >> > Hi! >> > >> > First post, kind of a noobie. I've been working with LXC and libvirt >> for a >> > few months now. Trying to do some interesting things with containers and >> > Android devices :D >> > Here's my entire domain definition: >> > >> > <domain type='lxc'> >> > <name>oshi32134</name> >> > <uuid>xxxxx</uuid> >> > <memory unit='KiB'>3145728</memory> >> > <currentMemory unit='KiB'>3145728</currentMemory> >> > <vcpu placement='static'>1</vcpu> >> > <resource> >> > <partition>/machine</partition> >> > </resource> >> > <os> >> > <type arch='i686'>exe</type> >> > <init>/sbin/init</init> >> > </os> >> > <clock offset='utc'/> >> > <on_poweroff>destroy</on_poweroff> >> > <on_reboot>restart</on_reboot> >> > <on_crash>destroy</on_crash> >> > <devices> >> > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> >> > <filesystem type='mount' accessmode='passthrough'> >> > <source dir='/some/valid/filesystem/location'/> >> > <target dir='/'/> >> > </filesystem> >> > <filesystem type='mount' accessmode='passthrough'> >> > <source dir='/another/valid/filesystem/location'/> >> > <target dir='/mnt/android'/> >> > </filesystem> >> > <interface type='bridge'> >> > <mac address='xx:xx:xx:xx:xx:xx'/> >> > <source bridge='br1'/> >> > </interface> >> > <console type='pty'> >> > <target type='lxc' port='0'/> >> > </console> >> > <hostdev mode='capabilities' type='misc'> >> > <source> >> > <char>/dev/kvm</char> >> > </source> >> > </hostdev> >> > <hostdev mode='subsystem' type='usb' managed='yes'> >> > <source> >> > <vendor id='0x04e8'/> >> > <product id='0x6860'/> >> > </source> >> > </hostdev> >> > </devices> >> > </domain> >> >> Your config looks fine here. >> >> > >> > Everything worked fine until I added the USB <hostdev> element. I'm >> > essentially trying to get access to a physical Android device connected >> to >> > the host from inside a container. When I go to start the container, I >> get >> > an error about Operation not permitted. Here's the relevant bits from >> > /var/log/libvirt/lxc/machine.log: >> > >> > 2014-04-11 22:46:40.491+0000: starting up >> > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin >> > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr >> /usr/lib/libvirt/libvirt_lxc >> > --name oshi32134 --console 24 --security=none --handshake 27 >> --background >> > --veth vnet1 >> > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 >> > 2014-04-11 22:46:40.597+0000: 685: error : >> > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device >> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >> > permitted >> > Unable to create device >> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >> > permitted >> >> Do you have AppArmour enabled on the machine. That seems like the >> most likely thing that would result in libvirt getting that permission >> error. >> >> Regards, >> Daniel >> -- >> |: http://berrange.com -o- >> http://www.flickr.com/photos/dberrange/ :| >> |: http://libvirt.org -o- >> http://virt-manager.org :| >> |: http://autobuild.org -o- >> http://search.cpan.org/~danberr/ :| >> |: http://entangle-photo.org -o- >> http://live.gnome.org/gtk-vnc :| >> > >
Filip Maj
2014-Apr-17 00:11 UTC
Re: [libvirt-users] LXC + USB passthrough = Operation not permitted
Further followups! We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2 code to try to figure out what libvirt is doing under the hood. Even though we have the log level set to 1 (info) in our libvirtd.conf, we are not seeing the VIR_DEBUG() [1] statements being printed out. There are tons of other presumably-debug lines of output showing up in our log. We are sort of expecting to see the output from [1] in our logs somewhere, given the type of usb mounting we are trying to do? [1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c05dfec6428cad927cd5751004a4f3afc67899de;hb=HEAD#l1396 On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil@saucelabs.com> wrote:> To follow up on this a little bit, tail'ing kern.log while trying to get > our little container up doesn't yield anything with apparmor complaining, > so, unless I'm looking in the wrong spots for apparmor logs (which I don't > think so, as I see other apparmor-related log entries in kern.log), I am > not entirely sure this is an apparmor issue at this point. > > > On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote: > >> Yeah, AppArmor is enabled, but I put everything (that I could find) into >> complain mode: >> >> $ sudo apparmor_status >> apparmor module is loaded. >> 12 profiles are loaded. >> 3 profiles are in enforce mode. >> lxc-container-default >> lxc-container-default-with-mounting >> lxc-container-default-with-nesting >> 9 profiles are in complain mode. >> /sbin/dhclient >> /usr/bin/lxc-start >> /usr/lib/NetworkManager/nm-dhcp-client.action >> /usr/lib/connman/scripts/dhclient-script >> /usr/lib/libvirt/virt-aa-helper >> /usr/sbin/libvirtd >> /usr/sbin/ntpd >> /usr/sbin/rsyslogd >> /usr/sbin/tcpdump >> 3 processes have profiles defined. >> 0 processes are in enforce mode. >> 2 processes are in complain mode. >> /usr/sbin/libvirtd (30419) >> /usr/sbin/ntpd (3418) >> 1 processes are unconfined but have a profile defined. >> /usr/sbin/rsyslogd (626) >> >> And still get issues. From libvirtd.log: >> >> 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 >> 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : >> Cannot recv data: Connection reset by peer >> 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : >> internal error: guest failed to start: Unable to create device >> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >> permitted >> >> 2014-04-16 22:19:10.964+0000: 30420: warning : >> virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in >> list of active USB devices >> >> Thanks in advance for any help, Daniel! >> >> Cheers, >> Fil >> >> >> On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote: >> >>> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote: >>> > Hi! >>> > >>> > First post, kind of a noobie. I've been working with LXC and libvirt >>> for a >>> > few months now. Trying to do some interesting things with containers >>> and >>> > Android devices :D >>> > Here's my entire domain definition: >>> > >>> > <domain type='lxc'> >>> > <name>oshi32134</name> >>> > <uuid>xxxxx</uuid> >>> > <memory unit='KiB'>3145728</memory> >>> > <currentMemory unit='KiB'>3145728</currentMemory> >>> > <vcpu placement='static'>1</vcpu> >>> > <resource> >>> > <partition>/machine</partition> >>> > </resource> >>> > <os> >>> > <type arch='i686'>exe</type> >>> > <init>/sbin/init</init> >>> > </os> >>> > <clock offset='utc'/> >>> > <on_poweroff>destroy</on_poweroff> >>> > <on_reboot>restart</on_reboot> >>> > <on_crash>destroy</on_crash> >>> > <devices> >>> > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> >>> > <filesystem type='mount' accessmode='passthrough'> >>> > <source dir='/some/valid/filesystem/location'/> >>> > <target dir='/'/> >>> > </filesystem> >>> > <filesystem type='mount' accessmode='passthrough'> >>> > <source dir='/another/valid/filesystem/location'/> >>> > <target dir='/mnt/android'/> >>> > </filesystem> >>> > <interface type='bridge'> >>> > <mac address='xx:xx:xx:xx:xx:xx'/> >>> > <source bridge='br1'/> >>> > </interface> >>> > <console type='pty'> >>> > <target type='lxc' port='0'/> >>> > </console> >>> > <hostdev mode='capabilities' type='misc'> >>> > <source> >>> > <char>/dev/kvm</char> >>> > </source> >>> > </hostdev> >>> > <hostdev mode='subsystem' type='usb' managed='yes'> >>> > <source> >>> > <vendor id='0x04e8'/> >>> > <product id='0x6860'/> >>> > </source> >>> > </hostdev> >>> > </devices> >>> > </domain> >>> >>> Your config looks fine here. >>> >>> > >>> > Everything worked fine until I added the USB <hostdev> element. I'm >>> > essentially trying to get access to a physical Android device >>> connected to >>> > the host from inside a container. When I go to start the container, I >>> get >>> > an error about Operation not permitted. Here's the relevant bits from >>> > /var/log/libvirt/lxc/machine.log: >>> > >>> > 2014-04-11 22:46:40.491+0000: starting up >>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin >>> > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr >>> /usr/lib/libvirt/libvirt_lxc >>> > --name oshi32134 --console 24 --security=none --handshake 27 >>> --background >>> > --veth vnet1 >>> > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 >>> > 2014-04-11 22:46:40.597+0000: 685: error : >>> > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device >>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >>> > permitted >>> > Unable to create device >>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >>> > permitted >>> >>> Do you have AppArmour enabled on the machine. That seems like the >>> most likely thing that would result in libvirt getting that permission >>> error. >>> >>> Regards, >>> Daniel >>> -- >>> |: http://berrange.com -o- >>> http://www.flickr.com/photos/dberrange/ :| >>> |: http://libvirt.org -o- >>> http://virt-manager.org :| >>> |: http://autobuild.org -o- >>> http://search.cpan.org/~danberr/ :| >>> |: http://entangle-photo.org -o- >>> http://live.gnome.org/gtk-vnc :| >>> >> >> >