libvirt users - Apr 2014 - LXC + USB passthrough = Operation not permitted

Hi!

First post, kind of a noobie. I've been working with LXC and libvirt for a
few months now. Trying to do some interesting things with containers and
Android devices :D

I'm running ubuntu 13.10 with LXC 1.0.1 and tried both libvirt 1.1.1 and
1.2.2 (backported from ubuntu-trusty), but with either version of libvirt
am getting issues as soon as I try to get access to USB devices inside the
container.

Relevant versions of stuff:

$ dpkg -l | grep -i lxc
ii  liblxc1                              1.0.1-0ubuntu1~ubuntu13.10.1~ppa1
i386         Linux Containers userspace tools (library)
ii  lxc                                  1.0.1-0ubuntu1~ubuntu13.10.1~ppa1
i386         Linux Containers userspace tools
ii  lxc-templates                        1.0.1-0ubuntu1~ubuntu13.10.1~ppa1
i386         Linux Containers userspace tools (templates)
ii  python3-lxc                          1.0.1-0ubuntu1~ubuntu13.10.1~ppa1
i386         Linux Containers userspace tools (Python 3.x bindings)

$ dpkg -l | grep libvirt
ii  libvirt-bin                          1.2.2-0ubuntu12
i386         programs for the libvirt library
ii  libvirt0                             1.2.2-0ubuntu12
i386         library for interfacing with different virtualization systems
ii  python-libvirt                       1.2.2-0ubuntu1
 i386         libvirt Python bindings

Here's my entire domain definition:

<domain type='lxc'>
  <name>oshi32134</name>
  <uuid>xxxxx</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='i686'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/some/valid/filesystem/location'/>
      <target dir='/'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/another/valid/filesystem/location'/>
      <target dir='/mnt/android'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='xx:xx:xx:xx:xx:xx'/>
      <source bridge='br1'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
    <hostdev mode='capabilities' type='misc'>
      <source>
        <char>/dev/kvm</char>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb'
managed='yes'>
      <source>
        <vendor id='0x04e8'/>
        <product id='0x6860'/>
  </source>
    </hostdev>
  </devices>
</domain>

Everything worked fine until I added the USB <hostdev> element. I'm
essentially trying to get access to a physical Android device connected to
the host from inside a container. When I go to start the container, I get
an error about Operation not permitted. Here's the relevant bits from
/var/log/libvirt/lxc/machine.log:

2014-04-11 22:46:40.491+0000: starting up
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc
--name oshi32134 --console 24 --security=none --handshake 27 --background
--veth vnet1
2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2
2014-04-11 22:46:40.597+0000: 685: error :
virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device
//var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
permitted
Unable to create device
//var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
permitted

I get the same output above when I use libvirt 1.1.1.

Tracing down through the libvirt code it looks like it fails when calling
`mknod`. It's clear that libvirt converts the product/vendor ids I
specified in the domain definition file to the appropriate USB bus
(/dev/bus/usb/002/003, verified by comparing against output of
`usb-devices`).

In a parallel effort, a colleague of mine set up a vanilla LXContainer on a
new install of the same version of ubuntu, and successfully was able to
communicate with the Android device from a container. The only
configuration tweaks he made was make sure to whitelist the cgroup with the
appropriate major/minor device number (associated with the Android device)
inside the LXC configuration file. From there the container was able to
appropriately create the device file as you attach/detach the Android
device.

Hopefully someone can shed some light, and appreciate your patience with me
as I am learning a lot of this stuff as I go :)

Thanks in advance for any help!
Fil

On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj
wrote:
> Hi!
> 
> First post, kind of a noobie. I've been working with LXC and libvirt
for a
> few months now. Trying to do some interesting things with containers and
> Android devices :D
> Here's my entire domain definition:
> 
> <domain type='lxc'>
>   <name>oshi32134</name>
>   <uuid>xxxxx</uuid>
>   <memory unit='KiB'>3145728</memory>
>   <currentMemory unit='KiB'>3145728</currentMemory>
>   <vcpu placement='static'>1</vcpu>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='i686'>exe</type>
>     <init>/sbin/init</init>
>   </os>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <devices>
>     <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
>     <filesystem type='mount'
accessmode='passthrough'>
>       <source dir='/some/valid/filesystem/location'/>
>       <target dir='/'/>
>     </filesystem>
>     <filesystem type='mount'
accessmode='passthrough'>
>       <source dir='/another/valid/filesystem/location'/>
>       <target dir='/mnt/android'/>
>     </filesystem>
>     <interface type='bridge'>
>       <mac address='xx:xx:xx:xx:xx:xx'/>
>       <source bridge='br1'/>
>     </interface>
>     <console type='pty'>
>       <target type='lxc' port='0'/>
>     </console>
>     <hostdev mode='capabilities' type='misc'>
>       <source>
>         <char>/dev/kvm</char>
>       </source>
>     </hostdev>
>     <hostdev mode='subsystem' type='usb'
managed='yes'>
>       <source>
>         <vendor id='0x04e8'/>
>         <product id='0x6860'/>
>   </source>
>     </hostdev>
>   </devices>
> </domain>
Your config looks fine here.
> 
> Everything worked fine until I added the USB <hostdev> element.
I'm
> essentially trying to get access to a physical Android device connected to
> the host from inside a container. When I go to start the container, I get
> an error about Operation not permitted. Here's the relevant bits from
> /var/log/libvirt/lxc/machine.log:
> 
> 2014-04-11 22:46:40.491+0000: starting up
> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc
> --name oshi32134 --console 24 --security=none --handshake 27 --background
> --veth vnet1
> 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2
> 2014-04-11 22:46:40.597+0000: 685: error :
> virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device
> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
> permitted
> Unable to create device
> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
> permitted
Do you have AppArmour enabled on the machine. That seems like the
most likely thing that would result in libvirt getting that permission
error.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Yeah, AppArmor is enabled, but I put everything (that I could find) into
complain mode:

$ sudo apparmor_status
apparmor module is loaded.
12 profiles are loaded.
3 profiles are in enforce mode.
   lxc-container-default
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
9 profiles are in complain mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/libvirt/virt-aa-helper
   /usr/sbin/libvirtd
   /usr/sbin/ntpd
   /usr/sbin/rsyslogd
   /usr/sbin/tcpdump
3 processes have profiles defined.
0 processes are in enforce mode.
2 processes are in complain mode.
   /usr/sbin/libvirtd (30419)
   /usr/sbin/ntpd (3418)
1 processes are unconfined but have a profile defined.
   /usr/sbin/rsyslogd (626)

And still get issues. From libvirtd.log:

2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2
2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 :
Cannot recv data: Connection reset by peer
2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 :
internal error: guest failed to start: Unable to create device
//var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
permitted

2014-04-16 22:19:10.964+0000: 30420: warning :
virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in
list of active USB devices

Thanks in advance for any help, Daniel!

Cheers,
Fil


On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange
<berrange@redhat.com>wrote:
> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
> > Hi!
> >
> > First post, kind of a noobie. I've been working with LXC and
libvirt for
> a
> > few months now. Trying to do some interesting things with containers
and
> > Android devices :D
> > Here's my entire domain definition:
> >
> > <domain type='lxc'>
> >   <name>oshi32134</name>
> >   <uuid>xxxxx</uuid>
> >   <memory unit='KiB'>3145728</memory>
> >   <currentMemory
unit='KiB'>3145728</currentMemory>
> >   <vcpu placement='static'>1</vcpu>
> >   <resource>
> >     <partition>/machine</partition>
> >   </resource>
> >   <os>
> >     <type arch='i686'>exe</type>
> >     <init>/sbin/init</init>
> >   </os>
> >   <clock offset='utc'/>
> >   <on_poweroff>destroy</on_poweroff>
> >   <on_reboot>restart</on_reboot>
> >   <on_crash>destroy</on_crash>
> >   <devices>
> >     <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
> >     <filesystem type='mount'
accessmode='passthrough'>
> >       <source dir='/some/valid/filesystem/location'/>
> >       <target dir='/'/>
> >     </filesystem>
> >     <filesystem type='mount'
accessmode='passthrough'>
> >       <source dir='/another/valid/filesystem/location'/>
> >       <target dir='/mnt/android'/>
> >     </filesystem>
> >     <interface type='bridge'>
> >       <mac address='xx:xx:xx:xx:xx:xx'/>
> >       <source bridge='br1'/>
> >     </interface>
> >     <console type='pty'>
> >       <target type='lxc' port='0'/>
> >     </console>
> >     <hostdev mode='capabilities' type='misc'>
> >       <source>
> >         <char>/dev/kvm</char>
> >       </source>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='usb'
managed='yes'>
> >       <source>
> >         <vendor id='0x04e8'/>
> >         <product id='0x6860'/>
> >   </source>
> >     </hostdev>
> >   </devices>
> > </domain>
>
> Your config looks fine here.
>
> >
> > Everything worked fine until I added the USB <hostdev> element.
I'm
> > essentially trying to get access to a physical Android device
connected
> to
> > the host from inside a container. When I go to start the container, I
get
> > an error about Operation not permitted. Here's the relevant bits
from
> > /var/log/libvirt/lxc/machine.log:
> >
> > 2014-04-11 22:46:40.491+0000: starting up
> > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr
/usr/lib/libvirt/libvirt_lxc
> > --name oshi32134 --console 24 --security=none --handshake 27
--background
> > --veth vnet1
> > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2
> > 2014-04-11 22:46:40.597+0000: 685: error :
> > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device
> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
> > permitted
> > Unable to create device
> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not
> > permitted
>
> Do you have AppArmour enabled on the machine. That seems like the
> most likely thing that would result in libvirt getting that permission
> error.
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-   
http://www.flickr.com/photos/dberrange/:|
> |: http://libvirt.org              -o-            
http://virt-manager.org:|
> |: http://autobuild.org       -o-        
http://search.cpan.org/~danberr/:|
> |: http://entangle-photo.org       -o-      
http://live.gnome.org/gtk-vnc:|
>