Hello! Okay i tried again with only staticly linked busybox: offlinehacker:~/ $ /home/offlinehacker/busybox/busybox BusyBox v1.17.1 (Debian 1:1.17.1-8) multi-call binary. Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko and others. Licensed under GPLv2. See source distribution for full notice. .... Again my id: uid=499(offlinehacker) gid=100(users) groups=100(users),1(wheel),57(networkmanager) My rootfs tree(/home/offlinehacker/busybox): busybox ├── [offlineh users ] busybox └── [offlineh users ] busybox-static_1.17.1-8_amd64.deb It works just fine as root and these folders gets created: busybox ├── [offlineh users ] busybox ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb ├── [root root ] dev ├── [root root ] .oldroot ├── [root root ] proc └── [root root ] sys When i start it with idmap with clean rootfs(dev proc sys and .oldroot deleted) i get this error, and it is a little bit different now: error: Failed to create domain from helloworld.xml error: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC And log is pretty similar: sep 06 11:24:56 laptop libvirtd[1542]: EVENT_POLL_UPDATE_HANDLE: watch=241 events=1 sep 06 11:24:57 laptop libvirtd[1542]: Skip interrupt, 1 140499747788544 sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 sep 06 11:24:57 laptop libvirtd[1542]: server=0x7fc8a60ddd60 client=0x7fc8a60e8bb0 msg=0x7fc8a60e6970 rerr=0x7fc89a32cd40 args=0x7fc8880160a0 ret=0x7fc888016030 sep 06 11:24:57 laptop libvirtd[1542]: priv=0x7fc8a60ea3a0 conn=(nil) sep 06 11:24:57 laptop libvirtd[1542]: name=lxc:/// sep 06 11:24:57 laptop libvirtd[1542]: Cannot recv data: Connection reset by peer sep 06 11:24:57 laptop libvirtd[1542]: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC Rootfs after failed creation looks like this: busybox ├── [offlineh users ] busybox ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb ├── [offlineh users ] .oldroot ├── [offlineh users ] proc └── [offlineh users ] sys I have debugging enabled, at least LIBVIRT_DEBUG is set to 1 and i get much more messages. If there's any my granular debug please let me know. PS: I forgot to mention my version of libvirt is 1.1.2 Thanks, Jaka! On Fri, Sep 6, 2013 at 3:41 AM, Gao feng <gaofeng@cn.fujitsu.com> wrote:> On 09/06/2013 03:15 AM, Jaka Hudoklin wrote: > > Hello! > > > > I'm testing user namespaces and I have quite some problem getting them > to work. > > > > First of all, I have user namespaces support enabled in kernel: > > > > offlinehacker:~/ $ uname -r > > 3.10.10 > > offlinehacker:~/ $ ls /proc/self/ns/ > > ipc@ mnt@ net@ pid@ user@ uts@ > > > > I created simple ubuntu rootfs and when I start container without idmap, > so without user namespace mappings, it works just fine: > > > > Libivrt config: > > > > <domain type='lxc'> > > <name>helloworld</name> > > <memory>102400</memory> > > <os> > > <type>exe</type> > > <init>/bin/dash</init> > > </os> > > <!--<idmap> > > <uid start='0' target='499' count='10'/> > > <gid start='0' target='100' count='10'/> > > </idmap>--> > > <devices> > > <console type='pty'/> > > <filesystem type='mount'> > > <source dir='/home/offlinehacker/rootfs'/> > > <target dir='/'/> > > </filesystem> > > </devices> > > </domain> > > > > Your configuration looks good. > > > This is how my rootfs looks like: > > > > offlinehacker:~/ $ ls -la rootfs > > drwxr-xr-x 23 offlinehacker users 4096 sep 5 19:06 ./ > > drwxr-xr-x 59 offlinehacker users 4096 sep 5 19:06 ../ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 bin/ > > drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 boot/ > > drwxr-xr-x 4 offlinehacker users 4096 avg 27 14:11 dev/ > > drwxr-xr-x 86 offlinehacker users 4096 sep 5 18:20 etc/ > > drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 home/ > > lrwxrwxrwx 1 offlinehacker users 33 avg 27 14:10 initrd.img > -> /boot/initrd.img-3.2.0-52-virtual > > drwxr-xr-x 18 offlinehacker users 4096 avg 27 14:10 lib/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:10 lib64/ > > drwx------ 2 offlinehacker users 4096 avg 27 14:11 lost+found/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 media/ > > drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 mnt/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 opt/ > > -rw-r--r-- 1 offlinehacker vboxusers 231671365 avg 27 14:12 > precise-server-cloudimg-amd64-root.tar.gz > > drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 proc/ > > drwx------ 2 offlinehacker users 4096 sep 3 23:47 root/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 run/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 sbin/ > > drwxr-xr-x 2 offlinehacker users 4096 mar 5 2012 selinux/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 srv/ > > drwxr-xr-x 2 offlinehacker users 4096 apr 14 2012 sys/ > > drwxrwxrwt 2 offlinehacker users 4096 sep 4 01:39 tmp/ > > drwxr-xr-x 10 offlinehacker users 4096 avg 27 14:09 usr/ > > drwxr-xr-x 12 offlinehacker users 4096 sep 5 18:10 var/ > > lrwxrwxrwx 1 offlinehacker users 29 avg 27 14:10 vmlinuz -> > boot/vmlinuz-3.2.0-52-virtual > > > > And this is who I am: > > offlinehacker:~/ $ id > > uid=499(offlinehacker) gid=67(libvirtd) > groups=100(users),1(wheel),57(networkmanager),67(libvirtd) > > Can this user exec //home/offlinehacker/rootfs/bin/dash successfully ? > > > > > When I create the container with idmap uncommented I get the following > error: > > > > offlinehacker:~/ $ virsh -c lxc:/// create helloworld.xml > > error: Failed to create domain from helloworld.xml > > error: internal error: guest failed to start: 2013-09-05 > 19:08:57.781+0000: 19036: debug > > > > And these are the logs: > > sep 05 19:08:52 laptop libvirtd[1542]: server=0x7fc8a60ddd60 > client=0x7fc8a60e68d0 msg=0x7fc8a60e9380 rerr=0x7fc89a32cd40 > args=0x7fc88800b4a0 ret=0x7fc88800a1c0 > > sep 05 19:08:52 laptop libvirtd[1542]: priv=0x7fc8a60e91f0 conn=(nil) > > sep 05 19:08:52 laptop libvirtd[1542]: name=lxc:/// > > sep 05 19:08:57 laptop libvirtd[1542]: Cannot recv data: Connection > reset by peer > > sep 05 19:08:58 laptop libvirtd[1542]: internal error: guest failed to > start: 2013-09-05 19:08:57.781+0000: 19036: debug > > > > Looks like .oldroot, dev, proc and sys gets created with mapped > permissions(499:100), but container fails to start. > > > > Please enable debug mode, I need more information. > > http://libvirt.org/logging.html > > Thansk >
And if needed, i can recompile from source and place custom debug handlers, but i need some pointers where it could have go wrong. Thanks, Jaka! On Fri, Sep 6, 2013 at 1:32 PM, Jaka Hudoklin <jakahudoklin@gmail.com>wrote:> Hello! > > Okay i tried again with only staticly linked busybox: > offlinehacker:~/ $ /home/offlinehacker/busybox/busybox > BusyBox v1.17.1 (Debian 1:1.17.1-8) multi-call binary. > Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko > and others. Licensed under GPLv2. > See source distribution for full notice. > .... > > Again my id: > uid=499(offlinehacker) gid=100(users) > groups=100(users),1(wheel),57(networkmanager) > > My rootfs tree(/home/offlinehacker/busybox): > busybox > ├── [offlineh users ] busybox > └── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > > It works just fine as root and these folders gets created: > busybox > ├── [offlineh users ] busybox > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > ├── [root root ] dev > ├── [root root ] .oldroot > ├── [root root ] proc > └── [root root ] sys > > When i start it with idmap with clean rootfs(dev proc sys and .oldroot > deleted) i get this error, and it is a little bit different now: > error: Failed to create domain from helloworld.xml > error: internal error: guest failed to start: 2013-09-06 > 11:24:57.088+0000: 5794: debug : virFileC > > And log is pretty similar: > sep 06 11:24:56 laptop libvirtd[1542]: EVENT_POLL_UPDATE_HANDLE: watch=241 > events=1 > sep 06 11:24:57 laptop libvirtd[1542]: Skip interrupt, 1 140499747788544 > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > sep 06 11:24:57 laptop libvirtd[1542]: server=0x7fc8a60ddd60 > client=0x7fc8a60e8bb0 msg=0x7fc8a60e6970 rerr=0x7fc89a32cd40 > args=0x7fc8880160a0 ret=0x7fc888016030 > sep 06 11:24:57 laptop libvirtd[1542]: priv=0x7fc8a60ea3a0 conn=(nil) > sep 06 11:24:57 laptop libvirtd[1542]: name=lxc:/// > sep 06 11:24:57 laptop libvirtd[1542]: Cannot recv data: Connection reset > by peer > sep 06 11:24:57 laptop libvirtd[1542]: internal error: guest failed to > start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC > > Rootfs after failed creation looks like this: > busybox > ├── [offlineh users ] busybox > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > ├── [offlineh users ] .oldroot > ├── [offlineh users ] proc > └── [offlineh users ] sys > > I have debugging enabled, at least LIBVIRT_DEBUG is set to 1 and i get > much more messages. If there's any my granular debug please let me know. > > PS: I forgot to mention my version of libvirt is 1.1.2 > > Thanks, Jaka! > > > On Fri, Sep 6, 2013 at 3:41 AM, Gao feng <gaofeng@cn.fujitsu.com> wrote: > >> On 09/06/2013 03:15 AM, Jaka Hudoklin wrote: >> > Hello! >> > >> > I'm testing user namespaces and I have quite some problem getting them >> to work. >> > >> > First of all, I have user namespaces support enabled in kernel: >> > >> > offlinehacker:~/ $ uname -r >> > 3.10.10 >> > offlinehacker:~/ $ ls /proc/self/ns/ >> > ipc@ mnt@ net@ pid@ user@ uts@ >> > >> > I created simple ubuntu rootfs and when I start container without >> idmap, so without user namespace mappings, it works just fine: >> > >> > Libivrt config: >> > >> > <domain type='lxc'> >> > <name>helloworld</name> >> > <memory>102400</memory> >> > <os> >> > <type>exe</type> >> > <init>/bin/dash</init> >> > </os> >> > <!--<idmap> >> > <uid start='0' target='499' count='10'/> >> > <gid start='0' target='100' count='10'/> >> > </idmap>--> >> > <devices> >> > <console type='pty'/> >> > <filesystem type='mount'> >> > <source dir='/home/offlinehacker/rootfs'/> >> > <target dir='/'/> >> > </filesystem> >> > </devices> >> > </domain> >> > >> >> Your configuration looks good. >> >> > This is how my rootfs looks like: >> > >> > offlinehacker:~/ $ ls -la rootfs >> > drwxr-xr-x 23 offlinehacker users 4096 sep 5 19:06 ./ >> > drwxr-xr-x 59 offlinehacker users 4096 sep 5 19:06 ../ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 bin/ >> > drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 boot/ >> > drwxr-xr-x 4 offlinehacker users 4096 avg 27 14:11 dev/ >> > drwxr-xr-x 86 offlinehacker users 4096 sep 5 18:20 etc/ >> > drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 home/ >> > lrwxrwxrwx 1 offlinehacker users 33 avg 27 14:10 initrd.img >> -> /boot/initrd.img-3.2.0-52-virtual >> > drwxr-xr-x 18 offlinehacker users 4096 avg 27 14:10 lib/ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:10 lib64/ >> > drwx------ 2 offlinehacker users 4096 avg 27 14:11 lost+found/ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 media/ >> > drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 mnt/ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 opt/ >> > -rw-r--r-- 1 offlinehacker vboxusers 231671365 avg 27 14:12 >> precise-server-cloudimg-amd64-root.tar.gz >> > drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 proc/ >> > drwx------ 2 offlinehacker users 4096 sep 3 23:47 root/ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 run/ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 sbin/ >> > drwxr-xr-x 2 offlinehacker users 4096 mar 5 2012 selinux/ >> > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 srv/ >> > drwxr-xr-x 2 offlinehacker users 4096 apr 14 2012 sys/ >> > drwxrwxrwt 2 offlinehacker users 4096 sep 4 01:39 tmp/ >> > drwxr-xr-x 10 offlinehacker users 4096 avg 27 14:09 usr/ >> > drwxr-xr-x 12 offlinehacker users 4096 sep 5 18:10 var/ >> > lrwxrwxrwx 1 offlinehacker users 29 avg 27 14:10 vmlinuz -> >> boot/vmlinuz-3.2.0-52-virtual >> > >> > And this is who I am: >> > offlinehacker:~/ $ id >> > uid=499(offlinehacker) gid=67(libvirtd) >> groups=100(users),1(wheel),57(networkmanager),67(libvirtd) >> >> Can this user exec //home/offlinehacker/rootfs/bin/dash successfully ? >> >> > >> > When I create the container with idmap uncommented I get the following >> error: >> > >> > offlinehacker:~/ $ virsh -c lxc:/// create helloworld.xml >> > error: Failed to create domain from helloworld.xml >> > error: internal error: guest failed to start: 2013-09-05 >> 19:08:57.781+0000: 19036: debug >> > >> > And these are the logs: >> > sep 05 19:08:52 laptop libvirtd[1542]: server=0x7fc8a60ddd60 >> client=0x7fc8a60e68d0 msg=0x7fc8a60e9380 rerr=0x7fc89a32cd40 >> args=0x7fc88800b4a0 ret=0x7fc88800a1c0 >> > sep 05 19:08:52 laptop libvirtd[1542]: priv=0x7fc8a60e91f0 conn=(nil) >> > sep 05 19:08:52 laptop libvirtd[1542]: name=lxc:/// >> > sep 05 19:08:57 laptop libvirtd[1542]: Cannot recv data: Connection >> reset by peer >> > sep 05 19:08:58 laptop libvirtd[1542]: internal error: guest failed to >> start: 2013-09-05 19:08:57.781+0000: 19036: debug >> > >> > Looks like .oldroot, dev, proc and sys gets created with mapped >> permissions(499:100), but container fails to start. >> > >> >> Please enable debug mode, I need more information. >> >> http://libvirt.org/logging.html >> >> Thansk >> > >
On 09/06/2013 07:32 PM, Jaka Hudoklin wrote:> Hello! > > Okay i tried again with only staticly linked busybox: > offlinehacker:~/ $ /home/offlinehacker/busybox/busybox > BusyBox v1.17.1 (Debian 1:1.17.1-8) multi-call binary. > Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko > and others. Licensed under GPLv2. > See source distribution for full notice. > .... > > Again my id: > uid=499(offlinehacker) gid=100(users) groups=100(users),1(wheel),57(networkmanager) > > My rootfs tree(/home/offlinehacker/busybox): > busybox > ├── [offlineh users ] busybox > └── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > > It works just fine as root and these folders gets created: > busybox > ├── [offlineh users ] busybox > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > ├── [root root ] dev > ├── [root root ] .oldroot > ├── [root root ] proc > └── [root root ] sys > > When i start it with idmap with clean rootfs(dev proc sys and .oldroot deleted) i get this error, and it is a little bit different now: > error: Failed to create domain from helloworld.xml > error: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC > > And log is pretty similar: > sep 06 11:24:56 laptop libvirtd[1542]: EVENT_POLL_UPDATE_HANDLE: watch=241 events=1 > sep 06 11:24:57 laptop libvirtd[1542]: Skip interrupt, 1 140499747788544 > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > sep 06 11:24:57 laptop libvirtd[1542]: server=0x7fc8a60ddd60 client=0x7fc8a60e8bb0 msg=0x7fc8a60e6970 rerr=0x7fc89a32cd40 args=0x7fc8880160a0 ret=0x7fc888016030 > sep 06 11:24:57 laptop libvirtd[1542]: priv=0x7fc8a60ea3a0 conn=(nil) > sep 06 11:24:57 laptop libvirtd[1542]: name=lxc:/// > sep 06 11:24:57 laptop libvirtd[1542]: Cannot recv data: Connection reset by peer > sep 06 11:24:57 laptop libvirtd[1542]: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC > > Rootfs after failed creation looks like this: > busybox > ├── [offlineh users ] busybox > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > ├── [offlineh users ] .oldroot > ├── [offlineh users ] proc > └── [offlineh users ] sys > > I have debugging enabled, at least LIBVIRT_DEBUG is set to 1 and i get much more messages. If there's any my granular debug please let me know. > > PS: I forgot to mention my version of libvirt is 1.1.2 >OK, I get it, Maybe you need this patch 1583dfda7c4e5ad71efe0615c06e5676528d8203 LXC: Don't mount securityfs when user namespace enabled Thanks
I applied your patch, but no success. What bothers me is that connection gets reseted. By the way, i'm using systemd, with process started in forking mode and as daemon. Could this cause any problems? This is my libvirtd.conf, if it helps anything: unix_sock_group = "libvirtd" unix_sock_rw_perms = "0770" auth_unix_ro = "none" auth_unix_rw = "none" Can you please tell me easy to setup distro with systemd that user namespaces work, so i can compare. Thanks! On Mon, Sep 9, 2013 at 3:08 AM, Gao feng <gaofeng@cn.fujitsu.com> wrote:> On 09/06/2013 07:32 PM, Jaka Hudoklin wrote: > > Hello! > > > > Okay i tried again with only staticly linked busybox: > > offlinehacker:~/ $ /home/offlinehacker/busybox/busybox > > BusyBox v1.17.1 (Debian 1:1.17.1-8) multi-call binary. > > Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko > > and others. Licensed under GPLv2. > > See source distribution for full notice. > > .... > > > > Again my id: > > uid=499(offlinehacker) gid=100(users) > groups=100(users),1(wheel),57(networkmanager) > > > > My rootfs tree(/home/offlinehacker/busybox): > > busybox > > ├── [offlineh users ] busybox > > └── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > > > > It works just fine as root and these folders gets created: > > busybox > > ├── [offlineh users ] busybox > > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > > ├── [root root ] dev > > ├── [root root ] .oldroot > > ├── [root root ] proc > > └── [root root ] sys > > > > When i start it with idmap with clean rootfs(dev proc sys and .oldroot > deleted) i get this error, and it is a little bit different now: > > error: Failed to create domain from helloworld.xml > > error: internal error: guest failed to start: 2013-09-06 > 11:24:57.088+0000: 5794: debug : virFileC > > > > And log is pretty similar: > > sep 06 11:24:56 laptop libvirtd[1542]: EVENT_POLL_UPDATE_HANDLE: > watch=241 events=1 > > sep 06 11:24:57 laptop libvirtd[1542]: Skip interrupt, 1 140499747788544 > > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > > sep 06 11:24:57 laptop libvirtd[1542]: server=0x7fc8a60ddd60 > client=0x7fc8a60e8bb0 msg=0x7fc8a60e6970 rerr=0x7fc89a32cd40 > args=0x7fc8880160a0 ret=0x7fc888016030 > > sep 06 11:24:57 laptop libvirtd[1542]: priv=0x7fc8a60ea3a0 conn=(nil) > > sep 06 11:24:57 laptop libvirtd[1542]: name=lxc:/// > > sep 06 11:24:57 laptop libvirtd[1542]: Cannot recv data: Connection > reset by peer > > sep 06 11:24:57 laptop libvirtd[1542]: internal error: guest failed to > start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC > > > > Rootfs after failed creation looks like this: > > busybox > > ├── [offlineh users ] busybox > > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > > ├── [offlineh users ] .oldroot > > ├── [offlineh users ] proc > > └── [offlineh users ] sys > > > > I have debugging enabled, at least LIBVIRT_DEBUG is set to 1 and i get > much more messages. If there's any my granular debug please let me know. > > > > PS: I forgot to mention my version of libvirt is 1.1.2 > > > > OK, I get it, Maybe you need this patch > > 1583dfda7c4e5ad71efe0615c06e5676528d8203 > LXC: Don't mount securityfs when user namespace enabled > > Thanks >