libvirt users - Jul 2013 - Re: Libvirt-lxc and systemd question

Warning - I'm fairly new to libvirt, lxc and systemd so there is a good 
chance I'm doing something terribly wrong here.  However, instead of 
continuing to struggle, I figured I would mail the list for some 
advice.  What I'm trying to accomplish is a libvirt-lxc, systemd-based 
container running on my system (Fedora 19).  I've read that sharing the 
underlying OS filesystem with the containers doesn't work, so I've 
installed a minimal Fedora 19 install in /srv/mycontainer.  Everything 
seems to work okay but what I'm struggling with is how to setup the 
initial accounts.  I've tried to attach to the container using
'nsenter'
(entering all the namespaces) but it doesn't appear that the bind mounts 
are in place.  For example, I see the /etc/passwd for my host OS, not 
the container.  Is there a better way to setup the initial accounts on 
the container?

Here is what I have installed:

$ rpm -qa | grep lxc
libvirt-daemon-driver-lxc-1.0.5.2-1.fc19.x86_64
libvirt-daemon-lxc-1.0.5.2-1.fc19.x86_64

$ rpm -qa | grep systemd
systemd-libs-204-9.fc19.x86_64
systemd-python-204-9.fc19.x86_64
systemd-sysv-204-9.fc19.x86_64
systemd-libs-204-9.fc19.i686
systemd-204-9.fc19.x86_64


Here is the scenario I'm trying to go through:

$ export LIBVIRT_DEFAULT_URI=lxc:///
$ getenforce
Enforcing

$ sudo yum -y --releasever=19 --nogpg --installroot=/srv/mycontainer 
--disablerepo='*' --enablerepo=fedora install systemd passwd yum 
fedora-release vim-minimal
... lots of output

$ ls /srv/mycontainer/
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root run  
sbin  srv  sys  tmp  usr  var

$ cat test2.xml
<domain type='lxc'>
   <name>test2</name>
   <memory>102400</memory>
   <os>
     <type arch='x86_64'>exe</type>
     <init>/bin/systemd</init>
   </os>
   <devices>
     <console type='pty'/>
     <filesystem type='mount'>
       <source dir='/srv/mycontainer'/>
       <target dir='/'/>
     </filesystem>
   </devices>
</domain>

$ virsh define test2.xml
Domain test2 defined from test2.xml

$ virsh start test2
Domain test2 started

# Attach to container to set account passwords
$ sudo nsenter -m -u -i -n -p -t `pgrep -f test2`
[sudo] password for mhicks:
[root@localhost /]# diff -q /srv/mycontainer/etc/passwd /etc/passwd
Files /srv/mycontainer/etc/passwd and /etc/passwd differ

Any ideas?

-Matt

On Mon, Jul 22, 2013 at 11:08:07AM -0400, Matt Hicks
wrote:
> Warning - I'm fairly new to libvirt, lxc and systemd so there is a
> good chance I'm doing something terribly wrong here.  However,
> instead of continuing to struggle, I figured I would mail the list
> for some advice.  What I'm trying to accomplish is a libvirt-lxc,
> systemd-based container running on my system (Fedora 19).  I've read
> that sharing the underlying OS filesystem with the containers
> doesn't work, so I've installed a minimal Fedora 19 install in
> /srv/mycontainer.  Everything seems to work okay but what I'm
> struggling with is how to setup the initial accounts.  I've tried to
> attach to the container using 'nsenter' (entering all the
> namespaces) but it doesn't appear that the bind mounts are in place.
> For example, I see the /etc/passwd for my host OS, not the
> container.  Is there a better way to setup the initial accounts on
> the container?
> 
> Here is what I have installed:
> 
> $ rpm -qa | grep lxc
> libvirt-daemon-driver-lxc-1.0.5.2-1.fc19.x86_64
> libvirt-daemon-lxc-1.0.5.2-1.fc19.x86_64
> 
> $ rpm -qa | grep systemd
> systemd-libs-204-9.fc19.x86_64
> systemd-python-204-9.fc19.x86_64
> systemd-sysv-204-9.fc19.x86_64
> systemd-libs-204-9.fc19.i686
> systemd-204-9.fc19.x86_64
> 
> 
> Here is the scenario I'm trying to go through:
> 
> $ export LIBVIRT_DEFAULT_URI=lxc:///
> $ getenforce
> Enforcing
> 
> $ sudo yum -y --releasever=19 --nogpg --installroot=/srv/mycontainer
> --disablerepo='*' --enablerepo=fedora install systemd passwd yum
> fedora-release vim-minimal
> ... lots of output
> 
> $ ls /srv/mycontainer/
> bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root
> run  sbin  srv  sys  tmp  usr  var
> 
> $ cat test2.xml
> <domain type='lxc'>
>   <name>test2</name>
>   <memory>102400</memory>
>   <os>
>     <type arch='x86_64'>exe</type>
>     <init>/bin/systemd</init>
>   </os>
>   <devices>
>     <console type='pty'/>
>     <filesystem type='mount'>
>       <source dir='/srv/mycontainer'/>
>       <target dir='/'/>
>     </filesystem>
>   </devices>
> </domain>
> 
> $ virsh define test2.xml
> Domain test2 defined from test2.xml
> 
> $ virsh start test2
> Domain test2 started
> 
> # Attach to container to set account passwords
> $ sudo nsenter -m -u -i -n -p -t `pgrep -f test2`
> [sudo] password for mhicks:
> [root@localhost /]# diff -q /srv/mycontainer/etc/passwd /etc/passwd
> Files /srv/mycontainer/etc/passwd and /etc/passwd differ
> 
> Any ideas?
Your pgrep is probably selecting the wrong process. You want to attach
to the 'systemd' process, but I think your pgrep will find the
'libvirt_lxc'
process instead.

You shoudn't really use nsenter at all - use

  virsh -c lxc:/// lxc-enter-namespace test2 /bin/sh

and it should "do the right thing" automatically finding the processes
and namespaces.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

On 07/22/2013 11:12 AM, Daniel P. Berrange wrote:
> On Mon, Jul 22, 2013 at 11:08:07AM -0400, Matt Hicks wrote:
>> Warning - I'm fairly new to libvirt, lxc and systemd so there is a
>> good chance I'm doing something terribly wrong here.  However,
>> instead of continuing to struggle, I figured I would mail the list
>> for some advice.  What I'm trying to accomplish is a libvirt-lxc,
>> systemd-based container running on my system (Fedora 19).  I've
read
>> that sharing the underlying OS filesystem with the containers
>> doesn't work, so I've installed a minimal Fedora 19 install in
>> /srv/mycontainer.  Everything seems to work okay but what I'm
>> struggling with is how to setup the initial accounts.  I've tried
to
>> attach to the container using 'nsenter' (entering all the
>> namespaces) but it doesn't appear that the bind mounts are in
place.
>> For example, I see the /etc/passwd for my host OS, not the
>> container.  Is there a better way to setup the initial accounts on
>> the container?
>>
>> Here is what I have installed:
>>
>> $ rpm -qa | grep lxc
>> libvirt-daemon-driver-lxc-1.0.5.2-1.fc19.x86_64
>> libvirt-daemon-lxc-1.0.5.2-1.fc19.x86_64
>>
>> $ rpm -qa | grep systemd
>> systemd-libs-204-9.fc19.x86_64
>> systemd-python-204-9.fc19.x86_64
>> systemd-sysv-204-9.fc19.x86_64
>> systemd-libs-204-9.fc19.i686
>> systemd-204-9.fc19.x86_64
>>
>>
>> Here is the scenario I'm trying to go through:
>>
>> $ export LIBVIRT_DEFAULT_URI=lxc:///
>> $ getenforce
>> Enforcing
>>
>> $ sudo yum -y --releasever=19 --nogpg --installroot=/srv/mycontainer
>> --disablerepo='*' --enablerepo=fedora install systemd passwd
yum
>> fedora-release vim-minimal
>> ... lots of output
>>
>> $ ls /srv/mycontainer/
>> bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root
>> run  sbin  srv  sys  tmp  usr  var
>>
>> $ cat test2.xml
>> <domain type='lxc'>
>>    <name>test2</name>
>>    <memory>102400</memory>
>>    <os>
>>      <type arch='x86_64'>exe</type>
>>      <init>/bin/systemd</init>
>>    </os>
>>    <devices>
>>      <console type='pty'/>
>>      <filesystem type='mount'>
>>        <source dir='/srv/mycontainer'/>
>>        <target dir='/'/>
>>      </filesystem>
>>    </devices>
>> </domain>
>>
>> $ virsh define test2.xml
>> Domain test2 defined from test2.xml
>>
>> $ virsh start test2
>> Domain test2 started
>>
>> # Attach to container to set account passwords
>> $ sudo nsenter -m -u -i -n -p -t `pgrep -f test2`
>> [sudo] password for mhicks:
>> [root@localhost /]# diff -q /srv/mycontainer/etc/passwd /etc/passwd
>> Files /srv/mycontainer/etc/passwd and /etc/passwd differ
>>
>> Any ideas?
> Your pgrep is probably selecting the wrong process. You want to attach
> to the 'systemd' process, but I think your pgrep will find the
'libvirt_lxc'
> process instead.
>
> You shoudn't really use nsenter at all - use
>
>    virsh -c lxc:/// lxc-enter-namespace test2 /bin/sh
>
> and it should "do the right thing" automatically finding the
processes
> and namespaces.
>
> Daniel
Thanks Daniel!

One note, when I first ran that (using sudo), I received the following 
SELinux denials:

type=AVC msg=audit(1374507059.429:625): avc:  denied  { transition } 
for  pid=8600 comm="virsh" path="/usr/bin/bash"
dev="dm-3" ino=1842877
scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 
tcontext=system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 tclass=process

type=SYSCALL msg=audit(1374507059.429:625): arch=x86_64 syscall=execve 
success=no exit=EACCES a0=7f87443a7a30 a1=7f87444287e0 a2=7fff38cd3c40 
a3=8 items=0 ppid=0 pid=8600 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 
egid=0 sgid=0 fsgid=0 ses=1 tty=pts0 comm=virsh exe=/usr/bin/virsh 
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)

However, if I put SELinux in permissive mode, the command works.  Is 
that expected or should I open a bug?

Also, still hitting some issues with the local account setup.  I'm not 
sure if this is related to my minimal install missing some components, 
but when I try and set the passwords on new accounts, I get a generic 
'System error':

sh-4.2# useradd myuser

sh-4.2# passwd myuser
Changing password for user myuser.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: System error

The same goes for switching users:

sh-4.2# su - myuser
su: System error

I've confirmed that an /etc/passwd and /etc/shadow entry exists for that 
user.

Console behavior is the login just fails with 'Incorrect login'.  I 
don't see anything of value in the host or container journal so not 
entirely sure where to look there...

Thanks again for your help

-Matt