Thierry Parmentelat
2016-Jul-10 11:26 UTC
[libvirt-users] lxc containers won't start in a f24 custom install - odd cgroup fs layout observed
Hi folks I use libvirt to programmatically spawn lxc containers I am facing an issue when migrating from fedora23 to fedora24 I use the stock kernel and libvirt version on both deployments, i.e.: f23: libvirt-1.2.18.3-2.fc23.x86_64 - kernel 4.5.7-202.fc23.x86_64 f24: libvirt-1.3.3.1-4.fc24.x86_64 - kernel 4.6.3-300.fc24.x86_64 First off, I need to outline that the host installation is done through some ad hoc procedure, as all this happens in the context of the centrally-managed PlanetLab global infrastructure I could not reproduce this problem on a host that is installed with the standard fedora install However I could use any clue that would point at a possible reason for this IMHO odd behaviour from libvirt, so that I can narrow down the potential flaws in my host installation procedure At first sight, it looks like the f24 deployment uses a different cgroup fs layout - see below and I suspect this could be the problem, but I am not aware of anything in my install. procedure that could cause that change, and as per this document https://libvirt.org/cgroups.html#systemdScope this change in the naming scheme looks odd - unless the doc is out of date of course So again, any clue is most welcome; many thanks in advance -- Thierry ------ All is fine with f23/libvirt-1.2.18.3-2.fc23.x86_64 However with f24/libvirt-1.3.3.1-4.fc24.x86_64 my lxc containers won't start and I am seeing this message in the libvirtd logfile 2016-07-09 11:52:33.416+0000: 3479: debug : virCgroupValidateMachineGroup:317 : Name 'lxc-3555-inrisl1.libvirt-lxc' for controller 'cpu' does not match 'inri_sl1', 'lxc-3555-inrisl1', 'inri_sl1.libvirt-lxc', '\ machine-lxc\x2dinri_sl1.scope' or 'machine-lxc\x2d3555\x2dinrisl1.scope' 2016-07-09 11:52:33.416+0000: 3479: debug : virCgroupNewDetectMachine:1501 : Failed to validate machine name for 'inri_sl1' driver 'lxc' 2016-07-09 11:52:33.416+0000: 3479: error : virLXCProcessStart:1501 : internal error: No valid cgroup for machine inri_sl1 At first it looked like the issue could be linked to the domain name having an underscore (example below uses inri_sl1 as a domain name) But trying with a regular name 'plain' without an underscore exhibits a similar issue ------------------------- Here's the set of files under /sys/fs/cgroup/memory that contains 'inri' in their name (other susbsytems have similar naming schemes in both cases) ========= f23 / libvirt-1.2.18.3-2.fc23.x86_64 (works fine) [root@vnode06 log]# cat /etc/fedora-release Fedora release 23 (Twenty Three) [root@vnode06 log]# uname -r 4.5.7-202.fc23.x86_64 [root@vnode06 log]# rpm -q libvirt libvirt-1.2.18.3-2.fc23.x86_64 [root@vnode06 log]# find /sys/fs/cgroup/memory/ -name '*inri*' /sys/fs/cgroup/memory/machine.slice/machine-lxc\x2dinri_sl1.scope ========= f24 / libvirt-1.3.3.1-4.fc24.x86_64 (container won't start) [root@vnode05 libvirt]# cat /etc/fedora-release Fedora release 24 (Twenty Four) [root@vnode05 libvirt]# uname -r 4.6.3-300.fc24.x86_64 [root@vnode05 libvirt]# rpm -q libvirt libvirt-1.3.3.1-4.fc24.x86_64 [root@vnode05 libvirt]# find /sys/fs/cgroup/memory/ -name '*inri*' /sys/fs/cgroup/memory/machine/lxc-3237-inrisl1.libvirt-lxc /sys/fs/cgroup/memory/machine/lxc-3555-inrisl1.libvirt-lxc /sys/fs/cgroup/memory/machine/lxc-2989-inrisl1.libvirt-lxc
Thierry Parmentelat
2016-Jul-10 13:02 UTC
Re: [libvirt-users] lxc containers won't start in a f24 custom install - odd cgroup fs layout observed
Answering my own issue: turned out I did not have the ‘systemd-container’ rpm installed in the host, and installing it solved the issue More generally, if like me you build your own fedora image, you might wish to know that the systemd rpm has been split apart in f24, and that it may be helpful to additionnally install explicitly systemd-udev systemd-container — Thierry> On 10 Jul 2016, at 13:26, Thierry Parmentelat <thierry.parmentelat@inria.fr> wrote: > > Hi folks > > I use libvirt to programmatically spawn lxc containers > I am facing an issue when migrating from fedora23 to fedora24 > I use the stock kernel and libvirt version on both deployments, i.e.: > f23: libvirt-1.2.18.3-2.fc23.x86_64 - kernel 4.5.7-202.fc23.x86_64 > f24: libvirt-1.3.3.1-4.fc24.x86_64 - kernel 4.6.3-300.fc24.x86_64 > > > First off, I need to outline that the host installation is done through some ad hoc procedure, > as all this happens in the context of the centrally-managed PlanetLab global infrastructure > > I could not reproduce this problem on a host that is installed with the standard fedora install > However I could use any clue that would point at a possible reason for this IMHO odd behaviour from libvirt, > so that I can narrow down the potential flaws in my host installation procedure > > At first sight, it looks like the f24 deployment uses a different cgroup fs layout - see below > and I suspect this could be the problem, but I am not aware of anything in my install. procedure that > could cause that change, and as per this document > https://libvirt.org/cgroups.html#systemdScope > this change in the naming scheme looks odd - unless the doc is out of date of course > > > So again, any clue is most welcome; many thanks in advance -- Thierry > > ------ > > All is fine with f23/libvirt-1.2.18.3-2.fc23.x86_64 > However with f24/libvirt-1.3.3.1-4.fc24.x86_64 my lxc containers won't start > and I am seeing this message in the libvirtd logfile > > 2016-07-09 11:52:33.416+0000: 3479: debug : virCgroupValidateMachineGroup:317 : Name 'lxc-3555-inrisl1.libvirt-lxc' for controller 'cpu' does not match 'inri_sl1', 'lxc-3555-inrisl1', 'inri_sl1.libvirt-lxc', '\ > machine-lxc\x2dinri_sl1.scope' or 'machine-lxc\x2d3555\x2dinrisl1.scope' > 2016-07-09 11:52:33.416+0000: 3479: debug : virCgroupNewDetectMachine:1501 : Failed to validate machine name for 'inri_sl1' driver 'lxc' > 2016-07-09 11:52:33.416+0000: 3479: error : virLXCProcessStart:1501 : internal error: No valid cgroup for machine inri_sl1 > > At first it looked like the issue could be linked to the domain name having an underscore (example below uses inri_sl1 as a domain name) > But trying with a regular name 'plain' without an underscore exhibits a similar issue > > ------------------------- > Here's the set of files under /sys/fs/cgroup/memory that contains 'inri' in their name > (other susbsytems have similar naming schemes in both cases) > > ========= f23 / libvirt-1.2.18.3-2.fc23.x86_64 (works fine) > > [root@vnode06 log]# cat /etc/fedora-release > Fedora release 23 (Twenty Three) > [root@vnode06 log]# uname -r > 4.5.7-202.fc23.x86_64 > [root@vnode06 log]# rpm -q libvirt > libvirt-1.2.18.3-2.fc23.x86_64 > [root@vnode06 log]# find /sys/fs/cgroup/memory/ -name '*inri*' > /sys/fs/cgroup/memory/machine.slice/machine-lxc\x2dinri_sl1.scope > > ========= f24 / libvirt-1.3.3.1-4.fc24.x86_64 (container won't start) > > [root@vnode05 libvirt]# cat /etc/fedora-release > Fedora release 24 (Twenty Four) > [root@vnode05 libvirt]# uname -r > 4.6.3-300.fc24.x86_64 > [root@vnode05 libvirt]# rpm -q libvirt > libvirt-1.3.3.1-4.fc24.x86_64 > [root@vnode05 libvirt]# find /sys/fs/cgroup/memory/ -name '*inri*' > /sys/fs/cgroup/memory/machine/lxc-3237-inrisl1.libvirt-lxc > /sys/fs/cgroup/memory/machine/lxc-3555-inrisl1.libvirt-lxc > /sys/fs/cgroup/memory/machine/lxc-2989-inrisl1.libvirt-lxc > > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users