CentOS virt - Jan 2021 - OS-level virtualization using LXC and systemd-nspawn containers

On 26.01.2021 18:41, Scott Dowdle wrote:
> Have you tried LXD?
Not yet. My first post on this mailing list
asked if anyone was using LXC in production:

Does anyone use LXC and/or systemd-nspawn
containers on RHEL 8 / CentOS 8 for production?

What are advantages and disadvantages of each of these technologies?

Can you share your experience with LXC and/or systemd-nspawn
for RHEL 8 / CentOS 8 operating system on the hardware node?

===========================================================
>> podman is replacement for Docker,
>> it is not replacement for OpenVZ 6 containers.
> Docker definitely targets "Application Containers"... with one
service per container.  podman says they can also do "System
Containers" by running systemd as the entry point.  Of course the vast
majority of pre-made container images you'll find in container image
repositories aren't built for that, but you can use distro provided images
and build a system container image out of them.  I have a simple recipe for
Fedora, CentOS, and Ubuntu.  I don't know how many people are using podman
in this capacity yet, and I don't know if it is mature or not for
production... but the limited testing I've done with it, has worked out
fairly well... using Fedora or CentOS Stream 8 as the host OS...
No problem, systemd-nspawn also has worked out fairly well, without
extra complexity, introduced by podman "System Containers" images.
> Yes, podman does still use it's own private network addressing, but I
guess that can be overcome by telling it to use the host network.  I haven't
tried that.  Not exactly like OpenVZ's container networking for sure.
I can't use host network for [system] containers.
Each container must have its own private network.
>> I have containers with 1.6 TiB of valuable data - podman
>> not designed to work in this mode and in such conditions.
> Persistent data really isn't an issue.  You just have to understand how
it works.  Plenty of people run long-term / persistent-data Docker and podman
containers...
Backuping persistent containers and restoring from backup - issue.
I don't want have deal with a mash of different images and layers.

Each my systemd-nspawn container located in separate filesystem:

# zfs list
NAME                  USED  AVAIL     REFER  MOUNTPOINT
tank                  531G  1.13T       96K  /tank
tank/containers       528G  1.13T      168K  /tank/containers
tank/containers/1    19.1G  1.13T     8.00G  /tank/containers/1
tank/containers/100  7.59G  1.13T     6.59G  /tank/containers/100
tank/containers/111   169G  1.13T     27.6G  /tank/containers/111
tank/containers/120  3.05G  1.13T     1.31G  /tank/containers/120
tank/containers/121  10.2G  1.13T     9.20G  /tank/containers/121
tank/containers/122  8.80G  1.13T     7.23G  /tank/containers/122
tank/containers/124  3.20G  1.13T     2.21G  /tank/containers/124
tank/containers/125  3.08G  1.13T     2.12G  /tank/containers/125
tank/containers/126  87.1G  1.13T     64.1G  /tank/containers/126
tank/containers/127   145G  1.13T      125G  /tank/containers/127
tank/containers/128  7.46G  1.13T     5.62G  /tank/containers/128
tank/containers/129  6.04G  1.13T     3.92G  /tank/containers/129
tank/containers/130  5.03G  1.13T     3.01G  /tank/containers/130
tank/containers/131  6.41G  1.13T     2.94G  /tank/containers/131
tank/containers/132  4.55G  1.13T     2.98G  /tank/containers/132
tank/containers/133  22.7G  1.13T     20.6G  /tank/containers/133
tank/containers/134  3.36G  1.13T     1.61G  /tank/containers/134
tank/containers/135  3.82G  1.13T     1.73G  /tank/containers/135
tank/containers/25   1.74G  1.13T      960M  /tank/containers/25
tank/containers/30   2.15G  1.13T     1.35G  /tank/containers/30
tank/containers/97   5.90G  1.13T     2.06G  /tank/containers/97
tank/containers/99   3.15G  1.13T     2.20G  /tank/containers/99

Each filesystem has many snapshots (24 hourly and 30 daily),
which are replicated to backup server, without the need to stop
each systemd-nspawn container for creating snapshot/backup of it.
>> So I have only two alternatives for OS-level virtualization:
>> LXC or systemd-nspawn.
> If CentOS is your target host, I'd guess that neither of those really
are a good solutions... simply because they aren't supported and upstream
doesn't care about anything other than podman for containers.
Upstream also doesn't support ZFS, but this is extraordinary file system
with excellent feature set.
> LXC varies from one distro to the next... with different kernels, and
different versions of libraries and management scripts.  Again, LXD on an Ubuntu
LTS host is probably the most stable... with Proxmox VE as a close second.  Both
of those upstreams care about system containers and put in a lot of effort to
make it work.
LXC/LXD for CentOS 8 and other Linux distros distributed
in the form of snap package. Inside snap - ordinary Ubuntu.
Google "Install LXC CentOS 8" for more details about this.
> Good luck.
Thank you.

Luck is need for me to find solutions of these bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1913734

https://bugzilla.redhat.com/show_bug.cgi?id=1913806

-- 
Best regards,
  Gena

Greetings,

----- Original Message -----
> Can you share your experience with LXC and/or systemd-nspawn
> for RHEL 8 / CentOS 8 operating system on the hardware node?
> I can't use host network for [system] containers.
> Each container must have its own private network.
In that case, perhaps you'd like docker/podman's private networking?
> Backuping persistent containers and restoring from backup - issue.
> I don't want have deal with a mash of different images and layers.
I haven't thought of backups.  I assume there are a number of backup
solutions for docker/podman containers but I'm completely ignorant.
> Each my systemd-nspawn container located in separate filesystem:
> 
> # zfs list
> NAME                  USED  AVAIL     REFER  MOUNTPOINT
> tank                  531G  1.13T       96K  /tank
> tank/containers       528G  1.13T      168K  /tank/containers
> tank/containers/1    19.1G  1.13T     8.00G  /tank/containers/1
Ok, so you are turning off SELinux and using ZFS too?  And you still want to
stay with EL?  Why?
> Upstream also doesn't support ZFS, but this is extraordinary file
system
> with excellent feature set.
Ubuntu and LXD do support ZFS and Canonical's lawyers seem happy to allow
ZFS to be bundled with Ubuntu by default.  You should get along nicely.

TYL,
-- 
Scott Dowdle
704 Church Street
Belgrade, MT 59714
(406)388-0827 [home]
(406)994-3931 [work]

26.01.2021 22:09, Gena Makhomed ?????:
> On 26.01.2021 18:41, Scott Dowdle wrote:
>
>> Have you tried LXD?
>
> Not yet. My first post on this mailing list
> asked if anyone was using LXC in production:
>
> Does anyone use LXC and/or systemd-nspawn
> containers on RHEL 8 / CentOS 8 for production?
>
>
Well, I use lxc on oracle linux 8 ( just because centos 8 is killed by 
redhat) in production, I just need to run yet another asterisk instance 
with different network settings on the same host.

Never tried systemd-nspawn, though.