CentOS - Mar 2021 - Bare metal vs. virtualization: Proxmox + Ceph + CentOS ?

Hi,

Last week I had a disaster which took me a few unnerving days to repair. My
main Internet-facing server is a bare-metal installation with CentOS 7. It
hosts four dozen web sites (or web applications) based on WordPress, Dolibarr,
OwnCloud, GEPI, and quite a number of mail accounts for ten different domains.
On sunday afternoon this machine had a hardware failure and proved to be
unrecoverable.

The good news is, I always have backups of everything. In that case, I have a
dedicated backup server (in a different datacenter in a different country). I?m
using Rsnapshot for incremental backups, so I had all data: websites, mail
accounts, database dumps, configurations, etc.

Now here?s the problem: it took me three and a half days of intense work to
restore everything and get everything running again. Three and a half days of
downtime is quite a stretch.

As far as I understand, my mistake was to use a bare-metal installation and not
a virtualized solution where I could simply restore a snapshot of a VM. Correct
me if I?m wrong.

Now I?m doing a lot of thinking and searching. Proxmox and Ceph look quite
promising. From what I can tell, the idea is not to use a big server but a
cluster of many small servers, and aggregate them like you would do with hard
disks in a RAID 10 array for example, only you would do this for the whole
system. And then install one or several CentOS 7 VMs on top of this setup.

Any advice from the pros before I dive head first into the  documentation?

Cheers from the sunny South of France,

Niki

-- 
Microlinux - Solutions informatiques durables
7, place de l'?glise - 30730 Montpezat
Site : https://www.microlinux.fr
Blog : https://blog.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32
Mob. : 06 51 80 12 12

Am 14.03.21 um 07:13 schrieb Nicolas Kovacs:
>
> Now here?s the problem: it took me three and a half days of intense work to
> restore everything and get everything running again. Three and a half days
of
> downtime is quite a stretch.
> 
What was the real problem? Why did you need days to restore
from backups? Maybe the new solution is attached here?

--
Leon

How many extra servers can you add to your setup? If I were in your
shoes, I would consider building a file server/NAS with fast
connection to your server(s). Then share the data to your services to
the server (NFS?), export the disk (iscsi) or some combination of
both. I hope someone can correct me but I think postfix has issues
with use accounts in NFS partitions.

Next step is building your web/mail/etc servers -- be them as VMs or
as all in the same baremetal -- as thin as possible so you can recover
that quickly (ansible?), mount data fileshares, and off you go. If you
are going the vm route you could ether save snapshots or build one of
those setups with two servers so in case one goes boink the other
takes over. This is also good for upgrading one of the VM servers: do
them on different days so you can see if there are problems.

If you cannot have more than one server, do run VMs and then put them
in a second set of disks so something happens to boot disk you can
recover.

On Sun, Mar 14, 2021 at 1:13 AM Nicolas Kovacs <info at microlinux.fr>
wrote:
>
> Hi,
>
> Last week I had a disaster which took me a few unnerving days to repair. My
> main Internet-facing server is a bare-metal installation with CentOS 7. It
> hosts four dozen web sites (or web applications) based on WordPress,
Dolibarr,
> OwnCloud, GEPI, and quite a number of mail accounts for ten different
domains.
> On sunday afternoon this machine had a hardware failure and proved to be
> unrecoverable.
>
> The good news is, I always have backups of everything. In that case, I have
a
> dedicated backup server (in a different datacenter in a different country).
I?m
> using Rsnapshot for incremental backups, so I had all data: websites, mail
> accounts, database dumps, configurations, etc.
>
> Now here?s the problem: it took me three and a half days of intense work to
> restore everything and get everything running again. Three and a half days
of
> downtime is quite a stretch.
>
> As far as I understand, my mistake was to use a bare-metal installation and
not
> a virtualized solution where I could simply restore a snapshot of a VM.
Correct
> me if I?m wrong.
>
> Now I?m doing a lot of thinking and searching. Proxmox and Ceph look quite
> promising. From what I can tell, the idea is not to use a big server but a
> cluster of many small servers, and aggregate them like you would do with
hard
> disks in a RAID 10 array for example, only you would do this for the whole
> system. And then install one or several CentOS 7 VMs on top of this setup.
>
> Any advice from the pros before I dive head first into the  documentation?
>
> Cheers from the sunny South of France,
>
> Niki
>
> --
> Microlinux - Solutions informatiques durables
> 7, place de l'?glise - 30730 Montpezat
> Site : https://www.microlinux.fr
> Blog : https://blog.microlinux.fr
> Mail : info at microlinux.fr
> T?l. : 04 66 63 10 32
> Mob. : 06 51 80 12 12
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos