Gluster users - Dec 2023 - Gluster -> Ceph

Il 14/12/2023 16:08, Joe Julian ha scritto:
> With ceph, if the placement database is corrupted, all your data is lost 
> (happened to my employer, once, losing 5PB of customer data).
 From what I've been told (by experts) it's really hard to make it 
happen. More if proper redundancy of MON and MDS daemons is implemented 
on quality HW.
> With Gluster, it's just files on disks, easily recovered.
I've already had to do it twice in a year with the coming third time 
that's the "definitive migration".
The first time there were too many little files, the second it seemed 
192GB RAM are not enough to handle 30 bricks per server, and now that I 
reduced to just 6 bricks per server (creating RAIDs) and created a brand 
new volume in august, I already find lots of FUSE-inaccessible files 
that doesn't heal. Should be impossible since I'm using "replica 3 
arbiter 1" over IPoIB with the three servers speaking directly via the 
switch. But it keeps happening. I really trusted Gluster promises, but 
currently what I (and, worse, the users) see is a 60-70% availability.

Neither Gluster nor Ceph are "backup solutions", so if the data is not
easily replaceable it's better to have it elsewhere. Better if offline.

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Universit? di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

On December 17, 2023 5:40:52 AM PST, Diego Zuccato <diego.zuccato at
unibo.it> wrote:
>Il 14/12/2023 16:08, Joe Julian ha scritto:
>
>> With ceph, if the placement database is corrupted, all your data is
lost (happened to my employer, once, losing 5PB of customer data).
>
>From what I've been told (by experts) it's really hard to make it
happen. More if proper redundancy of MON and MDS daemons is implemented on
quality HW.
>
LSI isn't exactly crap hardware. But when a flaw causes it to drop drives
under heavy load, the rebalance from dropped drives can cause that heavy load
causing a cascading failure. When the journal is never idle long enough to
checkpoint, it fills the partition and ends up corrupted and unrecoverable.

>Neither Gluster nor Ceph are "backup solutions", so if the data is
not easily replaceable it's better to have it elsewhere. Better if offline.
>
It's a nice idea but when you're dealing in petabytes of data, streaming
in as fast as your storage will allow, it's just not physically possible.