Gluster users - Aug 2017 - GlusterFS as virtual machine storage

Il 23-08-2017 18:51 Gionatan Danti ha scritto:
> Il 23-08-2017 18:14 Pavel Szalbot ha scritto:
>> Hi, after many VM crashes during upgrades of Gluster, losing network
>> connectivity on one node etc. I would advise running replica 2 with
>> arbiter.
> 
> Hi Pavel, this is bad news :(
> So, in your case at least, Gluster was not stable? Something as simple
> as an update would let it crash?
> 
>> I once even managed to break this setup (with arbiter) due to network
>> partitioning - one data node never healed and I had to restore from
>> backups (it was easier and kind of non-production). Be extremely
>> careful and plan for failure.
> 
> I would use VM locking via sanlock or virtlock, so a split brain
> should not cause simultaneous changes on both replicas. I am more
> concerned about volume heal time: what will happen if the standby node
> crashes/reboots? Will *all* data be re-synced from the master, or only
> changed bit will be re-synced? As stated above, I would like to avoid
> using sharding...
> 
> Thanks.
Hi all,
any other advice from who use (or do not use) Gluster as a replicated VM 
backend?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8

> This is true even if I manage locking at application level (via virlock 
> or sanlock)?
Yes. Gluster has it's own quorum, you can disable it but that's just a
recipe for a disaster.
> Also, on a two-node setup it is *guaranteed* for updates to one node to 
> put offline the whole volume?
I think so, but I never took the chance so who knows.
> On the other hand, a 3-way setup (or 2+arbiter) if free from all these 
> problems?
> 
Free from a lot of problems, but apparently not as good as a replica 3
volume. I can't comment on arbiter, I only have replica 3 clusters. I
can tell you that my colleagues setting up 2 nodes clusters have _a lot_
of problems.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170825/044b99af/attachment.sig>

On 8/25/2017 12:56 AM, Gionatan Danti wrote:
>
>
>> WK wrote:
>> 2 node plus Arbiter. You NEED the arbiter or a third node. Do NOT try 2
>> node with a VM
>
> This is true even if I manage locking at application level (via 
> virlock or sanlock)?

We ran Rep2 for years on 3.4.? It does work if you are really,really? 
careful,? But in a crash on one side, you might have lost some bits that 
were on the fly. The VM would then try to heal.
Without sharding, big VMs take a while because the WHOLE VM file has to 
be copied over. Then you might get Split-brain and have to stop the VM, 
pick the good one, make sure that is healed on both sides and then 
restart the VM.

Arbiter/Replica 3 prevents that. Sharding helps a lot as well by making 
the heals really quick, though in a Replica 2 with sharding you no 
longer have a nice big? .img file sitting on each brick in plain view 
and picking a split-brain winner is now WAY more complicated. You would 
have to re-assemble things.

We were quite good and fixing broken Gluster 3.4 nodes, but we are 
*much* happier with the Arbiter node and sharding. It is a huge difference.
We could go to Rep3 but we like the extra speed and we are comfortable 
with the Arb limitations (we also have excellent off cluster backups 
<grin>).

> Also, on a two-node setup it is *guaranteed* for updates to one node 
> to put offline the whole volume?
If you still have quorum turned on, then yes. One side goes and you are 
down.
> On the other hand, a 3-way setup (or 2+arbiter) if free from all these 
> problems?
>
Yes, you can lose one of the three nodes and after the pause, everything 
just continues. If you have a second failure before you can recover, 
then you have lost quorum.

If that second failure is the other actual replica, then you could get 
into a situation where the arbiter isn't happy with either copy when you 
come back up and of course the arbiter doesn't have a good copy itself. 
Pavel alluded to something like that when describing his problem.

That is where replica 3 helps. In theory, with replica 3, you could lose 
2 nodes and still have a reasonable copy of your VM, though you've lost 
quorum and are still down. At that point, *I* would kill the two bad 
nodes (STOMITH) to prevent them from coming back AND turn off quorum. 
You could then run on the single node until you can save/copy those VM 
images, preferably by migrating off that volume completely. Create a 
remote pool using SSHFS if you have nothing else available. THEN I would 
go back and fix the gluster cluster and migrate back into it.

Replica2/Replica3 does not matter if you lose your Gluster network 
switch, but again the Arb or Rep3 setup makes it easier to recover. I 
suppose the only advantage of Replica2 is that you can use a cross over 
cable and not worry about losing the switch, but bonding/teaming works 
well and there are bonding modes that don't require the same switch for 
the bond slaves. So you can build in some redundancy there as well.