similar to: Performance drop from 3.8 to 3.10

Status: We have a 3 node gluster cluster (proxmox based) - gluster 3.8.12 - Replica 3 - VM Hosting Only - Sharded Storage Or I should say we *had* a 3 node cluster, one node died today. Possibly I can recover it, in whcih case no issues, we just let it heal itself. For now its running happily on 2 nodes with no data loss - gluster for teh win! But its looking like I might have to replace the

Could you disable cluster.eager-lock and try again? -Krutika On Thu, Sep 21, 2017 at 6:31 PM, Lindsay Mathieson < lindsay.mathieson at gmail.com> wrote: > Upgraded recently from 3.8.15 to 3.10.5 and have seen a fairly substantial > drop in read/write perfomance > > env: > > - 3 node, replica 3 cluster > > - Private dedicated Network: 1Gx3, bond: balance-alb >

On Fri, Jun 9, 2017 at 12:41 PM, <lemonnierk at ulrar.net> wrote: > > I'm thinking the following: > > > > gluster volume remove-brick datastore4 replica 2 > > vna.proxmox.softlog:/tank/vmdata/datastore4 force > > > > gluster volume add-brick datastore4 replica 3 > > vnd.proxmox.softlog:/tank/vmdata/datastore4 > > I think that should work

On 9/06/2017 9:56 PM, Pranith Kumar Karampuri wrote: > > > gluster volume remove-brick datastore4 replica 2 > > vna.proxmox.softlog:/tank/vmdata/datastore4 force > > > > gluster volume add-brick datastore4 replica 3 > > vnd.proxmox.softlog:/tank/vmdata/datastore4 > > I think that should work perfectly fine yes, either that > or

> I'm thinking the following: > > gluster volume remove-brick datastore4 replica 2 > vna.proxmox.softlog:/tank/vmdata/datastore4 force > > gluster volume add-brick datastore4 replica 3 > vnd.proxmox.softlog:/tank/vmdata/datastore4 I think that should work perfectly fine yes, either that or directly use replace-brick ? -------------- next part -------------- A non-text

On 11/06/2017 10:01 AM, WK wrote: > You replaced vna with vnd but it is probably not fully healed yet cuz > you had 3.8T worth of chunks to copy. No, the heal had completed. Finished about 9 hours before I rebooted. > > So you had two good nodes (vnb and vng) working and you rebooted one > of them? Three good nodes - vnb, vng, vnh and one dead - vna from node vng: root at

Since my node died on friday I have a dead peer (vna) that needs to be removed. I had major issues this morning that I haven't resolve yet with all VM's going offline when I rebooted a node which I *hope * was due to quorum issues as I now have four peers in the cluster, one dead, three live. Confidence level is not high. -- Lindsay Mathieson

On 11/06/2017 6:42 PM, Atin Mukherjee wrote: > If the dead server doesn't host any volumes (bricks of volumes to be > specific) then you can actually remove the uuid entry from > /var/lib/glusterd from other nodes Is that just the file entry in "/var/lib/glusterd/peers" ? e.g I have: gluster peer status Number of Peers: 3 Hostname: vnh.proxmox.softlog

On 6/10/2017 5:12 PM, Lindsay Mathieson wrote: >> > > Three good nodes - vnb, vng, vnh and one dead - vna > > from node vng: > > root at vng:~# gluster peer status > Number of Peers: 3 > > Hostname: vna.proxmox.softlog > Uuid: de673495-8cb2-4328-ba00-0419357c03d7 > State: Peer in Cluster (Disconnected) > > Hostname: vnb.proxmox.softlog > Uuid:

On 6/10/2017 4:38 PM, Lindsay Mathieson wrote: > Since my node died on friday I have a dead peer (vna) that needs to be > removed. > > > I had major issues this morning that I haven't resolve yet with all > VM's going offline when I rebooted a node which I *hope * was due to > quorum issues as I now have four peers in the cluster, one dead, three > live. >

On 11/06/2017 10:46 AM, WK wrote: > I thought you had removed vna as defective and then ADDED in vnh as > the replacement? > > Why is vna still there? Because I *can't* remove it. It died, was unable to be brought up. The gluster peer detach command only works with live servers - A severe problem IMHO. -- Lindsay Mathieson

On Sun, 11 Jun 2017 at 06:25, Lindsay Mathieson <lindsay.mathieson at gmail.com> wrote: > On 11/06/2017 10:46 AM, WK wrote: > > I thought you had removed vna as defective and then ADDED in vnh as > > the replacement? > > > > Why is vna still there? > > Because I *can't* remove it. It died, was unable to be brought up. The > gluster peer detach command

On Sun, Jun 11, 2017 at 2:12 PM, Atin Mukherjee <amukherj at redhat.com> wrote: > > On Sun, 11 Jun 2017 at 06:25, Lindsay Mathieson < > lindsay.mathieson at gmail.com> wrote: > >> On 11/06/2017 10:46 AM, WK wrote: >> > I thought you had removed vna as defective and then ADDED in vnh as >> > the replacement? >> > >> > Why is vna

On Sun, 11 Jun 2017 at 16:35, Gandalf Corvotempesta < gandalf.corvotempesta at gmail.com> wrote: > > > Il 11 giu 2017 1:00 PM, "Atin Mukherjee" <amukherj at redhat.com> ha scritto: > > Yes. And please ensure you do this after bringing down all the glusterd > instances and then once the peer file is removed from all the nodes restart > glusterd on all the

On 13 June 2017 at 02:56, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote: > We can also do "gluster peer detach <hostname> force right? Just to be sure I setup a test 3 node vm gluster cluster :) then shut down one of the nodes and tried to remove it. root at gh1:~# gluster peer status Number of Peers: 2 Hostname: gh2.brian.softlog Uuid:

Il 11 giu 2017 1:00 PM, "Atin Mukherjee" <amukherj at redhat.com> ha scritto: Yes. And please ensure you do this after bringing down all the glusterd instances and then once the peer file is removed from all the nodes restart glusterd on all the nodes one after another. If you have to bring down all gluster instances before file removal, you also bring down the whole gluster

On Tue, 13 Jun 2017 at 06:39, Lindsay Mathieson <lindsay.mathieson at gmail.com> wrote: > > On 13 June 2017 at 02:56, Pranith Kumar Karampuri <pkarampu at redhat.com> > wrote: > >> We can also do "gluster peer detach <hostname> force right? > > > > Just to be sure I setup a test 3 node vm gluster cluster :) then shut down > one of the nodes

On Sun, 11 Jun 2017 at 16:26, Lindsay Mathieson <lindsay.mathieson at gmail.com> wrote: > On 11/06/2017 6:42 PM, Atin Mukherjee wrote: > > If the dead server doesn't host any volumes (bricks of volumes to be > specific) then you can actually remove the uuid entry from > /var/lib/glusterd from other nodes > > Is that just the file entry in

Hey List, I'm trying to test out using Gluster 3.4 for virtual machine disks. My enviroment consists of two Fedora 19 hosts with gluster and qemu/kvm installed. I have a single volume on gluster called vmdata that contains my qcow2 formated image created like this: qemu-img create -f qcow2 gluster://localhost/vmdata/test1.qcow 8G I'm able to boot my created virtual machine but in the

Hi I'm trying to setup a 3 node gluster, and am hitting huge performance bottlenecks. The 3 servers are connected over 10GB and glusterfs is set to create a 3 node replica. With a single VM performance was poor, but I could have lived with it. I tried to stress it by putting copies of a bunch of VMs on the servers and seeing what happened with parallel nodes.. network load never broke