similar to: Slow performance of gluster volume

Hi Krutika, Is it anything in the profile indicating what is causing this bottleneck? In case i can collect any other info let me know. Thanx On Sep 5, 2017 13:27, "Abi Askushi" <rightkicktech at gmail.com> wrote: Hi Krutika, Attached the profile stats. I enabled profiling then ran some dd tests. Also 3 Windows VMs are running on top this volume but did not do any stress

Hi Krutika, I already have a preallocated disk on VM. Now I am checking performance with dd on the hypervisors which have the gluster volume configured. I tried also several values of shard-block-size and I keep getting the same low values on write performance. Enabling client-io-threads also did not have any affect. The version of gluster I am using is glusterfs 3.8.12 built on May 11 2017

I tried to follow step from https://wiki.centos.org/SpecialInterestGroup/Storage to install latest gluster on the first node. It installed 3.10 and not 3.11. I am not sure how to install 3.11 without compiling it. Then when tried to start the gluster on the node the bricks were reported down (the other 2 nodes have still 3.8). No sure why. The logs were showing the below (even after rebooting the

I'm assuming you are using this volume to store vm images, because I see shard in the options list. Speaking from shard translator's POV, one thing you can do to improve performance is to use preallocated images. This will at least eliminate the need for shard to perform multiple steps as part of the writes - such as creating the shard and then writing to it and then updating the

Great to hear! ----- Original Message ----- > From: "Abi Askushi" <rightkicktech at gmail.com> > To: "Krutika Dhananjay" <kdhananj at redhat.com> > Cc: "gluster-user" <gluster-users at gluster.org> > Sent: Friday, September 8, 2017 7:01:00 PM > Subject: Re: [Gluster-users] Slow performance of gluster volume > > Following

Do you see any improvement with 3.11.1 as that has a patch that improves perf for this kind of a workload Also, could you disable eager-lock and check if that helps? I see that max time is being spent in acquiring locks. -Krutika On Wed, Sep 6, 2017 at 1:38 PM, Abi Askushi <rightkicktech at gmail.com> wrote: > Hi Krutika, > > Is it anything in the profile indicating what is

----- Original Message ----- > From: "Abi Askushi" <rightkicktech at gmail.com> > To: "Ben Turner" <bturner at redhat.com> > Cc: "Krutika Dhananjay" <kdhananj at redhat.com>, "gluster-user" <gluster-users at gluster.org> > Sent: Monday, September 11, 2017 1:40:42 AM > Subject: Re: [Gluster-users] Slow performance of

Following changes resolved the perf issue: Added the option /etc/glusterfs/glusterd.vol : option rpc-auth-allow-insecure on restarted glusterd Then set the volume option: gluster volume set vms server.allow-insecure on I am reaching now the max network bandwidth and performance of VMs is quite good. Did not upgrade the glusterd. As a next try I am thinking to upgrade gluster to 3.12 + test

Did not upgrade yet gluster. I am still using 3.8.12. Only the mentioned changes did provide the performance boost. >From which version to which version did you see such performance boost? I will try to upgrade and check difference also. On Sep 11, 2017 2:45 AM, "Ben Turner" <bturner at redhat.com> wrote: Great to hear! ----- Original Message ----- > From: "Abi

Hi Abi Can you please share your current transfer speeds after you made the change? Thank you. On Mon, Sep 11, 2017 at 9:55 AM, Ben Turner <bturner at redhat.com> wrote: > ----- Original Message ----- > > From: "Abi Askushi" <rightkicktech at gmail.com> > > To: "Ben Turner" <bturner at redhat.com> > > Cc: "Krutika Dhananjay"

OK my understanding is that with preallocated disks the performance with and without shard will be the same. In any case, please attach the volume profile[1], so we can see what else is slowing things down. -Krutika [1] - https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/#running-glusterfs-volume-profile-command On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi

Couple of things: 1. Like Darrell suggested, you should enable stat-prefetch and increase client and server event threads to 4. # gluster volume set <VOL> performance.stat-prefetch on # gluster volume set <VOL> client.event-threads 4 # gluster volume set <VOL> server.event-threads 4 2. Also glusterfs-3.10.1 and above has a shard performance bug fix -

Another update. I've setup a replica 3 volume without sharding and tried to install a VM on a qcow2 volume on that device; however the result is the same and the vm image has been corrupted, exactly at the same point. Here's the volume info of the create volume: Volume Name: gvtest Type: Replicate Volume ID: e2ddf694-ba46-4bc7-bc9c-e30803374e9d Status: Started Snapshot Count: 0 Number

Thanks for that input. Adding Niels since the issue is reproducible only with libgfapi. -Krutika On Thu, Jan 18, 2018 at 1:39 PM, Ing. Luca Lazzeroni - Trend Servizi Srl < luca at gvnet.it> wrote: > Another update. > > I've setup a replica 3 volume without sharding and tried to install a VM > on a qcow2 volume on that device; however the result is the same and the vm >

After other test (I'm trying to convice myself about gluster reliability :-) I've found that with performance.write-behind off the vm works without problem. Now I'll try with write-behind on and flush-behind on too. Il 18/01/2018 13:30, Krutika Dhananjay ha scritto: > Thanks for that input. Adding Niels since the issue is reproducible > only with libgfapi. > >

Hi, after our IRC chat I've rebuilt a virtual machine with FUSE based virtual disk. Everything worked flawlessly. Now I'm sending you the output of the requested getfattr command on the disk image: # file: TestFUSE-vda.qcow2 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x40ffafbbe987445692bb31295fa40105

Dear Krutika, Sorry for asking so naively but can you tell me on what factor do you base that the client and server event-threads parameters for a volume should be set to 4? Is this metric for example based on the number of cores a GlusterFS server has? I am asking because I saw my GlusterFS volumes are set to 2 and would like to set these parameters to something meaningful for performance

[Adding gluster-users] On Mon, Jun 19, 2017 at 8:16 PM, Chris Boot <bootc at bootc.net> wrote: > Hi folks, > > I have 3x servers in a "hyper-converged" oVirt 4.1.2 + GlusterFS 3.10 > configuration. My VMs run off a replica 3 arbiter 1 volume comprised of > 6 bricks, which themselves live on two SSDs in each of the servers (one > brick per SSD). The bricks are

Also to help isolate the component, could you answer these: 1. on a different volume with shard not enabled, do you see this issue? 2. on a plain 3-way replicated volume (no arbiter), do you see this issue? On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay <kdhananj at redhat.com> wrote: > Please share the volume-info output and the logs under /var/log/glusterfs/ > from all your

I've just done all the steps to reproduce the problem. Tha VM volume has been created via "qemu-img create -f qcow2 Test-vda2.qcow2 20G" on the gluster volume mounted via FUSE. I've tried also to create the volume with preallocated metadata, which moves the problem a bit far away (in time). The volume is a replice 3 arbiter 1 volume hosted on XFS bricks. Here are the