similar to: Changing performance.parallel-readdir to on causes CPU soft lockup and very high load all glusterd nodes

Hi, Just to update this thread. We updated from Gluster 3.12.2 to 3.12.3 which resolved the issue it seems. I checked the changelog but don't see anything that looks like this issue, but I'm glad it seems like it's OK now. Niels Hendriks On 14 November 2017 at 09:42, Niels Hendriks <niels at nuvini.com> wrote: > Hi, > > We're using a 3-node setup where GlusterFS

By chance... it's not 3 replica, 1 arbiter is it? See: https://bugzilla.redhat.com/show_bug.cgi?id=1512371 <https://bugzilla.redhat.com/show_bug.cgi?id=1512371> -- Sam McLeod https://smcleod.net https://twitter.com/s_mcleod -------------- next part -------------- An HTML attachment was scrubbed... URL:

Hi Serkan, I was wondering if you resolved your issue with the high CPU usage and hang after starting gluster? I'm setting up a 3 server (replica 3, arbiter 1), 300 volume, Gluster 3.12 cluster on CentOS 7 and am having what looks to be exactly the same issue as you. With no volumes created CPU usage / load is normal, but after creating all the volumes even with no data CPU and RAM usage

Thanks Milind, Yes I?m hanging out for CentOS?s Storage / Gluster SIG to release the packages for 3.12.1, I can see the packages were built a week ago but they?re still not on the repo :( -- Sam > On 18 Sep 2017, at 9:57 pm, Milind Changire <mchangir at redhat.com> wrote: > > Sam, > You might want to give glusterfs-3.12.1 a try instead. > > > >> On Fri, Sep

FYI - I've been testing the Gluster 3.12.1 packages with the help of the SIG maintainer and I can confirm that the logs are no longer being filled with NFS or null client errors after the upgrade. -- Sam McLeod @s_mcleod https://smcleod.net > On 18 Sep 2017, at 10:14 pm, Sam McLeod <mailinglists at smcleod.net> wrote: > > Thanks Milind, > > Yes I?m hanging out for

> Clarification for below logs: > > - 'dev_static' is the gluster volume. > - 'int-kube-01' is the gluster client. > - '10.51.70.151' is the first node in a three node (2 replica, 1 arbiter) gluster cluster. > - '/var/lib/kubelet/...../iss3dev-static' is a directory on the client that should be mounting

On Tue, Mar 20, 2018 at 8:57 AM, Sam McLeod <mailinglists at smcleod.net> wrote: > Hi Raghavendra, > > > On 20 Mar 2018, at 1:55 pm, Raghavendra Gowdappa <rgowdapp at redhat.com> > wrote: > > Aggregating large number of small writes by write-behind into large writes > has been merged on master: > https://github.com/gluster/glusterfs/issues/364 > >

Do the users have permission to see/interact with the directories, in addition to the files? On Mon, Nov 6, 2017 at 1:55 PM, Nithya Balachandran <nbalacha at redhat.com> wrote: > Hi, > > Please provide the gluster volume info. Do you see any errors in the > client mount log file (/var/log/glusterfs/var-lib-mountedgluster.log)? > > > Thanks, > Nithya > > On 6

Sam, For du ?sh on my newer volume, the result is 161T. The sum of the Used space in the df ?h output for all the bricks is ~163T. Close enough for me to believe everything is there. The total for used space in the df ?h of the mountpoint it 83T, roughly half what is used. Relevant lines from df ?h on server-A: Filesystem Size Used Avail Use% Mounted on /dev/sda1 59T 42T

Howdy, I'm setting up several gluster 3.12 clusters running on CentOS 7 and have having issues with glusterd.log and glustershd.log both being filled with errors relating to null client errors and client-callback functions. They seem to be related to high CPU usage across the nodes although I don't have a way of confirming that (suggestions welcomed!). in

Hi Niels, FYI - tested the install of the 3.13.0-1.el7 packages and all seems well with the install under CentOS 7. yum install -y https://cbs.centos.org/kojifiles/packages/centos-release-gluster313/1.0/1.el7.centos/noarch/centos-release-gluster313-1.0-1.el7.centos.noarch.rpm

We've got an issue with Gluster (3.12.x) where clients can't see directories that exist or are created within a mounted volume. We can see files, but not directories (with ls, find etc...) We can enter (cd) into directories, even if we can't see them. - Host typology is: 2 replica, 1 arbiter. - Volumes are: replicated and running on XFS on the hosts. - Clients are: GlusterFS native

Excellent description, thank you. With performance.write-behind-trickling-writes ON (default): ## 4k randwrite # fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=256MB --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 fio-3.1 Starting 1 process Jobs: 1

Also note, Sam's example is comparing apples and orchards. Feeding one person from an orchard is not as efficient as feeding one person an apple, but if you're feeding 10000 people... Also in question with the NFS example, how long until that chown was flushed? How long until another client could see those changes? That is ignoring the biggie, what happens when the NFS server goes down?

> On 6 Nov 2017, at 3:32 pm, Laura Bailey <lbailey at redhat.com> wrote: > > Do the users have permission to see/interact with the directories, in addition to the files? Yes, full access to directories and files. Also testing using the root user. > > On Mon, Nov 6, 2017 at 1:55 PM, Nithya Balachandran <nbalacha at redhat.com <mailto:nbalacha at redhat.com>>

Hi Raghavendra, > On 20 Mar 2018, at 1:55 pm, Raghavendra Gowdappa <rgowdapp at redhat.com> wrote: > > Aggregating large number of small writes by write-behind into large writes has been merged on master: > https://github.com/gluster/glusterfs/issues/364 <https://github.com/gluster/glusterfs/issues/364> > > Would like to know whether it helps for this usecase.

Hi, Please provide the gluster volume info. Do you see any errors in the client mount log file (/var/log/glusterfs/var-lib-mountedgluster.log)? Thanks, Nithya On 6 November 2017 at 05:13, Sam McLeod <mailinglists at smcleod.net> wrote: > We've got an issue with Gluster (3.12.x) where clients can't see > directories that exist or are created within a mounted volume. >

Sam, You might want to give glusterfs-3.12.1 a try instead. On Fri, Sep 15, 2017 at 6:42 AM, Sam McLeod <mailinglists at smcleod.net> wrote: > Howdy, > > I'm setting up several gluster 3.12 clusters running on CentOS 7 and have > having issues with glusterd.log and glustershd.log both being filled with > errors relating to null client errors and client-callback

We noticed something similar. Out of interest, does du -sh . show the same size? -- Sam McLeod (protoporpoise on IRC) https://smcleod.net https://twitter.com/s_mcleod Words are my own opinions and do not necessarily represent those of my employer or partners. > On 31 Jan 2018, at 12:47 pm, Freer, Eva B. <freereb at ornl.gov> wrote: > > After OS update to CentOS 7.4 or RedHat

On Tue, Mar 20, 2018 at 1:55 AM, TomK <tomkcpr at mdevsys.com> wrote: > On 3/19/2018 10:52 AM, Rik Theys wrote: > >> Hi, >> >> On 03/19/2018 03:42 PM, TomK wrote: >> >>> On 3/19/2018 5:42 AM, Ondrej Valousek wrote: >>> Removing NFS or NFS Ganesha from the equation, not very impressed on my >>> own setup either. For the writes