Gluster users - Apr 2020 - gnfs split brain when 1 server in 3x1 down (high load) - help request

On 16/04/20 6:54 pm, Erik Jacobson wrote:
>> The patch by itself is only making changes specific to AFR, so it
should not
>> affect other translators. But I wonder how readdir-ahead is enabled in
your
>> gnfs stack. All performance xlators are turned off in gnfs except
>> write-behind and AFAIK, there is no way to enable them via the CLI. Did
you
>> custom edit your gnfs volfile to add readdir-ahead? If yes, does the
crash
>> go-away if you remove the xlator from the nfs volfile?
> thank you. A quick reply. I will then go research how to do this,
> I've never hand edited a volume before. I've never even really
looked at
> the gnfs volfile before.
>
> There are no custom code changes or hand edits.
>
> More soon.
>
Okay, /var/lib/glusterd/nfs/nfs-server.vol is the file you want to look 
at if you are using gnfs.

-Ravi

Quick update just on how this got set.

gluster volume set cm_shared performance.parallel-readdir on

Is something we did turn on, thinking it might make our NFS services
faster and not knowing about it possibly being negative.

Below is a diff of the nfs volume file ON vs OFF. So I will simply turn
this OFF and do a test run. Does this look correct? I will start testing
with this turned OFF. Thank you!

[root at leader1 nfs]# diff -u /tmp/nfs-server.vol-ORIG nfs-server.vol
--- /tmp/nfs-server.vol-ORIG	2020-04-16 09:28:56.855309870 -0500
+++ nfs-server.vol	2020-04-16 09:29:14.267289600 -0500
@@ -60,21 +60,13 @@
     subvolumes cm_shared-client-0 cm_shared-client-1 cm_shared-client-2
 end-volume

-volume cm_shared-readdir-ahead-0
-    type performance/readdir-ahead
-    option rda-cache-limit 10MB
-    option rda-request-size 131072
-    option parallel-readdir on
-    subvolumes cm_shared-replicate-0
-end-volume
-
 volume cm_shared-dht
     type cluster/distribute
     option force-migration off
     option lock-migration off
     option lookup-optimize on
     option lookup-unhashed auto
-    subvolumes cm_shared-readdir-ahead-0
+    subvolumes cm_shared-replicate-0
 end-volume

 volume cm_shared-utime


On Thu, Apr 16, 2020 at 06:58:01PM +0530, Ravishankar N
wrote:
> 
> On 16/04/20 6:54 pm, Erik Jacobson wrote:
> > > The patch by itself is only making changes specific to AFR, so it
should not
> > > affect other translators. But I wonder how readdir-ahead is
enabled in your
> > > gnfs stack. All performance xlators are turned off in gnfs except
> > > write-behind and AFAIK, there is no way to enable them via the
CLI. Did you
> > > custom edit your gnfs volfile to add readdir-ahead? If yes, does
the crash
> > > go-away if you remove the xlator from the nfs volfile?
> > thank you. A quick reply. I will then go research how to do this,
> > I've never hand edited a volume before. I've never even really
looked at
> > the gnfs volfile before.
> > 
> > There are no custom code changes or hand edits.
> > 
> > More soon.
> > 
> Okay, /var/lib/glusterd/nfs/nfs-server.vol is the file you want to look at
> if you are using gnfs.
> 
> -Ravi