Gluster users - Jan 2020 - NFS clients show missing files while gluster volume rebalanced

We are using gluster 4.1.6. We are using gluster NFS (not ganesha).

Distributed/replicated with subvolume size 3 (6 total servers, 2
subvols).

The NFS clients use this for their root filesystem.

When I add 3 more gluster servers to add one more subvolume to the
storage volumes (so now subvolume size 3, 9 total servers, 3 total
subvolumes), the process gets started. 

ssh leader1 gluster volume add-brick cm_shared 172.23.0.9://data/brick_cm_shared
172.23.0.10://data/brick_cm_shared 172.23.0.11://data/brick_cm_shared

then

ssh leader1 gluster volume rebalance cm_shared start

The re-balance works. 'gluster volume status' shows re-balance in
progress.

However, existing gluster-NFS clients now show missing files and I can
no longer log into them (since NFS is their root). If you are logged in,
you can find that libraries are missing and general unhappiness with
random files now missing.

Is accessing a volume that is in the process of being re-balanced not
supported from a gluster NFS client? Or have I made an error?

Thank you for any help,

Erik

On January 30, 2020 2:20:34 AM GMT+02:00, Erik Jacobson <erik.jacobson at
hpe.com> wrote:
>We are using gluster 4.1.6. We are using gluster NFS (not ganesha).
>
>Distributed/replicated with subvolume size 3 (6 total servers, 2
>subvols).
>
>The NFS clients use this for their root filesystem.
>
>When I add 3 more gluster servers to add one more subvolume to the
>storage volumes (so now subvolume size 3, 9 total servers, 3 total
>subvolumes), the process gets started. 
>
>ssh leader1 gluster volume add-brick cm_shared
>172.23.0.9://data/brick_cm_shared 172.23.0.10://data/brick_cm_shared
>172.23.0.11://data/brick_cm_shared
>
>then
>
>ssh leader1 gluster volume rebalance cm_shared start
>
>The re-balance works. 'gluster volume status' shows re-balance in
>progress.
>
>However, existing gluster-NFS clients now show missing files and I can
>no longer log into them (since NFS is their root). If you are logged
>in,
>you can find that libraries are missing and general unhappiness with
>random files now missing.
>
>Is accessing a volume that is in the process of being re-balanced not
>supported from a gluster NFS client? Or have I made an error?
>
>Thank you for any help,
>
>Erik
>________
>
>Community Meeting Calendar:
>
>APAC Schedule -
>Every 2nd and 4th Tuesday at 11:30 AM IST
>Bridge: https://bluejeans.com/441850968
>
>NA/EMEA Schedule -
>Every 1st and 3rd Tuesday at 01:00 PM EDT
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users
Hi Erik,

Can you provide the volume info.
Also check the logs for the rebalance. There should be some valuable info on
what is going on.

Best Regards,
Strahil Nikolov

Closing the loop in case someone does a search on this...

I have an update. I am getting some time on 1,000 node soon so I have
started to validate jumping to gluster 7.2 on my small lab machine.

I switched the packages to my own build of gluster 7.2 with gnfs.
I re-installed my leader node (gluster/gnfs servers) and created
the volumes the same way as before. This includes heavy cache
optimization for the NFS services volume.

I can no longer duplicate this problem on gluster 7.2. I was able to
duplicate rebalance troubles on NFS clients every time on gluster
4.1.6.

I do have a couple questions on some rebalance errors, which I will send
in a separate email.

Erik

On Wed, Jan 29, 2020 at 06:20:34PM -0600, Erik Jacobson
wrote:
> We are using gluster 4.1.6. We are using gluster NFS (not ganesha).
> 
> Distributed/replicated with subvolume size 3 (6 total servers, 2
> subvols).
> 
> The NFS clients use this for their root filesystem.
> 
> When I add 3 more gluster servers to add one more subvolume to the
> storage volumes (so now subvolume size 3, 9 total servers, 3 total
> subvolumes), the process gets started. 
> 
> ssh leader1 gluster volume add-brick cm_shared
172.23.0.9://data/brick_cm_shared 172.23.0.10://data/brick_cm_shared
172.23.0.11://data/brick_cm_shared
> 
> then
> 
> ssh leader1 gluster volume rebalance cm_shared start
> 
> The re-balance works. 'gluster volume status' shows re-balance in
> progress.
> 
> However, existing gluster-NFS clients now show missing files and I can
> no longer log into them (since NFS is their root). If you are logged in,
> you can find that libraries are missing and general unhappiness with
> random files now missing.
> 
> Is accessing a volume that is in the process of being re-balanced not
> supported from a gluster NFS client? Or have I made an error?
> 
> Thank you for any help,
> 
> Erik