Nithya Balachandran
2018-Oct-16 14:13 UTC
[Gluster-users] Wrong volume size for distributed dispersed volume on 4.1.5
Hi, On 16 October 2018 at 18:20, <jring at mail.de> wrote:> Hi everybody, > > I have created a distributed dispersed volume on 4.1.5 under centos7 like > this a few days ago: > > gluster volume create data_vol1 disperse-data 4 redundancy 2 transport tcp > \ > \ > gf-p-d-01.isec.foobar.com:/bricks/brick1/brick \ > gf-p-d-03.isec.foobar.com:/bricks/brick1/brick \ > gf-p-d-04.isec.foobar.com:/bricks/brick1/brick \ > gf-p-k-01.isec.foobar.com:/bricks/brick1/brick \ > gf-p-k-03.isec.foobar.com:/bricks/brick1/brick \ > gf-p-k-04.isec.foobar.com:/bricks/brick1/brick \ > \ > gf-p-d-01.isec.foobar.com:/bricks/brick2/brick \ > gf-p-d-03.isec.foobar.com:/bricks/brick2/brick \ > gf-p-d-04.isec.foobar.com:/bricks/brick2/brick \ > gf-p-k-01.isec.foobar.com:/bricks/brick2/brick \ > gf-p-k-03.isec.foobar.com:/bricks/brick2/brick \ > gf-p-k-04.isec.foobar.com:/bricks/brick2/brick \ > \ > ... same for brick3 to brick9... > \ > gf-p-d-01.isec.foobar.com:/bricks/brick10/brick \ > gf-p-d-03.isec.foobar.com:/bricks/brick10/brick \ > gf-p-d-04.isec.foobar.com:/bricks/brick10/brick \ > gf-p-k-01.isec.foobar.com:/bricks/brick10/brick \ > gf-p-k-03.isec.foobar.com:/bricks/brick10/brick \ > gf-p-k-04.isec.foobar.com:/bricks/brick10/brick > > This worked nicely and resulted in the following filesystem: > [root at gf-p-d-01 ~]# df -h /data/ > Dateisystem Gr??e Benutzt Verf. Verw% Eingeh?ngt auf > gf-p-d-01.isec.foobar.com:/data_vol1 219T 2,2T 217T 2% /data > > Each of the bricks resides on its own 6TB disk with 1 big partition > formated with xfs. > > Yesterday a colleague looked at the filesystem and found some space > missing... > [root at gf-p-d-01 ~]# df -h /data/ > Filesystem Size Used Avail Use% Mounted on > gf-p-d-01.isec.foobar.com:/data_vol1 22T 272G 22T 2% /data > > Some googling brought the following bug report against 3.4 which looks > familiar: > > https://bugzilla.redhat.com/show_bug.cgi?id=1541830 > > So we did a quick grep shared-brick-count /var/lib/glusterd/vols/data_vol1/* > on all boxes and found that on 5 out of 6 boxes this was > shared-brick-count=0 for all bricks on remote boxes and 1 for local bricks. > > Is this the expected result or should we have all 1 everywhere (as the > quick fix script from the case sets it)? >No , this is fine. The shared-brick-count only needs to be 1 for the local bricks. The value for the remote bricks can be 0.> > Also on one box (the one where we created the volume from, btw) we have > shared-brick-count=0 for all remote bricks and 10 for the local bricks. >This is a problem. The shared-brick-count should be 1 for the local bricks here as well.> Is it possible that the bug from 3.4 still exists in 4.1.5 and should we > try the filter script which sets shared-brick-count=1 for all bricks? > >Can you try 1. restarting glusterd on all the nodes one after another (not at the same time) 2. Setting a volume option (say gluster volume set <volname> cluster.min-free-disk 11%) and see if it fixes the issue? Regards, Nithya> The volume is not currently in production so now would be the time to play > around and find the problem... > > TIA and regards, > > Joachim > > > ------------------------------------------------------------ > ------------------------------------- > FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSIT?T UND KOMFORT > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181016/3edfeddd/attachment.html>
jring at mail.de
2018-Oct-16 14:34 UTC
[Gluster-users] Wrong volume size for distributed dispersed volume on 4.1.5
Hi,> > So we did a quick grep shared-brick-count /var/lib/glusterd/vols/data_vol1/* on all boxes and found that on 5 out of 6 boxes this was shared-brick-count=0 for all bricks on remote boxes and 1 for local bricks. > > > > Is this the expected result or should we have all 1 everywhere (as the quick fix script from the case sets it)? > > No , this is fine. The shared-brick-count only needs to be 1 for the local bricks. The value for the remote bricks can be 0. > ? > > Also on one box (the one where we created the volume from, btw) we have shared-brick-count=0 for all remote bricks and 10 for the local bricks. > > This is a problem. The shared-brick-count should be 1 for the local bricks here as well. > ? > > Is it possible that the bug from 3.4 still exists in 4.1.5 and should we try the filter script which sets shared-brick-count=1 for all bricks? > > > > Can you try? > 1. restarting glusterd on all the nodes one after another (not at the same time) > 2. Setting a volume option (say gluster volume set <volname> cluster.min-free-disk 11%)? > > and see if it fixes the issue?Hi, ok, this was a quick fix - volume size is correct again and the shared-brick-count is correct everywhere. We'll duly note this in our wiki. Thanks a lot! Joachim ------------------------------------------------------------------------------------------------- FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSIT?T UND KOMFORT