jring at mail.de
2018-Oct-16 12:50 UTC
[Gluster-users] Wrong volume size for distributed dispersed volume on 4.1.5
Hi everybody, I have created a distributed dispersed volume on 4.1.5 under centos7 like this a few days ago: gluster volume create data_vol1 disperse-data 4 redundancy 2 transport tcp \ \ gf-p-d-01.isec.foobar.com:/bricks/brick1/brick \ gf-p-d-03.isec.foobar.com:/bricks/brick1/brick \ gf-p-d-04.isec.foobar.com:/bricks/brick1/brick \ gf-p-k-01.isec.foobar.com:/bricks/brick1/brick \ gf-p-k-03.isec.foobar.com:/bricks/brick1/brick \ gf-p-k-04.isec.foobar.com:/bricks/brick1/brick \ \ gf-p-d-01.isec.foobar.com:/bricks/brick2/brick \ gf-p-d-03.isec.foobar.com:/bricks/brick2/brick \ gf-p-d-04.isec.foobar.com:/bricks/brick2/brick \ gf-p-k-01.isec.foobar.com:/bricks/brick2/brick \ gf-p-k-03.isec.foobar.com:/bricks/brick2/brick \ gf-p-k-04.isec.foobar.com:/bricks/brick2/brick \ \ ... same for brick3 to brick9... \ gf-p-d-01.isec.foobar.com:/bricks/brick10/brick \ gf-p-d-03.isec.foobar.com:/bricks/brick10/brick \ gf-p-d-04.isec.foobar.com:/bricks/brick10/brick \ gf-p-k-01.isec.foobar.com:/bricks/brick10/brick \ gf-p-k-03.isec.foobar.com:/bricks/brick10/brick \ gf-p-k-04.isec.foobar.com:/bricks/brick10/brick This worked nicely and resulted in the following filesystem: [root at gf-p-d-01 ~]# df -h /data/ Dateisystem Gr??e Benutzt Verf. Verw% Eingeh?ngt auf gf-p-d-01.isec.foobar.com:/data_vol1 219T 2,2T 217T 2% /data Each of the bricks resides on its own 6TB disk with 1 big partition formated with xfs. Yesterday a colleague looked at the filesystem and found some space missing... [root at gf-p-d-01 ~]# df -h /data/ Filesystem Size Used Avail Use% Mounted on gf-p-d-01.isec.foobar.com:/data_vol1 22T 272G 22T 2% /data Some googling brought the following bug report against 3.4 which looks familiar: https://bugzilla.redhat.com/show_bug.cgi?id=1541830 So we did a quick grep shared-brick-count /var/lib/glusterd/vols/data_vol1/* on all boxes and found that on 5 out of 6 boxes this was shared-brick-count=0 for all bricks on remote boxes and 1 for local bricks. Is this the expected result or should we have all 1 everywhere (as the quick fix script from the case sets it)? Also on one box (the one where we created the volume from, btw) we have shared-brick-count=0 for all remote bricks and 10 for the local bricks. Is it possible that the bug from 3.4 still exists in 4.1.5 and should we try the filter script which sets shared-brick-count=1 for all bricks? The volume is not currently in production so now would be the time to play around and find the problem... TIA and regards, Joachim ------------------------------------------------------------------------------------------------- FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSIT?T UND KOMFORT
Nithya Balachandran
2018-Oct-16 14:13 UTC
[Gluster-users] Wrong volume size for distributed dispersed volume on 4.1.5
Hi, On 16 October 2018 at 18:20, <jring at mail.de> wrote:> Hi everybody, > > I have created a distributed dispersed volume on 4.1.5 under centos7 like > this a few days ago: > > gluster volume create data_vol1 disperse-data 4 redundancy 2 transport tcp > \ > \ > gf-p-d-01.isec.foobar.com:/bricks/brick1/brick \ > gf-p-d-03.isec.foobar.com:/bricks/brick1/brick \ > gf-p-d-04.isec.foobar.com:/bricks/brick1/brick \ > gf-p-k-01.isec.foobar.com:/bricks/brick1/brick \ > gf-p-k-03.isec.foobar.com:/bricks/brick1/brick \ > gf-p-k-04.isec.foobar.com:/bricks/brick1/brick \ > \ > gf-p-d-01.isec.foobar.com:/bricks/brick2/brick \ > gf-p-d-03.isec.foobar.com:/bricks/brick2/brick \ > gf-p-d-04.isec.foobar.com:/bricks/brick2/brick \ > gf-p-k-01.isec.foobar.com:/bricks/brick2/brick \ > gf-p-k-03.isec.foobar.com:/bricks/brick2/brick \ > gf-p-k-04.isec.foobar.com:/bricks/brick2/brick \ > \ > ... same for brick3 to brick9... > \ > gf-p-d-01.isec.foobar.com:/bricks/brick10/brick \ > gf-p-d-03.isec.foobar.com:/bricks/brick10/brick \ > gf-p-d-04.isec.foobar.com:/bricks/brick10/brick \ > gf-p-k-01.isec.foobar.com:/bricks/brick10/brick \ > gf-p-k-03.isec.foobar.com:/bricks/brick10/brick \ > gf-p-k-04.isec.foobar.com:/bricks/brick10/brick > > This worked nicely and resulted in the following filesystem: > [root at gf-p-d-01 ~]# df -h /data/ > Dateisystem Gr??e Benutzt Verf. Verw% Eingeh?ngt auf > gf-p-d-01.isec.foobar.com:/data_vol1 219T 2,2T 217T 2% /data > > Each of the bricks resides on its own 6TB disk with 1 big partition > formated with xfs. > > Yesterday a colleague looked at the filesystem and found some space > missing... > [root at gf-p-d-01 ~]# df -h /data/ > Filesystem Size Used Avail Use% Mounted on > gf-p-d-01.isec.foobar.com:/data_vol1 22T 272G 22T 2% /data > > Some googling brought the following bug report against 3.4 which looks > familiar: > > https://bugzilla.redhat.com/show_bug.cgi?id=1541830 > > So we did a quick grep shared-brick-count /var/lib/glusterd/vols/data_vol1/* > on all boxes and found that on 5 out of 6 boxes this was > shared-brick-count=0 for all bricks on remote boxes and 1 for local bricks. > > Is this the expected result or should we have all 1 everywhere (as the > quick fix script from the case sets it)? >No , this is fine. The shared-brick-count only needs to be 1 for the local bricks. The value for the remote bricks can be 0.> > Also on one box (the one where we created the volume from, btw) we have > shared-brick-count=0 for all remote bricks and 10 for the local bricks. >This is a problem. The shared-brick-count should be 1 for the local bricks here as well.> Is it possible that the bug from 3.4 still exists in 4.1.5 and should we > try the filter script which sets shared-brick-count=1 for all bricks? > >Can you try 1. restarting glusterd on all the nodes one after another (not at the same time) 2. Setting a volume option (say gluster volume set <volname> cluster.min-free-disk 11%) and see if it fixes the issue? Regards, Nithya> The volume is not currently in production so now would be the time to play > around and find the problem... > > TIA and regards, > > Joachim > > > ------------------------------------------------------------ > ------------------------------------- > FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSIT?T UND KOMFORT > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181016/3edfeddd/attachment.html>