Hi all,
We have a 4-node distributed cluster with 2 bricks per node running Gluster
7.7 + ZFS. We use directory quota to limit the space used by our members on
each project. Two days ago we noticed inconsistent space used reported by
Gluster in the quota list.
A small snippet of gluster volume quota vol list,
Path Hard-limit Soft-limit Used
Available Soft-limit exceeded? Hard-limit exceeded?
/projectA 5.0TB 80%(4.0TB) 3.1TB 1.9TB
No No
*/projectB 100.0TB 80%(80.0TB) 16383.4PB 740.9TB
No No*
/projectC 70.0TB 80%(56.0TB) 50.0TB 20.0TB
No No
The total space available in the cluster is 360TB, the quota for projectB
is 100TB and, as you can see, its reporting 16383.4PB used and 740TB
available (already decreased from 750TB).
There was an issue in Gluster 3.x related to the wrong directory quota (
https://lists.gluster.org/pipermail/gluster-users/2016-February/025305.html
and
https://lists.gluster.org/pipermail/gluster-users/2018-November/035374.html)
but it's marked as solved (not sure if the solution still applies).
*On projectB*
# getfattr -d -m . -e hex projectB
# file: projectB
trusted.gfid=0x3ca2bce0455945efa6662813ce20fc0c
trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f35e69800098ed9
trusted.glusterfs.dht=0xe1a4060c000000003ffffffe5ffffffc
trusted.glusterfs.mdata=0x010000000000000000000000005f355c59000000000939079f000000005ce2aff90000000007fdacb0000000005ce2aff90000000007fdacb0
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000ab0f227a860000000000478e33acffffffffffffc112
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set.1=0x0000640000000000ffffffffffffffff
trusted.glusterfs.quota.size.1=0x0000ab0f227a860000000000478e33acffffffffffffc112
*On projectA*
# getfattr -d -m . -e hex projectA
# file: projectA
trusted.gfid=0x05b09ded19354c0eb544d22d4659582e
trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f1aeb9f00044c64
trusted.glusterfs.dht=0xe1a4060c000000001fffffff3ffffffd
trusted.glusterfs.mdata=0x010000000000000000000000005f1ac6a10000000018f30a4e000000005c338fab0000000017a3135a000000005b0694fb000000001584a21b
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000067de3bbe20000000000000128610000000000033498
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set.1=0x0000460000000000ffffffffffffffff
trusted.glusterfs.quota.size.1=0x0000067de3bbe20000000000000128610000000000033498
Any idea on what's happening and how to fix it?
Thanks!
*Jo?o Ba?to*
---------------
*Scientific Computing and Software Platform*
Champalimaud Research
Champalimaud Center for the Unknown
Av. Bras?lia, Doca de Pedrou?os
1400-038 Lisbon, Portugal
fchampalimaud.org <https://www.fchampalimaud.org/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200814/2e33e56d/attachment.html>
Hi Jo?o, Based on your output it seems that the quota size is different on the 2 bricks. Have you tried to remove the quota and then recreate it ? Maybe it will be the easiest way to fix it. Best Regards, Strahil Nikolov ?? 14 ?????? 2020 ?. 4:35:14 GMT+03:00, "Jo?o Ba?to" <joao.bauto at neuro.fchampalimaud.org> ??????:>Hi all, > >We have a 4-node distributed cluster with 2 bricks per node running >Gluster >7.7 + ZFS. We use directory quota to limit the space used by our >members on >each project. Two days ago we noticed inconsistent space used reported >by >Gluster in the quota list. > >A small snippet of gluster volume quota vol list, > > Path Hard-limit Soft-limit Used >Available Soft-limit exceeded? Hard-limit exceeded? >/projectA 5.0TB 80%(4.0TB) 3.1TB 1.9TB > No No >*/projectB 100.0TB 80%(80.0TB) 16383.4PB 740.9TB > No No* >/projectC 70.0TB 80%(56.0TB) 50.0TB 20.0TB > No No > >The total space available in the cluster is 360TB, the quota for >projectB >is 100TB and, as you can see, its reporting 16383.4PB used and 740TB >available (already decreased from 750TB). > >There was an issue in Gluster 3.x related to the wrong directory quota >( >https://lists.gluster.org/pipermail/gluster-users/2016-February/025305.html > and >https://lists.gluster.org/pipermail/gluster-users/2018-November/035374.html) >but it's marked as solved (not sure if the solution still applies). > >*On projectB* ># getfattr -d -m . -e hex projectB ># file: projectB >trusted.gfid=0x3ca2bce0455945efa6662813ce20fc0c >trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f35e69800098ed9 >trusted.glusterfs.dht=0xe1a4060c000000003ffffffe5ffffffc >trusted.glusterfs.mdata=0x010000000000000000000000005f355c59000000000939079f000000005ce2aff90000000007fdacb0000000005ce2aff90000000007fdacb0 >trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000ab0f227a860000000000478e33acffffffffffffc112 >trusted.glusterfs.quota.dirty=0x3000 >trusted.glusterfs.quota.limit-set.1=0x0000640000000000ffffffffffffffff >trusted.glusterfs.quota.size.1=0x0000ab0f227a860000000000478e33acffffffffffffc112 > >*On projectA* ># getfattr -d -m . -e hex projectA ># file: projectA >trusted.gfid=0x05b09ded19354c0eb544d22d4659582e >trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f1aeb9f00044c64 >trusted.glusterfs.dht=0xe1a4060c000000001fffffff3ffffffd >trusted.glusterfs.mdata=0x010000000000000000000000005f1ac6a10000000018f30a4e000000005c338fab0000000017a3135a000000005b0694fb000000001584a21b >trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000067de3bbe20000000000000128610000000000033498 >trusted.glusterfs.quota.dirty=0x3000 >trusted.glusterfs.quota.limit-set.1=0x0000460000000000ffffffffffffffff >trusted.glusterfs.quota.size.1=0x0000067de3bbe20000000000000128610000000000033498 > >Any idea on what's happening and how to fix it? > >Thanks! >*Jo?o Ba?to* >--------------- > >*Scientific Computing and Software Platform* >Champalimaud Research >Champalimaud Center for the Unknown >Av. Bras?lia, Doca de Pedrou?os >1400-038 Lisbon, Portugal >fchampalimaud.org <https://www.fchampalimaud.org/>