Frank Ruehlemann
2018-Nov-19 16:25 UTC
[Gluster-users] Gluster 3.12.14: wrong quota in Distributed Dispersed Volume
Hi, we're running a Distributed Dispersed volume with Gluster 3.12.14 at Debian 9.6 (Stretch). We migrated our data (>300TB) from a pure Distributed volume into this Dispersed volume with cp, followed by multiple rsyncs. After the migration was successful we enabled quotas again with "gluster volume quota $VOLUME enable", which finished successfully. And we set our required quotas with "gluster volume quota $VOLUME limit-usage $PATH $QUOTA", which finished without errors too. But our "gluster volume quota $VOLUME list" shows wrong values. For example: A directory with ~170TB of data shows only 40.8TB Used. When we sum up all quoted directories we're way under the ~310TB that "df -h /$volume" shows. And "df -h /$volume/$directory" shows wrong values for nearly all directories. All 72 8TB-bricks and all quota deamons of the 6 servers are visible and online in "gluster volume status $VOLUME". In quotad.log I found multiple warnings like this:> [2018-11-16 09:21:25.738901] W [dict.c:636:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/features/quotad.so(+0x1d58) [0x7f6844be7d58] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/features/quotad.so(+0x2b92) [0x7f6844be8b92] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f684b0db640] ) 0-dict: dict is NULL [Invalid argument]In some brick logs I found those:> [2018-11-19 07:23:30.932327] I [MSGID: 120020] [quota.c:2198:quota_unlink_cbk] 0-$VOLUME-quota: quota context not set inode (gfid:f100f7a9-0779-4b4c-880f-c8b3b4bdc49d) [Invalid argument]and (replaced the volume name with "$VOLUME") those:> The message "W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-$VOLUME-quota: parent is NULL [Invalid argument]" repeated 13 times between [2018-11-19 15:28:54.089404] and [2018-11-19 15:30:12.792175] > [2018-11-19 15:31:34.559348] W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-$VOLUME-quota: parent is NULL [Invalid argument]I already found that setting the flag "trusted.glusterfs.quota.dirty" might help, but I'm unsure about the consequences that will be triggered. And I'm unsure about the necessary version flag. Has anyone an idea how to fix this? Best Regards, -- Frank R?hlemann IT-Systemtechnik UNIVERSIT?T ZU L?BECK IT-Service-Center Ratzeburger Allee 160 23562 L?beck Tel +49 451 3101 2034 Fax +49 451 3101 2004 ruehlemann at itsc.uni-luebeck.de www.itsc.uni-luebeck.de
Hari Gowtham
2018-Nov-20 06:08 UTC
[Gluster-users] Gluster 3.12.14: wrong quota in Distributed Dispersed Volume
Hi, Can you check if the quota crawl finished? Without it having finished the quota list will show incorrect values. Looking at the under accounting, it looks like the crawl is not yet finished ( it does take a lot of time as it has to crawl the whole filesystem). If the crawl has finished and the usage is still showing wrong values then there should be an accounting issue. The easy way to fix this is to try restarting quota. This will not cause any problems. The only downside is the limits won't hold true while the quota is disabled, till its enabled and the crawl finishes. Or you can try using the quota fsck script https://review.gluster.org/#/c/glusterfs/+/19179/ to fix your accounting issue. Regards, Hari. On Mon, Nov 19, 2018 at 10:05 PM Frank Ruehlemann <f.ruehlemann at uni-luebeck.de> wrote:> > Hi, > > we're running a Distributed Dispersed volume with Gluster 3.12.14 at > Debian 9.6 (Stretch). > > We migrated our data (>300TB) from a pure Distributed volume into this > Dispersed volume with cp, followed by multiple rsyncs. > After the migration was successful we enabled quotas again with "gluster > volume quota $VOLUME enable", which finished successfully. > And we set our required quotas with "gluster volume quota $VOLUME > limit-usage $PATH $QUOTA", which finished without errors too. > > But our "gluster volume quota $VOLUME list" shows wrong values. > For example: > A directory with ~170TB of data shows only 40.8TB Used. > When we sum up all quoted directories we're way under the ~310TB that > "df -h /$volume" shows. > And "df -h /$volume/$directory" shows wrong values for nearly all > directories. > > All 72 8TB-bricks and all quota deamons of the 6 servers are visible and > online in "gluster volume status $VOLUME". > > > In quotad.log I found multiple warnings like this: > > [2018-11-16 09:21:25.738901] W [dict.c:636:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/features/quotad.so(+0x1d58) [0x7f6844be7d58] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/features/quotad.so(+0x2b92) [0x7f6844be8b92] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f684b0db640] ) 0-dict: dict is NULL [Invalid argument] > > In some brick logs I found those: > > [2018-11-19 07:23:30.932327] I [MSGID: 120020] [quota.c:2198:quota_unlink_cbk] 0-$VOLUME-quota: quota context not set inode (gfid:f100f7a9-0779-4b4c-880f-c8b3b4bdc49d) [Invalid argument] > and (replaced the volume name with "$VOLUME") those: > > The message "W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-$VOLUME-quota: parent is NULL [Invalid argument]" repeated 13 times between [2018-11-19 15:28:54.089404] and [2018-11-19 15:30:12.792175] > > [2018-11-19 15:31:34.559348] W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-$VOLUME-quota: parent is NULL [Invalid argument] > > I already found that setting the flag "trusted.glusterfs.quota.dirty" might help, but I'm unsure about the consequences that will be triggered. And I'm unsure about the necessary version flag. > > Has anyone an idea how to fix this? > > Best Regards, > -- > Frank R?hlemann > IT-Systemtechnik > > UNIVERSIT?T ZU L?BECK > IT-Service-Center > > Ratzeburger Allee 160 > 23562 L?beck > Tel +49 451 3101 2034 > Fax +49 451 3101 2004 > ruehlemann at itsc.uni-luebeck.de > www.itsc.uni-luebeck.de > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-- Regards, Hari Gowtham.