Gluster users - Aug 2017 - Quotas not working after adding arbiter brick to replica 2

I tried to re-create manually my quotas but not even that works now. Running the
"limit-usage" command as showed below returns success:
$ sudo gluster volume quota myvolume limit-usage /userdirectory 50GB
volume quota : success
but when I list the quotas using "list" nothing appears.
What can I do to fix that issue with the quotas?
> -------- Original Message --------
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
to replica 2
> Local Time: August 2, 2017 2:35 PM
> UTC Time: August 2, 2017 12:35 PM
> From: mabi at protonmail.ch
> To: Sanoj Unnikrishnan <sunnikri at redhat.com>
> Gluster Users <gluster-users at gluster.org>
> Hi Sanoj,
> I copied over the quota.conf file from the affected volume (node 1) and
opened it up with a hex editor but can not recognize anything really except for
the first few header/version bytes. I have attached it within this mail
(compressed with bzip2) as requested.
> Should I recreate them manually? there where around 10 of them. Or is there
a hope of recovering these quotas?
> Regards,
> M.
>
>> -------- Original Message --------
>> Subject: Re: [Gluster-users] Quotas not working after adding arbiter
brick to replica 2
>> Local Time: August 2, 2017 1:06 PM
>> UTC Time: August 2, 2017 11:06 AM
>> From: sunnikri at redhat.com
>> To: mabi <mabi at protonmail.ch>
>> Gluster Users <gluster-users at gluster.org>
>>
>> Mabi,
>> We have fixed a couple of issues in the quota list path.
>> Could you also please attach the quota.conf file
(/var/lib/glusterd/vols/patchy/quota.conf)
>> (Ideally, the first few bytes would be ascii characters followed by 17
bytes per directory on which quota limit is set)
>> Regards,
>> Sanoj
>>
>> On Tue, Aug 1, 2017 at 1:36 PM, mabi <mabi at protonmail.ch>
wrote:
>>
>>> I also just noticed quite a few of the following warning messages
in the quotad.log log file:
>>> [2017-08-01 07:59:27.834202] W [MSGID: 108027]
[afr-common.c:2496:afr_discover_done] 0-myvolume-replicate-0: no read subvols
for (null)
>>>
>>>> -------- Original Message --------
>>>> Subject: [Gluster-users] Quotas not working after adding
arbiter brick to replica 2
>>>> Local Time: August 1, 2017 8:49 AM
>>>> UTC Time: August 1, 2017 6:49 AM
>>>> From: mabi at protonmail.ch
>>>> To: Gluster Users <gluster-users at gluster.org>
>>>> Hello,
>>>> As you might have read in my previous post on the mailing list
I have added an arbiter node to my GlusterFS 3.8.11 replica 2 volume. After some
healing issues and help of Ravi that could get fixed but now I just noticed that
my quotas are all gone.
>>>> When I run the following command:
>>>> glusterfs volume quota myvolume list
>>>> There is no output...
>>>> In the /var/log/glusterfs/quotad.log I can see the following
two lines when running the list command:
>>>> [2017-08-01 06:46:04.451765] W [dict.c:581:dict_unref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
[0x7fe868e21f3d]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x2d82)
[0x7fe868e22d82]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
[0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>>>> [2017-08-01 06:46:04.459154] W [dict.c:581:dict_unref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
[0x7fe868e21f3d]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x2d82)
[0x7fe868e22d82]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
[0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>>>> In case you need this info, I have added by arbiter node to the
replica 2 by using this command:
>>>> gluster volume add-brick myvolume replica 3 arbiter 1
arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>>> How can I get my quotas back working as before? I had defined
around 20 quotas on different directories of that volume.
>>>> Regards,
>>>> Mabi
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170803/0c6b7686/attachment.html>

Hi mabi,

This is a likely issue where the last gfid entry in the quota.conf file is
stale (because the directory was deleted with quota limit on it being
removed)
(https://review.gluster.org/#/c/16507/)

To fix the issue, we need to remove the last entry (last 17 bytes/ 16bytes
based on quota version) in the file.
Please use the below work around for the same until next upgrade.
you only need to change $vol to the name of volume.

==============================vol=<v5>
qconf=/var/lib/glusterd/vols/$vol/quota.conf
qconf_bk="$qconf".bk
cp $qconf $qconf_bk

grep "GlusterFS Quota conf | version: v1.2"
/var/lib/glusterd/vols/v5/quota.conf
if [ $? -eq 0 ];
then
        entry_size=17;
else
        entry_size=16;
fi

size=`ls -l $qconf | awk '{print $5}'`
(( size_new = size - entry_size ))
dd if=$qconf_bk of=$qconf bs=1 count=$size_new
gluster v quota v5 list
===================================
In the unlikely case that there are multiple stale entries in the end of
file you may have to run it multiple times
to fix the issue (each time one stale entry at the end is removed)


On Thu, Aug 3, 2017 at 1:17 PM, mabi <mabi at protonmail.ch> wrote:
> I tried to re-create manually my quotas but not even that works now.
> Running the "limit-usage" command as showed below returns
success:
>
> $ sudo gluster volume quota myvolume limit-usage /userdirectory 50GB
> volume quota : success
>
>
>
> but when I list the quotas using "list" nothing appears.
>
> What can I do to fix that issue with the quotas?
>
> -------- Original Message --------
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
> to replica 2
> Local Time: August 2, 2017 2:35 PM
> UTC Time: August 2, 2017 12:35 PM
> From: mabi at protonmail.ch
> To: Sanoj Unnikrishnan <sunnikri at redhat.com>
> Gluster Users <gluster-users at gluster.org>
>
> Hi Sanoj,
>
> I copied over the quota.conf file from the affected volume (node 1) and
> opened it up with a hex editor but can not recognize anything really except
> for the first few header/version bytes. I have attached it within this mail
> (compressed with bzip2) as requested.
>
> Should I recreate them manually? there where around 10 of them. Or is
> there a hope of recovering these quotas?
>
> Regards,
> M.
>
>
>
> -------- Original Message --------
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
> to replica 2
> Local Time: August 2, 2017 1:06 PM
> UTC Time: August 2, 2017 11:06 AM
> From: sunnikri at redhat.com
> To: mabi <mabi at protonmail.ch>
> Gluster Users <gluster-users at gluster.org>
>
> Mabi,
>
> We have fixed a couple of issues in the quota list path.
> Could you also please attach the quota.conf file
> (/var/lib/glusterd/vols/patchy/quota.conf)
> (Ideally, the first few bytes would be ascii characters followed by 17
> bytes per directory on which quota limit is set)
> Regards,
> Sanoj
>
> On Tue, Aug 1, 2017 at 1:36 PM, mabi <mabi at protonmail.ch> wrote:
>
>> I also just noticed quite a few of the following warning messages in
the
>> quotad.log log file:
>>
>> [2017-08-01 07:59:27.834202] W [MSGID: 108027]
>> [afr-common.c:2496:afr_discover_done] 0-myvolume-replicate-0: no read
>> subvols for (null)
>>
>>
>>
>>
>> -------- Original Message --------
>> Subject: [Gluster-users] Quotas not working after adding arbiter brick
to
>> replica 2
>> Local Time: August 1, 2017 8:49 AM
>> UTC Time: August 1, 2017 6:49 AM
>> From: mabi at protonmail.ch
>> To: Gluster Users <gluster-users at gluster.org>
>>
>> Hello,
>>
>> As you might have read in my previous post on the mailing list I have
>> added an arbiter node to my GlusterFS 3.8.11 replica 2 volume. After
some
>> healing issues and help of Ravi that could get fixed but now I just
noticed
>> that my quotas are all gone.
>>
>> When I run the following command:
>>
>> glusterfs volume quota myvolume list
>>
>> There is no output...
>>
>> In the /var/log/glusterfs/quotad.log I can see the following two lines
>> when running the list command:
>>
>> [2017-08-01 06:46:04.451765] W [dict.c:581:dict_unref]
>>
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
>> [0x7fe868e21f3d] -->/usr/lib/x86_64-linux-gnu/g
>> lusterfs/3.8.11/xlator/features/quotad.so(+0x2d82) [0x7fe868e22d82]
>> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
>> [0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>> [2017-08-01 06:46:04.459154] W [dict.c:581:dict_unref]
>>
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
>> [0x7fe868e21f3d] -->/usr/lib/x86_64-linux-gnu/g
>> lusterfs/3.8.11/xlator/features/quotad.so(+0x2d82) [0x7fe868e22d82]
>> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
>> [0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>>
>> In case you need this info, I have added by arbiter node to the replica
2
>> by using this command:
>>
>> gluster volume add-brick myvolume replica 3 arbiter 1
>> arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>
>>
>>
>> How can I get my quotas back working as before? I had defined around 20
>> quotas on different directories of that volume.
>>
>> Regards,
>> Mabi
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170804/6da883d2/attachment.html>

Thank you very much Sanoj, I ran your script once and it worked. I now have
quotas again...
Question: do you know in which release this issue will be fixed?
> -------- Original Message --------
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
to replica 2
> Local Time: August 4, 2017 3:28 PM
> UTC Time: August 4, 2017 1:28 PM
> From: sunnikri at redhat.com
> To: mabi <mabi at protonmail.ch>
> Gluster Users <gluster-users at gluster.org>
>
> Hi mabi,
> This is a likely issue where the last gfid entry in the quota.conf file is
stale (because the directory was deleted with quota limit on it being removed)
> (https://review.gluster.org/#/c/16507/)
> To fix the issue, we need to remove the last entry (last 17 bytes/ 16bytes
based on quota version) in the file.
> Please use the below work around for the same until next upgrade.
> you only need to change $vol to the name of volume.
> ==============================>
> vol=<v5>
> qconf=/var/lib/glusterd/vols/$vol/quota.conf
> qconf_bk="$qconf".bk
> cp $qconf $qconf_bk
> grep "GlusterFS Quota conf | version: v1.2"
/var/lib/glusterd/vols/v5/quota.conf
> if [ $? -eq 0 ];
> then
> entry_size=17;
> else
> entry_size=16;
> fi
> size=`ls -l $qconf | awk '{print $5}'`
> (( size_new = size - entry_size ))
> dd if=$qconf_bk of=$qconf bs=1 count=$size_new
> gluster v quota v5 list
> ===================================> In the unlikely case that there are
multiple stale entries in the end of file you may have to run it multiple times
> to fix the issue (each time one stale entry at the end is removed)
>
> On Thu, Aug 3, 2017 at 1:17 PM, mabi <mabi at protonmail.ch> wrote:
>
>> I tried to re-create manually my quotas but not even that works now.
Running the "limit-usage" command as showed below returns success:
>> $ sudo gluster volume quota myvolume limit-usage /userdirectory 50GB
>> volume quota : success
>> but when I list the quotas using "list" nothing appears.
>> What can I do to fix that issue with the quotas?
>>
>>> -------- Original Message --------
>>> Subject: Re: [Gluster-users] Quotas not working after adding
arbiter brick to replica 2
>>>
>>> Local Time: August 2, 2017 2:35 PM
>>> UTC Time: August 2, 2017 12:35 PM
>>> From: mabi at protonmail.ch
>>> To: Sanoj Unnikrishnan <sunnikri at redhat.com>
>>> Gluster Users <gluster-users at gluster.org>
>>> Hi Sanoj,
>>> I copied over the quota.conf file from the affected volume (node 1)
and opened it up with a hex editor but can not recognize anything really except
for the first few header/version bytes. I have attached it within this mail
(compressed with bzip2) as requested.
>>> Should I recreate them manually? there where around 10 of them. Or
is there a hope of recovering these quotas?
>>> Regards,
>>> M.
>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Gluster-users] Quotas not working after adding
arbiter brick to replica 2
>>>> Local Time: August 2, 2017 1:06 PM
>>>> UTC Time: August 2, 2017 11:06 AM
>>>> From: sunnikri at redhat.com
>>>> To: mabi <mabi at protonmail.ch>
>>>> Gluster Users <gluster-users at gluster.org>
>>>>
>>>> Mabi,
>>>> We have fixed a couple of issues in the quota list path.
>>>> Could you also please attach the quota.conf file
(/var/lib/glusterd/vols/patchy/quota.conf)
>>>> (Ideally, the first few bytes would be ascii characters
followed by 17 bytes per directory on which quota limit is set)
>>>> Regards,
>>>> Sanoj
>>>>
>>>> On Tue, Aug 1, 2017 at 1:36 PM, mabi <mabi at
protonmail.ch> wrote:
>>>>
>>>>> I also just noticed quite a few of the following warning
messages in the quotad.log log file:
>>>>> [2017-08-01 07:59:27.834202] W [MSGID: 108027]
[afr-common.c:2496:afr_discover_done] 0-myvolume-replicate-0: no read subvols
for (null)
>>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: [Gluster-users] Quotas not working after
adding arbiter brick to replica 2
>>>>>> Local Time: August 1, 2017 8:49 AM
>>>>>> UTC Time: August 1, 2017 6:49 AM
>>>>>> From: mabi at protonmail.ch
>>>>>> To: Gluster Users <gluster-users at gluster.org>
>>>>>> Hello,
>>>>>> As you might have read in my previous post on the
mailing list I have added an arbiter node to my GlusterFS 3.8.11 replica 2
volume. After some healing issues and help of Ravi that could get fixed but now
I just noticed that my quotas are all gone.
>>>>>> When I run the following command:
>>>>>> glusterfs volume quota myvolume list
>>>>>> There is no output...
>>>>>> In the /var/log/glusterfs/quotad.log I can see the
following two lines when running the list command:
>>>>>> [2017-08-01 06:46:04.451765] W [dict.c:581:dict_unref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
[0x7fe868e21f3d]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x2d82)
[0x7fe868e22d82]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
[0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>>>>>> [2017-08-01 06:46:04.459154] W [dict.c:581:dict_unref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
[0x7fe868e21f3d]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x2d82)
[0x7fe868e22d82]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
[0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>>>>>> In case you need this info, I have added by arbiter
node to the replica 2 by using this command:
>>>>>> gluster volume add-brick myvolume replica 3 arbiter 1
arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>> How can I get my quotas back working as before? I had
defined around 20 quotas on different directories of that volume.
>>>>>> Regards,
>>>>>> Mabi
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170804/eec5aa25/attachment.html>