Jorick Astrego
2016-Jan-27 15:03 UTC
[Gluster-users] Quota issue after upgrading from 3.6.4 to 3.6.6
Hi,
After upgrading one of a 3 node glusterfs cluster to 3.6.6, I cannot get
the HDD-1 volume to start properly on this server. The other servers are
still running 3.6.4 because I don't know if they will all fail after the
upgrade. There appears to be an issue with the quota fixed in 3.6.6
(https://bugzilla.redhat.com/show_bug.cgi?id=1259578) that may be the cause:
gluster volume status HDD1
Locking failed on 10.99.99.1. Please check log file for details.
Locking failed on 10.99.99.3. Please check log file for details.
[2016-01-27 14:52:11.268901] W
[marker-quota.c:1452:mq_release_parent_lock] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b]
(-->
/lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743]
))))) 0-HDD1-marker: An operation during quota updation of path
(/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d)
failed (Invalid argument)
[2016-01-27 14:52:11.284504] W
[marker-quota.c:1452:mq_release_parent_lock] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b]
(-->
/lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743]
))))) 0-HDD1-marker: An operation during quota updation of path
(/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d)
failed (Invalid argument)
[2016-01-27 14:52:11.288115] W
[marker-quota.c:1452:mq_release_parent_lock] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b]
(-->
/lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743]
))))) 0-HDD1-marker: An operation during quota updation of path
(/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d)
failed (Invalid argument)
[2016-01-27 14:52:11.292966] W
[marker-quota.c:1452:mq_release_parent_lock] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b]
(-->
/lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3]
(-->
/usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743]
))))) 0-HDD1-marker: An operation during quota updation of path
(/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d)
failed (Invalid argument)
Met vriendelijke groet, With kind regards,
Jorick Astrego
Netbulae Virtualization Experts
----------------
Tel: 053 20 30 270 info at netbulae.eu Staalsteden 4-3A KvK 08198180
Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01
----------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160127/fdab5f3a/attachment.html>
Atin Mukherjee
2016-Jan-27 16:44 UTC
[Gluster-users] Quota issue after upgrading from 3.6.4 to 3.6.6
Please share the glusterd log files along with cmd history log file from all the nodes. Thanks, Atin On 01/27/2016 08:33 PM, Jorick Astrego wrote:> Hi, > > After upgrading one of a 3 node glusterfs cluster to 3.6.6, I cannot get > the HDD-1 volume to start properly on this server. The other servers are > still running 3.6.4 because I don't know if they will all fail after the > upgrade. There appears to be an issue with the quota fixed in 3.6.6 > (https://bugzilla.redhat.com/show_bug.cgi?id=1259578) that may be the cause: > > gluster volume status HDD1 > Locking failed on 10.99.99.1. Please check log file for details. > Locking failed on 10.99.99.3. Please check log file for details. > > [2016-01-27 14:52:11.268901] W > [marker-quota.c:1452:mq_release_parent_lock] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] > (--> > /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] > ))))) 0-HDD1-marker: An operation during quota updation of path > (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) > failed (Invalid argument) > [2016-01-27 14:52:11.284504] W > [marker-quota.c:1452:mq_release_parent_lock] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] > (--> > /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] > ))))) 0-HDD1-marker: An operation during quota updation of path > (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) > failed (Invalid argument) > [2016-01-27 14:52:11.288115] W > [marker-quota.c:1452:mq_release_parent_lock] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] > (--> > /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] > ))))) 0-HDD1-marker: An operation during quota updation of path > (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) > failed (Invalid argument) > [2016-01-27 14:52:11.292966] W > [marker-quota.c:1452:mq_release_parent_lock] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] > (--> > /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] > (--> > /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] > ))))) 0-HDD1-marker: An operation during quota updation of path > (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) > failed (Invalid argument) > > > > ** > > Met vriendelijke groet, With kind regards, > > Jorick Astrego > * > Netbulae Virtualization Experts * > ------------------------------------------------------------------------ > Tel: 053 20 30 270 info at netbulae.eu Staalsteden 4-3A KvK 08198180 > Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
Jorick Astrego
2016-Feb-03 14:19 UTC
[Gluster-users] Quota issue after upgrading from 3.6.4 to 3.6.6
On 01/27/2016 05:44 PM, Atin Mukherjee wrote:> Please share the glusterd log files along with cmd history log file from > all the nodes. > > Thanks, > Atin > > On 01/27/2016 08:33 PM, Jorick Astrego wrote: >> Hi, >> >> After upgrading one of a 3 node glusterfs cluster to 3.6.6, I cannot get >> the HDD-1 volume to start properly on this server. The other servers are >> still running 3.6.4 because I don't know if they will all fail after the >> upgrade. There appears to be an issue with the quota fixed in 3.6.6 >> (https://bugzilla.redhat.com/show_bug.cgi?id=1259578) that may be the cause: >> >> gluster volume status HDD1 >> Locking failed on 10.99.99.1. Please check log file for details. >> Locking failed on 10.99.99.3. Please check log file for details. >> >> [2016-01-27 14:52:11.268901] W >> [marker-quota.c:1452:mq_release_parent_lock] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] >> (--> >> /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] >> ))))) 0-HDD1-marker: An operation during quota updation of path >> (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) >> failed (Invalid argument) >> [2016-01-27 14:52:11.284504] W >> [marker-quota.c:1452:mq_release_parent_lock] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] >> (--> >> /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] >> ))))) 0-HDD1-marker: An operation during quota updation of path >> (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) >> failed (Invalid argument) >> [2016-01-27 14:52:11.288115] W >> [marker-quota.c:1452:mq_release_parent_lock] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] >> (--> >> /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] >> ))))) 0-HDD1-marker: An operation during quota updation of path >> (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) >> failed (Invalid argument) >> [2016-01-27 14:52:11.292966] W >> [marker-quota.c:1452:mq_release_parent_lock] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f869c2094f6] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_release_parent_lock+0x271)[0x7f868a2b0bd1] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/marker.so(mq_fetch_child_size_and_contri+0x30b)[0x7f868a2b1e1b] >> (--> >> /lib64/libglusterfs.so.0(default_setxattr_cbk+0xa3)[0x7f869c210bc3] >> (--> >> /usr/lib64/glusterfs/3.6.6/xlator/features/access-control.so(posix_acl_setxattr_cbk+0xa3)[0x7f868acfb743] >> ))))) 0-HDD1-marker: An operation during quota updation of path >> (/52884f44-75f5-45ab-b909-a547e7c1c6f7/images/ce445275-4aa5-4f66-ad80-44259b425d98/33fd39ac-abbb-42df-8c9f-639c93e8c00d) >> failed (Invalid argument) >> >> >>Hi, Any progress on this? I hope you received the logs last week. Currently we have a lot of stability problems with the cluster after I tried a downgrade to the original version without resolving the errors. The node with the errors is still up as far as oVirt sees it and some hosts connect to it. After a couple of hours VM's start to pause with unkown storage errors. What would be the best aproach as we now have a degraded storage cluster? Wipe and reinitialize the node or are there things I could try first? Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info at netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ---------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160203/57811043/attachment.html>