David Gibbons
2014-Dec-01 13:05 UTC
[Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer working
Certainly, thank you for your response: Quotad is running on all nodes:> [root at gfs-a-1 ~]# ps aux | grep quotad > > root 3004 0.0 0.4 241368 68552 ? Ssl Nov30 0:07 >> /usr/local/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p >> /var/lib/glusterd/quotad/run/quotad.pid -l >> /usr/local/var/log/glusterfs/quotad.log -S >> /var/run/9d02605105ef0e74d913a4671c1143a1.socket --xlator-option >> *replicate*.data-self-heal=off --xlator-option >> *replicate*.metadata-self-heal=off --xlator-option >> *replicate*.entry-self-heal=off > >And the relevant output from gluster volume status shares per your request:> [root at gfs-a-1 ~]# gluster volume status shares | grep QuotaQuota Daemon on localhost N/A Y 3004 Quota Daemon on gfs-a-3 N/A Y> 32307Quota Daemon on gfs-a-4 N/A Y> 10818Quota Daemon on gfs-a-2 N/A Y> 12292No log entries are created in /var/log/glusterfs/quotad.log when I run a quota list; all of the log entries are from yesterday. They do indicate a version mis-match, although I can't seem to locate where that version is specified:> [2014-11-30 13:21:55.173081] I >> [client-handshake.c:1474:client_setvolume_cbk] 0-shares-client-14: Server >> and Client lk-version numbers are not same, reopening the fds > > [2014-11-30 13:21:55.173170] I >> [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-14: >> Server lk version = 1 > > [2014-11-30 13:21:55.178739] I [rpc-clnt.c:1729:rpc_clnt_reconfig] >> 0-shares-client-9: changing port to 49154 (from 0) > > [2014-11-30 13:21:55.181170] I >> [client-handshake.c:1677:select_server_supported_programs] >> 0-shares-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) > > [2014-11-30 13:21:55.181386] I >> [client-handshake.c:1462:client_setvolume_cbk] 0-shares-client-9: Connected >> to 172.16.10.13:49154, attached to remote volume >> '/mnt/a-3-shares-brick-3/brick'. > > [2014-11-30 13:21:55.181401] I >> [client-handshake.c:1474:client_setvolume_cbk] 0-shares-client-9: Server >> and Client lk-version numbers are not same, reopening the fds > > [2014-11-30 13:21:55.181535] I >> [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-9: >> Server lk version = 1 > >I see the operational mode for the volume as "3". I saw a non-related thread that indicated this number should be more digits on a cluster running 3.5.2. The other thread also indicated that quota may not work if the volume version number was not compatible with the quota version running on the cluster. I can't seem to find the link right now. It's almost as if the volume version did not get upgraded when the server version was upgraded. Is that possible? Cheers, Dave On Sun, Nov 30, 2014 at 11:46 PM, Krutika Dhananjay <kdhananj at redhat.com> wrote:> Hi, > > Could you confirm whether quotad (Quota Daemon) is online from the output > of `gluster volume status shares`? > > Also, could you share quota daemon's log file from the node where you > executed `quota list` command, which you will find at @ > /var/log/glusterfs/quotad.log? > > -Krutika > > ------------------------------ > > *From: *"David Gibbons" <david.c.gibbons at gmail.com> > *To: *"gluster-users" <gluster-users at gluster.org> > *Sent: *Sunday, November 30, 2014 9:57:02 PM > *Subject: *[Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no > longer working > > > Hi All, > > I performed a long-awaited upgrade from 3.4.1 to 3.5.2 today following the > instructions for an offline upgrade outlined here: > http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5 > > I ran the pre- and post- upgrade scripts as instructed, intending to move > the quotas over to the new version. The upgrade seemed to go well, the > volume is online and it appears to be functioning properly. > > When I attempt to check quotas, the list is empty: > >> [root at gfs-a-1 glusterfs]# gluster volume quota shares list >> >> Path Hard-limit Soft-limit Used >>> Available >> >> >>> -------------------------------------------------------------------------------- >> >> [root at gfs-a-1 glusterfs]# >> >> >> > And upon execution of that command, the cli.log file fills up with entries > like this. I am assuming it's one cli log entry per quota entry: > >> [2014-11-30 14:00:02.154143] W >>> [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is not >>> present in dict >> >> [2014-11-30 14:00:02.160507] W >>> [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is not >>> present in dict >> >> [2014-11-30 14:00:02.167947] W >>> [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is not >>> present in dic >> >> > So, it appears that somehow the quota database has become offline or > corrupt. Any thoughts on what I can do to resolve this? > > I have checked all of the binaries on all 4 machines in the cluster, and > they all appear to be running the correct version: > >> [root at gfs-a-1 glusterfs]# glusterfsd --version >> >> glusterfs 3.5.2 built on Nov 30 2014 08:16:37 >> >> > Cheers, > Dave > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141201/d103a527/attachment.html>
Krutika Dhananjay
2014-Dec-02 07:28 UTC
[Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer working
Hi, Are you sure the post-upgrade script ran to completion? Here is one way to confirm whether that is the case: check if the quota configured directories have an xattr called "trusted.glusterfs.quota.limit-set" set on them in the respective bricks. For example, here's what mine looks like: [root at haddock 1]# pwd /brick/1 [root at haddock 1]# getfattr -d -m . -e hex 1 # file: 1 security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000 trusted.gfid=0x57d0a561ca574d1cb0428f38d1c06e85 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000a00 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.limit-set=0x0000000006400000ffffffffffffffff trusted.glusterfs.quota.size=0x0000000000000a00 where /brick/1 is the brick directory and under it "1" is the name of one of the quota-configured directories. I believe your quota configurations are backed up at /var/tmp/glusterfs/quota-config-backup/vol_<volname> which you can use to get the quota-configured directory names. As for operating version, I think it is sufficient for it to be at 3 for the 3.5.x quota to work. -Krutika ----- Original Message -----> From: "David Gibbons" <david.c.gibbons at gmail.com> > To: "Krutika Dhananjay" <kdhananj at redhat.com> > Cc: "gluster-users" <gluster-users at gluster.org> > Sent: Monday, December 1, 2014 6:35:55 PM > Subject: Re: [Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer > working> Certainly, thank you for your response:> Quotad is running on all nodes:> > > [root at gfs-a-1 ~]# ps aux | grep quotad > > > > > > root 3004 0.0 0.4 241368 68552 ? Ssl Nov30 0:07 /usr/local/sbin/glusterfs > > > -s > > > localhost --volfile-id gluster/quotad -p > > > /var/lib/glusterd/quotad/run/quotad.pid -l > > > /usr/local/var/log/glusterfs/quotad.log -S > > > /var/run/9d02605105ef0e74d913a4671c1143a1.socket --xlator-option > > > *replicate*.data-self-heal=off --xlator-option > > > *replicate*.metadata-self-heal=off --xlator-option > > > *replicate*.entry-self-heal=off > > > > And the relevant output from gluster volume status shares per your request:> > [root at gfs-a-1 ~]# gluster volume status shares | grep Quota > > > Quota Daemon on localhost N/A Y 3004 > > > Quota Daemon on gfs-a-3 N/A Y 32307 > > > Quota Daemon on gfs-a-4 N/A Y 10818 > > > Quota Daemon on gfs-a-2 N/A Y 12292 > > No log entries are created in /var/log/glusterfs/quotad.log when I run a > quota list; all of the log entries are from yesterday. They do indicate a > version mis-match, although I can't seem to locate where that version is > specified:> > > [2014-11-30 13:21:55.173081] I > > > [client-handshake.c:1474:client_setvolume_cbk] > > > 0-shares-client-14: Server and Client lk-version numbers are not same, > > > reopening the fds > > > > > > [2014-11-30 13:21:55.173170] I > > > [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-14: > > > Server lk version = 1 > > > > > > [2014-11-30 13:21:55.178739] I [rpc-clnt.c:1729:rpc_clnt_reconfig] > > > 0-shares-client-9: changing port to 49154 (from 0) > > > > > > [2014-11-30 13:21:55.181170] I > > > [client-handshake.c:1677:select_server_supported_programs] > > > 0-shares-client-9: Using Program GlusterFS 3.3, Num (1298437), Version > > > (330) > > > > > > [2014-11-30 13:21:55.181386] I > > > [client-handshake.c:1462:client_setvolume_cbk] > > > 0-shares-client-9: Connected to 172.16.10.13:49154 , attached to remote > > > volume '/mnt/a-3-shares-brick-3/brick'. > > > > > > [2014-11-30 13:21:55.181401] I > > > [client-handshake.c:1474:client_setvolume_cbk] > > > 0-shares-client-9: Server and Client lk-version numbers are not same, > > > reopening the fds > > > > > > [2014-11-30 13:21:55.181535] I > > > [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-9: > > > Server > > > lk version = 1 > > > > I see the operational mode for the volume as "3". I saw a non-related thread > that indicated this number should be more digits on a cluster running 3.5.2. > The other thread also indicated that quota may not work if the volume > version number was not compatible with the quota version running on the > cluster. I can't seem to find the link right now.> It's almost as if the volume version did not get upgraded when the server > version was upgraded. Is that possible?> Cheers, > Dave> On Sun, Nov 30, 2014 at 11:46 PM, Krutika Dhananjay < kdhananj at redhat.com > > wrote:> > Hi, >> > Could you confirm whether quotad (Quota Daemon) is online from the output > > of > > `gluster volume status shares`? >> > Also, could you share quota daemon's log file from the node where you > > executed `quota list` command, which you will find at @ > > /var/log/glusterfs/quotad.log? >> > -Krutika >> > > From: "David Gibbons" < david.c.gibbons at gmail.com > > > > > > > To: "gluster-users" < gluster-users at gluster.org > > > > > > > Sent: Sunday, November 30, 2014 9:57:02 PM > > > > > > Subject: [Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer > > > working > > >> > > Hi All, > > >> > > I performed a long-awaited upgrade from 3.4.1 to 3.5.2 today following > > > the > > > instructions for an offline upgrade outlined here: > > > > > > http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5 > > >> > > I ran the pre- and post- upgrade scripts as instructed, intending to move > > > the > > > quotas over to the new version. The upgrade seemed to go well, the volume > > > is > > > online and it appears to be functioning properly. > > >> > > When I attempt to check quotas, the list is empty: > > >> > > > > [root at gfs-a-1 glusterfs]# gluster volume quota shares list > > > > > > > > > > > > > > > Path Hard-limit Soft-limit Used Available > > > > > > > > > > > > > > > -------------------------------------------------------------------------------- > > > > > > > > > > > > > > > [root at gfs-a-1 glusterfs]# > > > > > > > > > > > > > And upon execution of that command, the cli.log file fills up with > > > entries > > > like this. I am assuming it's one cli log entry per quota entry: > > >> > > > > [2014-11-30 14:00:02.154143] W > > > > > [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is > > > > > not > > > > > present in dict > > > > > > > > > > > > > > > [2014-11-30 14:00:02.160507] W > > > > > [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is > > > > > not > > > > > present in dict > > > > > > > > > > > > > > > [2014-11-30 14:00:02.167947] W > > > > > [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is > > > > > not > > > > > present in dic > > > > > > > > > > > > > So, it appears that somehow the quota database has become offline or > > > corrupt. > > > Any thoughts on what I can do to resolve this? > > >> > > I have checked all of the binaries on all 4 machines in the cluster, and > > > they > > > all appear to be running the correct version: > > >> > > > > [root at gfs-a-1 glusterfs]# glusterfsd --version > > > > > > > > > > > > > > > glusterfs 3.5.2 built on Nov 30 2014 08:16:37 > > > > > > > > > > > > > Cheers, > > > > > > Dave > > >> > > _______________________________________________ > > > > > > Gluster-users mailing list > > > > > > Gluster-users at gluster.org > > > > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141202/6266c396/attachment.html>