Steve Dainard
2016-Mar-02 18:37 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
>From the the client side logs I can see version info on mount:Final graph: +------------------------------------------------------------------------------+ 1: volume storage-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /storage 6: option client-version 3.7.6 7: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host 10.0.231.50 11: option remote-subvolume /mnt/raid6-storage/storage 12: option transport-type socket 13: option send-gids true 14: end-volume 15: 16: volume storage-client-1 17: type protocol/client 18: option clnt-lk-version 1 19: option volfile-checksum 0 20: option volfile-key /storage 21: option client-version 3.7.6 22: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 23: option fops-version 1298437 24: option ping-timeout 42 25: option remote-host 10.0.231.51 26: option remote-subvolume /mnt/raid6-storage/storage 27: option transport-type socket 28: option send-gids true 29: end-volume 30: 31: volume storage-client-2 32: type protocol/client 33: option clnt-lk-version 1 34: option volfile-checksum 0 35: option volfile-key /storage 36: option client-version 3.7.6 37: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 38: option fops-version 1298437 39: option ping-timeout 42 40: option remote-host 10.0.231.52 41: option remote-subvolume /mnt/raid6-storage/storage 42: option transport-type socket 43: option send-gids true 44: end-volume 45: 46: volume storage-client-3 47: type protocol/client 48: option clnt-lk-version 1 49: option volfile-checksum 0 50: option volfile-key /storage 51: option client-version 3.7.6 52: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 53: option fops-version 1298437 54: option ping-timeout 42 55: option remote-host 10.0.231.53 56: option remote-subvolume /mnt/raid6-storage/storage 57: option transport-type socket 58: option send-gids true 59: end-volume 60: 61: volume storage-client-4 62: type protocol/client 63: option ping-timeout 42 64: option remote-host 10.0.231.54 65: option remote-subvolume /mnt/raid6-storage/storage 66: option transport-type socket 67: option send-gids true 68: end-volume 69: 70: volume storage-client-5 71: type protocol/client 72: option ping-timeout 42 73: option remote-host 10.0.231.55 74: option remote-subvolume /mnt/raid6-storage/storage 75: option transport-type socket 76: option send-gids true 77: end-volume 78: 79: volume storage-dht 80: type cluster/distribute 81: subvolumes storage-client-0 storage-client-1 storage-client-2 storage-client-3 storage-client-4 storage-client-5 82: end-volume But not the client op-version, how can I retrieve this info? Thanks On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote:> Hi Steve, > > Which version you have upgraded client, could you tell us client > op-version after upgrade ? > > > have you upgraded all of your clients ? > > > Thanks, > Gaurav > > > ----- Original Message ----- > From: "Steve Dainard" <sdainard at spd1.com> > To: "gluster-users at gluster.org List" <gluster-users at gluster.org> > Sent: Wednesday, March 2, 2016 1:10:27 AM > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more > connected clients cannot support the feature being set > > Gluster 3.7.6 > 'storage' is a distributed volume > > # gluster volume set storage rebal-throttle lazy > volume set: failed: One or more connected clients cannot support the > feature being set. These clients need to be upgraded or disconnected before > running this command again > > I found a client connected using version 3.6.7 so I upgraded & > umount/mount the gluster volume on the client but I'm still getting this > error. > > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v > 3.7.6 and I get a few returns from the client above, all dated last week. > > I've run 'gluster volume status storage clients' and checked the connected > clients manually, they're all running 3.7.6. > > /var/log/gluster/etc-glusterfs-glusterd.vol.log: > [2016-03-01 19:23:20.180821] E [MSGID: 106022] > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] > 0-management: One or more c > lients don't support the required op-version > [2016-03-01 19:23:20.180853] E [MSGID: 106301] > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of > operation 'Volume Set' fa > iled on localhost : One or more connected clients cannot support the > feature being set. These clients need to be upgraded or disconnected be > fore running this command again > > Also tried setting the diagnostics.brick-log-level logging level and got > the same error. > > /var/lib/glusterd/vols/storage/info: > type=0 > count=6 > status=1 > sub_count=0 > stripe_count=1 > replica_count=1 > disperse_count=0 > redundancy_count=0 > version=26 > transport-type=0 > volume-id=26d355cb-c486-481f-ac16-e25390e73775 > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c > password> op-version=3 > client-op-version=3 > quota-version=1 > parent_volname=N/A > restored_from_snap=00000000-0000-0000-0000-000000000000 > snap-max-hard-limit=256 > features.quota-deem-statfs=on > features.inode-quota=on > diagnostics.brick-log-level=WARNING > features.quota=on > performance.readdir-ahead=on > performance.cache-size=1GB > performance.stat-prefetch=on > brick-0=10.0.231.50:-mnt-raid6-storage-storage > brick-1=10.0.231.51:-mnt-raid6-storage-storage > brick-2=10.0.231.52:-mnt-raid6-storage-storage > brick-3=10.0.231.53:-mnt-raid6-storage-storage > brick-4=10.0.231.54:-mnt-raid6-storage-storage > brick-5=10.0.231.55:-mnt-raid6-storage-storage > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160302/92d39f6c/attachment.html>
Atin Mukherjee
2016-Mar-03 01:37 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
Could you share glusterd statedump file? Run kill -SIGUSR1 $(pidof glusterd) and post that there would be a statedump file created in /var/run/gluster. -Atin Sent from one plus one On 03-Mar-2016 12:07 am, "Steve Dainard" <sdainard at spd1.com> wrote:> From the the client side logs I can see version info on mount: > > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume storage-client-0 > 2: type protocol/client > 3: option clnt-lk-version 1 > 4: option volfile-checksum 0 > 5: option volfile-key /storage > 6: option client-version 3.7.6 > 7: option process-uuid > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 > 8: option fops-version 1298437 > 9: option ping-timeout 42 > 10: option remote-host 10.0.231.50 > 11: option remote-subvolume /mnt/raid6-storage/storage > 12: option transport-type socket > 13: option send-gids true > 14: end-volume > 15: > 16: volume storage-client-1 > 17: type protocol/client > 18: option clnt-lk-version 1 > 19: option volfile-checksum 0 > 20: option volfile-key /storage > 21: option client-version 3.7.6 > 22: option process-uuid > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 > 23: option fops-version 1298437 > 24: option ping-timeout 42 > 25: option remote-host 10.0.231.51 > 26: option remote-subvolume /mnt/raid6-storage/storage > 27: option transport-type socket > 28: option send-gids true > 29: end-volume > 30: > 31: volume storage-client-2 > 32: type protocol/client > 33: option clnt-lk-version 1 > 34: option volfile-checksum 0 > 35: option volfile-key /storage > 36: option client-version 3.7.6 > 37: option process-uuid > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 > 38: option fops-version 1298437 > 39: option ping-timeout 42 > 40: option remote-host 10.0.231.52 > 41: option remote-subvolume /mnt/raid6-storage/storage > 42: option transport-type socket > 43: option send-gids true > 44: end-volume > 45: > 46: volume storage-client-3 > 47: type protocol/client > 48: option clnt-lk-version 1 > 49: option volfile-checksum 0 > 50: option volfile-key /storage > 51: option client-version 3.7.6 > 52: option process-uuid > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 > 53: option fops-version 1298437 > 54: option ping-timeout 42 > 55: option remote-host 10.0.231.53 > 56: option remote-subvolume /mnt/raid6-storage/storage > 57: option transport-type socket > 58: option send-gids true > 59: end-volume > 60: > 61: volume storage-client-4 > 62: type protocol/client > 63: option ping-timeout 42 > 64: option remote-host 10.0.231.54 > 65: option remote-subvolume /mnt/raid6-storage/storage > 66: option transport-type socket > 67: option send-gids true > 68: end-volume > 69: > 70: volume storage-client-5 > 71: type protocol/client > 72: option ping-timeout 42 > 73: option remote-host 10.0.231.55 > 74: option remote-subvolume /mnt/raid6-storage/storage > 75: option transport-type socket > 76: option send-gids true > 77: end-volume > 78: > 79: volume storage-dht > 80: type cluster/distribute > 81: subvolumes storage-client-0 storage-client-1 storage-client-2 > storage-client-3 storage-client-4 storage-client-5 > 82: end-volume > > > But not the client op-version, how can I retrieve this info? > > Thanks > > On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote: > >> Hi Steve, >> >> Which version you have upgraded client, could you tell us client >> op-version after upgrade ? >> >> >> have you upgraded all of your clients ? >> >> >> Thanks, >> Gaurav >> >> >> ----- Original Message ----- >> From: "Steve Dainard" <sdainard at spd1.com> >> To: "gluster-users at gluster.org List" <gluster-users at gluster.org> >> Sent: Wednesday, March 2, 2016 1:10:27 AM >> Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more >> connected clients cannot support the feature being set >> >> Gluster 3.7.6 >> 'storage' is a distributed volume >> >> # gluster volume set storage rebal-throttle lazy >> volume set: failed: One or more connected clients cannot support the >> feature being set. These clients need to be upgraded or disconnected before >> running this command again >> >> I found a client connected using version 3.6.7 so I upgraded & >> umount/mount the gluster volume on the client but I'm still getting this >> error. >> >> I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep >> -v 3.7.6 and I get a few returns from the client above, all dated last week. >> >> I've run 'gluster volume status storage clients' and checked the >> connected clients manually, they're all running 3.7.6. >> >> /var/log/gluster/etc-glusterfs-glusterd.vol.log: >> [2016-03-01 19:23:20.180821] E [MSGID: 106022] >> [glusterd-utils.c:10154:glusterd_check_client_op_version_support] >> 0-management: One or more c >> lients don't support the required op-version >> [2016-03-01 19:23:20.180853] E [MSGID: 106301] >> [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of >> operation 'Volume Set' fa >> iled on localhost : One or more connected clients cannot support the >> feature being set. These clients need to be upgraded or disconnected be >> fore running this command again >> >> Also tried setting the diagnostics.brick-log-level logging level and got >> the same error. >> >> /var/lib/glusterd/vols/storage/info: >> type=0 >> count=6 >> status=1 >> sub_count=0 >> stripe_count=1 >> replica_count=1 >> disperse_count=0 >> redundancy_count=0 >> version=26 >> transport-type=0 >> volume-id=26d355cb-c486-481f-ac16-e25390e73775 >> username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c >> password>> op-version=3 >> client-op-version=3 >> quota-version=1 >> parent_volname=N/A >> restored_from_snap=00000000-0000-0000-0000-000000000000 >> snap-max-hard-limit=256 >> features.quota-deem-statfs=on >> features.inode-quota=on >> diagnostics.brick-log-level=WARNING >> features.quota=on >> performance.readdir-ahead=on >> performance.cache-size=1GB >> performance.stat-prefetch=on >> brick-0=10.0.231.50:-mnt-raid6-storage-storage >> brick-1=10.0.231.51:-mnt-raid6-storage-storage >> brick-2=10.0.231.52:-mnt-raid6-storage-storage >> brick-3=10.0.231.53:-mnt-raid6-storage-storage >> brick-4=10.0.231.54:-mnt-raid6-storage-storage >> brick-5=10.0.231.55:-mnt-raid6-storage-storage >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160303/c43d505f/attachment.html>
Gaurav Garg
2016-Mar-03 06:29 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
Hi Steve, As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd) command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file. Thanks, ~Gaurav ----- Original Message ----- From: "Steve Dainard" <sdainard at spd1.com> To: "Gaurav Garg" <ggarg at redhat.com> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org> Sent: Thursday, March 3, 2016 12:07:25 AM Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set>From the the client side logs I can see version info on mount:Final graph: +------------------------------------------------------------------------------+ 1: volume storage-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /storage 6: option client-version 3.7.6 7: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host 10.0.231.50 11: option remote-subvolume /mnt/raid6-storage/storage 12: option transport-type socket 13: option send-gids true 14: end-volume 15: 16: volume storage-client-1 17: type protocol/client 18: option clnt-lk-version 1 19: option volfile-checksum 0 20: option volfile-key /storage 21: option client-version 3.7.6 22: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 23: option fops-version 1298437 24: option ping-timeout 42 25: option remote-host 10.0.231.51 26: option remote-subvolume /mnt/raid6-storage/storage 27: option transport-type socket 28: option send-gids true 29: end-volume 30: 31: volume storage-client-2 32: type protocol/client 33: option clnt-lk-version 1 34: option volfile-checksum 0 35: option volfile-key /storage 36: option client-version 3.7.6 37: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 38: option fops-version 1298437 39: option ping-timeout 42 40: option remote-host 10.0.231.52 41: option remote-subvolume /mnt/raid6-storage/storage 42: option transport-type socket 43: option send-gids true 44: end-volume 45: 46: volume storage-client-3 47: type protocol/client 48: option clnt-lk-version 1 49: option volfile-checksum 0 50: option volfile-key /storage 51: option client-version 3.7.6 52: option process-uuid template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 53: option fops-version 1298437 54: option ping-timeout 42 55: option remote-host 10.0.231.53 56: option remote-subvolume /mnt/raid6-storage/storage 57: option transport-type socket 58: option send-gids true 59: end-volume 60: 61: volume storage-client-4 62: type protocol/client 63: option ping-timeout 42 64: option remote-host 10.0.231.54 65: option remote-subvolume /mnt/raid6-storage/storage 66: option transport-type socket 67: option send-gids true 68: end-volume 69: 70: volume storage-client-5 71: type protocol/client 72: option ping-timeout 42 73: option remote-host 10.0.231.55 74: option remote-subvolume /mnt/raid6-storage/storage 75: option transport-type socket 76: option send-gids true 77: end-volume 78: 79: volume storage-dht 80: type cluster/distribute 81: subvolumes storage-client-0 storage-client-1 storage-client-2 storage-client-3 storage-client-4 storage-client-5 82: end-volume But not the client op-version, how can I retrieve this info? Thanks On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote:> Hi Steve, > > Which version you have upgraded client, could you tell us client > op-version after upgrade ? > > > have you upgraded all of your clients ? > > > Thanks, > Gaurav > > > ----- Original Message ----- > From: "Steve Dainard" <sdainard at spd1.com> > To: "gluster-users at gluster.org List" <gluster-users at gluster.org> > Sent: Wednesday, March 2, 2016 1:10:27 AM > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more > connected clients cannot support the feature being set > > Gluster 3.7.6 > 'storage' is a distributed volume > > # gluster volume set storage rebal-throttle lazy > volume set: failed: One or more connected clients cannot support the > feature being set. These clients need to be upgraded or disconnected before > running this command again > > I found a client connected using version 3.6.7 so I upgraded & > umount/mount the gluster volume on the client but I'm still getting this > error. > > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v > 3.7.6 and I get a few returns from the client above, all dated last week. > > I've run 'gluster volume status storage clients' and checked the connected > clients manually, they're all running 3.7.6. > > /var/log/gluster/etc-glusterfs-glusterd.vol.log: > [2016-03-01 19:23:20.180821] E [MSGID: 106022] > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] > 0-management: One or more c > lients don't support the required op-version > [2016-03-01 19:23:20.180853] E [MSGID: 106301] > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of > operation 'Volume Set' fa > iled on localhost : One or more connected clients cannot support the > feature being set. These clients need to be upgraded or disconnected be > fore running this command again > > Also tried setting the diagnostics.brick-log-level logging level and got > the same error. > > /var/lib/glusterd/vols/storage/info: > type=0 > count=6 > status=1 > sub_count=0 > stripe_count=1 > replica_count=1 > disperse_count=0 > redundancy_count=0 > version=26 > transport-type=0 > volume-id=26d355cb-c486-481f-ac16-e25390e73775 > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c > password> op-version=3 > client-op-version=3 > quota-version=1 > parent_volname=N/A > restored_from_snap=00000000-0000-0000-0000-000000000000 > snap-max-hard-limit=256 > features.quota-deem-statfs=on > features.inode-quota=on > diagnostics.brick-log-level=WARNING > features.quota=on > performance.readdir-ahead=on > performance.cache-size=1GB > performance.stat-prefetch=on > brick-0=10.0.231.50:-mnt-raid6-storage-storage > brick-1=10.0.231.51:-mnt-raid6-storage-storage > brick-2=10.0.231.52:-mnt-raid6-storage-storage > brick-3=10.0.231.53:-mnt-raid6-storage-storage > brick-4=10.0.231.54:-mnt-raid6-storage-storage > brick-5=10.0.231.55:-mnt-raid6-storage-storage > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >