Steve Dainard
2016-Mar-03 22:05 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
FYI Gluster storage node hostnames are gluster0[1-6].
Full dump attached. I see a few clients not on 30706. Most notably the two
debian 7 servers (using packages from gluster.org) seem to be running lower
op versions than the centos7 machines (every other client in 10.0.231.0/24
subnet).
glusterd.client1.identifier=10.0.231.10:1023 <-- *debian 7*, glusterfs
3.7.6 built on Feb 4 2016 06:25:19
glusterd.client1.volname=storage
glusterd.client1.max-op-version=30603
glusterd.client1.min-op-version=1
glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
glusterd.client2.volnameglusterd.client2.max-op-version=0
glusterd.client2.min-op-version=0
glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual
mounts)
glusterd.client3.volnameglusterd.client3.max-op-version=0
glusterd.client3.min-op-version=0
glusterd.client4.identifier=10.0.231.11:1022 <---* debian 7*, glusterfs
3.7.6 built on Feb 4 2016 06:25:19
glusterd.client4.volname=storage
glusterd.client4.max-op-version=30603
glusterd.client4.min-op-version=1
glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual
mounts)
glusterd.client5.volnameglusterd.client5.max-op-version=0
glusterd.client5.min-op-version=0
glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
glusterd.client6.volnameglusterd.client6.max-op-version=0
glusterd.client6.min-op-version=0
glusterd.client7.identifier=10.0.231.50:65529
glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
glusterd.client7.max-op-version=30706
glusterd.client7.min-op-version=1
...
Debian package info:
apt-cache policy glusterfs-client
glusterfs-client:
Installed: 3.7.6-2
Candidate: 3.7.6-2
Version table:
*** 3.7.6-2 0
500
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
wheezy/main amd64 Packages
Thanks
On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at redhat.com> wrote:
> Hi Steve,
>
> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof
> glusterd) command. it will create .dump file in /var/run/gluster/
> directory. client-op-version information will be present in dump file.
>
> Thanks,
> ~Gaurav
>
> ----- Original Message -----
> From: "Steve Dainard" <sdainard at spd1.com>
> To: "Gaurav Garg" <ggarg at redhat.com>
> Cc: "gluster-users at gluster.org List" <gluster-users at
gluster.org>
> Sent: Thursday, March 3, 2016 12:07:25 AM
> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
> connected clients cannot support the feature being set
>
> From the the client side logs I can see version info on mount:
>
> Final graph:
>
>
+------------------------------------------------------------------------------+
> 1: volume storage-client-0
> 2: type protocol/client
> 3: option clnt-lk-version 1
> 4: option volfile-checksum 0
> 5: option volfile-key /storage
> 6: option client-version 3.7.6
> 7: option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
> 8: option fops-version 1298437
> 9: option ping-timeout 42
> 10: option remote-host 10.0.231.50
> 11: option remote-subvolume /mnt/raid6-storage/storage
> 12: option transport-type socket
> 13: option send-gids true
> 14: end-volume
> 15:
> 16: volume storage-client-1
> 17: type protocol/client
> 18: option clnt-lk-version 1
> 19: option volfile-checksum 0
> 20: option volfile-key /storage
> 21: option client-version 3.7.6
> 22: option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
> 23: option fops-version 1298437
> 24: option ping-timeout 42
> 25: option remote-host 10.0.231.51
> 26: option remote-subvolume /mnt/raid6-storage/storage
> 27: option transport-type socket
> 28: option send-gids true
> 29: end-volume
> 30:
> 31: volume storage-client-2
> 32: type protocol/client
> 33: option clnt-lk-version 1
> 34: option volfile-checksum 0
> 35: option volfile-key /storage
> 36: option client-version 3.7.6
> 37: option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
> 38: option fops-version 1298437
> 39: option ping-timeout 42
> 40: option remote-host 10.0.231.52
> 41: option remote-subvolume /mnt/raid6-storage/storage
> 42: option transport-type socket
> 43: option send-gids true
> 44: end-volume
> 45:
> 46: volume storage-client-3
> 47: type protocol/client
> 48: option clnt-lk-version 1
> 49: option volfile-checksum 0
> 50: option volfile-key /storage
> 51: option client-version 3.7.6
> 52: option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
> 53: option fops-version 1298437
> 54: option ping-timeout 42
> 55: option remote-host 10.0.231.53
> 56: option remote-subvolume /mnt/raid6-storage/storage
> 57: option transport-type socket
> 58: option send-gids true
> 59: end-volume
> 60:
> 61: volume storage-client-4
> 62: type protocol/client
> 63: option ping-timeout 42
> 64: option remote-host 10.0.231.54
> 65: option remote-subvolume /mnt/raid6-storage/storage
> 66: option transport-type socket
> 67: option send-gids true
> 68: end-volume
> 69:
> 70: volume storage-client-5
> 71: type protocol/client
> 72: option ping-timeout 42
> 73: option remote-host 10.0.231.55
> 74: option remote-subvolume /mnt/raid6-storage/storage
> 75: option transport-type socket
> 76: option send-gids true
> 77: end-volume
> 78:
> 79: volume storage-dht
> 80: type cluster/distribute
> 81: subvolumes storage-client-0 storage-client-1 storage-client-2
> storage-client-3 storage-client-4 storage-client-5
> 82: end-volume
>
>
> But not the client op-version, how can I retrieve this info?
>
> Thanks
>
> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com>
wrote:
>
> > Hi Steve,
> >
> > Which version you have upgraded client, could you tell us client
> > op-version after upgrade ?
> >
> >
> > have you upgraded all of your clients ?
> >
> >
> > Thanks,
> > Gaurav
> >
> >
> > ----- Original Message -----
> > From: "Steve Dainard" <sdainard at spd1.com>
> > To: "gluster-users at gluster.org List" <gluster-users at
gluster.org>
> > Sent: Wednesday, March 2, 2016 1:10:27 AM
> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
> > connected clients cannot support the feature being set
> >
> > Gluster 3.7.6
> > 'storage' is a distributed volume
> >
> > # gluster volume set storage rebal-throttle lazy
> > volume set: failed: One or more connected clients cannot support the
> > feature being set. These clients need to be upgraded or disconnected
> before
> > running this command again
> >
> > I found a client connected using version 3.6.7 so I upgraded &
> > umount/mount the gluster volume on the client but I'm still
getting this
> > error.
> >
> > I've run grep "accepted client from"
/var/log/glusterfs/bricks/* | grep
> -v
> > 3.7.6 and I get a few returns from the client above, all dated last
week.
> >
> > I've run 'gluster volume status storage clients' and
checked the
> connected
> > clients manually, they're all running 3.7.6.
> >
> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:
> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]
> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]
> > 0-management: One or more c
> > lients don't support the required op-version
> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]
> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of
> > operation 'Volume Set' fa
> > iled on localhost : One or more connected clients cannot support the
> > feature being set. These clients need to be upgraded or disconnected
be
> > fore running this command again
> >
> > Also tried setting the diagnostics.brick-log-level logging level and
got
> > the same error.
> >
> > /var/lib/glusterd/vols/storage/info:
> > type=0
> > count=6
> > status=1
> > sub_count=0
> > stripe_count=1
> > replica_count=1
> > disperse_count=0
> > redundancy_count=0
> > version=26
> > transport-type=0
> > volume-id=26d355cb-c486-481f-ac16-e25390e73775
> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
> > password> > op-version=3
> > client-op-version=3
> > quota-version=1
> > parent_volname=N/A
> > restored_from_snap=00000000-0000-0000-0000-000000000000
> > snap-max-hard-limit=256
> > features.quota-deem-statfs=on
> > features.inode-quota=on
> > diagnostics.brick-log-level=WARNING
> > features.quota=on
> > performance.readdir-ahead=on
> > performance.cache-size=1GB
> > performance.stat-prefetch=on
> > brick-0=10.0.231.50:-mnt-raid6-storage-storage
> > brick-1=10.0.231.51:-mnt-raid6-storage-storage
> > brick-2=10.0.231.52:-mnt-raid6-storage-storage
> > brick-3=10.0.231.53:-mnt-raid6-storage-storage
> > brick-4=10.0.231.54:-mnt-raid6-storage-storage
> > brick-5=10.0.231.55:-mnt-raid6-storage-storage
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160303/f0ca9fbf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: glusterdump.1590.dump.1457041019
Type: application/octet-stream
Size: 25133 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160303/f0ca9fbf/attachment.obj>
Atin Mukherjee
2016-Mar-04 01:28 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
-Atin Sent from one plus one On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard at spd1.com> wrote:> > FYI Gluster storage node hostnames are gluster0[1-6]. > > Full dump attached. I see a few clients not on 30706. Most notably thetwo debian 7 servers (using packages from gluster.org) seem to be running lower op versions than the centos7 machines (every other client in 10.0.231.0/24 subnet).> > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs3.7.6 built on Feb 4 2016 06:25:19> glusterd.client1.volname=storage > glusterd.client1.max-op-version=30603This is running with 3.6.3 and that's why volume set fails. Unmount this client and upgrade and remount it back.> glusterd.client1.min-op-version=1 > > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)> glusterd.client2.volname> glusterd.client2.max-op-version=0 > glusterd.client2.min-op-version=0 > > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actualmounts)> glusterd.client3.volname> glusterd.client3.max-op-version=0 > glusterd.client3.min-op-version=0 > > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs3.7.6 built on Feb 4 2016 06:25:19> glusterd.client4.volname=storage > glusterd.client4.max-op-version=30603 > glusterd.client4.min-op-version=1 > > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actualmounts)> glusterd.client5.volname> glusterd.client5.max-op-version=0 > glusterd.client5.min-op-version=0 > > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)> glusterd.client6.volname> glusterd.client6.max-op-version=0 > glusterd.client6.min-op-version=0 > > glusterd.client7.identifier=10.0.231.50:65529 >glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage> glusterd.client7.max-op-version=30706 > glusterd.client7.min-op-version=1 > > ... > > Debian package info: > apt-cache policy glusterfs-client > glusterfs-client: > Installed: 3.7.6-2 > Candidate: 3.7.6-2 > Version table: > *** 3.7.6-2 0 > 500http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ wheezy/main amd64 Packages> > > Thanks > > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >> Hi Steve, >> >> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidofglusterd) command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.>> >> Thanks, >> ~Gaurav >> >> ----- Original Message ----- >> From: "Steve Dainard" <sdainard at spd1.com> >> To: "Gaurav Garg" <ggarg at redhat.com> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org> >> Sent: Thursday, March 3, 2016 12:07:25 AM >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One ormore connected clients cannot support the feature being set>> >> From the the client side logs I can see version info on mount: >> >> Final graph: >>+------------------------------------------------------------------------------+>> 1: volume storage-client-0 >> 2: type protocol/client >> 3: option clnt-lk-version 1 >> 4: option volfile-checksum 0 >> 5: option volfile-key /storage >> 6: option client-version 3.7.6 >> 7: option process-uuid >>template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0>> 8: option fops-version 1298437 >> 9: option ping-timeout 42 >> 10: option remote-host 10.0.231.50 >> 11: option remote-subvolume /mnt/raid6-storage/storage >> 12: option transport-type socket >> 13: option send-gids true >> 14: end-volume >> 15: >> 16: volume storage-client-1 >> 17: type protocol/client >> 18: option clnt-lk-version 1 >> 19: option volfile-checksum 0 >> 20: option volfile-key /storage >> 21: option client-version 3.7.6 >> 22: option process-uuid >>template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0>> 23: option fops-version 1298437 >> 24: option ping-timeout 42 >> 25: option remote-host 10.0.231.51 >> 26: option remote-subvolume /mnt/raid6-storage/storage >> 27: option transport-type socket >> 28: option send-gids true >> 29: end-volume >> 30: >> 31: volume storage-client-2 >> 32: type protocol/client >> 33: option clnt-lk-version 1 >> 34: option volfile-checksum 0 >> 35: option volfile-key /storage >> 36: option client-version 3.7.6 >> 37: option process-uuid >>template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0>> 38: option fops-version 1298437 >> 39: option ping-timeout 42 >> 40: option remote-host 10.0.231.52 >> 41: option remote-subvolume /mnt/raid6-storage/storage >> 42: option transport-type socket >> 43: option send-gids true >> 44: end-volume >> 45: >> 46: volume storage-client-3 >> 47: type protocol/client >> 48: option clnt-lk-version 1 >> 49: option volfile-checksum 0 >> 50: option volfile-key /storage >> 51: option client-version 3.7.6 >> 52: option process-uuid >>template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0>> 53: option fops-version 1298437 >> 54: option ping-timeout 42 >> 55: option remote-host 10.0.231.53 >> 56: option remote-subvolume /mnt/raid6-storage/storage >> 57: option transport-type socket >> 58: option send-gids true >> 59: end-volume >> 60: >> 61: volume storage-client-4 >> 62: type protocol/client >> 63: option ping-timeout 42 >> 64: option remote-host 10.0.231.54 >> 65: option remote-subvolume /mnt/raid6-storage/storage >> 66: option transport-type socket >> 67: option send-gids true >> 68: end-volume >> 69: >> 70: volume storage-client-5 >> 71: type protocol/client >> 72: option ping-timeout 42 >> 73: option remote-host 10.0.231.55 >> 74: option remote-subvolume /mnt/raid6-storage/storage >> 75: option transport-type socket >> 76: option send-gids true >> 77: end-volume >> 78: >> 79: volume storage-dht >> 80: type cluster/distribute >> 81: subvolumes storage-client-0 storage-client-1 storage-client-2 >> storage-client-3 storage-client-4 storage-client-5 >> 82: end-volume >> >> >> But not the client op-version, how can I retrieve this info? >> >> Thanks >> >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >> > Hi Steve, >> > >> > Which version you have upgraded client, could you tell us client >> > op-version after upgrade ? >> > >> > >> > have you upgraded all of your clients ? >> > >> > >> > Thanks, >> > Gaurav >> > >> > >> > ----- Original Message ----- >> > From: "Steve Dainard" <sdainard at spd1.com> >> > To: "gluster-users at gluster.org List" <gluster-users at gluster.org> >> > Sent: Wednesday, March 2, 2016 1:10:27 AM >> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more >> > connected clients cannot support the feature being set >> > >> > Gluster 3.7.6 >> > 'storage' is a distributed volume >> > >> > # gluster volume set storage rebal-throttle lazy >> > volume set: failed: One or more connected clients cannot support the >> > feature being set. These clients need to be upgraded or disconnectedbefore>> > running this command again >> > >> > I found a client connected using version 3.6.7 so I upgraded & >> > umount/mount the gluster volume on the client but I'm still gettingthis>> > error. >> > >> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* |grep -v>> > 3.7.6 and I get a few returns from the client above, all dated lastweek.>> > >> > I've run 'gluster volume status storage clients' and checked theconnected>> > clients manually, they're all running 3.7.6. >> > >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log: >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022] >> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] >> > 0-management: One or more c >> > lients don't support the required op-version >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301] >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of >> > operation 'Volume Set' fa >> > iled on localhost : One or more connected clients cannot support the >> > feature being set. These clients need to be upgraded or disconnected be >> > fore running this command again >> > >> > Also tried setting the diagnostics.brick-log-level logging level andgot>> > the same error. >> > >> > /var/lib/glusterd/vols/storage/info: >> > type=0 >> > count=6 >> > status=1 >> > sub_count=0 >> > stripe_count=1 >> > replica_count=1 >> > disperse_count=0 >> > redundancy_count=0 >> > version=26 >> > transport-type=0 >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775 >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c >> > password>> > op-version=3 >> > client-op-version=3 >> > quota-version=1 >> > parent_volname=N/A >> > restored_from_snap=00000000-0000-0000-0000-000000000000 >> > snap-max-hard-limit=256 >> > features.quota-deem-statfs=on >> > features.inode-quota=on >> > diagnostics.brick-log-level=WARNING >> > features.quota=on >> > performance.readdir-ahead=on >> > performance.cache-size=1GB >> > performance.stat-prefetch=on >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160304/b20ea20b/attachment.html>