Steve Dainard
2016-Mar-04 16:53 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
Except that even when I unmount gluster from both Debian clients, I'm still getting the same error, and a dump of those clients after unmounting still shows: glusterd.client1.identifier=10.0.231.10:1023 glusterd.client1.volname=storage glusterd.client1.max-op-version=30603 glusterd.client1.min-op-version=1 glusterd.client4.identifier=10.0.231.11:1022 glusterd.client4.volname=storage glusterd.client4.max-op-version=30603 glusterd.client4.min-op-version=1 On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherjee83 at gmail.com> wrote:> -Atin > Sent from one plus one > On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard at spd1.com> wrote: > > > > FYI Gluster storage node hostnames are gluster0[1-6]. > > > > Full dump attached. I see a few clients not on 30706. Most notably the > two debian 7 servers (using packages from gluster.org) seem to be running > lower op versions than the centos7 machines (every other client in > 10.0.231.0/24 subnet). > > > > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs > 3.7.6 built on Feb 4 2016 06:25:19 > > glusterd.client1.volname=storage > > glusterd.client1.max-op-version=30603 > This is running with 3.6.3 and that's why volume set fails. Unmount this > client and upgrade and remount it back. > > > glusterd.client1.min-op-version=1 > > > > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02 > (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) > > glusterd.client2.volname> > glusterd.client2.max-op-version=0 > > glusterd.client2.min-op-version=0 > > > > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual > mounts) > > glusterd.client3.volname> > glusterd.client3.max-op-version=0 > > glusterd.client3.min-op-version=0 > > > > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs > 3.7.6 built on Feb 4 2016 06:25:19 > > glusterd.client4.volname=storage > > glusterd.client4.max-op-version=30603 > > glusterd.client4.min-op-version=1 > > > > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual > mounts) > > glusterd.client5.volname> > glusterd.client5.max-op-version=0 > > glusterd.client5.min-op-version=0 > > > > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04 > (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) > > glusterd.client6.volname> > glusterd.client6.max-op-version=0 > > glusterd.client6.min-op-version=0 > > > > glusterd.client7.identifier=10.0.231.50:65529 > > > glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage > > glusterd.client7.max-op-version=30706 > > glusterd.client7.min-op-version=1 > > > > ... > > > > Debian package info: > > apt-cache policy glusterfs-client > > glusterfs-client: > > Installed: 3.7.6-2 > > Candidate: 3.7.6-2 > > Version table: > > *** 3.7.6-2 0 > > 500 > http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ > wheezy/main amd64 Packages > > > > > > Thanks > > > > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at redhat.com> wrote: > >> > >> Hi Steve, > >> > >> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof > glusterd) command. it will create .dump file in /var/run/gluster/ > directory. client-op-version information will be present in dump file. > >> > >> Thanks, > >> ~Gaurav > >> > >> ----- Original Message ----- > >> From: "Steve Dainard" <sdainard at spd1.com> > >> To: "Gaurav Garg" <ggarg at redhat.com> > >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org> > >> Sent: Thursday, March 3, 2016 12:07:25 AM > >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or > more connected clients cannot support the feature being set > >> > >> From the the client side logs I can see version info on mount: > >> > >> Final graph: > >> > +------------------------------------------------------------------------------+ > >> 1: volume storage-client-0 > >> 2: type protocol/client > >> 3: option clnt-lk-version 1 > >> 4: option volfile-checksum 0 > >> 5: option volfile-key /storage > >> 6: option client-version 3.7.6 > >> 7: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 > >> 8: option fops-version 1298437 > >> 9: option ping-timeout 42 > >> 10: option remote-host 10.0.231.50 > >> 11: option remote-subvolume /mnt/raid6-storage/storage > >> 12: option transport-type socket > >> 13: option send-gids true > >> 14: end-volume > >> 15: > >> 16: volume storage-client-1 > >> 17: type protocol/client > >> 18: option clnt-lk-version 1 > >> 19: option volfile-checksum 0 > >> 20: option volfile-key /storage > >> 21: option client-version 3.7.6 > >> 22: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 > >> 23: option fops-version 1298437 > >> 24: option ping-timeout 42 > >> 25: option remote-host 10.0.231.51 > >> 26: option remote-subvolume /mnt/raid6-storage/storage > >> 27: option transport-type socket > >> 28: option send-gids true > >> 29: end-volume > >> 30: > >> 31: volume storage-client-2 > >> 32: type protocol/client > >> 33: option clnt-lk-version 1 > >> 34: option volfile-checksum 0 > >> 35: option volfile-key /storage > >> 36: option client-version 3.7.6 > >> 37: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 > >> 38: option fops-version 1298437 > >> 39: option ping-timeout 42 > >> 40: option remote-host 10.0.231.52 > >> 41: option remote-subvolume /mnt/raid6-storage/storage > >> 42: option transport-type socket > >> 43: option send-gids true > >> 44: end-volume > >> 45: > >> 46: volume storage-client-3 > >> 47: type protocol/client > >> 48: option clnt-lk-version 1 > >> 49: option volfile-checksum 0 > >> 50: option volfile-key /storage > >> 51: option client-version 3.7.6 > >> 52: option process-uuid > >> > template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 > >> 53: option fops-version 1298437 > >> 54: option ping-timeout 42 > >> 55: option remote-host 10.0.231.53 > >> 56: option remote-subvolume /mnt/raid6-storage/storage > >> 57: option transport-type socket > >> 58: option send-gids true > >> 59: end-volume > >> 60: > >> 61: volume storage-client-4 > >> 62: type protocol/client > >> 63: option ping-timeout 42 > >> 64: option remote-host 10.0.231.54 > >> 65: option remote-subvolume /mnt/raid6-storage/storage > >> 66: option transport-type socket > >> 67: option send-gids true > >> 68: end-volume > >> 69: > >> 70: volume storage-client-5 > >> 71: type protocol/client > >> 72: option ping-timeout 42 > >> 73: option remote-host 10.0.231.55 > >> 74: option remote-subvolume /mnt/raid6-storage/storage > >> 75: option transport-type socket > >> 76: option send-gids true > >> 77: end-volume > >> 78: > >> 79: volume storage-dht > >> 80: type cluster/distribute > >> 81: subvolumes storage-client-0 storage-client-1 storage-client-2 > >> storage-client-3 storage-client-4 storage-client-5 > >> 82: end-volume > >> > >> > >> But not the client op-version, how can I retrieve this info? > >> > >> Thanks > >> > >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote: > >> > >> > Hi Steve, > >> > > >> > Which version you have upgraded client, could you tell us client > >> > op-version after upgrade ? > >> > > >> > > >> > have you upgraded all of your clients ? > >> > > >> > > >> > Thanks, > >> > Gaurav > >> > > >> > > >> > ----- Original Message ----- > >> > From: "Steve Dainard" <sdainard at spd1.com> > >> > To: "gluster-users at gluster.org List" <gluster-users at gluster.org> > >> > Sent: Wednesday, March 2, 2016 1:10:27 AM > >> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more > >> > connected clients cannot support the feature being set > >> > > >> > Gluster 3.7.6 > >> > 'storage' is a distributed volume > >> > > >> > # gluster volume set storage rebal-throttle lazy > >> > volume set: failed: One or more connected clients cannot support the > >> > feature being set. These clients need to be upgraded or disconnected > before > >> > running this command again > >> > > >> > I found a client connected using version 3.6.7 so I upgraded & > >> > umount/mount the gluster volume on the client but I'm still getting > this > >> > error. > >> > > >> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | > grep -v > >> > 3.7.6 and I get a few returns from the client above, all dated last > week. > >> > > >> > I've run 'gluster volume status storage clients' and checked the > connected > >> > clients manually, they're all running 3.7.6. > >> > > >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log: > >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022] > >> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] > >> > 0-management: One or more c > >> > lients don't support the required op-version > >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301] > >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of > >> > operation 'Volume Set' fa > >> > iled on localhost : One or more connected clients cannot support the > >> > feature being set. These clients need to be upgraded or disconnected > be > >> > fore running this command again > >> > > >> > Also tried setting the diagnostics.brick-log-level logging level and > got > >> > the same error. > >> > > >> > /var/lib/glusterd/vols/storage/info: > >> > type=0 > >> > count=6 > >> > status=1 > >> > sub_count=0 > >> > stripe_count=1 > >> > replica_count=1 > >> > disperse_count=0 > >> > redundancy_count=0 > >> > version=26 > >> > transport-type=0 > >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775 > >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c > >> > password> >> > op-version=3 > >> > client-op-version=3 > >> > quota-version=1 > >> > parent_volname=N/A > >> > restored_from_snap=00000000-0000-0000-0000-000000000000 > >> > snap-max-hard-limit=256 > >> > features.quota-deem-statfs=on > >> > features.inode-quota=on > >> > diagnostics.brick-log-level=WARNING > >> > features.quota=on > >> > performance.readdir-ahead=on > >> > performance.cache-size=1GB > >> > performance.stat-prefetch=on > >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage > >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage > >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage > >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage > >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage > >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage > >> > > >> > > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users at gluster.org > >> > http://www.gluster.org/mailman/listinfo/gluster-users > >> > > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160304/a1205a86/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterdump.1590.dump.1457110194 Type: application/octet-stream Size: 24836 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160304/a1205a86/attachment.obj>
Steve Dainard
2016-Mar-04 17:02 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
And I'll mention again, those Debian clients are not running 3.6. As shown
in previous email, they are both running:
# apt-cache policy glusterfs-client
glusterfs-client:
Installed: 3.7.6-2
Candidate: 3.7.6-2
Version table:
*** 3.7.6-2 0
500
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
wheezy/main
amd64 Packages
# glusterfs --version
glusterfs 3.7.6 built on Feb 4 2016 06:25:19
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
On Fri, Mar 4, 2016 at 8:53 AM, Steve Dainard <sdainard at spd1.com>
wrote:
> Except that even when I unmount gluster from both Debian clients, I'm
> still getting the same error, and a dump of those clients after unmounting
> still shows:
>
> glusterd.client1.identifier=10.0.231.10:1023
> glusterd.client1.volname=storage
> glusterd.client1.max-op-version=30603
> glusterd.client1.min-op-version=1
>
> glusterd.client4.identifier=10.0.231.11:1022
> glusterd.client4.volname=storage
> glusterd.client4.max-op-version=30603
> glusterd.client4.min-op-version=1
>
>
>
>
>
> On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherjee83 at
gmail.com
> > wrote:
>
>> -Atin
>> Sent from one plus one
>> On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard at
spd1.com> wrote:
>> >
>> > FYI Gluster storage node hostnames are gluster0[1-6].
>> >
>> > Full dump attached. I see a few clients not on 30706. Most notably
the
>> two debian 7 servers (using packages from gluster.org) seem to be
>> running lower op versions than the centos7 machines (every other client
in
>> 10.0.231.0/24 subnet).
>> >
>> > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7,
glusterfs
>> 3.7.6 built on Feb 4 2016 06:25:19
>> > glusterd.client1.volname=storage
>> > glusterd.client1.max-op-version=30603
>> This is running with 3.6.3 and that's why volume set fails. Unmount
this
>> client and upgrade and remount it back.
>>
>> > glusterd.client1.min-op-version=1
>> >
>> > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02
>> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
>> > glusterd.client2.volname>> >
glusterd.client2.max-op-version=0
>> > glusterd.client2.min-op-version=0
>> >
>> > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no
actual
>> mounts)
>> > glusterd.client3.volname>> >
glusterd.client3.max-op-version=0
>> > glusterd.client3.min-op-version=0
>> >
>> > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7,
glusterfs
>> 3.7.6 built on Feb 4 2016 06:25:19
>> > glusterd.client4.volname=storage
>> > glusterd.client4.max-op-version=30603
>> > glusterd.client4.min-op-version=1
>> >
>> > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no
actual
>> mounts)
>> > glusterd.client5.volname>> >
glusterd.client5.max-op-version=0
>> > glusterd.client5.min-op-version=0
>> >
>> > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04
>> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
>> > glusterd.client6.volname>> >
glusterd.client6.max-op-version=0
>> > glusterd.client6.min-op-version=0
>> >
>> > glusterd.client7.identifier=10.0.231.50:65529
>> >
>>
glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
>> > glusterd.client7.max-op-version=30706
>> > glusterd.client7.min-op-version=1
>> >
>> > ...
>> >
>> > Debian package info:
>> > apt-cache policy glusterfs-client
>> > glusterfs-client:
>> > Installed: 3.7.6-2
>> > Candidate: 3.7.6-2
>> > Version table:
>> > *** 3.7.6-2 0
>> > 500
>>
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
>> wheezy/main amd64 Packages
>> >
>> >
>> > Thanks
>> >
>> > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at
redhat.com> wrote:
>> >>
>> >> Hi Steve,
>> >>
>> >> As atin pointed out to take statedump by running #kill
-SIGUSR1
>> $(pidof glusterd) command. it will create .dump file in
/var/run/gluster/
>> directory. client-op-version information will be present in dump file.
>> >>
>> >> Thanks,
>> >> ~Gaurav
>> >>
>> >> ----- Original Message -----
>> >> From: "Steve Dainard" <sdainard at spd1.com>
>> >> To: "Gaurav Garg" <ggarg at redhat.com>
>> >> Cc: "gluster-users at gluster.org List"
<gluster-users at gluster.org>
>> >> Sent: Thursday, March 3, 2016 12:07:25 AM
>> >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed:
One or
>> more connected clients cannot support the feature being set
>> >>
>> >> From the the client side logs I can see version info on mount:
>> >>
>> >> Final graph:
>> >>
>>
+------------------------------------------------------------------------------+
>> >> 1: volume storage-client-0
>> >> 2: type protocol/client
>> >> 3: option clnt-lk-version 1
>> >> 4: option volfile-checksum 0
>> >> 5: option volfile-key /storage
>> >> 6: option client-version 3.7.6
>> >> 7: option process-uuid
>> >>
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>> >> 8: option fops-version 1298437
>> >> 9: option ping-timeout 42
>> >> 10: option remote-host 10.0.231.50
>> >> 11: option remote-subvolume /mnt/raid6-storage/storage
>> >> 12: option transport-type socket
>> >> 13: option send-gids true
>> >> 14: end-volume
>> >> 15:
>> >> 16: volume storage-client-1
>> >> 17: type protocol/client
>> >> 18: option clnt-lk-version 1
>> >> 19: option volfile-checksum 0
>> >> 20: option volfile-key /storage
>> >> 21: option client-version 3.7.6
>> >> 22: option process-uuid
>> >>
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>> >> 23: option fops-version 1298437
>> >> 24: option ping-timeout 42
>> >> 25: option remote-host 10.0.231.51
>> >> 26: option remote-subvolume /mnt/raid6-storage/storage
>> >> 27: option transport-type socket
>> >> 28: option send-gids true
>> >> 29: end-volume
>> >> 30:
>> >> 31: volume storage-client-2
>> >> 32: type protocol/client
>> >> 33: option clnt-lk-version 1
>> >> 34: option volfile-checksum 0
>> >> 35: option volfile-key /storage
>> >> 36: option client-version 3.7.6
>> >> 37: option process-uuid
>> >>
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>> >> 38: option fops-version 1298437
>> >> 39: option ping-timeout 42
>> >> 40: option remote-host 10.0.231.52
>> >> 41: option remote-subvolume /mnt/raid6-storage/storage
>> >> 42: option transport-type socket
>> >> 43: option send-gids true
>> >> 44: end-volume
>> >> 45:
>> >> 46: volume storage-client-3
>> >> 47: type protocol/client
>> >> 48: option clnt-lk-version 1
>> >> 49: option volfile-checksum 0
>> >> 50: option volfile-key /storage
>> >> 51: option client-version 3.7.6
>> >> 52: option process-uuid
>> >>
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
>> >> 53: option fops-version 1298437
>> >> 54: option ping-timeout 42
>> >> 55: option remote-host 10.0.231.53
>> >> 56: option remote-subvolume /mnt/raid6-storage/storage
>> >> 57: option transport-type socket
>> >> 58: option send-gids true
>> >> 59: end-volume
>> >> 60:
>> >> 61: volume storage-client-4
>> >> 62: type protocol/client
>> >> 63: option ping-timeout 42
>> >> 64: option remote-host 10.0.231.54
>> >> 65: option remote-subvolume /mnt/raid6-storage/storage
>> >> 66: option transport-type socket
>> >> 67: option send-gids true
>> >> 68: end-volume
>> >> 69:
>> >> 70: volume storage-client-5
>> >> 71: type protocol/client
>> >> 72: option ping-timeout 42
>> >> 73: option remote-host 10.0.231.55
>> >> 74: option remote-subvolume /mnt/raid6-storage/storage
>> >> 75: option transport-type socket
>> >> 76: option send-gids true
>> >> 77: end-volume
>> >> 78:
>> >> 79: volume storage-dht
>> >> 80: type cluster/distribute
>> >> 81: subvolumes storage-client-0 storage-client-1
storage-client-2
>> >> storage-client-3 storage-client-4 storage-client-5
>> >> 82: end-volume
>> >>
>> >>
>> >> But not the client op-version, how can I retrieve this info?
>> >>
>> >> Thanks
>> >>
>> >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at
redhat.com> wrote:
>> >>
>> >> > Hi Steve,
>> >> >
>> >> > Which version you have upgraded client, could you tell us
client
>> >> > op-version after upgrade ?
>> >> >
>> >> >
>> >> > have you upgraded all of your clients ?
>> >> >
>> >> >
>> >> > Thanks,
>> >> > Gaurav
>> >> >
>> >> >
>> >> > ----- Original Message -----
>> >> > From: "Steve Dainard" <sdainard at
spd1.com>
>> >> > To: "gluster-users at gluster.org List"
<gluster-users at gluster.org>
>> >> > Sent: Wednesday, March 2, 2016 1:10:27 AM
>> >> > Subject: [Gluster-users] gluster 3.7.6 volume set:
failed: One or
>> more
>> >> > connected clients cannot support the feature being set
>> >> >
>> >> > Gluster 3.7.6
>> >> > 'storage' is a distributed volume
>> >> >
>> >> > # gluster volume set storage rebal-throttle lazy
>> >> > volume set: failed: One or more connected clients cannot
support the
>> >> > feature being set. These clients need to be upgraded or
disconnected
>> before
>> >> > running this command again
>> >> >
>> >> > I found a client connected using version 3.6.7 so I
upgraded &
>> >> > umount/mount the gluster volume on the client but I'm
still getting
>> this
>> >> > error.
>> >> >
>> >> > I've run grep "accepted client from"
/var/log/glusterfs/bricks/* |
>> grep -v
>> >> > 3.7.6 and I get a few returns from the client above, all
dated last
>> week.
>> >> >
>> >> > I've run 'gluster volume status storage
clients' and checked the
>> connected
>> >> > clients manually, they're all running 3.7.6.
>> >> >
>> >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:
>> >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]
>> >> >
[glusterd-utils.c:10154:glusterd_check_client_op_version_support]
>> >> > 0-management: One or more c
>> >> > lients don't support the required op-version
>> >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]
>> >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management:
Staging of
>> >> > operation 'Volume Set' fa
>> >> > iled on localhost : One or more connected clients cannot
support the
>> >> > feature being set. These clients need to be upgraded or
disconnected
>> be
>> >> > fore running this command again
>> >> >
>> >> > Also tried setting the diagnostics.brick-log-level
logging level and
>> got
>> >> > the same error.
>> >> >
>> >> > /var/lib/glusterd/vols/storage/info:
>> >> > type=0
>> >> > count=6
>> >> > status=1
>> >> > sub_count=0
>> >> > stripe_count=1
>> >> > replica_count=1
>> >> > disperse_count=0
>> >> > redundancy_count=0
>> >> > version=26
>> >> > transport-type=0
>> >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775
>> >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
>> >> > password>> >> > op-version=3
>> >> > client-op-version=3
>> >> > quota-version=1
>> >> > parent_volname=N/A
>> >> > restored_from_snap=00000000-0000-0000-0000-000000000000
>> >> > snap-max-hard-limit=256
>> >> > features.quota-deem-statfs=on
>> >> > features.inode-quota=on
>> >> > diagnostics.brick-log-level=WARNING
>> >> > features.quota=on
>> >> > performance.readdir-ahead=on
>> >> > performance.cache-size=1GB
>> >> > performance.stat-prefetch=on
>> >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage
>> >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage
>> >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage
>> >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage
>> >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage
>> >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Gluster-users mailing list
>> >> > Gluster-users at gluster.org
>> >> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160304/bb06c899/attachment.html>
Atin Mukherjee
2016-Mar-05 07:20 UTC
[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
This is really weird, if you unmount the clients then statedump shouldn't be showing the same. Was unmount successful? Do you see any related error log entry in mount & glusterd log? -Atin Sent from one plus one On 04-Mar-2016 10:23 pm, "Steve Dainard" <sdainard at spd1.com> wrote:> Except that even when I unmount gluster from both Debian clients, I'm > still getting the same error, and a dump of those clients after unmounting > still shows: > > glusterd.client1.identifier=10.0.231.10:1023 > glusterd.client1.volname=storage > glusterd.client1.max-op-version=30603 > glusterd.client1.min-op-version=1 > > glusterd.client4.identifier=10.0.231.11:1022 > glusterd.client4.volname=storage > glusterd.client4.max-op-version=30603 > glusterd.client4.min-op-version=1 > > > > > > On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <atin.mukherjee83 at gmail.com > > wrote: > >> -Atin >> Sent from one plus one >> On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard at spd1.com> wrote: >> > >> > FYI Gluster storage node hostnames are gluster0[1-6]. >> > >> > Full dump attached. I see a few clients not on 30706. Most notably the >> two debian 7 servers (using packages from gluster.org) seem to be >> running lower op versions than the centos7 machines (every other client in >> 10.0.231.0/24 subnet). >> > >> > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs >> 3.7.6 built on Feb 4 2016 06:25:19 >> > glusterd.client1.volname=storage >> > glusterd.client1.max-op-version=30603 >> This is running with 3.6.3 and that's why volume set fails. Unmount this >> client and upgrade and remount it back. >> >> > glusterd.client1.min-op-version=1 >> > >> > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02 >> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) >> > glusterd.client2.volname>> > glusterd.client2.max-op-version=0 >> > glusterd.client2.min-op-version=0 >> > >> > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual >> mounts) >> > glusterd.client3.volname>> > glusterd.client3.max-op-version=0 >> > glusterd.client3.min-op-version=0 >> > >> > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs >> 3.7.6 built on Feb 4 2016 06:25:19 >> > glusterd.client4.volname=storage >> > glusterd.client4.max-op-version=30603 >> > glusterd.client4.min-op-version=1 >> > >> > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual >> mounts) >> > glusterd.client5.volname>> > glusterd.client5.max-op-version=0 >> > glusterd.client5.min-op-version=0 >> > >> > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04 >> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs) >> > glusterd.client6.volname>> > glusterd.client6.max-op-version=0 >> > glusterd.client6.min-op-version=0 >> > >> > glusterd.client7.identifier=10.0.231.50:65529 >> > >> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage >> > glusterd.client7.max-op-version=30706 >> > glusterd.client7.min-op-version=1 >> > >> > ... >> > >> > Debian package info: >> > apt-cache policy glusterfs-client >> > glusterfs-client: >> > Installed: 3.7.6-2 >> > Candidate: 3.7.6-2 >> > Version table: >> > *** 3.7.6-2 0 >> > 500 >> http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/ >> wheezy/main amd64 Packages >> > >> > >> > Thanks >> > >> > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >> >> >> Hi Steve, >> >> >> >> As atin pointed out to take statedump by running #kill -SIGUSR1 >> $(pidof glusterd) command. it will create .dump file in /var/run/gluster/ >> directory. client-op-version information will be present in dump file. >> >> >> >> Thanks, >> >> ~Gaurav >> >> >> >> ----- Original Message ----- >> >> From: "Steve Dainard" <sdainard at spd1.com> >> >> To: "Gaurav Garg" <ggarg at redhat.com> >> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org> >> >> Sent: Thursday, March 3, 2016 12:07:25 AM >> >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or >> more connected clients cannot support the feature being set >> >> >> >> From the the client side logs I can see version info on mount: >> >> >> >> Final graph: >> >> >> +------------------------------------------------------------------------------+ >> >> 1: volume storage-client-0 >> >> 2: type protocol/client >> >> 3: option clnt-lk-version 1 >> >> 4: option volfile-checksum 0 >> >> 5: option volfile-key /storage >> >> 6: option client-version 3.7.6 >> >> 7: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0 >> >> 8: option fops-version 1298437 >> >> 9: option ping-timeout 42 >> >> 10: option remote-host 10.0.231.50 >> >> 11: option remote-subvolume /mnt/raid6-storage/storage >> >> 12: option transport-type socket >> >> 13: option send-gids true >> >> 14: end-volume >> >> 15: >> >> 16: volume storage-client-1 >> >> 17: type protocol/client >> >> 18: option clnt-lk-version 1 >> >> 19: option volfile-checksum 0 >> >> 20: option volfile-key /storage >> >> 21: option client-version 3.7.6 >> >> 22: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0 >> >> 23: option fops-version 1298437 >> >> 24: option ping-timeout 42 >> >> 25: option remote-host 10.0.231.51 >> >> 26: option remote-subvolume /mnt/raid6-storage/storage >> >> 27: option transport-type socket >> >> 28: option send-gids true >> >> 29: end-volume >> >> 30: >> >> 31: volume storage-client-2 >> >> 32: type protocol/client >> >> 33: option clnt-lk-version 1 >> >> 34: option volfile-checksum 0 >> >> 35: option volfile-key /storage >> >> 36: option client-version 3.7.6 >> >> 37: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0 >> >> 38: option fops-version 1298437 >> >> 39: option ping-timeout 42 >> >> 40: option remote-host 10.0.231.52 >> >> 41: option remote-subvolume /mnt/raid6-storage/storage >> >> 42: option transport-type socket >> >> 43: option send-gids true >> >> 44: end-volume >> >> 45: >> >> 46: volume storage-client-3 >> >> 47: type protocol/client >> >> 48: option clnt-lk-version 1 >> >> 49: option volfile-checksum 0 >> >> 50: option volfile-key /storage >> >> 51: option client-version 3.7.6 >> >> 52: option process-uuid >> >> >> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0 >> >> 53: option fops-version 1298437 >> >> 54: option ping-timeout 42 >> >> 55: option remote-host 10.0.231.53 >> >> 56: option remote-subvolume /mnt/raid6-storage/storage >> >> 57: option transport-type socket >> >> 58: option send-gids true >> >> 59: end-volume >> >> 60: >> >> 61: volume storage-client-4 >> >> 62: type protocol/client >> >> 63: option ping-timeout 42 >> >> 64: option remote-host 10.0.231.54 >> >> 65: option remote-subvolume /mnt/raid6-storage/storage >> >> 66: option transport-type socket >> >> 67: option send-gids true >> >> 68: end-volume >> >> 69: >> >> 70: volume storage-client-5 >> >> 71: type protocol/client >> >> 72: option ping-timeout 42 >> >> 73: option remote-host 10.0.231.55 >> >> 74: option remote-subvolume /mnt/raid6-storage/storage >> >> 75: option transport-type socket >> >> 76: option send-gids true >> >> 77: end-volume >> >> 78: >> >> 79: volume storage-dht >> >> 80: type cluster/distribute >> >> 81: subvolumes storage-client-0 storage-client-1 storage-client-2 >> >> storage-client-3 storage-client-4 storage-client-5 >> >> 82: end-volume >> >> >> >> >> >> But not the client op-version, how can I retrieve this info? >> >> >> >> Thanks >> >> >> >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >> >> >> > Hi Steve, >> >> > >> >> > Which version you have upgraded client, could you tell us client >> >> > op-version after upgrade ? >> >> > >> >> > >> >> > have you upgraded all of your clients ? >> >> > >> >> > >> >> > Thanks, >> >> > Gaurav >> >> > >> >> > >> >> > ----- Original Message ----- >> >> > From: "Steve Dainard" <sdainard at spd1.com> >> >> > To: "gluster-users at gluster.org List" <gluster-users at gluster.org> >> >> > Sent: Wednesday, March 2, 2016 1:10:27 AM >> >> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or >> more >> >> > connected clients cannot support the feature being set >> >> > >> >> > Gluster 3.7.6 >> >> > 'storage' is a distributed volume >> >> > >> >> > # gluster volume set storage rebal-throttle lazy >> >> > volume set: failed: One or more connected clients cannot support the >> >> > feature being set. These clients need to be upgraded or disconnected >> before >> >> > running this command again >> >> > >> >> > I found a client connected using version 3.6.7 so I upgraded & >> >> > umount/mount the gluster volume on the client but I'm still getting >> this >> >> > error. >> >> > >> >> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | >> grep -v >> >> > 3.7.6 and I get a few returns from the client above, all dated last >> week. >> >> > >> >> > I've run 'gluster volume status storage clients' and checked the >> connected >> >> > clients manually, they're all running 3.7.6. >> >> > >> >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log: >> >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022] >> >> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support] >> >> > 0-management: One or more c >> >> > lients don't support the required op-version >> >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301] >> >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of >> >> > operation 'Volume Set' fa >> >> > iled on localhost : One or more connected clients cannot support the >> >> > feature being set. These clients need to be upgraded or disconnected >> be >> >> > fore running this command again >> >> > >> >> > Also tried setting the diagnostics.brick-log-level logging level and >> got >> >> > the same error. >> >> > >> >> > /var/lib/glusterd/vols/storage/info: >> >> > type=0 >> >> > count=6 >> >> > status=1 >> >> > sub_count=0 >> >> > stripe_count=1 >> >> > replica_count=1 >> >> > disperse_count=0 >> >> > redundancy_count=0 >> >> > version=26 >> >> > transport-type=0 >> >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775 >> >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c >> >> > password>> >> > op-version=3 >> >> > client-op-version=3 >> >> > quota-version=1 >> >> > parent_volname=N/A >> >> > restored_from_snap=00000000-0000-0000-0000-000000000000 >> >> > snap-max-hard-limit=256 >> >> > features.quota-deem-statfs=on >> >> > features.inode-quota=on >> >> > diagnostics.brick-log-level=WARNING >> >> > features.quota=on >> >> > performance.readdir-ahead=on >> >> > performance.cache-size=1GB >> >> > performance.stat-prefetch=on >> >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage >> >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage >> >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage >> >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage >> >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage >> >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage >> >> > >> >> > >> >> > _______________________________________________ >> >> > Gluster-users mailing list >> >> > Gluster-users at gluster.org >> >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> > >> > >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160305/6187ed13/attachment.html>