thr3ads.net - Gluster users - [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set [Mar 2016]

If this information is useful, please help other people find it:
Share via:

Gaurav Garg

2016-Mar-03 06:29 UTC

[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

Hi Steve,

As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof
glusterd)  command. it will create .dump file in /var/run/gluster/ directory.
client-op-version information will be present in dump file.

Thanks,
~Gaurav

----- Original Message -----
From: "Steve Dainard" <sdainard at spd1.com>
To: "Gaurav Garg" <ggarg at redhat.com>
Cc: "gluster-users at gluster.org List" <gluster-users at
gluster.org>
Sent: Thursday, March 3, 2016 12:07:25 AM
Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
connected clients cannot support the feature being set
>From the the client side logs I can see version info on mount:
Final graph:
+------------------------------------------------------------------------------+
  1: volume storage-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key /storage
  6:     option client-version 3.7.6
  7:     option process-uuid
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 42
 10:     option remote-host 10.0.231.50
 11:     option remote-subvolume /mnt/raid6-storage/storage
 12:     option transport-type socket
 13:     option send-gids true
 14: end-volume
 15:
 16: volume storage-client-1
 17:     type protocol/client
 18:     option clnt-lk-version 1
 19:     option volfile-checksum 0
 20:     option volfile-key /storage
 21:     option client-version 3.7.6
 22:     option process-uuid
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
 23:     option fops-version 1298437
 24:     option ping-timeout 42
 25:     option remote-host 10.0.231.51
 26:     option remote-subvolume /mnt/raid6-storage/storage
 27:     option transport-type socket
 28:     option send-gids true
 29: end-volume
 30:
 31: volume storage-client-2
 32:     type protocol/client
 33:     option clnt-lk-version 1
 34:     option volfile-checksum 0
 35:     option volfile-key /storage
 36:     option client-version 3.7.6
 37:     option process-uuid
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
 38:     option fops-version 1298437
 39:     option ping-timeout 42
 40:     option remote-host 10.0.231.52
 41:     option remote-subvolume /mnt/raid6-storage/storage
 42:     option transport-type socket
 43:     option send-gids true
 44: end-volume
 45:
 46: volume storage-client-3
 47:     type protocol/client
 48:     option clnt-lk-version 1
 49:     option volfile-checksum 0
 50:     option volfile-key /storage
 51:     option client-version 3.7.6
 52:     option process-uuid
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
 53:     option fops-version 1298437
 54:     option ping-timeout 42
 55:     option remote-host 10.0.231.53
 56:     option remote-subvolume /mnt/raid6-storage/storage
 57:     option transport-type socket
 58:     option send-gids true
 59: end-volume
 60:
 61: volume storage-client-4
 62:     type protocol/client
 63:     option ping-timeout 42
 64:     option remote-host 10.0.231.54
 65:     option remote-subvolume /mnt/raid6-storage/storage
 66:     option transport-type socket
 67:     option send-gids true
 68: end-volume
 69:
 70: volume storage-client-5
 71:     type protocol/client
 72:     option ping-timeout 42
 73:     option remote-host 10.0.231.55
 74:     option remote-subvolume /mnt/raid6-storage/storage
 75:     option transport-type socket
 76:     option send-gids true
 77: end-volume
 78:
 79: volume storage-dht
 80:     type cluster/distribute
 81:     subvolumes storage-client-0 storage-client-1 storage-client-2
storage-client-3 storage-client-4 storage-client-5
 82: end-volume


But not the client op-version, how can I retrieve this info?

Thanks

On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com> wrote:
> Hi Steve,
>
> Which version you have upgraded client, could you tell us client
> op-version after upgrade ?
>
>
> have you upgraded all of your clients ?
>
>
> Thanks,
> Gaurav
>
>
> ----- Original Message -----
> From: "Steve Dainard" <sdainard at spd1.com>
> To: "gluster-users at gluster.org List" <gluster-users at
gluster.org>
> Sent: Wednesday, March 2, 2016 1:10:27 AM
> Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
> connected clients cannot support the feature being set
>
> Gluster 3.7.6
> 'storage' is a distributed volume
>
> # gluster volume set storage rebal-throttle lazy
> volume set: failed: One or more connected clients cannot support the
> feature being set. These clients need to be upgraded or disconnected before
> running this command again
>
> I found a client connected using version 3.6.7 so I upgraded &
> umount/mount the gluster volume on the client but I'm still getting
this
> error.
>
> I've run grep "accepted client from"
/var/log/glusterfs/bricks/* | grep -v
> 3.7.6 and I get a few returns from the client above, all dated last week.
>
> I've run 'gluster volume status storage clients' and checked
the connected
> clients manually, they're all running 3.7.6.
>
> /var/log/gluster/etc-glusterfs-glusterd.vol.log:
> [2016-03-01 19:23:20.180821] E [MSGID: 106022]
> [glusterd-utils.c:10154:glusterd_check_client_op_version_support]
> 0-management: One or more c
> lients don't support the required op-version
> [2016-03-01 19:23:20.180853] E [MSGID: 106301]
> [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of
> operation 'Volume Set' fa
> iled on localhost : One or more connected clients cannot support the
> feature being set. These clients need to be upgraded or disconnected be
> fore running this command again
>
> Also tried setting the diagnostics.brick-log-level logging level and got
> the same error.
>
> /var/lib/glusterd/vols/storage/info:
> type=0
> count=6
> status=1
> sub_count=0
> stripe_count=1
> replica_count=1
> disperse_count=0
> redundancy_count=0
> version=26
> transport-type=0
> volume-id=26d355cb-c486-481f-ac16-e25390e73775
> username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
> password> op-version=3
> client-op-version=3
> quota-version=1
> parent_volname=N/A
> restored_from_snap=00000000-0000-0000-0000-000000000000
> snap-max-hard-limit=256
> features.quota-deem-statfs=on
> features.inode-quota=on
> diagnostics.brick-log-level=WARNING
> features.quota=on
> performance.readdir-ahead=on
> performance.cache-size=1GB
> performance.stat-prefetch=on
> brick-0=10.0.231.50:-mnt-raid6-storage-storage
> brick-1=10.0.231.51:-mnt-raid6-storage-storage
> brick-2=10.0.231.52:-mnt-raid6-storage-storage
> brick-3=10.0.231.53:-mnt-raid6-storage-storage
> brick-4=10.0.231.54:-mnt-raid6-storage-storage
> brick-5=10.0.231.55:-mnt-raid6-storage-storage
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>

Steve Dainard

2016-Mar-03 22:05 UTC

head link

[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

FYI Gluster storage node hostnames are gluster0[1-6].

Full dump attached. I see a few clients not on 30706. Most notably the two
debian 7 servers (using packages from gluster.org) seem to be running lower
op versions than the centos7 machines (every other client in 10.0.231.0/24
subnet).

glusterd.client1.identifier=10.0.231.10:1023 <-- *debian 7*, glusterfs
3.7.6 built on Feb  4 2016 06:25:19
glusterd.client1.volname=storage
glusterd.client1.max-op-version=30603
glusterd.client1.min-op-version=1

glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
glusterd.client2.volnameglusterd.client2.max-op-version=0
glusterd.client2.min-op-version=0

glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual
mounts)
glusterd.client3.volnameglusterd.client3.max-op-version=0
glusterd.client3.min-op-version=0

glusterd.client4.identifier=10.0.231.11:1022 <---* debian 7*, glusterfs
3.7.6 built on Feb  4 2016 06:25:19
glusterd.client4.volname=storage
glusterd.client4.max-op-version=30603
glusterd.client4.min-op-version=1

glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual
mounts)
glusterd.client5.volnameglusterd.client5.max-op-version=0
glusterd.client5.min-op-version=0

glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
glusterd.client6.volnameglusterd.client6.max-op-version=0
glusterd.client6.min-op-version=0

glusterd.client7.identifier=10.0.231.50:65529
glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
glusterd.client7.max-op-version=30706
glusterd.client7.min-op-version=1

...

Debian package info:
apt-cache policy glusterfs-client
glusterfs-client:
  Installed: 3.7.6-2
  Candidate: 3.7.6-2
  Version table:
 *** 3.7.6-2 0
        500
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
wheezy/main amd64 Packages


Thanks

On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at redhat.com> wrote:
> Hi Steve,
>
> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof
> glusterd)  command. it will create .dump file in /var/run/gluster/
> directory. client-op-version information will be present in dump file.
>
> Thanks,
> ~Gaurav
>
> ----- Original Message -----
> From: "Steve Dainard" <sdainard at spd1.com>
> To: "Gaurav Garg" <ggarg at redhat.com>
> Cc: "gluster-users at gluster.org List" <gluster-users at
gluster.org>
> Sent: Thursday, March 3, 2016 12:07:25 AM
> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
> connected clients cannot support the feature being set
>
> From the the client side logs I can see version info on mount:
>
> Final graph:
>
>
+------------------------------------------------------------------------------+
>   1: volume storage-client-0
>   2:     type protocol/client
>   3:     option clnt-lk-version 1
>   4:     option volfile-checksum 0
>   5:     option volfile-key /storage
>   6:     option client-version 3.7.6
>   7:     option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>   8:     option fops-version 1298437
>   9:     option ping-timeout 42
>  10:     option remote-host 10.0.231.50
>  11:     option remote-subvolume /mnt/raid6-storage/storage
>  12:     option transport-type socket
>  13:     option send-gids true
>  14: end-volume
>  15:
>  16: volume storage-client-1
>  17:     type protocol/client
>  18:     option clnt-lk-version 1
>  19:     option volfile-checksum 0
>  20:     option volfile-key /storage
>  21:     option client-version 3.7.6
>  22:     option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>  23:     option fops-version 1298437
>  24:     option ping-timeout 42
>  25:     option remote-host 10.0.231.51
>  26:     option remote-subvolume /mnt/raid6-storage/storage
>  27:     option transport-type socket
>  28:     option send-gids true
>  29: end-volume
>  30:
>  31: volume storage-client-2
>  32:     type protocol/client
>  33:     option clnt-lk-version 1
>  34:     option volfile-checksum 0
>  35:     option volfile-key /storage
>  36:     option client-version 3.7.6
>  37:     option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>  38:     option fops-version 1298437
>  39:     option ping-timeout 42
>  40:     option remote-host 10.0.231.52
>  41:     option remote-subvolume /mnt/raid6-storage/storage
>  42:     option transport-type socket
>  43:     option send-gids true
>  44: end-volume
>  45:
>  46: volume storage-client-3
>  47:     type protocol/client
>  48:     option clnt-lk-version 1
>  49:     option volfile-checksum 0
>  50:     option volfile-key /storage
>  51:     option client-version 3.7.6
>  52:     option process-uuid
>
>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
>  53:     option fops-version 1298437
>  54:     option ping-timeout 42
>  55:     option remote-host 10.0.231.53
>  56:     option remote-subvolume /mnt/raid6-storage/storage
>  57:     option transport-type socket
>  58:     option send-gids true
>  59: end-volume
>  60:
>  61: volume storage-client-4
>  62:     type protocol/client
>  63:     option ping-timeout 42
>  64:     option remote-host 10.0.231.54
>  65:     option remote-subvolume /mnt/raid6-storage/storage
>  66:     option transport-type socket
>  67:     option send-gids true
>  68: end-volume
>  69:
>  70: volume storage-client-5
>  71:     type protocol/client
>  72:     option ping-timeout 42
>  73:     option remote-host 10.0.231.55
>  74:     option remote-subvolume /mnt/raid6-storage/storage
>  75:     option transport-type socket
>  76:     option send-gids true
>  77: end-volume
>  78:
>  79: volume storage-dht
>  80:     type cluster/distribute
>  81:     subvolumes storage-client-0 storage-client-1 storage-client-2
> storage-client-3 storage-client-4 storage-client-5
>  82: end-volume
>
>
> But not the client op-version, how can I retrieve this info?
>
> Thanks
>
> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com>
wrote:
>
> > Hi Steve,
> >
> > Which version you have upgraded client, could you tell us client
> > op-version after upgrade ?
> >
> >
> > have you upgraded all of your clients ?
> >
> >
> > Thanks,
> > Gaurav
> >
> >
> > ----- Original Message -----
> > From: "Steve Dainard" <sdainard at spd1.com>
> > To: "gluster-users at gluster.org List" <gluster-users at
gluster.org>
> > Sent: Wednesday, March 2, 2016 1:10:27 AM
> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
> > connected clients cannot support the feature being set
> >
> > Gluster 3.7.6
> > 'storage' is a distributed volume
> >
> > # gluster volume set storage rebal-throttle lazy
> > volume set: failed: One or more connected clients cannot support the
> > feature being set. These clients need to be upgraded or disconnected
> before
> > running this command again
> >
> > I found a client connected using version 3.6.7 so I upgraded &
> > umount/mount the gluster volume on the client but I'm still
getting this
> > error.
> >
> > I've run grep "accepted client from"
/var/log/glusterfs/bricks/* | grep
> -v
> > 3.7.6 and I get a few returns from the client above, all dated last
week.
> >
> > I've run 'gluster volume status storage clients' and
checked the
> connected
> > clients manually, they're all running 3.7.6.
> >
> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:
> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]
> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]
> > 0-management: One or more c
> > lients don't support the required op-version
> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]
> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of
> > operation 'Volume Set' fa
> > iled on localhost : One or more connected clients cannot support the
> > feature being set. These clients need to be upgraded or disconnected
be
> > fore running this command again
> >
> > Also tried setting the diagnostics.brick-log-level logging level and
got
> > the same error.
> >
> > /var/lib/glusterd/vols/storage/info:
> > type=0
> > count=6
> > status=1
> > sub_count=0
> > stripe_count=1
> > replica_count=1
> > disperse_count=0
> > redundancy_count=0
> > version=26
> > transport-type=0
> > volume-id=26d355cb-c486-481f-ac16-e25390e73775
> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
> > password> > op-version=3
> > client-op-version=3
> > quota-version=1
> > parent_volname=N/A
> > restored_from_snap=00000000-0000-0000-0000-000000000000
> > snap-max-hard-limit=256
> > features.quota-deem-statfs=on
> > features.inode-quota=on
> > diagnostics.brick-log-level=WARNING
> > features.quota=on
> > performance.readdir-ahead=on
> > performance.cache-size=1GB
> > performance.stat-prefetch=on
> > brick-0=10.0.231.50:-mnt-raid6-storage-storage
> > brick-1=10.0.231.51:-mnt-raid6-storage-storage
> > brick-2=10.0.231.52:-mnt-raid6-storage-storage
> > brick-3=10.0.231.53:-mnt-raid6-storage-storage
> > brick-4=10.0.231.54:-mnt-raid6-storage-storage
> > brick-5=10.0.231.55:-mnt-raid6-storage-storage
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160303/f0ca9fbf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: glusterdump.1590.dump.1457041019
Type: application/octet-stream
Size: 25133 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160303/f0ca9fbf/attachment.obj>

Gluster users - Mar 2016 - gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set