Gluster users - Jul 2017 - op-version for reset-brick (Was: Re: [ovirt-users] Upgrading HC from 4.0 to 4.1)

On Wed, Jul 5, 2017 at 8:16 PM, Gianluca Cecchi <gianluca.cecchi at
gmail.com>
wrote:
>
>
> On Wed, Jul 5, 2017 at 7:42 AM, Sahina Bose <sabose at redhat.com>
wrote:
>
>>
>>
>>> ...
>>>
>>> then the commands I need to run would be:
>>>
>>> gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
>>> start
>>> gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
>>> gl01.localdomain.local:/gluster/brick3/export commit force
>>>
>>> Correct?
>>>
>>
>> Yes, correct. gl01.localdomain.local should resolve correctly on all 3
>> nodes.
>>
>
>
> It fails at first step:
>
>  [root at ovirt01 ~]# gluster volume reset-brick export
> ovirt01.localdomain.local:/gluster/brick3/export start
> volume reset-brick: failed: Cannot execute command. The cluster is
> operating at version 30712. reset-brick command reset-brick start is
> unavailable in this version.
> [root at ovirt01 ~]#
>
> It seems somehow in relation with this upgrade not of the commercial
> solution Red Hat Gluster Storage
> https://access.redhat.com/documentation/en-US/Red_Hat_
> Storage/3.1/html/Installation_Guide/chap-Upgrading_Red_Hat_Storage.html
>
> So ti seems I have to run some command of type:
>
> gluster volume set all cluster.op-version XXXXX
>
> with XXXXX > 30712
>
> It seems that latest version of commercial Red Hat Gluster Storage is 3.1
> and its op-version is indeed 30712..
>
> So the question is which particular op-version I have to set and if the
> command can be set online without generating disruption....
>
It should have worked with the glusterfs 3.10 version from Centos repo.
Adding gluster-users for help on the op-version

>
> Thanks,
> Gianluca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170705/95cce7a9/attachment.html>

On Wed, Jul 5, 2017 at 5:02 PM, Sahina Bose <sabose at redhat.com> wrote:
>
>
> On Wed, Jul 5, 2017 at 8:16 PM, Gianluca Cecchi <gianluca.cecchi at
gmail.com
> > wrote:
>
>>
>>
>> On Wed, Jul 5, 2017 at 7:42 AM, Sahina Bose <sabose at
redhat.com> wrote:
>>
>>>
>>>
>>>> ...
>>>>
>>>> then the commands I need to run would be:
>>>>
>>>> gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
>>>> start
>>>> gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
>>>> gl01.localdomain.local:/gluster/brick3/export commit force
>>>>
>>>> Correct?
>>>>
>>>
>>> Yes, correct. gl01.localdomain.local should resolve correctly on
all 3
>>> nodes.
>>>
>>
>>
>> It fails at first step:
>>
>>  [root at ovirt01 ~]# gluster volume reset-brick export
>> ovirt01.localdomain.local:/gluster/brick3/export start
>> volume reset-brick: failed: Cannot execute command. The cluster is
>> operating at version 30712. reset-brick command reset-brick start is
>> unavailable in this version.
>> [root at ovirt01 ~]#
>>
>> It seems somehow in relation with this upgrade not of the commercial
>> solution Red Hat Gluster Storage
>> https://access.redhat.com/documentation/en-US/Red_Hat_Storag
>> e/3.1/html/Installation_Guide/chap-Upgrading_Red_Hat_Storage.html
>>
>> So ti seems I have to run some command of type:
>>
>> gluster volume set all cluster.op-version XXXXX
>>
>> with XXXXX > 30712
>>
>> It seems that latest version of commercial Red Hat Gluster Storage is
3.1
>> and its op-version is indeed 30712..
>>
>> So the question is which particular op-version I have to set and if the
>> command can be set online without generating disruption....
>>
>
> It should have worked with the glusterfs 3.10 version from Centos repo.
> Adding gluster-users for help on the op-version
>
>
>>
>> Thanks,
>> Gianluca
>>
>
>
It seems op-version is not updated automatically by default, so that it can
manage mixed versions while you update one by one...

I followed what described here:
https://gluster.readthedocs.io/en/latest/Upgrade-Guide/op_version/


- Get current version:

[root at ovirt01 ~]# gluster volume get all cluster.op-version
Option                                  Value

------                                  -----

cluster.op-version                      30712

[root at ovirt01 ~]#


- Get maximum version I can set for current setup:

[root at ovirt01 ~]# gluster volume get all cluster.max-op-version
Option                                  Value

------                                  -----

cluster.max-op-version                  31000

[root at ovirt01 ~]#


- Get op version information for all the connected clients:

[root at ovirt01 ~]# gluster volume status all clients | grep ":49" |
awk
'{print $4}' | sort | uniq -c
     72 31000
[root at ovirt01 ~]#

--> ok


- Update op-version

[root at ovirt01 ~]# gluster volume set all cluster.op-version 31000
volume set: success
[root at ovirt01 ~]#


- Verify:
[root at ovirt01 ~]# gluster volume get all cluster.op-versionOption
                       Value
------                                  -----

cluster.op-version                      31000

[root at ovirt01 ~]#

--> ok

[root at ovirt01 ~]# gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful

[root at ovirt01 ~]# gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
gl01.localdomain.local:/gluster/brick3/export commit force
volume reset-brick: failed: Commit failed on ovirt02.localdomain.local.
Please check log file for details.
Commit failed on ovirt03.localdomain.local. Please check log file for
details.
[root at ovirt01 ~]#

[root at ovirt01 bricks]# gluster volume info export

Volume Name: export
Type: Replicate
Volume ID: b00e5839-becb-47e7-844f-6ce6ce1b7153
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gl01.localdomain.local:/gluster/brick3/export
Brick2: ovirt02.localdomain.local:/gluster/brick3/export
Brick3: ovirt03.localdomain.local:/gluster/brick3/export (arbiter)
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 6
network.ping-timeout: 30
user.cifs: off
nfs.disable: on
performance.strict-o-direct: on
[root at ovirt01 bricks]# gluster volume reset-brick export
ovirt02.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful
[root at ovirt01 bricks]# gluster volume reset-brick export
ovirt02.localdomain.local:/gluster/brick3/export
gl02.localdomain.local:/gluster/brick3/export commit force
volume reset-brick: failed: Commit failed on localhost. Please check log
file for details.
[root at ovirt01 bricks]#

I proceed (I have actually nothing on export volume...)

[root at ovirt01 bricks]# gluster volume reset-brick export
ovirt02.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful
[root at ovirt01 bricks]# gluster volume reset-brick export
ovirt02.localdomain.local:/gluster/brick3/export
gl02.localdomain.local:/gluster/brick3/export commit force
volume reset-brick: failed: Commit failed on localhost. Please check log
file for details.
[root at ovirt01 bricks]#

Again error


[root at ovirt01 bricks]# gluster volume info export

Volume Name: export
Type: Replicate
Volume ID: b00e5839-becb-47e7-844f-6ce6ce1b7153
Status: Started
Snapshot Count: 0
Number of Bricks: 0 x (2 + 1) = 2
Transport-type: tcp
Bricks:
Brick1: gl01.localdomain.local:/gluster/brick3/export
Brick2: ovirt03.localdomain.local:/gluster/brick3/export
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 6
network.ping-timeout: 30
user.cifs: off
nfs.disable: on
performance.strict-o-direct: on
[root at ovirt01 bricks]#


The last

[root at ovirt01 bricks]# gluster volume reset-brick export
ovirt03.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful
[root at ovirt01 bricks]# gluster volume reset-brick export
ovirt03.localdomain.local:/gluster/brick3/export
gl03.localdomain.local:/gluster/brick3/export commit force
volume reset-brick: failed: Commit failed on localhost. Please check log
file for details.
[root at ovirt01 bricks]#

again error


[root at ovirt01 bricks]# gluster volume info export

Volume Name: export
Type: Replicate
Volume ID: b00e5839-becb-47e7-844f-6ce6ce1b7153
Status: Started
Snapshot Count: 0
Number of Bricks: 0 x (2 + 1) = 1
Transport-type: tcp
Bricks:
Brick1: gl01.localdomain.local:/gluster/brick3/export
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 6
network.ping-timeout: 30
user.cifs: off
nfs.disable: on
performance.strict-o-direct: on
[root at ovirt01 bricks]#

See here for gluster log in gzip format....
https://drive.google.com/file/d/0BwoPbcrMv8mvQmlYZjAySTZKTzQ/view?usp=sharing

The first command executed at 14:57 and the other two at 15:04

This is what seen by oVirt right now for the volume
https://drive.google.com/file/d/0BwoPbcrMv8mvNFAyd043TnNwSEU/view?usp=sharing

(After the first command I saw 2 of 3 up)

Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170705/22bcde12/attachment.html>

On Wed, Jul 5, 2017 at 8:32 PM, Sahina Bose <sabose at redhat.com> wrote:
>
>
> On Wed, Jul 5, 2017 at 8:16 PM, Gianluca Cecchi <gianluca.cecchi at
gmail.com
> > wrote:
>
>>
>>
>> On Wed, Jul 5, 2017 at 7:42 AM, Sahina Bose <sabose at
redhat.com> wrote:
>>
>>>
>>>
>>>> ...
>>>>
>>>> then the commands I need to run would be:
>>>>
>>>> gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
>>>> start
>>>> gluster volume reset-brick export
ovirt01.localdomain.local:/gluster/brick3/export
>>>> gl01.localdomain.local:/gluster/brick3/export commit force
>>>>
>>>> Correct?
>>>>
>>>
>>> Yes, correct. gl01.localdomain.local should resolve correctly on
all 3
>>> nodes.
>>>
>>
>>
>> It fails at first step:
>>
>>  [root at ovirt01 ~]# gluster volume reset-brick export
>> ovirt01.localdomain.local:/gluster/brick3/export start
>> volume reset-brick: failed: Cannot execute command. The cluster is
>> operating at version 30712. reset-brick command reset-brick start is
>> unavailable in this version.
>> [root at ovirt01 ~]#
>>
>> It seems somehow in relation with this upgrade not of the commercial
>> solution Red Hat Gluster Storage
>> https://access.redhat.com/documentation/en-US/Red_Hat_Storag
>> e/3.1/html/Installation_Guide/chap-Upgrading_Red_Hat_Storage.html
>>
>> So ti seems I have to run some command of type:
>>
>> gluster volume set all cluster.op-version XXXXX
>>
>> with XXXXX > 30712
>>
>> It seems that latest version of commercial Red Hat Gluster Storage is
3.1
>> and its op-version is indeed 30712..
>>
>> So the question is which particular op-version I have to set and if the
>> command can be set online without generating disruption....
>>
>
> It should have worked with the glusterfs 3.10 version from Centos repo.
> Adding gluster-users for help on the op-version
>
This definitely means your cluster op-version is running < 3.9.0

         if (conf->op_version < GD_OP_VERSION_3_9_0
&&
            strcmp (cli_op, "GF_REPLACE_OP_COMMIT_FORCE"))
{
                snprintf (msg, sizeof (msg), "Cannot execute command. The
"
                          "cluster is operating at version %d. reset-brick
"
                          "command %s is unavailable in this
version.",

conf->op_version,
                          gd_rb_op_to_str
(cli_op));
                ret -1;
                goto
out;
        }

What's the version of gluster bits are you running across the gluster
cluster? Please note cluster.op-version is not exactly the same as of rpm
version and with every upgrades it's recommended to bump up the op-version.

>
>>
>> Thanks,
>> Gianluca
>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170705/d51d2c88/attachment.html>