thr3ads.net - Gluster users - [Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3

If this information is useful, please help other people find it:
Share via:

Paolo Margara

2017-Jul-20 09:04 UTC

[Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3_lock

Hi list,

recently I've noted a strange behaviour of my gluster storage, sometimes
while executing a simple command like "gluster volume status
vm-images-repo" as a response I got "Another transaction is in
progress
for vm-images-repo. Please try again after sometime.". This situation
does not get solved simply waiting for but I've to restart glusterd on
the node that hold (and does not release) the lock, this situation occur
randomly after some days. In the meanwhile, prior and after the issue
appear, everything is working as expected.

I'm using gluster 3.8.12 on CentOS 7.3, the only relevant information
that I found on the log file (etc-glusterfs-glusterd.vol.log) of my
three nodes are the following:

* node1, at the moment the issue begins:

[2017-07-19 15:07:43.130203] W
[glusterd-locks.c:572:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
[0x7f373f25f00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
[0x7f373f250a25]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
[0x7f373f2f548f] ) 0-management: Lock for vm-images-repo held by
2c6f154f-efe3-4479-addc-b2021aa9d5df
[2017-07-19 15:07:43.128242] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume vm-images-repo
[2017-07-19 15:07:43.130244] E [MSGID: 106119]
[glusterd-op-sm.c:3782:glusterd_op_ac_lock] 0-management: Unable to
acquire lock for vm-images-repo
[2017-07-19 15:07:43.130320] E [MSGID: 106376]
[glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler returned: -1
[2017-07-19 15:07:43.130665] E [MSGID: 106116]
[glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Locking
failed on virtnode-0-1-gluster. Please check log file for details.
[2017-07-19 15:07:43.131293] E [MSGID: 106116]
[glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Locking
failed on virtnode-0-2-gluster. Please check log file for details.
[2017-07-19 15:07:43.131360] E [MSGID: 106151]
[glusterd-syncop.c:1884:gd_sync_task_begin] 0-management: Locking Peers
Failed.
[2017-07-19 15:07:43.132005] E [MSGID: 106116]
[glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Unlocking
failed on virtnode-0-2-gluster. Please check log file for details.
[2017-07-19 15:07:43.132182] E [MSGID: 106116]
[glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Unlocking
failed on virtnode-0-1-gluster. Please check log file for details.

* node2, at the moment the issue begins:

[2017-07-19 15:07:43.131975] W
[glusterd-locks.c:572:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
[0x7f17b5b9e00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
[0x7f17b5b8fa25]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
[0x7f17b5c3448f] ) 0-management: Lock for vm-images-repo held by
d9047ecd-26b5-467b-8e91-50f76a0c4d16
[2017-07-19 15:07:43.132019] E [MSGID: 106119]
[glusterd-op-sm.c:3782:glusterd_op_ac_lock] 0-management: Unable to
acquire lock for vm-images-repo
[2017-07-19 15:07:43.133568] W
[glusterd-locks.c:686:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
[0x7f17b5b9e00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2b712)
[0x7f17b5b8f712]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd082a)
[0x7f17b5c3482a] ) 0-management: Lock owner mismatch. Lock for vol
vm-images-repo held by d9047ecd-26b5-467b-8e91-50f76a0c4d16
[2017-07-19 15:07:43.133597] E [MSGID: 106118]
[glusterd-op-sm.c:3845:glusterd_op_ac_unlock] 0-management: Unable to
release lock for vm-images-repo
The message "E [MSGID: 106376] [glusterd-op-sm.c:7775:glusterd_op_sm]
0-management: handler returned: -1" repeated 3 times between [2017-07-19
15:07:42.976193] and [2017-07-19 15:07:43.133646]

* node3, at the moment the issue begins:

[2017-07-19 15:07:42.976593] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume vm-images-repo
[2017-07-19 15:07:43.129941] W
[glusterd-locks.c:572:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
[0x7f6133f5b00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
[0x7f6133f4ca25]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
[0x7f6133ff148f] ) 0-management: Lock for vm-images-repo held by
d9047ecd-26b5-467b-8e91-50f76a0c4d16
[2017-07-19 15:07:43.129981] E [MSGID: 106119]
[glusterd-op-sm.c:3782:glusterd_op_ac_lock] 0-management: Unable to
acquire lock for vm-images-repo
[2017-07-19 15:07:43.130034] E [MSGID: 106376]
[glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler returned: -1
[2017-07-19 15:07:43.130131] E [MSGID: 106275]
[glusterd-rpc-ops.c:876:glusterd_mgmt_v3_lock_peers_cbk_fn]
0-management: Received mgmt_v3 lock RJT from uuid:
2c6f154f-efe3-4479-addc-b2021aa9d5df
[2017-07-19 15:07:43.130710] W
[glusterd-locks.c:686:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
[0x7f6133f5b00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2b712)
[0x7f6133f4c712]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd082a)
[0x7f6133ff182a] ) 0-management: Lock owner mismatch. Lock for vol
vm-images-repo held by d9047ecd-26b5-467b-8e91-50f76a0c4d16
[2017-07-19 15:07:43.130733] E [MSGID: 106118]
[glusterd-op-sm.c:3845:glusterd_op_ac_unlock] 0-management: Unable to
release lock for vm-images-repo
[2017-07-19 15:07:43.130771] E [MSGID: 106376]
[glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler returned: -1

The thing that is really strange is that in this case the uuid of node3
is d9047ecd-26b5-467b-8e91-50f76a0c4d16!

The mapping nodename-uuid is:

* (node1) virtnode-0-0-gluster: 2c6f154f-efe3-4479-addc-b2021aa9d5df

* (node2) virtnode-0-1-gluster: e93ebee7-5d95-4100-a9df-4a3e60134b73

* (node3) virtnode-0-2-gluster: d9047ecd-26b5-467b-8e91-50f76a0c4d16

In this case restarting glusterd on node3 usually solve the issue.

What could be the root cause of this behavior? How can I fix this once
and for all?

If needed I could provide the full log file.


Greetings,

    Paolo Margara

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170720/08a2eecf/attachment.html>

Atin Mukherjee

2017-Jul-20 09:38 UTC

head link

[Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3_lock

Please share the cmd_history.log file from all the storage nodes.

On Thu, Jul 20, 2017 at 2:34 PM, Paolo Margara <paolo.margara at
polito.it>
wrote:
> Hi list,
>
> recently I've noted a strange behaviour of my gluster storage,
sometimes
> while executing a simple command like "gluster volume status
> vm-images-repo" as a response I got "Another transaction is in
progress for
> vm-images-repo. Please try again after sometime.". This situation does
not
> get solved simply waiting for but I've to restart glusterd on the node
that
> hold (and does not release) the lock, this situation occur randomly after
> some days. In the meanwhile, prior and after the issue appear, everything
> is working as expected.
>
> I'm using gluster 3.8.12 on CentOS 7.3, the only relevant information
that
> I found on the log file (etc-glusterfs-glusterd.vol.log) of my three
> nodes are the following:
>
> * node1, at the moment the issue begins:
>
> [2017-07-19 15:07:43.130203] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock]
> (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
> [0x7f373f25f00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
> [0x7f373f250a25]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
> [0x7f373f2f548f] ) 0-management: Lock for vm-images-repo held by
> 2c6f154f-efe3-4479-addc-b2021aa9d5df
> [2017-07-19 15:07:43.128242] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
> 0-management: Received status volume req for volume vm-images-repo
> [2017-07-19 15:07:43.130244] E [MSGID: 106119]
[glusterd-op-sm.c:3782:glusterd_op_ac_lock]
> 0-management: Unable to acquire lock for vm-images-repo
> [2017-07-19 15:07:43.130320] E [MSGID: 106376]
[glusterd-op-sm.c:7775:glusterd_op_sm]
> 0-management: handler returned: -1
> [2017-07-19 15:07:43.130665] E [MSGID: 106116]
> [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Locking
> failed on virtnode-0-1-gluster. Please check log file for details.
> [2017-07-19 15:07:43.131293] E [MSGID: 106116]
> [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Locking
> failed on virtnode-0-2-gluster. Please check log file for details.
> [2017-07-19 15:07:43.131360] E [MSGID: 106151]
[glusterd-syncop.c:1884:gd_sync_task_begin]
> 0-management: Locking Peers Failed.
> [2017-07-19 15:07:43.132005] E [MSGID: 106116]
> [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Unlocking
> failed on virtnode-0-2-gluster. Please check log file for details.
> [2017-07-19 15:07:43.132182] E [MSGID: 106116]
> [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Unlocking
> failed on virtnode-0-1-gluster. Please check log file for details.
>
> * node2, at the moment the issue begins:
>
> [2017-07-19 15:07:43.131975] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock]
> (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
> [0x7f17b5b9e00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
> [0x7f17b5b8fa25]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
> [0x7f17b5c3448f] ) 0-management: Lock for vm-images-repo held by
> d9047ecd-26b5-467b-8e91-50f76a0c4d16
> [2017-07-19 15:07:43.132019] E [MSGID: 106119]
[glusterd-op-sm.c:3782:glusterd_op_ac_lock]
> 0-management: Unable to acquire lock for vm-images-repo
> [2017-07-19 15:07:43.133568] W
[glusterd-locks.c:686:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
> [0x7f17b5b9e00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2b712)
> [0x7f17b5b8f712]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd082a)
> [0x7f17b5c3482a] ) 0-management: Lock owner mismatch. Lock for vol
> vm-images-repo held by d9047ecd-26b5-467b-8e91-50f76a0c4d16
> [2017-07-19 15:07:43.133597] E [MSGID: 106118]
[glusterd-op-sm.c:3845:glusterd_op_ac_unlock]
> 0-management: Unable to release lock for vm-images-repo
> The message "E [MSGID: 106376] [glusterd-op-sm.c:7775:glusterd_op_sm]
> 0-management: handler returned: -1" repeated 3 times between
[2017-07-19
> 15:07:42.976193] and [2017-07-19 15:07:43.133646]
>
> * node3, at the moment the issue begins:
>
> [2017-07-19 15:07:42.976593] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
> 0-management: Received status volume req for volume vm-images-repo
> [2017-07-19 15:07:43.129941] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock]
> (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
> [0x7f6133f5b00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
> [0x7f6133f4ca25]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
> [0x7f6133ff148f] ) 0-management: Lock for vm-images-repo held by
> d9047ecd-26b5-467b-8e91-50f76a0c4d16
> [2017-07-19 15:07:43.129981] E [MSGID: 106119]
[glusterd-op-sm.c:3782:glusterd_op_ac_lock]
> 0-management: Unable to acquire lock for vm-images-repo
> [2017-07-19 15:07:43.130034] E [MSGID: 106376]
[glusterd-op-sm.c:7775:glusterd_op_sm]
> 0-management: handler returned: -1
> [2017-07-19 15:07:43.130131] E [MSGID: 106275] [glusterd-rpc-ops.c:876:
> glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock
> RJT from uuid: 2c6f154f-efe3-4479-addc-b2021aa9d5df
> [2017-07-19 15:07:43.130710] W
[glusterd-locks.c:686:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
> [0x7f6133f5b00f]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2b712)
> [0x7f6133f4c712]
-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd082a)
> [0x7f6133ff182a] ) 0-management: Lock owner mismatch. Lock for vol
> vm-images-repo held by d9047ecd-26b5-467b-8e91-50f76a0c4d16
> [2017-07-19 15:07:43.130733] E [MSGID: 106118]
[glusterd-op-sm.c:3845:glusterd_op_ac_unlock]
> 0-management: Unable to release lock for vm-images-repo
> [2017-07-19 15:07:43.130771] E [MSGID: 106376]
[glusterd-op-sm.c:7775:glusterd_op_sm]
> 0-management: handler returned: -1
>
> The thing that is really strange is that in this case the uuid of node3 is
> d9047ecd-26b5-467b-8e91-50f76a0c4d16!
>
> The mapping nodename-uuid is:
>
> * (node1) virtnode-0-0-gluster: 2c6f154f-efe3-4479-addc-b2021aa9d5df
>
> * (node2) virtnode-0-1-gluster: e93ebee7-5d95-4100-a9df-4a3e60134b73
>
> * (node3) virtnode-0-2-gluster: d9047ecd-26b5-467b-8e91-50f76a0c4d16
>
> In this case restarting glusterd on node3 usually solve the issue.
>
> What could be the root cause of this behavior? How can I fix this once
> and for all?
>
> If needed I could provide the full log file.
>
>
> Greetings,
>
>     Paolo Margara
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170720/1ef7fee2/attachment.html>

Paolo Margara

2017-Jul-20 10:24 UTC

head link

[Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3_lock

In attachment the requested logs for all the three nodes.

thanks,

    Paolo


Il 20/07/2017 11:38, Atin Mukherjee ha scritto:> Please share the cmd_history.log file from all the storage nodes.
>
> On Thu, Jul 20, 2017 at 2:34 PM, Paolo Margara
> <paolo.margara at polito.it <mailto:paolo.margara at
polito.it>> wrote:
>
>     Hi list,
>
>     recently I've noted a strange behaviour of my gluster storage,
>     sometimes while executing a simple command like "gluster volume
>     status vm-images-repo" as a response I got "Another
transaction is
>     in progress for vm-images-repo. Please try again after sometime.".
>     This situation does not get solved simply waiting for but I've to
>     restart glusterd on the node that hold (and does not release) the
>     lock, this situation occur randomly after some days. In the
>     meanwhile, prior and after the issue appear, everything is working
>     as expected.
>
>     I'm using gluster 3.8.12 on CentOS 7.3, the only relevant
>     information that I found on the log file
>     (etc-glusterfs-glusterd.vol.log) of my three nodes are the following:
>
>     * node1, at the moment the issue begins:
>
>     [2017-07-19 15:07:43.130203] W
>     [glusterd-locks.c:572:glusterd_mgmt_v3_lock]
>     (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
>     [0x7f373f25f00f]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
>     [0x7f373f250a25]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
>     [0x7f373f2f548f] ) 0-management: Lock for vm-images-repo held by
>     2c6f154f-efe3-4479-addc-b2021aa9d5df
>     [2017-07-19 15:07:43.128242] I [MSGID: 106499]
>     [glusterd-handler.c:4349:__glusterd_handle_status_volume]
>     0-management: Received status volume req for volume vm-images-repo
>     [2017-07-19 15:07:43.130244] E [MSGID: 106119]
>     [glusterd-op-sm.c:3782:glusterd_op_ac_lock] 0-management: Unable
>     to acquire lock for vm-images-repo
>     [2017-07-19 15:07:43.130320] E [MSGID: 106376]
>     [glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler
>     returned: -1
>     [2017-07-19 15:07:43.130665] E [MSGID: 106116]
>     [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management:
>     Locking failed on virtnode-0-1-gluster. Please check log file for
>     details.
>     [2017-07-19 15:07:43.131293] E [MSGID: 106116]
>     [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management:
>     Locking failed on virtnode-0-2-gluster. Please check log file for
>     details.
>     [2017-07-19 15:07:43.131360] E [MSGID: 106151]
>     [glusterd-syncop.c:1884:gd_sync_task_begin] 0-management: Locking
>     Peers Failed.
>     [2017-07-19 15:07:43.132005] E [MSGID: 106116]
>     [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management:
>     Unlocking failed on virtnode-0-2-gluster. Please check log file
>     for details.
>     [2017-07-19 15:07:43.132182] E [MSGID: 106116]
>     [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management:
>     Unlocking failed on virtnode-0-1-gluster. Please check log file
>     for details.
>
>     * node2, at the moment the issue begins:
>
>     [2017-07-19 15:07:43.131975] W
>     [glusterd-locks.c:572:glusterd_mgmt_v3_lock]
>     (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
>     [0x7f17b5b9e00f]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
>     [0x7f17b5b8fa25]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
>     [0x7f17b5c3448f] ) 0-management: Lock for vm-images-repo held by
>     d9047ecd-26b5-467b-8e91-50f76a0c4d16
>     [2017-07-19 15:07:43.132019] E [MSGID: 106119]
>     [glusterd-op-sm.c:3782:glusterd_op_ac_lock] 0-management: Unable
>     to acquire lock for vm-images-repo
>     [2017-07-19 15:07:43.133568] W
>     [glusterd-locks.c:686:glusterd_mgmt_v3_unlock]
>     (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
>     [0x7f17b5b9e00f]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2b712)
>     [0x7f17b5b8f712]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd082a)
>     [0x7f17b5c3482a] ) 0-management: Lock owner mismatch. Lock for vol
>     vm-images-repo held by d9047ecd-26b5-467b-8e91-50f76a0c4d16
>     [2017-07-19 15:07:43.133597] E [MSGID: 106118]
>     [glusterd-op-sm.c:3845:glusterd_op_ac_unlock] 0-management: Unable
>     to release lock for vm-images-repo
>     The message "E [MSGID: 106376]
>     [glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler
>     returned: -1" repeated 3 times between [2017-07-19
>     15:07:42.976193] and [2017-07-19 15:07:43.133646]
>
>     * node3, at the moment the issue begins:
>
>     [2017-07-19 15:07:42.976593] I [MSGID: 106499]
>     [glusterd-handler.c:4349:__glusterd_handle_status_volume]
>     0-management: Received status volume req for volume vm-images-repo
>     [2017-07-19 15:07:43.129941] W
>     [glusterd-locks.c:572:glusterd_mgmt_v3_lock]
>     (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
>     [0x7f6133f5b00f]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2ba25)
>     [0x7f6133f4ca25]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd048f)
>     [0x7f6133ff148f] ) 0-management: Lock for vm-images-repo held by
>     d9047ecd-26b5-467b-8e91-50f76a0c4d16
>     [2017-07-19 15:07:43.129981] E [MSGID: 106119]
>     [glusterd-op-sm.c:3782:glusterd_op_ac_lock] 0-management: Unable
>     to acquire lock for vm-images-repo
>     [2017-07-19 15:07:43.130034] E [MSGID: 106376]
>     [glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler
>     returned: -1
>     [2017-07-19 15:07:43.130131] E [MSGID: 106275]
>     [glusterd-rpc-ops.c:876:glusterd_mgmt_v3_lock_peers_cbk_fn]
>     0-management: Received mgmt_v3 lock RJT from uuid:
>     2c6f154f-efe3-4479-addc-b2021aa9d5df
>     [2017-07-19 15:07:43.130710] W
>     [glusterd-locks.c:686:glusterd_mgmt_v3_unlock]
>     (-->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x3a00f)
>     [0x7f6133f5b00f]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0x2b712)
>     [0x7f6133f4c712]
>     -->/usr/lib64/glusterfs/3.8.12/xlator/mgmt/glusterd.so(+0xd082a)
>     [0x7f6133ff182a] ) 0-management: Lock owner mismatch. Lock for vol
>     vm-images-repo held by d9047ecd-26b5-467b-8e91-50f76a0c4d16
>     [2017-07-19 15:07:43.130733] E [MSGID: 106118]
>     [glusterd-op-sm.c:3845:glusterd_op_ac_unlock] 0-management: Unable
>     to release lock for vm-images-repo
>     [2017-07-19 15:07:43.130771] E [MSGID: 106376]
>     [glusterd-op-sm.c:7775:glusterd_op_sm] 0-management: handler
>     returned: -1
>
>     The thing that is really strange is that in this case the uuid of
>     node3 is d9047ecd-26b5-467b-8e91-50f76a0c4d16!
>
>     The mapping nodename-uuid is:
>
>     * (node1) virtnode-0-0-gluster: 2c6f154f-efe3-4479-addc-b2021aa9d5df
>
>     * (node2) virtnode-0-1-gluster: e93ebee7-5d95-4100-a9df-4a3e60134b73
>
>     * (node3) virtnode-0-2-gluster: d9047ecd-26b5-467b-8e91-50f76a0c4d16
>
>     In this case restarting glusterd on node3 usually solve the issue.
>
>     What could be the root cause of this behavior? How can I fix this
>     once and for all?
>
>     If needed I could provide the full log file.
>
>
>     Greetings,
>
>         Paolo Margara
>
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     http://lists.gluster.org/mailman/listinfo/gluster-users
>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170720/71420655/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node1-cmd_history.log.bz2
Type: application/x-bzip
Size: 73249 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170720/71420655/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node2-cmd_history.log.bz2
Type: application/x-bzip
Size: 59657 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170720/71420655/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node3-cmd_history.log.bz2
Type: application/x-bzip
Size: 283831 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170720/71420655/attachment-0005.bin>

Reasonably Related Threads

Search for more maybe matching threads

Gluster users - Jul 2017 - glusterd-locks.c:572:glusterd_mgmt_v3_lock

[Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3_lock

[Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3_lock

[Gluster-users] glusterd-locks.c:572:glusterd_mgmt_v3_lock

Reasonably Related Threads