Krishnan Parthasarathi
2015-Jun-10 08:27 UTC
[Gluster-users] Unable to get lock for uuid peers
> Hi all, > I two servers with 3.7.1 and have the same problem of this issue: > http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20693 > > My servers packages: > # rpm -qa | grep gluster | sort > glusterfs-3.7.1-1.el6.x86_64 > glusterfs-api-3.7.1-1.el6.x86_64 > glusterfs-cli-3.7.1-1.el6.x86_64 > glusterfs-client-xlators-3.7.1-1.el6.x86_64 > glusterfs-fuse-3.7.1-1.el6.x86_64 > glusterfs-geo-replication-3.7.1-1.el6.x86_64 > glusterfs-libs-3.7.1-1.el6.x86_64 > glusterfs-server-3.7.1-1.el6.x86_64 > > Command: > # gluster volume status > Another transaction is in progress. Please try again after sometime. > > > In /var/log/gluster/etc-glusterfs-glusterd.vol.log I found: > > [2015-06-09 16:12:38.949842] E [glusterd-utils.c:164:glusterd_lock] > 0-management: Unable to get lock for uuid: > 99a41a2a-2ce5-461c-aec0-510bd5b37bf2, lock held by: > 04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c > [2015-06-09 16:12:38.949864] E [glusterd-syncop.c:1766:gd_sync_task_begin] > 0-management: Unable to acquire lock > > I check the files: > From server 1: > # cat /var/lib/glusterd/peers/04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c > uuid=04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c > state=3 > hostname1=192.168.61.101 > > From server 2: > # cat /var/lib/glusterd/peers/99a41a2a-2ce5-461c-aec0-510bd5b37bf2 > uuid=99a41a2a-2ce5-461c-aec0-510bd5b37bf2 > state=3 > hostname1=192.168.61.100Could you attach the complete glusterd log file and cmd-history.log file under /var/log/glusterfs directory? Could you provide a more detailed listing of things you did before hitting this issue?
On 06/10/2015 10:27 AM, Krishnan Parthasarathi wrote:>> Hi all, >> I two servers with 3.7.1 and have the same problem of this issue: >> http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20693 >> >> My servers packages: >> # rpm -qa | grep gluster | sort >> glusterfs-3.7.1-1.el6.x86_64 >> glusterfs-api-3.7.1-1.el6.x86_64 >> glusterfs-cli-3.7.1-1.el6.x86_64 >> glusterfs-client-xlators-3.7.1-1.el6.x86_64 >> glusterfs-fuse-3.7.1-1.el6.x86_64 >> glusterfs-geo-replication-3.7.1-1.el6.x86_64 >> glusterfs-libs-3.7.1-1.el6.x86_64 >> glusterfs-server-3.7.1-1.el6.x86_64 >> >> Command: >> # gluster volume status >> Another transaction is in progress. Please try again after sometime. >> >> >> In /var/log/gluster/etc-glusterfs-glusterd.vol.log I found: >> >> [2015-06-09 16:12:38.949842] E [glusterd-utils.c:164:glusterd_lock] >> 0-management: Unable to get lock for uuid: >> 99a41a2a-2ce5-461c-aec0-510bd5b37bf2, lock held by: >> 04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c >> [2015-06-09 16:12:38.949864] E [glusterd-syncop.c:1766:gd_sync_task_begin] >> 0-management: Unable to acquire lock >> >> I check the files: >> From server 1: >> # cat /var/lib/glusterd/peers/04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c >> uuid=04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c >> state=3 >> hostname1=192.168.61.101 >> >> From server 2: >> # cat /var/lib/glusterd/peers/99a41a2a-2ce5-461c-aec0-510bd5b37bf2 >> uuid=99a41a2a-2ce5-461c-aec0-510bd5b37bf2 >> state=3 >> hostname1=192.168.61.100 > Could you attach the complete glusterd log file and cmd-history.log > file under /var/log/glusterfs directory? Could you provide a more > detailed listing of things you did before hitting this issue?Hi Krishnan, thanks to a quick answer. In attach you can found the two log you request: cmd_history.log etc-glusterfs-glusterd.vol.log We use the gluster volume as openstack nova, glance, cinder backend. The volume is configured using 2 bricks mounted by an iscsi device: [root at cld-stg-01 glusterfs]# gluster volume info volume-nova-prod Volume Name: volume-nova-prod Type: Distribute Volume ID: 4bbef4c8-0441-4e81-a2c5-559401adadc0 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 192.168.61.100:/brickOpenstack/nova-prod/mpathb Brick2: 192.168.61.101:/brickOpenstack/nova-prod/mpathb Options Reconfigured: storage.owner-gid: 162 storage.owner-uid: 162 Last week we update openstack from havana to icehouse and we rename the storage hosts but we didn't change the IP. All volume have been created using ip addresses. So last week we stop all services (openstack gluster and also iscsi). We change the name in DNS of private ip of the 2 nics. We reboot the storage servers We start agian iscsi, multipath, glusterd process. We have to stop and start the volumes, but after that everything works fine. Now we don't observe any other problems except this. We have a nagios probe which check the volume status each 5 minutes to ensure all gluster process is working fine and so we find this problem I post. Cheer Sergio -------------- next part -------------- A non-text attachment was scrubbed... Name: cmd_history.log Type: text/x-log Size: 161884 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150610/2af2a659/attachment-0002.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: etc-glusterfs-glusterd.vol.log Type: text/x-log Size: 875307 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150610/2af2a659/attachment-0003.bin>