Mohammed Rafi K C
2017-May-31 11:38 UTC
[Gluster-users] Snapshot auto-delete unmount problem
Can you give us more logs on this issue. Also by any chance did somebody unmount the lv's in any cases ? Regards Rafi KC On 05/31/2017 03:00 PM, Gary Lloyd wrote:> Hi I am having a problem deleting snapshots, gluster is failing to > unmount them. I am running centos 7.3 with gluster-3.10.2-1 > > here is some log output: > > [2017-05-31 09:21:39.961371] W [MSGID: 106057] > [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] > 0-management: Snap volume > 331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick1-b > not found [Invalid argument] > [2017-05-31 09:21:51.520811] W [MSGID: 106112] > [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management: > Soft-limit (value = 27) of volume shares1 is reached. Deleting > snapshot Snap_GMT-2017.05.31-09.20.04. > [2017-05-31 09:21:51.531729] E [MSGID: 106095] > [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: > umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 > failed (No such file or directory) [No such file or directory] > [2017-05-31 09:22:00.540373] E [MSGID: 106038] > [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] > 0-management: umount failed for path > /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 (brick: > /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such > file or directory. > [2017-05-31 09:22:02.442048] W [MSGID: 106033] > [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: > Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/, > err: Directory not empty. More than one glusterd running on this node. > [Directory not empty] > [2017-05-31 09:22:02.443336] W [MSGID: 106039] > [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] > 0-management: Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04 > [2017-05-31 09:22:02.444038] I [MSGID: 106144] > [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick > /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port 49157 > > > > Can anyone help ? > > Thanks > > > /Gary Lloyd/ > ________________________________________________ > I.T. Systems:Keele University > Finance & IT Directorate > Keele:Staffs:IC1 Building:ST5 5NB:UK > ________________________________________________ > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170531/f93bacce/attachment.html>
Here is the glusterd.log file as it has just happened with a manual removal of a snapshot. The lvs exist even after the snapshot gets removed. I'm also running samba on these with shadowcopy. Thanks *Gary Lloyd* ________________________________________________ I.T. Systems:Keele University Finance & IT Directorate Keele:Staffs:IC1 Building:ST5 5NB:UK +44 1782 733063 <%2B44%201782%20733073> ________________________________________________ On 31 May 2017 at 12:38, Mohammed Rafi K C <rkavunga at redhat.com> wrote:> Can you give us more logs on this issue. Also by any chance did somebody > unmount the lv's in any cases ? > > > Regards > > Rafi KC > > On 05/31/2017 03:00 PM, Gary Lloyd wrote: > > Hi I am having a problem deleting snapshots, gluster is failing to unmount > them. I am running centos 7.3 with gluster-3.10.2-1 > > here is some log output: > > [2017-05-31 09:21:39.961371] W [MSGID: 106057] [glusterd-snapshot-utils.c: > 410:glusterd_snap_volinfo_find] 0-management: Snap volume > 331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps- > 331ec972f90d494d8a86dd4f69d718b7-brick1-b not found [Invalid argument] > [2017-05-31 09:21:51.520811] W [MSGID: 106112] [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] > 0-management: Soft-limit (value = 27) of volume shares1 is reached. > Deleting snapshot Snap_GMT-2017.05.31-09.20.04. > [2017-05-31 09:21:51.531729] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount] > 0-management: umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 > failed (No such file or directory) [No such file or directory] > [2017-05-31 09:22:00.540373] E [MSGID: 106038] [glusterd-snapshot.c:2895: > glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path > /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 (brick: > /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such > file or directory. > [2017-05-31 09:22:02.442048] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] > 0-management: Failed to rmdir: /run/gluster/snaps/ > 4f980da64dec424ba0b48d6d36c4c54e/, err: Directory not empty. More than > one glusterd running on this node. [Directory not empty] > [2017-05-31 09:22:02.443336] W [MSGID: 106039] > [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management: > Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04 > [2017-05-31 09:22:02.444038] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove] > 0-pmap: removing brick /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b > on port 49157 > > > > Can anyone help ? > > Thanks > > > *Gary Lloyd* > ________________________________________________ > I.T. Systems:Keele University > Finance & IT Directorate > Keele:Staffs:IC1 Building:ST5 5NB:UK > ________________________________________________ > > > _______________________________________________ > Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170531/b76502a3/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterd.log Type: text/x-log Size: 262889 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170531/b76502a3/attachment.bin>
Incase anyone is interested this issue was caused by turning on brick multiplexing. Switching it off made the problem go away... *Gary Lloyd* ________________________________________________ I.T. Systems:Keele University Finance & IT Directorate Keele:Staffs:IC1 Building:ST5 5NB:UK +44 1782 733063 <%2B44%201782%20733073> ________________________________________________ On 31 May 2017 at 14:46, Gary Lloyd <g.lloyd at keele.ac.uk> wrote:> Here is the glusterd.log file as it has just happened with a manual > removal of a snapshot. > The lvs exist even after the snapshot gets removed. I'm also running samba > on these with shadowcopy. > > Thanks > > *Gary Lloyd* > ________________________________________________ > I.T. Systems:Keele University > Finance & IT Directorate > Keele:Staffs:IC1 Building:ST5 5NB:UK > +44 1782 733063 <%2B44%201782%20733073> > ________________________________________________ > > On 31 May 2017 at 12:38, Mohammed Rafi K C <rkavunga at redhat.com> wrote: > >> Can you give us more logs on this issue. Also by any chance did somebody >> unmount the lv's in any cases ? >> >> >> Regards >> >> Rafi KC >> >> On 05/31/2017 03:00 PM, Gary Lloyd wrote: >> >> Hi I am having a problem deleting snapshots, gluster is failing to >> unmount them. I am running centos 7.3 with gluster-3.10.2-1 >> >> here is some log output: >> >> [2017-05-31 09:21:39.961371] W [MSGID: 106057] >> [glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management: >> Snap volume 331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snap >> s-331ec972f90d494d8a86dd4f69d718b7-brick1-b not found [Invalid argument] >> [2017-05-31 09:21:51.520811] W [MSGID: 106112] >> [glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management: >> Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot >> Snap_GMT-2017.05.31-09.20.04. >> [2017-05-31 09:21:51.531729] E [MSGID: 106095] >> [glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting >> /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 failed (No >> such file or directory) [No such file or directory] >> [2017-05-31 09:22:00.540373] E [MSGID: 106038] >> [glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management: >> umount failed for path /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 >> (brick: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): >> No such file or directory. >> [2017-05-31 09:22:02.442048] W [MSGID: 106033] >> [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management: >> Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/, >> err: Directory not empty. More than one glusterd running on this node. >> [Directory not empty] >> [2017-05-31 09:22:02.443336] W [MSGID: 106039] >> [glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management: >> Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04 >> [2017-05-31 09:22:02.444038] I [MSGID: 106144] >> [glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick >> /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port >> 49157 >> >> >> >> Can anyone help ? >> >> Thanks >> >> >> *Gary Lloyd* >> ________________________________________________ >> I.T. Systems:Keele University >> Finance & IT Directorate >> Keele:Staffs:IC1 Building:ST5 5NB:UK >> ________________________________________________ >> >> >> _______________________________________________ >> Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170713/1b0f59f7/attachment.html>