FĂ©lix de Lelelis
2015-Mar-31 07:05 UTC
[Gluster-users] Metadata filesystem XFS gluster 3.6
Hi, I had a problem with a filesystem xfs on gluster. The filesystem metadata was filled: Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: space map metadata: unable to allocate new metadata block Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: 252:2: metadata operation 'dm_thin_insert_block' failed: error = -28 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: 252:2: aborting current metadata transaction Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: 252:2: switching pool to read-only mode Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: block 0x701830 ("xfs_buf_iodone_callbacks") error 5 numblks 8 Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of device Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: process_bio_read_only: dm_thin_find_block() failed: error = -5 Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of device Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: process_bio_read_only: dm_thin_find_block() failed: error = -5 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: block 0x68047c ("xlog_iodone") error 5 numblks 64 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): xfs_do_force_shutdown(0x2) called from line 1170 of file fs/xfs/xfs_log.c. Return address 0xffffffffa012a4c1 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Log I/O Error Detected. Shutting down filesystem Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Please umount the filesystem and rectify the problem(s) Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of device Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: process_bio_read_only: dm_thin_find_block() failed: error = -5 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: block 0x6804bc ("xlog_iodone") error 5 numblks 64 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): xfs_do_force_shutdown(0x2) called from line 1170 of file fs/xfs/xfs_log.c. Return address 0xffffffffa012a4c1 Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of device Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): xfs_log_force: error 5 returned. Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: process_bio_read_only: dm_thin_find_block() failed: error = -5 Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of device Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: process_bio_read_only: dm_thin_find_block() failed: error = -5 Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of device Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: process_bio_read_only: dm_thin_find_block() failed: error = -5 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: block 0x6804fc ("xlog_iodone") error 5 numblks 64 Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): xfs_do_force_shutdown(0x2) called from line 1170 of file fs/xfs/xfs_log.c. Return address 0xffffffffa012a4c After that, gluster was shutdown and with it the 2 server are shtudown too. The lvm partition was missing and so far I haven't been able restore the file system. All data is missing?? I don't understand the situation and I don't know if it's due a xfs filesystem or glusterfs fail. Someone it has been this situation? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150331/9858bd68/attachment.html>
Am Di, 31.03.2015, 09:05 schrieb F?lix de Lelelis:> > I had a problem with a filesystem xfs on gluster. The filesystem metadata > was filled [...]I'm not sure if we've had exactly the same issue, but recently our GlusterFS failed due to one XFS brick being filled completely (about 60 KB free). Might be related? I haven't found a proper solution for it yet. I assumed that "cluster.min_free_disk" would do the trick, but that only triggers warnings to be written to the logs, but doesn't prevent the bricks from being filled up beyond XFS "comfort zone" ;) Regards, Pb
On 03/31/2015 09:05 AM, F?lix de Lelelis wrote:> Hi, > > I had a problem with a filesystem xfs on gluster. The filesystem > metadata was filled: > > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: space map > metadata: unable to allocate new metadata block > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: 252:2: > metadata operation 'dm_thin_insert_block' failed: error = -28 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: 252:2: > aborting current metadata transaction > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: 252:2: > switching pool to read-only mode > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: > block 0x701830 ("xfs_buf_iodone_callbacks") error 5 numblks 8 > Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of > device > Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: > process_bio_read_only: dm_thin_find_block() failed: error = -5 > Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of > device > Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: > process_bio_read_only: dm_thin_find_block() failed: error = -5 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: > block 0x68047c ("xlog_iodone") error 5 numblks 64 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): > xfs_do_force_shutdown(0x2) called from line 1170 of file > fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c1 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Log I/O Error > Detected. Shutting down filesystem > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Please umount the > filesystem and rectify the problem(s) > Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of > device > Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: > process_bio_read_only: dm_thin_find_block() failed: error = -5 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: > block 0x6804bc ("xlog_iodone") error 5 numblks 64 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): > xfs_do_force_shutdown(0x2) called from line 1170 of file > fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c1 > Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of > device > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): xfs_log_force: error > 5 returned. > Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: > process_bio_read_only: dm_thin_find_block() failed: error = -5 > Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of > device > Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: > process_bio_read_only: dm_thin_find_block() failed: error = -5 > Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access beyond end of > device > Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0, want=562056, limit=24576 > Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin: > process_bio_read_only: dm_thin_find_block() failed: error = -5 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata I/O error: > block 0x6804fc ("xlog_iodone") error 5 numblks 64 > Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): > xfs_do_force_shutdown(0x2) called from line 1170 of file > fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c > > > > After that, gluster was shutdown and with it the 2 server are shtudown > too. The lvm partition was missing and so far I haven't been able > restore the file system. All data is missing?? > I don't understand the situation and I don't know if it's due a xfs > filesystem or glusterfs fail. Someone it has been this situation? > > Thanks. > >Hi, Do you run thin provisioned on LVM? There are some discussions about this:> https://www.redhat.com/archives/linux-lvm/2014-December/msg00015.html> You definitely ran out of metadata space. Which version of the kernel > and lvm2 userspace are you using? > > See the "Metadata space exhaustion" section of the lvmthin manpage in a > recent lvm2 release to guide you on how to recover. > > Also, once you've gotten past ths you really should configure lvm2 to > autoextend the thin-pool (both data and metadata) as needed in response > to low watermark, etc. See "Automatically extend thin pool LV" in > lvmthin manpage.Maybe related to:> https://bugzilla.redhat.com/show_bug.cgi?id=1097948Hope this helps. Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info at netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ---------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150331/e5db366f/attachment.html>