Wojciech Turek
2009-Jan-22 15:44 UTC
mds reports that OST is full (ENOSPC error -28 ) but df tells different
Hello, RHEL4 Kernel 2.6.9-67.0.22smp Lustre-1.6.6 Lustre MDS report following error: Jan 22 15:20:40 mds01.beowulf.cluster kernel: LustreError: 24680:0:(lov_request.c:692:lov_update_create_set()) error creating fid 0xeb79c9d sub-object on OST idx 4/1: rc = -28 Which I translate as that one of the OST (index 4/1) is full and has no space left on device. OSS seem to be consistent and says: Jan 22 15:21:15 storage08.beowulf.cluster kernel: LustreError: 23507:0:(filter_io_26.c:721:filter_commitrw_write()) error starting transaction: rc = -30 Which I translate as Client would like to write to an existing file but it can''t because file system is read only. The OST device is still mounted with rw option /etc/mtab contains line - /dev/dm-8 /mnt/ddn-data/ost7 lustre rw 0 0 I tried to create new objects on this OST (by explicitly striping to it `lfs setstripe -c1 -i4 testdir`) but it seem that lustre automatically skips it and goes to the next one. I think that lustre finds out that OST(idx4) is full end it prevents clients from writing to it and allows them to read only access, is that correct? Now the main question is why Lustre thinks that OST(idx4) is full? df command on the OSS mounting this OST tells something opposite: df -h /dev/dm-8 3.6T 3.0T 465G 87% /mnt/ddn-data/ost7 df -i /dev/dm-8 244195328 1055690 243139638 1% /mnt/ddn-data/ost7 So it seem that there is still 465Gbytes left and plenty of inodes on that OST. Is it possible that this OST have meny orphaned objects which takes all the available space? Is there a way of reclaiming back this free space? Regards, Wojciech -- tunefs.lustre --print /dev/dm-8 checking for existing Lustre data: found CONFIGS/mountdata Reading CONFIGS/mountdata Target: ddn_data-OST0004 Index: 4 Lustre FS: ddn_data Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro,extents,mballoc _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Brian J. Murrell
2009-Jan-22 16:02 UTC
[Lustre-discuss] mds reports that OST is full (ENOSPC error -28 ) but df tells different
On Thu, 2009-01-22 at 15:44 +0000, Wojciech Turek wrote:> Hello,Hi,> Lustre MDS report following error: > Jan 22 15:20:40 mds01.beowulf.cluster kernel: LustreError: > 24680:0:(lov_request.c:692:lov_update_create_set()) error creating fid > 0xeb79c9d sub-object on OST idx 4/1: rc = -28-28 is ENOSPC.> Which I translate as that one of the OST (index 4/1) is full and has > no space left on device.Yes.> OSS seem to be consistent and says: > Jan 22 15:21:15 storage08.beowulf.cluster kernel: LustreError: > 23507:0:(filter_io_26.c:721:filter_commitrw_write()) error starting > transaction: rc = -30Hrm. I''m not sure a -30 (EROFS) would translate to a -28 to the MDS. I think it would also be a -30. So are you sure you are looking at correlating messages? The timestamps, if the two nodes are in sync also seem to indicate a lack of correlation with 35s of disparity. Perhaps there is an actual -28 in the OSS log prior to the -30 one?> Which I translate as Client would like to write to an existing file > but it can''t because file system is read only.Indeed. But why is it read-only? There should be an event in the OSS log saying that it was turning the filesystem read-only.> The OST device is still mounted with rw optionYeah. That''s just the state at mount time. Lustre will set a device read-only in the case of filesystem errors, as one example.> Now the main question is why Lustre thinks that OST(idx4) is full?No, I think the main question is why is it read-only. The full situation may have been transient where it filled up momentarily and then some objects were removed. In any case, this is a secondary issue and really only need be considered once the read-only situation is understood.> Is it possible that this OST have meny orphaned objects which takes > all the available space?That would be reflected in the df. If you suspect there may be orphan objects though, you could lfsck to verify and clean.> Is there a way of reclaiming back this free space?If you mean orphaned OST objects, then lfsck. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090122/d90d212a/attachment.bin
Wojciech Turek
2009-Jan-22 17:27 UTC
Re: mds reports that OST is full (ENOSPC error -28 ) but df tells different
Hi Brian, Brian J. Murrell wrote: On Thu, 2009-01-22 at 15:44 +0000, Wojciech Turek wrote: Hello, Hi, Lustre MDS report following error: Jan 22 15:20:40 mds01.beowulf.cluster kernel: LustreError: 24680:0:(lov_request.c:692:lov_update_create_set()) error creating fid 0xeb79c9d sub-object on OST idx 4/1: rc = -28 -28 is ENOSPC. Which I translate as that one of the OST (index 4/1) is full and has no space left on device. Yes. OSS seem to be consistent and says: Jan 22 15:21:15 storage08.beowulf.cluster kernel: LustreError: 23507:0:(filter_io_26.c:721:filter_commitrw_write()) error starting transaction: rc = -30 Hrm. I''m not sure a -30 (EROFS) would translate to a -28 to the MDS. I think it would also be a -30. So are you sure you are looking at correlating messages? The timestamps, if the two nodes are in sync also seem to indicate a lack of correlation with 35s of disparity. Perhaps there is an actual -28 in the OSS log prior to the -30 one? Yes you are right, there is plenty of messages with -30 in the logs and probably they are not related to -28. Which I translate as Client would like to write to an existing file but it can''t because file system is read only. Indeed. But why is it read-only? There should be an event in the OSS log saying that it was turning the filesystem read-only. The OST device is still mounted with rw option Yeah. That''s just the state at mount time. Lustre will set a device read-only in the case of filesystem errors, as one example. Now the main question is why Lustre thinks that OST(idx4) is full? No, I think the main question is why is it read-only. The full situation may have been transient where it filled up momentarily and then some objects were removed. In any case, this is a secondary issue and really only need be considered once the read-only situation is understood. Thank you for puting me in right track. I found these in the syslog Jan 22 10:18:40 storage08.beowulf.cluster kernel: LDISKFS-fs error (device dm-8): mb_free_blocks: double-free of inode 16203779''s block 688627718(bit 8198 in group 21015) Jan 22 10:18:40 storage08.beowulf.cluster kernel: Jan 22 10:18:40 storage08.beowulf.cluster kernel: Remounting filesystem read-only Is this means that the file system may be corrupted? I am going to run fsck -f on this device and try to mount it back, is that a right procedure? I did not find any errors on my S2A9500 storage, so I am not sure when this corruption could occur. Is it possible that this OST have meny orphaned objects which takes all the available space? That would be reflected in the df. If you suspect there may be orphan objects though, you could lfsck to verify and clean. Is there a way of reclaiming back this free space? If you mean orphaned OST objects, then lfsck. b. Cheers Wojciech _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Brian J. Murrell
2009-Jan-22 18:08 UTC
[Lustre-discuss] mds reports that OST is full (ENOSPC error -28 ) but df tells different
On Thu, 2009-01-22 at 17:27 +0000, Wojciech Turek wrote:> Hi Brian,Hi Wojciech,> Yes you are right, there is plenty of messages with -30 in the logs > and probably they are not related to -28.Indeed.> Thank you for puting me in right track.NP.> I found these in the syslog > Jan 22 10:18:40 storage08.beowulf.cluster kernel: LDISKFS-fs error > (device dm-8): mb_free_blocks: double-free of inode 16203779''s block > 688627718(bit 8198 in group 21015)Ouch.> Is this means that the file system may be corrupted?Yes, I''m afraid so. Do you have any storage hardware events to correlate some corruption?> I am going to run fsck -f on this device and try to mount it back, is > that a right procedure?Just because I am a paranoid old man I tend to add "-n" to my first fsck just to see how bad it is going to be and then proceed based on that.> I did not find any errors on my S2A9500 storage, so I am not sure > when this corruption could occur.Hrm. I guess that answers my previous question. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090122/a022fbdd/attachment.bin
Wojciech Turek
2009-Jan-22 18:19 UTC
Re: mds reports that OST is full (ENOSPC error -28 ) but df tells different
Hi Brian, I have tried to umount the OST(idx4) and the server LBUGed I attache LBUG below. Jan 22 17:32:30 storage08 kernel: Lustre: Failing over ddn_data-OST0004 Jan 22 17:32:30 storage08 kernel: Lustre: Skipped 1 previous similar message Jan 22 17:32:30 storage08 kernel: Lustre: *** setting obd ddn_data-OST0004 device ''unknown-block(253,8)'' read-only *** Jan 22 17:32:30 storage08 kernel: Lustre: Skipped 1 previous similar message Jan 22 17:32:30 storage08 kernel: Turning device dm-8 (0xfd00008) read-only Jan 22 17:32:30 storage08 kernel: Lustre: ddn_data-OST0004: shutting down for failover; client state will be preserved. Jan 22 17:32:30 storage08 kernel: LustreError: 8667:0:(lprocfs_status.c:779:lprocfs_free_client_stats()) ASSERTION(client_stat->nid_exp_ref_count == 0) failed:count 1 Jan 22 17:32:30 storage08 kernel: LustreError: 8667:0:(lprocfs_status.c:779:lprocfs_free_client_stats()) LBUG Jan 22 17:32:30 storage08 kernel: Lustre: 8667:0:(linux-debug.c:185:libcfs_debug_dumpstack()) showing stack for process 8667 Jan 22 17:32:30 storage08 kernel: Lustre: 8667:0:(linux-debug.c:185:libcfs_debug_dumpstack()) Skipped 14 previous similar messages Jan 22 17:32:30 storage08 kernel: umount R running task 0 8667 8581 (NOTLB) Jan 22 17:32:30 storage08 kernel: 0000000000000246 000001025f0598c0 0000000000000002 ffffffff80462cc0 Jan 22 17:32:30 storage08 kernel: ffffffff8045ecc0 ffffffff8013817a 0000003000000010 000001025f0599c8 Jan 22 17:32:30 storage08 kernel: 000001025f059908 000001025f0599d8 Jan 22 17:32:30 storage08 kernel: Call Trace:{free_block+285} {__kernel_text_address+26} Jan 22 17:32:30 storage08 kernel: {show_trace+375} {show_stack+241} Jan 22 17:32:30 storage08 kernel: {:libcfs:lbug_with_loc+115} {:obdclass:lprocfs_free_client_stats+228} Jan 22 17:32:30 storage08 kernel: {remove_proc_entry+443} {:obdclass:lprocfs_free_per_client_stats+145} Jan 22 17:32:30 storage08 kernel: {:obdfilter:filter_cleanup+539} {:obdclass:class_decref+1526} Jan 22 17:32:30 storage08 kernel: {__wake_up+54} {:obdclass:obd_zombie_impexp_cull+154} Jan 22 17:32:30 storage08 kernel: {:obdclass:class_detach+701} {:obdclass:class_process_config+6033} Jan 22 17:32:30 storage08 kernel: {:obdclass:class_manual_cleanup+2676} Jan 22 17:32:30 storage08 kernel: {:obdclass:lustre_end_log+1353} {:obdclass:class_name2dev+154} Jan 22 17:32:30 storage08 kernel: {:obdclass:server_put_super+1232} Jan 22 17:32:30 storage08 kernel: {invalidate_list+49} {dispose_list+192} Jan 22 17:32:30 storage08 kernel: {invalidate_inodes+185} {generic_shutdown_super+198} Jan 22 17:32:30 storage08 kernel: {kill_anon_super+9} {deactivate_super+95} Jan 22 17:32:31 storage08 kernel: {sys_umount+965} {sys_newstat+17} Jan 22 17:32:31 storage08 kernel: {error_exit+0} {system_call+126} Jan 22 17:32:31 storage08 kernel: Jan 22 17:32:31 storage08 kernel: LustreError: 137-5: UUID ''ddn_data-OST0004_UUID'' is not available for connect (stopping) Jan 22 17:32:31 storage08 kernel: LustreError: Skipped 35 previous similar messages Jan 22 17:32:31 storage08 kernel: LustreError: dumping log to /tmp/lustre-log.1232645550.8667 Jan 22 17:33:29 storage08 kernel: LustreError: 23499:0:(ldlm_lib.c:1619:target_send_reply_msg()) @@@ processing error (-19) req@00000103d2408c00 x1609/t0 o8->>@>:0/0 lens 240/0 e 0 to 0 dl 1232646009 ref 1 fl Interpret:/0/0 rc -19/0 Jan 22 17:33:29 storage08 kernel: LustreError: 23499:0:(ldlm_lib.c:1619:target_send_reply_msg()) Skipped 1278 previous similar messages Jan 22 17:33:48 storage08 kernel: LustreError: 137-5: UUID ''ddn_data-OST0004_UUID'' is not available for connect (stopping) Jan 22 17:33:48 storage08 kernel: LustreError: Skipped 578 previous similar messages Any clue if it is related to the read-only state of the OST0004? Regards, Wojciech Wojciech Turek wrote: Hi Brian, Brian J. Murrell wrote: On Thu, 2009-01-22 at 15:44 +0000, Wojciech Turek wrote: Hello, Hi, Lustre MDS report following error: Jan 22 15:20:40 mds01.beowulf.cluster kernel: LustreError: 24680:0:(lov_request.c:692:lov_update_create_set()) error creating fid 0xeb79c9d sub-object on OST idx 4/1: rc = -28 -28 is ENOSPC. Which I translate as that one of the OST (index 4/1) is full and has no space left on device. Yes. OSS seem to be consistent and says: Jan 22 15:21:15 storage08.beowulf.cluster kernel: LustreError: 23507:0:(filter_io_26.c:721:filter_commitrw_write()) error starting transaction: rc = -30 Hrm. I''m not sure a -30 (EROFS) would translate to a -28 to the MDS. I think it would also be a -30. So are you sure you are looking at correlating messages? The timestamps, if the two nodes are in sync also seem to indicate a lack of correlation with 35s of disparity. Perhaps there is an actual -28 in the OSS log prior to the -30 one? Yes you are right, there is plenty of messages with -30 in the logs and probably they are not related to -28. Which I translate as Client would like to write to an existing file but it can''t because file system is read only. Indeed. But why is it read-only? There should be an event in the OSS log saying that it was turning the filesystem read-only. The OST device is still mounted with rw option Yeah. That''s just the state at mount time. Lustre will set a device read-only in the case of filesystem errors, as one example. Now the main question is why Lustre thinks that OST(idx4) is full? No, I think the main question is why is it read-only. The full situation may have been transient where it filled up momentarily and then some objects were removed. In any case, this is a secondary issue and really only need be considered once the read-only situation is understood. Thank you for puting me in right track. I found these in the syslog Jan 22 10:18:40 storage08.beowulf.cluster kernel: LDISKFS-fs error (device dm-8): mb_free_blocks: double-free of inode 16203779''s block 688627718(bit 8198 in group 21015) Jan 22 10:18:40 storage08.beowulf.cluster kernel: Jan 22 10:18:40 storage08.beowulf.cluster kernel: Remounting filesystem read-only Is this means that the file system may be corrupted? I am going to run fsck -f on this device and try to mount it back, is that a right procedure? I did not find any errors on my S2A9500 storage, so I am not sure when this corruption could occur. Is it possible that this OST have meny orphaned objects which takes all the available space? That would be reflected in the df. If you suspect there may be orphan objects though, you could lfsck to verify and clean. Is there a way of reclaiming back this free space? If you mean orphaned OST objects, then lfsck. b. Cheers Wojciech _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Brian J. Murrell
2009-Jan-22 18:41 UTC
[Lustre-discuss] mds reports that OST is full (ENOSPC error -28 ) but df tells different
On Thu, 2009-01-22 at 18:19 +0000, Wojciech Turek wrote:> Hi Brian, > > I have tried to umount the OST(idx4) and the server LBUGed > > I attache LBUG below.This is an excellent example of how to use bugzilla to find existing bugs. This one looks like bug 16887 which is fixed in 1.6.7. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090122/125dd522/attachment.bin
Wojciech Turek
2009-Jan-22 18:45 UTC
Re: mds reports that OST is full (ENOSPC error -28 ) but df tells different
I am sorry, I should have looked in there before spamming here. Thank you Brian, Wojciech Brian J. Murrell wrote: On Thu, 2009-01-22 at 18:19 +0000, Wojciech Turek wrote: Hi Brian, I have tried to umount the OST(idx4) and the server LBUGed I attache LBUG below. This is an excellent example of how to use bugzilla to find existing bugs. This one looks like bug 16887 which is fixed in 1.6.7. b. _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Brian J. Murrell
2009-Jan-22 19:20 UTC
[Lustre-discuss] mds reports that OST is full (ENOSPC error -28 ) but df tells different
On Thu, 2009-01-22 at 18:45 +0000, Wojciech Turek wrote:> I am sorry, I should have looked in there before spamming here.No worries. It was just suggestion to all that bugzilla will frequently get one answers quicker than waiting for a response here. Not that you shouldn''t feel free to post here if you wish. Realize of course that it will just take longer to get an answer. Just composing an e-mail can take longer than searching bugzilla in most cases. :-) Cheers, b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090122/8d466147/attachment.bin
Andreas Dilger
2009-Jan-23 19:43 UTC
[Lustre-discuss] mds reports that OST is full (ENOSPC error -28 ) but df tells different
On Jan 22, 2009 17:27 +0000, Wojciech Turek wrote:> Thank you for puting me in right track. I found these in the syslog > Jan 22 10:18:40 storage08.beowulf.cluster kernel: LDISKFS-fs error (device dm-8): mb_free_blocks: double-free of inode 16203779''s block 688627718(bit 8198 in group 21015) > Jan 22 10:18:40 storage08.beowulf.cluster kernel: Remounting filesystem read-only> Is this means that the file system may be corrupted? I am going to run > fsck -f on this device and try to mount it back, is that a right > procedure?It might only be an error in memory and not on disk, based on previous cases of this message. It might relate to the ENOSPC problem causing some obscure error path to try and clean up after failing to allocate data. There were previous reports of this message, so this is another thing to check in Bugzilla if the version of Lustre you are running has the fixes for this problem or not. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.