Thomas Roth
2008-Jul-02 13:14 UTC
[Lustre-discuss] no file creation on not yet full Lustre
Hi all, on a Lustre FS v1.6.5 with Debian Etch kernel 2.6.22 that is 94% full, I can''t create any more files. Each OST has still room for ~ 2GB: #> lfs df UUID 1K-blocks Used Available Use% Mounted on MDT0000_UUID 495497804 669376 466511924 0% /lustre[MDT:0] OST0000_UUID 2440596776 2316586108 1942896 94% /lustre[OST:0] OST0001_UUID 2928715484 2779554112 2680056 94% /lustre[OST:1] OST0002_UUID 2440596776 2316342644 2186360 94% /lustre[OST:2] OST0003_UUID 2928715484 2779758712 2475456 94% /lustre[OST:3] OST0004_UUID: inactive device OST0005_UUID: inactive device OST0006_UUID 2440596776 2316400736 2128268 94% /lustre[OST:6] OST0007_UUID 2928715484 2779972968 2261196 94% /lustre[OST:7] OST0008_UUID 2440596776 2316306052 2214696 94% /lustre[OST:8] OST0009_UUID 2928715484 2779558976 2666004 94% /lustre[OST:9] filesystem summary: 21477249040 20384480308 18554932 94% /lustre And it''s not the amount of inodes, either: #> lfs df -i UUID Inodes IUsed IFree IUse% Mounted on MDT0000_UUID 128657164 4950058 123707106 3% /lustre[MDT:0] OST0000_UUID 2384256 452839 1931417 18% /lustre[OST:0] OST0001_UUID 2861056 543317 2317739 18% /lustre[OST:1] OST0002_UUID 2384256 452336 1931920 18% /lustre[OST:2] OST0003_UUID 2861056 542826 2318230 18% /lustre[OST:3] OST0004_UUID: inactive device OST0005_UUID: inactive device OST0006_UUID 2384256 452735 1931521 18% /lustre[OST:6] OST0007_UUID 2861056 543414 2317642 18% /lustre[OST:7] OST0008_UUID 2384256 452335 1931921 18% /lustre[OST:8] OST0009_UUID 2861056 542851 2318205 18% /lustre[OST:9] filesystem summary: 128657164 4950058 123707106 3% /lustre The MDT has a log message (possibly) on this: LustreError: 4586:0:(mds_open.c:416:mds_create_objects()) error creating objects for inode 110854263: rc = -5 Later on, there are more Errors like LustreError: 4581:0:(mds_open.c:710:mds_finish_open()) mds_create_objects: rc = -28 There is also rc = -22. By the way, does anybody know what these numbers mean/where to look them up? In my search through the net I once again stumbled over ras.lustre.org and error.lustre.org but without success. It''s a pity, I must by now have Gigabytes of LustreError messages but I have never found what part to paste there... The error while trying to write was: ~# dd if=/dev/zero of=/lustre/Null dd: opening `/lustre/Null'': Input/output error I tried to make a new file go to a specific OST: ~# lfs setstripe /lustre/a -i 3 error on ioctl 0x4008669a for ''/lustre/a'' (3): No space left on device error: setstripe: create stripe file failed On the OSTs I don''t see any corresponding entry in the logs. There is an old bug that might be related: bugzilla.lustre.org/show_bug.cgi?id=12768 . At least it left similar traces in the log. But it has been resolved a year ago. So, anybody has got an idea what goes wrong here? Many thanks, Thomas
Thomas Roth
2008-Jul-02 13:32 UTC
[Lustre-discuss] no file creation on not yet full Lustre
Oh, that was one of the (many) things I skipped when reading the manual. Seems to have done the trick, thanks a lot. Thomas Guy Coates wrote:> Thomas Roth wrote: >> Hi all, >> >> on a Lustre FS v1.6.5 with Debian Etch kernel 2.6.22 that is 94% full, I >> can''t create any more files. >> Each OST has still room for ~ 2GB: >> > > Hi Thomas, > > Are you running into the 5% of disk space ext3 reserves for root? > > I think the manual says you can do: > > tune2fs -m 0 /dev/XXX on your OSTs and MDSs to recover the space. > > > Guy >-- -------------------------------------------------------------------- Thomas Roth Department: Informationstechnologie Location: SB3 1.262 Phone: +49-6159-71 1453 Fax: +49-6159-71 2986 Gesellschaft f?r Schwerionenforschung mbH Planckstra?e 1 D-64291 Darmstadt www.gsi.de Gesellschaft mit beschr?nkter Haftung Sitz der Gesellschaft: Darmstadt Handelsregister: Amtsgericht Darmstadt, HRB 1528 Gesch?ftsf?hrer: Professor Dr. Horst St?cker Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph, Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
D. Marc Stearman
2008-Jul-02 15:27 UTC
[Lustre-discuss] no file creation on not yet full Lustre
On Jul 2, 2008, at 6:14 AM, Thomas Roth wrote:> The MDT has a log message (possibly) on this: > > LustreError: 4586:0:(mds_open.c:416:mds_create_objects()) error > creating > objects for inode 110854263: rc = -5 > > Later on, there are more Errors like > LustreError: 4581:0:(mds_open.c:710:mds_finish_open()) > mds_create_objects: rc = -28 > > There is also rc = -22. By the way, does anybody know what these > numbers > mean/where to look them up? In my search through the net I once again > stumbled over ras.lustre.org and error.lustre.org but without > success. > It''s a pity, I must by now have Gigabytes of LustreError messages > but I > have never found what part to paste there...The file that lists the error messages is errno.h Under RHEL5 there are many, but the main ones are these: /usr/include/asm-generic/errno-base.h /usr/include/asm-generic/errno.h You can do a "find /usr/include | grep errno" to find all of them. -Marc ---- D. Marc Stearman LC Lustre Administration Lead marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641