Katya Tutlyaeva
2010-Jul-08 12:47 UTC
[Lustre-discuss] No space left on device on not full filesystem
Hello,
We have running lustre 1.8.1 and have met "No space lest on
device"
error when uploading 500 Gb small files (less then 100 Kb each).
The problem seems to depends on the number of files. If we remove one
file, we can create one new file, even with Gb size; but if we haven''t
remove something we can''t create even very little file, as an example
using touch command:
[client]$ touch newfile
touch: cannot touch `newfile'': No space left on device
[client]$ rm oldfile
rm: remove regular file `qqq''? y
[client]$ dd bs=100M count=10 < /dev/zero > qqq
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 15.9882 s, 65.6 MB/s
beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq.tmp
qqq.tmp: No space left on device.
beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq
^C5+0 records in
5+0 records out
524288000 bytes (524 MB) copied, 7.66803 s, 68.4 MB/s
But there are a lot of free space both on OSTs and MDS:
[client]$ lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 43.8G 813.4M 40.5G 1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 867.7G 45.8G 777.8G 5% /mnt/lustre[OST:0]
lustre-OST0001_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:1]
lustre-OST0002_UUID 916.9G 44.9G 825.5G 4% /mnt/lustre[OST:2]
lustre-OST0003_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:3]
lustre-OST0004_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:4]
lustre-OST0005_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:5]
lustre-OST0006_UUID 916.9G 44.5G 825.9G 4% /mnt/lustre[OST:6]
lustre-OST0007_UUID 916.9G 44.6G 825.8G 4% /mnt/lustre[OST:7]
lustre-OST0008_UUID 916.9G 44.5G 825.8G 4% /mnt/lustre[OST:8]
filesystem summary: 8.0T 403.8G 7.2T 4% /mnt/lustre
There are not additional information in dmesg, no new messeges in
/tmp/lustre-log also.
Would you be kind enough to help me, to give some advice?
Thank you!
Best regards,
Katya
Wojciech Turek
2010-Jul-08 16:14 UTC
[Lustre-discuss] No space left on device on not full filesystem
Hi, Looks like your file system is running out of inodes. Please do lfs df -i on client and copy the output here. best regards, Wojciech On 8 July 2010 13:47, Katya Tutlyaeva <ordi at xgl.pereslavl.ru> wrote:> Hello, > > We have running lustre 1.8.1 and have met "No space lest on device" > error when uploading 500 Gb small files (less then 100 Kb each). > The problem seems to depends on the number of files. If we remove one > file, we can create one new file, even with Gb size; but if we haven''t > remove something we can''t create even very little file, as an example > using touch command: > [client]$ touch newfile > touch: cannot touch `newfile'': No space left on device > [client]$ rm oldfile > rm: remove regular file `qqq''? y > [client]$ dd bs=100M count=10 < /dev/zero > qqq > 10+0 records in > 10+0 records out > 1048576000 bytes (1.0 GB) copied, 15.9882 s, 65.6 MB/s > beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq.tmp > qqq.tmp: No space left on device. > beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq > ^C5+0 records in > 5+0 records out > 524288000 bytes (524 MB) copied, 7.66803 s, 68.4 MB/s > > But there are a lot of free space both on OSTs and MDS: > [client]$ lfs df -h > UUID bytes Used Available Use% Mounted on > lustre-MDT0000_UUID 43.8G 813.4M 40.5G 1% /mnt/lustre[MDT:0] > lustre-OST0000_UUID 867.7G 45.8G 777.8G 5% /mnt/lustre[OST:0] > lustre-OST0001_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:1] > lustre-OST0002_UUID 916.9G 44.9G 825.5G 4% /mnt/lustre[OST:2] > lustre-OST0003_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:3] > lustre-OST0004_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:4] > lustre-OST0005_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:5] > lustre-OST0006_UUID 916.9G 44.5G 825.9G 4% /mnt/lustre[OST:6] > lustre-OST0007_UUID 916.9G 44.6G 825.8G 4% /mnt/lustre[OST:7] > lustre-OST0008_UUID 916.9G 44.5G 825.8G 4% /mnt/lustre[OST:8] > > filesystem summary: 8.0T 403.8G 7.2T 4% /mnt/lustre > > There are not additional information in dmesg, no new messeges in > /tmp/lustre-log also. > > Would you be kind enough to help me, to give some advice? > Thank you! > > Best regards, > Katya > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- -- Wojciech Turek Assistant System Manager High Performance Computing Service University of Cambridge -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100708/305e0d2f/attachment.html
Kevin Van Maren
2010-Jul-08 16:15 UTC
[Lustre-discuss] No space left on device on not full filesystem
Two possibilities seem likely: 1) You have run out of inodes. Do "df -i", or "lfs df -i" 2) It could be a grant issue. With 700GB free on each OST, unless you have a lot of clients, it looks like a grant leak issue. See: https://bugzilla.lustre.org/show_bug.cgi?id=22755 Kevin Katya Tutlyaeva wrote:> Hello, > > We have running lustre 1.8.1 and have met "No space lest on device" > error when uploading 500 Gb small files (less then 100 Kb each). > The problem seems to depends on the number of files. If we remove one > file, we can create one new file, even with Gb size; but if we haven''t > remove something we can''t create even very little file, as an example > using touch command: > [client]$ touch newfile > touch: cannot touch `newfile'': No space left on device > [client]$ rm oldfile > rm: remove regular file `qqq''? y > [client]$ dd bs=100M count=10 < /dev/zero > qqq > 10+0 records in > 10+0 records out > 1048576000 bytes (1.0 GB) copied, 15.9882 s, 65.6 MB/s > beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq.tmp > qqq.tmp: No space left on device. > beda:/mnt/lustre# dd bs=100M count=10 < /dev/zero > qqq > ^C5+0 records in > 5+0 records out > 524288000 bytes (524 MB) copied, 7.66803 s, 68.4 MB/s > > But there are a lot of free space both on OSTs and MDS: > [client]$ lfs df -h > UUID bytes Used Available Use% Mounted on > lustre-MDT0000_UUID 43.8G 813.4M 40.5G 1% /mnt/lustre[MDT:0] > lustre-OST0000_UUID 867.7G 45.8G 777.8G 5% /mnt/lustre[OST:0] > lustre-OST0001_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:1] > lustre-OST0002_UUID 916.9G 44.9G 825.5G 4% /mnt/lustre[OST:2] > lustre-OST0003_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:3] > lustre-OST0004_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:4] > lustre-OST0005_UUID 916.9G 44.9G 825.4G 4% /mnt/lustre[OST:5] > lustre-OST0006_UUID 916.9G 44.5G 825.9G 4% /mnt/lustre[OST:6] > lustre-OST0007_UUID 916.9G 44.6G 825.8G 4% /mnt/lustre[OST:7] > lustre-OST0008_UUID 916.9G 44.5G 825.8G 4% /mnt/lustre[OST:8] > > filesystem summary: 8.0T 403.8G 7.2T 4% /mnt/lustre > > There are not additional information in dmesg, no new messeges in > /tmp/lustre-log also. > > Would you be kind enough to help me, to give some advice? > Thank you! > > Best regards, > Katya > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Brian J. Murrell
2010-Jul-08 16:22 UTC
[Lustre-discuss] No space left on device on not full filesystem
On Thu, 2010-07-08 at 16:47 +0400, Katya Tutlyaeva wrote:> Hello, > > We have running lustre 1.8.1 and have met "No space lest on device" > error when uploading 500 Gb small files (less then 100 Kb each).What does "lfs df" and "lfs df -i" on a client report? b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100708/3b794905/attachment.bin
Katya Tutlyaeva
2010-Jul-09 08:51 UTC
[Lustre-discuss] No space left on device on not full filesystem
Thank you all for answers! Really, I have running out of inodes, [client]$ lfs df -i UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 13111096 13111096 0 100% /mnt/lustre[MDT:0] lustre-OST0000_UUID 57778176 1457730 56320446 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 61054976 1457660 59597316 2% /mnt/lustre[OST:1] lustre-OST0002_UUID 61054976 1457725 59597251 2% /mnt/lustre[OST:2] lustre-OST0003_UUID 61054976 1457397 59597579 2% /mnt/lustre[OST:3] lustre-OST0004_UUID 61054976 1457619 59597357 2% /mnt/lustre[OST:4] lustre-OST0005_UUID 61054976 1457623 59597353 2% /mnt/lustre[OST:5] lustre-OST0006_UUID 61054976 1455512 59599464 2% /mnt/lustre[OST:6] lustre-OST0007_UUID 61054976 1455511 59599465 2% /mnt/lustre[OST:7] lustre-OST0008_UUID 61054976 1455479 59599497 2% /mnt/lustre[OST:8] filesystem summary: 13111096 13111096 0 100% /mnt/lustre Would you be so kind to give me the advice How can I tune number of inodes without loss of data? And which number of inodes I need? I?m newbie in Lustre.. If I''ll set -N 20000000 to create 2 millions of inodes it will take 2000000*512/(1024*1024) = 9765.625 MB of MDS capacity? (Inode size is near 512 bytes, isn''t it?) Where can I look the inode size? Thank again! Regards, Katya *From:* turek.wojciech at gmail.com [mailto:turek.wojciech at gmail.com] *On Behalf Of *Wojciech Turek *Sent:* Thursday, July 08, 2010 8:14 PM *To:* Katya Tutlyaeva *Cc:* lustre-discuss at lists.lustre.org *Subject:* Re: [Lustre-discuss] No space left on device on not full filesystem Hi, Looks like your file system is running out of inodes. Please do lfs df -i on client and copy the output here. best regards, _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss -- -- Wojciech Turek Assistant System Manager High Performance Computing Service University of Cambridge
Brian J. Murrell
2010-Jul-09 15:45 UTC
[Lustre-discuss] No space left on device on not full filesystem
On Fri, 2010-07-09 at 12:51 +0400, Katya Tutlyaeva wrote:> > Really, I have running out of inodes,Indeed, as many of us had suspected.> [client]$ lfs df -i > > UUID Inodes IUsed IFree IUse% Mounted on > > lustre-MDT0000_UUID 13111096 13111096 0 100% /mnt/lustre[MDT:0]> Would you be so kind to give me the advice How can I tune number of > inodes without loss of data?The safest method is to create a new, larger MDT and shut down the filesystem and rsync the existing MDT to the new MDT, paying attention to the process in the manual for doing this, in particular, preserving extended attributes.> And which number of inodes I need?That depends on how many files you expect your Lustre system to contain.> I?m > newbie in Lustre.. If I''ll set -N 20000000 to create 2 millions of > inodes it will take 2000000*512/(1024*1024) = 9765.625 MB of MDS > capacity? (Inode size is near 512 bytes, isn''t it?) Where can I look the > inode size?I believe there are sizing guidelines in the manual as well. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100709/09b722e3/attachment.bin