Alex
2008-Oct-29 09:49 UTC
[Lustre-discuss] max file size on lustre - No space left on device after 37GB!
Hello, Reading Lustre Manual (section 33. System Limits) i could not find an answer to my problem. I am tring to create a 50GB testfile on my lustre file system using dd. After 37GB (which is the max size of one of my OST), dd is stopping complaining about FREE SPACE!!! [root at rs1 ~]# uname -rm 2.6.18-53.1.14.el5_lustre.1.6.5.1smp i686 [root at rs1 ~]# Before dd usage: [root at rs1 ~]# lfs df -h UUID bytes Used Available Use% Mounted on testfs-MDT0000_UUID 130.4G 460.0M 122.5G 0% /mnt/lustre[MDT:0] testfs-OST0000_UUID 18.3G 1.4G 16.0G 7% /mnt/lustre[OST:0] testfs-OST0001_UUID 18.3G 444.5M 17.0G 2% /mnt/lustre[OST:1] testfs-OST0002_UUID 36.7G 448.5M 34.4G 1% /mnt/lustre[OST:2] testfs-OST0003_UUID 36.7G 448.5M 34.4G 1% /mnt/lustre[OST:3] filesystem summary: 110.0G 2.7G 101.7G 2% /mnt/lustre [root at rs1 ~]# So, i have 101GB free -> start dd and create a 50GB file on /mnt/lustre: [root at rs1 ~]# dd if=/dev/zero of=/mnt/lustre/test2.img bs=100M count=500 dd: writing `/mnt/lustre/test2.img'': No space left on device 352+0 records in 351+0 records out 36909240320 bytes (37 GB) copied, 3496.8 seconds, 10.6 MB/s [root at rs1 ~]# list again free space usage: [root at rs1 ~]# lfs df -h UUID bytes Used Available Use% Mounted on testfs-MDT0000_UUID 130.4G 460.0M 122.5G 0% /mnt/lustre[MDT:0] testfs-OST0000_UUID 18.3G 1.4G 16.0G 7% /mnt/lustre[OST:0] testfs-OST0001_UUID 18.3G 444.5M 17.0G 2% /mnt/lustre[OST:1] testfs-OST0002_UUID 36.7G 34.8G 4.0M 94% /mnt/lustre[OST:2] testfs-OST0003_UUID 36.7G 448.5M 34.4G 1% /mnt/lustre[OST:3] filesystem summary: 110.0G 37.1G 67.4G 33% /mnt/lustre [root at rs1 ~]# As you see, testfs-OST0002_UUID is full, but on /mnt/lustre I have enough free space - 67GB (on other OSTs) in order to allow dd to continue and create my file. How can i avoid dd to stop and allow it to continue to complete on other OSTs? Regards, Alx
Brian J. Murrell
2008-Oct-29 10:46 UTC
[Lustre-discuss] max file size on lustre - No space left on device after 37GB!
On Wed, 2008-10-29 at 11:49 +0200, Alex wrote:> Hello,Hi,> Reading Lustre Manual (section 33. System Limits) i could not find an answer > to my problem. > > I am tring to create a 50GB testfile on my lustre file system using dd. After > 37GB (which is the max size of one of my OST), dd is stopping complaining > about FREE SPACE!!!> list again free space usage: > > [root at rs1 ~]# lfs df -h > UUID bytes Used Available Use% Mounted on > testfs-MDT0000_UUID 130.4G 460.0M 122.5G 0% /mnt/lustre[MDT:0] > testfs-OST0000_UUID 18.3G 1.4G 16.0G 7% /mnt/lustre[OST:0] > testfs-OST0001_UUID 18.3G 444.5M 17.0G 2% /mnt/lustre[OST:1] > testfs-OST0002_UUID 36.7G 34.8G 4.0M 94% /mnt/lustre[OST:2] > testfs-OST0003_UUID 36.7G 448.5M 34.4G 1% /mnt/lustre[OST:3] > filesystem summary: 110.0G 37.1G 67.4G 33% /mnt/lustre > [root at rs1 ~]# > > As you see, testfs-OST0002_UUID is full,Yes. That''s why you got the -ENOSPC> but on /mnt/lustre I have enough free > space - 67GB (on other OSTs) in order to allow dd to continue and create my > file. How can i avoid dd to stop and allow it to continue to complete on > other OSTs?You need to investigate striping. Lustre''s default striping policy is 1, which is no striping. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20081029/f9830acf/attachment.bin
Alex
2008-Oct-29 11:07 UTC
[Lustre-discuss] max file size on lustre - No space left on device after 37GB!
> > As you see, testfs-OST0002_UUID is full, > > Yes. That''s why you got the -ENOSPC > > > but on /mnt/lustre I have enough free > > space - 67GB (on other OSTs) in order to allow dd to continue and create > > my file. How can i avoid dd to stop and allow it to continue to complete > > on other OSTs? > > You need to investigate striping. Lustre''s default striping policy is > 1, which is no striping.Hi Brian, Thanks for your reply. How can i change "Lustre default striping policy"? On http://manual.lustre.org/manual/LustreManual16_HTML/index.html i cannot find the answer. Regards, Alx
Alex
2008-Oct-29 15:46 UTC
[Lustre-discuss] max file size on lustre - No space left on device after 37GB!
On Wednesday 29 October 2008 13:07, Alex wrote:> > > As you see, testfs-OST0002_UUID is full, > > > > Yes. That''s why you got the -ENOSPC > > > > > but on /mnt/lustre I have enough free > > > space - 67GB (on other OSTs) in order to allow dd to continue and > > > create my file. How can i avoid dd to stop and allow it to continue to > > > complete on other OSTs? > > > > You need to investigate striping. Lustre''s default striping policy is > > 1, which is no striping. > > Hi Brian, > > Thanks for your reply. How can i change "Lustre default striping policy"? > On http://manual.lustre.org/manual/LustreManual16_HTML/index.html i cannot > find the answer. >I found the answer ... googling. http://manual.lustre.org is a crap! [root at rs1 ~]# lfs setstripe -c -1 /mnt/lustre/ (spread across all available OST''s) Regards, Alx
Michael MacDonald
2008-Oct-29 16:54 UTC
[Lustre-discuss] max file size on lustre - No space left on device after 37GB!
On Wed, 2008-10-29 at 17:46 +0200, Alex wrote:> > Thanks for your reply. How can i change "Lustre default striping policy"? > > On http://manual.lustre.org/manual/LustreManual16_HTML/index.html i cannot > > find the answer. > > > > I found the answer ... googling. http://manual.lustre.org is a crap! > [root at rs1 ~]# lfs setstripe -c -1 /mnt/lustre/ (spread across all available > OST''s) >Is "Striping and I/O Options" too obscure a name? http://manual.lustre.org/manual/LustreManual16_HTML/StripingAndIOOptions.html There''s a subsection named "lfs setstripe - Setting File Layouts" ... http://manual.lustre.org/manual/LustreManual16_HTML/StripingAndIOOptions.html#50643011_78664 Mike