Sebastian Reitenbach
2010-Jan-29 07:34 UTC
[Lustre-discuss] striping files to different OSTs
Hi, I am new to lustre, just did a test setup with a MGS/MGS and two OST servers. I run lustre 1.8.1 on SLES 11 x86_64. reading chapter 1.4.1 of the Lustre manual, I see that files should get striped to different OSTs, and that files can become larger than the size of one OST. However, when I use dd if=/dev/zero of=outfile.img then I get an error dd: writing `outfile.img'': No space left on device after the first OST is full, the second one is still empty, see below: srv3:/lustre/foo # lfs df -h UUID bytes Used Available Use% Mounted on foo-MDT0000_UUID 446.1M 16.6M 404.0M 3% /lustre/foo[MDT:0] foo-OST0000_UUID 9.8G 22.7M 9.3G 0% /lustre/foo[OST:0] foo-OST0001_UUID 9.8G 22.7M 9.3G 0% /lustre/foo[OST:1] filesystem summary: 19.7G 45.4M 18.6G 0% /lustre/foo UUID bytes Used Available Use% Mounted on bar-MDT0000_UUID 425.5M 16.4M 384.7M 3% /lustre/bar[MDT:0] bar-OST0000_UUID 9.8G 22.7M 9.3G 0% /lustre/bar[OST:0] bar-OST0001_UUID 9.8G 22.7M 9.3G 0% /lustre/bar[OST:1] filesystem summary: 19.7G 45.4M 18.6G 0% /lustre/bar srv3:/lustre/foo # dd if=/dev/zero of=outfile.img bs=4M dd: writing `outfile.img'': No space left on device 2388+0 records in 2387+0 records out 10012561408 bytes (10 GB) copied, 221.014 s, 45.3 MB/s srv3:/lustre/foo # lfs df -h UUID bytes Used Available Use% Mounted on foo-MDT0000_UUID 446.1M 16.6M 404.0M 3% /lustre/foo[MDT:0] foo-OST0000_UUID 9.8G 9.3G 4.0K 94% /lustre/foo[OST:0] foo-OST0001_UUID 9.8G 22.7M 9.3G 0% /lustre/foo[OST:1] filesystem summary: 19.7G 9.4G 9.3G 47% /lustre/foo UUID bytes Used Available Use% Mounted on bar-MDT0000_UUID 425.5M 16.4M 384.7M 3% /lustre/bar[MDT:0] bar-OST0000_UUID 9.8G 22.7M 9.3G 0% /lustre/bar[OST:0] bar-OST0001_UUID 9.8G 22.7M 9.3G 0% /lustre/bar[OST:1] filesystem summary: 19.7G 45.4M 18.6G 0% /lustre/bar srv3:/lustre/foo # dd if=/dev/zero of=outfile2.img bs=4M dd: writing `outfile2.img'': No space left on device 2388+0 records in 2387+0 records out 10012565504 bytes (10 GB) copied, 331.793 s, 30.2 MB/s srv3:/lustre/foo # lfs df -h UUID bytes Used Available Use% Mounted on foo-MDT0000_UUID 446.1M 16.6M 404.0M 3% /lustre/foo[MDT:0] foo-OST0000_UUID 9.8G 9.3G 4.0K 94% /lustre/foo[OST:0] foo-OST0001_UUID 9.8G 9.3G 4.0K 94% /lustre/foo[OST:1] filesystem summary: 19.7G 18.7G 8.0K 94% /lustre/foo UUID bytes Used Available Use% Mounted on bar-MDT0000_UUID 425.5M 16.4M 384.7M 3% /lustre/bar[MDT:0] bar-OST0000_UUID 9.8G 22.7M 9.3G 0% /lustre/bar[OST:0] bar-OST0001_UUID 9.8G 22.7M 9.3G 0% /lustre/bar[OST:1] filesystem summary: 19.7G 45.4M 18.6G 0% /lustre/bar The filesystems were created like this on the MGS/MDS: mkfs.lustre --mgs --mdt --fsname=foo --reformat /dev/xvdb1 mkfs.lustre --mdt --fsname=bar --mgsnode=10.0.0.81 at tcp --reformat /dev/xvdb2 mount -t lustre /dev/xvdb1 /lustre/foo-mgs-mdt mount -t lustre /dev/xvdb2 /lustre/bar-mdt on OST1: mke2fs -O journal_dev -b 4096 /dev/xvdd mke2fs -O journal_dev -b 4096 /dev/xvde mkfs.lustre --fsname=foo --param="failover.node=10.0.0.86 at tcp" -- mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdd - E stride=32" /dev/xvdb1 mkfs.lustre --fsname=bar --param="failover.node=10.0.0.86 at tcp" -- mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvde - E stride=32" /dev/xvdb2 on OST2: mke2fs -O journal_dev -b 4096 /dev/xvdf mke2fs -O journal_dev -b 4096 /dev/xvdg mkfs.lustre --fsname=foo --param="failover.node=10.0.0.85 at tcp" -- mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdf - E stride=32" /dev/xvdc1 mkfs.lustre --fsname=bar --param="failover.node=10.0.0.85 at tcp" -- mgsnode=10.0.0.81 at tcp --ost --reformat --mkfsoptions "-j -J device=/dev/xvdg - E stride=32" /dev/xvdc2 On the OST hosts I use pacemaker to mout the filesystems. On the clients I mounted the filesystems: mount -t lustre 10.0.0.81 at tcp:/foo /lustre/foo mount -t lustre 10.0.0.81 at tcp:/bar /lustre/bar Anything I did wrong or misunderstood? Maybe I need to tune a parameter of the file system to activate that striping? regards, Sebastian
Sebastian Reitenbach
2010-Jan-29 09:23 UTC
[Lustre-discuss] striping files to different OSTs
Hi, On Friday 29 January 2010 08:34:50 am Sebastian Reitenbach wrote:> Hi, > > I am new to lustre, just did a test setup with a MGS/MGS and two OST > servers. I run lustre 1.8.1 on SLES 11 x86_64. > reading chapter 1.4.1 of the Lustre manual, I see that files should get > striped to different OSTs, and that files can become larger than the size > of one OST. > > However, when I use dd if=/dev/zero of=outfile.img then I get an error > dd: writing `outfile.img'': No space left on device > after the first OST is full, the second one is still empty, see below: >Answering myself here: After being pointed to the right direction, I found reading chapter 24 regarding file striping, very informative. cheers Sebastian