Andreas Dilger
2006-May-19 07:36 UTC
[Lustre-discuss] Handbook && "No space left on device"
On Mar 22, 2005 17:16 +0100, Alessandro wrote:> Do you know which is the ratio between MDS storage space and OST storage > space? > With some tests, it seems to be 1:20 (I need 1GB for the size of the MDS > device to storage 20GB of file data in OST device) > Perhaps it depends by dir structures? Or not? > Where can I find this info? (I think there are not many ''technical-low > level document'' about Lustre)There is the Lustre FAQ at http://www.clusterfs.com/faq.html. This particular question isn''t answered there though. The answer is that by default (in the current version, fixed in the next release) the MDS allocates an inode per 8192 bytes of the device it is created in, regardless of the file size. So the number of inodes you get depends on the MDS device size. The number of MDS inodes you need depends entirely upon how large your average file size is. Most Lustre installations have quite large files (in the multi-MB range). In order to increase the number of inodes for a given MDS size, you can specify "--mkfsoptions=''-i 4096''" on the --add mds line in the LMC script. The size of the MDS would be approximately (1.5 is safety margin): 1.5 * sum(OST size) / average_file_size Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Mc Carthy, Fergal
2006-May-19 07:36 UTC
[Lustre-discuss] Handbook && "No space left on device"
Alessandro, If you check the recent archives for this mailing list you should find a mail that asks similar questions which was responded to with useful info. That would be a good place to start for looking into the ratios. One thing to consider is adding the "-T news" (or "-i 4096") string as an mkfsoptions setting to the MDS device. This will cause the device formatting to allocate many more inodes (1 for every 4k of disk space) in the MDS device. Fergal. -- Fergal.McCarthy@HP.com (The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as "HP CONFIDENTIAL".) -----Original Message----- From: lustre-discuss-admin@lists.clusterfs.com [mailto:lustre-discuss-admin@lists.clusterfs.com] On Behalf Of Alessandro Sent: 22 March 2005 16:17 To: lustre-discuss@lists.clusterfs.com Subject: Re: [Lustre-discuss] Handbook && "No space left on device" Thanks for your attention. Now it seems it''s all OK (...) Do you know which is the ratio between MDS storage space and OST storage space? With some tests, it seems to be 1:20 (I need 1GB for the size of the MDS device to storage 20GB of file data in OST device) Perhaps it depends by dir structures? Or not? Where can I find this info? (I think there are not many ''technical-low level=20 document'' about Lustre) ----- Original Message -----=20 Subject: RE: [Lustre-discuss] Handbook && "No space left on device" Alessandro, I suspect the problem may be the declaration of your MDS device. You have only allocated 50000k for the size of the MDS loopback device. As a quick way to verify this, what does df -i say for the Lustre file system? If it says that there are no free inodes, then that is the problem. I would recommend using a larger MDS loopback device. As for other information sources, I would recommend checking out the Lustre Wiki (https://wiki.clusterfs.com/lustre) for more info that may be of help. Fergal. -- Fergal.McCarthy@HP.com (The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as "HP CONFIDENTIAL".) -----Original Message----- From: lustre-discuss-admin@lists.clusterfs.com [mailto:lustre-discuss-admin@lists.clusterfs.com] On Behalf Of Alessandro Sent: 22 March 2005 13:14 To: lustre-discuss@lists.clusterfs.com Subject: [Lustre-discuss] Handbook && "No space left on device" Hi to all, I use a real disk (a single partition as /dev/sdc1) as an EXT3 storage filesystem (about 200GByte). When I mount to /mnt/lustre, I can copy only a part of what I want, then I have a lot of messages: (example: ''cp -rfv /usr /mnt/lustre'') cp: cannot create directory `/mnt/lustre/usr/libexec'': No space left on device cp: cannot create directory `/mnt/lustre/usr/sbin'': No space left on device cp: cannot create directory `/mnt/lustre/usr/share'': No space left on device cp: cannot create directory `/mnt/lustre/usr/X11R6'': No space left on device cp: cannot create directory `/mnt/lustre/usr/dict'': No space left on device cp: cannot create directory `/mnt/lustre/usr/etc'': No space left on device cp: cannot create directory `/mnt/lustre/usr/games'': No space left on device . but df says me I wrote only 1% of /mnt/lustre Why I cannot write to 50%(for example) of /mnt/lustre? Which parameters must I configure? Where can I find an handbook useful in providing an in-depth understanding of Lustre? http://www.lustre.org/docs/lustre.pdf doesn''t work!!! Useful info: Lustre 1.2.4 Red Hat 9 (kernel Vanilla 2.4.24) My configuration script: lmc -o ale.xml --add node --node server_cvs lmc -m ale.xml --add net --node server_cvs --nid server_cvs --nettype tcp # Configure MDS lmc -m ale.xml --add mds --node server_cvs --mds mds-test --fstype ext3 --dev /tmp/mds-test --size 50000 # Configure OST lmc -m ale.xml --add ost --node server_cvs --ost ost1-test --fstype ext3 --dev /dev/sdc1 # Configure client lmc -m ale.xml --add mtpt --node server_cvs --path /mnt/lustre --mds mds-test --ost ost1-test Ale _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.clusterfs.com https://lists.clusterfs.com/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.clusterfs.com https://lists.clusterfs.com/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.clusterfs.com https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
Thanks for your attention. Now it seems it''s all OK (...) Do you know which is the ratio between MDS storage space and OST storage space? With some tests, it seems to be 1:20 (I need 1GB for the size of the MDS device to storage 20GB of file data in OST device) Perhaps it depends by dir structures? Or not? Where can I find this info? (I think there are not many ''technical-low level document'' about Lustre) ----- Original Message ----- Subject: RE: [Lustre-discuss] Handbook && "No space left on device" Alessandro, I suspect the problem may be the declaration of your MDS device. You have only allocated 50000k for the size of the MDS loopback device. As a quick way to verify this, what does df -i say for the Lustre file system? If it says that there are no free inodes, then that is the problem. I would recommend using a larger MDS loopback device. As for other information sources, I would recommend checking out the Lustre Wiki (https://wiki.clusterfs.com/lustre) for more info that may be of help. Fergal. -- Fergal.McCarthy@HP.com (The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as "HP CONFIDENTIAL".) -----Original Message----- From: lustre-discuss-admin@lists.clusterfs.com [mailto:lustre-discuss-admin@lists.clusterfs.com] On Behalf Of Alessandro Sent: 22 March 2005 13:14 To: lustre-discuss@lists.clusterfs.com Subject: [Lustre-discuss] Handbook && "No space left on device" Hi to all, I use a real disk (a single partition as /dev/sdc1) as an EXT3 storage filesystem (about 200GByte). When I mount to /mnt/lustre, I can copy only a part of what I want, then I have a lot of messages: (example: ''cp -rfv /usr /mnt/lustre'') cp: cannot create directory `/mnt/lustre/usr/libexec'': No space left on device cp: cannot create directory `/mnt/lustre/usr/sbin'': No space left on device cp: cannot create directory `/mnt/lustre/usr/share'': No space left on device cp: cannot create directory `/mnt/lustre/usr/X11R6'': No space left on device cp: cannot create directory `/mnt/lustre/usr/dict'': No space left on device cp: cannot create directory `/mnt/lustre/usr/etc'': No space left on device cp: cannot create directory `/mnt/lustre/usr/games'': No space left on device . but df says me I wrote only 1% of /mnt/lustre Why I cannot write to 50%(for example) of /mnt/lustre? Which parameters must I configure? Where can I find an handbook useful in providing an in-depth understanding of Lustre? http://www.lustre.org/docs/lustre.pdf doesn''t work!!! Useful info: Lustre 1.2.4 Red Hat 9 (kernel Vanilla 2.4.24) My configuration script: lmc -o ale.xml --add node --node server_cvs lmc -m ale.xml --add net --node server_cvs --nid server_cvs --nettype tcp # Configure MDS lmc -m ale.xml --add mds --node server_cvs --mds mds-test --fstype ext3 --dev /tmp/mds-test --size 50000 # Configure OST lmc -m ale.xml --add ost --node server_cvs --ost ost1-test --fstype ext3 --dev /dev/sdc1 # Configure client lmc -m ale.xml --add mtpt --node server_cvs --path /mnt/lustre --mds mds-test --ost ost1-test Ale _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.clusterfs.com https://lists.clusterfs.com/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.clusterfs.com https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
Mc Carthy, Fergal
2006-May-19 07:36 UTC
[Lustre-discuss] Handbook && "No space left on device"
Alessandro, I suspect the problem may be the declaration of your MDS device. You have only allocated 50000k for the size of the MDS loopback device. As a quick way to verify this, what does df -i say for the Lustre file system? If it says that there are no free inodes, then that is the problem. I would recommend using a larger MDS loopback device. As for other information sources, I would recommend checking out the Lustre Wiki (https://wiki.clusterfs.com/lustre) for more info that may be of help. Fergal. -- Fergal.McCarthy@HP.com (The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as "HP CONFIDENTIAL".) -----Original Message----- From: lustre-discuss-admin@lists.clusterfs.com [mailto:lustre-discuss-admin@lists.clusterfs.com] On Behalf Of Alessandro Sent: 22 March 2005 13:14 To: lustre-discuss@lists.clusterfs.com Subject: [Lustre-discuss] Handbook && "No space left on device" Hi to all, I use a real disk (a single partition as /dev/sdc1) as an EXT3 storage=20 filesystem (about 200GByte). When I mount to /mnt/lustre, I can copy only a part of what I want, then I=20 have a lot of messages: (example: ''cp -rfv /usr /mnt/lustre'') cp: cannot create directory `/mnt/lustre/usr/libexec'': No space left on=20 device cp: cannot create directory `/mnt/lustre/usr/sbin'': No space left on device cp: cannot create directory `/mnt/lustre/usr/share'': No space left on device cp: cannot create directory `/mnt/lustre/usr/X11R6'': No space left on device cp: cannot create directory `/mnt/lustre/usr/dict'': No space left on device cp: cannot create directory `/mnt/lustre/usr/etc'': No space left on device cp: cannot create directory `/mnt/lustre/usr/games'': No space left on device .. but df says me I wrote only 1% of /mnt/lustre Why I cannot write to 50%(for example) of /mnt/lustre? Which parameters must I configure? Where can I find an handbook useful in providing an in-depth understanding=20 of Lustre? http://www.lustre.org/docs/lustre.pdf doesn''t work!!! Useful info: Lustre 1.2.4 Red Hat 9 (kernel Vanilla 2.4.24) My configuration script: lmc -o ale.xml --add node --node server_cvs lmc -m ale.xml --add net --node server_cvs --nid server_cvs --nettype tcp # Configure MDS lmc -m ale.xml --add mds --node server_cvs --mds mds-test --fstype=20 ext3 --dev /tmp/mds-test --size 50000 # Configure OST lmc -m ale.xml --add ost --node server_cvs --ost ost1-test --fstype=20 ext3 --dev /dev/sdc1 # Configure client lmc -m ale.xml --add mtpt --node server_cvs --path /mnt/lustre --mds=20 mds-test --ost ost1-test Ale=20 _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.clusterfs.com https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
Fergal is correct. The tell-tale message is: cp: cannot create directory `/mnt/lustre/usr/sbin'': No space left on device -ENOSPC can mean both "out of blocks" as well as "out of inodes". Although a directory uses both, it''s an MDS issue in either case. lustre.pdf was causing more confusion than it was preventing, so it was removed from the web site. It''s not an operational manual, and it''s doesn''t describe the current state of the system. It describes an architecture that we are incrementally achieving, but does not yet exist. -Phil On 3/22/2005 9:16, Mc Carthy, Fergal wrote:> Alessandro, > > I suspect the problem may be the declaration of your MDS device. You > have only allocated 50000k for the size of the MDS loopback device. > > As a quick way to verify this, what does df -i say for the Lustre file > system? If it says that there are no free inodes, then that is the > problem. > > I would recommend using a larger MDS loopback device. > > As for other information sources, I would recommend checking out the > Lustre Wiki (https://wiki.clusterfs.com/lustre) for more info that may > be of help. > > Fergal. > > -- > > Fergal.McCarthy@HP.com > > (The contents of this message and any attachments to it are confidential > and may be legally privileged. If you have received this message in > error you should delete it from your system immediately and advise the > sender. To any recipient of this message within HP, unless otherwise > stated, you should consider this message and attachments as "HP > CONFIDENTIAL".) > > > -----Original Message----- > From: lustre-discuss-admin@lists.clusterfs.com > [mailto:lustre-discuss-admin@lists.clusterfs.com] On Behalf Of > Alessandro > Sent: 22 March 2005 13:14 > To: lustre-discuss@lists.clusterfs.com > Subject: [Lustre-discuss] Handbook && "No space left on device" > > Hi to all, > I use a real disk (a single partition as /dev/sdc1) as an EXT3 storage > filesystem (about 200GByte). > When I mount to /mnt/lustre, I can copy only a part of what I want, then > I > have a lot of messages: > (example: ''cp -rfv /usr /mnt/lustre'') > > cp: cannot create directory `/mnt/lustre/usr/libexec'': No space left on > device > cp: cannot create directory `/mnt/lustre/usr/sbin'': No space left on > device > cp: cannot create directory `/mnt/lustre/usr/share'': No space left on > device > cp: cannot create directory `/mnt/lustre/usr/X11R6'': No space left on > device > cp: cannot create directory `/mnt/lustre/usr/dict'': No space left on > device > cp: cannot create directory `/mnt/lustre/usr/etc'': No space left on > device > cp: cannot create directory `/mnt/lustre/usr/games'': No space left on > device > .. > but df says me I wrote only 1% of /mnt/lustre > > Why I cannot write to 50%(for example) of /mnt/lustre? > Which parameters must I configure? > Where can I find an handbook useful in providing an in-depth > understanding > of Lustre? > http://www.lustre.org/docs/lustre.pdf doesn''t work!!! > > Useful info: > Lustre 1.2.4 > Red Hat 9 (kernel Vanilla 2.4.24) > > My configuration script: > > lmc -o ale.xml --add node --node server_cvs > lmc -m ale.xml --add net --node server_cvs --nid server_cvs --nettype > tcp > > # Configure MDS > > lmc -m ale.xml --add mds --node server_cvs --mds mds-test --fstype > ext3 --dev /tmp/mds-test --size 50000 > > # Configure OST > > lmc -m ale.xml --add ost --node server_cvs --ost ost1-test --fstype > ext3 --dev /dev/sdc1 > > # Configure client > > lmc -m ale.xml --add mtpt --node server_cvs --path /mnt/lustre --mds > mds-test --ost ost1-test > > > > Ale > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss >
Hi to all, I use a real disk (a single partition as /dev/sdc1) as an EXT3 storage filesystem (about 200GByte). When I mount to /mnt/lustre, I can copy only a part of what I want, then I have a lot of messages: (example: ''cp -rfv /usr /mnt/lustre'') cp: cannot create directory `/mnt/lustre/usr/libexec'': No space left on device cp: cannot create directory `/mnt/lustre/usr/sbin'': No space left on device cp: cannot create directory `/mnt/lustre/usr/share'': No space left on device cp: cannot create directory `/mnt/lustre/usr/X11R6'': No space left on device cp: cannot create directory `/mnt/lustre/usr/dict'': No space left on device cp: cannot create directory `/mnt/lustre/usr/etc'': No space left on device cp: cannot create directory `/mnt/lustre/usr/games'': No space left on device .. but df says me I wrote only 1% of /mnt/lustre Why I cannot write to 50%(for example) of /mnt/lustre? Which parameters must I configure? Where can I find an handbook useful in providing an in-depth understanding of Lustre? http://www.lustre.org/docs/lustre.pdf doesn''t work!!! Useful info: Lustre 1.2.4 Red Hat 9 (kernel Vanilla 2.4.24) My configuration script: lmc -o ale.xml --add node --node server_cvs lmc -m ale.xml --add net --node server_cvs --nid server_cvs --nettype tcp # Configure MDS lmc -m ale.xml --add mds --node server_cvs --mds mds-test --fstype ext3 --dev /tmp/mds-test --size 50000 # Configure OST lmc -m ale.xml --add ost --node server_cvs --ost ost1-test --fstype ext3 --dev /dev/sdc1 # Configure client lmc -m ale.xml --add mtpt --node server_cvs --path /mnt/lustre --mds mds-test --ost ost1-test Ale