Hi, I'm busy doing a bit of testing with an ocfs2 partition on an iSCSI target. I'm using Ubuntu 9.10 (Karmic) with the latest kernel module and tools included with that release (tools version 1.4.2 I believe). And I made the fs with the following command: mkfs.ocfs2 -L mailstore1 -N 4 -T mail /dev/sdb1 Initially the fs showed 1.2G of 20G already used. Creating files with dd and deleting them again puts the space usage back down normally. I ran bonnie++ 1.96 (threaded version) overnight and now it shows 3.1G used when the only thing on the fs is the lost+found directory. I ran fsck.ocfs2 on the partition but it found no errors or orphaned inodes as was happening before upgrading Ubuntu. Is this extra space usage after a lot of writing/deleting expected? If so, can I calculate how much space I'm like to have to sacrifice to this (the 20GB is for testing, the final iSCSI target would likely be about 1.5TB)? The iSCSI target is on a SAN provided by our hosting company, so it would help if we can have a more accurate prediction of the space we're doing to need to buy. Also, are there any other options I can use when creating the fs that might be better for usage as a mail store? We hit about 700 concurrent dovecot connections at peak. We use Maildir++ and most messages are probably between 10KB and 200KB, with some being up to 20MB with attachments etc. Any other advice/experiences related to this sort of usage would also be appreciated. Thanks Guy -- Don't just do something...sit there! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20091113/9bf6b7a1/attachment.html
Sunil Mushran
2009-Nov-13 21:18 UTC
[Ocfs2-users] Space used on file system although empty.
In the mail configuration, the default journal size is 256MB. (For datafiles, it is 64MB). 4 slots * 256MB is 1G used up in journals. Note that mkfs.ocfs2 allows users to specify a custom journal size too. OCFS2 has dynamic inode allocators. Meaning it does not allocate space for inodes during format but does so during runtime. However, once space has been carved out for inodes, it is not reclaimed back for data. So your bonnie++ run must have created a lot of files. That space will be reused when new files are created in the fs. You can look at the system directory using debugfs.ocfs2 to confirm my theory. Go with block/cluster size 4K/4K. While a smaller blocksize will save you space per-inode (as each inode is block-sized), the io thruput will also be lower. So it is best to with 4K blocksize, and enable inline-data (default feature) to store the small files (3K or so) in the inode itself. Sunil Guy wrote:> Hi, > > I'm busy doing a bit of testing with an ocfs2 partition on an iSCSI > target. I'm using Ubuntu 9.10 (Karmic) with the latest kernel module > and tools included with that release (tools version 1.4.2 I believe). > And I made the fs with the following command: > > mkfs.ocfs2 -L mailstore1 -N 4 -T mail /dev/sdb1 > > Initially the fs showed 1.2G of 20G already used. Creating files with > dd and deleting them again puts the space usage back down normally. I > ran bonnie++ 1.96 (threaded version) overnight and now it shows 3.1G > used when the only thing on the fs is the lost+found directory. > I ran fsck.ocfs2 on the partition but it found no errors or orphaned > inodes as was happening before upgrading Ubuntu. > > Is this extra space usage after a lot of writing/deleting expected? > If so, can I calculate how much space I'm like to have to sacrifice to > this (the 20GB is for testing, the final iSCSI target would likely be > about 1.5TB)? The iSCSI target is on a SAN provided by our hosting > company, so it would help if we can have a more accurate prediction of > the space we're doing to need to buy. > > Also, are there any other options I can use when creating the fs that > might be better for usage as a mail store? > We hit about 700 concurrent dovecot connections at peak. We use > Maildir++ and most messages are probably between 10KB and 200KB, with > some being up to 20MB with attachments etc. > > Any other advice/experiences related to this sort of usage would also > be appreciated. > > Thanks > Guy >
Sunil Mushran
2009-Nov-16 18:40 UTC
[Ocfs2-users] Space used on file system although empty.
http://oss.oracle.com/~smushran/.debug/scripts/stat_sysdir.sh Run this script against the device and save the output to a file. Then look at the inode_alloc:XXXX . There will be one for each slot. Bitmap Total: 18432 Used: 1114 Free: 17318 That's the number of inodes, used and free. Add these numbers for all slots to get the full picture. Guy wrote:> Hi Sunil, > > 2009/11/13 Sunil Mushran <sunil.mushran at oracle.com > <mailto:sunil.mushran at oracle.com>> > > OCFS2 has dynamic inode allocators. Meaning it does not allocate > space for inodes during > format but does so during runtime. However, once space has been > carved out for inodes, it > is not reclaimed back for data. So your bonnie++ run must have > created a lot of files. > That space will be reused when new files are created in the fs. > You can look at the system > directory using debugfs.ocfs2 to confirm my theory. > > > Can you tell me how I'd get that particular bit of information from > debugfs.ocfs2? I'm afraid I'm not very familiar with it and the man > page isn't enlightening me at this point. > > Is there also a way I could calculate roughly how much space will go > to this based on the number of existing files on the current mailstore? > > Thanks > Guy > > -- > Don't just do something...sit there!