Ocfs2 users - Nov 2009 - Space used on file system although empty.

Hi,

I'm busy doing a bit of testing with an ocfs2 partition on an iSCSI target.
I'm using Ubuntu 9.10 (Karmic) with the latest kernel module and tools
included with that release (tools version 1.4.2 I believe). And I made the
fs with the following command:

mkfs.ocfs2 -L mailstore1 -N 4 -T mail /dev/sdb1

Initially the fs showed 1.2G of 20G already used. Creating files with dd and
deleting them again puts the space usage back down normally. I ran bonnie++
1.96 (threaded version) overnight and now it shows 3.1G used when the only
thing on the fs is the lost+found directory.
I ran fsck.ocfs2 on the partition but it found no errors or orphaned inodes
as was happening before upgrading Ubuntu.

Is this extra space usage after a lot of writing/deleting expected?
If so, can I calculate how much space I'm like to have to sacrifice to this
(the 20GB is for testing, the final iSCSI target would likely be about
1.5TB)? The iSCSI target is on a SAN provided by our hosting company, so it
would help if we can have a more accurate prediction of the space we're
doing to need to buy.

Also, are there any other options I can use when creating the fs that might
be better for usage as a mail store?
We hit about 700 concurrent dovecot connections at peak. We use Maildir++
and most messages are probably between 10KB and 200KB, with some being up to
20MB with attachments etc.

Any other advice/experiences related to this sort of usage would also be
appreciated.

Thanks
Guy

-- 
Don't just do something...sit there!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20091113/9bf6b7a1/attachment.html

In the mail configuration, the default journal size is 256MB. (For 
datafiles, it is 64MB).
4 slots * 256MB is 1G used up in journals. Note that mkfs.ocfs2 allows 
users to specify
a custom journal size too.

OCFS2 has dynamic inode allocators. Meaning it does not allocate space 
for inodes during
format but does so during runtime. However, once space has been carved 
out for inodes, it
is not reclaimed back for data. So your bonnie++ run must have created a 
lot of files.
That space will be reused when new files are created in the fs. You can 
look at the system
directory using debugfs.ocfs2 to confirm my theory.

Go with block/cluster size 4K/4K. While a smaller blocksize will save 
you space per-inode
(as each inode is block-sized), the io thruput will also be lower. So it 
is best to with
4K blocksize, and enable inline-data (default feature) to store the 
small files (3K or so)
in the inode itself.

Sunil

Guy wrote:
> Hi,
>
> I'm busy doing a bit of testing with an ocfs2 partition on an iSCSI 
> target. I'm using Ubuntu 9.10 (Karmic) with the latest kernel module 
> and tools included with that release (tools version 1.4.2 I believe). 
> And I made the fs with the following command:
>
> mkfs.ocfs2 -L mailstore1 -N 4 -T mail /dev/sdb1
>
> Initially the fs showed 1.2G of 20G already used. Creating files with 
> dd and deleting them again puts the space usage back down normally. I 
> ran bonnie++ 1.96 (threaded version) overnight and now it shows 3.1G 
> used when the only thing on the fs is the lost+found directory.
> I ran fsck.ocfs2 on the partition but it found no errors or orphaned 
> inodes as was happening before upgrading Ubuntu.
>
> Is this extra space usage after a lot of writing/deleting expected?
> If so, can I calculate how much space I'm like to have to sacrifice to 
> this (the 20GB is for testing, the final iSCSI target would likely be 
> about 1.5TB)? The iSCSI target is on a SAN provided by our hosting 
> company, so it would help if we can have a more accurate prediction of 
> the space we're doing to need to buy.
>
> Also, are there any other options I can use when creating the fs that 
> might be better for usage as a mail store?
> We hit about 700 concurrent dovecot connections at peak. We use 
> Maildir++ and most messages are probably between 10KB and 200KB, with 
> some being up to 20MB with attachments etc.
>
> Any other advice/experiences related to this sort of usage would also 
> be appreciated.
>
> Thanks
> Guy
>

http://oss.oracle.com/~smushran/.debug/scripts/stat_sysdir.sh

Run this script against the device and save the output to a file.

Then look at the inode_alloc:XXXX . There will be one for each slot.

    Bitmap Total: 18432   Used: 1114   Free: 17318

That's the number of inodes, used and free. Add these numbers for all slots
to get the full picture.


Guy wrote:
> Hi Sunil,
>
> 2009/11/13 Sunil Mushran <sunil.mushran at oracle.com 
> <mailto:sunil.mushran at oracle.com>>
>
>     OCFS2 has dynamic inode allocators. Meaning it does not allocate
>     space for inodes during
>     format but does so during runtime. However, once space has been
>     carved out for inodes, it
>     is not reclaimed back for data. So your bonnie++ run must have
>     created a lot of files.
>     That space will be reused when new files are created in the fs.
>     You can look at the system
>     directory using debugfs.ocfs2 to confirm my theory.
>
>
> Can you tell me how I'd get that particular bit of information from 
> debugfs.ocfs2? I'm afraid I'm not very familiar with it and the man
> page isn't enlightening me at this point.
>
> Is there also a way I could calculate roughly how much space will go 
> to this based on the number of existing files on the current mailstore? 
>  
> Thanks
> Guy
>
> -- 
> Don't just do something...sit there!