Greetings, We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if anyone can suggest what does fragment (block/8) in the ufs2 mean and how this parameter works. I know It's better to read the full ufs2 specification, but hope that someone here can give a hint. Please advice with optimizations or tricks. Thank you very much. -- With best regards, | The Power to Serve Nguyen Tam Chinh | http://www.FreeBSD.org
Nguyen Tam Chinh wrote: [snipped]> Please advice with optimizations or tricks. [...]Did you already looked at 'man 7 tuning'? HTH, Philipp -- www.familie-ost.info/~pj
On 2007-Jun-27 14:11:19 +0400, Nguyen Tam Chinh <unixvn@gmail.com> wrote:> We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well.Short answer: No. Longer answer: FreeBSD and UFS2 have been tweaked to support large numbers of files in larger filesystems and there are no hard limits that you will exceed by having 500,000,000 files in a >1TB FS. However, you will not be able to fsck the FS on an i386 system and will need a lot of RAM+SWAP on amd64 or SPARC64. fsck will also take a _long_ time (hours) to run. Depending on how the files are organised, you may run into severe performance problems with directory searching.> From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works.I suggest you read /usr/share/doc/smm/05.fastfs/paper.ascii.gz Whilst this paper discusses UFS1, the basics remain the same. I have tried using a 4K/0.5K UFS1 filesystem in the past and found the performance was very poor. UFS2 was based on 16K/2K and I would expect it to perform even worse with 4K/0.5K. I would suggest you try 8K/1K. BTW, in sizing your system, you will need to allow for both the last space when the file sizes are rounded up to a multiple of the fragment size, as well as the inode size (256 bytes). If you have 1TB of data, it's likely that you will have another 0.5-1TB of overheads. Overall, I suggest you look at an alternative way to store the data. -- Peter Jeremy -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20070630/08a7a8e8/attachment.pgp
> We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read the > full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks. > Thank you very much.Try zfs on amd64 unless your app doesn't work well with zfs or your organization doesn't allow current. Current is remarkably stable taking into account zfs is fairly new and ported from solaris and running on current. I'm using it on a 8.2 TB nexsan storage and no crashes during testing and a limited time in production. Some years ago I used FreeBSD (5.2) as nfs-server (using ufs2) on approx. 15 partitions ranging from 400 GB to 2 TB in size. If the server for some reason had crashed the webservers were unable to access the nfs-mounted partitions during the period the server did a snapshot of a partition, in order to perform a background-fsck and thus our website was down. So ufs2 does not scale well. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
Nguyen Tam Chinh wrote:> Greetings, > > We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read the > full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks. > Thank you very much. >If all else fails; try "divide and conquer" by having one filesystem per subdirectory. Unless you plan on having all files in a single dir?! Also look at how the squid proxy stores its files using a hashed dir structure. Another alternative is storing the data in a database if you don't need direct random RW file access..
> approx. 15 partitions ranging from 400 GB to 2 TB in size. If the > server for some reason had crashed the webservers were unable tothe question is about the reason it crashed...> access the nfs-mounted partitions during the period the server did a > snapshot of a partition, in order to perform a background-fsck and > thus our website was down. So ufs2 does not scale well.
> I have tried using a 4K/0.5K UFS1 filesystem in the past and found the > performance was very poor. UFS2 was based on 16K/2K and I would expect > it to perform even worse with 4K/0.5K. I would suggest you try 8K/1K. >not for small files. you are light with large files but it's not THAT bad as you say. i reagularly use 4K/0.5 UFS but not for everything if i require good fast speed for big files. for really big files i make 32/4 filesystem with very little inodes
> > We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read theexactly as a block/cluster in windows. fragment is the smallest allocation block. "block" is a group of 8 fragments to make allocation faster and smarter.> full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks.please DO NOT make single partition like that. try to divide it to 3-4 partitions. it will work on a single one but waiting for fsck will kill you ;) AFAIK fsck time grows nonlinearly with fs size to some extent.. options for newfs will be like that newfs -m <A> -i <B> -b 4096 -f 512 -U /dev/partition where A is space left. with mostly small files and huge partition don't worry to set it 1 or even 0. B - size of disk(bytes)/amount of inodes default is probably 2048, you may use 1024 or 4096 for your case - make rough estimate how much files will you have (you told between 4 and 0.5k, but what average?). making too much inodes=wasted space (128 bytes/inode), making too little=big problem :) another question - HOW do you plan to make backups of such data? with dump rsync tar etc. it's clearly "mission impossible". feel free to mail me i had such cases not 5E8 but over 1E8 files :)
> > approx. 15 partitions ranging from 400 GB to 2 TB in size. If the > > server for some reason had crashed the webservers were unable to > > the question is about the reason it crashed... > > > access the nfs-mounted partitions during the period the server did a > > snapshot of a partition, in order to perform a background-fsck and > > thus our website was down. So ufs2 does not scale well.Reasons not related to the nfs-server itself. FreeBSD itself was rock-solid. It was firmware-related on the storage-side. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
>> > snapshot of a partition, in order to perform a background-fsck and >> > thus our website was down. So ufs2 does not scale well. > > Reasons not related to the nfs-server itself. FreeBSD itself was > rock-solid. It was firmware-related on the storage-side. >i always use software mirror concat or both in FreeBSD. always works, 10 times cheaper, fully portable and (yes true) comparable in speed. in some cases - faster.
On Wednesday, 27 June 2007 at 14:11:19 +0400, Nguyen Tam Chinh wrote:> Greetings, > > We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read the > full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks. > Thank you very much. > > -- > With best regards, | The Power to Serve > Nguyen Tam Chinh | http://www.FreeBSD.org > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"I am not aware of any ZFS results on such tasks, may be you will be the one who share them ;) However RaiserFS whould be the best choise on such spesific case. It's not available on FreeBSD currently. I don't think UFS can handle a huge amount of small files effectively. Of course gjournal could be an option for fsck problems, but how do you plan to backup or sync this storage? -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================