CentOS - Jul 2013 - What FileSystems for large stores and very very large stores?

I was learning about the different FS exists.
I was working on systems that ReiserFS was the star but since there is 
no longer support from the creator there are other consolidations to be 
done.
I want to ask about couple FS options.
EXT4 which is amazing for one node but for more it's another story.
I have heard about GFS2 and GlusterFS and read the docs and official 
materials from RH on them.
In the RH docs it states the EXT4 limit files per directory is 65k and I 
had a directory which was pretty loaded with files and I am unsure 
exactly what was the size but I am almost sure it was larger the 65k 
files per directory.

I was considering using GlusterFS for a very large storage system with 
NFS front.
I am still unsure EXT4 should or shouldn't be able to handle more then 
16TB since the linux kernel ext4 docs at: 
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt in section 2.1
it states: * ability to use filesystems > 16TB (e2fsprogs support not 
available yet).
so can I use it or not?? if there are no tools to handle this size then 
I cannot trust it.

I want to create a storage with more then 16TB based on GlusterFS since 
it allows me to use 2-3 rings FS which will allow me to put the storage 
in a form of:
1 client -> HA NFS servers -> GlusterFS cluster.

it seems to more that GlusterFS is a better choice then Swift since RH 
do provide support for it.

Every response will be appreciated.

Thanks,
Eliezer

Eliezer Croitoru wrote:
<snip>
> I was considering using GlusterFS for a very large storage system with
> NFS front.
> I am still unsure EXT4 should or shouldn't be able to handle more then
> 16TB since the linux kernel ext4 docs at:
> https://www.kernel.org/doc/Documentation/filesystems/ext4.txt in section
> 2.1
> it states: * ability to use filesystems > 16TB (e2fsprogs support not
> available yet). so can I use it or not?? if there are no tools to handle
> this size then I cannot trust it.
I would not go over 16TB (actually, I should say that I have not gone over
it, and I have several LARGE RAID boxes). The tools aren't there, or
don't
work usefully (days to check a filesystem isn't
"useful").
>
> I want to create a storage with more then 16TB based on GlusterFS since
> it allows me to use 2-3 rings FS which will allow me to put the storage
> in a form of:
> 1 client -> HA NFS servers -> GlusterFS cluster.
We tried out glusterfs a couple years ago, but I gather, from my manage
and the user who tried it, that there were some issues. I have no idea how
one would fsck a glusterfs.

*shrug* Why not stay under 16TB, and just mount the filesystems where you
want? Were you looking at someone creating a single file larger than that?

      mark
      mark

On Fri, Jul 5, 2013 at 8:45 AM, Eliezer Croitoru <eliezer at ngtech.co.il>
wrote:
> I was learning about the different FS exists.
> I was working on systems that ReiserFS was the star but since there is
> no longer support from the creator there are other consolidations to be
> done.
> I want to ask about couple FS options.
> EXT4 which is amazing for one node but for more it's another story.
> I have heard about GFS2 and GlusterFS and read the docs and official
> materials from RH on them.
> In the RH docs it states the EXT4 limit files per directory is 65k and I
> had a directory which was pretty loaded with files and I am unsure
> exactly what was the size but I am almost sure it was larger the 65k
> files per directory.
>
> I was considering using GlusterFS for a very large storage system with
> NFS front.
> I am still unsure EXT4 should or shouldn't be able to handle more then
> 16TB since the linux kernel ext4 docs at:
> https://www.kernel.org/doc/Documentation/filesystems/ext4.txt in section
2.1
> it states: * ability to use filesystems > 16TB (e2fsprogs support not
> available yet).
> so can I use it or not?? if there are no tools to handle this size then
> I cannot trust it.
>
> I want to create a storage with more then 16TB based on GlusterFS since
> it allows me to use 2-3 rings FS which will allow me to put the storage
> in a form of:
> 1 client -> HA NFS servers -> GlusterFS cluster.
If you really only have one client, you might look at ceph for
distributed block storage with xfs on top so you don't need to run
through fuse.  Or if your application can be changed to use the s3
interface you could get away from the posix filesystem bottlenecks
completely.

No experience with this stuff - just sounds like the promising
up-and-coming thing...

--
   Les Mikesell
     lesmikesell at gmail.com

----- Original Message -----
| I was learning about the different FS exists.
| I was working on systems that ReiserFS was the star but since there
| is
| no longer support from the creator there are other consolidations to
| be
| done.
| I want to ask about couple FS options.
| EXT4 which is amazing for one node but for more it's another story.
| I have heard about GFS2 and GlusterFS and read the docs and official
| materials from RH on them.
| In the RH docs it states the EXT4 limit files per directory is 65k
| and I
| had a directory which was pretty loaded with files and I am unsure
| exactly what was the size but I am almost sure it was larger the 65k
| files per directory.
| 
| I was considering using GlusterFS for a very large storage system
| with
| NFS front.
| I am still unsure EXT4 should or shouldn't be able to handle more
| then
| 16TB since the linux kernel ext4 docs at:
| https://www.kernel.org/doc/Documentation/filesystems/ext4.txt in
| section 2.1
| it states: * ability to use filesystems > 16TB (e2fsprogs support not
| available yet).
| so can I use it or not?? if there are no tools to handle this size
| then
| I cannot trust it.
| 
| I want to create a storage with more then 16TB based on GlusterFS
| since
| it allows me to use 2-3 rings FS which will allow me to put the
| storage
| in a form of:
| 1 client -> HA NFS servers -> GlusterFS cluster.
| 
| it seems to more that GlusterFS is a better choice then Swift since
| RH
| do provide support for it.
| 
| Every response will be appreciated.
| 
| Thanks,
| Eliezer

As someone who has some rather large volumes for research storage I will say
that ALL of the file systems have limitations, *especially* in the case of
failures.  I have typical volumes that range from 16TB up to 48TB and the big
issue is when it comes to performing file system checks.  You see, there is a
lot of information that gets loaded into memory in order to perform a file
system check.  A number of years ago I was unable to perform a EXT4 file system
check on a 15TB volume without consuming over 32GB of memory on a file system
with very few files.  At the time, the file server only had 8GB of memory, so
this presented a problem.

However, while this problem was solvable it was also subject to usage.  The file
system in question only had large files on it.  These files were typically
gigabytes in size, but for another filer, this time with 48GB of memory but a
tens of millions of very small files, the file system check for it took nearly
96GB of memory in order to perform a file system check.

So far, without a doubt, XFS has been the best "overall" file system
for our usages, but YMMV.  It would seem that Red Hat is also pushing it as the
file system of choice going forward until something better ( btrfs *snicker* )
comes along.  XFS is also the recommended file system for use with GlusterFS so
that makes it an easy choice too.

GlusterFS itself has some H/A built in.  You can talk to any of the GlusterFS
servers via NFS and it will fully operate in an active/active manner so your
diagram would be 1 client -> Gluster Cluster (via protocols supported by
Gluster NFS/CIFS/NATIVE).  I have found it to be rather fragile as well in some
respects and performance for some of my workloads just don't map well to it
even though it looks like they should gain some benefit.  However, it does work
seemingly well for other workloads and it is being actively developed.

GlusterFS also allows you to "import" existing file systems at a later
time.  So feel free to start off with a standard XFS volume, but be mindful of
the XFS options that GlusterFS requires, namely the inode size being 64K, then
if you decide to add cluster to your storage infrastructure you can perform the
said "import" function and then start replication or distributed file
serving from Gluster.

-- 
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier at sfu.ca
Website : http://www.sfu.ca/itservices

?A successful person is one who can lay a solid foundation from the bricks
others have thrown at them.? -David Brinkley via Luke Shaw

OK so back to the issue in hands.
The issue is that I have a mail storage for more then 65k users per
domain and the ext4 doesn't support this size of directory list.
The reiser FS indeed fits for the purpose but ext4 doesn't even start to
scratch it.
Now the real question is that:
What FS will you use for dovecot backhand to store a domain with more
then 65k users?

Eliezer

On 07/05/2013 04:45 PM, Eliezer Croitoru wrote:
> I was learning about the different FS exists.
> I was working on systems that ReiserFS was the star but since there is 
> no longer support from the creator there are other consolidations to be 
> done.
> I want to ask about couple FS options.
> EXT4 which is amazing for one node but for more it's another story.
> I have heard about GFS2 and GlusterFS and read the docs and official 
> materials from RH on them.
> In the RH docs it states the EXT4 limit files per directory is 65k and I 
> had a directory which was pretty loaded with files and I am unsure 
> exactly what was the size but I am almost sure it was larger the 65k 
> files per directory.
> 
> I was considering using GlusterFS for a very large storage system with 
> NFS front.
> I am still unsure EXT4 should or shouldn't be able to handle more then 
> 16TB since the linux kernel ext4 docs at: 
> https://www.kernel.org/doc/Documentation/filesystems/ext4.txt in section
2.1
> it states: * ability to use filesystems > 16TB (e2fsprogs support not 
> available yet).
> so can I use it or not?? if there are no tools to handle this size then 
> I cannot trust it.
> 
> I want to create a storage with more then 16TB based on GlusterFS since 
> it allows me to use 2-3 rings FS which will allow me to put the storage 
> in a form of:
> 1 client -> HA NFS servers -> GlusterFS cluster.
> 
> it seems to more that GlusterFS is a better choice then Swift since RH 
> do provide support for it.
> 
> Every response will be appreciated.
> 
> Thanks,
> Eliezer
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>