Hello, We're starting to set up a 4 node gluster system. I'm currently trying to decide on the low-level options, including what filesystem to use. For various reasons I would be more comfortable with XFS over ext4, but I read in the 'Introduction to Gluster' that 'XFS (can be slow)'. I haven't found any other details about this, and wondered if anyone has more information or experience of using gluster with XFS. Or if anything has changed with 3.1. We don't want it to be slow, and I'm happy enough using ext4 if necessary, but just wanted to see what others thought first. Thanks David -- David Lloyd V Consultants www.v-consultants.co.uk
We run two somewhat large gluster clusters in production on xfs with great success. I had to go with xfs as ext4 doesn't support large enough file systems. Make sure you mount your xfs partitions with 64bit inode support and use only 64bit OS's. I'm still running 2.0.9 however the performance is pretty good. We use ours to store media for our website and with our smaller two server four brick 60tb cluster I can easily push 800mbit of http traffic with an average object size of 2-3megs. Not bad for a bunch of slow sata disks! Liam On Nov 15, 2010 2:53 AM, "David Lloyd" <david.lloyd at v-consultants.co.uk> wrote:> Hello, > > We're starting to set up a 4 node gluster system. I'm currently trying > to decide on the low-level options, including what filesystem to use. > > For various reasons I would be more comfortable with XFS over ext4, > but I read in the 'Introduction to Gluster' that 'XFS (can be slow)'. > > I haven't found any other details about this, and wondered if anyone > has more information or experience of using gluster with XFS. Or if > anything has changed with 3.1. We don't want it to be slow, and I'm > happy enough using ext4 if necessary, but just wanted to see what > others thought first. > > Thanks > David > > -- > David Lloyd > V Consultants > www.v-consultants.co.uk > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Our smaller cluster, 60tb, stores media data and acts as our CDN feed system. Its a pretty simple setup, the front end is two Dell 1950 servers running Apache mounting gluster via the fuse client. We use bonded gigabit ethernet on the back side to two supermicro 4u 24 bay servers. Each server has another supermicro 4u 24 bay chassis hanging off the back connected via SAS. Both servers are mirrors of each other. Drives are desktop Seagate 1.5tb drives connected to a 3ware 9690 SAS card. We make one huge 24 drive raid6 volume (~30tb) as a brick and use gluster to glue it all together. Performace is decent - we've pushed nearly 800mbit of web traffic with it. Our Juniper firewall only has gigabit anyway so I don't know how much more I could push if I went to 10g. One weird thing I've noticed is 500mbit of web traffic is about double that on the backend which is why we use bonded ethernet for the backend. Another trick we do is our two frontend webservers only mount one server each - so webserver A only mounts gluster server A. We found that the over head of gluster constantly verifing the files were in sync added a ~20% overhead. All the clients that actually write the data of course mount both servers so the files mirrored correctly. Email me privately if you want more detail. Liam On Nov 15, 2010 6:57 AM, "Rudi Ahlers" <Rudi at softdux.com> wrote: