First post! We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster. We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes. My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be. Any advice would be much appreciated! Cheers, Anthony
I do not have such experiance but if its critical business and if devices are ready to install , register it http://www.gluster.com/ and get 30 day free supoort from glusterfs guys and go on . -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Anthony Goddard Sent: Tuesday, December 29, 2009 11:55 PM To: gluster-users at gluster.org Subject: [Gluster-users] volume sizes First post! We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster. We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes. My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be. Any advice would be much appreciated! Cheers, Anthony _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Anthony, Gluster can take the smaller ( 6TB ) volumes and aggregate them into a large Gluster volume ( as seen from the clients ). So that takes care of managebility on the client side of things. On the server side, once you make those smaller 6 TB volumes, you will depend on RAID to rebuild the disk behind it, so its good to have a smaller partition. Since you are using RAID and not Gluster replication, it might just make sense to have smaller RAID partitions. If instead you were using Gluster replication and resulting recovery, it would happen at first access of the file and the size of the Gluster volume or the backend native FS volume or the RAID ( or raw ) partition behind it would not be much of a consideration. Regards, Tejas. ----- Original Message ----- From: "Anthony Goddard" <agoddard at mbl.edu> To: gluster-users at gluster.org Sent: Wednesday, December 30, 2009 3:24:35 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: [Gluster-users] volume sizes First post! We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster. We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes. My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be. Any advice would be much appreciated! Cheers, Anthony _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
We have a very similar setup. We have a 6 x 24 bay gluster cluster with 36TB per node. We use 3ware raid cards with raid6 over all the 24 drives making a ~32TB usable per node. We have our gluster cluster setup like raid 10, so 3 nodes stripped together and then mirrored to the other 3 nodes. Performance is very good as so is the reliability which was more important to us then performance. I thought about breaking it into smaller pieces but it gets complicated very quick so i went with the simpler is better setup. We also grow about 1tb a week of data so i have to add 1-2 nodes a year which is a huge pain in the butt since gluster doesnt make it very easy to do. (ie building the directory structure on each new node) Doing a ls -agl on the root of our cluster takes well over a week - we have around 50+ million files in there. The only downside is the rebuild time whenever we loose a drive. The 3ware controller with such a large array takes about a week to rebuild from any one drive failure. Of course, with raid6, we can loose two drives without any data loss. Luckily we've never lost two or more drives within the same week. However if we DID for whatever reason loose the whole array we can always pull the data of the other mirror node. I do very closely watch the SMART output of each drive and proactively replace any drive which starts to show any signs of failing or read/write errors. I have a smaller cluster of 4 x 24 bay 36TB per node. This array pushes well over 500mbit of traffic almost 24/7 with almost zero issues. I've been very happy with how well it performs. I do notice that during an array rebuild after a failed drive the IOwait time on the server is a bit higher but over all it does very well. If you would like more information on my setup or what hardware/software i run please feel free to contact me privately. thanks, liam On Tue, Dec 29, 2009 at 1:54 PM, Anthony Goddard <agoddard at mbl.edu> wrote:> First post! > We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster. > We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes. > > My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be. > > Any advice would be much appreciated! > > > Cheers, > Anthony > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
Thanks, Raghvendra. Anthony, Its a "lazy" self-heal mechanism, if you will. If one wants it all done right away, an ls -alR will access each file and hence cause the rebuild of the whole glusterfs volume which _may_, like you mentioned, be spread across disk partitions, LVM/RAID luns or even server nodes. Even after all that, only the files impacted in the volume would need to be rebuilt - although there might be some difference in overheads for different sized and configured Glusterfs volumes. It might be interesting to check - we have not done numbers on this. Let me check with the person who is more familiar with this area of code than me and he may be able to suggest some ballpark numbers till we run some real numbers. Meanwhile, if you do some tests, please share the numbers with the community. Regards, Tejas. ----- Original Message ----- From: "Raghavendra G" <raghavendra at gluster.com> To: "Anthony Goddard" <agoddard at mbl.edu> Cc: "Tejas N. Bhise" <tejas at gluster.com>, "gluster-users" <gluster-users at gluster.org> Sent: Wednesday, December 30, 2009 9:10:23 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: Re: [Gluster-users] volume sizes Hi Anthony, On Wed, Dec 30, 2009 at 6:30 PM, Anthony Goddard < agoddard at mbl.edu > wrote: Hi Tejas, Thanks for the advice. I will be using RAID as well as gluster replication I think.. as we'll only need to sacrifice 1 drive per raid set to add a bit of extra redundancy. The rebuild happens at the first access of a file, does this mean that the entire brick/node is rebuilt upon an initial file access? No, only the file which is accessed is rebuilt. That is the reason we recursively access all the files using 'ls -laR' on mount point. I think this is what I've seen from using gluster previously. If this is the case, it would rebuild the entire volume which could span many raid volumes or even machines, is this correct? If this is the case, then the underlying disk wouldn't have any effect at all, but if it's spanned over multiple machines and it only needs to rebuild one machine (or multiple volumes on one machine) it only needs to rebuild one volume. I don't know if that made any sense.. haha.. but if it did, any insights into whether the size of the volumes (aside from RAID rebuilds) will have a positive effect on glusters rebuild operations? Cheers, Ant. On Dec 30, 2009, at 2:56 AM, Tejas N. Bhise wrote:> Anthony, > > Gluster can take the smaller ( 6TB ) volumes and aggregate them into a large Gluster volume ( as seen from the clients ). So that takes care of managebility on the client side of things. On the server side, once you make those smaller 6 TB volumes, you will depend on RAID to rebuild the disk behind it, so its good to have a smaller partition. Since you are using RAID and not Gluster replication, it might just make sense to have smaller RAID partitions. > > If instead you were using Gluster replication and resulting recovery, it would happen at first access of the file and the size of the Gluster volume or the backend native FS volume or the RAID ( or raw ) partition behind it would not be much of a consideration. > > Regards, > Tejas. > > ----- Original Message ----- > From: "Anthony Goddard" < agoddard at mbl.edu > > To: gluster-users at gluster.org > Sent: Wednesday, December 30, 2009 3:24:35 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi > Subject: [Gluster-users] volume sizes > > First post! > We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster. > We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes. > > My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be. > > Any advice would be much appreciated! > > > Cheers, > Anthony > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users_______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Raghavendra G