Hi, Can anyone tell me about their experiences with 1PB and larger gluster installations. We are thinking of doing 1PB with 10x 100PB storage servers and would appreciate some reassurance :). We plan to scale to....above and beyond.... Ta Andrew
On 02/06/2012 01:41 PM, Andrew Holway wrote:> Hi, > > Can anyone tell me about their experiences with 1PB and larger gluster > installations. We are thinking of doing 1PB with 10x 100PB storage > servers and would appreciate some reassurance :). We plan to scale > to....above and beyond....Greetings Andrew: We've got a bit of experience in scaling to (many large) bricks for gluster. The big issues are reliable/working network connections, RAIDs, etc. Currently you should look carefully at some sort of replication mechanism (if you are storing 1PB of data, recovery time is likely important). 100TB bricks aren't a problem (as long as they are built sanely), though I'd probably suggest subdividing smaller so you can take better advantage of finer grain parallelism. The issue on design of the bricks goes to the bandwidth and the height of the storage bandwidth wall (time required to read/write the entire 100TB). For most designs we see (cascading arrays with SAS/FC), this is going to be problematic at best. But thats another story (and we are biased because of what we do). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: landman at scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
My suggestion is to look at using multiple clusters. Downside is that your application need to be able to load balance on these clusters but in terms of scaling, recovery , rebalance and other issues it is a better approach then one large 1PB cluster On Mon, Feb 6, 2012 at 10:41 AM, Andrew Holway <andrew.holway at gmail.com> wrote:> Hi, > > Can anyone tell me about their experiences with 1PB and larger gluster > installations. We are thinking of doing 1PB with 10x 100PB storage > servers and would appreciate some reassurance :). We plan to scale > to....above and beyond.... > > Ta > > Andrew > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
I agree with Lauro: Redundancy != Backup Except in special cases such as an HPC cluster where result data can be huge and difficult to backup but the data can be recomputed without much fuss if it is lost, you need backups. Even in the case you decide you don't need them then everyone who uses/owns/creates the data much be made aware of the risk. Some HPC centers tell their users outright that they don't back up and their users are on their own. Jeff White - Linux/Unix Systems Engineer University of Pittsburgh - CSSD On 02/17/2012 11:55 AM, John Lauro wrote:> Just a warning about not doing backups because things are so reliable... > > Due to all the redundancy, I don't recall the last time I used a backup > for a hardware failure. That said, I do need them for user error... > > Unless those two copies of the data are protected from each other and not > replicated instantly... a user error will mess up both copies as quickly > as one. > > >> -----Original Message----- >> From: gluster-users-bounces at gluster.org [mailto:gluster-users- >> bounces at gluster.org] On Behalf Of Nathan Stratton >> Sent: Friday, February 17, 2012 10:45 AM >> To: Matty >> Cc: gluster-users at gluster.org >> Subject: Re: [Gluster-users]>1PB >> >> On Fri, 17 Feb 2012, Matty wrote: >> >>> How are folks backing up large amounts of data in their gluster file >>> systems? Replication? Snapshots and archival? As file systems grow to >>> 1PB the conventional backup to disk / tape methodology needs to >>> change. We are putting a lot of thought into this subject here. >> At lest on our end, we don't have backups.... We just make sure we have > 2 >> copies of the data on RAID6 hardware. >> >>> <> >> Nathan Stratton CTO, BlinkMind, Inc. >> nathan at robotics.net nathan at blinkmind.com >> http://www.robotics.net http://www.blinkmind.com >> >>> - Ryan >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Total capability > 1PB, each node has 12 disks and capability of each disk is 1.8TB. We haven't build so larger cluster. The glusterfs's performance is testing on cluster with 4 nodes. -----Original Message----- From: ?? [mailto:renqiang at 360buy.com] Sent: Monday, February 20, 2012 7:40 PM To: 'Nathan Stratton'; 'Song' Cc: gluster-users at gluster.org; 'Andrew Holway' Subject: ??: [Gluster-users] >1PB Can you tell me the total capability of your glusterfs, and how many disks each node has? And your cluster's speed? -----????----- ???: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] ?? Nathan Stratton ????: 2012?2?17? 22:35 ???: Song ??: gluster-users at gluster.org; 'Andrew Holway' ??: Re: [Gluster-users] >1PB On Fri, 17 Feb 2012, Song wrote:> Hi, > > We have the same question. How many nodes are suitable in one gulsterfs? > > For example, volume type: DHT + replication(=3) > There are 240 nodes server. Server information is: > disk: 12*1.8T > network: Gigabit Ethernet > > 40 nodes * 6 clusters? > 60 nodes * 4 clusters? > 80 nodes * 3 clusters? > 120 nodes * 2 clusters? > 240 nodes * 1 cluster? > > What are limitations to scale larger cluster of glusterfs?Gluster can easily support a large number of nodes, the question is how much you care about the underlying data. We store 2 copies of the data and our underlying hardware is RAID6 allowing us to lose 2 disks on each server. As the number of nodes grows that chance of losing a node becomes higher, but one of the beautiful things about Gluster is you still have access to all the other data that is NOT on the lost pair of servers. We have found that your best bet is to split the data over as many servers as possible provides the best up time.><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users