We''re looking at deploying 1.6 sometime early next year. I just have a few questions regarding deployment. 1. How critical is the OS on the OST''s. What I mean is traditionally I''ll have a storage box and the data is always separate from the OS. So if I lose the OS I don''t really care, I re-install and copy back over my /etc/exports file. With Lustre I''m not sure I can do that on an OST. 2. Rebalancing. I did a search and it seems that rebalancing is done by hand. IE mv /lustre/data /tmp; mv /tmp/data /lustre; I see that 1.6 has support for auto balancing but I think that just means it will favor OST''s with more space. If I use 90% of my cluster on 4 OST''s and then add two more OST''s Those two OST''s will handle most of the load until they approach the capacity of the other 4. How do other people handle rebalancing? Thanks, -- Daniel Leaberry Senior Systems Administrator iArchives Tel: 801-494-6528 Cell: 801-376-6411 Email: dleaberry@iarchives.com
Hi Daniel, > -----Original Message----- > From: lustre-discuss-bounces@clusterfs.com > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of > Daniel Leaberry > Sent: Tuesday, August 22, 2006 1:41 PM > To: lustre-discuss@clusterfs.com > Subject: [Lustre-discuss] Misc questions on deployment > > We''re looking at deploying 1.6 sometime early next year. I > just have a few questions regarding deployment. > > 1. How critical is the OS on the OST''s. What I mean is > traditionally I''ll have a storage box and the data is always > separate from the OS. So if I lose the OS I don''t really > care, I re-install and copy back over my /etc/exports file. > With Lustre I''m not sure I can do that on an OST. With Lustre you can also - the configuration is stored on the disk. Failover even allows you to do this "live"! > 2. Rebalancing. I did a search and it seems that rebalancing > is done by hand. IE mv /lustre/data /tmp; mv /tmp/data > /lustre; I see that 1.6 has support for auto balancing but I > think that just means it will favor OST''s with more space. > If I use 90% of my cluster on 4 OST''s and then add two more > OST''s Those two OST''s will handle most of the load until > they approach the capacity of the other 4. How do other > people handle rebalancing? Your understanding of what 1.6 will do is correct. There will be on additional feature coming this fall which allows you to mark certain directories (and descendants) to be stored on a particular named collection (aka "pool") of OST''s. I cannot really answer how other systems solve this problem. - Peter - > > > Thanks, > > -- > Daniel Leaberry > Senior Systems Administrator > iArchives > Tel: 801-494-6528 > Cell: 801-376-6411 > Email: dleaberry@iarchives.com > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >
Regarding rebalancing, it would be helpful if a utility to rebalance the file-system can be provided. The rebalance utility might be time-consuming which moves objects across OSTs and maps inodes to the new location of the objects (in MDS). The rebalance operation might be done online in which case the object needs to be locked/unlocked before/after migration respectively or offline in which case FS has to be unmounted. Just a thought.. Regards, -Kums>>> "Peter J. Braam" <braam@clusterfs.com> 8/22/2006 5:59 PM >>>Hi Daniel, > -----Original Message----- > From: lustre-discuss-bounces@clusterfs.com > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of > Daniel Leaberry > Sent: Tuesday, August 22, 2006 1:41 PM > To: lustre-discuss@clusterfs.com > Subject: [Lustre-discuss] Misc questions on deployment > > We''re looking at deploying 1.6 sometime early next year. I > just have a few questions regarding deployment. > > 1. How critical is the OS on the OST''s. What I mean is > traditionally I''ll have a storage box and the data is always > separate from the OS. So if I lose the OS I don''t really > care, I re-install and copy back over my /etc/exports file. > With Lustre I''m not sure I can do that on an OST. With Lustre you can also - the configuration is stored on the disk. Failover even allows you to do this "live"! > 2. Rebalancing. I did a search and it seems that rebalancing > is done by hand. IE mv /lustre/data /tmp; mv /tmp/data > /lustre; I see that 1.6 has support for auto balancing but I > think that just means it will favor OST''s with more space. > If I use 90% of my cluster on 4 OST''s and then add two more > OST''s Those two OST''s will handle most of the load until > they approach the capacity of the other 4. How do other > people handle rebalancing? Your understanding of what 1.6 will do is correct. There will be on additional feature coming this fall which allows you to mark certain directories (and descendants) to be stored on a particular named collection (aka "pool") of OST''s. I cannot really answer how other systems solve this problem. - Peter - > > > Thanks, > > -- > Daniel Leaberry > Senior Systems Administrator > iArchives > Tel: 801-494-6528 > Cell: 801-376-6411 > Email: dleaberry@iarchives.com > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Kumaran Rajaram wrote:> Regarding rebalancing, it would be helpful if a utility to rebalance the > file-system can be provided. The rebalance utility might be > time-consuming which moves objects across OSTs and maps inodes to the > new location of the objects (in MDS). The rebalance operation might be > done online in which case the object needs to be locked/unlocked > before/after migration respectively or offline in which case FS has to > be unmounted. > > Just a thought.. > > Regards, > -Kums >While out looking at the bugzilla yesterday I found this KB entry on rebalancing complete with a rudimentary script. I actually found a lot of good answers that weren''t in the FAQ in the KB. Thanks for providing that resource.> >>>> "Peter J. Braam" <braam@clusterfs.com> 8/22/2006 5:59 PM >>> >>>> > Hi Daniel, > > > -----Original Message----- > > From: lustre-discuss-bounces@clusterfs.com > > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of > > Daniel Leaberry > > Sent: Tuesday, August 22, 2006 1:41 PM > > To: lustre-discuss@clusterfs.com > > Subject: [Lustre-discuss] Misc questions on deployment > > > > We''re looking at deploying 1.6 sometime early next year. I > > just have a few questions regarding deployment. > > > > 1. How critical is the OS on the OST''s. What I mean is > > traditionally I''ll have a storage box and the data is always > > separate from the OS. So if I lose the OS I don''t really > > care, I re-install and copy back over my /etc/exports file. > > With Lustre I''m not sure I can do that on an OST. > > With Lustre you can also - the configuration is stored on the disk. > Failover even allows you to do this "live"! > > > 2. Rebalancing. I did a search and it seems that rebalancing > > is done by hand. IE mv /lustre/data /tmp; mv /tmp/data > > /lustre; I see that 1.6 has support for auto balancing but I > > think that just means it will favor OST''s with more space. > > If I use 90% of my cluster on 4 OST''s and then add two more > > OST''s Those two OST''s will handle most of the load until > > they approach the capacity of the other 4. How do other > > people handle rebalancing? > > Your understanding of what 1.6 will do is correct. There will be on > additional feature coming this fall which allows you to mark certain > directories (and descendants) to be stored on a particular named > collection (aka "pool") of OST''s. > > I cannot really answer how other systems solve this problem. > > > - Peter - > >Thank you Peter for your answers. The marking of directories for certain OST''s sounds like exactly what we would use. We process rolls of microfilm and for disaster recovery purposes it would work far better if each roll was on one OST. Otherwise the loss of an OST would mean we could lose 2-3 files per roll which would be terrible.