Pat Haley
2012-Nov-14 17:45 UTC
[Gluster-users] Using local writes with gluster for temporary storage
Hi, We have a cluster with 130 compute nodes with an NAS-type central storage under gluster (3 bricks, ~50TB). When we run large number of ocean models we can run into bottlenecks with many jobs trying to write to our central storage. It was suggested to us that we could also used gluster to unite the disks on the compute nodes into a single "disk" in which files would be written locally. Then we could move the files after the runs were complete in a more sequential manner (thus avoiding overloading the network). What was originally suggested (the NUFA policy) has since been deprecated. What would be the recommended method of accomplishing our goal in the latest version of Gluster? And where can we find documentation on it? We have seen the following links, but would be interested in any more pointers you may have. Thanks. http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6 http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/ http://www.gluster.org/2012/05/back-door-async-replication/ https://github.com/jdarcy/bypass -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Frank Sonntag
2012-Nov-14 19:16 UTC
[Gluster-users] Using local writes with gluster for temporary storage
Hello, We are interested in exactly the same use case and I am very keen to hear how to do this with gluster after v3.2. Thanks, Frank Frank Sonntag Meteorologist, MetOcean Solutions Ltd PO Box 441, New Plymouth, New Zealand 4340 T: +64 7-825 0540 www.metocean.co.nz On 15/11/2012, at 6:45 AM, Pat Haley wrote:> > Hi, > > We have a cluster with 130 compute nodes with an NAS-type > central storage under gluster (3 bricks, ~50TB). When we > run large number of ocean models we can run into bottlenecks > with many jobs trying to write to our central storage. > It was suggested to us that we could also used gluster to > unite the disks on the compute nodes into a single "disk" > in which files would be written locally. Then we could > move the files after the runs were complete in a more > sequential manner (thus avoiding overloading the network). > > What was originally suggested (the NUFA policy) has since > been deprecated. What would be the recommended method > of accomplishing our goal in the latest version of Gluster? > And where can we find documentation on it? > > We have seen the following links, but would be interested > in any more pointers you may have. Thanks. > > http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6 > > http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/ > > http://www.gluster.org/2012/05/back-door-async-replication/ > > https://github.com/jdarcy/bypass > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email: phaley at mit.edu > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213 http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users
Frank Sonntag
2012-Nov-14 23:46 UTC
[Gluster-users] Using local writes with gluster for temporary storage
I can't find any official announcement that NUFA is deprecated. The code is still present in 3.3. Does anyone know what the plans are re. this feature? Frank Sonntag Meteorologist, MetOcean Solutions Ltd PO Box 441, New Plymouth, New Zealand 4340 T: +64 7-825 0540 www.metocean.co.nz On 15/11/2012, at 6:45 AM, Pat Haley wrote:> > Hi, > > We have a cluster with 130 compute nodes with an NAS-type > central storage under gluster (3 bricks, ~50TB). When we > run large number of ocean models we can run into bottlenecks > with many jobs trying to write to our central storage. > It was suggested to us that we could also used gluster to > unite the disks on the compute nodes into a single "disk" > in which files would be written locally. Then we could > move the files after the runs were complete in a more > sequential manner (thus avoiding overloading the network). > > What was originally suggested (the NUFA policy) has since > been deprecated. What would be the recommended method > of accomplishing our goal in the latest version of Gluster? > And where can we find documentation on it? > > We have seen the following links, but would be interested > in any more pointers you may have. Thanks. > > http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6 > > http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/ > > http://www.gluster.org/2012/05/back-door-async-replication/ > > https://github.com/jdarcy/bypass > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email: phaley at mit.edu > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213 http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users
Jeff Darcy
2012-Nov-15 11:41 UTC
[Gluster-users] Using local writes with gluster for temporary storage
On Nov 14, 2012, at 12:45 PM, Pat Haley wrote:> We have a cluster with 130 compute nodes with an NAS-type > central storage under gluster (3 bricks, ~50TB). When we > run large number of ocean models we can run into bottlenecks > with many jobs trying to write to our central storage. > It was suggested to us that we could also used gluster to > unite the disks on the compute nodes into a single "disk" > in which files would be written locally. Then we could > move the files after the runs were complete in a more > sequential manner (thus avoiding overloading the network). > > What was originally suggested (the NUFA policy) has since > been deprecated. What would be the recommended method > of accomplishing our goal in the latest version of Gluster? > And where can we find documentation on it? > > We have seen the following links, but would be interested > in any more pointers you may have. Thanks. > > http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6 > > http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/ > > http://www.gluster.org/2012/05/back-door-async-replication/ > > https://github.com/jdarcy/bypasss the author of the last two, I guess I should weigh in here. There's a danger with these kinds of solutions, which is the creation of conflicts. What if two clients write to their local copies using bypass? The subsequent self-heal will fail due to split brain. Sometimes this isn't an issue because of the way applications work, or can be avoided by locking the file, but otherwise it's a real danger. Moving from replication to distribution, NUFA is deprecated in truth regardless of the official status. There's no CLI support, and it lacks many bug fixes and enhancements - e.g. dynamic reconfiguration, rebalance, replace-brick - that have been applied to DHT. It builds and appears to work, but in my brief tests just now didn't seem to distribute files any differently than DHT would. All is not lost, however. You can get a limited kind of NUFA-like behavior by using user-specified layouts. Here's some background: http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/ http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/ A while back, I added a feature to mark the extended attributes that we use as having been set by the user, so that we'll leave them alone when we rebalance. What you can do is set these attributes on a directory so that the local node "claims" practically all of the consistent-hashing space and new files will be placed there but still findable by all of the usual methods and fully integrated with all the rest of DHT. When your app is done, you can just set the layout type back to normal and let DHT recalaculate it (there's a magic xattr to do this for a single directory so you don't need to do it across the whole volume). If you actually want to migrate the files around physically, you could do it the same way we do - check whether the file should be somewhere else, if so create a copy and rename over the original. I know that all sounds a bit hackish and cumbersome, but it would work and not risk incompatibility with the rest of GlusterFS. I even have some scripts to do some similar things, so if you'd like to collaborate with me on automating some of this I'd be glad to.