Gluster users - Nov 2012 - Using local writes with gluster for temporary storage

Hi,

We have a cluster with 130 compute nodes with an NAS-type
central storage under gluster (3 bricks, ~50TB).  When we
run large number of ocean models we can run into bottlenecks
with many jobs trying to write to our central storage.
It was suggested to us that we could also used gluster to
unite the disks on the compute nodes into a single "disk"
in which files would be written locally.  Then we could
move the files after the runs were complete in a more
sequential manner (thus avoiding overloading the network).

What was originally suggested (the NUFA policy) has since
been deprecated.  What would be the recommended method
of accomplishing our goal in the latest version of Gluster?
And where can we find documentation on it?

We have seen the following links, but would be interested
in any more pointers you may have.  Thanks.

http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6

http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/

http://www.gluster.org/2012/05/back-door-async-replication/

https://github.com/jdarcy/bypass

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

Hello,

We are interested in exactly the same use case and I am very keen to hear how to
do this with gluster after v3.2.

Thanks,
Frank

Frank Sonntag
Meteorologist, MetOcean Solutions Ltd
PO Box 441, New Plymouth, New Zealand 4340
T: +64 7-825 0540
www.metocean.co.nz


On 15/11/2012, at 6:45 AM, Pat Haley wrote:
> 
> Hi,
> 
> We have a cluster with 130 compute nodes with an NAS-type
> central storage under gluster (3 bricks, ~50TB).  When we
> run large number of ocean models we can run into bottlenecks
> with many jobs trying to write to our central storage.
> It was suggested to us that we could also used gluster to
> unite the disks on the compute nodes into a single "disk"
> in which files would be written locally.  Then we could
> move the files after the runs were complete in a more
> sequential manner (thus avoiding overloading the network).
> 
> What was originally suggested (the NUFA policy) has since
> been deprecated.  What would be the recommended method
> of accomplishing our goal in the latest version of Gluster?
> And where can we find documentation on it?
> 
> We have seen the following links, but would be interested
> in any more pointers you may have.  Thanks.
> 
>
http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6
> 
> http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/
> 
> http://www.gluster.org/2012/05/back-door-async-replication/
> 
> https://github.com/jdarcy/bypass
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:  phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

I can't find any official announcement that NUFA is deprecated. The code is
still present in 3.3.
Does anyone know what the plans are re. this feature?


Frank Sonntag
Meteorologist, MetOcean Solutions Ltd
PO Box 441, New Plymouth, New Zealand 4340
T: +64 7-825 0540
www.metocean.co.nz



On 15/11/2012, at 6:45 AM, Pat Haley wrote:
> 
> Hi,
> 
> We have a cluster with 130 compute nodes with an NAS-type
> central storage under gluster (3 bricks, ~50TB).  When we
> run large number of ocean models we can run into bottlenecks
> with many jobs trying to write to our central storage.
> It was suggested to us that we could also used gluster to
> unite the disks on the compute nodes into a single "disk"
> in which files would be written locally.  Then we could
> move the files after the runs were complete in a more
> sequential manner (thus avoiding overloading the network).
> 
> What was originally suggested (the NUFA policy) has since
> been deprecated.  What would be the recommended method
> of accomplishing our goal in the latest version of Gluster?
> And where can we find documentation on it?
> 
> We have seen the following links, but would be interested
> in any more pointers you may have.  Thanks.
> 
>
http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6
> 
> http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/
> 
> http://www.gluster.org/2012/05/back-door-async-replication/
> 
> https://github.com/jdarcy/bypass
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:  phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

On Nov 14, 2012, at 12:45 PM, Pat Haley wrote:
> We have a cluster with 130 compute nodes with an NAS-type
> central storage under gluster (3 bricks, ~50TB).  When we
> run large number of ocean models we can run into bottlenecks
> with many jobs trying to write to our central storage.
> It was suggested to us that we could also used gluster to
> unite the disks on the compute nodes into a single "disk"
> in which files would be written locally.  Then we could
> move the files after the runs were complete in a more
> sequential manner (thus avoiding overloading the network).
>
> What was originally suggested (the NUFA policy) has since
> been deprecated.  What would be the recommended method
> of accomplishing our goal in the latest version of Gluster?
> And where can we find documentation on it?
>
> We have seen the following links, but would be interested
> in any more pointers you may have.  Thanks.
>
>
http://thr3ads.net/gluster-users/2012/06/1941337-how-to-enable-nufa-in-3.2.6
>
> http://blog.aeste.my/2012/05/15/glusterfs-3-2-updates/
>
> http://www.gluster.org/2012/05/back-door-async-replication/
>
> https://github.com/jdarcy/bypass

s the author of the last two, I guess I should weigh in here. There's a
danger with these kinds of solutions, which is the creation of  
conflicts. What
if two clients write to their local copies using bypass? The subsequent
self-heal will fail due to split brain. Sometimes this isn't an issue  
because
of the way applications work, or can be avoided by locking the file, but
otherwise it's a real danger.

Moving from replication to distribution, NUFA is deprecated in truth
regardless of the official status. There's no CLI support, and it  
lacks many
bug fixes and enhancements - e.g. dynamic reconfiguration, rebalance,
replace-brick - that have been applied to DHT. It builds and appears  
to work,
but in my brief tests just now didn't seem to distribute files any  
differently
than DHT would.

All is not lost, however. You can get a limited kind of NUFA-like  
behavior by
using user-specified layouts. Here's some background:

         http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
         http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/

A while back, I added a feature to mark the extended attributes that  
we use as
having been set by the user, so that we'll leave them alone when we  
rebalance.
What you can do is set these attributes on a directory so that the  
local node
"claims" practically all of the consistent-hashing space and new files
will be
placed there but still findable by all of the usual methods and fully
integrated with all the rest of DHT. When your app is done, you can  
just set
the layout type back to normal and let DHT recalaculate it (there's a  
magic
xattr to do this for a single directory so you don't need to do it  
across the
whole volume).  If you actually want to migrate the files around  
physically,
you could do it the same way we do - check whether the file should be
somewhere else, if so create a copy and rename over the original.

I know that all sounds a bit hackish and cumbersome, but it would work  
and not
risk incompatibility with the rest of GlusterFS. I even have some  
scripts to
do some similar things, so if you'd like to collaborate with me on  
automating
some of this I'd be glad to.