Gluster users - Sep 2013 - Enabling Apache Hadoop on GlusterFS: glusterfs-hadoop 2.1 released

Hi Folks

We are pleased to announce a major update to the glusterfs-hadoop project with
the release of version 2.1. The glusterfs-hadoop project, available at The
glusterfs-hadoop project team, provides an Apache licensed Hadoop FileSystem
plugin which enables Apache Hadoop 1.x and 2.x to run directly on top of
GlusterFS. This release includes a re-architected plugin which now extends
existing functionality within Hadoop to run on local and POSIX File Systems.

-- Overview --

Apache Hadoop has a pluggable FileSystem Architecture. This means that if you
have a filesystem or object store that you would like to use with Hadoop, you
can create a Hadoop FileSystem plugin for it which will act as a mediator
between the generic Hadoop FileSystem interface and your filesystem of choice. A
popular example would be that over a million Hadoop clusters are spun up on
Amazon every year, a lot of which use Amazon S3 as the Hadoop FileSystem.

In order to configure the plugin, a specific deployment configuration is
required. Firstly, it is required that the Hadoop JobTracker and TaskTrackers
(or the Hadoop 2.x equivalents) are installed on servers within the gluster
trusted storage pool for a given gluster volume. The JobTracker uses the plugin
to query the extended attributes for job input files in gluster to ascertain
file placement as well as the distribution of file replicas across the cluster.
The TaskTrackers use the plugin to leverage a local fuse mount of the gluster
volume in order to access the data required for the tasks. When the JobTracker
receives a Hadoop job, it uses the locality information it ascertains via the
plugin to send the tasks for the Hadoop Job to Hadoop TaskTrackers on servers
that have the data required for the task within their local bricks. This ensures
data is read from disk and not over the network. Please see the attached diagram
which provides an overview of the entire solution for a Hadoop 1.x deployment.

The community project, along with the documentation and available releases, is
hosted within the Gluster Forge at http://forge.gluster.org/hadoop. The
glusterfs-hadoop project will also be available within the Fedora 20 release
later this year, alongside fellow Fedora newcomer Apache Hadoop and the already
available gluster project. The glusterfs-hadoop project team welcomes
contributions and participation from the broader community.

Stay tuned for upcoming posts around GlusterFS integration into the Apache
Ambari and Fedora projects.

Regards
The glusterfs-hadoop project team
-------------- next part --------------
A non-text attachment was scrubbed...
Name: glusterfs-hadoop.png
Type: image/png
Size: 194453 bytes
Desc: not available
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130905/d695f554/attachment.png>
-------------- next part --------------
_______________________________________________
Announce mailing list
Announce at gluster.org
http://supercolony.gluster.org/mailman/listinfo/announce

On Thu, Sep 5, 2013 at 2:53 PM, Stephen Watt <swatt at redhat.com> wrote:
> Hi Folks
>
> We are pleased to announce a major update to the glusterfs-hadoop project
> with the release of version 2.1. The glusterfs-hadoop project, available at
> The glusterfs-hadoop project team, provides an Apache licensed Hadoop
> FileSystem plugin which enables Apache Hadoop 1.x and 2.x to run directly
> on top of GlusterFS. This release includes a re-architected plugin which
> now extends existing functionality within Hadoop to run on local and POSIX
> File Systems.
>
> -- Overview --
>
> Apache Hadoop has a pluggable FileSystem Architecture. This means that if
> you have a filesystem or object store that you would like to use with
> Hadoop, you can create a Hadoop FileSystem plugin for it which will act as
> a mediator between the generic Hadoop FileSystem interface and your
> filesystem of choice. A popular example would be that over a million Hadoop
> clusters are spun up on Amazon every year, a lot of which use Amazon S3 as
> the Hadoop FileSystem.
>
> In order to configure the plugin, a specific deployment configuration is
> required. Firstly, it is required that the Hadoop JobTracker and
> TaskTrackers (or the Hadoop 2.x equivalents) are installed on servers
> within the gluster trusted storage pool for a given gluster volume. The
> JobTracker uses the plugin to query the extended attributes for job input
> files in gluster to ascertain file placement as well as the distribution of
> file replicas across the cluster. The TaskTrackers use the plugin to
> leverage a local fuse mount of the gluster volume in order to access the
> data required for the tasks. When the JobTracker receives a Hadoop job, it
> uses the locality information it ascertains via the plugin to send the
> tasks for the Hadoop Job to Hadoop TaskTrackers on servers that have the
> data required for the task within their local bricks. This ensures data is
> read from disk and not over the network. Please see the attached diagram
> which provides an overview of the entire solution for a Hadoop 1.x
> deployment.
>
> The community project, along with the documentation and available
> releases, is hosted within the Gluster Forge at
> http://forge.gluster.org/hadoop. The glusterfs-hadoop project will also
> be available within the Fedora 20 release later this year, alongside fellow
> Fedora newcomer Apache Hadoop and the already available gluster project.
> The glusterfs-hadoop project team welcomes contributions and participation
> from the broader community.
>
> Stay tuned for upcoming posts around GlusterFS integration into the Apache
> Ambari and Fedora projects.
>
> Regards
> The glusterfs-hadoop project team
> _______________________________________________
> Announce mailing list
> Announce at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/announce
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
Congratulations! This is great news!!

Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130905/8beabc1e/attachment.html>