Gluster users - Apr 2011 - How to configure with mixed LAN addresses?

Hi,

I didn't title this well, so let me phrase that better. With 2-system
replicated mirror, there is a speed and reliability advantage in joining
those two systems with a crossover cable and putting those two nics in their
own private address space. Gluster-as-server is happy to do that. 

Gluster-as-client isn't happy in this arrangement, when the client is coming
in from the normal LAN addresses rather than the private addresses the
mirroring is using. This appears to be because the default behavior is for
the client to download the volume file from the servers, which then contains
the private IPs, which aren't available from the client's perspective.

I knon in past versions of gluster the choice of the client downloading that
file, or having its own copy, was configurable. It's not clear from the
current stripped-down documentation whether that's still the case. Or if it
is, I don't find discussion of how to configure the client to work right in
this sort of setup. As it is, the client just hangs on the initial mount
request, and sits there consuming 100% of cpu until killed. Of course,
better behavior would be for the client to throw an error and quit in a
polite way - hanging at 100% cpu is just wrong.

The error the client does throw is "dangling volume. check volfile".
When I
search on that all I find is discussions of how it's often a bogus error.
Maybe it's real here? I can't find the instructions on how to fix it if
so,
or even a definition of "dangling volume."

Happily, using an NFS client works just fine in this arrangement. Maybe
gluster just isn't designed for two mirrors to share a private LAN, but not
having a switch between them removes a point of failure, and is faster. Even
where there's a private LAN for storage, it should be better to have yet
another address space for a crossover between mirrors, so I'm hoping
there's
a way to get remote clients happy with this setup.

Best,
Whit

gluster-users-bounces at gluster.org wrote on 04/19/2011 11:07:09 AM:
> I didn't title this well, so let me phrase that better. With 2-system
> replicated mirror, there is a speed and reliability advantage in joining
> those two systems with a crossover cable and putting those two nics in
their
> own private address space. Gluster-as-server is happy to do that.
>
> Gluster-as-client isn't happy in this arrangement, when the client is
coming
> in from the normal LAN addresses rather than the private addresses the
> mirroring is using. This appears to be because the default behavior is
for
> the client to download the volume file from the servers, which then
contains
> the private IPs, which aren't available from the client's
perspective.
>
> I knon in past versions of gluster the choice of the client downloading
that
> file, or having its own copy, was configurable. It's not clear from the
> current stripped-down documentation whether that's still the case. Or
if
it
> is, I don't find discussion of how to configure the client to work
right
in
> this sort of setup. As it is, the client just hangs on the initial mount
> request, and sits there consuming 100% of cpu until killed. Of course,
> better behavior would be for the client to throw an error and quit in a
> polite way - hanging at 100% cpu is just wrong.
So... I've a very similar configuration, and while it is strongly
discouraged by Gluster proper, we just place the configured client files on
the machines and do it that way.

-greg

On Tue, Apr 19, 2011 at 04:56:57PM -0400, Jeff Darcy wrote:
> > But with nfs client use, and/or gluster client use locally on one of
the
> > mirroring gluster servers, it still looks like best practice to use a
> > crossover to me. Or do I have the logic wrong?
> 
> In those cases you're correct so long as there are only two servers
> (another case I tend not to focus on BTW).  That didn't seem like what
> you were talking about, though, since you mentioned both client-side
> volfiles (which wouldn't be used in the NFS case) and private addresses
> (which wouldn't be hidden from clients which are also servers).  I
> therefore concluded that you must be talking about native protocol
> between two servers and separate clients.
I'm talking about a mixed environment. I've presently got - for the same
two
servers - gluster-mirrored storage used by clients on the servers, as well
as gluster-mirrored storage by nfs clients elsewhere on a LAN. I would
_also_ like to use gluster clients on the LAN, to connect to storage on
these boxes, but so far those don't work for me in this configuration. (At
least with 3.1.4 clients as they compile on Ubuntu 8.04.4. Maybe the bug I'm
hitting isn't in the tree I'm barking up here.)

While it's exciting that gluster can handle huge amounts of data for the
likes of Pandora, there are many shops with smaller data storage needs,
whose main goal is consolidating systems in some approximation of an HA
environment. Largish SAS RAID arrays are cheap now, especially if you don't
care much about spindle speed. Servers with a couple of 8-core CPUs are also
cheap. So the temptation is: Take a couple of such servers, put a crossover
between them, and mirror some of their disk storage for HA. Use some of that
storage for servers elsewhere on the LAN, some locally. Use some of the
local storage for kvm vms hosted on these same servers....

What was a whole rack of servers a few years ago can easily fold down to
just a couple of boxes like this today. All the excitement seems to be about
huge clouds, but small clouds are fun too, and useful.

Best,
Whit