Gluster users - Oct 2012 - Gluster-users Digest, Vol 54, Issue 27

On 10/21/2012 02:18 PM, Israel Shirk wrote:
> Haris, try the NFS mount.  Gluster typically triggers healing through
> the client, so if you skip the client, nothing heals.
Not true anymore. With 3.3 there's a self-heal daemon that will handle
the heals. You do risk reading stale data if you don't read through the
client though.

It's not going to trigger this by writing to the brick itself, and doing
'ls -l' or stat is kind of a mess.
> The native Gluster client tends to be really @#$@#$@# stupid.  It'll
> send reads to Singapore while you're in Virginia (and there are bricks
> 0.2ms away),
False. The client will read from the first-to-respond. Yes, if Singapore
is responding faster than Virginia you might want to figure out why
Virginia is so overloaded that it's taking more than 200ms to respond,
but really that shouldn't be the case.

Totally agree.  Should not be the case.  Wish it was false.  But when it
suddenly sends EVERYTHING to Singapore from Virginia, not touching the
servers in Virginia AT ALL, and you can mount them using NFS and it works
great, I gotta point my finger at the client.  You can disagree however
much you want, but I'm talking from very frustrating experiences here.
> then when healing is needed it will take a bunch of time to do that,
> all the while it's blocking your application or web server, which
> under heavy loads will cause your entire application to buckle.
False. 3.3 uses granular locking which won't block your application.

Blocking your application as in, taking so many file descriptors due to lag
that the application runs out of file descriptors or connections and locks
up.
> The NFS client is dumb, which in my mind is a lot better - it'll just
> do what you tell it to do and allow you to compensate for connectivity
> issues yourself using something like Linux-HA.
The "NFS client" is probably more apt than you meant. It is both
GlusterFS client and NFS server, and it connects to the bricks and
performs reads and self-heal in exactly the same way as the fuse client.

It works, where the native Gluster client wakes you up for hours in the
middle of the night.  You can argue about internals, I'm just saying one
works, the other fails miserably.
>
> You have to keep in mind when using gluster that 99% of the people
> using it are running their tests on a single server (see the recent
> notes about how testing patches are only performed on a single server),
False. There are many more testers than that, most of which are outside
of the development team.

You are always right and I am always wrong.  Congratulations :)

I'm simply saying that I keep hearing that Gluster is supposed to work
great on distributed applications (as in distributed to more than one
place), but the reality of the situation is that it's really buggy and
nobody is willing to acknowledge it.  There are big problems with using
Gluster in this way, and I can't see that being the case when it's being
tested for this.  If you're having better luck with it in a high-traffic
production environment, by all means update the docs so others don't have
to go through the downtime that results from Gluster's undocumented issues.
> and most applications don't distribute or mirror to bricks more than a
> few hundred yards away.  Their idea of geo-replication is that you
> send your writes to the other side of the world (which may or may not
> be up at the moment), then twiddle your thumbs for a while and hope it
> gets back to you.  So, that said, it's possible to get it to work, and
> it's almost better than lsyncd, but it'll still make you cry
periodically.
>
> Ok, back to happy time :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121022/3f2d1a5c/attachment.html>

On Mon, Oct 22, 2012 at 08:55:18AM -0600, Israel Shirk wrote:
> I'm simply saying that I keep hearing that Gluster is supposed to work
great on
> distributed applications (as in distributed to more than one place), but
the
> reality of the situation is that it's really buggy and nobody is
willing to
> acknowledge it.  There are big problems with using Gluster in this way, and
I
> can't see that being the case when it's being tested for this.
Israel, you need to distinguish between LAN and WAN. Gluster's normal
replication is for LAN. Gluster's geo-replication is for WAN. Now, I
don't
know where you "keep hearing that Gluster is supposed to work great"
using
normal replication over a WAN. But it's not from anyone close to the
project. So you shouldn't complain to the project about this. You might
suggest a correction to whoever you're hearing it from, though, especially
if they're saying something so misleading in print or a blog where others
may get confused by it.

Best,
Whit

I'm sorry you took that email as an "I'm always right" sort of
thing.
You were telling a user that things were a certain way and for the 
majority of users that I encounter on a daily basis, it's just not true. 
I wanted to set the record straight so that user can make a properly 
informed decision. If you'd put your absolutes in a context of "I tried
to do it this way but it didn't work for me", then I wouldn't have
had a
leg to stand on.

I hang out on the IRC and help people all day long for the last couple 
of years. I've seen many more people with positive experiences than 
negative ones. When something doesn't work as designed, I always 
encourage bug reports. If you have found bugs, please report them. I 
just did a search and can't find any that you're attached to. There are 
a whopping 34 open bugs for geo-replicate and 14 of those are feature 
requests. Of the remaining 20, 5 haven't been fixed yet. That doesn't 
look like "it's really buggy" to me (looking at geo-replication
because
that's the tool designed to be "distributed to more than one
place"
which is what I believe we're discussing).

The reason I, and most of the people that I work with that also have 
successfully used GlusterFS, don't update the docs to tell you how to do 
it successfully is because we just followed the docs and it worked. 
There are, indeed, people for whom that's not true. There have been, on 
occasion, people even in the IRC channel that had problems that were so 
far outside the norm that we couldn't help them. Obviously I can't tell 
you why they have problems because we couldn't figure out their problem. 
Most of the people who do have problems missed at least one step in the 
installation process as documented.

If you've found a bug but are having troubles isolating it, feel free to 
ask for help here or on the IRC channel (#gluster on freenode). I love 
finding bugs so we can all have a better tool to use.

On 10/22/2012 07:55 AM, Israel Shirk wrote:
> On 10/21/2012 02:18 PM, Israel Shirk wrote:
> > Haris, try the NFS mount.  Gluster typically triggers healing through
> > the client, so if you skip the client, nothing heals.
> Not true anymore. With 3.3 there's a self-heal daemon that will handle
> the heals. You do risk reading stale data if you don't read through the
> client though.
>
> It's not going to trigger this by writing to the brick itself, and 
> doing 'ls -l' or stat is kind of a mess.
>
> > The native Gluster client tends to be really @#$@#$@# stupid. 
It'll
> > send reads to Singapore while you're in Virginia (and there are
bricks
> > 0.2ms away),
> False. The client will read from the first-to-respond. Yes, if Singapore
> is responding faster than Virginia you might want to figure out why
> Virginia is so overloaded that it's taking more than 200ms to respond,
> but really that shouldn't be the case.
>
> Totally agree.  Should not be the case.  Wish it was false.  But when 
> it suddenly sends EVERYTHING to Singapore from Virginia, not touching 
> the servers in Virginia AT ALL, and you can mount them using NFS and 
> it works great, I gotta point my finger at the client.  You can 
> disagree however much you want, but I'm talking from very frustrating 
> experiences here.
>
> > then when healing is needed it will take a bunch of time to do that,
> > all the while it's blocking your application or web server, which
> > under heavy loads will cause your entire application to buckle.
> False. 3.3 uses granular locking which won't block your application.
>
> Blocking your application as in, taking so many file descriptors due 
> to lag that the application runs out of file descriptors or 
> connections and locks up.
>
> > The NFS client is dumb, which in my mind is a lot better - it'll
just
> > do what you tell it to do and allow you to compensate for connectivity
> > issues yourself using something like Linux-HA.
> The "NFS client" is probably more apt than you meant. It is both
> GlusterFS client and NFS server, and it connects to the bricks and
> performs reads and self-heal in exactly the same way as the fuse client.
>
> It works, where the native Gluster client wakes you up for hours in 
> the middle of the night.  You can argue about internals, I'm just 
> saying one works, the other fails miserably.
>
> >
> > You have to keep in mind when using gluster that 99% of the people
> > using it are running their tests on a single server (see the recent
> > notes about how testing patches are only performed on a single
server),
> False. There are many more testers than that, most of which are outside
> of the development team.
>
> You are always right and I am always wrong.  Congratulations :)
>
> I'm simply saying that I keep hearing that Gluster is supposed to work 
> great on distributed applications (as in distributed to more than one 
> place), but the reality of the situation is that it's really buggy and 
> nobody is willing to acknowledge it.  There are big problems with 
> using Gluster in this way, and I can't see that being the case when 
> it's being tested for this.  If you're having better luck with it
in a
> high-traffic production environment, by all means update the docs so 
> others don't have to go through the downtime that results from 
> Gluster's undocumented issues.
>
> > and most applications don't distribute or mirror to bricks more
than a
> > few hundred yards away.  Their idea of geo-replication is that you
> > send your writes to the other side of the world (which may or may not
> > be up at the moment), then twiddle your thumbs for a while and hope it
> > gets back to you.  So, that said, it's possible to get it to work,
and
> > it's almost better than lsyncd, but it'll still make you cry
periodically.
> >
> > Ok, back to happy time :)
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121022/764219e6/attachment.html>

Israel - thank you for taking the time to write out more thoughtful responses.
See below for my responses inline.

----- Original Message -----
> On 10/21/2012 02:18 PM, Israel Shirk wrote:
> > Haris, try the NFS mount. Gluster typically triggers healing
> > through
> > the client, so if you skip the client, nothing heals.
> Not true anymore. With 3.3 there's a self-heal daemon that will
> handle
> the heals. You do risk reading stale data if you don't read through
> the
> client though.
> It's not going to trigger this by writing to the brick itself, and
> doing 'ls -l' or stat is kind of a mess.
Actually, the self-heal daemon should work server-side now, and it shouldn't
require an FS stat. That was one of the key features of 3.3. If you notice this
not working properly, please file a bug, as that is not expected behavior.
> > False. The client will read from the first-to-respond. Yes, if
> > Singapore
> >is responding faster than Virginia you might want to figure out why
> >Virginia is so overloaded that it's taking more than 200ms to
> >respond,
> >but really that shouldn't be the case.
> Totally agree. Should not be the case. Wish it was false. But when it
> suddenly sends EVERYTHING to Singapore from Virginia, not touching
> the servers in Virginia AT ALL, and you can mount them using NFS and
> it works great, I gotta point my finger at the client. You can
> disagree however much you want, but I'm talking from very
> frustrating experiences here.
GlusterFS algorithm is first-to-respond, as Joe suggested. I guess the question
is, why are you seeing this behavior? That shouldn't be the case. I'd
like to see some investigation of your particular setup to do some root cause
analysis.

A secondary question would be why you're not using geo-rep for that kind of
setup, but I'd like to get an answer to the first question first.
> > then when healing is needed it will take a bunch of time to do
> > that,
> > all the while it's blocking your application or web server, which
> > under heavy loads will cause your entire application to buckle.
> False. 3.3 uses granular locking which won't block your application.
> Blocking your application as in, taking so many file descriptors due
> to lag that the application runs out of file descriptors or
> connections and locks up.
This sounds like it stems from your replication issues, above. Solving the first
problem should obviate this one.

As always, please report any bugs you find to
https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

-JM 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121022/0f292edb/attachment.html>