Gluster users - Sep 2014 - complete f......p thanks to glusterfs...applause, you crashed weeks of work

until a few days ago my opinion about glusterfs was " working but
stable", now i would just call it the a versatile data and time blackhole.

Though i don't even feel like the dev's read the gluster-users list, i
suggest you shot yourself and just do it like truecrypt ( big disclaimer: this
software is insecure, use another product, NO PRODUCTION USE).

It started with the usual issues ,not syncing(3.2) , shd fails(3.2-3.3), peer
doesnt reconnect(3.3), ssl keys have to be 2048-bit fixed size and all keys have
to bey verywhere(all versions....which noob programmed that ??), only control
connection is encrypted,   etc. etc. i kept calm, resynced,recreated, already
gave up.. VERY OFTEN..

At a certain point it also used tons of diskspace due to not deleting files in
the ".glusterfs" directory , (but still being connected and up serving
volumes)

IT WAS A LONG AND PAINFUL SYNCING PROCESS until i thought i was happy ;)

But now the master-fail happened:
 (and i already know you can't pop out a simple solution, but yeah come,
write your mess.. i'll describe it for you)

Due to an Online-resizing lvm/XFS glusterfs (i watch the logs nearly all the
time) i discovered "mismacthing disk layouts" , realizing also that

server1 was up and happy when you mount from it, but server2 spew input/output
errors on several directories (for now just in that volume),

i tried to rename one directory, it created a recursive loop inside XFS (e.g.
BIGGEST FILE-SYSTEM FAIL : TWO INODES linking to one dir , ideally containing
another)
i got at least the XFS loop solved.

Then the pre-last resort option came up.. deleted the volumes, cleaned all xattr
on that ~2T ... and recreated the volumes, since shd seems to work somehow since
3.4
guess what happened ?? i/o errors on server2 on and on , before i could mount on
server1 from server 2 without i/o errors..not now..

Really i would like to love this project, but right now i'm in the mood for
a killswitch (for the whole project), the aim is good, the way glusterfs tries
to achieve this is just poor..tons of senseless logs, really , even your worst
*insertBigCorp* DB server will spit less logs, glusterfs in the default setting
is just eating your diskspace with logs, there is no option to rate-limit ,
everytime you start a volume it logs the volume config... sometimes i feel like
git would be the way to go, not only for the logs (git-annex ;) ) .

now i realized through "ls -R 1>/dev/null" that this happend on ALL
volumes in the cluster, an known problem "can't stat folders".

Maybe anyone has a suggestion , except "create a new clean volume and move
all your TB's" .

Regards

My first suggestion, that's not phrased the very carefully chosen words I
usually use in order to be sure to be read as "nice", would be to stop
screwing the system design by randomly doing things to the bricks that you
clearly don't understand and instead let the software do what it was
designed to do.

I phrase it harshly because you started your email with hostility, blaming
developers that demonstrate more talent before breakfast than you enumerate in
your description below over, apparently, you're three year long experience
(based on the version history).

Most of the problems you describe can be caused directly by your
"fixes" and cannot be caused by resizing xfs.

More inline

On September 1, 2014 8:49:46 PM PDT, Bencho Naut <all at b1nch0.us.to>
wrote:
>until a few days ago my opinion about glusterfs was " working but
>stable", now i would just call it the a versatile data and time
>blackhole.
>
>Though i don't even feel like the dev's read the gluster-users list,
i
>suggest you shot yourself and just do it like truecrypt ( big
>disclaimer: this software is insecure, use another product, NO
>PRODUCTION USE).
>
>It started with the usual issues ,not syncing(3.2) , shd
>fails(3.2-3.3), peer doesnt reconnect(3.3), ssl keys have to be
>2048-bit fixed size and all keys have to bey verywhere(all
>versions....which noob programmed that ??), only control connection is
>encrypted,   etc. etc. i kept calm, resynced,recreated, already gave
>up.. VERY OFTEN..
>
>At a certain point it also used tons of diskspace due to not deleting
>files in the ".glusterfs" directory , (but still being connected
and up
>serving volumes)
>
>IT WAS A LONG AND PAINFUL SYNCING PROCESS until i thought i was happy
>;)
>
>But now the master-fail happened:
>(and i already know you can't pop out a simple solution, but yeah come,
>write your mess.. i'll describe it for you)
>
>Due to an Online-resizing lvm/XFS glusterfs (i watch the logs nearly
>all the time) i discovered "mismacthing disk layouts" , realizing
also
>that 
Mismatching layouts comes from having two fully populated servers with no xattrs
as you describe creating later.
>
>server1 was up and happy when you mount from it, but server2 spew
>input/output errors on several directories (for now just in that
>volume),
Illogical as there should be no difference in operation regardless of which
server provides the client configuration. Did you somehow get the vols tree out
of sync between servers?
>
>i tried to rename one directory, it created a recursive loop inside XFS
>(e.g. BIGGEST FILE-SYSTEM FAIL : TWO INODES linking to one dir ,
>ideally containing another)
>i got at least the XFS loop solved.
>
>Then the pre-last resort option came up.. deleted the volumes, cleaned
>all xattr on that ~2T ... and recreated the volumes, since shd seems to
>work somehow since 3.4
>guess what happened ?? i/o errors on server2 on and on , before i could
>mount on server1 from server 2 without i/o errors..not now..
>
>Really i would like to love this project, but right now i'm in the mood
>for a killswitch (for the whole project), the aim is good, the way
I'm sure you can get a refund for the software. 
>glusterfs tries to achieve this is just poor..tons of senseless logs,
Perhaps you're getting "tons"  because you've already broken
it. I don't great too much in my logs.
>really , even your worst *insertBigCorp* DB server will spit less logs,
Absolutely. In fact I think the bigger the Corp, the smaller and more obfuscated
the logs. I guess that's a good thing?
>glusterfs in the default setting is just eating your diskspace with
>logs, there is no option to rate-limit , everytime you start a volume
>it logs the volume config... sometimes i feel like git would be the way
>to go, not only for the logs (git-annex ;) ) .
>
>now i realized through "ls -R 1>/dev/null" that this happend on
ALL
>volumes in the cluster, an known problem "can't stat folders".
>
>Maybe anyone has a suggestion , except "create a new clean volume and
>move all your TB's" .
BTW... You never backed up your accusation that there is a "data black
hole".

This can be solved. It's a complete mess though, by this account, and
won't be easy. Your gfids and dht mappings may be mismatched, you may have
directories and files with the same path on different bricks, and who knows what
state your .glusterfs directory is in. Hang out in #gluster on irc and I'll
help you as I can. I'm just getting back from vacation so I'll have a
bunch of my own work to catch up on,  too.
>
>Regards
>_______________________________________________
>Gluster-users mailing list
>Gluster-users at gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hmmm, if this message supposed to be some kind of criticism of glusterfs
devs and product, than it is too emotional to take it seriously.

Just another community member...


2014-09-02 6:49 GMT+03:00 Bencho Naut <all at b1nch0.us.to>:
> until a few days ago my opinion about glusterfs was " working but
stable",
> now i would just call it the a versatile data and time blackhole.
>
> Though i don't even feel like the dev's read the gluster-users
list, i
> suggest you shot yourself and just do it like truecrypt ( big disclaimer:
> this software is insecure, use another product, NO PRODUCTION USE).
>
> It started with the usual issues ,not syncing(3.2) , shd fails(3.2-3.3),
> peer doesnt reconnect(3.3), ssl keys have to be 2048-bit fixed size and all
> keys have to bey verywhere(all versions....which noob programmed that ??),
> only control connection is encrypted,   etc. etc. i kept calm,
> resynced,recreated, already gave up.. VERY OFTEN..
>
> At a certain point it also used tons of diskspace due to not deleting
> files in the ".glusterfs" directory , (but still being connected
and up
> serving volumes)
>
> IT WAS A LONG AND PAINFUL SYNCING PROCESS until i thought i was happy ;)
>
> But now the master-fail happened:
>  (and i already know you can't pop out a simple solution, but yeah
come,
> write your mess.. i'll describe it for you)
>
> Due to an Online-resizing lvm/XFS glusterfs (i watch the logs nearly all
> the time) i discovered "mismacthing disk layouts" , realizing
also that
>
> server1 was up and happy when you mount from it, but server2 spew
> input/output errors on several directories (for now just in that volume),
>
> i tried to rename one directory, it created a recursive loop inside XFS
> (e.g. BIGGEST FILE-SYSTEM FAIL : TWO INODES linking to one dir , ideally
> containing another)
> i got at least the XFS loop solved.
>
> Then the pre-last resort option came up.. deleted the volumes, cleaned all
> xattr on that ~2T ... and recreated the volumes, since shd seems to work
> somehow since 3.4
> guess what happened ?? i/o errors on server2 on and on , before i could
> mount on server1 from server 2 without i/o errors..not now..
>
> Really i would like to love this project, but right now i'm in the mood
> for a killswitch (for the whole project), the aim is good, the way
> glusterfs tries to achieve this is just poor..tons of senseless logs,
> really , even your worst *insertBigCorp* DB server will spit less logs,
> glusterfs in the default setting is just eating your diskspace with logs,
> there is no option to rate-limit , everytime you start a volume it logs the
> volume config... sometimes i feel like git would be the way to go, not only
> for the logs (git-annex ;) ) .
>
> now i realized through "ls -R 1>/dev/null" that this happend
on ALL
> volumes in the cluster, an known problem "can't stat
folders".
>
> Maybe anyone has a suggestion , except "create a new clean volume and
move
> all your TB's" .
>
> Regards
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>


-- 
Best regards,
Roman.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140902/1e862d3e/attachment.html>

Could you please provide logs of the bricks and mounts.

Pranith
On 09/02/2014 09:19 AM, Bencho Naut wrote:
> until a few days ago my opinion about glusterfs was " working but
stable", now i would just call it the a versatile data and time blackhole.
>
> Though i don't even feel like the dev's read the gluster-users
list, i suggest you shot yourself and just do it like truecrypt ( big
disclaimer: this software is insecure, use another product, NO PRODUCTION USE).
>
> It started with the usual issues ,not syncing(3.2) , shd fails(3.2-3.3),
peer doesnt reconnect(3.3), ssl keys have to be 2048-bit fixed size and all keys
have to bey verywhere(all versions....which noob programmed that ??), only
control connection is encrypted,   etc. etc. i kept calm, resynced,recreated,
already gave up.. VERY OFTEN..
>
> At a certain point it also used tons of diskspace due to not deleting files
in the ".glusterfs" directory , (but still being connected and up
serving volumes)
>
> IT WAS A LONG AND PAINFUL SYNCING PROCESS until i thought i was happy ;)
>
> But now the master-fail happened:
>   (and i already know you can't pop out a simple solution, but yeah
come, write your mess.. i'll describe it for you)
>
> Due to an Online-resizing lvm/XFS glusterfs (i watch the logs nearly all
the time) i discovered "mismacthing disk layouts" , realizing also
that
>
> server1 was up and happy when you mount from it, but server2 spew
input/output errors on several directories (for now just in that volume),
>
> i tried to rename one directory, it created a recursive loop inside XFS
(e.g. BIGGEST FILE-SYSTEM FAIL : TWO INODES linking to one dir , ideally
containing another)
> i got at least the XFS loop solved.
>
> Then the pre-last resort option came up.. deleted the volumes, cleaned all
xattr on that ~2T ... and recreated the volumes, since shd seems to work somehow
since 3.4
> guess what happened ?? i/o errors on server2 on and on , before i could
mount on server1 from server 2 without i/o errors..not now..
>
> Really i would like to love this project, but right now i'm in the mood
for a killswitch (for the whole project), the aim is good, the way glusterfs
tries to achieve this is just poor..tons of senseless logs, really , even your
worst *insertBigCorp* DB server will spit less logs, glusterfs in the default
setting is just eating your diskspace with logs, there is no option to
rate-limit , everytime you start a volume it logs the volume config... sometimes
i feel like git would be the way to go, not only for the logs (git-annex ;) ) .
>
> now i realized through "ls -R 1>/dev/null" that this happend
on ALL volumes in the cluster, an known problem "can't stat
folders".
>
> Maybe anyone has a suggestion , except "create a new clean volume and
move all your TB's" .
>
> Regards
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

I know it hurts to lose data, and I know it hurts if it seems that it's
the fault of someone else's code/program. As FOSS developer it's often
annoying that users don't do anything to make our lives easier: They
don't
help, they don't pay - they don't even say thank you in most cases.

But when *anything* fails them, there's suddenly lots of time to offload
your frustration on the devs.

However, for the sake of completeness, may I ask, how much you have
supported/paid the developers at all in order to feel like having the
right to insult them like this?


At our facility, we are using GlusterFS successfully for production, and
so far it was a breeze to set up, has never failed us - and the devs are
helpful if asked.
If you need professional support for your setup, there's always the
possibility to ask RedHat for a storage subscription.


Enough troll-feeding for today.

Regards,
Pb

> ssl keys have to be 2048-bit fixed size
No, they don't.
> all keys have to bey verywhere(all versions....which noob programmed
> that ??)
That noob would be me.

t's not necessary to have the same key on all servers, but using
different ones would be even more complex and confusing for users.
Instead, the servers authenticate to one another using a single
identity.  According to SSL 101, anyone authenticating as an identity
needs the key for that identity, because it's really the key - not the
publicly readable cert - that guarantees authenticity.

If you want to set up a separate key+cert for each server, each one
having a "CA" file for the others, you certainly can and it works.
However, you'll still have to deal with distributing those new certs.
That's inherent to how SSL works.  Instead of forcing a particular PKI
or cert-distribution scheme on users, the GlusterFS SSL implementation
is specifically intended to let users make those choices.
> only control connection is encrypted
That's not true.  There are *separate* options to control encryption
for the data path, and in fact that code's much older.  Why separate?
Because the data-path usage of SSL is based on a different identity
model - probably more what you expected, with a separate identity per
client instead of a shared one between servers.
> At a certain point it also used tons of diskspace due to not deleting
> files in the ".glusterfs" directory , (but still being connected
and
> up serving volumes)
For a long time, the only internal conditions that might have caused
the .glusterfs links not to be cleaned up were about 1000x less common
than similar problems which arise when users try to manipulate files
directly on the bricks.  Perhaps if you could describe what you were
doing on the bricks, we could help identify what was going on and
suggest safer ways of achieving the same goals.
> IT WAS A LONG AND PAINFUL SYNCING PROCESS until i thought i was happy
> ;)
Syncing what?  I'm guessing a bit here, but it sounds like you were
trying to do the equivalent of a replace-brick (or perhaps rebalance) by
hand.  As you've clearly discovered, such attempts are fraught with
peril.  Again, with some more constructive engagement perhaps we can
help guide you toward safer solutions.
> Due to an Online-resizing lvm/XFS glusterfs (i watch the logs nearly
> all the time) i discovered "mismacthing disk layouts" , realizing
also
> that
>
> server1 was up and happy when you mount from it, but server2 spew
> input/output errors on several directories (for now just in that
> volume),
The "mismatching layout" messages are usually the result of extended
attributes that are missing from one brick's copy of a directory.  It's
possible that the XFS resize code is racy, in the sense that extended
attributes become unavailable at some stage even though the directory
itself is still accessible.  I suggest that you follow up on that bug
with the XFS developers, who are sure to be much more polite and
responsive than we are.
> i tried to rename one directory, it created a recursive loop inside
> XFS (e.g.  BIGGEST FILE-SYSTEM FAIL : TWO INODES linking to one dir ,
> ideally containing another) i got at least the XFS loop solved.
Another one for the XFS developers.
> Then the pre-last resort option came up.. deleted the volumes, cleaned
> all xattr on that ~2T ... and recreated the volumes, since shd seems
> to work somehow since 3.4
You mention that you cleared all xattrs.  Did you also clear out
.glusterfs?  In general, using anything but a completely empty directory
tree as a brick can be a bit problematic.
> Maybe anyone has a suggestion , except "create a new clean volume and
> move all your TB's" .
More suggestions might have been available if you had sought them
earlier.  At this point, none of us can tell what state your volume is
in, and there are many indications that it's probably a state none of us
have never seen or anticipated.  As you've found, attempting random
fixes in such a situation often makes things worse.  It would be
irresponsible for us to suggest that you go down even more unknown and
untried paths.  Our first priority should be to get things back to a
known and stable state.  Unfortunately at this point the only such
state at this point would seem to be a clean volume.