Gluster users - Apr 2014 - Slow healing times on large cinder and nova volumes

Hi guys,

x-posted from irc.

We're having an issue on our prod openstack environment, which is backed by
gluster using two replicas (I know.  I wasn't given a choice.)

We lost storage on one of the replica servers and so had to replace failed
bricks.  The heal operation on Cinder and Nova volumes is coming up on the
two-week mark and it seems as if it will never catch up and finish.

Nova heal info shows a constantly fluctuating list with multiple heals on many
of the files, as if it's trying to keep up with deltas.  It's at 860GB
of 1.1TB.

Cinder doesn't really seem to progress. It's at about 1.9T out of 6T
utilized, though the total sparse file size totals about 30T.  It also has done
multiple heals on the some files.

I seem to be down to just watching it spin.  Any help or tips?

Thanks,

Larry Schmid | Principal Cloud Engineer

IO

M +1.602.316.8639  |  O +1.602.273.5431
E   lschmid at io.com | io.com



Founded in 2007, IO is a worldwide leader in software defined data center
technology, services and solutions that enable businesses and governments to
intelligently control their information.

The communication contained in this e-mail is confidential and is intended only
for the named recipient(s) and may contain information that is privileged,
proprietary, attorney work product or exempt from disclosure under applicable
law. If you have received this message in error, or are not the named
recipient(s), please note that any form of distribution, copying or use of this
communication or the information in it is strictly prohibited and may be
unlawful. Please immediately notify the sender of the error, and delete this
communication including any attached files from your system. Thank you for your
cooperation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140421/e3d6d7d4/attachment.html>

Could you attach log files please.
You said the bricks are replaced. In case of brick-replacement, index based
self-heal doesn't work so full self-heal needs to be triggered using
"gluster volume heal <volname> full". Could you confirm if that
command is issued?

Pranith
----- Original Message -----
> From: "Larry Schmid" <lschmid at io.com>
> To: gluster-users at gluster.org
> Sent: Tuesday, April 22, 2014 4:07:39 AM
> Subject: [Gluster-users] Slow healing times on large cinder and nova
volumes
> 
> 
> 
> Hi guys,
> 
> 
> 
> x-posted from irc.
> 
> 
> 
> We're having an issue on our prod openstack environment, which is
backed by
> gluster using two replicas (I know. I wasn't given a choice.)
> 
> 
> 
> We lost storage on one of the replica servers and so had to replace failed
> bricks. The heal operation on Cinder and Nova volumes is coming up on the
> two-week mark and it seems as if it will never catch up and finish.
> 
> 
> 
> Nova heal info shows a constantly fluctuating list with multiple heals on
> many of the files, as if it's trying to keep up with deltas. It?s at
860GB
> of 1.1TB.
> 
> 
> 
> Cinder doesn't really seem to progress. It's at about 1.9T out of
6T
> utilized, though the total sparse file size totals about 30T. It also has
> done multiple heals on the some files.
> 
> 
> 
> I seem to be down to just watching it spin. Any help or tips?
> 
> 
> 
> Thanks,
> 
> 
> 
> Larry Schmid | Principal Cloud Engineer
> 
> 
> 
> IO
> 
> 
> 
> M +1.602.316.8639 | O +1.602.273.5431
> 
> E lschmid at io.com | io.com
> 
> 
> 
> 
> Founded in 2007, IO is a worldwide leader in software defined data center
> technology, services and solutions that enable businesses and governments
to
> intelligently control their information.
> 
> The communication contained in this e-mail is confidential and is intended
> only for the named recipient(s) and may contain information that is
> privileged, proprietary, attorney work product or exempt from disclosure
> under applicable law. If you have received this message in error, or are
not
> the named recipient(s), please note that any form of distribution, copying
> or use of this communication or the information in it is strictly
prohibited
> and may be unlawful. Please immediately notify the sender of the error, and
> delete this communication including any attached files from your system.
> Thank you for your cooperation.
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users