Gluster users - Oct 2014 - Perfomance issue on a 90+% full file system

Hello,
My glusterfs-3.4.2-1.el6 is having a performance issue. It was working fine
until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for the
last 10 months. This then dropped to 40Mb/s. Since nothing changed on the
system, I focused on the transition to the 90% full file system.  I also found
that the 6 undelaying XFS files systems were pretty fragmented (~56%).

We are using gluster to achieve a large flat file system. This is a single
server/node configuration, so no network issues are involved.

As the problem is on a production system, I setup a smaller test system. I
monitored performance and was able to duplicate the problem (90MB/s up to 90%
full, then a drop off in performance thereafter). The closer to 100% full, the
lower the throughput.

I then started deleting content from the test server. I was surprised to find
that the performance did NOT increase - it stayed the same. I took the test
system to file system 50% full but still saw 40Mb/s!

The test I set up was designed to fragment the XFS partitions (to mimic the
production system state). I defragmented online successfully, however
performance did not increase.

I am currently trying a rebalance across the 6 XFS partitions to see if that
helps.

I was wondering if anyone remembers a problem like this? Is there a chance that
the rebalance will get me back to the normally seen performance? Would upgrading
gluster fix this?

Thanks,
Bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141006/8acb1ff3/attachment.html>

Yup, pretty common for us.  Once we hit ~90% on either of our two
production clusters (107 TB usable each), performance takes a beating.

I don't consider this a problem, per se.  Most file systems (clustered
or otherwise) are the same.  I consider a high water mark for any
production file system to be 80% (and I consider that vendor
agnostic), at which time action should be taken to begin clean up.
That's good sysadminning 101.

-Dan

----------------
Dan Mons
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au


On 7 October 2014 08:36, Ellison, Bob <bob.ellison at ccur.com>
wrote:
> Hello,
>
> My glusterfs-3.4.2-1.el6 is having a performance issue. It was working fine
> until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for
> the last 10 months. This then dropped to 40Mb/s. Since nothing changed on
> the system, I focused on the transition to the 90% full file system.  I
also
> found that the 6 undelaying XFS files systems were pretty fragmented
(~56%).
>
>
>
> We are using gluster to achieve a large flat file system. This is a single
> server/node configuration, so no network issues are involved.
>
>
>
> As the problem is on a production system, I setup a smaller test system. I
> monitored performance and was able to duplicate the problem (90MB/s up to
> 90% full, then a drop off in performance thereafter). The closer to 100%
> full, the lower the throughput.
>
>
>
> I then started deleting content from the test server. I was surprised to
> find that the performance did NOT increase ? it stayed the same. I took the
> test system to file system 50% full but still saw 40Mb/s!
>
>
>
> The test I set up was designed to fragment the XFS partitions (to mimic the
> production system state). I defragmented online successfully, however
> performance did not increase.
>
>
>
> I am currently trying a rebalance across the 6 XFS partitions to see if
that
> helps.
>
>
>
> I was wondering if anyone remembers a problem like this? Is there a chance
> that the rebalance will get me back to the normally seen performance? Would
> upgrading gluster fix this?
>
>
>
> Thanks,
>
> Bob
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users