This is a widely perceived feature/bug of gluster. It also affects other
distributed filesystems, tho generally not as much.
We've done 2 things to address this. One is a distributed 'du' that
is
clusterfork'ed out to the storage nodes and compiles the results. This is
realtime and will provide data to that point. If you're interested in it,
let
me know and I can provide the code to do this. However, it requires
clusterfork, some per-site config, and is specific to 'du', altho it
could be
modified to support other shell commands.
Here's the difference in performance on a fairly busy gluster system (4
storage nodes, 8 volumes, 340TB, 60% used)
====================14:54:09 root at hpc-s:/som
1226 $ time du -sh abusch/*
694M abusch/MATS-gtf
^C
real 3m58.098s <--- killed after ~4m
user 0m0.033s
sys 0m0.351s
14:58:24 root at hpc-s:/som
1227 $ gfdu abusch/\*
INFO: Corrected gluster starting path: [/som/abusch/*]
About to execute [/root/bin/cf --script --tar=GLSRV "du -s
/raid1/som/abusch/*
; du -s /raid2/som/abusch/*; "]
Go? [yN]y
INFO: For raw results [cd /root/cf/CF-du--s--raid1-som-
abu-14.58.38_2013-10-08]
Size: File|Dir
693.8203 M /som/abusch/MATS-gtf
1.5292 G /som/abusch/MISO-gffs
764.5117 M /som/abusch/MISO-gffs-v2
23.8720 G /som/abusch/deepSeq
25.2845 G /som/abusch/genomes
5.4239 G /som/abusch/index
16.8011 G /som/abusch/index2
-----------------------------------
74.3348 G Total
time was ~4s
====================
The other approach is with the RobinHood Policy Engine
<http://sourceforge.net/apps/trac/robinhood> which runs on a cron and
recurses
thru your FS, taking X hours, but compiles that info into a MySQL DB that is
instantly responsive (but could be slightly out of date). NTL, it's a very
helpful tool to detect hotspots and ZOTfiles (Zillions Of Tiny files)
We are using it to monitor NFS volumes, Gluster, and Fraunhofer FSs.
It is a very slick system and a student (Adam Brenner) is modifying it to
generate better stats via the web interface.
See his github and the robinhood trac:
https://github.com/abrenner/robinhood-multifs-web
http://sourceforge.net/apps/trac/robinhood
On Tuesday, October 08, 2013 09:07:52 AM Anders Salling Andersen
wrote:> Hi all i have a 50tb glusterfs replicated setup, with Many small files. My
> metadata is very slow ex. Du -sh takes over 24 hours. Is there a Way to
> make faster metadata ?
>
> Regards Anders.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131008/fea421e6/attachment.html>