justglusterfs at gmail.com
2014-Oct-08 08:45 UTC
[Gluster-users] glusterfs replica volume self heal lots of small file very very slow！！why？how to improve？
Hi all?
I do the following test?
I create a glusterfs replica volume (replica count is 2 ) with two server
node(server A and server B)?use XFS as the underlying filesystem? then mount
the volume in client node?
then? I shut down the network of server A node? in client node? I copy a
dir?which has a lot of small files?? the dir size is 2.9GByte?
when copy finish? I unmount the volume from the client? then I start the
network of server A node? now? glusterfs self-heal-daemon start heal dir
from server B to server A?
in the end? I find the self-heal-daemon heal the dir use 40 minutes?
It's too slow? why?
I find out related options with self-heal? as follow?
cluster.self-heal-window-size
cluster.self-heal-readdir-size
cluster.background-self-heal-count
then I config :
cluster.self-heal-window-size is 1024(max value)
cluster.self-heal-readdir-size is 131072(max value)
and then do the same test case, find this times heal the dir use 35
minutes, The effective is not obvious,
I want to ask, If there are better ways to improve replica volume self heal
lots of small file performance??
thanks?
justglusterfs at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141008/1126b580/attachment.html>
Joe Julian
2014-Oct-08 14:30 UTC
[Gluster-users] [Gluster-devel] glusterfs replica volume self heal lots of small file very very slow！！why？how to improve？
On 10/8/2014 1:45 AM, justglusterfs at gmail.com wrote:> Hi all? > I do the following test? > I create a glusterfs replica volume (replica count is 2 ) with two > server node(server A and server B)?use XFS as the underlying > filesystem? then mount the volume in client node? > then? I shut down the network of server A node? in client > node? I copy a dir?which has a lot of small files?? the dir size > is *2.9GByte?* > when copy finish? I unmount the volume from the client? then I > start the network of server A node? now? glusterfs > self-heal-daemon start heal dir from server B to server A? > in the end? I find the self-heal-daemon heal the dir *use > 40 m**inutes? * *It's too slow? why?* > * > * > I find out related options with self-heal? as follow? > *cluster.self-heal-window-size* > *cluster.self-heal-readdir-size* > *cluster.background-self-heal-count* > * > * > ** > * then I config :* > *cluster.self-heal-window-size is 1024(max value)* > *cluster.self-heal-readdir-size is 131072(max value)* > ** > * and then do the same test case, find this times heal the > dir use 35 minutes, **The effective is not obvious, * > * > * > * > * > * I want to ask, If there are better ways to improve replica volume > self heal lots of small file performance??* >If you need it healed that quickly, mount the volume on a client and walk the tree with "find -exec stat {} \; > /dev/null" -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141008/ae4166ee/attachment.html>
Pranith Kumar Karampuri
2014-Oct-15 15:29 UTC
[Gluster-users] [Gluster-devel] glusterfs replica volume self heal lots of small file very very slow！！why？how to improve？
On 10/08/2014 02:15 PM, justglusterfs at gmail.com wrote:> Hi all? > I do the following test? > I create a glusterfs replica volume (replica count is 2 ) with two > server node(server A and server B)?use XFS as the underlying > filesystem? then mount the volume in client node? > then? I shut down the network of server A node? in client > node? I copy a dir?which has a lot of small files?? the dir size > is *2.9GByte?* > when copy finish? I unmount the volume from the client? then I > start the network of server A node? now? glusterfs > self-heal-daemon start heal dir from server B to server A? > in the end? I find the self-heal-daemon heal the dir *use > 40 m**inutes? * *It's too slow? why?* > I find out related options with self-heal? as follow?Sorry for the delay in response. It is better to enable profile on the volume and find what is happening. Just before restoring the network connection, enable profiling using 'gluster volume profile <volname> start' Let the healing complete. Give the output of 'gluster volume profile <volname> info' output. We can check that to figure out what is happening. Pranith> *cluster.self-heal-window-size* > *cluster.self-heal-readdir-size* > *cluster.background-self-heal-count* > * > * > ** > * then I config :* > *cluster.self-heal-window-size is 1024(max value)* > *cluster.self-heal-readdir-size is 131072(max value)* > ** > * and then do the same test case, find this times heal the > dir use 35 minutes, **The effective is not obvious, * > * > * > * > * > * I want to ask, If there are better ways to improve replica volume > self heal lots of small file performance??* > thanks? > > ------------------------------------------------------------------------ > justglusterfs at gmail.com > > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-devel-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141015/1d98ac7e/attachment.html>