Merlin Morgenstern
2015-Aug-24 13:20 UTC
[Gluster-users] How to partition directory structur for 300K files?
I am running into trouble while syncing (rsync, cp ... ) my files to glusterfs. After about 50K files, one machine dies and has to be rebooted. As there are about 300K files in one directory, I am thinking about to cluster that in a directory structure in order to overcome that problem. e.g. /0001/filename /0002/filename That would cut down the amount of files in one directory. However this is something I would like to avoid if possible due to SEO - changing the url of the file brings a lot of trouble. The system underneith are 2 seperate VM instances, each running ubuntu 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 retina with capable SSD and 1G internal network between the VMs. Thank you for any help on this. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150824/295b868f/attachment.html>
Mathieu Chateau
2015-Aug-24 14:01 UTC
[Gluster-users] How to partition directory structur for 300K files?
Hello, did you do some basic tuning to help anyway ? using latest version ? in replication or only distributed ? Why using NFS and not native fuse client to mount volume? did you install VM tools (if using VMware fusion) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern <merlin.morgenstern at gmail.com> :> I am running into trouble while syncing (rsync, cp ... ) my files to > glusterfs. After about 50K files, one machine dies and has to be rebooted. > > As there are about 300K files in one directory, I am thinking about to > cluster that in a directory structure in order to overcome that problem. > > e.g. /0001/filename /0002/filename > > That would cut down the amount of files in one directory. However this is > something I would like to avoid if possible due to SEO - changing the url > of the file brings a lot of trouble. > > The system underneith are 2 seperate VM instances, each running ubuntu > 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 > retina with capable SSD and 1G internal network between the VMs. > > Thank you for any help on this. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150824/0428141e/attachment.html>