So, we're trying to serve home directories, and some directories with applications from our gluster cluster, mounting as glusterfs. This is stuff we've done quite happily from nfs previously. The main use of the gluster is to serve large image files and it's great for that but the piddling little files that in the homedirs and applications are much slower (~10x) than the previous nfs. This is measured as the time it takes for the application to start, rather than anything low-level. It feels to me that it's the same kind of thing that's been mentioned here before, but it doesn't seem to have been answered. Should we expect this behaviour? Is there anything we can do to improve things? We're on 3.1.1, haven't done much in the way of tuning from defaults. Cheers David -- David Lloyd V Consultants www.v-consultants.co.uk
In my opinion, you will only see this improved when GlusterFS WAN Replication is available. How on earth is this related, you ask? Here's what I think is happening in your case: - NFS is not a distributed FS. There is one host, an NFS Server. When you access a file, the standard getAttr() happens (you can also monitor it with nfsstat), and you then start getting the file data. - GlusterFS is distributed! There is the case that your file could be re-written on any of the nodes in the cluster, that host this file. In order to make sure you get the most up-to-date file, when you try to access the file, GlusterFS will check with ALL nodes. The more machines you have in the cluster, in theory, the longer this might take. - How this is related to WAN Replication? In WAN replication this process takes much longer because it happens on the WAN, and makes GlusterFS unsuitable for WAN scenarios. I know some work is being done in this direction, and I have a feeling it will greatly benefit LAN scenarios as well, where very small files are involved. I guess some sort of state caching on the clients will happen, or some of the checks will be skippable (by configuration), with some "eventual replication" being allowed. Anyway this is all theoretical - I have no knowledge of the inner workings of GlusterFS. On Jan 27, 2011, at 2:15 AM, David Lloyd wrote:> So, we're trying to serve home directories, and some directories with > applications from our gluster cluster, mounting as glusterfs. This is stuff > we've done quite happily from nfs previously. > > The main use of the gluster is to serve large image files and it's great for > that but the piddling little files that in the homedirs and applications are > much slower (~10x) than the previous nfs. This is measured as the time it > takes for the application to start, rather than anything low-level. > > It feels to me that it's the same kind of thing that's been mentioned here > before, but it doesn't seem to have been answered. > > Should we expect this behaviour? > Is there anything we can do to improve things? > > We're on 3.1.1, haven't done much in the way of tuning from defaults. > > Cheers > David > > -- > David Lloyd > V Consultants > www.v-consultants.co.uk > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Hi, I made the transition in a hurry a week ago from an NFS server to Gluster for home folder exports (story should be in the mail archives of the list). We have about 40 PCs with Linux, which now mount the home folders directly from Gluster (via GlusterFS and no longer via NFS). Gluster also exports the storage for our virtual machines. The performance on the first-day for the desktops was a flop, not so much because of Gluster, but because of the disks. After changing from low-rpm "green" hard disks to better disks set in raid, the desktop experience came back to normal -- no really noticeable drop in performance. One week later, and nobody is complaining... Looking at iostat on the underlying Gluster "bricks", the disk is not getting loaded very much if you look at how much data is being written -- maybe a few hundred KB per second -- but the transactions per second iostat shows, never drops below 30-40, and I've seen it as high as 300-400. Since the performance at the desktop for now is acceptable, I haven't put more time into this for now. I plan on making two tests when time permits: mounting NFS from Gluster (Gluster's NFS export); trying to set up Gluster with client-side caching. I think a longer term solution for me will come down to creating a virtual machine: user files will be stored inside, and for Gluster, it will become just one big file on it's storage. This will also probably become necessary as security comes back to mind. Regards, Andrew On Thu, Jan 27, 2011 at 12:15 AM, David Lloyd <david.lloyd at v-consultants.co.uk> wrote:> So, we're trying to serve home directories, and some directories with > applications from our gluster cluster, mounting as glusterfs. This is stuff > we've done quite happily from nfs previously. > > The main use of the gluster is to serve large image files and it's great for > that but the piddling little files that in the homedirs and applications are > much slower (~10x) than the previous nfs. This is measured as the time it > takes for the application to start, rather than anything low-level. > > It feels to me that it's the same kind of thing that's been mentioned here > before, but it doesn't seem to have been answered. > > Should we expect this behaviour? > Is there anything we can do to improve things? > > We're on 3.1.1, haven't done much in the way of tuning from defaults. > > Cheers > David > > -- > David Lloyd > V Consultants > www.v-consultants.co.uk > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >