Hi all, I have read some recent post about performance issues, complaining about the fuse driver and recomended NFS.. although my final goal is replicate volume, I'm now just doing some test for reference. my basic benchmark is dd if=/dev/zero of=ddtest bs=1M count=1000 conv=fsync running directly on server xfs partition give me 120-130 MB/s .. OK on client NFS mount using _KERNEL_ nfs server give me 100-112MB/s .. OK but a pure "nfs replacement" gluster config (one brick distribute) give me sometimes 112 and sometimes only 64 !!! any idea why so unstable? I was watching top,iotop and iftop .. and no idea, the resources were free but the real problem comes with my second benchmark, untar the kernel sources tar xjf /usr/portage/distfiles/linux-3.2.tar.bz2 on server localy .. 16sec kernel nfs mount .. 1:30 gluster nfs mount .. 23min !!! what??? is this the small files issue? so much? even on nfs? my last quick test with gluster native (fuse) mount, give me 46MB/s for dd (really slow for non replicate) and 3:58 for the sources untar (not the best, but much much better than gluster nfs) I have done no tunning at all yet. But I thought I will get much better numbers for the one brick distribute. And expected tuning with the replica.. server is CPU E5-1650 v3 @ 3.50GHz with 4GB RAM, 7200rpm hdd client is old i5 661 @ 3.33GHz, 4GB RAM gigabit ethernet glusterfs-3.7.4 XFS filesystem on brick partition Pavel
On 29/02/16 15:25, Pavel Riha wrote:> Hi all, > > I have read some recent post about performance issues, complaining > about the fuse driver and recomended NFS.. > > although my final goal is replicate volume, I'm now just doing some > test for reference. > > my basic benchmark is > dd if=/dev/zero of=ddtest bs=1M count=1000 conv=fsync > > running directly on server xfs partition give me 120-130 MB/s .. OK > > on client NFS mount using _KERNEL_ nfs server give me 100-112MB/s .. OK > > but a pure "nfs replacement" gluster config (one brick distribute) > give me sometimes 112 and sometimes only 64 !!! any idea why so > unstable? > I was watching top,iotop and iftop .. and no idea, the resources were > free > > but the real problem comes with my second benchmark, untar the kernel > sources > tar xjf /usr/portage/distfiles/linux-3.2.tar.bz2 > > on server localy .. 16sec > kernel nfs mount .. 1:30 > gluster nfs mount .. 23min !!! what??? > > is this the small files issue? so much? even on nfs? > > > my last quick test with gluster native (fuse) mount, give me > 46MB/s for dd (really slow for non replicate) and > 3:58 for the sources untar (not the best, but much much better than > gluster nfs) > > > I have done no tunning at all yet. But I thought I will get much > better numbers for the one brick distribute. And expected tuning with > the replica.. > > > server is CPU E5-1650 v3 @ 3.50GHz with 4GB RAM, 7200rpm hdd > client is old i5 661 @ 3.33GHz, 4GB RAM > gigabit ethernet > glusterfs-3.7.4 > XFS filesystem on brick partition >Can u provide more details about volume configuration (gluster vol info) ?> > > > Pavel > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Pavel Riha wrote on 29/02/2016 09:55:> > I have read some recent post about performance issues, complaining > about the fuse driver and recomended NFS.. > > although my final goal is replicate volume, I'm now just doing some > test for reference. > > my basic benchmark is dd if=/dev/zero of=ddtest bs=1M count=1000 > conv=fsync > > running directly on server xfs partition give me 120-130 MB/s .. OK > > on client NFS mount using _KERNEL_ nfs server give me 100-112MB/s .. > OK > > but a pure "nfs replacement" gluster config (one brick distribute) > give me sometimes 112 and sometimes only 64 !!! any idea why so > unstable? I was watching top,iotop and iftop .. and no idea, the > resources were free > > but the real problem comes with my second benchmark, untar the kernel > sources tar xjf /usr/portage/distfiles/linux-3.2.tar.bz2 > > on server localy .. 16sec kernel nfs mount .. 1:30 gluster nfs > mount .. 23min !!! what??? > > is this the small files issue? so much? even on nfs? > > my last quick test with gluster native (fuse) mount, give me 46MB/s > for dd (really slow for non replicate) and 3:58 for the sources untar > (not the best, but much much better than gluster nfs) > > I have done no tunning at all yet. But I thought I will get much > better numbers for the one brick distribute. And expected tuning with > the replica.. > > server is CPU E5-1650 v3 @ 3.50GHz with 4GB RAM, 7200rpm hdd client > is old i5 661 @ 3.33GHz, 4GB RAM gigabit ethernet glusterfs-3.7.4 XFS > filesystem on brick partitionJust a quick "me too". Not sure what if Using a 2-brick replica over 2x bonded (round robin) 1Gb ethernet. Mount is on one of the brick nodes. A kernel un-tar seems to really stress Gluster. Extraction to Gluster NFS mount: # time tar xf /tmp/linux-4.4.tar real 13m46.308s user 0m1.224s sys 0m19.608s Extraction to Gluster FUSE mount: # time tar xf /tmp/linux-4.4.tar real 9m10.413s user 0m1.964s sys 0m16.740s Extraction to local hardware-RAID backed FS (where the bricks are): # time tar xf /tmp/linux-4.4.tar real 0m2.771s user 0m0.284s sys 0m2.396s Ronny -- Ronny Adsetts Technical Director Amazing Internet Ltd, London t: +44 20 8977 8943 w: www.amazinginternet.com Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ Registered in England. Company No. 4042957 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: OpenPGP digital signature URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160303/3ac16e60/attachment.sig>