Hi all, I know my hardware is not high-performance. But during my gluster-tests there are some remarkable results... My node are: node1: Home-brewn home-server based on an atom. node2: Old heater named Pentium4 node3: Turion64-based laptop The first have their disks via sata and are connected via Gigabit-Ethernet, the laptop has only pata and 100MBit. I did run dbench with the same parameters but varying number of clients on the raw disks of the bricks and also on the fuse-mounted volume in its different stages of 1) two nodes, two bricks, 2) two nodes, four bricks and 3) three nodes, 6bricks. The results of Throughput and Maximum Latency are graphed in the attached pdf[*]. The third node was only used with the three-node-setup. I find it remarkable that the local disk is faster by a factor of two in throughput and faster by a factor of ten(!) in latency. The first I could explain as the write-access has to go through the network and to the disk and might thus get a performance-penalty on this old, single-core system. But the high latency is very strange... I didn't do any special optimization, mounted all the brick-partitions as ext4 with defaults,user_xattr,noatime and there was nothing else going on on the cpus and network apart from normal desktop and background-music. Did I do something wrong, did I miss some optimization or is this to be a expected performance? Thanks for your input, Arnold [*] I hope that makes it through to the list and doesn't offend because of its size. -------------- next part -------------- A non-text attachment was scrubbed... Name: bigreplication.pdf Type: application/pdf Size: 45792 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120213/a50096be/attachment.pdf> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120213/a50096be/attachment.sig>
On Mon, Feb 13, 2012 at 12:37:10AM +0100, Arnold Krille wrote:> 1) two nodes, two bricks, 2) two nodes, four bricks and 3) three > nodes, 6bricks.Are these distributed only, or replicated? If I understand it right (and I'm quite new to gluster myself), whenever you do a read on a replicated volume, gluster dispatches the operation to both nodes, waits for both results to come back, and checks they are the same (and if not, works out which is wrong and kicks off a self-heal operation) http://www.youtube.com/watch?v=AsgtE7Ph2_k And of course, writes have to be dispatched to both nodes, and won't complete until the slowest has finished. This may be the reason for your poor latency. I don't know how dbench operates - is it entirely single-threaded or does it attempt to run concurrent operations to simulate concurrent clients? I would except to get higher throughput in the latter case.> I find it remarkable that the local disk is faster by a factor of two in > throughput and faster by a factor of ten(!) in latency.As well as trying non-replicated gluster, I think you should try NFS as a baseline for what dbench might be able to achieve on a network-attached filesystem. As Scotty used to say, "ya canna change the laws of physics", or something like that. Regards, Brian.
Hi, thanks for your answer and sorry for the probably broken thread. My Kmail2 seems to contain some bugs regarding mail-headers when using filtering... On 00:00:00 you wrote:> On Mon, Feb 13, 2012 at 12:37:10AM +0100, Arnold Krille wrote: > > 1) two nodes, two bricks, 2) two nodes, four bricks and 3) three > > nodes, 6bricks. > > Are these distributed only, or replicated?Two bricks: replicated. Four and six bricks: replicated+distributed.> If I understand it right (and I'm quite new to gluster myself), whenever you > do a read on a replicated volume, gluster dispatches the operation to both > nodes, waits for both results to come back, and checks they are the same > (and if not, works out which is wrong and kicks off a self-heal operation) > > http://www.youtube.com/watch?v=AsgtE7Ph2_k > > And of course, writes have to be dispatched to both nodes, and won't > complete until the slowest has finished. This may be the reason for your > poor latency.I understand that writes have to happen on all (running) replicas and only return when the last finished (like the C-protocol with drbd). But reads can (and should) happen from the nearest only. Or from the fastest. With two nodes you can't decide which node has the 'true' data except to check for the attributes. But for content reading you would have to have three replica or checksums...> I don't know how dbench operates - is it entirely single-threaded or does it > attempt to run concurrent operations to simulate concurrent clients? I > would except to get higher throughput in the latter case.dbench is the filesystem-part of a samba-server with N clients doing typical windows-users stuff. So dbench with one client is single-threaded (actually single-processed) afaik. So running with more simulated clients is multiple processes accessing lots of files for lots of operations. A far better indicator of fs-performance then a simple dd on a big file...> > I find it remarkable that the local disk is faster by a factor of two in > > throughput and faster by a factor of ten(!) in latency. > As well as trying non-replicated gluster, I think you should try NFS as a > baseline for what dbench might be able to achieve on a network-attached > filesystem.NFS on these nodes is limited by the Gigabit-Network-Performance and the disk and results in min(120MBit, ~100MBit) from network and disk. But I will run dbench on the nfs shares (without gluster) this evening.> As Scotty used to say, "ya canna change the laws of physics", or something > like that.Thats why I did the local-disk tests for the same graph. And Scotty is actually the guy who did bent the laws of physics quite a lot:-P Have fun, Arnold -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120213/a64d9038/attachment.sig>