Wipe_Out
2012-Apr-30 17:39 UTC
[Gluster-users] Some benchmarks for anyone that's interested..
Submitting again without the attachment because the list moderators don't
appear to be clearing the queue.. So I pasted the CSV below..
---------- Forwarded message ----------
From: Wipe_Out <wipe_out at users.sourceforge.net>
Date: 29 April 2012 19:35
Subject: Some benchmarks for anyone that's interested..
To: gluster-users at gluster.org
Hi all,
Here are some benchmarks I have just done in a small 2 node cluster with 3
x 500GB drives.. OS on drive 1 and the other two used for testing..
Interconnect is 1Gb Ethernet..
Tests are:
Single Disk (direct, no gluster)
Gluster Replicated
Gluster Striped Replicated
Gluster Distributed Replicated
Gluster Stripe
Glusterfs version is 3.3 Beta3..
Simple test with bonnie++ with 8GB size (double the memory of the nodes)..
Composite CSV attached which is probably the easiest way to compare
results..
*Single Disk
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 8G 288 98 76689 11 32637 6 2030 92 91483 5 206.1
2
Latency 29634us 226ms 318ms 95196us 27554us 1017ms
Version 1.96 ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 12687 46 +++++ +++ +++++ +++ 21140 76 +++++ +++ +++++ +++
Latency 113us 652us 494us 100us 23us 51us
1.96,1.96,localhost.localdomain,1,1335711943,8G,,288,98,76689,11,32637,6,2030,92,91483,5,206.1,2,16,,,,,12687,46,+++++,+++,+++++,+++,21140,76,+++++,+++,+++++,+++,29634us,226ms,318ms,95196us,27554us,1017ms,113us,652us,494us,100us,23us,51us
Gluster Replicated
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 8G 12 12 63474 12 21709 5 2522 96 56940 3 157.4 2
Latency 711ms 189ms 385ms 14804us 256ms 1303ms
Version 1.96 ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 336 2 5160 3 452 2 304 2 1548 1 309 1
Latency 1035ms 4804us 4355ms 1440ms 10184us 8349ms
1.96,1.96,localhost.localdomain,1,1335706557,8G,,12,12,63474,12,21709,5,2522,96,56940,3,157.4,2,16,,,,,336,2,5160,3,452,2,304,2,1548,1,309,1,711ms,189ms,385ms,14804us,256ms,1303ms,1035ms,4804us,4355ms,1440ms,10184us,8349ms
Striped Replicated
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 8G 12 11 40659 7 15032 3 1915 75 63220 3 248.0 2
Latency 744ms 1539ms 530ms 178ms 455ms 165ms
Version 1.96 ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 167 1 1103 0 231 0 151 1 807 1 194 0
Latency 1263ms 60415us 1308ms 1194ms 8964us 850ms
1.96,1.96,localhost.localdomain,1,1335709868,8G,,12,11,40659,7,15032,3,1915,75,63220,3,248.0,2,16,,,,,167,1,1103,0,231,0,151,1,807,1,194,0,744ms,1539ms,530ms,178ms,455ms,165ms,1263ms,60415us,1308ms,1194ms,8964us,850ms
Distribute Replicate
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 8G 12 11 56218 11 20631 6 2126 85 55694 3 151.9 1
Latency 785ms 205ms 387ms 196ms 287ms 1277ms
Version 1.96 ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 254 2 963 0 387 1 221 1 1397 1 307 1
Latency 473ms 20351us 198ms 984ms 252ms 487ms
1.96,1.96,localhost.localdomain,1,1335720170,8G,,12,11,56218,11,20631,6,2126,85,55694,3,151.9,1,16,,,,,254,2,963,0,387,1,221,1,1397,1,307,1,785ms,205ms,387ms,196ms,287ms,1277ms,473ms,20351us,198ms,984ms,252ms,487ms
Stripe
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 8G 10 9 48514 9 18294 5 1992 79 50725 3 285.0 3
Latency 1036ms 751ms 441ms 200ms 249ms 252ms
Version 1.96 ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 280 1 2017 1 342 0 265 1 1466 2 269 0
Latency 2082ms 28109us 7048ms 6559ms 9363us 17815ms
1.96,1.96,localhost.localdomain,1,1335721555,8G,,10,9,48514,9,18294,5,1992,79,50725,3,285.0,3,16,,,,,280,1,2017,1,342,0,265,1,1466,2,269,0,1036ms,751ms,441ms,200ms,249ms,252ms,2082ms,28109us,7048ms,6559ms,9363us,17815ms
*
--- CSV Start ---
Description,format_version,bonnie_version,name,concurrency,seed,file_size,io_chunk_size,putc,putc_cpu,put_block,put_block_cpu,rewrite,rewrite_cpu,getc,getc_cpu,get_block,get_block_cpu,seeks,seeks_cpu,num_files,max_size,min_size,num_dirs,file_chunk_size,seq_create,seq_create_cpu,seq_stat,seq_stat_cpu,seq_del,seq_del_cpu,ran_create,ran_create_cpu,ran_stat,ran_stat_cpu,ran_del,ran_del_cpu,putc_latency,put_block_latency,rewrite_latency,getc_latency,get_block_latency,seeks_latency,seq_create_latency,seq_stat_latency,seq_del_latency,ran_create_latency,ran_stat_latency,ran_del_latency
Single
Disk,1.96,1.96,localhost.localdomain,1,1335711943,8G,,288,98,76689,11,32637,6,2030,92,91483,5,206.1,2,16,,,,,12687,46,+++++,+++,+++++,+++,21140,76,+++++,+++,+++++,+++,29634us,226ms,318ms,95196us,27554us,1017ms,113us,652us,494us,100us,23us,51us
Gluster
Replicated,1.96,1.96,localhost.localdomain,1,1335706557,8G,,12,12,63474,12,21709,5,2522,96,56940,3,157.4,2,16,,,,,336,2,5160,3,452,2,304,2,1548,1,309,1,711ms,189ms,385ms,14804us,256ms,1303ms,1035ms,4804us,4355ms,1440ms,10184us,8349ms
Gluster Striped
Replicated,1.96,1.96,localhost.localdomain,1,1335709868,8G,,12,11,40659,7,15032,3,1915,75,63220,3,248.0,2,16,,,,,167,1,1103,0,231,0,151,1,807,1,194,0,744ms,1539ms,530ms,178ms,455ms,165ms,1263ms,60415us,1308ms,1194ms,8964us,850ms
Gluster Distributed
Replicated,1.96,1.96,localhost.localdomain,1,1335720170,8G,,12,11,56218,11,20631,6,2126,85,55694,3,151.9,1,16,,,,,254,2,963,0,387,1,221,1,1397,1,307,1,785ms,205ms,387ms,196ms,287ms,1277ms,473ms,20351us,198ms,984ms,252ms,487ms
Gluster
Stripe,1.96,1.96,localhost.localdomain,1,1335721555,8G,,10,9,48514,9,18294,5,1992,79,50725,3,285.0,3,16,,,,,280,1,2017,1,342,0,265,1,1466,2,269,0,1036ms,751ms,441ms,200ms,249ms,252ms,2082ms,28109us,7048ms,6559ms,9363us,17815ms
--- CSV End ---
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120430/d95952c7/attachment.html>
lejeczek
2012-May-02 08:52 UTC
[Gluster-users] Some benchmarks for anyone that's interested..
thanks for posting I'd be curious to see what kind of disproportion you get between: raw fs / single brick volume with local fuse mountpoint which effectively points back to the same raw fs from my quick tests I saw massive gap between the two thanks On 30/04/12 18:39, Wipe_Out wrote:> Submitting again without the attachment because the list > moderators don't appear to be clearing the queue.. So I > pasted the CSV below.. > > ---------- Forwarded message ---------- > From: *Wipe_Out* <wipe_out at users.sourceforge.net > <mailto:wipe_out at users.sourceforge.net>> > Date: 29 April 2012 19:35 > Subject: Some benchmarks for anyone that's interested.. > To: gluster-users at gluster.org > <mailto:gluster-users at gluster.org> > > > Hi all, > > Here are some benchmarks I have just done in a small 2 > node cluster with 3 x 500GB drives.. OS on drive 1 and the > other two used for testing.. Interconnect is 1Gb Ethernet.. > > Tests are: > Single Disk (direct, no gluster) > Gluster Replicated > Gluster Striped Replicated > Gluster Distributed Replicated > Gluster Stripe > > Glusterfs version is 3.3 Beta3.. > > Simple test with bonnie++ with 8GB size (double the memory > of the nodes).. Composite CSV attached which is probably > the easiest way to compare results.. > > *Single Disk > Version 1.96 ------Sequential Output------ --Sequential > Input- --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > localhost.locald 8G 288 98 76689 11 32637 6 2030 > 92 91483 5 206.1 2 > Latency 29634us 226ms 318ms 95196us 27554us1017ms > Version 1.96 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- > -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 12687 46 +++++ +++ +++++ +++ 21140 76 +++++ +++ +++++ +++ > Latency 113us 652us 494us 100us 23us 51us > 1.96,1.96,localhost.localdomain,1,1335711943,8G,,288,98,76689,11,32637,6,2030,92,91483,5,206.1,2,16,,,,,12687,46,+++++,+++,+++++,+++,21140,76,+++++,+++,+++++,+++,29634us,226ms,318ms,95196us,27554us,1017ms,113us,652us,494us,100us,23us,51us > > Gluster Replicated > Version 1.96 ------Sequential Output------ --Sequential > Input- --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > localhost.locald 8G12 12 63474 12 21709 5 2522 96 > 56940 3 157.4 2 > Latency 711ms 189ms 385ms 14804us 256ms1303ms > Version 1.96 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- > -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 336 2 5160 3 452 2 304 2 1548 1 309 1 > Latency 1035ms4804us4355ms1440ms 10184us8349ms > 1.96,1.96,localhost.localdomain,1,1335706557,8G,,12,12,63474,12,21709,5,2522,96,56940,3,157.4,2,16,,,,,336,2,5160,3,452,2,304,2,1548,1,309,1,711ms,189ms,385ms,14804us,256ms,1303ms,1035ms,4804us,4355ms,1440ms,10184us,8349ms > > Striped Replicated > Version 1.96 ------Sequential Output------ --Sequential > Input- --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > localhost.locald 8G12 11 40659 7 15032 3 1915 75 > 63220 3 248.0 2 > Latency 744ms1539ms 530ms 178ms 455ms 165ms > Version 1.96 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- > -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 167 1 1103 0 231 0 151 1 807 1 194 0 > Latency 1263ms 60415us1308ms1194ms8964us 850ms > 1.96,1.96,localhost.localdomain,1,1335709868,8G,,12,11,40659,7,15032,3,1915,75,63220,3,248.0,2,16,,,,,167,1,1103,0,231,0,151,1,807,1,194,0,744ms,1539ms,530ms,178ms,455ms,165ms,1263ms,60415us,1308ms,1194ms,8964us,850ms > > Distribute Replicate > Version 1.96 ------Sequential Output------ --Sequential > Input- --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > localhost.locald 8G12 11 56218 11 20631 6 2126 85 > 55694 3 151.9 1 > Latency 785ms 205ms 387ms 196ms 287ms1277ms > Version 1.96 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- > -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 254 2 963 0 387 1 221 1 1397 1 307 1 > Latency 473ms 20351us 198ms 984ms 252ms 487ms > 1.96,1.96,localhost.localdomain,1,1335720170,8G,,12,11,56218,11,20631,6,2126,85,55694,3,151.9,1,16,,,,,254,2,963,0,387,1,221,1,1397,1,307,1,785ms,205ms,387ms,196ms,287ms,1277ms,473ms,20351us,198ms,984ms,252ms,487ms > > Stripe > Version 1.96 ------Sequential Output------ --Sequential > Input- --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > localhost.locald 8G10 9 48514 9 18294 5 1992 79 > 50725 3 285.0 3 > Latency 1036ms 751ms 441ms 200ms 249ms 252ms > Version 1.96 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- > -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 280 1 2017 1 342 0 265 1 1466 2 269 0 > Latency 2082ms 28109us7048ms6559ms9363us 17815ms > 1.96,1.96,localhost.localdomain,1,1335721555,8G,,10,9,48514,9,18294,5,1992,79,50725,3,285.0,3,16,,,,,280,1,2017,1,342,0,265,1,1466,2,269,0,1036ms,751ms,441ms,200ms,249ms,252ms,2082ms,28109us,7048ms,6559ms,9363us,17815ms* > > > --- CSV Start --- > Description,format_version,bonnie_version,name,concurrency,seed,file_size,io_chunk_size,putc,putc_cpu,put_block,put_block_cpu,rewrite,rewrite_cpu,getc,getc_cpu,get_block,get_block_cpu,seeks,seeks_cpu,num_files,max_size,min_size,num_dirs,file_chunk_size,seq_create,seq_create_cpu,seq_stat,seq_stat_cpu,seq_del,seq_del_cpu,ran_create,ran_create_cpu,ran_stat,ran_stat_cpu,ran_del,ran_del_cpu,putc_latency,put_block_latency,rewrite_latency,getc_latency,get_block_latency,seeks_latency,seq_create_latency,seq_stat_latency,seq_del_latency,ran_create_latency,ran_stat_latency,ran_del_latency > Single > Disk,1.96,1.96,localhost.localdomain,1,1335711943,8G,,288,98,76689,11,32637,6,2030,92,91483,5,206.1,2,16,,,,,12687,46,+++++,+++,+++++,+++,21140,76,+++++,+++,+++++,+++,29634us,226ms,318ms,95196us,27554us,1017ms,113us,652us,494us,100us,23us,51us > Gluster > Replicated,1.96,1.96,localhost.localdomain,1,1335706557,8G,,12,12,63474,12,21709,5,2522,96,56940,3,157.4,2,16,,,,,336,2,5160,3,452,2,304,2,1548,1,309,1,711ms,189ms,385ms,14804us,256ms,1303ms,1035ms,4804us,4355ms,1440ms,10184us,8349ms > Gluster Striped > Replicated,1.96,1.96,localhost.localdomain,1,1335709868,8G,,12,11,40659,7,15032,3,1915,75,63220,3,248.0,2,16,,,,,167,1,1103,0,231,0,151,1,807,1,194,0,744ms,1539ms,530ms,178ms,455ms,165ms,1263ms,60415us,1308ms,1194ms,8964us,850ms > Gluster Distributed > Replicated,1.96,1.96,localhost.localdomain,1,1335720170,8G,,12,11,56218,11,20631,6,2126,85,55694,3,151.9,1,16,,,,,254,2,963,0,387,1,221,1,1397,1,307,1,785ms,205ms,387ms,196ms,287ms,1277ms,473ms,20351us,198ms,984ms,252ms,487ms > Gluster > Stripe,1.96,1.96,localhost.localdomain,1,1335721555,8G,,10,9,48514,9,18294,5,1992,79,50725,3,285.0,3,16,,,,,280,1,2017,1,342,0,265,1,1466,2,269,0,1036ms,751ms,441ms,200ms,249ms,252ms,2082ms,28109us,7048ms,6559ms,9363us,17815ms > --- CSV End --- > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120502/1eec9169/attachment.html>