----- Original Message -----> From: "Joe Julian" <joe at julianfamily.org> > To: "Punit Dambiwal" <hypunit at gmail.com>, gluster-users at gluster.org, "Humble Devassy Chirammal" > <humble.devassy at gmail.com> > Sent: Monday, February 16, 2015 3:32:31 PM > Subject: Re: [Gluster-users] Gluster performance on the small files > > > On 02/12/2015 10:58 PM, Punit Dambiwal wrote: > > > > Hi, > > I have seen the gluster performance is dead slow on the small files...even i > am using the SSD....it's too bad performance....even i am getting better > performance in my SAN with normal SATA disk... > > I am using distributed replicated glusterfs with replica count=2...i have all > SSD disks on the brick... > > > > root at vm3:~# dd bs=64k count=4k if=/dev/zero of=test oflag=dsync > > 4096+0 records in > > 4096+0 records out > > 268435456 bytes (268 MB) copied, 57.3145 s, 4.7 MB/s >This seems pretty slow, even if you are using gigabit. Here is what I get: [root at gqac031 smallfile]# dd bs=64k count=4k if=/dev/zero of=/gluster-emptyvol/test oflag=dsync 4096+0 records in 4096+0 records out 268435456 bytes (268 MB) copied, 10.5965 s, 25.3 MB/s FYI this is on my 2 node pure replica + spinning disks(RAID 6, this is not setup for smallfile workloads. For smallfile I normally use RAID 10) + 10G. The single threaded DD process is defiantly a bottle neck here, the power in distributed systems is doing things in parallel across clients / threads. You may want to try smallfile: http://www.gluster.org/community/documentation/index.php/Performance_Testing Smallfile command used - python /small-files/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 64 --files 10000 --top /gluster-emptyvol/ --pause 1000 --host-set "client1, client2" total threads = 16 total files = 157100 total data = 9.589 GB 98.19% of requested files processed, minimum is 70.00 41.271602 sec elapsed time 3806.491454 files/sec 3806.491454 IOPS 237.905716 MB/sec If you wanted to do something similar with DD you could do: <my script> for i in `seq 1..4` do dd bs=64k count=4k if=/dev/zero of=/gluster-emptyvol/test$i oflag=dsync & done for pid in $(pidof dd); do while kill -0 "$pid"; do sleep 0.1 done done # time myscript.sh Then do the math to figure out the MB / sec of the system. -b> > > root at vm3:~# dd bs=64k count=4k if=/dev/zero of=test conv=fdatasync > > 4096+0 records in > > 4096+0 records out > > 268435456 bytes (268 MB) copied, 1.80093 s, 149 MB/s > > > > How small is your VM image? The image is the file that GlusterFS is serving, > not the small files within it. Perhaps the filesystem you're using within > your VM is inefficient with regard to how it handles disk writes. > > I believe your concept of "small file" performance is misunderstood, as is > often the case with this phrase. The "small file" issue has to do with the > overhead of finding and checking the validity of any file, but with a small > file the percentage of time doing those checks is proportionally greater. > With your VM image, that file is already open. There are no self-heal checks > or lookups that are happening in your tests, so that overhead is not the > problem. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Punit Dambiwal
2015-Feb-17 02:03 UTC
[Gluster-users] Gluster performance on the small files
Hi Vijay, Please find the volume info here :- [root at cpu01 ~]# gluster volume info Volume Name: ds01 Type: Distributed-Replicate Volume ID: 369d3fdc-c8eb-46b7-a33e-0a49f2451ff6 Status: Started Number of Bricks: 48 x 2 = 96 Transport-type: tcp Bricks: Brick1: cpu01:/bricks/1/vol1 Brick2: cpu02:/bricks/1/vol1 Brick3: cpu03:/bricks/1/vol1 Brick4: cpu04:/bricks/1/vol1 Brick5: cpu01:/bricks/2/vol1 Brick6: cpu02:/bricks/2/vol1 Brick7: cpu03:/bricks/2/vol1 Brick8: cpu04:/bricks/2/vol1 Brick9: cpu01:/bricks/3/vol1 Brick10: cpu02:/bricks/3/vol1 Brick11: cpu03:/bricks/3/vol1 Brick12: cpu04:/bricks/3/vol1 Brick13: cpu01:/bricks/4/vol1 Brick14: cpu02:/bricks/4/vol1 Brick15: cpu03:/bricks/4/vol1 Brick16: cpu04:/bricks/4/vol1 Brick17: cpu01:/bricks/5/vol1 Brick18: cpu02:/bricks/5/vol1 Brick19: cpu03:/bricks/5/vol1 Brick20: cpu04:/bricks/5/vol1 Brick21: cpu01:/bricks/6/vol1 Brick22: cpu02:/bricks/6/vol1 Brick23: cpu03:/bricks/6/vol1 Brick24: cpu04:/bricks/6/vol1 Brick25: cpu01:/bricks/7/vol1 Brick26: cpu02:/bricks/7/vol1 Brick27: cpu03:/bricks/7/vol1 Brick28: cpu04:/bricks/7/vol1 Brick29: cpu01:/bricks/8/vol1 Brick30: cpu02:/bricks/8/vol1 Brick31: cpu03:/bricks/8/vol1 Brick32: cpu04:/bricks/8/vol1 Brick33: cpu01:/bricks/9/vol1 Brick34: cpu02:/bricks/9/vol1 Brick35: cpu03:/bricks/9/vol1 Brick36: cpu04:/bricks/9/vol1 Brick37: cpu01:/bricks/10/vol1 Brick38: cpu02:/bricks/10/vol1 Brick39: cpu03:/bricks/10/vol1 Brick40: cpu04:/bricks/10/vol1 Brick41: cpu01:/bricks/11/vol1 Brick42: cpu02:/bricks/11/vol1 Brick43: cpu03:/bricks/11/vol1 Brick44: cpu04:/bricks/11/vol1 Brick45: cpu01:/bricks/12/vol1 Brick46: cpu02:/bricks/12/vol1 Brick47: cpu03:/bricks/12/vol1 Brick48: cpu04:/bricks/12/vol1 Brick49: cpu01:/bricks/13/vol1 Brick50: cpu02:/bricks/13/vol1 Brick51: cpu03:/bricks/13/vol1 Brick52: cpu04:/bricks/13/vol1 Brick53: cpu01:/bricks/14/vol1 Brick54: cpu02:/bricks/14/vol1 Brick55: cpu03:/bricks/14/vol1 Brick56: cpu04:/bricks/14/vol1 Brick57: cpu01:/bricks/15/vol1 Brick58: cpu02:/bricks/15/vol1 Brick59: cpu03:/bricks/15/vol1 Brick60: cpu04:/bricks/15/vol1 Brick61: cpu01:/bricks/16/vol1 Brick62: cpu02:/bricks/16/vol1 Brick63: cpu03:/bricks/16/vol1 Brick64: cpu04:/bricks/16/vol1 Brick65: cpu01:/bricks/17/vol1 Brick66: cpu02:/bricks/17/vol1 Brick67: cpu03:/bricks/17/vol1 Brick68: cpu04:/bricks/17/vol1 Brick69: cpu01:/bricks/18/vol1 Brick70: cpu02:/bricks/18/vol1 Brick71: cpu03:/bricks/18/vol1 Brick72: cpu04:/bricks/18/vol1 Brick73: cpu01:/bricks/19/vol1 Brick74: cpu02:/bricks/19/vol1 Brick75: cpu03:/bricks/19/vol1 Brick76: cpu04:/bricks/19/vol1 Brick77: cpu01:/bricks/20/vol1 Brick78: cpu02:/bricks/20/vol1 Brick79: cpu03:/bricks/20/vol1 Brick80: cpu04:/bricks/20/vol1 Brick81: cpu01:/bricks/21/vol1 Brick82: cpu02:/bricks/21/vol1 Brick83: cpu03:/bricks/21/vol1 Brick84: cpu04:/bricks/21/vol1 Brick85: cpu01:/bricks/22/vol1 Brick86: cpu02:/bricks/22/vol1 Brick87: cpu03:/bricks/22/vol1 Brick88: cpu04:/bricks/22/vol1 Brick89: cpu01:/bricks/23/vol1 Brick90: cpu02:/bricks/23/vol1 Brick91: cpu03:/bricks/23/vol1 Brick92: cpu04:/bricks/23/vol1 Brick93: cpu01:/bricks/24/vol1 Brick94: cpu02:/bricks/24/vol1 Brick95: cpu03:/bricks/24/vol1 Brick96: cpu04:/bricks/24/vol1 Options Reconfigured: nfs.disable: off user.cifs: enable auth.allow: 10.10.0.* performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 server.allow-insecure: on [root at cpu01 ~]# Thanks, punit On Tue, Feb 17, 2015 at 6:16 AM, Ben Turner <bturner at redhat.com> wrote:> ----- Original Message ----- > > From: "Joe Julian" <joe at julianfamily.org> > > To: "Punit Dambiwal" <hypunit at gmail.com>, gluster-users at gluster.org, > "Humble Devassy Chirammal" > > <humble.devassy at gmail.com> > > Sent: Monday, February 16, 2015 3:32:31 PM > > Subject: Re: [Gluster-users] Gluster performance on the small files > > > > > > On 02/12/2015 10:58 PM, Punit Dambiwal wrote: > > > > > > > > Hi, > > > > I have seen the gluster performance is dead slow on the small > files...even i > > am using the SSD....it's too bad performance....even i am getting better > > performance in my SAN with normal SATA disk... > > > > I am using distributed replicated glusterfs with replica count=2...i > have all > > SSD disks on the brick... > > > > > > > > root at vm3:~# dd bs=64k count=4k if=/dev/zero of=test oflag=dsync > > > > 4096+0 records in > > > > 4096+0 records out > > > > 268435456 bytes (268 MB) copied, 57.3145 s, 4.7 MB/s > > > > This seems pretty slow, even if you are using gigabit. Here is what I get: > > [root at gqac031 smallfile]# dd bs=64k count=4k if=/dev/zero > of=/gluster-emptyvol/test oflag=dsync > 4096+0 records in > 4096+0 records out > 268435456 bytes (268 MB) copied, 10.5965 s, 25.3 MB/s > > FYI this is on my 2 node pure replica + spinning disks(RAID 6, this is not > setup for smallfile workloads. For smallfile I normally use RAID 10) + 10G. > > The single threaded DD process is defiantly a bottle neck here, the power > in distributed systems is doing things in parallel across clients / > threads. You may want to try smallfile: > > > http://www.gluster.org/community/documentation/index.php/Performance_Testing > > Smallfile command used - python /small-files/smallfile/smallfile_cli.py > --operation create --threads 8 --file-size 64 --files 10000 --top > /gluster-emptyvol/ --pause 1000 --host-set "client1, client2" > > total threads = 16 > total files = 157100 > total data = 9.589 GB > 98.19% of requested files processed, minimum is 70.00 > 41.271602 sec elapsed time > 3806.491454 files/sec > 3806.491454 IOPS > 237.905716 MB/sec > > If you wanted to do something similar with DD you could do: > > <my script> > for i in `seq 1..4` > do > dd bs=64k count=4k if=/dev/zero of=/gluster-emptyvol/test$i > oflag=dsync & > done > for pid in $(pidof dd); do > while kill -0 "$pid"; do > sleep 0.1 > done > done > > # time myscript.sh > > Then do the math to figure out the MB / sec of the system. > > -b > > > > > > > root at vm3:~# dd bs=64k count=4k if=/dev/zero of=test conv=fdatasync > > > > 4096+0 records in > > > > 4096+0 records out > > > > 268435456 bytes (268 MB) copied, 1.80093 s, 149 MB/s > > > > > > > > How small is your VM image? The image is the file that GlusterFS is > serving, > > not the small files within it. Perhaps the filesystem you're using within > > your VM is inefficient with regard to how it handles disk writes. > > > > I believe your concept of "small file" performance is misunderstood, as > is > > often the case with this phrase. The "small file" issue has to do with > the > > overhead of finding and checking the validity of any file, but with a > small > > file the percentage of time doing those checks is proportionally greater. > > With your VM image, that file is already open. There are no self-heal > checks > > or lookups that are happening in your tests, so that overhead is not the > > problem. > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150217/27304c1f/attachment.html>