Hello people, For those have *GlusterFS running over GigE, what is the throughput you have for write data?* I have 3 nodes, running GNU/Linux Ubuntu Server 10.04, GlusterFS 3.0.2. The network is GigE, dedicated switch. I am experiencing maximum 2 MB/sec running a "dd" over a client mount point. Thank you very much -- Henrique Haas +55 (51) 3028.6602 www.dz6web.com
Wow, that doesn't sound right! I have a gluster cluster at home using old P4 hardware, and cheap soho off the shelf gigabit switches, and I can get ~35MB/s writes with dd: dd if=/dev/zero of=/vol/scratch/foo/test.img bs=1M count=64 64+0 records in 64+0 records out 67108864 bytes (67 MB) copied, 1.76719 s, 38.0 MB/s dd if=/dev/zero of=/vol/scratch/foo/test.img bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 29.3242 s, 36.6 MB/s And that's with no tuning of gigabit, i.e. no jumbo frames or anything, everything is stock, and off the shelf. One of my machines might even be using an old pci 33mhz bus cheap realtek gigabit card. The above numbers were on a straight distribute between two nodes, 1 brick each. On my other volume that's doing replicate between the two nodes, I get about 18MB/s (same hardware as above). Doug On Fri, Sep 10, 2010 at 12:16 PM, Henrique Haas <henrique at dz6web.com> wrote:> Hello people, > > For those have *GlusterFS running over GigE, what is the throughput you have > for write data?* > I have 3 nodes, running GNU/Linux Ubuntu Server 10.04, GlusterFS 3.0.2. > The network is GigE, dedicated switch. > > I am experiencing maximum 2 MB/sec running a "dd" over a client mount point. > > Thank you very much > > -- > Henrique Haas > +55 (51) 3028.6602 > www.dz6web.com > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >-- Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html
Something is bad wrong. I have two servers w/8 SATA drives each setup as replicate/distribute and I'm getting over 30Mbps on my writes (for larger files of course). Nothing special about the machines (2.5Ghz Dual Core, 8Gb memory, SuperMicro 8-port non-RAID SATA controller). Regards, Larry Bates vitalEsafe, Inc. ------------------------------ Message: 5 Date: Fri, 10 Sep 2010 13:16:52 -0300 From: Henrique Haas <henrique at dz6web.com> Subject: [Gluster-users] GlusterFS Performance gigE To: gluster-users at gluster.org Message-ID: <AANLkTi=DdkWx_b0oyQbvbMRCeBN-k2--O4A6YHeiLJbZ at mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Hello people, For those have *GlusterFS running over GigE, what is the throughput you have for write data?* I have 3 nodes, running GNU/Linux Ubuntu Server 10.04, GlusterFS 3.0.2. The network is GigE, dedicated switch. I am experiencing maximum 2 MB/sec running a "dd" over a client mount point. Thank you very much -- Henrique Haas +55 (51) 3028.6602 www.dz6web.com
I agree something is wrong. Here is what I am getting on a more ISP grade setup (but no SAN sadly) also with no Ethernet tuning : dd if=/dev/zero of=/usr/local/sites/test.test bs=1M count=64 64+0 records in 64+0 records out 67108864 bytes (67 MB) copied, 0.754847 seconds, 88.9 MB/s dd if=/dev/zero of=/usr/local/sites/test.test bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 13.9273 seconds, 77.1 MB/s I did some other testing using tiobench, iozone, and dt with similar (albeit MUCH more detailed) results. The setup above is using cluster/replicate between 2 remote bricks (raid10 SAS 15K). Also this was run during a low load window while there was still some light load on the webservers so this would be slightly higher if I could stop our webservers but that isn't a option for a few more days at least. Also we use lsycnd (inotify based) to sync changed files off to remote backup servers to overcome the limits on wirespeed with gluster (our remote nodes are 100Mb) so that has some brick/disk overhead as well. One thing you could do to help troubleshoot is run a speed test and use atop or sar tools to debug and also get a snapshot of the test for review. -- Chris Layton Senior Systems Engineer WorldSpice Technologies chrisl at worldspice.net 901-843-9300 On Fri, 2010-09-10 at 13:16 -0300, Henrique Haas wrote:> Hello people, > > For those have *GlusterFS running over GigE, what is the throughput you have > for write data?* > I have 3 nodes, running GNU/Linux Ubuntu Server 10.04, GlusterFS 3.0.2. > The network is GigE, dedicated switch. > > I am experiencing maximum 2 MB/sec running a "dd" over a client mount point. > > Thank you very much > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users