Thank you for the reply, but this kind of config change nothing concerning
the replication transfer rate.
Note that glusterfs cpu usage on all box takes less than 3% of CPU.
All the box can write data with a transfer rate at 800MB/s, but i'm still
blocked at 30MB/s
Any help/suggestion is welcome
Joel
On Thu, Oct 29, 2009 at 9:04 AM, Christian Marnitz <
christian.marnitz at icesmedia.de> wrote:
> Hi,
>
> take a look to the following performance-features:
>
> ###################################################
> ##### performance
> ###################################################
> volume iot
> type performance/io-threads
> option thread-count 16 # deault is 16
> option min-threads 32
> option max-threads 256
> subvolumes mega-replicator
> end-volume
>
> ##### writebehind - write aggregation
> volume wb
> type performance/write-behind
> option cache-size 3MB # default is equal to aggregate-size
> option flush-behind on # default is 'off'
> subvolumes iot
> end-volume
>
> ##### read-ahead - read aggregation
> volume readahead
> type performance/read-ahead
> #option page-size 1048576 # 128kb is the default option
> option page-count 4 # 2 is default option
> option force-atime-update off # default is off
> subvolumes wb
> end-volume
>
>
> ###### no chaching - disable if need realtime
> volume ioc
> type performance/io-cache
> option cache-size 128MB # default is 32MB
> option page-size 1MB #128KB is default option
> option cache-timeout 1 # default is 1
> subvolumes wb
> end-volume
>
> ---
>
> Here is the info written, what happens behind:
>
>
>
http://gluster.com/community/documentation/index.php/Translators/performance/readahead
>
>
http://gluster.com/community/documentation/index.php/Translators/performance/writebehind
>
>
>
http://gluster.com/community/documentation/index.php/Translators/performance/io-threads
>
>
http://gluster.com/community/documentation/index.php/Translators/performance/io-cache
>
>
> I personally use writebehind and io-threads. It depends on your requests,
> what you need.
>
> Greetings,
> Christian
>
>
>
>
> -----Urspr?ngliche Nachricht-----
> Von: gluster-users-bounces at gluster.org [mailto:
> gluster-users-bounces at gluster.org] Im Auftrag von joel vennin
> Gesendet: Donnerstag, 29. Oktober 2009 07:44
> An: gluster-users at gluster.org
> Betreff: Re: [Gluster-users] Slow Replication over infiniband
>
> Please, some one can give me clue concerning the increase of the transfer
> rate, I ready to hack code if necessary.
>
> Thank you.
>
> On Tue, Oct 27, 2009 at 3:30 PM, joel vennin <joel.vennin at
gmail.com>
> wrote:
>
> > Hi, I've a setup composed with 3 nodes(zodiac1, zodiac2, zodiac3)
> >
> > Node config: 8 cores W5580 at 3.20GHz, 76 Go RAM, 40 To, infiniband
> > mlx4_0 (20 Gb) Gluster Version: 2.0.7 Kernel Linux 2.6.30 / Fuse 2.7.4
> >
> >
> > Config file:
> >
> > #----------------------------------------------------------
> > # SERVER SIDE
> > volume posix-1
> > type storage/posix
> > option directory /data/yacht-data2 end-volume
> >
> > volume locks-1
> > type features/locks
> > subvolumes posix-1
> > end-volume
> >
> >
> > volume brick-1
> > type performance/io-threads
> > option thread-count 4
> > subvolumes locks-1
> > end-volume
> >
> > volume server
> > type protocol/server
> > option transport-type ib-verbs
> > option transport.ib-verbs.device-name mlx4_0
> > option auth.addr.brick-1.allow *
> > subvolumes brick-1 #brick-2
> > end-volume
> >
> >
> > # CLIENT NODE DECLARATION
> >
> > #
> > # ZODIAC 1 CLIENT
> > volume zbrick1
> > type protocol/client
> > option transport-type ib-verbs
> > option remote-host 192.168.3.200
> > option remote-subvolume brick-1 end-volume
> >
> > # ZODIAC 2
> > volume zbrick2
> > type protocol/client
> > option transport-type ib-verbs
> > option remote-host 192.168.3.201
> > option remote-subvolume brick-1 end-volume
> >
> > # ZODIAC 3
> > volume zbrick3
> > type protocol/client
> > option transport-type ib-verbs
> > option remote-host 192.168.3.202
> > option remote-subvolume brick-1 end-volume
> >
> > # MEGA REPLICATE
> > volume mega-replicator
> > type cluster/replicate
> > subvolumes zbrick1 zbrick2 zbrick3 end-volume
> > #----------------------------------------------------------
> >
> > Command on each server to start glusterfs:
> > glusterfs -f myreplica.vol /mnt/data/
> >
> >
> >
> > The scenario that I use:
> > zodiac1 and zodiac2 are correctly synchronized. Once every thing
> > is ok for zodiac1 and zodiac2, I decide to start the zodiac3. I order
> > to synchronize it, I do on the box ls -aLR /mnt/data. So the
> > replication start but the transfer rate is really slow: around 30 MB/s
> > !
> >
> > Have you an idea how can I increase this rate ?
> >
> >
> > Thank you.
> >
> >
> >
> >
> >
>
>