Gluster users - Apr 2009 - help me saturate my reads with a stripe of 4 machines

I'm having trouble getting stripe to perform as well as I'd expect.
I've managed to get it to work pretty well with 2 machines in a stripe
set with these files:

server.vol:
volume posix
  type storage/posix
  option directory /tmp/gluster
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume io-threads
  type performance/io-threads
  option thread-count 16
  subvolumes locks
end-volume

volume read-ahead
  type performance/read-ahead
  option page-count 16
  subvolumes io-threads
end-volume

volume write-behind
  type performance/write-behind
  option cache-size 1MB
  subvolumes read-ahead
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.write-behind.allow *
  subvolumes write-behind
end-volume

client.vol:
volume machine01
  type protocol/client
  option transport-type tcp
  option remote-host machine01
  option remote-subvolume write-behind
end-volume

volume machine02
  type protocol/client
  option transport-type tcp
  option remote-host machine02
  option remote-subvolume write-behind
end-volume

volume stripe
  type cluster/stripe
  option block-size 1MB
  subvolumes machine01 machine02
end-volume

volume read-ahead
  type performance/read-ahead
  option page-count 16
  subvolumes stripe
end-volume

volume write-behind
  type performance/write-behind
  option cache-size 1MB
  subvolumes read-ahead
end-volume


I can saturate my link when catting a 4gb file to /dev/null at around
112MB/s. I'm having trouble, however, getting this to happen when I've
got 4 machines in the stripe. I can only sustain about a 60-80MB/s
read, though I manage to do a 112MB/s write. Anyone see anything
obvious? Here's my client.vol file, and the server.vol is the same as
above:


volume machine01
  type protocol/client
  option transport-type tcp
  option remote-host machine01
  option remote-subvolume write-behind
end-volume

volume machine02
  type protocol/client
  option transport-type tcp
  option remote-host machine02
  option remote-subvolume write-behind
end-volume

volume machine03
  type protocol/client
  option transport-type tcp
  option remote-host machine03
  option remote-subvolume write-behind
end-volume

volume machine04
  type protocol/client
  option transport-type tcp
  option remote-host machine04
  option remote-subvolume write-behind
end-volume

volume stripe
  type cluster/stripe
  option block-size 1MB
  subvolumes machine01 machine02 machine03 machine04
end-volume

volume read-ahead
  type performance/read-ahead
  option page-count 16
  subvolumes stripe
end-volume

volume write-behind
  type performance/write-behind
  option cache-size 1MB
  subvolumes read-ahead
end-volume

On Wed, Apr 29, 2009 at 6:03 PM, Erick Tryzelaar
<idadesub at users.sourceforge.net> wrote:
> I'm having trouble getting stripe to perform as well as I'd expect.
> I've managed to get it to work pretty well with 2 machines in a stripe
> set with these files:
Some more details. I've found that if I make my stripe set block size
64MB+ or higher, I can saturate my read speed. The smaller the block
size, the lower the total bandwidth used. Since I'm essentially
serializing my reads from each machine, I think I'm hiding the cost
that gluster has when it's merging all the data from the stripes. At
the smaller stripe sizes the cost of merging the data dominates, and
ends up eating half the available bandwidth.