Gluster users - Sep 2009 - Replicate performance (cluster/replicate, performance/io-cache)

Greetings,

I've just started experimenting with glusterfs and am new to this list, so
apologies if this has all been covered before! (a quick browse through the
archives didn't reveal anything though)

I'm attempting to use glusterfs to manage replication between a pair of
geographically separated test servers (communicating over an ssh tunnel).
I realise this isn't exactly an optimal configuration. However, I need
fast read performance but don't particularly care about write performance,
and it doesn't really matter if I'm reading 30-second-old data (my usage
is mainly for webserver replication, so the data won't be changing very
often anyway).

Now, I've set up gluster as per the attached configurations. I believe all
the appropriate performance translators are configured, notably including
the io-cache. The read-subvolume is set to a posix storage on the local
machine.

Still, in spite of the local read-subvolume and caching, successive read
operations on the same (tiny) file, well within the cache-timeout period
always take a minimum time equivalent to four round trips to complete.

Can anyone explain why this is? I understand there may be one transaction
to validate the cache, but what else is going on here? Furthermore, is it
possible to defer/disable this re-validation (within a timeout)
altogether?

Thanks in advance for any help!

Cheers,
Alex.


-- 
Alex Craven
krei at it.net.au
-------------- next part --------------
volume loopback
  type storage/posix
  option directory /cluster/test
end-volume

volume remote-achaemenes
 	type protocol/client
 	option transport-type tcp
 	option remote-host 127.0.0.1
 	option remote-port 6997
 	option remote-subvolume brick
	option username test_cluster
	option password xxxxxxxx
	option transport-timeout 10
end-volume

volume replicate
	type cluster/replicate
	subvolumes  loopback remote-achaemenes
	option read-subvolume loopback
	option lock-node loopback
	### DANGER BELOW! 
        # the following are set temporarily, in an attempt to identify the
        # cause of poor read performance issues
	option self-heal off
	option data-self-heal off
	option metadata-self-heal off
	option entry-self-heal off
	option data-change-log off
	option metadata-change-log off
	option entry-change-log off
	option data-lock-server-count 0
	option entry-lock-server-count 0
end-volume

volume writebehind
	type performance/write-behind
	option window-size 1MB
	option flush-behind on
	subvolumes replicate
end-volume

volume cache
	type performance/io-cache
	option cache-size 256MB
	subvolumes writebehind
	option priority *.php:3,*.html:2,*:1
	option cache-timeout 10
	option force-revalidate-timeout 30
	option force-atime-update off
end-volume
-------------- next part --------------
volume posix
  type storage/posix
  option directory /cluster/test
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.login.brick.allow test_cluster
  option auth.login.test_cluster.password xxxxxxxx
  option listen-port 6996
  option bind-address 127.0.0.1
  subvolumes brick
end-volume

On 09/14/2009 08:27 AM, Alex Craven wrote:
> I've just started experimenting with glusterfs and am new to this list,
so
> apologies if this has all been covered before! (a quick browse through the
> archives didn't reveal anything though)
>
> I'm attempting to use glusterfs to manage replication between a pair of
> geographically separated test servers (communicating over an ssh tunnel).
> I realise this isn't exactly an optimal configuration. However, I need
> fast read performance but don't particularly care about write
performance,
> and it doesn't really matter if I'm reading 30-second-old data (my
usage
> is mainly for webserver replication, so the data won't be changing very
> often anyway).
>
> Now, I've set up gluster as per the attached configurations. I believe
all
> the appropriate performance translators are configured, notably including
> the io-cache. The read-subvolume is set to a posix storage on the local
> machine.
>
> Still, in spite of the local read-subvolume and caching, successive read
> operations on the same (tiny) file, well within the cache-timeout period
> always take a minimum time equivalent to four round trips to complete.
>    
The read subvolume only helps on read(). For every other request, such 
as open(), all of the replicas get involved. I think you are looking for 
the dr sync functionality they list on the roadmap for GlusterFS 2.2.
> Can anyone explain why this is? I understand there may be one transaction
> to validate the cache, but what else is going on here? Furthermore, is it
> possible to defer/disable this re-validation (within a timeout)
> altogether?luster.org/cgi-bin/mailman/listinfo/gluster-users
>    
Not that I know of - and I think there would be many gotchas of doing 
such a thing - however, for certain use cases, it would be kind of cool 
if they could relax the restrictions and/or move operations such as 
consistency checks and self-heal to the background to be run 
asynchronously. For example, if I am doing open(..., O_RDONLY), read(), 
close(), then I may not care about the state of other volumes, as long 
as I see a consistent snapshot that doesn't break half-way through my 
read()s...

Cheers,
mark

-- 
Mark Mielke<mark at mielke.cc>