Gluster users - Oct 2009 - 3-layer structure and the bonnie rewrite problem

Hello list

First of all: Good work and thanks for GlusterFS!

I'm totally new to GlusterFS, but i like it a lot and think about
migrating my NFS setup completely to GlusterFS. But i ran into some
problems with my chosen structure. Hopefully someone can help out.

The first questions: i ran into some performance issues with a certain
structure/setup and like to know (before i continue testing) wheter this
is a valid approach or simply "dont do". In my tests (dd and bonnie++)
the targeted setup (3-layer, below) shows a realy poor performance
(especially with the bonnie-rewrite-test, already discussed in this
list, but not in this context, i think). So: does this approach
(3-layer) does make any sense with GlusterFS ?


Here a draft of the structure:

Config1 (regular):
***********
[Client-Node]
      ^
      |
  GlusterFS
      |
      v
[Storage1 <-replicate-> Storage2]
***********

Config2 (3-layer):
***********
[Client-Node]
      ^
      |
  GlusterFS
      |
      v
[Re-Exporter/Proxy-Node]
      ^
      |
  GlusterFS
      |
      v
[Storage1 <-replicate-> Storage2, Storage3]
***********


The "Config2" is the targeted structure. I do require the exporter
layer
as a proxy to keep the vol-files on the clients substituable and simple
and by this - most important ;) - to reduce my administrative effort.
Imagine the following:
* Having 3 webserver-client-nodes, on two of them multiple websites, on
third a single huge website with huge IO and space requirements.
* on exporter i do assemble the storage
  /srv/web mounts storage[1,2]:/srv/web
  /srv/web/client3 mounts storage3:/srv/web/client3
* the resulting space from exporter's point of view looks like:
  /srv/web/client1 (from storage[1,2])
  /srv/web/client2 (from storage[1,2])
  /srv/web/client3 (from storage3)
* all clients do mount:
  exporter:/srv/web/<hostname>
Actually it's a little more complex but i hope this shows what i would
like to have.

I run debian-lenny with 2.6.26-bpo.1-xen-686-kernel, tried both
GlusterFS 2.0.4 and 2.0.6 on FUSE 2.7.4. I can provide straces, the
configs i recently used (tried a lot already), bonnie-results and a
build-dump for GlusterFS (created as debian packages).


Another question: Compared to NFS, even in the regular-structure
(above), the bonnie-rewrite tests showed a poor performance. NFS is
about 2-3 times faster wheras most of the other tests (throughput and
such) showed far better results for GlusterFS. I read the threads on
this mailing list about the "bonnie-problem", but they didnt finish
with
any solution, so I'd like to know wheter there is any news there. Again,
i could provide straces, configs, bonnie-results, ...


Thanks
Greets from Berlin
Ulrich

Hello list

First of all: Good work and thanks for GlusterFS!

I'm totally new to GlusterFS, but i like it a lot and think about
migrating my NFS setup completely to GlusterFS. But i ran into some
problems with my chosen structure. Hopefully someone can help out.

The first questions: i ran into some performance issues with a certain
structure/setup and like to know (before i continue testing) wheter this
is a valid approach or simply "dont do". In my tests (dd and bonnie++)
the targeted setup (3-layer, below) shows a realy poor performance
(especially with the bonnie-rewrite-test, already discussed in this
list, but not in this context, i think). So: does this approach
(3-layer) does make any sense with GlusterFS ?


Here a draft of the structure:

Config1 (regular):
***********
[Client-Node]
      ^
      |
  GlusterFS
      |
      v
[Storage1 <-replicate-> Storage2]
***********

Config2 (3-layer):
***********
[Client-Node]
      ^
      |
  GlusterFS
      |
      v
[Re-Exporter/Proxy-Node]
      ^
      |
  GlusterFS
      |
      v
[Storage1 <-replicate-> Storage2, Storage3]
***********


The "Config2" is the targeted structure. I do require the exporter
layer
as a proxy to keep the vol-files on the clients substituable and simple
and by this - most important ;) - to reduce my administrative effort.
Imagine the following:
* Having 3 webserver-client-nodes, on two of them multiple websites, on
third a single huge website with huge IO and space requirements.
* on exporter i do assemble the storage
  /srv/web mounts storage[1,2]:/srv/web
  /srv/web/client3 mounts storage3:/srv/web/client3
* the resulting space from exporter's point of view looks like:
  /srv/web/client1 (from storage[1,2])
  /srv/web/client2 (from storage[1,2])
  /srv/web/client3 (from storage3)
* all clients do mount:
  exporter:/srv/web/<hostname>
Actually it's a little more complex but i hope this shows what i would
like to have.

I run debian-lenny with 2.6.26-bpo.1-xen-686-kernel, tried both
GlusterFS 2.0.4 and 2.0.6 on FUSE 2.7.4. I can provide straces, the
configs i recently used (tried a lot already), bonnie-results and a
build-dump for GlusterFS (created as debian packages).


Another question: Compared to NFS, even in the regular-structure
(above), the bonnie-rewrite tests showed a poor performance. NFS is
about 2-3 times faster wheras most of the other tests (throughput and
such) showed far better results for GlusterFS. I read the threads on
this mailing list about the "bonnie-problem", but they didnt finish
with
any solution, so I'd like to know wheter there is any news there. Again,
i could provide straces, configs, bonnie-results, ...


Thanks
Greets from Berlin
Ulrich

(sent again, googlemail != gmail when sending to a mailing list)