Gluster users - Jul 2010 - Ext4 safe for production use with gluster?

As part of my current research for upcoming setup, I started reading
up on filesystems (was actually looking up fencing which leds to node
crash and file integrity on different fs) to see if there would be any
significant improvement to using a different filesystem for the
glusterfs storage nodes.

Initially, the reading suggested that ext4 would provide a significant
performance boost. Quite possibly since it delays write longer,
latencies of the network storage is hidden even better especially for
temporary files.

After subsequent reading, it seems that the cost of the massive delay
allocation that could blow up really bad in the event the system
crash.

However, this may also be limited to poorly written applications, i.e.
since some massive arguments last year over Firefox 3.0, sqlite and
ext4 problems, ext4 now detects situations where it should flush to
disk immediately to avoid zero length files on crash.

Assuming I'm using the most likely disk writing applications I'm using
is KVM, exim, dovecot, mysql and postgresql, would ext4 be safe for
production use or should I stick to ext3?

On 07/07/2010 08:37 AM, Emmanuel Noobadmin wrote:
> Initially, the reading suggested that ext4 would provide a significant
> performance boost. Quite possibly since it delays write longer,
> latencies of the network storage is hidden even better especially for
> temporary files.
>
> After subsequent reading, it seems that the cost of the massive delay
> allocation that could blow up really bad in the event the system
> crash.
> Assuming I'm using the most likely disk writing applications I'm
using
> is KVM, exim, dovecot, mysql and postgresql, would ext4 be safe for
> production use or should I stick to ext3?
We set up a small test case in our environment to test Gluster / ext4 in 
a simple 4-node client-replication setup.  After running it through the 
regular Bonnie / IOZone / FFSB tests, we determined that it _worked_, 
but that compared to ext3, we saw some strange timing results overall 
(wierd lag spikes, etc).  Unfortunately the project was scrapped early 
on (for external reasons), and no further investigation was done.  YMMV.

If you do go ahead and put together a test suite, i'm sure i'm not the 
only one that would be interested in seeing the results. :)

-- 
Daniel Maher <dma+gluster AT witbe DOT net>

On 07/07/2010 02:37 AM, Emmanuel Noobadmin wrote:
> As part of my current research for upcoming setup, I started reading
> up on filesystems (was actually looking up fencing which leds to node
> crash and file integrity on different fs) to see if there would be any
> significant improvement to using a different filesystem for the
> glusterfs storage nodes.
> 
> Initially, the reading suggested that ext4 would provide a significant
> performance boost. Quite possibly since it delays write longer,
> latencies of the network storage is hidden even better especially for
> temporary files.
> 
> After subsequent reading, it seems that the cost of the massive delay
> allocation that could blow up really bad in the event the system
> crash.
> 
> However, this may also be limited to poorly written applications, i.e.
> since some massive arguments last year over Firefox 3.0, sqlite and
> ext4 problems, ext4 now detects situations where it should flush to
> disk immediately to avoid zero length files on crash.
> 
> Assuming I'm using the most likely disk writing applications I'm
using
> is KVM, exim, dovecot, mysql and postgresql, would ext4 be safe for
> production use or should I stick to ext3?
A bunch of ext4/xfs/etc. maintainers are in my group.  The "party
line"
is that ext4 can be made suitable for production use *if* you have all
of the latest patches (not just ext4 itself but block layer etc.) *and*
set the right options.  IIRC the default versions and options shipped
with most distributions - including older versions of RHEL and Fedora -
are probably not the ones you want.  The downside is that if you want
greater data safety you pay for it in performance, and some of the
performance regressions associated with switching to safer defaults have
been widely discussed.  If XFS is an option for you, it might be worth
considering because it balances these safety and performance needs a
little better.  Otherwise, I'd recommend careful research and
configuration of ext4, because these are the kinds of problems you
probably won't catch in a synthetic testing environment and you really
don't want to be debugging data-integrity problems just after the Big
Power Hit.