Gluster users - Jun 2009 - Low Performance Problems

Hello, I'm evaluating GlusterFS as a storage backend for webcontent
behind a cache cluster and I feel I'm not getting the performance I
should. My test hardware is:
client:
dual opteron 2.2ghz, 4gb ram, 2x36.5gb scsi in hardware raid1 ext3

servers:
pentium d 2.8ghz, 1gb ram, 4x500gb sata in software raid5 ext3

the servers are connected via gigabit ethernet on a dedicated switch
with no other traffic. All servers are running Ubuntu 8.04 64bit. I
tried GlusterFS 2.0.1 and 2.0.2 with similar results.

my glusterfs config is as follows, all translators with default options:
server:
posix
locks
io-threads
server(tcp)

client:
clients(tcp)
[distribute]
read-ahead
write-behind
io-threads

I ran some tests copying a 227mb fileset consisting of 21690 image files
 with an average size of 8kb. For comparison I used a NFSv3 share. Here
are the results:

NFS:
write: 74s
read:  36s

GlusterFS 1 Server:
write: 332s
read:   59s

GlusterFS 2 Servers with Distribute:
write: 331s
read:   60s

I monitored Loadaverage, CPU and Diskusage on all machines and found no
bottleneck there. Network problems are also not an option as NFS does
way better. Am I right in expecting better performance from GlusterFS
and does anyone know where I could start in improving?

Thanks,
Martin

On Thu, 18 Jun 2009 15:02:34 +0200
Martin Reissner <mreissner at wavecon.de> wrote:
> [...]
> NFS:
> write: 74s
> read:  36s
> 
> GlusterFS 1 Server:
> write: 332s
> read:   59s
> 
> GlusterFS 2 Servers with Distribute:
> write: 331s
> read:   60s
Can you produce the same test for replicate, too?
A really interesting setup for people who want to get rid of NFS...
In theory the minimum time (caused by a FE network) should be below 23s for
read and write or maybe 46s for replicate write case. I know you have GBit
ethernet, but your disks won't cope with that anyway, so one would be
content
with a factor 2 in real life. Nevertheless your test really shows that NFS is
not that bad.
A local-disk FUSE fs would be an interesting comparison, too.
-- 
Regards,
Stephan

Alright, some updates for the record...

I noticed that my client box was in fact running Ubuntu 7.10 with Fuse
2.7.0, so I switched to another box with 8.04 (Fuse 2.7.2). The write
performance doubled and I could move the fileset in ~162s now. Read
performance was same as before.

I then tried the GlusterFS patched fuse-2.7.4glfs11 but it wouldn't
compile on 8.04. The error was:
/usr/local/src/fuse-2.7.4glfs11/kernel/dir.c: In function ?iattr_to_fattr?:
/usr/local/src/fuse-2.7.4glfs11/kernel/dir.c:1027: error: ?struct iattr?
has no member named ?ia_file?

but thats a different story. So I went back to the 7.10 box and
installed the glusterpatched fuse there. Here are the results with
distribute on two servers:

write: 90s
read:  60s

Writes increased again but still far from the NFS performance and reads
are the same as before. The weirdest thing about all this is that read
performance didn't increase at all when using two servers with
distribute compared to only one. Looks like there is a bottleneck on the
client either in GlusterFS or Fuse...

Martin

Martin Reissner wrote:
> Hello, I'm evaluating GlusterFS as a storage backend for webcontent
> behind a cache cluster and I feel I'm not getting the performance I
> should. My test hardware is:
> client:
> dual opteron 2.2ghz, 4gb ram, 2x36.5gb scsi in hardware raid1 ext3
> 
> servers:
> pentium d 2.8ghz, 1gb ram, 4x500gb sata in software raid5 ext3
> 
> the servers are connected via gigabit ethernet on a dedicated switch
> with no other traffic. All servers are running Ubuntu 8.04 64bit. I
> tried GlusterFS 2.0.1 and 2.0.2 with similar results.
> 
> my glusterfs config is as follows, all translators with default options:
> server:
> posix
> locks
> io-threads
> server(tcp)
> 
> client:
> clients(tcp)
> [distribute]
> read-ahead
> write-behind
> io-threads
> 
> I ran some tests copying a 227mb fileset consisting of 21690 image files
>  with an average size of 8kb. For comparison I used a NFSv3 share. Here
> are the results:
> 
> NFS:
> write: 74s
> read:  36s
> 
> GlusterFS 1 Server:
> write: 332s
> read:   59s
> 
> GlusterFS 2 Servers with Distribute:
> write: 331s
> read:   60s
> 
> I monitored Loadaverage, CPU and Diskusage on all machines and found no
> bottleneck there. Network problems are also not an option as NFS does
> way better. Am I right in expecting better performance from GlusterFS
> and does anyone know where I could start in improving?
> 
> Thanks,
> Martin
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>