Gluster users - Jun 2012 - Performance optimization tips Gluster 3.3? (small files / directory listings)

Hi,

I'm using Gluster 3.3.0-1.el6.x86_64, on two storage nodes, replicated mode
(fs1, fs2)
Node specs: CentOS 6.2 Intel Quad Core 2.8Ghz, 4Gb ram, 3ware raid, 2x500GB
sata 7200rpm (RAID1 for os), 6x1TB sata 7200rpm (RAID10 for /data), 1Gbit
network

I've it mounted data partition to web1 a Dual Quad 2.8Ghz, 8Gb ram, using
glusterfs. (also tried NFS -> Gluster mount)

We have 50Gb of files, ~800'000 files in 3 levels of directories (max 2000
directories in one folder)

My main problem is speed of directory indexes "ls -alR"  on the
gluster
mount takes 23 minutes every time.

It don't seem like any directory listing information cache, with regular
NFS (not gluster) between web1<->fs1, this takes 6m13s first time, and
5m13s there after.

Gluster mount is 4+ times slower for directory indexing performance vs pure
NFS to single server, is this as expected?
I understand there is a lot more calls involved checking both nodes but I'm
just looking for a reality check regarding this.

Any suggestions of how I can speed this up?

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120606/b1f93ad8/attachment.html>

Could you post the logs of the mount process so that we can analyse what is
going on.
Did you have data on bricks before you created the volume? Did you upgrade from
3.2?

Pranith
----- Original Message -----
From: "olav johansen" <luxis2012 at gmail.com>
To: gluster-users at gluster.org
Sent: Thursday, June 7, 2012 8:02:14 AM
Subject: [Gluster-users] Performance optimization tips Gluster 3.3? (small	files
/ directory listings)


Hi, 

I'm using Gluster 3.3.0-1.el6.x86_64, on two storage nodes, replicated mode
(fs1, fs2)
Node specs: CentOS 6.2 Intel Quad Core 2.8Ghz, 4Gb ram, 3ware raid, 2x500GB sata
7200rpm (RAID1 for os), 6x1TB sata 7200rpm (RAID10 for /data), 1Gbit network

I've it mounted data partition to web1 a Dual Quad 2.8Ghz, 8Gb ram, using
glusterfs. (also tried NFS -> Gluster mount)

We have 50Gb of files, ~800'000 files in 3 levels of directories (max 2000
directories in one folder)

My main problem is speed of directory indexes "ls -alR" on the gluster
mount takes 23 minutes every time.

It don't seem like any directory listing information cache, with regular NFS
(not gluster) between web1<->fs1, this takes 6m13s first time, and 5m13s
there after.

Gluster mount is 4+ times slower for directory indexing performance vs pure NFS
to single server, is this as expected?
I understand there is a lot more calls involved checking both nodes but I'm
just looking for a reality check regarding this.

Any suggestions of how I can speed this up? 

Thanks, 



_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Hi,
Sorry this reply won't be of any help to your problem, but I am too curious
to understand how it can be even slower if monting using Gluster client which I
would expect always be quicker than NFS or anything else.
If you find the reason port it back to the list and share with us please. I
think this directory index issues has been reported already for systems with
many files.

Regards,

Fernando

From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at
gluster.org] On Behalf Of olav johansen
Sent: 07 June 2012 03:32
To: gluster-users at gluster.org
Subject: [Gluster-users] Performance optimization tips Gluster 3.3? (small files
/ directory listings)

Hi,

I'm using Gluster 3.3.0-1.el6.x86_64, on two storage nodes, replicated mode
(fs1, fs2)
Node specs: CentOS 6.2 Intel Quad Core 2.8Ghz, 4Gb ram, 3ware raid, 2x500GB sata
7200rpm (RAID1 for os), 6x1TB sata 7200rpm (RAID10 for /data), 1Gbit network

I've it mounted data partition to web1 a Dual Quad 2.8Ghz, 8Gb ram, using
glusterfs. (also tried NFS -> Gluster mount)

We have 50Gb of files, ~800'000 files in 3 levels of directories (max 2000
directories in one folder)

My main problem is speed of directory indexes "ls -alR"  on the
gluster mount takes 23 minutes every time.

It don't seem like any directory listing information cache, with regular NFS
(not gluster) between web1<->fs1, this takes 6m13s first time, and 5m13s
there after.

Gluster mount is 4+ times slower for directory indexing performance vs pure NFS
to single server, is this as expected?
I understand there is a lot more calls involved checking both nodes but I'm
just looking for a reality check regarding this.

Any suggestions of how I can speed this up?

Thanks,

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120607/e18c35f6/attachment.html>