Gluster users - Dec 2012 - Does brick fs play a large role on listing files client side?

Hello everyone,

We are in the process of evaluating GlusterFS to replace an ocfs2 deployment as
backing storage for a web application server cluster.

My ocfs2 deployment was 8 x 512GB Crucial M4 SSDs in RAID0 connected to an LSI
9260-8i with DRBD protocol C pushed out as an iSCSI LUN via IET.

My GlusterFS deployment right now is 8 x 512GB OCZ Vertex 4 (no RAID) connected
to Dell PERC H710, formatted as XFS and put together into a distributed volume.

Client side, I'm seeing much higher response times from GlusterFS. However,
listing directories with millions of files take much longer with GlusterFS than
with ocfs2. Is this to be expected? Would I see an improvement with ext4 on the
bricks?

I would appreciate any advice.

Thanks,
Michael



______________________________________________________________________________________
Michael Kushnir
System Architect / Engineer
Communications Engineering Branch
Lister Hill National Center for Biomedical Communications
National Library of Medicine

8600 Rockville Pike, Building 38A, Floor 10
Besthesda, MD 20894

Phone: 301-435-3219
Email: michael.kushnir at nih.gov<mailto:michael.kushnir at nih.gov>

[VMW-LGO-CERTIFIED-PRO-5-K]

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121204/25ebf8b3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 8449 bytes
Desc: image001.jpg
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121204/25ebf8b3/attachment.jpg>

On Dec 4, 2012, at 5:30 PM, Kushnir, Michael (NIH/NLM/LHC) [C] wrote:
> My GlusterFS deployment right now is 8 x 512GB OCZ Vertex 4 (no RAID)
connected to Dell PERC H710, formatted as XFS and put together into a
distributed volume.
Hi,

Are you just using a single brick? Gluster is a scale-out NAS file system so is
usually used when you want to want to aggregate the disk performance and disk
space of many machines into a singe Global Name Space.

ocfs (cluster filesystem) is more for when you have a single disk volume
attached via SCSI to many machines. More than one machine cannot for instance
access the same ext4 filesystem concurrently. ocfs provides a locking mechanism
allowing many systems to access the same SCSI device at the same time.

Gluster is to NFS as OCFS is to EXT4 (kinda).

The lag your getting might be due to FUSE (Filesystem in Userspace). FUSE allows
weird and wonderful filesystems to be mounted in userspace meaning kernel
support is not required. This is typically much slower than kernel enabled
filesystems.

Cheers,

Andrew

Thanks for the reply,
> Are you just using a single brick? Gluster is a scale-out NAS file system
so is usually used when you want to want to aggregate the disk performance and
disk space of many machines into a singe Global Name Space.
I currently have one server with 8 bricks. Once I get through evaluation, we
will expand to multiple servers with 24 bricks each. We are looking to have a
replica count of 2  for each brick eventually.

On my gluster server, I can run an ls against /export/*/imgs and get file
listings from each brick in seconds. However, on my client, I run ls against the
/imgs/ directory on the gluster volume and wait days. Even if I mount the
gluster volume on the storage server itself, ls takes a long long time.

So, what are my options for improving the speed of directory listing on gluster
clients? Would changing brick FS to ext4 make a difference in the time it takes
to list on the client? Should I try mounting the volume over NFS? Something
else?

Thanks,
Michael


-----Original Message-----
From: Andrew Holway [mailto:a.holway at syseleven.de] 
Sent: Tuesday, December 04, 2012 4:47 PM
To: Kushnir, Michael (NIH/NLM/LHC) [C]
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Does brick fs play a large role on listing files
client side?


On Dec 4, 2012, at 5:30 PM, Kushnir, Michael (NIH/NLM/LHC) [C] wrote:
> My GlusterFS deployment right now is 8 x 512GB OCZ Vertex 4 (no RAID)
connected to Dell PERC H710, formatted as XFS and put together into a
distributed volume.
Hi,

Are you just using a single brick? Gluster is a scale-out NAS file system so is
usually used when you want to want to aggregate the disk performance and disk
space of many machines into a singe Global Name Space.

ocfs (cluster filesystem) is more for when you have a single disk volume
attached via SCSI to many machines. More than one machine cannot for instance
access the same ext4 filesystem concurrently. ocfs provides a locking mechanism
allowing many systems to access the same SCSI device at the same time.

Gluster is to NFS as OCFS is to EXT4 (kinda).

The lag your getting might be due to FUSE (Filesystem in Userspace). FUSE allows
weird and wonderful filesystems to be mounted in userspace meaning kernel
support is not required. This is typically much slower than kernel enabled
filesystems.

Cheers,

Andrew