samba - Jan 2003 - Slow performance with lots of files in one directory

Hello,

I've got performance problems with copying small files over to a Samba
share in a directory that has lots of small files (10000 to 20000
files). It takes too long time to copy new files (they drip in at a fast
pace), and smbd eats a lot of CPU time.

Is there any way to make Samba run faster in this situation?

Cheers,

-- 
Anders Nordby
Aftenposten AS, Systemteknisk avd.

Tlf.: +47 22864083
Fax: +47 22864074

Anders Nordby writes:
> I've got performance problems with copying small files over to a Samba
> share in a directory that has lots of small files (10000 to 20000
> files). It takes too long time to copy new files (they drip in at a
> fast pace), and smbd eats a lot of CPU time.
This could be not so much a Samba problem as a Unix kernel problem.
Traditional Unix filesystems (UFS, FFS, ext2, et c.) do not deal well
with very full directories. See Maurice Bach's book *Design of the Unix
Operating System* and M. K. McKusick's *Design and Implementation of the
4.4BSD Operating System*. These are both just great books.

If you are on Linux, try using one of the new filesystems like ReiserFS,
XFS or JFS. Among other abilities, they can handle extremely full
directories better.



-- 
Chris Palmer   Systems Programmer   GeneEd

At 15:04 09/01/2003, Chris Palmer wrote:
>Anders Nordby writes:
>
>> I've got performance problems with copying small files over to a
Samba
>> share in a directory that has lots of small files (10000 to 20000
>> files). It takes too long time to copy new files (they drip in at a
>> fast pace), and smbd eats a lot of CPU time.
>
>This could be not so much a Samba problem as a Unix kernel problem.
>Traditional Unix filesystems (UFS, FFS, ext2, et c.) do not deal well
>with very full directories. See Maurice Bach's book *Design of the Unix
>Operating System* and M. K. McKusick's *Design and Implementation of the
>4.4BSD Operating System*. These are both just great books.
>
>If you are on Linux, try using one of the new filesystems like ReiserFS,
>XFS or JFS. Among other abilities, they can handle extremely full
>directories better.
What about EXT3 file system?

Thank you 

Roberto



>-- 
>Chris Palmer   Systems Programmer   GeneEd
>
>-- 
>To unsubscribe from this list go to the following URL and read the
>instructions:  http://lists.samba.org/mailman/listinfo/samba

Roberto Jo?o Lopes Garcia writes:
> What about EXT3 file system?
ext3 is ext2 plus journalling, and so not fundamentally different.


-- 
Chris Palmer   Systems Programmer   GeneEd

Unless you are a programmer, I am afraid the only thing you can do is to
modify how the files are stored in that directory.  I had the files on a
ext3 RAID5 with lots of memory config and any type of access to that
directory would bring smb to a crawl.  I even tried putting the files on a
separate XFS RAID5 server and mount the directory, but seemed to just make
it worse even with a 1Gig connection between the servers.  The files that I
store are from our in-house imaging program.  Our file names were all
numeric so it was just a case of changing the name structure from 123456.TIF
to /3/2/1/456.TIF.  In the new file name format, a directory has no more
than 999+10 directory entries.  Now the system is working better than ever.

Regards
-------------------------
Gerald Drouillard
Owner and Consultant
Drouillard & Associates, Inc.
http://www.Drouillard.ca
> -----Original Message-----
> From: samba-admin@lists.samba.org [mailto:samba-admin@lists.samba.org]On
> Behalf Of Anders Nordby
> Sent: Thursday, January 09, 2003 10:19 AM
> To: samba@lists.samba.org
> Subject: [Samba] Slow performance with lots of files in one directory
>
>
> Hello,
>
> I've got performance problems with copying small files over to a Samba
> share in a directory that has lots of small files (10000 to 20000
> files). It takes too long time to copy new files (they drip in at a fast
> pace), and smbd eats a lot of CPU time.
>
> Is there any way to make Samba run faster in this situation?
>
> Cheers,
>
> --
> Anders Nordby
> Aftenposten AS, Systemteknisk avd.
>
> Tlf.: +47 22864083
> Fax: +47 22864074
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  http://lists.samba.org/mailman/listinfo/samba