Simon.Harris@nationwide.co.uk
2004-Feb-05 18:41 UTC
[Samba] massive performance problems if transferring many sma ll files
-----BEGIN PGP SIGNED MESSAGE----- ******************************************************************** The contents of this email are intended exclusively for the addressee. If you are not the addressee you must not read use or disclose the email contents ; you should notify us immediately [ by clicking "Reply" ] and delete this email. Nationwide monitors e-mails to ensure its systems operate effectively and to minimise the risk of viruses. Whilst it has taken reasonable steps to scan this email, it does not accept liability for any virus that may be contained in it. ******************************************************************** Having had similar problems with not much success I would be interested in the outcome. Seems that Solaris with Samba with large number of files in a single directory is not a good mix! Doing a network trace the, time it takes samba to respond goes up significantly with the more files in that directory. In tests with an application doing basically a findfirst for every file gave the following results: Directory with 3000 files Reply for 'file found': 16ms Reply for 'file not found': 47ms Directory with 9000 files Reply for 'file found': 31 - 62ms Reply for 'file not found': 125ms Directory with ~17000 files Reply for 'file found': 100 - 200ms Reply for 'file not found': 250ms These were rough timings but you can see the trend. Also, can't find option 'mangle method'. Cheers, Simon - -----Original Message----- From: steffen.k@gmx.at [mailto:steffen.k@gmx.at] Sent: 05 February 2004 10:29 To: abartlet@samba.org Cc: samba@lists.samba.org Subject: Re: [Samba] massive performance problems if transferring many small files> On Thu, 2004-02-05 at 20:12, steffen.k@gmx.at wrote: > > Hi, > > > > If I try to transfer about 27.000 small HTML-Files from myWindows-PC to> our > > central SAMBA-Server I'll get massive performance problems. First the > > performance looks ok but after a time period of 3 minutes it goes slow > do > wn - > > appr. one file per second will be transferred. Has anyone an idea what I > can tune > > to avoid this annoying behaviour. > > > - samba version 2.2.8a (64bit compiled) > > Are all/many of these files in the same directory?Yes, it's a flat directory hierarchy.> I would suggest that Samba 3.0 might handle this situation better, or in > 2.2 set 'mangle method = hash2'.I'll try it.... Are there any Solaris kernel variables they interfere with this behaviour and needs to be adjusted? Regards, Steffen - -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba -----BEGIN PGP SIGNATURE----- Version: PGPsdk version 1.1.1 (C) 1997 Pretty Good Privacy, Inc. iQEVAwUBQCKQjm4wJNdvwuyFAQG3Ygf/UECbdjJ89Dlqn/6uvxNmgbgvIZ03QRHY vMBfqc/WfGyH9xJ/0YDbKr+zDppPeBx4vyDIhCVMxt3khQgceFoT0NwE3JRA5Rqx m4yZ9ccosMNcWCJgFFyKRUPy2kcH+he/Aooi+UXyISbAluGTEAGpmrrbiEotCuQC kFyXlCQBxwJfcM8VSgUvBtUXvXay6taEpvQmLSWGG9MMRtOZnRYiDFtpvEOsrerX wHjwWidLKLyhfEcLFtHu93THfiRDSK59rb8xdko3Rn58Tg9jgjUMbWWgBfRT38nR TWJp7kd1m4sm/BQcDHt9fU4LiDh6hVi2FJysOR6I5P4gJEWG4Kosfg==oqA1 -----END PGP SIGNATURE-----
Simon.Harris@nationwide.co.uk
2004-Feb-12 10:05 UTC
[Samba] massive performance problems if transferring many sma ll files
-----BEGIN PGP SIGNED MESSAGE-----
********************************************************************
The contents of this email are intended exclusively for the
addressee. If you are not the addressee you must not read
use or disclose the email contents ; you should notify us
immediately [ by clicking "Reply" ] and delete this email.
Nationwide monitors e-mails to ensure its systems operate
effectively and to minimise the risk of viruses. Whilst it has
taken reasonable steps to scan this email, it does not
accept liability for any virus that may be contained in it.
********************************************************************
What does mangle method hash2 do differently?
My case involves similar amount of files in a single directory in the
format:
five-digits[0-9].{ABC}
(where I can have three extensions for same five-digits)
This is managed by an application so will not get duplicates.
Is there anything I can tune to speed-up the findfirst that my application
is doing for every file?
Am using the defaults in samba.conf!
- -----Original Message-----
From: abartlet@samba.org [mailto:abartlet@samba.org]
Sent: 05 February 2004 20:48
To: steffen.k@gmx.at
Cc: samba@lists.samba.org; abartlet@samba.org
Subject: Re: [Samba] massive performance problems if transferring many small
files
Security warning. Details in WARNING.TXT about the possible problem.
- --------------------------------------------------------------------
On Thu, 2004-02-05 at 21:34, Steffen Kauka wrote:> > On Thu, 2004-02-05 at 20:12, steffen.k@gmx.at wrote:
> > > Hi,
> > >
> > > If I try to transfer about 27.000 small HTML-Files from my
Windows-PC
to> > our
> > > central SAMBA-Server I'll get massive performance problems.
First the
> > > performance looks ok but after a time period of 3 minutes it goes
slow
> > do
> > wn -
> > > appr. one file per second will be transferred. Has anyone an idea
what
I> > can tune
> > > to avoid this annoying behaviour.
> >
> > > - samba version 2.2.8a (64bit compiled)
> >
> > Are all/many of these files in the same directory?
>
> Yes, it's a flat directory hierarchy.
Because unix is case sensitive, and windows is not, this is almost a
pathalogical case for Samba.
Samba must scan the *entire* directory, to see if there is a matching
file (of potentially different case), so it can say 'sorry, file by that
name already'. Naturally, this isn't exactly fast as you approach 27000
files...
> > I would suggest that Samba 3.0 might handle this situation better, or
in
> > 2.2 set 'mangle method = hash2'.
>
> I'll try it....
The hash2 method is not only much faster, it has a much lower collision
rate. This helps if for some reason, the 8.3 names are being used.
If you can put the files into a hierarchy, then things will be *much*
better. Or, for this copy only, you might want to turn 'case sensitive
= yes' on in your smb.conf - however the implications of that are nasty
for normal windows operations (and perhaps even the copy, depending on
what you use).
Another quick hack might simply be to zip the files up, and run 'unzip'
on the server.
Andrew Bartlett
- --
Andrew Bartlett abartlet@pcug.org.au
Manager, Authentication Subsystems, Samba Team abartlet@samba.org
Student Network Administrator, Hawker College abartlet@hawkerc.net
http://samba.org http://build.samba.org http://hawkerc.net
- -- To unsubscribe from this list go to the following URL and read the
instructions: http://lists.samba.org/mailman/listinfo/samba
-----BEGIN PGP SIGNATURE-----
Version: PGPsdk version 1.1.1 (C) 1997 Pretty Good Privacy, Inc.
iQEVAwUBQCtSBG4wJNdvwuyFAQE0Ogf/ejDWK4i+kj4lPnlkA4diWY3DRzWgkMwo
qNBS06B/2eQt5YL0uEL7k1kNNkCfxaIQvQ5ivtOb3rL9Tb/NQ2x7VIIVnsQVV8gl
cLkv2wFHyhatnvagEySRaBvGoizGjOkDdWTKLU1VjLHsd82dNLmQqPKSFNAtkAiZ
cNh9E7wQwhgKrAd6wWXBJZMljXKY/eSizdgeSx7kOPL0UtHz1ozD518PZdU+XZXJ
lDsYXdUckdvlzMsNwqpcY42DCU8OUEjmqlDmrazVaLLVpHy4FLSKzNyggePhUXjg
z8wMEm9ggquwhRg/hLCVI5Nqyd2+qrs8tsgZ+MV33ftQsNO0HQywjQ==L9/L
-----END PGP SIGNATURE-----