Hi
I'm currently experiencing an excessive amount of CPU usage by my
smbd daemons, and was wondering what I could do to reduce it.
The gear I'm using is...
- Server is a dual CPU SGI Octane with 1Gb of memory, and no other
processes running apart from Samba.
- The disk space being served is a 300Gb Ciprico RAID volume with a
truly fast access speed.
- The server has a Gigabit ethernet connection to our switch.
- Samba 2.0.7 (SGI Freeware pre-compiled and packaged binary).
In my opinion, that should be plenty of resources to serve fifteen or
so NT workstations (that's about what we are using). However, during
even medium usage periods, there can be a dozen or more individual
smbd processes running, each consuming up to 70% CPU (if they manage
to clam themselves to the top of the pile they will take that much,
though generally competition between themselves limits them to 20% or
so... with the server CPU use at 100% full-time).
After many days of playing with this problem I have a few pieces of
information that lead me to believe that Samba (as it is configured on
my server at the moment) is doing too much work. I will not be at all
surprised to find that I have overlooked some configuration flag, but
I am at a loss just now about what to try next!
- I have experimented with "max xmit" and discovered that a value of
around 8k gives the best CPU performance for a given file copy,
but it's still fairly high utilisation.
- Experimented with locks and oplocks and level 2 oplocks, but found
that the default values were about the best (and that's what we
were using when the problem was first noticed).
- Experimented with file transfers and discovered something
interesting... the copy of a 1Gb file proceeded at around 7.5Mb
per second, which is in line with the 9Mbps we achieved with FTP
(which is always expected to be a bit faster than a more complex
protocol). However, when copying 10,000 files of 10k each (only
100Mb) it took AGES, and CPU use went towards 100% for the smbd
thread handling the requests.
- Unfortunately, our typical usage of the Samba file system is more
accurately modeled by the test using thousands of small files... a
general render for a small chunk of one of our 3D scenes may use
1000 to 2000 small scene description files (which are around 4kb
each), plus 400 to 1000 texture image files (averaging 5Mb, though
some are MUCH bigger). There are also many intermediate stages
that get written back to the Samba volume in order to be re-used
by other processes, so it's not just one-way traffic. And of
course, our rendered images go back there too... they can be up to
fifteen Mb each.
- Accessing a file which resides in a directory that has 10,000
other files in it appears to be much slower and much more CPU
intensive than performing the same operation on a file in a nearly
empty directory.
- We have used "par" to analyse the system calls made by the Samba
daemon during heavy loads, and it seems to be doing a lot of
excess directory reading. For the technically-minded, ngetdents()
is called LOTS of times. Although it is rather expensive to call,
there doesn't seem to be much caching of the results... is there a
way to cache directory contents? I have activated the "getwd
cache" option but that only seems to cache the tree walk, not the
dir contents themselves.
If your opinion is that our usage patterns are aggravating the
problem, I would agree with you! However, I really NEED to use Samba
here as part of this setup. It is the only thing that will bridge the
gap seamlessly, and apart from slowdown caused by over-use of server
CPUs, it is performing flawlessly (ie, we have all the access to the
files we need, in a straight-forward and natural manner).
We are using a heterogeneous network comprised of SGI and NT
workstations and Samba is the only solution we have found that can fit
into our production pipeline.
I have tried an NFS based system (my current Samba setup replaces our
old system, which used Maestro). Although the NFS system was not
flexible enough for our needs, even at peak use it did not use
anywhere near the amount of CPU that my Samba daemons are currently
using.
We have used Samba in the past for other projects, and it worked fine,
but the user software was different and made use of fewer, larger
files. Testing our old server under our new conditions yielded the
same result... over-use of CPU. I didn't notice the differences in our
usage patterns before I committed to using it for this project... but
now I'm neck deep and I really need help!
Yes, I have read the Samba Performance Tuning document and have obeyed
all those recommendations.
If you could please offer me any advice on how to tune Samba for high
frequency, small size file access,
Justen
--
.-----------------------.------------.--------------------------.
| Justen Marshall | | e-mail: justen@al.com.au |
| Technical Director | | phone: +61-2-9383-4831 |
| Animal Logic Pty Ltd | | fax: +61-2-9383-4801 |
|-----------------------^------------^--------------------------|
| Athiesm is a non-prophet organization. |
`---------------------------------------------------------------'
Justen, On Wed, 4 Oct 2000 13:50:30 +1000, Justen Marshall wrote:>I'm currently experiencing an excessive amount of CPU usage by my >smbd daemons, and was wondering what I could do to reduce it.> - Experimented with file transfers and discovered something > interesting... the copy of a 1Gb file proceeded at around 7.5Mb > per second, which is in line with the 9Mbps we achieved with FTP > (which is always expected to be a bit faster than a more complex > protocol). However, when copying 10,000 files of 10k each (only > 100Mb) it took AGES, and CPU use went towards 100% for the smbd > thread handling the requests.Well, most unixes do perform really bad when it comes to directories of such size.>If you could please offer me any advice on how to tune Samba for high >frequency, small size file access,I guess you checked the standard options for such cases: no "hide [dot] files" no "veto files" no "dont descend" wide links = yes getwd cache = yes change notify timeout = <as much as possible> debug level = <as low as possible> Do you have a lot of locks? Dir you consider setting "lock directory" to a virtual disk in memory? I don't know your OS but at least one of the OSs I used in the past had a tuning option in itself to cache more directory entries. Might be worth to check with your "local dealer". Regards, Robert -- --------------------------------------------------------------- Robert.Dahlem@gmx.net Fax +49-69-432647 --------------------------------------------------------------- Sent using PMMail (http://www.pmmail2000.com) - fast, decent, email software; far better than Outlook. Try it sometime.
If the poster could describe the size and contents of the directory, we could do a little experiment with ufs and samba to see if the problem lies in the OS or Samba... Older Solaris systems were slow updating heavily-hit directories, which hits the Windows "rename, copy/edit, save, delete" sequence right between the eyes, but didn't hurt "copy *" operations, so the data and the use of it are relevant. And, as always, if the data is structured like 27july1944.jsmit.23, it should be changed to a layout like july1944/27/jsmit.23, so as to keep the directoiry sizes small and bounded... --dave -- David Collier-Brown, | Always do right. This will gratify some people 185 Ellerslie Ave., | and astonish the rest. -- Mark Twain Willowdale, Ontario | //www.oreilly.com/catalog/samba/author.html Work: (905) 415-2849 Home: (416) 223-8968 Email: davecb@canada.sun.com
Uhhh, I don't think your computer is fast enough. Steve On Wed, 4 Oct 2000, Justen Marshall wrote:> Hi > > I'm currently experiencing an excessive amount of CPU usage by my > smbd daemons, and was wondering what I could do to reduce it. > > The gear I'm using is... > > - Server is a dual CPU SGI Octane with 1Gb of memory, and no other > processes running apart from Samba. > > - The disk space being served is a 300Gb Ciprico RAID volume with a > truly fast access speed. > > - The server has a Gigabit ethernet connection to our switch. > > - Samba 2.0.7 (SGI Freeware pre-compiled and packaged binary). > > In my opinion, that should be plenty of resources to serve fifteen or > so NT workstations (that's about what we are using). However, during > even medium usage periods, there can be a dozen or more individual > smbd processes running, each consuming up to 70% CPU (if they manage > to clam themselves to the top of the pile they will take that much, > though generally competition between themselves limits them to 20% or > so... with the server CPU use at 100% full-time). > > After many days of playing with this problem I have a few pieces of > information that lead me to believe that Samba (as it is configured on > my server at the moment) is doing too much work. I will not be at all > surprised to find that I have overlooked some configuration flag, but > I am at a loss just now about what to try next! > > - I have experimented with "max xmit" and discovered that a value of > around 8k gives the best CPU performance for a given file copy, > but it's still fairly high utilisation. > > - Experimented with locks and oplocks and level 2 oplocks, but found > that the default values were about the best (and that's what we > were using when the problem was first noticed). > > - Experimented with file transfers and discovered something > interesting... the copy of a 1Gb file proceeded at around 7.5Mb > per second, which is in line with the 9Mbps we achieved with FTP > (which is always expected to be a bit faster than a more complex > protocol). However, when copying 10,000 files of 10k each (only > 100Mb) it took AGES, and CPU use went towards 100% for the smbd > thread handling the requests. > > - Unfortunately, our typical usage of the Samba file system is more > accurately modeled by the test using thousands of small files... a > general render for a small chunk of one of our 3D scenes may use > 1000 to 2000 small scene description files (which are around 4kb > each), plus 400 to 1000 texture image files (averaging 5Mb, though > some are MUCH bigger). There are also many intermediate stages > that get written back to the Samba volume in order to be re-used > by other processes, so it's not just one-way traffic. And of > course, our rendered images go back there too... they can be up to > fifteen Mb each. > > - Accessing a file which resides in a directory that has 10,000 > other files in it appears to be much slower and much more CPU > intensive than performing the same operation on a file in a nearly > empty directory. > > - We have used "par" to analyse the system calls made by the Samba > daemon during heavy loads, and it seems to be doing a lot of > excess directory reading. For the technically-minded, ngetdents() > is called LOTS of times. Although it is rather expensive to call, > there doesn't seem to be much caching of the results... is there a > way to cache directory contents? I have activated the "getwd > cache" option but that only seems to cache the tree walk, not the > dir contents themselves. > > If your opinion is that our usage patterns are aggravating the > problem, I would agree with you! However, I really NEED to use Samba > here as part of this setup. It is the only thing that will bridge the > gap seamlessly, and apart from slowdown caused by over-use of server > CPUs, it is performing flawlessly (ie, we have all the access to the > files we need, in a straight-forward and natural manner). > > We are using a heterogeneous network comprised of SGI and NT > workstations and Samba is the only solution we have found that can fit > into our production pipeline. > > I have tried an NFS based system (my current Samba setup replaces our > old system, which used Maestro). Although the NFS system was not > flexible enough for our needs, even at peak use it did not use > anywhere near the amount of CPU that my Samba daemons are currently > using. > > We have used Samba in the past for other projects, and it worked fine, > but the user software was different and made use of fewer, larger > files. Testing our old server under our new conditions yielded the > same result... over-use of CPU. I didn't notice the differences in our > usage patterns before I committed to using it for this project... but > now I'm neck deep and I really need help! > > Yes, I have read the Samba Performance Tuning document and have obeyed > all those recommendations. > > If you could please offer me any advice on how to tune Samba for high > frequency, small size file access, > > Justen > -- > .-----------------------.------------.--------------------------. > | Justen Marshall | | e-mail: justen@al.com.au | > | Technical Director | | phone: +61-2-9383-4831 | > | Animal Logic Pty Ltd | | fax: +61-2-9383-4801 | > |-----------------------^------------^--------------------------| > | Athiesm is a non-prophet organization. | > `---------------------------------------------------------------' > >
A few months ago, we did some testing of Samba with 3000 files in a directory. I wrote a simple program that would create 3000 zero byte files, named 1, 2, 3, etc. Using a Win98 client we tested this with Samba 2.0x running under Solaris 7.0, OpenBSD, Linux w/EXT2, and Linux with the ReiserFS. In every instance the file creation got slower and slower, and the CPU usage would gradually increase to the point of 80-100%, this only with a single Win98 client! We've got a few NT40 servers that have more than 20000 files in a single directory, that don't perform this badly. We'd like to replace these servers with Linux boxes using Samba, but unless we re-organize these files into multiple directories (which means a bunch of custom software needs to be rewritten), this isn't possible. This definately is a problem that should be a looked into, this is not the first time I've seen someone complain about this. Spliting the files up into multiple directories, is just a temporary workaround, not a solution. Dave Dezinski> If the poster could describe the size and contents of the directory, >we could do a little experiment with ufs and samba to see if the >problem lies in the OS or Samba... > Older Solaris systems were slow updating heavily-hit directories, >which hits the Windows "rename, copy/edit, save, delete" sequence >right between the eyes, but didn't hurt "copy *" operations, so >the data and the use of it are relevant. > > And, as always, if the data is structured like 27july1944.jsmit.23, >it should be changed to a layout like july1944/27/jsmit.23, so as >to keep the directoiry sizes small and bounded... > >--dave
> On Wed, 4 Oct 2000, Justen Marshall wrote: > > [ lots of things about small files ] >I missed the start of the thread so please forgive me if you've already done this... what kind of performance do you get running running Unix commands on the Octane ?? The following is an O2000 4 CPU+Cipricos, and an Indigo 2 both with 6.5.9f e.g. say I have a small directory with a few average sized files... small for our industry anyway :-) hxpro@mutha 43% timex ls > /dev/null real 0.13 user 0.11 sys 0.02 hxpro@mutha 44% timex ls -l > /dev/null real 2.17 user 0.90 sys 1.27 hxpro@mutha 45% ls -l | wc 12224 110009 809081 hxpro@mutha 46% timex du -sk 99349072 . real 0.52 user 0.03 sys 0.49 hxpro@mutha 49% timex find . -name foo -print real 0.47 user 0.04 sys 0.43 the long listing is taking longer due to NIS server lookup, but then I expect nsd to cache that result and go from cache for the user id -> name translation etc. Repeatedly running the ls -l the 2nd time is hxpro@mutha 54% timex ls -l > /dev/null real 1.41 user 0.89 sys 0.52 running via NFSv3 UDP (155 ATM) ... hxpro@wallace 5% timex ls > /dev/null real 0.59 user 0.34 sys 0.08 hxpro@wallace 6% timex ls -l > /dev/null real 19.18 user 4.60 sys 7.34 hxpro@wallace 11% timex du -sk 99349056 . real 11.40 user 0.13 sys 4.83 hxpro@wallace 9% timex find . -name foo -print real 11.25 user 0.19 sys 3.97 nfsd at 10-15% CPU (R12K@400) On the SGI nfsd is a kernel thread place holder, smbd is a user level process so it will take more time to schedule and context switch etc. It would be nice to have kernel level smbd hooks and or multithreaded smbd but that's not an easy port/recode. I agree a directory cache in the application layer would be nice, you could look at the kernel parameters xfs_refcache_percent, nbuf, ncsize and try increasing them (disclaimer we don't do much with them here so your milage may vary ...) Kevin -- | Kevin Wheatley | These are the opinions of nobody | | Technical Services Manager | and are not shared by my employers | | Cinesite Digital Studios | |