samba - May 2006 - Performance issue on AIX when deleting files in a directory with a large number of files

Server:
- AIX5.3 with JFS2 file systems
- IBM pSeries 520 (1.9GHz p5+ CPU)
- Samba 3.0.22 (and the same problem exists in 3.0.21a)
- I tested Samba compiled from sources with GCC 4.0.2 and 3.4.3 same result
for both. I also downloaded the AIX binary from
http://us4.samba.org/samba/ftp/Binary_Packages/AIX/ and it has the same
problem as well.

Client:
MS Windows XP SP2


We have an application that creates a large number of files in a single
directory. At peak times the number of files in that directory get up around
150,000. The files are around 10-200KB in size. When I try to delete files
out of that directory then things get really slow ... up to 2-3 seconds per
file ... and the CPU utilization on the server goes up near 100%.
If I delete files out of that directory directly on the box then it goes
very quickly so it seems like this is definitely a Samba issue.

I tried the suggestions on this page:
http://mirrors.uol.com.br/pub/samba/HOWTO/Samba-LargeDirectory-HOWTO

But that didn't make any difference either.

I also put together a Linux test box (1.5GHz P4, Ubuntu Dapper Beta 7, Samba
3.0.22, ext3 file system) and the performance issue is not present on that
platform.

-Claus
____________________________________________
Claus Lund
Systems Developer

Department of Taxes
Information Systems
133 State Street
Montpelier, Vermont 05633-1401
(802) 828-3735

On Wed, May 10, 2006 at 10:17:53AM -0400, Claus Lund
wrote:
> 
> We have an application that creates a large number of files in a single
> directory. At peak times the number of files in that directory get up
around
> 150,000. The files are around 10-200KB in size. When I try to delete files
> out of that directory then things get really slow ... up to 2-3 seconds per
> file ... and the CPU utilization on the server goes up near 100%.
> If I delete files out of that directory directly on the box then it goes
> very quickly so it seems like this is definitely a Samba issue.
This may be the client rescanning the directory after a delete.
If the application is braindamaged enough to create 150k files
in a directory it's stupid enough to rescan also.
> I tried the suggestions on this page:
> http://mirrors.uol.com.br/pub/samba/HOWTO/Samba-LargeDirectory-HOWTO
> 
> But that didn't make any difference either.
> 
> I also put together a Linux test box (1.5GHz P4, Ubuntu Dapper Beta 7,
Samba
> 3.0.22, ext3 file system) and the performance issue is not present on that
> platform.
Hmmm. So the problem doesn't occur on a Linux+ext3 box, but does
on an AIX+jfs box - with the same version of Samba....

I can see a pointer to the problem here, and it doesn't look like
Samba to me :-).

Jeremy.

Hi Bill,

I already tried updating AIX.
I'm on 5300-04 (only a couple of security related fixes added since the tech
level) on my production box (we're using Samba 3.0.21a on that box). And
I'm
on the latest Technology Level + all the latest fixes on my test box (and
that's where I currently have 3.0.22).

Both machines are setup with JFS2 and neither use inline logs.

I am not all that familar with debugging tools like truss (I haven't really
done any real programming since my college days ... 6-7 years ago). But I
just ran:
truss -f -o /datavg/smb_truss.out /opt/Samba/3.0.22/sbin/smbd -D

I used Windows Explorer to browse to my test directory (with about 30K files
in it), highlighted all the files and hit delete. I let it delete a few
files and then cancelled. The truss file is about 185MB (4MB zipped). Do you
have a place where I can upload that?

-Claus
> > >
> > > But the nfs or local access isn't performing the same access
pattern
> > > that Samba is by being driven by the client. I'm guessing
that if you
> > > performed the same actions locally that the client is requesting
> > > Samba perform you'd get the same results (in fact you *must*
- as
> > > all of Samba is userspace, there's no magic in what Samba is
doing
> > > here - it's doing what the client requests from userspace).
> > >
> > > My money is still on the kernel, as driven in this access
pattern.
> > >
> >
> > Yep. I'd be curious what a truss of a smbd process shows for
access on
> that
> > filesystem.
> >
> > Is this jfs2 using an inline log? Just curious...
> >
>
> Claus, can you run "oslevel -r" and send me the results. I think
this may
> have been fixed in 5300-03.
>
> 5300-04-3 is available, I'd consider that as well...
>
>
> Cheers,
>
> Bill
>