samba - Apr 2004 - Directory listing with 11k files very slow to list

Hello,

I have a samba server which has a directory with 11764 files. (The
program stores information about jobs and each job has 3 files, nothing
I can do about it).

When I do a listing of that directory using smbclient, it takes 15+
seconds to complete, same thing on Windows XP.

Does anyone have any ideas on how I can speed this up?

I did a log on log level 10, it is accessible at
http://www.enger.org/backup.log (10761841 bytes).

Timeline I have noticed:

[0 sec] Received the request
[0 sec] Starts running on all files:
[2004/04/19 16:31:51, 8] lib/util.c:is_in_path(1496)
  is_in_path: A0000009.job
[2004/04/19 16:31:51, 8] lib/util.c:is_in_path(1500)
  is_in_path: no name list.
[2 sec] starts doing this on all files:
[2004/04/19 16:31:53, 5] smbd/trans2.c:get_lanman2_dir_entry(606)
  get_lanman2_dir_entry found Detail/A000MOON.job fname=A000MOON.job
[2004/04/19 16:31:53, 8] smbd/trans2.c:get_lanman2_dir_entry(521)
  get_lanman2_dir_entry:readdir on dirptr 0x87f3960 now at offset 8
[2004/04/19 16:31:53, 10] lib/ms_fnmatch.c:ms_fnmatch(240)
  ms_fnmatch(*,A000RAJA.pfl) -> 0
[2004/04/19 16:31:53, 8] smbd/dosmode.c:dos_mode(122)
  dos_mode: Detail/A000RAJA.pfl
[2004/04/19 16:31:53, 8] lib/util.c:is_in_path(1496)
  is_in_path: Detail/A000RAJA.pfl
[2004/04/19 16:31:53, 8] lib/util.c:is_in_path(1500)
  is_in_path: no name list.
[2004/04/19 16:31:53, 8] smbd/dosmode.c:dos_mode(170)
  dos_mode returning
[27 sec] finishes off
[2004/04/19 16:32:18, 9] smbd/trans2.c:send_trans2_replies(178)
  t2_rep: params_sent_thistime = 8, data_sent_thistime = 4200,
useable_space = 131012
[2004/04/19 16:32:18, 9] smbd/trans2.c:send_trans2_replies(180)
  t2_rep: params_to_send = 8, data_to_send = 4200, paramsize = 8,
datasize = 4200
[2004/04/19 16:32:18, 6] lib/util_sock.c:write_socket(407)
  write_socket(5,4268)
[2004/04/19 16:32:18, 6] lib/util_sock.c:write_socket(410)
  write_socket(5,4268) wrote 4268
[2004/04/19 16:32:18, 3] smbd/trans2.c:call_trans2findnext(1369)
  SMBtrans2 mask=* directory=Detail dirtype=22 numentries=35

If anyone has any ideas that would be great.

I have tried Samba 2.2.7a and 3.0.2 which both have the same issue. Log
is from 3.0.2. Running Fedora Core 1 and ext3 file system.

Thanks,
	Matthew Enger
	m.enger@xi.com.au



-- 
Matthew Enger
m.enger@xi.com.au
Mob: 0412 463 080
Direct: (03) 9747 4001
X Integration
A Netcruiser Pty Ltd business
Ph: 1300 730 997
Fax: 1300 136 720

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthew Enger wrote:

| Hello,
|
| I have a samba server which has a directory with 11764 files. (The
| program stores information about jobs and each job has 3 files,
| nothing I can do about it).
|
| When I do a listing of that directory using smbclient, it takes 15+
|  seconds to complete, same thing on Windows XP.
|
| Does anyone have any ideas on how I can speed this up?
|
| I did a log on log level 10, it is accessible at
| http://www.enger.org/backup.log (10761841 bytes).
|
| Timeline I have noticed:
|
| [0 sec] Received the request [0 sec] Starts running on all files:
| [2004/04/19 16:31:51, 8] lib/util.c:is_in_path(1496) is_in_path:
| A0000009.job [2004/04/19 16:31:51, 8] lib/util.c:is_in_path(1500)
| is_in_path: no name list. [2 sec] starts doing this on all files:
| [2004/04/19 16:31:53, 5] smbd/trans2.c:get_lanman2_dir_entry(606)
| get_lanman2_dir_entry found Detail/A000MOON.job fname=A000MOON.job
| [2004/04/19 16:31:53, 8] smbd/trans2.c:get_lanman2_dir_entry(521)
| get_lanman2_dir_entry:readdir on dirptr 0x87f3960 now at offset 8
| [2004/04/19 16:31:53, 10] lib/ms_fnmatch.c:ms_fnmatch(240)
| ms_fnmatch(*,A000RAJA.pfl) -> 0 [2004/04/19 16:31:53, 8]
| smbd/dosmode.c:dos_mode(122) dos_mode: Detail/A000RAJA.pfl
| [2004/04/19 16:31:53, 8] lib/util.c:is_in_path(1496) is_in_path:
| Detail/A000RAJA.pfl [2004/04/19 16:31:53, 8]
| lib/util.c:is_in_path(1500) is_in_path: no name list. [2004/04/19
| 16:31:53, 8] smbd/dosmode.c:dos_mode(170) dos_mode returning [27
| sec] finishes off [2004/04/19 16:32:18, 9]
| smbd/trans2.c:send_trans2_replies(178) t2_rep: params_sent_thistime
| = 8, data_sent_thistime = 4200, useable_space = 131012 [2004/04/19
| 16:32:18, 9] smbd/trans2.c:send_trans2_replies(180) t2_rep:
| params_to_send = 8, data_to_send = 4200, paramsize = 8, datasize | 4200
[2004/04/19 16:32:18, 6] lib/util_sock.c:write_socket(407)
| write_socket(5,4268) [2004/04/19 16:32:18, 6]
| lib/util_sock.c:write_socket(410) write_socket(5,4268) wrote 4268
| [2004/04/19 16:32:18, 3] smbd/trans2.c:call_trans2findnext(1369)
| SMBtrans2 mask=* directory=Detail dirtype=22 numentries=35
|
| If anyone has any ideas that would be great.
|
| I have tried Samba 2.2.7a and 3.0.2 which both have the same issue.
| Log is from 3.0.2. Running Fedora Core 1 and ext3 file system.
|
| Thanks, Matthew Enger m.enger@xi.com.au
|
|
|
Not much can be done for this I'm afriad.  I have a directory with
40k(!) files in it.  Guess how long that takes to open.

<flame suit on>I've found a small speed boost by putting the share on
a reiserFS partition.  Yeah, yeah, data corruption blah blah blah.
I've been running it for over a year without a single problem.  That
is not to say I don't backup the new data hourly onto an ext3
partition.  :)

- --
Sean Kennedy
PGP public key: http://tpno.org/keys/0xFC1C377F.asc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFAhGmJIjyA6vwcN38RAoSbAJ9AsU17hMY1mEffeLV+OnmcAzTAdwCdHj6x
8Qc22t0kUUGOw2zd8y1ddHs=H81U
-----END PGP SIGNATURE-----

> | I have a samba server which has a directory with 11764 files. (The
> | program stores information about jobs and each job has 3 files,
> | nothing I can do about it).
> | When I do a listing of that directory using smbclient, it takes 15+
> |  seconds to complete, same thing on Windows XP.
> | Does anyone have any ideas on how I can speed this up?
Three letters - XFS.  We had terrible problems like this, we ditched
ext2/3 and moved to XFS.  Things are much better now.