Hi all, I'm debugging a performance problem for a client. Needless to say, the organisation of their data is such that this problem is really annoying. Here's the situation: Samba 3.0.x PDC in a mixed 9x/nt/xp environment, using acls. When trying to delete a large number of folders from a folder, the 'preparing to delete' stage takes about half hour. It's simple to repeat: set up samba pdc, join an xp client into the domain, map a share with some 4000 folders and try to delete them. Strace of a smbd proccess in the 'preparing to delete' stage looks like this: stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 gettimeofday({1090703949, 903367}, NULL) = 0 stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 getxattr("/mnt/test/test/0220", "system.posix_acl_access", 0x11fffe940, 132) = -1 ENODATA (No data available) stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 getxattr("/mnt/test/test/0220", "system.posix_acl_default", 0x11fffe940, 132) = -1 ENODATA (No data available) stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 getxattr("/mnt/test/test/0220", "user.SAMBA_PAI", 0x1207f0aa0, 1024) = -1 ENODATA (No data available) For every folder it wants to delete, it walks all the folders doing this. So for 4k folders we get 16 milion such tests, which take waay too much time. Are these loops necessary? Can they be optimised? We've tested ext3+acl and xfs as an underlying filesystem, with no difference. We also noticed this behaviour on win98 as a client. Tested on 3.0.2a and 3.0.4, smb.conf attached. In my own testings i've noticed that with security=share and with anon r/w public share samba doesn't do any getxattr calls and this whole tests completes within minutes, as it should. Also, on a linux client mounting the share via smbmount deleting also takes just minutes. -- Jure Peèar
On Sun, 25 Jul 2004 18:15:14 +0200 Jure Pe?ar <pegasus@nerv.eu.org> wrote:> Samba 3.0.x PDC in a mixed 9x/nt/xp environment, using acls. When trying > to delete a large number of folders from a folder, the 'preparing to > delete' stage takes about half hour. > > It's simple to repeat: set up samba pdc, join an xp client into the > domain, map a share with some 4000 folders and try to delete them. > > Strace of a smbd proccess in the 'preparing to delete' stage looks like > this: > > stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 > gettimeofday({1090703949, 903367}, NULL) = 0 > stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 > getxattr("/mnt/test/test/0220", "system.posix_acl_access", 0x11fffe940, > 132)= -1 ENODATA (No data available) > stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 > getxattr("/mnt/test/test/0220", "system.posix_acl_default", 0x11fffe940, > 132) = -1 ENODATA (No data available) > stat("/mnt/test/test/0220", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 > getxattr("/mnt/test/test/0220", "user.SAMBA_PAI", 0x1207f0aa0, 1024) = -1 > ENODATA (No data available) > > For every folder it wants to delete, it walks all the folders doing this. > So for 4k folders we get 16 milion such tests, which take waay too much > time. Are these loops necessary? Can they be optimised? > > We've tested ext3+acl and xfs as an underlying filesystem, with no > difference. We also noticed this behaviour on win98 as a client. > > Tested on 3.0.2a and 3.0.4, smb.conf attached. >More discoveries here: I have folder 'test' with 4k folders named from 0001 to 4096 (seq -w 4096 fed to mkdir). If i select folder 'test' and delete it, the 'preparing to delete' stage takes just seconds and the delete itself takes about 160s, which is about right. BUT if i open folder 'test', select all of the directories there and try to delete them, it takes almost half an hour to get the conformation dialog box, while smbd process is doing what above strace shows and eating all the available cpu. Any ideas why? -- Jure Pe?ar
On Mon, 26 Jul 2004 20:23:39 +0200 Jure Peèar <pegasus@nerv.eu.org> wrote:> On Mon, 26 Jul 2004 06:19:16 -0700 > "Esh, Andrew" <Andrew_Esh@adaptec.com> wrote: > > > Has an ACL cache been implemented in Jeremy's version of ACL support? > > I know Matt Zinkevicius had one in his ACL code, and it drastically > > improved performance in situations where the folder path was being > > traversed during a permissions inheritance check. > > ACL cache sounds exactly like what we'd need. Are there any patches > floating aronund? I'm very willing to test them.We have some more interestings finds regarding this matter: The situation we're having problems with is having shares on samba PDC box; this way, ACL lookups are a showstopper. If we set up another samba 3.0 box as a domain member to that existing PDC and set up some shares on it, the performance is as it should be. Deleting a ~4k folders folder now takes only ~10s for 'preparing to delete' and 70s for actual delete. Right now we're trying to understand why this problem only comes up with shares on the PDC box. -- Jure Peèar