joop.martens@lekkerland.nl
2005-Jun-28 15:38 UTC
Fw: [Samba] Missing files on Windows client side
Jeremy,
I think we found the part of source where the problem resides:
smbd/trans2.c
+1444 for (i=0;(i<maxentries) && !finished &&
!out_of_space;i++)
{
+1445 BOOL got_exact_match = False;
+1446
+1447 /* this is a heuristic to avoid seeking the dirptr
except when
+1448 absolutely necessary. It allows for a
filename of about 40 chars */
+1449 if (space_remaining < DIRLEN_GUESS &&
numentries >
0) {
+1450 out_of_space = True;
+1451 finished = False;
+1452 } else {
+1453 finished = !get_lanman2_dir_entry(conn,
+1454 inbuf, outbuf,
+1455 mask,dirtype,info_level,
+1456
requires_resume_key,dont_descend,
+1457 &p,pdata,space_remaining,
&out_of_space, &got_exact_match,
+1458 &last_entry_off);
+1459 }
+1460
+1461 if (finished && out_of_space)
+1462 finished = False;
+1463
+1464 if (!finished && !out_of_space)
+1465 numentries++;
+1466
+1467 /*
+1468 * As an optimisation if we know we aren't looking
+1469 * for a wildcard name (ie. the name matches the
wildcard exactly)
+1470 * then we can finish on any (first) match.
+1471 * This speeds up large directory searches. JRA.
+1472 */
+1473
+1474 if(got_exact_match)
+1475 finished = True;
+1476
+1477 space_remaining = max_data_bytes -
PTR_DIFF(p,pdata);
+1478 DEBUG(5,("call_trans2findfirst - space remaining
%d numentries %d\n", space_remaining, numentries));
+1479 }
at line 1478 we added an extra debug line to see what's going on.
with every file the space_remaining is going down from 16384
(max_data_bytes) to 0 in varied steps of 100 till 150 bytes.
just before the out of space error we see the following:
space_remaining=4 bytes or something like that. And at the same time
numentries=121
get_lanman2_dir_entry found ./DEBITEUR.03062005.13
fname=DEBITEUR.03062005.13
[2005/06/28 16:32:41, 5] smbd/trans2.c:call_trans2findfirst(1478)
call_trans2findfirst - space remaining 412 numentries 118
<-------
[2005/06/28 16:32:41, 8] smbd/trans2.c:get_lanman2_dir_entry(869)
get_lanman2_dir_entry:readdir on dirptr 0x2003f598 now at offset 0
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode(283)
dos_mode: ./DEBITEUR.03062005.14
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode_from_sbuf(151)
dos_mode_from_sbuf returning
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode(315)
dos_mode returning
[2005/06/28 16:32:41, 5] smbd/trans2.c:get_lanman2_dir_entry(959)
get_lanman2_dir_entry found ./DEBITEUR.03062005.14
fname=DEBITEUR.03062005.14
[2005/06/28 16:32:41, 5] smbd/trans2.c:call_trans2findfirst(1478)
call_trans2findfirst - space remaining 276 numentries 119
<-------
[2005/06/28 16:32:41, 8] smbd/trans2.c:get_lanman2_dir_entry(869)
get_lanman2_dir_entry:readdir on dirptr 0x2003f598 now at offset 0
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode(283)
dos_mode: ./DEBITEUR.03062005.15
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode_from_sbuf(151)
dos_mode_from_sbuf returning
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode(315)
dos_mode returning
[2005/06/28 16:32:41, 5] smbd/trans2.c:get_lanman2_dir_entry(959)
get_lanman2_dir_entry found ./DEBITEUR.03062005.15
fname=DEBITEUR.03062005.15
[2005/06/28 16:32:41, 5] smbd/trans2.c:call_trans2findfirst(1478)
call_trans2findfirst - space remaining 140 numentries 120
[2005/06/28 16:32:41, 8] smbd/trans2.c:get_lanman2_dir_entry(869)
get_lanman2_dir_entry:readdir on dirptr 0x2003f598 now at offset 0
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode(283)
dos_mode: ./DEBITEUR.03062005.16
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode_from_sbuf(151)
dos_mode_from_sbuf returning
[2005/06/28 16:32:41, 8] smbd/dosmode.c:dos_mode(315)
dos_mode returning
[2005/06/28 16:32:41, 5] smbd/trans2.c:get_lanman2_dir_entry(959)
get_lanman2_dir_entry found ./DEBITEUR.03062005.16
fname=DEBITEUR.03062005.16
[2005/06/28 16:32:41, 5] smbd/trans2.c:call_trans2findfirst(1478)
call_trans2findfirst - space remaining 4 numentries 121
<-------
[2005/06/28 16:32:41, 5] smbd/trans2.c:call_trans2findfirst(1478)
call_trans2findfirst - space remaining 4 numentries 121
<------- ?
[2005/06/28 16:32:41, 9] smbd/trans2.c:send_trans2_replies(556)
t2_rep: params_sent_thistime = 10, data_sent_thistime = 16380,
useable_space = 131010
[2005/06/28 16:32:41, 9] smbd/trans2.c:send_trans2_replies(558)
t2_rep: params_to_send = 10, data_to_send = 16380, paramsize = 10,
datasize = 16380
[2005/06/28 16:32:41, 6] lib/util_sock.c:write_socket(449)
write_socket(24,16452)
[2005/06/28 16:32:41, 6] lib/util_sock.c:write_socket(452)
write_socket(24,16452) wrote 16452
[2005/06/28 16:32:41, 4] smbd/trans2.c:call_trans2findfirst(1519)
SMBtrans2 mask=* directory=./ dirtype=22 numentries=121
[2005/06/28 16:32:41, 6] smbd/process.c:process_smb(1090)
got message type 0x0 of len 0x7a
[2005/06/28 16:32:41, 3] smbd/process.c:process_smb(1091)
Transaction 9 of length 126
So the next time that get_lanman2_dir_entry is called there is not enough
memory to put in the next entry.
Every time this out of space error occurs we can see that the file is
called with get_lanman2_dir_entry, but is not send to windows.
Then the function call_trans2findfirst goes further with the remaining
files and run well till the space_remaining is too low again and then
another file is lost in the dir output.
We also notice that maxentries=1644 and max_data_bytes=16384. maxentries
appeared to be calculated. But I could not found out so far
how max_data_bytes was calculated. Can this be increased? Or could it be
that DIRLEN_GUESS is too small?
So my idea is that in AIX 5.2 ML5/6 + latest fixes the number of bytes used
for direntries consumes more memory that in linux.
Can you help us further with this?
Met vriendelijke groet, / With kind regards,
J. Martens
Afd. Automatisering van Lekkerland Nederland en Conway Belgi?
Dep. ICT from Lekkerland The Netherlands and Conway Belgium
E-mail: joop.martens@lekkerland.nl
Jeremy Allison
<jra@samba.org>
To
27/06/2005 17:54 joop.martens@lekkerland.nl
cc
jra@samba.org
Please respond to Subject
Jeremy Allison Re: Fw: [Samba] Missing files on
<jra@samba.org> Windows client side
On Mon, Jun 27, 2005 at 09:42:25AM +0200, joop.martens@lekkerland.nl
wrote:>
> Jeremy,
>
> Is there something I can do for you to help you solving this problem?
> Because the problem also exists on Irix and AIX with all latest fixes
> installed i don't think this is a bug within AIX.
>
> I could open a software call at IBM but then i have to be sure that
it's
> not a Samba problem.
>
> Met vriendelijke groet, / With kind regards,
Talk to James Peach at SGI. I'd like to get him to reproduce your
problem on IRIX before I think it's a Samba problem.
He's on the mailing list archives.
Jeremy.
----------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you
are not the intended receipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.
Deze e-mail kan vertrouwelijke en/of persoonlijke informatie bevatten. Als
u niet de bedoelde ontvanger bent, of deze e-mail per ongeluk heeft
gekregen, breng dan de verzender ogenblikkelijk hiervan op de hoogte en
vernietig dit bericht.
Het kopieren, in de openbaarheid brengen of verspreiden van de inhoud van
deze e-mail is ten strengste verboden.
Ce courriel peut contenir de l?information confidentielle et/ou
privil?gi?e. Si vous n??tes pas le r?cepteur intentionn? (ou vous avez re?u
ce courriel par erreur) svp, avertissez imm?diatement l??metteur et
d?truisez ce courriel.
Toute copie non autoris?e, divulgation d?information ou distribution du
contenu de ce courriel sont strictement interdites.
----------------------------------------------------------------------------------------------------------------------------------------------------------------
On Tue, Jun 28, 2005 at 05:38:26PM +0200, joop.martens@lekkerland.nl wrote:> So the next time that get_lanman2_dir_entry is called there is not enough > memory to put in the next entry. > Every time this out of space error occurs we can see that the file is > called with get_lanman2_dir_entry, but is not send to windows.That is expected. Not the following code used to detect out of space: if (PTR_DIFF(p,pdata) > space_remaining) { /* Move the dirptr back to prev_dirpos */ dptr_SeekDir(conn->dirptr, prev_dirpos); *out_of_space = True; DEBUG(9,("get_lanman2_dir_entry: out of space\n")); return False; /* Not finished - just out of space */ } So, at the out of space error we seek the dirptr back to the position just before we read the file entry from the directory. See this code above at the start of the loop : /* Needed if we run out of space */ long curr_dirpos = prev_dirpos = dptr_TellDir(conn->dirptr); dname = dptr_ReadDirName(conn->dirptr,&curr_dirpos,&sbuf); The logic goes as follows : Save prev_dirpos (using TellDir) ReadDirName to get next file. If space available, add to output buffer. If space not available, seek back to prev_dirpos (so the net ReadDirName will return the same name again) and then return. The next call from the client, FindNext, goes into get_lanman2_dir_entry() and reads the next name - which should be the same as the one rejected previously for out of space as we seeked back to the point before we read it. I have tested this logic on Linux, and on a large directory I get the "out of space" errors in the log too, but never lose any files as the next client findnext() call returns the previous name rejected. I still suspect either the telldir or seekdir calls on your version of AIX of having a problem here. Jeremy.