Our software utilizes an OS X Server (Snow Leopard 10.6.8) for file
read and writes. We gather files out of many directories, process
them, and then write files back to the server.
We had a long time end user who utilizes our software with an XP box
upgrade from XP to Windows 7.
Now with Windows 7 the SMBD is crashing routinely under heavy loads.
Whatever information it was working on at the time is corrupted. EVEN
WORSE the SMBD respawns silently under a new process and reconnects,
so we don't know that the smb server crashed or that the data has
become corrupt.
The problem is intermitant, but frequent. I'll throw 16 duplicates of
the same file at the system and it will fail on, say, files 11 and
16. This is the same file under a different name.
Here's a sample crash log (they're all about the same.)
Process: smbd [96275]
Path: /usr/sbin/smbd
Identifier: smbd
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: smbd [871]
Date/Time: 2012-04-17 14:24:53.924 -0500
OS Version: Mac OS X Server 10.6.8 (10K549)
Report Version: 6
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Crashed Thread: 0
Thread 0 Crashed:
0 libSystem.B.dylib 0x00007fff8a25ce20 strcmp + 80
1 odsam.dylib 0x000000010008ee95 0x100088000 +
28309
2 odsam.dylib 0x000000010008f1e4 0x100088000 +
29156
3 odsam.dylib 0x000000010008f3f6 0x100088000 +
29686
4 odsam.dylib 0x0000000100093455 init_module +
15567
5 odsam.dylib 0x00000001000941b1 init_module +
18987
6 smbd 0x0000000137429f2d
pdb_getsampwnam + 32
7 odsam.dylib 0x000000010035f78d init_module +
8191
8 smbd 0x000000013748f8b1
add_mapping_entry + 2591
9 smbd 0x0000000137496ab7
auth_ntlmssp_start + 575
10 smbd 0x000000013736fc09
ntlmssp_set_password + 4447
11 smbd 0x000000013736e476
ntlmssp_update + 583
12 smbd 0x000000013749680e
auth_ntlmssp_update + 43
13 smbd 0x00000001373cee41
rpc_pipe_register_commands + 907
14 smbd 0x00000001373d0140
api_pipe_bind_auth3 + 772
15 smbd 0x00000001373c96d6
init_rpc_pipe_hnd + 3771
16 smbd 0x00000001373c9ffd
init_rpc_pipe_hnd + 6114
17 smbd 0x00000001372ec819
reply_pipe_write_and_X + 360
18 smbd 0x000000013730026b
reply_write_and_X + 499
19 smbd 0x000000013732e695
schedule_deferred_open_smb_message + 2481
20 smbd 0x000000013732fd87 smbd_process
+ 2360
21 smbd 0x00000001374f8183 main + 4456
22 smbd 0x00000001372bccf0 start + 52
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000c10 rbx: 0x000000010010c580 rcx:
0x0000000102a00000 rdx: 0x00000000000000c1
rdi: 0x0000000000000000 rsi: 0x00000001001143f0 rbp:
0x00007fff5fbf9800 rsp: 0x00007fff5fbf97e8
r8: 0x0000000060c10410 r9: 0x000000000000001d r10:
0x00000000102a050b r11: 0x00007fff8a2604b9
r12: 0x0000000000000000 r13: 0x0000000000000000 r14:
0x00007fff5fbf9980 r15: 0x0000000137600df8
rip: 0x00007fff8a25ce20 rfl: 0x0000000000010202 cr2:
0x0000000000000000
Binary Images:
0x100088000 - 0x10009afff odsam.dylib ??? (???)
<39DE0ED4-5FAB-4D22-2DA7-B2D87EF4EE4F> /usr/lib/samba/pdb/odsam.dylib
0x1000f2000 - 0x1000f8fff darwin_streams.dylib ???
(???) <733DDD25-1228-5364-7930-69B65F048322> /usr/lib/samba/vfs/
darwin_streams.dylib
0x100356000 - 0x100363ff7 odsam.dylib ??? (???)
<217BA59F-853C-EE95-2129-C315B11D11DC> /usr/lib/samba/auth/odsam.dylib
0x10037e000 - 0x100386ff7 com.apple.ByteRangeLocking
1.1.2 (1.1.2) <71331B90-233E-F3FA-DDB0-E257800D2F8F> /System/Library/
PrivateFrameworks/ByteRangeLocking.framework/Versions/A/ByteRangeLocking
0x1005bb000 - 0x1005c0fff darwinacl.dylib ??? (???)
<4DD79B8E-4F66-FA89-E5DA-D6EEDC4044E5> /usr/lib/samba/vfs/
darwinacl.dylib
0x1005c5000 - 0x1005c6fff notify_kqueue.dylib ???
(???) <CBB7D7DC-3D11-B628-6BBF-503A24548363> /usr/lib/samba/vfs/
notify_kqueue.dylib
0x1372bc000 - 0x1375ebff7 +smbd ??? (???)
<27BF9F95-6C18-4AAB-30B0-5F681DE2CF2F> /usr/sbin/smbd
0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???)
<486E6C61-1197-CC7C-2197-82CE505102D7> /usr/lib/dyld
0x7fff80dc3000 - 0x7fff80dc3ff7 com.apple.CoreServices 44
(44) <DC7400FB-851E-7B8A-5BF6-6F50094302FB> /System/Library/Frameworks/
CoreServices.framework/Versions/A/CoreServices
0x7fff80dc4000 - 0x7fff80dc8ff7 libmathCommon.A.dylib
315.0.0 (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> /
usr/lib/system/libmathCommon.A.dylib
0x7fff80ea3000 - 0x7fff80eceff7 libxslt.1.dylib 3.24.0
(compatibility 3.0.0) <8AB4CA9E-435A-33DA-7041-904BA7FA11D5> /usr/lib/
libxslt.1.dylib
0x7fff80f6d000 - 0x7fff8102afff
com.apple.CoreServices.OSServices 359.2 (359.2)
<BBB8888E-18DE-5D09-3C3A-F4C029EC7886> /System/Library/Frameworks/
CoreServices.framework/Versions/A/Frameworks/OSServices.framework/
Versions/A/OSServices
0x7fff8145e000 - 0x7fff81474fef libbsm.0.dylib ??? (???)
<42D3023A-A1F7-4121-6417-FCC6B51B3E90> /usr/lib/libbsm.0.dylib
0x7fff8175f000 - 0x7fff81799fff libcups.2.dylib 2.8.0
(compatibility 2.0.0) <7982734A-B66B-44AA-DEEC-364D2C10009B> /usr/lib/
libcups.2.dylib
0x7fff8179a000 - 0x7fff8182afff com.apple.SearchKit 1.3.0
(1.3.0) <4175DC31-1506-228A-08FD-C704AC9DF642> /System/Library/
Frameworks/CoreServices.framework/Versions/A/Frameworks/
SearchKit.framework/Versions/A/SearchKit
0x7fff81c3a000 - 0x7fff81c3ffff libpam.2.dylib 3.0.0
(compatibility 3.0.0) <97F037FC-0CD8-D4B3-8133-7D7013791F86> /usr/lib/
libpam.2.dylib
0x7fff8266e000 - 0x7fff8268eff7
com.apple.DirectoryService.Framework 3.6 (621.11) <AD76C757-6701-
BDB5-631E-1CB77D669586> /System/Library/Frameworks/
DirectoryService.framework/Versions/A/DirectoryService
0x7fff826db000 - 0x7fff826ecff7 libz.1.dylib 1.2.3
(compatibility 1.0.0) <97019C74-161A-3488-41EC-A6CA8738418C> /usr/lib/
libz.1.dylib
0x7fff826ed000 - 0x7fff827a3ff7 libobjc.A.dylib 227.0.0
(compatibility 1.0.0) <03140531-3B2D-1EBA-DA7F-E12CC8F63969> /usr/lib/
libobjc.A.dylib
0x7fff82a23000 - 0x7fff82b42fe7 libcrypto.0.9.8.dylib 0.9.8
(compatibility 0.9.8) <14115D29-432B-CF02-6B24-A60CC533A09E> /usr/lib/
libcrypto.0.9.8.dylib
0x7fff82b85000 - 0x7fff82c02fef libstdc++.6.dylib 7.9.0
(compatibility 7.0.0) <35ECA411-2C08-FD7D-11B1-1B7A04921A5C> /usr/lib/
libstdc++.6.dylib
0x7fff82c60000 - 0x7fff82eeafe7 com.apple.security 6.1.2
(55002) <FD0B5AD4-74DB-7ED8-90D3-6EC56FFA8557> /System/Library/
Frameworks/Security.framework/Versions/A/Security
0x7fff82eeb000 - 0x7fff82f8bfff com.apple.LaunchServices
362.3 (362.3) <B90B7C31-FEF8-3C26-BFB3-D8A48BD2C0DA> /System/Library/
Frameworks/CoreServices.framework/Versions/A/Frameworks/
LaunchServices.framework/Versions/A/LaunchServices
0x7fff83050000 - 0x7fff83384fef
com.apple.CoreServices.CarbonCore 861.39 (861.39) <1386A24D-
DD15-5903-057E-4A224FAF580B> /System/Library/Frameworks/
CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/
Versions/A/CarbonCore
0x7fff836cf000 - 0x7fff8370afff com.apple.AE 496.5 (496.5)
<208DF391-4DE6-81ED-C697-14A2930D1BC6> /System/Library/Frameworks/
CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
0x7fff8370b000 - 0x7fff8370cff7
com.apple.TrustEvaluationAgent 1.1 (1) <5952A9FA-
BC2B-16EF-91A7-43902A5C07B6> /System/Library/PrivateFrameworks/
TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
0x7fff83977000 - 0x7fff839b4fff com.apple.LDAPFramework 2.0
(120.1) <54A6769E-D7E2-DBE2-EA61-87B9EA355DA4> /System/Library/
Frameworks/LDAP.framework/Versions/A/LDAP
0x7fff839b5000 - 0x7fff839c3ff7 libkxld.dylib ??? (???)
<8145A534-95CC-9F3C-B78B-AC9898F38C6F> /usr/lib/system/libkxld.dylib
0x7fff84343000 - 0x7fff84349ff7 com.apple.DiskArbitration
2.3 (2.3) <857F6E43-1EF4-7D53-351B-10DE0A8F992A> /System/Library/
Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x7fff8434a000 - 0x7fff84396fff libauto.dylib ??? (???)
<F7221B46-DC4F-3153-CE61-7F52C8C293CF> /usr/lib/libauto.dylib
0x7fff84428000 - 0x7fff84450fff com.apple.DictionaryServices
1.1.2 (1.1.2) <E9269069-93FA-2B71-F9BA-FDDD23C4A65E> /System/Library/
Frameworks/CoreServices.framework/Versions/A/Frameworks/
DictionaryServices.framework/Versions/A/DictionaryServices
0x7fff859d8000 - 0x7fff85a38fe7 com.apple.framework.IOKit
2.0 (???) <4F071EF0-8260-01E9-C641-830E582FA416> /System/Library/
Frameworks/IOKit.framework/Versions/A/IOKit
0x7fff85a39000 - 0x7fff85b31ff7 libiconv.2.dylib 7.0.0
(compatibility 7.0.0) <44AADE50-15BC-BC6B-BEF0-5029A30766AC> /usr/lib/
libiconv.2.dylib
0x7fff85bd7000 - 0x7fff85c21ff7 com.apple.Metadata 10.6.3
(507.15) <DE238BE4-5E22-C4D5-CF5C-3D50FDEE4701> /System/Library/
Frameworks/CoreServices.framework/Versions/A/Frameworks/
Metadata.framework/Versions/A/Metadata
0x7fff873dc000 - 0x7fff873fdfff libresolv.9.dylib 41.1.0
(compatibility 1.0.0) <9410EC7F-4D24-6740-AFEE-90405750FAD7> /usr/lib/
libresolv.9.dylib
0x7fff87883000 - 0x7fff87895fe7 libsasl2.2.dylib 3.15.0
(compatibility 3.0.0) <76B83C8D-8EFE-4467-0F75-275648AFED97> /usr/lib/
libsasl2.2.dylib
0x7fff87dac000 - 0x7fff87de9ff7 libssl.0.9.8.dylib 0.9.8
(compatibility 0.9.8) <F743389F-F25A-A77D-4FCA-D6B01AF2EE6D> /usr/lib/
libssl.0.9.8.dylib
0x7fff87dea000 - 0x7fff87e9afff edu.mit.Kerberos 6.5.11
(6.5.11) <085D80F5-C9DC-E252-C21B-03295E660C91> /System/Library/
Frameworks/Kerberos.framework/Versions/A/Kerberos
0x7fff88242000 - 0x7fff88316fe7 com.apple.CFNetwork 454.12.4
(454.12.4) <C83E2BA1-1818-B3E8-5334-860AD21D1C80> /System/Library/
Frameworks/CoreServices.framework/Versions/A/Frameworks/
CFNetwork.framework/Versions/A/CFNetwork
0x7fff8861a000 - 0x7fff88731fef libxml2.2.dylib 10.3.0
(compatibility 10.0.0) <1B27AFDD-DF87-2009-170E-C129E1572E8B> /usr/lib/
libxml2.2.dylib
0x7fff887b2000 - 0x7fff88970fff libicucore.A.dylib 40.0.0
(compatibility 1.0.0) <4274FC73-A257-3A56-4293-5968F3428854> /usr/lib/
libicucore.A.dylib
0x7fff88983000 - 0x7fff88992fff com.apple.NetFS 3.2.2
(3.2.2) <7CCBD70E-BF31-A7A7-DB98-230687773145> /System/Library/
Frameworks/NetFS.framework/Versions/A/NetFS
0x7fff8a217000 - 0x7fff8a258fff
com.apple.SystemConfiguration 1.10.8 (1.10.2) <78D48D27-A9C4-62CA-2803-
D0BBED82855A> /System/Library/Frameworks/SystemConfiguration.framework/
Versions/A/SystemConfiguration
0x7fff8a259000 - 0x7fff8a41afef libSystem.B.dylib 125.2.11
(compatibility 1.0.0) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/
libSystem.B.dylib
0x7fff8a434000 - 0x7fff8a5abfe7 com.apple.CoreFoundation
6.6.6 (550.44) <BB4E5158-E47A-39D3-2561-96CB49FA82D4> /System/Library/
Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x7fff8a5ac000 - 0x7fff8a665fff libsqlite3.dylib 9.6.0
(compatibility 9.0.0) <2C5ED312-E646-9ADE-73A9-6199A2A43150> /usr/lib/
libsqlite3.dylib
0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???)
<9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib
----
We turned up the logging on the SMB protocol, and found this around
the time of the crash:
----------------------------------------------------------------------------------------------------------------------------------------------------------
[2012/04/20 15:19:34, 0, pid=225] /SourceCache/samba/samba-235.7/samba/
source/auth/auth_odsam.c:getusernode(85)
dsGetRecordEntry gave -14085 [eDSRefTableCSBPAllocError]
[2012/04/20 15:19:34, 2, pid=225] /SourceCache/samba/samba-235.7/samba/
source/auth/auth.c:check_ntlm_password(319)
check_ntlm_password: Authentication for user [rip] -> [rip] FAILED
with error NT_STATUS_NO_SUCH_USER
[2012/04/20 15:19:34, 3, pid=225] /SourceCache/samba/samba-235.7/samba/
source/smbd/sec_ctx.c:pop_sec_ctx(448)
pop_sec_ctx (1025, 20) - sec_ctx_stack_ndx = 0
[2012/04/20 15:19:34, 3, pid=225] /SourceCache/samba/samba-235.7/samba/
source/rpc_server/srv_pipe_hnd.c:process_complete_pdu(827)
process_complete_pdu: DCE/RPC fault sent on pipe winreg
----------------------
Now keep in mind that we've been authenticated as "rip" all along,
it
just decided some 19,000,000 UNIX System calls (not an exageration)
into the process that it couldn't locate the user "rip."
We're desperate for a solution, but so far all the tweaks we've done
on both the smb.conf and the Windows 7 registry have come up empty.
Can anyone point us in the direction of a solution?