paul at gorilla-computing.de
2012-Aug-01 16:01 UTC
[Samba] Subject: samba 3.6.6: unstable network behaviour on win7 clients + segfault in smbstatus
Dear list,
I'm experiencing some severe problems on three independent standalone
samba file servers:
1. Strange behaviour on Windows 7 Clients
- sometimes users can't delete files or folders - Samba would delete
the files (they're gone off the filesystem) but still show them in the
explorer with a zero filesize, throwing a 'permission denied' message if
one would try to delete them a second time
- a workaround for this seems to be to MOVE the files to the desktop
instead and then delete them locally?!
- when some heavy program (i.e. AutoCAD) attempts to save a document
it starts a series of file operations (create temp, rename, rename,
delete) it often wouldn't succeed the first but the second or third time
you press the 'Save' button (error 'can't save .dwg file. save
as .tmp!')
2. 'smbstatus' on the server quits with a segmentation fault when trying
to list the locked files. it wouldn't do that if there are no or just a
few files open. But as soon as there is enough traffic it does so very
reliable:
$tail /var/log/messages
...
Aug 1 08:52:36 data kernel: [47608.306972] smbstatus[5331]: segfault at
7f808615a2c0 ip 00007f807f2244cf sp 00007fff5c4697b8 error 4 in
*libc-2.13.so*[7f807f10d000+17d000]
- it sounds like a network problem (like some kind of timeout maybe) but
its reproducable on three different servers in three different offices
on three different hardware setups, some pure gigabit networks, some
mixed (but all same software). the biggest office has the most problems
though.
- it seems that smbstatus' segfault and the weird network behaviour
depend on each other - I haven't seen any of the errors alone
- I tested the memory and the CPU without error and any other
application on the servers behaves normally
- the system run succesfully for over six months, before the error
started to appear
- deleting the SAMBA cache didn't seem to change anything
- it doesn't depend on a recent switch of kernel versions - same
behaviour on 3.1.0-1 to 3.2.0-3
I tried to raise the loglevel (= 5) to watch one of those fileoperations
fail, but its a lot of output that seems mostly normal to me. there're
some 'file not found' messages though:
[2012/07/31 18:33:25.609210, 5] smbd/open.c:1690(open_file_ntcreate)
open_file_ntcreate: FILE_OPEN requested for file Temp/111127_test.dwk
and file doesn't exist.
I'm not sure if its related but if I see the address of smbstatus'
segfault and then read the strace output of one of those smbd processes
it fails to query, I see a lot of failed readv system calls that point
to a very nearby address:
readv(10, 0x7fb6d80ca960, 1) = -1 EAGAIN (Resource
temporarily unavailable)
This error (if it is one) appears very often in the strace output,
sometimes every 15 to 20 lines (which adds to a lot!)
I couldn't really find out if its the same block though as I don't know
how to determine which process occupies a given memory address
(searching through the/proc/ID/maps output seemed rather cumbersome -
I'm sure there is a better way..)
I really doubt its a bug in 'libc', but maybe there is a wrong pointer
reference in whereever this system call is triggered? At least to me
those mistakes slip easy of the pen and the error code seems reasonable.
The samba server depends on an awful lot of libraries and I'm definitely
not skilled enough to debug it by myself.
Its quite hard to track down and I didn't want to open up a bugreport
without asking for help first. Especially as I think IF it was a bug the
internet should be full of posts about it already..
Many thanks in advance. If you need more information please let me know.
Paul
-- System Information:
Debian Release: wheezy/sid
APT prefers testing
APT policy: (990, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 3.2.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Versions of packages samba depends on:
ii adduser 3.113+nmu3
ii debconf [debconf-2.0] 1.5.44
ii dpkg 1.16.4.3
ii libacl1 2.2.51-8
ii libattr1 1:2.4.46-8
ii libc6 2.13-33
ii libcap2 1:2.22-1
ii libcomerr2 1.42.4-3
ii libcups2 1.5.3-1
ii libgssapi-krb5-2 1.10.1+dfsg-1
ii libk5crypto3 1.10.1+dfsg-1
ii libkrb5-3 1.10.1+dfsg-1
ii libldap-2.4-2 2.4.31-1
ii libpam-modules 1.1.3-7.1
ii libpam-runtime 1.1.3-7.1
ii libpam0g 1.1.3-7.1
ii libpopt0 1.16-7
ii libtalloc2 2.0.7+git20120207-1
ii libtdb1 1.2.10-2
ii libwbclient0 2:3.6.6-2
ii lsb-base 4.1+Debian7
ii procps 1:3.3.3-2
ii samba-common 2:3.6.6-2
ii update-inetd 4.43
ii zlib1g 1:1.2.7.dfsg-13
Versions of packages samba recommends:
ii logrotate 3.8.1-4
ii tdb-tools 1.2.10-2
Versions of packages samba suggests:
pn ctdb <none>
pn ldb-tools <none>
pn openbsd-inetd | inet-superserver <none>
pn smbldap-tools <none>
/etc/samba/smb.conf
[global]
passdb backend = ldapsam:ldap://127.0.0.1/
ldap suffix = dc=domain,dc=tld
ldap admin dn = cn=admin,dc=domain,dc=tld
ldap user suffix = ou=Users
ldap group suffix = ou=Group
ldap machine suffix = ou=Machine
ldap suffix = dc=domain,dc=tld
ldap ssl = no
ldap passwd sync = Yes
max protocol = SMB2
client ntlmv2 auth = yes
send spnego principal = no
wins support = no
wins server = 192.168.150.2
preferred master = no
domain master = no
dns proxy = no
workgroup = HQ
server string = %h
interfaces = 127.0.0.1 192.168.150.1
bind interfaces only = yes
log file = /var/log/samba/log.%m
log level = 1
max log size = 1000
syslog = 0
panic action = /usr/share/samba/panic-action %d
security = user
encrypt passwords = true
obey pam restrictions = yes
unix password sync = no
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n
*Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
pam password change = yes
printing = cups
printcap name = cups
load printers = yes
socket options = TCP_NODELAY
time server = yes
invalid users = root
hosts allow = 192.168.150.0/24
shared libraries:
$cat /proc/smbd-ID/maps (excerpt)
7f4a13136000-7f4a13141000 r-xp 00000000 09:01 25952422
/lib/x86_64-linux-gnu/libnss_files-2.13.so
7f4a13342000-7f4a1334c000 r-xp 00000000 09:01 25952387
/lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f4a1354d000-7f4a13554000 r-xp 00000000 09:01 25952317
/lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f4a14378000-7f4a14b59000 rw-s 00000000 00:0e 11473
/var/run/samba/locking.tdb
7f4a14b59000-7f4a14b91000 rw-s 00000000 00:0e 11471
/var/run/samba/connections.tdb
7f4a14b91000-7f4a14bc3000 rw-s 00000000 00:0e 11470
/var/run/samba/sessionid.tdb
7f4a14bc3000-7f4a14c2b000 rw-s 00000000 00:0e 11455
/var/run/samba/gencache.tdb
7f4a14c2b000-7f4a14c93000 rw-s 00000000 00:0e 11456
/var/run/samba/gencache_notrans.tdb
7f4a14c93000-7f4a14c95000 r-xp 00000000 09:01 2236086
/usr/lib/x86_64-linux-gnu/gconv/IBM850.so
7f4a14e96000-7f4a14efd000 rw-s 00000000 09:01 23724812
/var/lib/samba/account_policy.tdb
7f4a14efd000-7f4a14f32000 r--s 00000000 09:01 23728297
/var/cache/nscd/group
7f4a14f32000-7f4a14f67000 r--s 00000000 09:01 23728231
/var/cache/nscd/passwd
7f4a14f67000-7f4a14f6a000 r-xp 00000000 09:01 2229366
/usr/lib/x86_64-linux-gnu/gconv/UTF-16.so
7f4a1516b000-7f4a151af000 r-xp 00000000 09:01 25952280
/lib/x86_64-linux-gnu/libdbus-1.so.3.7.1
7f4a153b1000-7f4a153b4000 r-xp 00000000 09:01 25952259
/lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7f4a155b4000-7f4a155c5000 r-xp 00000000 09:01 2228425
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7f4a157c6000-7f4a157d6000 r-xp 00000000 09:01 2234113
/usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
7f4a159d7000-7f4a15a58000 r-xp 00000000 09:01 25952349
/lib/x86_64-linux-gnu/libm-2.13.so
7f4a15c59000-7f4a15c69000 r-xp 00000000 09:01 2229380
/usr/lib/x86_64-linux-gnu/libavahi-client.so.3.2.9
7f4a15e6b000-7f4a15e77000 r-xp 00000000 09:01 2228355
/usr/lib/x86_64-linux-gnu/libavahi-common.so.3.5.3
7f4a16078000-7f4a1607b000 r-xp 00000000 09:01 25952286
/lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f4a1627c000-7f4a16284000 r-xp 00000000 09:01 2231897
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f4a16485000-7f4a1649c000 r-xp 00000000 09:01 25952447
/lib/x86_64-linux-gnu/libpthread-2.13.so
7f4a166a1000-7f4a1671b000 r-xp 00000000 09:01 25952333
/lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7f4a1691f000-7f4a169d8000 r-xp 00000000 09:01 2231131
/usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
7f4a16bdf000-7f4a16bf9000 r-xp 00000000 09:01 2231285
/usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f4a16dfa000-7f4a16f77000 r-xp 00000000 09:01 25952257
/lib/x86_64-linux-gnu/libc-2.13.so
7f4a17181000-7f4a17197000 r-xp 00000000 09:01 25952262
/lib/x86_64-linux-gnu/libz.so.1.2.7
7f4a17398000-7f4a173a4000 r-xp 00000000 09:01 2228403
/usr/lib/x86_64-linux-gnu/libwbclient.so.0
7f4a175a6000-7f4a175b7000 r-xp 00000000 09:01 2230176
/usr/lib/x86_64-linux-gnu/libtdb.so.1.2.10
7f4a177b7000-7f4a177c2000 r-xp 00000000 09:01 2228234
/usr/lib/x86_64-linux-gnu/libtalloc.so.2.0.7
7f4a179c2000-7f4a179cd000 r-xp 00000000 09:01 25952455
/lib/x86_64-linux-gnu/libpopt.so.0.0.0
7f4a17bcf000-7f4a17bd6000 r-xp 00000000 09:01 25952466
/lib/x86_64-linux-gnu/librt-2.13.so
7f4a17dd7000-7f4a17dd9000 r-xp 00000000 09:01 25952391
/lib/x86_64-linux-gnu/libdl-2.13.so
7f4a17fdb000-7f4a17ff0000 r-xp 00000000 09:01 25952371
/lib/x86_64-linux-gnu/libnsl-2.13.so
7f4a181f3000-7f4a18206000 r-xp 00000000 09:01 25952269
/lib/x86_64-linux-gnu/libresolv-2.13.so
7f4a18409000-7f4a1840d000 r-xp 00000000 09:01 25821223
/lib/libcap.so.2.22
7f4a1860d000-7f4a18611000 r-xp 00000000 09:01 25952344
/lib/x86_64-linux-gnu/libattr.so.1.1.0
7f4a18812000-7f4a1881a000 r-xp 00000000 09:01 25952311
/lib/x86_64-linux-gnu/libacl.so.1.1.0
7f4a18a1b000-7f4a18a28000 r-xp 00000000 09:01 25952341
/lib/x86_64-linux-gnu/libpam.so.0.83.0
7f4a18c29000-7f4a18c31000 r-xp 00000000 09:01 25952290
/lib/x86_64-linux-gnu/libcrypt-2.13.so
7f4a18e60000-7f4a18eb3000 r-xp 00000000 09:01 2231630
/usr/lib/x86_64-linux-gnu/libcups.so.2
7f4a190b8000-7f4a190bb000 r-xp 00000000 09:01 25952296
/lib/x86_64-linux-gnu/libcom_err.so.2.1
7f4a192bc000-7f4a192e2000 r-xp 00000000 09:01 2229329
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f4a194e5000-7f4a195ae000 r-xp 00000000 09:01 2231878
/usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f4a197b9000-7f4a197f5000 r-xp 00000000 09:01 2228790
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f4a199f8000-7f4a19a06000 r-xp 00000000 09:01 2228823
/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
7f4a19c07000-7f4a19c53000 r-xp 00000000 09:01 2228822
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3 (2.4.31-1)
7f4a19e58000-7f4a19e78000 r-xp 00000000 09:01 25952488
/lib/x86_64-linux-gnu/ld-2.13.so
7f4a19e85000-7f4a19e9d000 rw-s 00000000 00:0e 11472
/var/run/samba/brlock.tdb
7f4a19e9d000-7f4a19eab000 rw-s 00000000 00:0e 11449
/var/run/samba/messages.tdb
7f4a19eab000-7f4a19f12000 rw-s 00000000 09:01 23724655
/var/lib/samba/share_info.tdb
7f4a19f13000-7f4a19f15000 rw-s 00000000 00:0e 12357
/var/run/samba/serverid.tdb
7f4a19f15000-7f4a19f16000 rw-s 00000000 00:0e 11475
/var/run/samba/notify_onelevel.tdb
7f4a19f16000-7f4a19f17000 rw-s 00000000 00:0e 11474
/var/run/samba/notify.tdb
7f4a19f2d000-7f4a19f62000 r--s 00000000 09:01 23727081
/var/cache/nscd/hosts
7f4a19f62000-7f4a19f82000 r--s 00000000 09:01 2228610
/usr/share/samba/lowcase.dat
7f4a19f82000-7f4a19fa2000 r--s 00000000 09:01 2228613
/usr/share/samba/upcase.dat
7f4a19fa2000-7f4a1a00b000 rw-s 00000000 09:01 23724558
/var/lib/samba/secrets.tdb
7f4a1a00b000-7f4a1a012000 r--s 00000000 09:01 2233616
/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f4a1a012000-7f4a1a032000 r--s 00000000 09:01 2228610
/usr/share/samba/lowcase.dat
7f4a1a032000-7f4a1a052000 r--s 00000000 09:01 2228613
/usr/share/samba/upcase.dat
7f4a1a06f000-7f4a1a072000 rw-s 00000000 00:0e 11476
/var/run/samba/printer_list.tdb
7f4a1a074000-7f4a1a075000 rw-s 00000000 09:01 23724106
/var/cache/samba/login_cache.tdb
7f4a1a077000-7f4a1a078000 r--p 0001f000 09:01 25952488
/lib/x86_64-linux-gnu/ld-2.13.so
Possibly Parallel Threads
Wisdom of the Ancients
