thr3ads.net - samba - [Samba] "samba-tool backup offline" stuck [Nov 2019]

If this information is useful, please help other people find it:
Share via:

Andrea Venturoli

2019-Nov-13 08:23 UTC

[Samba] "samba-tool backup offline" stuck

On 2019-11-11 09:41, Andrew Bartlett wrote:
>>> and what is the full backtrace on that tdbbackup?
>>
>> You mean: compile with debug symbols, attach gdb and get stack trace?
>> If so, I'll do it, but I'll need some time.
> 
> Yes, that's what I need.
Please, find it below:

(gdb) bt
#0  0x0000000800fcdab8 in _fcntl () from /lib/libc.so.7
#1  0x0000000800c68908 in __thr_fcntl (fd=3, cmd=13)
     at /usr/src/lib/libthr/thread/thr_syscalls.c:202
#2  0x0000000800ed2127 in fcntl (fd=3, cmd=13) at 
/usr/src/lib/libc/sys/fcntl.c:54
#3  0x000000080082c83c in fcntl_lock (tdb=<optimized out>,
rw=<optimized
out>, off=8,
     len=<optimized out>, waitflag=<optimized out>) at 
../../common/lock.c:58
#4  tdb_brlock (tdb=0x80161f000, rw_type=3, offset=8, len=1, 
flags=<optimized out>)
     at ../../common/lock.c:200
#5  0x000000080082cd19 in tdb_nest_lock (tdb=0x80161f000, offset=8, 
ltype=3,
     flags=TDB_LOCK_WAIT) at ../../common/lock.c:390
#6  0x0000000800831808 in _tdb_transaction_start (tdb=0x80161f000, 
lockflags=TDB_LOCK_WAIT)
     at ../../common/transaction.c:490
#7  0x000000000040188b in backup_tdb (
     old_name=0x7fffffffddd6 "/var/db/samba4/private/secrets.ldb",
     new_name=0x80161c000
"/var/db/samba4/private/secrets.ldb.copy.tdb",
     hash_size=<optimized out>, nolock=<optimized out>, 
readonly=<optimized out>)
     at ../../tools/tdbbackup.c:157
#8  0x0000000000401528 in main (argc=<optimized out>, argv=0x7fffffffdb40)
     at ../../tools/tdbbackup.c:360


Don't know if this is enough, but I'll get more info if needed (just 
tell me which info).

  bye & Thanks
	av.

Andrew Bartlett

2019-Nov-13 08:30 UTC

head link

[Samba] "samba-tool backup offline" stuck

On Wed, 2019-11-13 at 09:23 +0100, Andrea Venturoli
wrote:> On 2019-11-11 09:41, Andrew Bartlett wrote:
> 
> > > > and what is the full backtrace on that tdbbackup?
> > > 
> > > You mean: compile with debug symbols, attach gdb and get stack
trace?
> > > If so, I'll do it, but I'll need some time.
> > 
> > Yes, that's what I need.
> 
> Please, find it below:
> (gdb) bt
> #0  0x0000000800fcdab8 in _fcntl () from /lib/libc.so.7
> #1  0x0000000800c68908 in __thr_fcntl (fd=3, cmd=13)
>      at /usr/src/lib/libthr/thread/thr_syscalls.c:202
> #2  0x0000000800ed2127 in fcntl (fd=3, cmd=13) at 
> /usr/src/lib/libc/sys/fcntl.c:54
> #3  0x000000080082c83c in fcntl_lock (tdb=<optimized out>,
rw=<optimized
> out>, off=8,
>      len=<optimized out>, waitflag=<optimized out>) at 
> ../../common/lock.c:58
> #4  tdb_brlock (tdb=0x80161f000, rw_type=3, offset=8, len=1, 
> flags=<optimized out>)
>      at ../../common/lock.c:200
> #5  0x000000080082cd19 in tdb_nest_lock (tdb=0x80161f000, offset=8, 
> ltype=3,
>      flags=TDB_LOCK_WAIT) at ../../common/lock.c:390
> #6  0x0000000800831808 in _tdb_transaction_start (tdb=0x80161f000, 
> lockflags=TDB_LOCK_WAIT)
>      at ../../common/transaction.c:490
> #7  0x000000000040188b in backup_tdb (
>      old_name=0x7fffffffddd6
"/var/db/samba4/private/secrets.ldb",
>      new_name=0x80161c000
"/var/db/samba4/private/secrets.ldb.copy.tdb",
>      hash_size=<optimized out>, nolock=<optimized out>, 
> readonly=<optimized out>)
>      at ../../tools/tdbbackup.c:157
> #8  0x0000000000401528 in main (argc=<optimized out>,
argv=0x7fffffffdb40)
>      at ../../tools/tdbbackup.c:360
> 
> 
> Don't know if this is enough, but I'll get more info if needed
(just
> tell me which info).
Thanks that shows what is normally the behaviour.

I was worried someone had patched fcntl for flock(), as some on Hurd
were trying to do (we told them not do, quite sternly).  

Regarding your problem, on linux I use lslocks to debug from here, you
can normally work out which process is holding the lock, which lock it
is waiting on (if any, marked with a *) and gdb that to work out what
is the matter with it, and perhaps why it is in that situation.

I'm not sure if that is available and within your skills set, but that
is how to progress this. 

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Andrea Venturoli

2019-Nov-13 12:11 UTC

head link

[Samba] "samba-tool backup offline" stuck

On 2019-11-13 09:30, Andrew Bartlett wrote:
> Regarding your problem, on linux I use lslocks to debug from here, you
> can normally work out which process is holding the lock, which lock it
> is waiting on (if any, marked with a *) and gdb that to work out what
> is the matter with it, and perhaps why it is in that situation.
Hmm...

I have no lslocks on FreeBSD; will lsof do?

In case, here's an excerpt:> samba     72462      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72462      root   46u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72466      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72466      root    8u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72469      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72469      root   32u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> winbindd  72475      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> winbindd  72475      root   12u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72476      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72476      root   25u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72487      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72487      root    8u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72488      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72488      root    8u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72489      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72489      root    8u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72494      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> smbd      72494      root    8u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72496      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72496      root   46u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72497      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72497      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72692      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72692      root   46u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72696      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72696      root   46u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72697      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72697      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72882      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     72882      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     73535      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     73535      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     73536      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     73536      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     73895      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     73895      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     76100      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     76100      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     76103      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     76103      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     85298      root  txt     VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
> samba     85298      root   35u    VREG       4,1302659313            
430080   32007 /usr/jails/dc/var/db/samba4/private/secrets.tdb
IIUIC, it seems I have around 20 processes using secrets.tdb read/write, 
but no locks!

So I tried stopping Samba completely: now no process has secrets.tdb 
open; still tdbbackup gets stuck in _fcntl!!!

I'm open to suggestions...

  bye & Thanks
	av.

samba - Nov 2019 - "samba-tool backup offline" stuck

[Samba] "samba-tool backup offline" stuck

[Samba] "samba-tool backup offline" stuck

[Samba] "samba-tool backup offline" stuck

Maybe Matching Threads