samba - Oct 2013 - Many "smbd" process with "D" uninterruptible sleep status

I got a samba server(Linux) to store some data in my PC(windows)
frequently.  And last night I found that samba can't work. After logined to
samba server and found that there's 91 smbd processes with status
"D"
totally.
# ps
    169 root         0        SW   [pdflush]
    170 root         0        DW   [pdflush]
....
29534 simon  20608 D    /sbin/smbd -D
29548 root       2792 D    /bin/sync
30160 simon  20608 D    /sbin/smbd -D
30474 simon  20608 D    /sbin/smbd -D
30496 simon  18676 D    /sbin/smbd -D
30673 simon  20504 D    /sbin/smbd -D
30810 simon  20504 D    /sbin/smbd -D
31302 simon  20608 D    /sbin/smbd -D
31965 simon  20608 D    /sbin/smbd -D
32288 simon  20608 D    /sbin/smbd -D
...
And after 5 mins the result of "ps" is the same.

The result of "top" shows below. The load average is VERY HIGHT and
CPU is
busy for io.
# top
Mem: 366504K used, 143900K free, 0K shrd, 14216K buff, 293224K cached
CPU:  1.7% usr  3.5% sys  0.0% nic  0.0% idle 92.9% io  1.7% irq  0.0% sirq
Load average: 103.10 103.03 102.65 1/149 5614
PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND
 5532  3325 root     R     3000  0.5  3.5 top
 2421     1 simon  D    20712  4.0  0.0 /sbin/smbd -D
  594     1 simon  D    20712  4.0  0.0 /sbin/smbd -D
 4130     1 simon  D    20712  4.0  0.0 /sbin/smbd -D
 1261     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
23452     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
30474     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
21641     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
 9053     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
 3068     1 simon  D    20608  4.0  0.0 /sbin/smbd -D

After reboot, the samba works fine. Until now I cannot reproduce yet.
I can't figure out what's going on.
Is it kind of bugs in kernel or samba?
Does pdflush crash first, then affect smbd and sync to crash?

Please help,
Thanks very much.


I used some options in smb.conf:
  max smbd process = 100
  max connections = 100

Linux Samba Server:
  Linux 2.6.31.8
  Samba 3.5.6
  Software RAID 1


Simon

On Thu, Oct 17, 2013 at 03:06:22PM +0800, Simon Wang
wrote:
> I got a samba server(Linux) to store some data in my PC(windows)
> frequently.  And last night I found that samba can't work. After
logined to
> samba server and found that there's 91 smbd processes with status
"D"
> totally.
> # ps
>     169 root         0        SW   [pdflush]
>     170 root         0        DW   [pdflush]
> ....
> 29534 simon  20608 D    /sbin/smbd -D
> 29548 root       2792 D    /bin/sync
> 30160 simon  20608 D    /sbin/smbd -D
> 30474 simon  20608 D    /sbin/smbd -D
> 30496 simon  18676 D    /sbin/smbd -D
> 30673 simon  20504 D    /sbin/smbd -D
> 30810 simon  20504 D    /sbin/smbd -D
> 31302 simon  20608 D    /sbin/smbd -D
> 31965 simon  20608 D    /sbin/smbd -D
> 32288 simon  20608 D    /sbin/smbd -D
> ...
> And after 5 mins the result of "ps" is the same.
> 
> The result of "top" shows below. The load average is VERY HIGHT
and CPU is
> busy for io.
Long-running D is almost always a kernel problem. D means
uninterruptible wait. smbd made some syscall that does not
return for a long time. With a working kernel no user space
process should be able to do that. Look at your syslog
(/var/log/messages or /var/log/syslog or such) for kernel
problems. Something else might be extremely slow storage
like a USB1 memory stick. It should eventually come back,
but this could look the same.

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de

Monday I got the same problem but in other situation.

We have 2 physical server (node1 et node2) and a software Lifekeeper 
(software doing HA, network raid and failover)

Monday : node2 crash
Lifekeeper do a auto-failover and all service start on node1
but node1 take high loading for unknow reason and samba not responding 
correctly.
And I see the same case than you, many smbd process in state D.

solution : hardware reset of server.

I'm afraid because we plan to replace these servers by new servers with 
samba 4 with same configuration.
I don't know if this issue is due to number of client who try to reconnect 
or other thing.

best regards

        St?phane

-----------------------------------
St?phane PURNELLE                         Admin. Syst?mes et R?seaux 
Service Informatique       Corman S.A.           Tel : 00 32 (0)87/342467

samba-bounces at lists.samba.org wrote on 17/10/2013 09:06:22:
> De : Simon Wang <simonnn.wang at gmail.com>
> A : samba at lists.samba.org, 
> Date : 17/10/2013 09:06
> Objet : [Samba] Many "smbd" process with "D"
uninterruptible sleep
status
> Envoy? par : samba-bounces at lists.samba.org
> 
> I got a samba server(Linux) to store some data in my PC(windows)
> frequently.  And last night I found that samba can't work. After
logined
to
> samba server and found that there's 91 smbd processes with status
"D"
> totally.
> # ps
>     169 root         0        SW   [pdflush]
>     170 root         0        DW   [pdflush]
> ....
> 29534 simon  20608 D    /sbin/smbd -D
> 29548 root       2792 D    /bin/sync
> 30160 simon  20608 D    /sbin/smbd -D
> 30474 simon  20608 D    /sbin/smbd -D
> 30496 simon  18676 D    /sbin/smbd -D
> 30673 simon  20504 D    /sbin/smbd -D
> 30810 simon  20504 D    /sbin/smbd -D
> 31302 simon  20608 D    /sbin/smbd -D
> 31965 simon  20608 D    /sbin/smbd -D
> 32288 simon  20608 D    /sbin/smbd -D
> ...
> And after 5 mins the result of "ps" is the same.
> 
> The result of "top" shows below. The load average is VERY HIGHT
and CPU
is
> busy for io.
> # top
> Mem: 366504K used, 143900K free, 0K shrd, 14216K buff, 293224K cached
> CPU:  1.7% usr  3.5% sys  0.0% nic  0.0% idle 92.9% io  1.7% irq  0.0% 
sirq
> Load average: 103.10 103.03 102.65 1/149 5614
> PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND
>  5532  3325 root     R     3000  0.5  3.5 top
>  2421     1 simon  D    20712  4.0  0.0 /sbin/smbd -D
>   594     1 simon  D    20712  4.0  0.0 /sbin/smbd -D
>  4130     1 simon  D    20712  4.0  0.0 /sbin/smbd -D
>  1261     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
> 23452     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
> 30474     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
> 21641     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
>  9053     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
>  3068     1 simon  D    20608  4.0  0.0 /sbin/smbd -D
> 
> After reboot, the samba works fine. Until now I cannot reproduce yet.
> I can't figure out what's going on.
> Is it kind of bugs in kernel or samba?
> Does pdflush crash first, then affect smbd and sync to crash?
> 
> Please help,
> Thanks very much.
> 
> 
> I used some options in smb.conf:
>   max smbd process = 100
>   max connections = 100
> 
> Linux Samba Server:
>   Linux 2.6.31.8
>   Samba 3.5.6
>   Software RAID 1
> 
> 
> Simon
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba