Hi All,
We're currently trying to use samba in an embedded
application, so we're working with samba 3.0.2a
compiled for PPC.
We have a test application which will basically write,
then read back and compare data files on the disk.
The problems we see occur when we try using just over
3500 small files (< 10k each). It seems that after
about 200 or so iterations of this test, we start to
see the smbd process's VmRss size grow. Once it
starts growing, it continues to grow until an Out of
Memory condition occurs, somewhere around 270
iterations.
We've tried this same test on the following versions
of samba as well, and received the same result:
samba-2.2.7a
samba-2.2.8
samba-2.2.12
Here's our smb.conf file:
<snip>
[global]
workgroup = WORKGROUP
server string
log file = /var/samba/log.%m
max log size = 50
security = share
encrypt passwords = yes
smb passwd file = /etc/samba/smbpasswd
socket options = TCP_NODELAY SO_RCVBUF=16384
SO_SNDBUF=16384
dns proxy = no
browsable = yes
guest account = nobody
guest ok = yes
create mask = 777
directory mask = 777
force directory mode = 0777
force create mode 0777
read size = 65536
disable spoolss = yes
[Share]
path = /share
public = yes
writable = yes
veto files = /lost+found/
hide files = /lost+found/
[Printer]
path = /var/lp/
printer name = printer
writable = yes
public = yes
printable = yes
print command = lpr -r %s
</snip>
Here's some output during the test:
At the start of the test, here is the process listing:
<snip>
/ # ps
PID Uid VmSize Stat Command
1 root 520 S ini
2 root SWN [ksoftirqd/0]
3 root SW< [events/0]
4 root SW< [khelper]
9 root SW< [kthread]
32 root SW< [kblockd/0]
45 root SW [khubd]
84 root SW [pdflush]
85 root SW [pdflush]
87 root SW< [aio/0]
86 root DW [kswapd0]
645 root SW [mtdblockd]
646 root SW [ftld]
647 root SW [nftld]
677 root SWN [jffs2_gcd_mtd4]
765 root 328 S /sbin/syslogd -m 0 -s 40 -b
0
830 root 1820 S /sbin/smbd
891 root 836 S /bin/sh
971 root 3104 S /sbin/smbd
972 root 736 R ps
</snip>
This was right after a boot, then manually shutting
down nmbd, and various other small daemons that were
unnecessary for the test
2 hours later, here's the output:
<snip>
/ # ps
PID Uid VmSize Stat Command
1 root 520 S ini
2 root SWN [ksoftirqd/0]
3 root SW< [events/0]
4 root SW< [khelper]
9 root SW< [kthread]
32 root SW< [kblockd/0]
45 root SW [khubd]
84 root SW [pdflush]
85 root SW [pdflush]
87 root SW< [aio/0]
86 root DW [kswapd0]
645 root SW [mtdblockd]
646 root SW [ftld]
647 root SW [nftld]
677 root SWN [jffs2_gcd_mtd4]
765 root 328 S /sbin/syslogd -m 0 -s 40 -b
0
830 root 1820 S /sbin/smbd
891 root 888 S /bin/sh
971 root 3104 S /sbin/smbd
1007 root 736 R ps
</snip>
Then, around iteration number 200, we see this:
<snip>
/ # ps
PID Uid VmSize Stat Command
1 root 924 S ini
2 root SWN [ksoftirqd/0]
3 root SW< [events/0]
4 root SW< [khelper]
9 root SW< [kthread]
32 root SW< [kblockd/0]
45 root SW [khubd]
84 root SW [pdflush]
85 root SW [pdflush]
87 root SW< [aio/0]
86 root DW [kswapd0]
645 root SW [mtdblockd]
646 root SW [ftld]
647 root SW [nftld]
677 root SWN [jffs2_gcd_mtd4]
765 root 664 S /sbin/syslogd -m 0 -s 40 -b
0
830 root 3296 S /sbin/smbd
891 root 2756 S /bin/sh
971 root 6176 D /sbin/smbd
1071 root 736 R ps
</snip>
This last time, I actually stopped the test,
disconnected the network drive, and reconnected the
network drive, so the new process listing looks like
this:
<snip>
/ # ps
PID Uid VmSize Stat Command
1 root 924 S ini
2 root SWN [ksoftirqd/0]
3 root SW< [events/0]
4 root SW< [khelper]
9 root SW< [kthread]
32 root SW< [kblockd/0]
45 root SW [khubd]
84 root SW [pdflush]
85 root SW [pdflush]
87 root SW< [aio/0]
86 root SW [kswapd0]
645 root SW [mtdblockd]
646 root SW [ftld]
647 root SW [nftld]
677 root SWN [jffs2_gcd_mtd4]
765 root 668 S /sbin/syslogd -m 0 -s 40 -b
0
830 root 3612 S /sbin/smbd
891 root 2820 S /bin/sh
1076 root 1996 S /sbin/smbd
1077 root 736 R ps
</snip>
If I restart the test, and come back just 10
iterations later, I see this:
<snip>
/ # ps
PID Uid VmSize Stat Command
1 root 1636 S ini
2 root SWN [ksoftirqd/0]
3 root SW< [events/0]
4 root SW< [khelper]
9 root SW< [kthread]
32 root SW< [kblockd/0]
45 root SW [khubd]
84 root SW [pdflush]
85 root SW [pdflush]
87 root SW< [aio/0]
86 root DW [kswapd0]
645 root SW [mtdblockd]
646 root SW [ftld]
647 root SW [nftld]
677 root SWN [jffs2_gcd_mtd4]
765 root 688 S /sbin/syslogd -m 0 -s 40 -b
0
830 root 4004 S /sbin/smbd
891 root 3640 S /bin/sh
1076 root 14156 D /sbin/smbd
1137 root 736 R ps
</snip>
Due to size constraints, attempts to use versions of
samba after 3.0.2a aren't entirely feasibly, mostly
due to external dependencies on other libraries.
I guess I'm just wondering two things:
1. Has anyone else seen this behaviour? (My google
searches haven't been turning up much in the way of
results, if yours turn up better, please send me the
keywords you used in your search as well)
2. What other information can I provide that would
help with tracking down the root cause of this? Where
else should I be looking
We have kernel 2.6.12-rc2 running on this board
compiled for a PowerPC 603e (MPC8241 processor) as
well.
Thanks in advance,
Anthony
On Tue, Jun 28, 2005 at 03:55:04PM -0400, Anthony Russello wrote:> Hi All, > > We're currently trying to use samba in an embedded > application, so we're working with samba 3.0.2a > compiled for PPC. > > We have a test application which will basically write, > then read back and compare data files on the disk. > > The problems we see occur when we try using just over > 3500 small files (< 10k each). It seems that after > about 200 or so iterations of this test, we start to > see the smbd process's VmRss size grow. Once it > starts growing, it continues to grow until an Out of > Memory condition occurs, somewhere around 270 > iterations. > > We've tried this same test on the following versions > of samba as well, and received the same result: > > samba-2.2.7a > samba-2.2.8 > samba-2.2.123.0.2a is very old. Can you attempt the same with a 3.0.14a at least. Thanks, Jeremy.
Okay, If I manually edit include/config.h to change the iconv stuff to #undef and commented out, it now builds (might be an configure issue there). That setup is currently running, it'll take several hours before we hit the error condition (if we hit it with this build). I'll get back to you tomorrow with the results. Thanks again for responding so quickly. Cheers, Anthony --- Anthony Russello <arussello@rogers.com> wrote:> > --- Jeremy Allison <jra@samba.org> wrote: > > > > > 3.0.2a is very old. Can you attempt the same with > a > > 3.0.14a at least. > > Hi Jeremy, > > Attempts to configure samba 3.0.14a with > --without-libiconv result in the error: > > <snip> > checking whether pututline returns pointer... yes > checking for ut_syslen in utmpx... no > configure: error: argument to --with-libiconv must > be > a directory > </snip> > > <snip> > Linking bin/smbd > lib/iconv.o(.text+0x2a8): In function `sys_iconv': > : undefined reference to `libiconv' > lib/iconv.o(.text+0x2d8): In function `sys_iconv': > : undefined reference to `libiconv' > lib/iconv.o(.text+0x5a0): In function > `smb_iconv_open': > : undefined reference to `libiconv_open' > lib/iconv.o(.text+0x5bc): In function > `smb_iconv_open': > : undefined reference to `libiconv_open' > lib/iconv.o(.text+0x5f4): In function > `smb_iconv_open': > : undefined reference to `libiconv_open' > lib/iconv.o(.text+0x610): In function > `smb_iconv_open': > : undefined reference to `libiconv_open' > lib/iconv.o(.text+0x8d0): In function > `smb_iconv_close': > : undefined reference to `libiconv_close' > lib/iconv.o(.text+0x8e0): In function > `smb_iconv_close': > : undefined reference to `libiconv_close' > lib/iconv.o(.text+0x8f0): In function > `smb_iconv_close': > : undefined reference to `libiconv_close' > collect2: ld returned 1 exit status > make: *** [bin/smbd] Error 1 > stripping using powerpc-603e-linux-gnu-strip > </snip> > > This worked fine in samba 3.0.2a. Unfortunately, > due > to space limitations in our flash filesystem on our > embedded device, we can't include libiconv. > > Any ideas? > > Thanks for getting back to me so soon. > > Anthony >
> As you're doing this on an embedded system as Irecall > you> might want to cut down on the stat cache (which can > grow > unlimited on normal systems).I just read up on the stat cache options, and from what I saw, the default stat cache size is 50KB. Is that incorrect?> To turn it off set : > > stat cache = FalseI'll give this a try. It should be noted that I was able to reproduce the same issue on a dual P3 550, with 512MB of RAM. It took a fair bit longer to reach the point of failure, but it did fail. I believe we reached around 1300 iterations of writing 3500+ small files, then reading those back to compare them. Thanks, Anthony