Hello,
i experience freezing of my FreeBSD machine when performing certain
operations
on a Samba share.
Technical info:
- FreeBSD 8.2 Release 64 Bit (it also happened with 8.2 RC3)
- Samba 3.5.6.1
- Athlon II Quadcore, 4 GB Ram
- 1 SSD with a ZFS pool (No.0) containing the FreeBSD system
- 12x2TB RaidZ2 pool (No.1) for data, created on 12 GEOM eli encrypted
partitions on 12 disks,
shared to a Windows 7 PC with Samba,
8 of the disks are attached to 2 Marvell SATA controllers, 4 to the
onboard controller
- ZPool v15, ZFS v4
Scenarios (checked using top):
A:
When copying files from one directory in pool 1 to another, the free
memory drops from
about 3700M to abaout 200M in the process, but seems to stabilize then.
B:
When copying the files onto a Windows machine using the Samba share,
the free memory seems to stabilize at about 100M.
C:
When computing a hashvalue of files from the share on Windows or doing a
binary compare to copies of the files stored on the Windows PC (using
Total Commander),
the free memory on the FreeBSD machine drops even lower and shortly
after the BSD system freezes.
Here is the last top output i got via ssh:
/last pid: 1328; load averages: 4.53, 2.23, 0.99 up 0+00:04:39
22:07:50
263 processes: 43 running, 201 sleeping, 19 waiting
CPU: 0.9% user, 0.0% nice, 23.1% system, 4.2% interrupt, 71.9% idle
Mem: 720K Active, 516M Wired, 144K Cache, 320K Buf, *39M Free*
Swap: 4096M Total, 12M Used, 4084M Free, 3008K In, 5124K Out
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 4 171 ki31 0K 64K RUN 0 15:54 303.61% idle
1321 root 1 52 0 27812K 704K swread 1 0:24 14.26% smbd
12 root 19 -60 - 0K 304K WAIT 0 0:21 12.45% intr
16 root 1 48 - 0K 16K psleep 2 0:01 3.76%
pagedaemon
3 root 1 -8 - 0K 16K RUN 0 0:06 3.27% g_up
4 root 1 -8 - 0K 16K - 3 0:05 2.69% g_down
0 root 108 -8 0 0K 1712K - 0 1:02 1.86% kernel
8 root 6 -8 - 0K 88K tx->tx 1 0:00 1.27% zfskern
1268 root 1 44 - 0K 16K geli:w 1 0:03 0.98%
g_eli[1] gpt
1225 root 1 45 - 0K 16K RUN 3 0:02 0.98%
g_eli[3] gpt
1267 root 1 44 - 0K 16K geli:w 0 0:02 0.98%
g_eli[0] gpt
1237 root 1 44 - 0K 16K RUN 0 0:02 0.88%
g_eli[0] gpt
1214 root 1 44 - 0K 16K RUN 2 0:02 0.88%
g_eli[2] gpt
1244 root 1 44 - 0K 16K RUN 2 0:02 0.78%
g_eli[2] gpt
1243 root 1 44 - 0K 16K RUN 1 0:02 0.78%
g_eli[1] gpt
1212 root 1 44 - 0K 16K RUN 0 0:02 0.78%
g_eli[0] gpt
1215 root 1 44 - 0K 16K RUN 3 0:02 0.78%
g_eli[3] gpt
1213 root 1 44 - 0K 16K RUN 1 0:02 0.78%
g_eli[1] gpt
1240 root 1 44 - 0K 16K RUN 3 0:02 0.78%
g_eli[3] gpt
1217 root 1 44 - 0K 16K RUN 0 0:02 0.78%
g_eli[0] gpt
1242 root 1 44 - 0K 16K RUN 0 0:02 0.68%
g_eli[0] gpt
1238 root 1 44 - 0K 16K RUN 1 0:02 0.68%
g_eli[1] gpt
1248 root 1 44 - 0K 16K RUN 1 0:02 0.68%
g_eli[1] gpt
1252 root 1 44 - 0K 16K RUN 0 0:02 0.68%
g_eli[0] gpt
1249 root 1 44 - 0K 16K RUN 2 0:02 0.68%
g_eli[2] gpt
1269 root 1 44 - 0K 16K geli:w 2 0:02 0.68%
g_eli[2] gpt/
It looks like a caching problem to me, but i don't know how to fix it.
I am also a bit confused, since i don't see an obvious difference
between scenario B and C.
I had a similar setup with 5 disks RaidZ1 and Samba running on 8.1 Release,
and never experienced such a freeze.
Does anyone have advice on how to get rid of this problem?
Best regards,
Henner Heck
*Config files*
----------------------------------------------------------------------------------
*loader.conf*
ahci_load="YES"
hw.hptrr.attach_generic=0
mvs_load="YES"
aio_load="YES"
kern.geom.label.gptid.enable=0
geom_label_load="YES"
geom_eli_load="YES"
zfs_load="YES"
hint.acpi_throttle.0.disabled=1
hw.pci.do_power_nodriver=3
vfs.root.mountfrom="zfs:pool0/ROOT/freebsd"
*rc.conf* (a bit anonymized)
defaultrouter="xxx.xxx.xxx.xxx"
hostname="hostname"
ifconfig_em0="inet xxx.xxx.xxx.xxx netmask xxx.xxx.xxx.xxx"
keymap="german.iso"
zfs_enable="YES"
sshd_enable="YES"
geli_autodetach="NO"
samba_enable="YES"
powerd_enable="YES"
*sysctl.conf*
No active entries
*smb.conf* (a bit anonymized)
[global]
workgroup = AG
server string = Praetoria
log file = /var/log/log.%m
log level = 0
max log size = 50
security = user
encrypt passwords = yes
use sendfile = true
min receivefile size = 131072
aio read size = 16384
aio write size = 16384
#needed for aio to work
write cache size = 0
[user1]
comment = Share for user user1
path = /volumes/pool1/users/user1
valid users = user1
public = no
writable = yes
printable = no
browseable = yes
Kostik Belousov
2011-Feb-22 22:21 UTC
FreeBSD 8.2 Release, ZFS + Samba, running out of memory
On Tue, Feb 22, 2011 at 10:55:37PM +0100, Henner Heck wrote:> > Hello, > > i experience freezing of my FreeBSD machine when performing certain > operations > on a Samba share. > > Technical info: > - FreeBSD 8.2 Release 64 Bit (it also happened with 8.2 RC3) > - Samba 3.5.6.1 > - Athlon II Quadcore, 4 GB Ram > - 1 SSD with a ZFS pool (No.0) containing the FreeBSD system > - 12x2TB RaidZ2 pool (No.1) for data, created on 12 GEOM eli encrypted > partitions on 12 disks, > shared to a Windows 7 PC with Samba, > 8 of the disks are attached to 2 Marvell SATA controllers, 4 to the > onboard controller > - ZPool v15, ZFS v4 > > Scenarios (checked using top): > > A: > When copying files from one directory in pool 1 to another, the free > memory drops from > about 3700M to abaout 200M in the process, but seems to stabilize then. > > B: > When copying the files onto a Windows machine using the Samba share, > the free memory seems to stabilize at about 100M. > > C: > When computing a hashvalue of files from the share on Windows or doing a > binary compare to copies of the files stored on the Windows PC (using > Total Commander), > the free memory on the FreeBSD machine drops even lower and shortly > after the BSD system freezes. > Here is the last top output i got via ssh: > > /last pid: 1328; load averages: 4.53, 2.23, 0.99 up 0+00:04:39 > 22:07:50 > 263 processes: 43 running, 201 sleeping, 19 waiting > CPU: 0.9% user, 0.0% nice, 23.1% system, 4.2% interrupt, 71.9% idle > Mem: 720K Active, 516M Wired, 144K Cache, 320K Buf, *39M Free* > Swap: 4096M Total, 12M Used, 4084M Free, 3008K In, 5124K Out > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 4 171 ki31 0K 64K RUN 0 15:54 303.61% idle > 1321 root 1 52 0 27812K 704K swread 1 0:24 14.26% smbd > 12 root 19 -60 - 0K 304K WAIT 0 0:21 12.45% intr > 16 root 1 48 - 0K 16K psleep 2 0:01 3.76% > pagedaemon > 3 root 1 -8 - 0K 16K RUN 0 0:06 3.27% g_up > 4 root 1 -8 - 0K 16K - 3 0:05 2.69% g_down > 0 root 108 -8 0 0K 1712K - 0 1:02 1.86% kernel > 8 root 6 -8 - 0K 88K tx->tx 1 0:00 1.27% zfskern > 1268 root 1 44 - 0K 16K geli:w 1 0:03 0.98% > g_eli[1] gpt > 1225 root 1 45 - 0K 16K RUN 3 0:02 0.98% > g_eli[3] gpt > 1267 root 1 44 - 0K 16K geli:w 0 0:02 0.98% > g_eli[0] gpt > 1237 root 1 44 - 0K 16K RUN 0 0:02 0.88% > g_eli[0] gpt > 1214 root 1 44 - 0K 16K RUN 2 0:02 0.88% > g_eli[2] gpt > 1244 root 1 44 - 0K 16K RUN 2 0:02 0.78% > g_eli[2] gpt > 1243 root 1 44 - 0K 16K RUN 1 0:02 0.78% > g_eli[1] gpt > 1212 root 1 44 - 0K 16K RUN 0 0:02 0.78% > g_eli[0] gpt > 1215 root 1 44 - 0K 16K RUN 3 0:02 0.78% > g_eli[3] gpt > 1213 root 1 44 - 0K 16K RUN 1 0:02 0.78% > g_eli[1] gpt > 1240 root 1 44 - 0K 16K RUN 3 0:02 0.78% > g_eli[3] gpt > 1217 root 1 44 - 0K 16K RUN 0 0:02 0.78% > g_eli[0] gpt > 1242 root 1 44 - 0K 16K RUN 0 0:02 0.68% > g_eli[0] gpt > 1238 root 1 44 - 0K 16K RUN 1 0:02 0.68% > g_eli[1] gpt > 1248 root 1 44 - 0K 16K RUN 1 0:02 0.68% > g_eli[1] gpt > 1252 root 1 44 - 0K 16K RUN 0 0:02 0.68% > g_eli[0] gpt > 1249 root 1 44 - 0K 16K RUN 2 0:02 0.68% > g_eli[2] gpt > 1269 root 1 44 - 0K 16K geli:w 2 0:02 0.68% > g_eli[2] gpt/ > > It looks like a caching problem to me, but i don't know how to fix it. > I am also a bit confused, since i don't see an obvious difference > between scenario B and C. > I had a similar setup with 5 disks RaidZ1 and Samba running on 8.1 Release, > and never experienced such a freeze. > > Does anyone have advice on how to get rid of this problem?Try the patch from rev. 218795. If it indeed help, we would need an errara notice. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110222/eaa27fc0/attachment.pgp
Martin Matuska
2011-Feb-22 22:31 UTC
FreeBSD 8.2 Release, ZFS + Samba, running out of memory
There looks like a known bug of not activating memory pages if using sendfile(2). This was fixed by kib@ in revision 218795 of stable/8. Please try the following patch and report the result: http://people.freebsd.org/~mm/patches/releng_8_2/218795.zfs.patch D?a 22.02.2011 22:55, Henner Heck wrote / nap?sal(a):> > Hello, > > i experience freezing of my FreeBSD machine when performing certain > operations > on a Samba share. > > Technical info: > - FreeBSD 8.2 Release 64 Bit (it also happened with 8.2 RC3) > - Samba 3.5.6.1 > - Athlon II Quadcore, 4 GB Ram > - 1 SSD with a ZFS pool (No.0) containing the FreeBSD system > - 12x2TB RaidZ2 pool (No.1) for data, created on 12 GEOM eli encrypted > partitions on 12 disks, > shared to a Windows 7 PC with Samba, > 8 of the disks are attached to 2 Marvell SATA controllers, 4 to the > onboard controller > - ZPool v15, ZFS v4 > > Scenarios (checked using top): > > A: > When copying files from one directory in pool 1 to another, the free > memory drops from > about 3700M to abaout 200M in the process, but seems to stabilize then. > > B: > When copying the files onto a Windows machine using the Samba share, > the free memory seems to stabilize at about 100M. > > C: > When computing a hashvalue of files from the share on Windows or doing a > binary compare to copies of the files stored on the Windows PC (using > Total Commander), > the free memory on the FreeBSD machine drops even lower and shortly > after the BSD system freezes.
Jeremy Chadwick
2011-Feb-22 22:43 UTC
FreeBSD 8.2 Release, ZFS + Samba, running out of memory
On Tue, Feb 22, 2011 at 10:55:37PM +0100, Henner Heck wrote:> i experience freezing of my FreeBSD machine when performing certain > operations > on a Samba share. > > Technical info: > - FreeBSD 8.2 Release 64 Bit (it also happened with 8.2 RC3) > - Samba 3.5.6.1 > - Athlon II Quadcore, 4 GB Ram > - 1 SSD with a ZFS pool (No.0) containing the FreeBSD system > - 12x2TB RaidZ2 pool (No.1) for data, created on 12 GEOM eli encrypted > partitions on 12 disks, > shared to a Windows 7 PC with Samba, > 8 of the disks are attached to 2 Marvell SATA controllers, 4 to the > onboard controller > - ZPool v15, ZFS v4 > > Scenarios (checked using top): > > A: > When copying files from one directory in pool 1 to another, the free > memory drops from > about 3700M to abaout 200M in the process, but seems to stabilize then. > > B: > When copying the files onto a Windows machine using the Samba share, > the free memory seems to stabilize at about 100M. > > C: > When computing a hashvalue of files from the share on Windows or doing a > binary compare to copies of the files stored on the Windows PC (using > Total Commander), > the free memory on the FreeBSD machine drops even lower and shortly > after the BSD system freezes. > Here is the last top output i got via ssh: > > ... > > Does anyone have advice on how to get rid of this problem? > > ... > > *smb.conf* (a bit anonymized) > > [global] > use sendfile = trueSet this to "false" and the problem might go away. You can try some of the patches others have recommended as well (which will permit you to use sendfile going forward), but if you don't want to deal with patches, the workaround should be fine. If you still experience issues after that, possibly some loader.conf tuning for ZFS is needed. Also, one question: when you say "freezes", do you mean "completely locks up hard" or "the system seems alive but seems to be spending all of its CPU time doing something else"? Try hitting NumLock on the system's keyboard to see if the LED toggles on/off; if it does, the system isn't frozen, but is catatonic to some degree. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |