Does anyone else use ZFS to store TM backups? I find that whenever my laptop (over wifi!) starts a TM the ZFS machine it's backing up to grinds to a halt.. Other systems streaming stuff over NFS from it also tend to stall.. I presume that TM is doing something which causes ZFS some issues but I'm not sure how to find out what the real problem is let alone how to fix it.. I am running FreeBSD midget.dons.net.au 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #8 r217094M: Sat Jan 8 11:15:07 CST 2011 darius@midget.dons.net.au:/usr/obj/usr/src/sys/MIDGET amd64 It is a 5 disk RAIDZ1 with 1.29Tb free using WD10EADS drives. I don't see any SMART errors or ZFS warnings. I have the following ZFS related tunables vfs.zfs.arc_max="3072M" vfs.zfs.prefetch_disable="1" vfs.zfs.txg.timeout=5 vfs.zfs.cache_flush_disable=1 Any help appreciated, thanks :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C
Hi, On 4/28/11 4:03 PM, Daniel O'Connor wrote:> Does anyone else use ZFS to store TM backups? > > I find that whenever my laptop (over wifi!) starts a TM the ZFS machine it's backing up to grinds to a halt.. Other systems streaming stuff over NFS from it also tend to stall..Are you using zfs compression? If so, try turning that off. I have a pool with a couple of filesystems with gzip-9 compression enabled. Whenever I write (using zfs receive, it is a backup server) to one of those volumes the whole pool stalls with lots of disk activity. Even creating a snapshot on another filesystem within the same pool lasts a couple of minutes. Does anyone know how to make this perform a little better? It's only writing small amounts (70-100 ops/s, 1 MB/s) on an otherwise idle pool. Still the drive leds blink like crazy. One of my two CPU cores is maxed out, the other is idle. I suppose it won't get any faster (it's CPU bound because of the heavy gzip compression), but why is the pool so slow? Is zfs receive using synchonous writes? Sorry for maybe being offtopic :-) Regards, Thomas
I doubt the issues you are encountering have much to do with ZFS. It sounds like you are using TimeMachine over NFS. Obviously, Apple does not support that configuration: http://www.google.com/search?q=time+machine+nfs+site:apple.com In my opinion, TimeMachine should only be used with block storage. If you use any kind of file-sharing protocol (AFP, SMB/CIFS or NFS), TimeMachine is implemented using a sparse disk image broken into hundreds or thousands of separate files. This is a hack at best. Time machine works very well with locally attached storage, but if you need to use network storage, you might want to try iSCSI: http://thegreyblog.blogspot.com/2010/02/using-zfs-with-apple-time-machine.html http://people.freebsd.org/~rse/iscsi/iscsi.txt On Apr 28, 2011, at 7:03 AM, Daniel O'Connor wrote:> Does anyone else use ZFS to store TM backups? > > I find that whenever my laptop (over wifi!) starts a TM the ZFS machine it's backing up to grinds to a halt.. Other systems streaming stuff over NFS from it also tend to stall.. > > I presume that TM is doing something which causes ZFS some issues but I'm not sure how to find out what the real problem is let alone how to fix it.. > > I am running FreeBSD midget.dons.net.au 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #8 r217094M: Sat Jan 8 11:15:07 CST 2011 darius@midget.dons.net.au:/usr/obj/usr/src/sys/MIDGET amd64 > > It is a 5 disk RAIDZ1 with 1.29Tb free using WD10EADS drives. > > I don't see any SMART errors or ZFS warnings. > > I have the following ZFS related tunables > > vfs.zfs.arc_max="3072M" > vfs.zfs.prefetch_disable="1" > vfs.zfs.txg.timeout=5 > vfs.zfs.cache_flush_disable=1 > > Any help appreciated, thanks :) > > -- > Daniel O'Connor software and network engineer > for Genesis Software - http://www.gsoft.com.au > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C > > > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
On Apr 28, 2011, at 7:03 AM, Daniel O'Connor wrote:> Does anyone else use ZFS to store TM backups? > > I find that whenever my laptop (over wifi!) starts a TM the ZFS machine it's backing up to grinds to a halt.. Other systems streaming stuff over NFS from it also tend to stall.. > > I presume that TM is doing something which causes ZFS some issues but I'm not sure how to find out what the real problem is let alone how to fix it..Caveat: Time Machine is a viable backup system, but you're outside the configurations which it supports. TM really wants the backup to be located on an HFS+ filesystem because it makes very extensive use of hard links-- including hard links to directories-- which are not widely supported by other Unix filesystems. Even if you choose to continue using ZFS storage, please note that you'll obtain significantly better results by using AFP instead of NFS filesharing. Wikipedia has useful info here: http://en.wikipedia.org/wiki/Time_Machine_%28Mac_OS%29 http://en.wikipedia.org/wiki/Apple_Filing_Protocol Regards, -- -Chuck
On Thu, Apr 28, 2011 at 11:33:22PM +0930, Daniel O'Connor wrote:> Does anyone else use ZFS to store TM backups? > > I find that whenever my laptop (over wifi!) starts a TM the ZFS machine it's backing up to grinds to a halt.. Other systems streaming stuff over NFS from it also tend to stall.. > > I presume that TM is doing something which causes ZFS some issues but I'm not sure how to find out what the real problem is let alone how to fix it.. > > I am running FreeBSD midget.dons.net.au 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #8 r217094M: Sat Jan 8 11:15:07 CST 2011 darius@midget.dons.net.au:/usr/obj/usr/src/sys/MIDGET amd64 > > It is a 5 disk RAIDZ1 with 1.29Tb free using WD10EADS drives. > > I don't see any SMART errors or ZFS warnings. > > I have the following ZFS related tunables > > vfs.zfs.arc_max="3072M" > vfs.zfs.prefetch_disable="1" > vfs.zfs.txg.timeout=5 > vfs.zfs.cache_flush_disable=1Are the last two actually *working* in /boot/loader.conf? Can you verify by looking at them via sysctl? AFAIK they shouldn't work, since they lack double-quotes around the values. Parsing errors are supposed to throw you back to the loader prompt. See loader.conf(5) for the syntax. I'm also not sure why you're setting cache_flush_disable at all.> Any help appreciated, thanks :)Others seem to be battling stating that "NFS doesn't work for TM", but that isn't what you're complaining about. You're complaining that FreeBSD with ZFS + NFS performs extremely poorly when trying to do backups from an OS X client using TM (writing to the NFS mount). I have absolutely no experience with TM or OS X, so if it's actually a client-level problem (which I'm doubting) I can't help you there. Just sort of a ramble here at different things... It would be useful to provide ZFS ARC sysctl data from the FreeBSD system where you're seeing performance issues. "sysctl -a kstat.zfs.misc.arcstats" should suffice. You should also try executing "zpool iostat -v 1" during the TM backup to see if there's a particular device which is behaving poorly. There have been reports of ZFS pools behaving poorly when a single device within the pool has slow I/O (e.g. 5 hard disks, one of which has internal issues, resulting in the entire pool performing horribly). You should let this run for probably 60-120 seconds to get an idea. Given your parameters above (assuming vfs.zfs.txg.timeout IS in fact 5!), you should see "bursts" of writes every 5 seconds. I know that there are some things on ZFS that perform badly overall. Anything that involves excessive/large numbers of files (not file sizes, but actual files themselves) seems to perform not-so-great with ZFS. For example, Maildir on ZFS = piss-poor performance. There are ways to work around this issue (if I remember correctly, by adding a dedicated "log" device to your ZFS pool, but be aware your log devices need to be reliable (if you have a single log device and it fails the entire pool is damaged, if I remember right)), but I don't consider it feasible. So if TM is creating tons of files on the NFS mount (backed by ZFS), then I imagine the performance isn't so great. Could you please provide the following sysctl values? Thanks. kern.maxvnodes kern.minvnodes vfs.freevnodes vfs.numvnodes If the FreeBSD machine has a wireless card in it, if at all possible could you try ruling that out by hooking up wired Ethernet instead? It's probably not the cause, but worth trying anyway. If you have a home router or something doing 802.11, don't bother with this idea. Next, you COULD try using Samba/CIFS on the FreeBSD box to see if you can narrow the issue down to bad NFS performance. Please see this post of mine about tuning Samba on FreeBSD (backed by ZFS) to get extremely good performance. Many people responded and said their performance drastically improved (you can see the thread yourself). The trick is AIO. You can ignore the part about setting vm.kmem_size in loader.conf; that advice is now old/deprecated (does not pertain to you given the date of your kernel), and vfs.zfs.txg.write_limit_override is something you shouldn't mess with unless absolutely needed to leave it default: http://lists.freebsd.org/pipermail/freebsd-stable/2011-February/061642.html Finally, when was the last time this FreeBSD machine was rebooted? Some people have seen horrible performance that goes away after a reboot. There's some speculation that memory fragmentation has something to do with it. I simply don't know. I'm not telling you to reboot the box (please don't; it would be more useful if it could be kept up in case folks want to do analysis of it). -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |