We are a CentOS shop, and have the lucky, fortunate problem of having ever-increasing amounts of data to manage. EXT3/4 becomes tough to manage when you start climbing, especially when you have to upgrade, so we're contemplating switching to ZFS. As of last spring, it appears that ZFS On Linux http://zfsonlinux.org/ calls itself production ready despite a version number of 0.6.2, and being acknowledged as unstable on 32 bit systems. However, given the need to do backups, zfs send sounds like a godsend over rsync which is running into scaling problems of its own. (EG: Nightly backups are being threatened by the possibility of taking over 24 hours per backup) Was wondering if anybody here could weigh in with real-life experience? Performance/scalability? -Ben PS: I joined their mailing list recently, will be watching there as well. We will, of course, be testing for a while before "making the switch".
On 10/24/2013 1:41 PM, Lists wrote:> Was wondering if anybody here could weigh in with real-life experience? > Performance/scalability?I've only used ZFS on Solaris and FreeBSD. some general observations... 1) you need a LOT of ram for decent performance on large zpools. 1GB ram above your basic system/application requirements per terabyte of zpool is not unreasonable. 2) don't go overboard with snapshots. a few 100 are probably OK, but 1000s (*) will really drag down the performance of operations that enumerate file systems. 3) NEVER let a zpool fill up above about 70% full, or the performance really goes downhill. 4) I prefer using striped mirrors (aka raid10) over raidz/z2, but my applications are primarily database. (*) ran into a guy who had 100s of zfs 'file systems' (mount points), per user home directories, and was doing nightly snapshots going back several years, and his zfs commands were taking a long long time to do anything, and he couldn't figure out why. I think he had over 10,000 filesystems * snapshots. -- john r pierce 37N 122W somewhere on the middle of the left coast
On Thu, Oct 24, 2013 at 4:41 PM, Lists <lists at benjamindsmith.com> wrote:> We are a CentOS shop, and have the lucky, fortunate problem of having > ever-increasing amounts of data to manage. EXT3/4 becomes tough to > manage when you start climbing, especially when you have to upgrade, so > we're contemplating switching to ZFS. >You didn't mention XFS. Just curious if you considered it or not.> > As of last spring, it appears that ZFS On Linux http://zfsonlinux.org/ > calls itself production ready despite a version number of 0.6.2, and > being acknowledged as unstable on 32 bit systems. > > However, given the need to do backups, zfs send sounds like a godsend > over rsync which is running into scaling problems of its own. (EG: > Nightly backups are being threatened by the possibility of taking over > 24 hours per backup) > > Was wondering if anybody here could weigh in with real-life experience? > Performance/scalability? > > -Ben > > PS: I joined their mailing list recently, will be watching there as > well. We will, of course, be testing for a while before "making the > switch". > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-- ---~~.~~--- Mike // SilverTip257 //
On 10/25/2013, 05:00 , centos-request at centos.org wrote:> We are a CentOS shop, and have the lucky, fortunate problem of having > ever-increasing amounts of data to manage. EXT3/4 becomes tough to > manage when you start climbing, especially when you have to upgrade, so > we're contemplating switching to ZFS. > > As of last spring, it appears that ZFS On Linuxhttp://zfsonlinux.org/ > calls itself production ready despite a version number of 0.6.2, and > being acknowledged as unstable on 32 bit systems. > > However, given the need to do backups, zfs send sounds like a godsend > over rsync which is running into scaling problems of its own. (EG: > Nightly backups are being threatened by the possibility of taking over > 24 hours per backup) > > Was wondering if anybody here could weigh in with real-life experience? > Performance/scalability? > > -BenFWIW, I manage a small IT shop with a redundant pair of ZFS file servers running the zfsonlinux.org package on 64-bit ScientificLinux-6 platforms. CentOS-6 would work just as well. Installing it with yum couldn't be simpler, but configuring it takes a bit of reading and experimentation. I reserved a bit more than 1GByte of RAM for each TByte of disk. One machine (20 useable TBytes in raid-z3) is the SMB server for all of the clients, and the other machine (identically configured) sits in the background acting as a hot spare. Users tell me that performance is quite good. After about 2 months of testing, there have been no problems whatsoever, although I'll admit the servers do not operate under much stress. There is a cron job on each machine that does a scrub every Sunday. The old ext4 primary file servers have been shut down and the ZFS boxes put into production, although one of the old ext4 servers will remain rsync'd to the new machines for a few more months (just in case). The new servers have the zfsonlinux repositories configured for manual updates, but the two machines tend to be left alone unless there are important security updates or new features I need. To keep the two servers in sync I use 'lsyncd' which is essentially a front-end for rsync that cuts down thrashing and overhead dramatically by excluding the full filesystem scan and using inotify to figure out what to sync. This allows almost-real-time syncing of the backup machine. (BTW, you need to crank the resources for inotify waaaaay up for large filesystems with a couple million files.) So far, so good. I still have a *lot* to learn about ZFS and its feature set, but for now it's doing the job very nicely. I don't miss the long ext4 periodic fsck's one bit :-) YMMV, of course, Chuck
On Thu, Oct 24, 2013 at 01:41:17PM -0700, Lists wrote:> We are a CentOS shop, and have the lucky, fortunate problem of having > ever-increasing amounts of data to manage. EXT3/4 becomes tough to > manage when you start climbing, especially when you have to upgrade, so > we're contemplating switching to ZFS. > > As of last spring, it appears that ZFS On Linux http://zfsonlinux.org/ > calls itself production ready despite a version number of 0.6.2, and > being acknowledged as unstable on 32 bit systems. > > However, given the need to do backups, zfs send sounds like a godsend > over rsync which is running into scaling problems of its own. (EG: > Nightly backups are being threatened by the possibility of taking over > 24 hours per backup) > > Was wondering if anybody here could weigh in with real-life experience? > Performance/scalability? > > -Ben > > PS: I joined their mailing list recently, will be watching there as > well. We will, of course, be testing for a while before "making the > switch".Joining the discussion late, and don't really have anything to contribute on the ZFSonLinux side of things... At $DAYJOB we have been running ZFS via Nexenta (previously via Solaris 10) for many years. We have about 5PB of this and the primary use case is for backups and handling of imagery. For the most part, we really, really like ZFS. My feeling is that ZFS itself (at least in the *Solaris form) is rock solid and stable. Other pieces of the stack -- namely SMB/CIFS and some of the management tools provided by the various vendors are a bit more questionable. We spend a bit more time fighting weirdnesses with things higher up the stack than we do say on our NetApp environment. Too be expected. I'm waiting for Red Hat or someone else to come out and support ZFS -- perhaps unlikely due to legality questions, but if I could marry the power of ZFS with the software stack in Linux (Samba!!), I'd be mighty happy. Yes -- we could run Samba on our Nexenta boxes, but it isn't "supported". Echo'ing what many others say: - ZFS is memory hungry. All of our PRD boxes have 144GB of memory in them, and some have SSD's for ZIL or L2ARC depending on the workload. - Powerful redundancy is possible. Our environment is built on top of Dell MD1200 JBOD's all dual pathed up to dual LSI SAS switches. Our vdev's (RAID groups) are sized to match the number of JBODs with the invididual disks spread across each JBOD. We use triple parity RAID (RAIDZ3) and as such can lose three entire JBODs without suffering any data loss. We actually had one JBOD go flaky on us and were able to hot yank it out, put in a new one with zero downtime (and much shorter resilver/rebuild times than you'd get with regular RAID). - We make heavy use of snapshots and clones. Probably have 200-300 on some sysems and we use them to do release management for collections of imagery. Very powerful and haven't run into performance issues yet. * Snapshots let us take "diffs" between versions quite easily. We then stream these diffs to an identical ZFS system at a DR site and merge in the changes. Our network pipe isn't big enough yet to do this quickly, so we typically just plug in another SAS JBOD with a zpool on it, stream the diffs there as a flat file, sneakernet the JBOD to the DR site, plug it in, import the zpool and slurp in the differences. Pretty cool. As I mentioned, we have run into a few weird quirks. Mainly around stability of the management GUI (or lack of basic features like "useful" SNMP based monitoring), performance with CIFS and oddnesses like high system load in certain edge cases. Some general rough edges I suppose that we've been OK dealing with. The Nexenta guys are super smart, but of course they're a smaller shop and don't have the resources behind them that CentOS does with Red Hat. My guess is that this would be exacerbated to some extent on the Linux platform at this point. I personally wouldn't want to use ZFS on Linux for our customer data serving workloads, but might consider it for something purely internal. Ray