I''m currently thinking about rolling a variant of http://www.napp-it.org/napp-it/all-in-one/index_en.html with remote backup (via snapshot and send) to 2-3 other (HP N40L-based) zfs boxes for production in our organisation. The systems themselves would be either Dell or Supermicro (latter with ZIL/L2ARC on SSD, plus SAS disks (pools as mirrors) all with hardware pass-through). The idea is to use zfs for data integrity and backup via data snapshot (especially important data will be also back-up''d via conventional DLT tapes). Before I test thisi -- Is anyone using this is in production? Any caveats? Can I actually have a year''s worth of snapshots in zfs without too much performance degradation? Thanks.
On 09/18/2012 04:31 PM, Eugen Leitl wrote:> > I''m currently thinking about rolling a variant of > > http://www.napp-it.org/napp-it/all-in-one/index_en.html > > with remote backup (via snapshot and send) to 2-3 > other (HP N40L-based) zfs boxes for production in > our organisation. The systems themselves would > be either Dell or Supermicro (latter with ZIL/L2ARC > on SSD, plus SAS disks (pools as mirrors) all with > hardware pass-through). > > The idea is to use zfs for data integrity and > backup via data snapshot (especially important > data will be also back-up''d via conventional DLT > tapes). > > Before I test thisi -- > > Is anyone using this is in production? Any caveats?Well, obviously nobody has used a setup *exactly* like yours in production. That being said, I fail to see any obvious problems with it. The HP N40L boxes are wonderful for backup purposes, since they can easily push >1Gbit/s in ZFS throughput with a very small footprint.> Can I actually have a year''s worth of snapshots in > zfs without too much performance degradation?Each additional dataset (not sure about snapshots, though) increases boot times slightly, however, I''ve seen pools with several hundred datasets without any serious issues, so yes, it is possible. Be prepared, though, that the data volumes might be substantial (depending on your overall data turn-around per unit time between the snapshots). Cheers, -- Saso
On 9/18/2012 10:31 AM, Eugen Leitl wrote:> I''m currently thinking about rolling a variant of > > http://www.napp-it.org/napp-it/all-in-one/index_en.html > > with remote backup (via snapshot and send) to 2-3 > other (HP N40L-based) zfs boxes for production in > our organisation. The systems themselves would > be either Dell or Supermicro (latter with ZIL/L2ARC > on SSD, plus SAS disks (pools as mirrors) all with > hardware pass-through). > > The idea is to use zfs for data integrity and > backup via data snapshot (especially important > data will be also back-up''d via conventional DLT > tapes). > > Before I test thisi -- > > Is anyone using this is in production? Any caveats? >I run an all-in-one and it works fine. Supermicro x9scl-f with 32gb ECC ram. 20 is for the openindiana SAN, with an ibm m1015 passed through via vmdirect (pci passthru). 4 SAS nearline drives in 2x2 mirror config in a jbod chassis. 2 samsung 830 128gb ssds as l2arc. The main caveat is to order the VMs properly for auto-start (assuming you use that as I do.) The OI VM goes first, and I give a good 120 seconds before starting the other VMs. For auto shutdown, all VMs but OI do suspend, OI does shutdown. The big caveat: do NOT use iSCSI for the datastore, use NFS. Maybe there''s a way to fix this, but I found that on start up, ESXi would time out the iSCSI datastore mount before the virtualized SAN VM was up and serving the share - bad news. NFS seems to be more resilient there. vmxnet3 vnics should work fine for OI VM, but might want to stick to e1000.> Can I actually have a year''s worth of snapshots in > zfs without too much performance degradation? >Dunno about that.
On Sep 18, 2012, at 10:40 AM, Dan Swartzendruber wrote:> On 9/18/2012 10:31 AM, Eugen Leitl wrote: >> I''m currently thinking about rolling a variant of >> >> http://www.napp-it.org/napp-it/all-in-one/index_en.html >> >> with remote backup (via snapshot and send) to 2-3 >> other (HP N40L-based) zfs boxes for production in >> our organisation. The systems themselves would >> be either Dell or Supermicro (latter with ZIL/L2ARC >> on SSD, plus SAS disks (pools as mirrors) all with >> hardware pass-through). >> >> The idea is to use zfs for data integrity and >> backup via data snapshot (especially important >> data will be also back-up''d via conventional DLT >> tapes). >> >> Before I test thisi -- >> >> Is anyone using this is in production? Any caveats? >> > I run an all-in-one and it works fine. Supermicro x9scl-f with 32gb ECC ram. 20 is for the openindiana SAN, with an ibm m1015 passed through via vmdirect (pci passthru). 4 SAS nearline drives in 2x2 mirror config in a jbod chassis. 2 samsung 830 128gb ssds as l2arc. The main caveat is to order the VMs properly for auto-start (assuming you use that as I do.) The OI VM goes first, and I give a good 120 seconds before starting the other VMs. For auto shutdown, all VMs but OI do suspend, OI does shutdown. The big caveat: do NOT use iSCSI for the datastore, use NFS. Maybe there''s a way to fix this, but I found that on start up, ESXi would time out the iSCSI datastore mount before the virtualized SAN VM was up and serving the share - bad news. NFS seems to be more resilient there. vmxnet3 vnics should work fine for OI VM, but might want to stick to e1000. >> Can I actually have a year''s worth of snapshots in >> zfs without too much performance degradation? >> > Dunno about that.I did something similar: http://churnd.wordpress.com/2011/06/27/zfsesxi-all-in-one-part-1/ Works great? need to bump up the RAM to 32GB.
On Sep 18, 2012, at 10:40 AM, Dan Swartzendruber wrote: On 9/18/2012 10:31 AM, Eugen Leitl wrote: I''m currently thinking about rolling a variant of http://www.napp-it.org/napp-it/all-in-one/index_en.html with remote backup (via snapshot and send) to 2-3 other (HP N40L-based) zfs boxes for production in our organisation. The systems themselves would be either Dell or Supermicro (latter with ZIL/L2ARC on SSD, plus SAS disks (pools as mirrors) all with hardware pass-through). The idea is to use zfs for data integrity and backup via data snapshot (especially important data will be also back-up''d via conventional DLT tapes). Before I test thisi -- Is anyone using this is in production? Any caveats? I run an all-in-one and it works fine. Supermicro x9scl-f with 32gb ECC ram. 20 is for the openindiana SAN, with an ibm m1015 passed through via vmdirect (pci passthru). 4 SAS nearline drives in 2x2 mirror config in a jbod chassis. 2 samsung 830 128gb ssds as l2arc. The main caveat is to order the VMs properly for auto-start (assuming you use that as I do.) The OI VM goes first, and I give a good 120 seconds before starting the other VMs. For auto shutdown, all VMs but OI do suspend, OI does shutdown. The big caveat: do NOT use iSCSI for the datastore, use NFS. Maybe there''s a way to fix this, but I found that on start up, ESXi would time out the iSCSI datastore mount before the virtualized SAN VM was up and serving the share - bad news. NFS seems to be more resilient there. vmxnet3 vnics should work fine for OI VM, but might want to stick to e1000. Can I actually have a year''s worth of snapshots in zfs without too much performance degradation? Dunno about that. I did something similar: http://churnd.wordpress.com/2011/06/27/zfsesxi-all-in-one-part-1/ Works great? need to bump up the RAM to 32GB. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120918/6baa2139/attachment.html>
On 18 sept. 2012, at 16:40, Dan Swartzendruber <dswartz at druber.com> wrote:> On 9/18/2012 10:31 AM, Eugen Leitl wrote: >> I''m currently thinking about rolling a variant of >> >> http://www.napp-it.org/napp-it/all-in-one/index_en.html >> >> with remote backup (via snapshot and send) to 2-3 >> other (HP N40L-based) zfs boxes for production in >> our organisation. The systems themselves would >> be either Dell or Supermicro (latter with ZIL/L2ARC >> on SSD, plus SAS disks (pools as mirrors) all with >> hardware pass-through). >> >> The idea is to use zfs for data integrity and >> backup via data snapshot (especially important >> data will be also back-up''d via conventional DLT >> tapes). >> >> Before I test thisi -- >> >> Is anyone using this is in production? Any caveats? >> > I run an all-in-one and it works fine. Supermicro x9scl-f with 32gb ECC ram. 20 is for the openindiana SAN, with an ibm m1015 passed through via vmdirect (pci passthru). 4 SAS nearline drives in 2x2 mirror config in a jbod chassis. 2 samsung 830 128gb ssds as l2arc. The main caveat is to order the VMs properly for auto-start (assuming you use that as I do.) The OI VM goes first, and I give a good 120 seconds before starting the other VMs. For auto shutdown, all VMs but OI do suspend, OI does shutdown. The big caveat: do NOT use iSCSI for the datastore, use NFS. Maybe there''s a way to fix this, but I found that on start up, ESXi would time out the iSCSI datastore mount before the virtualized SAN VM was up and serving the share - bad news. NFS seems to be more resilient there. vmxnet3 vnics should work fine for OI VM, but might want to stick to e1000. >> Can I actually have a year''s worth of snapshots in >> zfs without too much performance degradation? >> > Dunno about that.This concords with my experience after building a few custom appliances with similar configurations. For the backup side of things, stop and think about the actual use cases for keeping a year''s worth of snapshots. Generally speaking, restore requests are for data that is relatively hot and has been live some time in the current quarter. I think that you could limit your snapshot retention to something smaller, and pull the files back from tape if you go past that. One detail missing from this calculation is the frequency of snapshots. A year''s worth of hourly snapshots is huge for a little box like the HP NXXL machines. A year''s worth of daily snapshots is more in the domain of the reasonable. For reference, though I have one that retains 4 weeks of replicated hourly snapshots without complaint. (8Gb/4x2Tb raidz1) The bigger issue you''ll run into will be data sizing as a year''s worth of snapshot basically means that you''re keeping a journal of every single write that''s occurred over the year. If you are running VM Images, this can also mean that you''re retaining a years worth of writes to your OS swap file - something of exceedingly little value. You might want to consider moving the swap files to a separate virtual disk on a different volume. If you''re running ESXi with a vSphere license, I''d recommend looking at VDR (free with the vCenter license) for backing up the VMs to the little HPs since you get compressed and deduplicated backups that will minimize the replication bandwidth requirements. Much depends on what you''re optimizing for. If it''s RTO (bring VMs back online very quickly) then replicating the primary NFS datastore is great - just point a server at the replicated NFS store, import the VM and start. With an RPO that coincides with your snapshot frequency. Cheers, Erik
On Sep 18, 2012, at 7:31 AM, Eugen Leitl <eugen at leitl.org> wrote:> > Can I actually have a year''s worth of snapshots in > zfs without too much performance degradation?I''ve got 6 years of snapshots with no degradation :-) In general, there is not a direct correlation between snapshot count and performance. -- richard -- illumos Day & ZFS Day, Oct 1-2, 2012 San Fransisco www.zfsday.com Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120918/77c71bac/attachment.html>
I''ve installed a good number of all-in-one ZFS solutions, mostly based around NexentaStor and VMWare ESXi on HP ProLiant hardware. An example of this is documented on Server Fault at: http://serverfault.com/a/398579/13325 -- Edmund White ewwhite at mac.com On 9/18/12 10:31 AM, "Eugen Leitl" <eugen at leitl.org> wrote:> >I''m currently thinking about rolling a variant of > >http://www.napp-it.org/napp-it/all-in-one/index_en.html > >with remote backup (via snapshot and send) to 2-3 >other (HP N40L-based) zfs boxes for production in >our organisation. The systems themselves would >be either Dell or Supermicro (latter with ZIL/L2ARC >on SSD, plus SAS disks (pools as mirrors) all with >hardware pass-through). > >The idea is to use zfs for data integrity and >backup via data snapshot (especially important >data will be also back-up''d via conventional DLT >tapes). > >Before I test thisi -- > >Is anyone using this is in production? Any caveats? > >Can I actually have a year''s worth of snapshots in >zfs without too much performance degradation? > >Thanks. >_______________________________________________ >zfs-discuss mailing list >zfs-discuss at opensolaris.org >http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Richard Elling wrote:> On Sep 18, 2012, at 7:31 AM, Eugen Leitl <eugen at leitl.org > <mailto:eugen at leitl.org>> wrote: >> >> Can I actually have a year''s worth of snapshots in >> zfs without too much performance degradation? > > I''ve got 6 years of snapshots with no degradation :-)$ zfs list -t snapshot -r export/home | wc -l 1951 $ echo 1951 / 365 | bc -l 5.34520547945205479452 $ So you''re slightly ahead of my 5.3 years of daily snapshots:-) -- Andrew Gabriel
On Tue, 18 Sep 2012, Erik Ableson wrote:> > The bigger issue you''ll run into will be data sizing as a year''s > worth of snapshot basically means that you''re keeping a journal of > every single write that''s occurred over the year. If you are runningThe above is not a correct statement. The snapshot only preserves the file-level differences between the points in time. A snapshot does not preserve "every single write". Zfs does not even send every single write to underlying disk. In some usage models, the same file may be re-written 100 times between snapshots, or might not ever appear in any snapshot. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Tue, Sep 18, 2012 at 2:02 PM, Bob Friesenhahn < bfriesen at simple.dallas.tx.us> wrote:> On Tue, 18 Sep 2012, Erik Ableson wrote: > >> >> The bigger issue you''ll run into will be data sizing as a year''s worth of >> snapshot basically means that you''re keeping a journal of every single >> write that''s occurred over the year. If you are running >> > > The above is not a correct statement. The snapshot only preserves the > file-level differences between the points in time. A snapshot does not > preserve "every single write". Zfs does not even send every single write > to underlying disk. In some usage models, the same file may be re-written > 100 times between snapshots, or might not ever appear in any snapshot. > > >Depending on how frequently you''re taking snapshots, your change rate, and how long you keep the snapshots around, it may very well be true. It''s not universally true, but it''s also no universally false. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120918/525e1486/attachment.html>
On Tue, Sep 18, 2012 at 05:30:56PM +0200, Erik Ableson wrote:> > If you''re running ESXi with a vSphere license, I''d recommend looking at VDR (free with the vCenter license) for backing up the VMs to the little HPs since you get compressed and deduplicated backups that will minimize the replication bandwidth requirements. >Don''t look at VDR. It''s known to be very buggy and corrupt itself in no time. Also it''s known to do bad restores overwriting *wrong* VMs. VMware also killed it and replaced it with another product. -- Pasi
On 09/19/12 02:38 AM, Sa?o Kiselkov wrote:> On 09/18/2012 04:31 PM, Eugen Leitl wrote: > >> Can I actually have a year''s worth of snapshots in >> zfs without too much performance degradation? > Each additional dataset (not sure about snapshots, though) increases > boot times slightly, however, I''ve seen pools with several hundred > datasets without any serious issues, so yes, it is possible. Be > prepared, though, that the data volumes might be substantial (depending > on your overall data turn-around per unit time between the snapshots).The boot overhead for many (in my case >1200) filesystems isn''t as bad as it was. On our original Thumper I had to amalgamate all our user home directories into one filesystem due to slow boot. Now I have split them again to send over a slow WAN... Large numbers of snapshots (10''s of thousands) don''t appear to impact boot times. -- Ian.