Roy Sigurd Karlsbakk
2011-Feb-01 02:16 UTC
[zfs-discuss] ZFS and L2ARC memory requirements?
> Even *with* an L2ARC, your memory requirements are *substantial*, > because the L2ARC itself needs RAM. 8 GB is simply inadequate for your > test.With 50TB storage, and 1TB if L2ARC, with no dedup, what amount of ARC would you would you recommeend? And then, _with_ dedup, what would you recommemend? Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
Roy Sigurd Karlsbakk
2011-Feb-01 02:40 UTC
[zfs-discuss] ZFS and L2ARC memory requirements?
----- Original Message -----> > Even *with* an L2ARC, your memory requirements are *substantial*, > > because the L2ARC itself needs RAM. 8 GB is simply inadequate for > > your > > test. > > With 50TB storage, and 1TB if L2ARC, with no dedup, what amount of ARC > would you would you recommeend? > > And then, _with_ dedup, what would you recommemend?make that 100TB of storage Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
On Jan 31, 2011, at 6:16 PM, Roy Sigurd Karlsbakk wrote:>> Even *with* an L2ARC, your memory requirements are *substantial*, >> because the L2ARC itself needs RAM. 8 GB is simply inadequate for your >> test. > > With 50TB storage, and 1TB if L2ARC, with no dedup, what amount of ARC would you would you recommeend?Just the L2ARC directory can consume 24GB of ARC for 8KB records.> And then, _with_ dedup, what would you recommemend?Depends on the specific OS release, but I think you should plan on using dedup where the cost/GB is expensive relative to the cost/IOPS. -- richard
On 01/31/11 06:40 PM, Roy Sigurd Karlsbakk wrote:> ----- Original Message ----- > >>> Even *with* an L2ARC, your memory requirements are *substantial*, >>> because the L2ARC itself needs RAM. 8 GB is simply inadequate for >>> your >>> test. >>> >> With 50TB storage, and 1TB if L2ARC, with no dedup, what amount of ARC >> would you would you recommeend? >>First off... a *big* caveat. I am *not* a tuning expert. We have people in our company who can help you out from operational experience if you want to configure a system like this -- one of them -- Richard Elling -- is a frequently seen face around here. That said, I''m going to respond from my *very* rough understanding of how these structures play together.... So, I''d say: Alot. 1TB for ARC sounds like a rather largeish amount. I don''t know offhand the typical ratio of ARC -> L2ARC, but note that every entry in the L2ARC requires at least some book-keeping in the ARC (which is in RAM). I''ve seen people say that you can have anywhere from 10x RAM to 20x RAM for L2ARC. It sounds like this means between 50GB and 100GB *just* for an L2ARC of this size. That''s without dedup.>> And then, _with_ dedup, what would you recommemend? >> > make that 100TB of storage >With 100TB of storage, fully consumed, your DDT is going to need to be about 500GB (assuming 64K block size, which may or may not be a good average). That whole DDT will fit into the L2ARC above, so you probably can get by with just the 50-100GB of RAM. But I recommend allocating *more* than that, because you really don''t want *every* write to the DDT to go to L2ARC, and you really *do* want to have some memory available for things besides just the ARC. Generally, this feels like a 256GB memory configuration to me. Fundamentally, the best way to reduce the memory impact is to use dedup much more sparingly, and configure a much smaller L2ARC. You also need to analyze your workload to see if you''ll benefit from having L2ARC apart from the DDT itself. - Garrett
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Roy Sigurd Karlsbakk > > > Even *with* an L2ARC, your memory requirements are *substantial*, > > because the L2ARC itself needs RAM. 8 GB is simply inadequate for your > > test. > > With 50TB storage, and 1TB if L2ARC, with no dedup, what amount of ARC > would you would you recommeend?Without dedup and without L2ARC, the amount of ram you require is unrelated to amount of storage you have. Your ram requirement depends on what applications you run. Any excess ram you have will be used for ARC (that is, L1 ARC) and therefore used to benefit performance. So excess ram is always good. Do not be a cheapskate with ram. Regardless of whether you use ZFS, or any other filesystem, or any other OS, even windows or linux. Excess ram is always a good thing. It always improves stability and improves performance. If you are using a laptop and not serving anything and performance is not a major concern and you''re free to reboot whenever you want, then you can survive on 2G of ram. But a server presumably DOES stuff and you don''t want to reboot frequently. I''d recommend 4G minimally, 8G standard, and if you run any applications (databases, web servers, symantec products) then add more. And if you use dedup, or l2arc, then add more.> And then, _with_ dedup, what would you recommemend?If you have dedup enabled, add slightly under 3G ram for every 1TB unique data in your pool on top of whatever you''ve selected for your base ram configuration.
100TB storage? Cool! What is the hardware? How many discs? Gief me ze hardware! :oP -- This message posted from opensolaris.org
Roy Sigurd Karlsbakk
2011-Feb-04 22:06 UTC
[zfs-discuss] ZFS and L2ARC memory requirements?
> 100TB storage? Cool! What is the hardware? How many discs? Gief me ze > hardware! :oP> 100TB storage? Cool! What is the hardware? How many discs? Gief me ze hardware! :oPWe have two 100TB boxes running as Bacula storage agents, each a supermicro box with 40 disks (2 2,5" 500GB drives internally for the rpool, 2 SSDs for SLOG). The remaining 35 slots are used for two L2ARC devices (Micron, former Cruzial RealSSD C300). The drives are WD Black, and they seem fairly stable, but we have gotten some bad iostat from some of them (though, those are returned). The storage consists mainly of 11 VDEVs, 7 2TB drives each, in RAIDz2. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
Roy Sigurd Karlsbakk
2011-Feb-05 05:20 UTC
[zfs-discuss] ZFS and L2ARC memory requirements?
> We have two 100TB boxes running as Bacula storage agents, each a > supermicro box with 40 disks (2 2,5" 500GB drives internally for the > rpool, 2 SSDs for SLOG). The remaining 35 slots are used for two L2ARC > devices (Micron, former Cruzial RealSSD C300). The drives are WD > Black, and they seem fairly stable, but we have gotten some bad iostat > from some of them (though, those are returned). The storage consists > mainly of 11 VDEVs, 7 2TB drives each, in RAIDz2.Add a 45 drive JBOD to that to make the math work. The JBOD is a similar supermicro box, but without the mobo, attached with SAS. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.