Does anyone know what the minimum value for zfs_arc_max should be set to? Does it depend on the amount of memory on the system, and - if so - is there a formula, or percentage, to use to determine what the minimum value is? Thanks Richard Paynter Confidentiality Note: This e-mail, including any attachment to it, may contain material that is confidential, proprietary, privileged and/or "Protected Health Information," within the meaning of the regulations under the Health Insurance Portability & Accountability Act as amended. If it is not clear that you are the intended recipient, you are hereby notified that you have received this transmittal in error, and any review, dissemination, distribution or copying of this e-mail, including any attachment to it, is strictly prohibited. If you have received this e-mail in error, please immediately return it to the sender and delete it from your system. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120516/18252472/attachment-0001.html>
On May 16, 2012, at 12:35 PM, "Paynter, Richard" <Richard.Paynter at infocrossing.com> wrote:> Does anyone know what the minimum value for zfs_arc_max should be set to? Does it depend on the amount of memory on the system, and ? if so ? is there a formula, or percentage, to use to determine what the minimum value is?It depends on the OS. In general, it is set to size of RAM - 1GB. For many Solaris releases, there is a bug that limits the effective use to 7/8 of RAM. -- richard -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120516/78b42263/attachment.html>
On Wed, May 16, 2012 at 3:35 PM, Paynter, Richard <Richard.Paynter at infocrossing.com> wrote:> Does anyone know what the minimum value for zfs_arc_max should be set to? > Does it depend on the amount of memory on the system, and ? if so ? is there > a formula, or percentage, to use to determine what the minimum value is?I would assume that it should not be set any lower than `kstat zfs::arcstats:c_min`, which I assume you can set with zfs_arc_min in /etc/system, but that does not really answer your question. Under Solaris 10 I have set it as low as 1 GB with no (apparent) ill effects except the desired reduction in the ARC and performance associated with such. This was NOT for production, but purely testing. Why are you trying to tune the ARC as _low_ as possible? In my experience the ARC gives up memory readily for other uses. The only place I _had_ to tune the ARC in production was a couple systems running an app that checks for free memory _before_ trying to allocate it. If the ARC has all but 1 GB in use, the app (which is looking for about 2.5 GB) does not see enough free memory and tosses an out of memory error without even trying to actually get the memory it needs. I have the ARC max tuned to 4 GB on these boxes with 16 GB RAM. Instead of my ARC using 10-12 GB it is stuck with only 4 GB, but the application does not quite with an error. Yes, I have flagged this behavior for the vendor, they don''t seem to care. -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) -> Assistant Technical Director, LoneStarCon 3 (http://lonestarcon3.org/) -> Sound Coordinator, Schenectady Light Opera Company ( http://www.sloctheater.org/ ) -> Technical Advisor, Troy Civic Theatre Company -> Technical Advisor, RPI Players
On Thu, 17 May 2012, Paul Kraus wrote:> > Why are you trying to tune the ARC as _low_ as possible? In my > experience the ARC gives up memory readily for other uses. The only > place I _had_ to tune the ARC in production was a couple systems > running an app that checks for free memory _before_ trying to allocate > it. If the ARC has all but 1 GB in use, the app (which is looking forOn my system I adjusted the ARC down due to running user-space applications with very bursty short-term large memory usage. Reducing the ARC assured that there would be no contention between zfs ARC and the applications. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
I''ll agree with Bob on this. A specific use case is a VirtualBox server hosting lots of guests. I even made a point of mentioning this tunable in the Solaris 10 Virtualization Essentials section on vbox :) There are several other use cases as well. Bob Bob Sent from my iPad On May 17, 2012, at 9:03 AM, Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:> On Thu, 17 May 2012, Paul Kraus wrote: >> >> Why are you trying to tune the ARC as _low_ as possible? In my >> experience the ARC gives up memory readily for other uses. The only >> place I _had_ to tune the ARC in production was a couple systems >> running an app that checks for free memory _before_ trying to allocate >> it. If the ARC has all but 1 GB in use, the app (which is looking for > > On my system I adjusted the ARC down due to running user-space applications with very bursty short-term large memory usage. Reducing the ARC assured that there would be no contention between zfs ARC and the applications. > > Bob > -- > Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 05/17/12 15:03, Bob Friesenhahn wrote:> On Thu, 17 May 2012, Paul Kraus wrote: >> >> Why are you trying to tune the ARC as _low_ as possible? In my >> experience the ARC gives up memory readily for other uses. The only >> place I _had_ to tune the ARC in production was a couple systems >> running an app that checks for free memory _before_ trying to allocate >> it. If the ARC has all but 1 GB in use, the app (which is looking for > > On my system I adjusted the ARC down due to running user-space > applications with very bursty short-term large memory usage. Reducing > the ARC assured that there would be no contention between zfs ARC and > the applications.If the system is running one app which expects to do lots of application level caching (and in theory, the app should be able to work out what''s worth caching and what isn''t better than any filesystem underneath it can guess), then you should be planning your memory usage accordingly. For example, a database server, you probably want to allocate much of the system''s memory to the database cache (in the case of Oracle, the SGA), leaving enough for a smaller ZFS arc and the memory required by the OS and app. Depends on the system and database size, but something like 50% SGA, 25% ZFS ARC, 25% for everything else might be an example, with the SGA disproportionally bigger on larger systems with larger databases. On my desktop system (supposed to be 8GB RAM, but currently 6GB due to a dead DIMM), I have knocked the ARC down to 1GB. I used to find the ARC wouldn''t shrink in size until system had got to the point of crawling along showing anon page-ins, and some app (usually firefox or thunderbird) had already become too difficult to use. I must admit I did this a long time ago, and ZFS''s shrinking of the ARC may be more proactive now than it was back then, but I don''t notice any ZFS performance issues with the ARC restricted to 1GB on a desktop system. It may have increase scrub times, but that happens when I''m in bed, so I don''t care. -- Andrew
On May 17, 2012, at 5:28 AM, Paul Kraus wrote:> On Wed, May 16, 2012 at 3:35 PM, Paynter, Richard > <Richard.Paynter at infocrossing.com> wrote: > >> Does anyone know what the minimum value for zfs_arc_max should be set to? >> Does it depend on the amount of memory on the system, and ? if so ? is there >> a formula, or percentage, to use to determine what the minimum value is? > > I would assume that it should not be set any lower than `kstat > zfs::arcstats:c_min`, which I assume you can set with zfs_arc_min in > /etc/system, but that does not really answer your question.arc_c_max and arc_c_min are used for different things. If you make significant changes there, then it is a good idea to set both zfs_arc_min, zfs_arc_max, and zfs_arc_meta_limit as a set. NB, the minimum size of these is 64MB. -- richard -- ZFS Performance and Training Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/3849da0d/attachment.html>
I have a customer who wants to set zfs_arc_max to 1G for a 16G system, and 2G for a 32G system. Both of these are SAP and/or Oracle db servers. They apparently want to maximize the amount of memory available for the applications. My question, now, is whether it would be advisable to do this, and - if not - what the impact might be. The 32G system is an M3000; the 16G system is a V490. Both are running Solaris 10 Update 8, patched to 147440-09. I have a feeling that setting zfs_arc_max this low is not a good idea, but would like to get some corroborating information/references, if possible. Thanks Rick From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Richard Elling Sent: Thursday, May 17, 2012 11:24 AM To: Paul Kraus Cc: ZFS Discussions Subject: Re: [zfs-discuss] zfs_arc_max values On May 17, 2012, at 5:28 AM, Paul Kraus wrote: On Wed, May 16, 2012 at 3:35 PM, Paynter, Richard <Richard.Paynter at infocrossing.com> wrote: Does anyone know what the minimum value for zfs_arc_max should be set to? Does it depend on the amount of memory on the system, and - if so - is there a formula, or percentage, to use to determine what the minimum value is? I would assume that it should not be set any lower than `kstat zfs::arcstats:c_min`, which I assume you can set with zfs_arc_min in /etc/system, but that does not really answer your question. arc_c_max and arc_c_min are used for different things. If you make significant changes there, then it is a good idea to set both zfs_arc_min, zfs_arc_max, and zfs_arc_meta_limit as a set. NB, the minimum size of these is 64MB. -- richard -- ZFS Performance and Training Richard.Elling at RichardElling.com +1-760-896-4422 Confidentiality Note: This e-mail, including any attachment to it, may contain material that is confidential, proprietary, privileged and/or "Protected Health Information," within the meaning of the regulations under the Health Insurance Portability & Accountability Act as amended. If it is not clear that you are the intended recipient, you are hereby notified that you have received this transmittal in error, and any review, dissemination, distribution or copying of this e-mail, including any attachment to it, is strictly prohibited. If you have received this e-mail in error, please immediately return it to the sender and delete it from your system. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/d5b9c79b/attachment-0001.html>
hi it will depend on how zfs is used in the system 1)root zpool only? 2)Db files also? 3)Is this RAC configuration? regards On 5/17/2012 3:19 PM, Paynter, Richard wrote:> > I have a customer who wants to set zfs_arc_max to 1G for a 16G system, > and 2G for a 32G system. Both of these are SAP and/or Oracle db > servers. They apparently want to maximize the amount of memory > available for the applications. > > My question, now, is whether it would be advisable to do this, and -- > if not -- what the impact might be. > > The 32G system is an M3000; the 16G system is a V490. Both are > running Solaris 10 Update 8, patched to 147440-09. > > I have a feeling that setting zfs_arc_max this low is not a good idea, > but would like to get some corroborating information/references, if > possible. > > Thanks > > Rick > > *From:*zfs-discuss-bounces at opensolaris.org > [mailto:zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Richard Elling > *Sent:* Thursday, May 17, 2012 11:24 AM > *To:* Paul Kraus > *Cc:* ZFS Discussions > *Subject:* Re: [zfs-discuss] zfs_arc_max values > > On May 17, 2012, at 5:28 AM, Paul Kraus wrote: > > > > On Wed, May 16, 2012 at 3:35 PM, Paynter, Richard > <Richard.Paynter at infocrossing.com > <mailto:Richard.Paynter at infocrossing.com>> wrote: > > > Does anyone know what the minimum value for zfs_arc_max should be set to? > > Does it depend on the amount of memory on the system, and -- if so > -- is there > > a formula, or percentage, to use to determine what the minimum > value is? > > > I would assume that it should not be set any lower than `kstat > zfs::arcstats:c_min`, which I assume you can set with zfs_arc_min in > /etc/system, but that does not really answer your question. > > arc_c_max and arc_c_min are used for different things. If you make > significant > > changes there, then it is a good idea to set both zfs_arc_min, > zfs_arc_max, and > > zfs_arc_meta_limit as a set. > > NB, the minimum size of these is 64MB. > > -- richard > > -- > > ZFS Performance and Training > > Richard.Elling at RichardElling.com <mailto:Richard.Elling at RichardElling.com> > > +1-760-896-4422 > > > > Confidentiality Note: This e-mail, including any attachment to it, may > contain material that is confidential, proprietary, privileged and/or > "Protected Health Information," within the meaning of the regulations > under the Health Insurance Portability & Accountability Act as > amended. If it is not clear that you are the intended recipient, you > are hereby notified that you have received this transmittal in error, > and any review, dissemination, distribution or copying of this e-mail, > including any attachment to it, is strictly prohibited. If you have > received this e-mail in error, please immediately return it to the > sender and delete it from your system. Thank you. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/7be14fc1/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: laotsao.vcf Type: text/x-vcard Size: 600 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/7be14fc1/attachment.vcf>
sorry what is data storage? ASS U ME the root is local hdd regards On 5/17/2012 3:31 PM, Hung-Sheng Tsao Ph.D. wrote:> > hi > it will depend on how zfs is used in the system > 1)root zpool only? > 2)Db files also? > 3)Is this RAC configuration? > regards > > On 5/17/2012 3:19 PM, Paynter, Richard wrote: >> >> I have a customer who wants to set zfs_arc_max to 1G for a 16G >> system, and 2G for a 32G system. Both of these are SAP and/or Oracle >> db servers. They apparently want to maximize the amount of memory >> available for the applications. >> >> My question, now, is whether it would be advisable to do this, and ? >> if not ? what the impact might be. >> >> The 32G system is an M3000; the 16G system is a V490. Both are >> running Solaris 10 Update 8, patched to 147440-09. >> >> I have a feeling that setting zfs_arc_max this low is not a good >> idea, but would like to get some corroborating >> information/references, if possible. >> >> Thanks >> >> Rick >> >> *From:*zfs-discuss-bounces at opensolaris.org >> [mailto:zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Richard >> Elling >> *Sent:* Thursday, May 17, 2012 11:24 AM >> *To:* Paul Kraus >> *Cc:* ZFS Discussions >> *Subject:* Re: [zfs-discuss] zfs_arc_max values >> >> On May 17, 2012, at 5:28 AM, Paul Kraus wrote: >> >> >> >> On Wed, May 16, 2012 at 3:35 PM, Paynter, Richard >> <Richard.Paynter at infocrossing.com >> <mailto:Richard.Paynter at infocrossing.com>> wrote: >> >> >> Does anyone know what the minimum value for zfs_arc_max should be set to? >> >> Does it depend on the amount of memory on the system, and ? if so >> ? is there >> >> a formula, or percentage, to use to determine what the minimum >> value is? >> >> >> I would assume that it should not be set any lower than `kstat >> zfs::arcstats:c_min`, which I assume you can set with zfs_arc_min in >> /etc/system, but that does not really answer your question. >> >> arc_c_max and arc_c_min are used for different things. If you make >> significant >> >> changes there, then it is a good idea to set both zfs_arc_min, >> zfs_arc_max, and >> >> zfs_arc_meta_limit as a set. >> >> NB, the minimum size of these is 64MB. >> >> -- richard >> >> -- >> >> ZFS Performance and Training >> >> Richard.Elling at RichardElling.com >> <mailto:Richard.Elling at RichardElling.com> >> >> +1-760-896-4422 >> >> >> >> Confidentiality Note: This e-mail, including any attachment to it, >> may contain material that is confidential, proprietary, privileged >> and/or "Protected Health Information," within the meaning of the >> regulations under the Health Insurance Portability & Accountability >> Act as amended. If it is not clear that you are the intended >> recipient, you are hereby notified that you have received this >> transmittal in error, and any review, dissemination, distribution or >> copying of this e-mail, including any attachment to it, is strictly >> prohibited. If you have received this e-mail in error, please >> immediately return it to the sender and delete it from your system. >> Thank you. >> >> >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > -- >-- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/f71a2b74/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: laotsao.vcf Type: text/x-vcard Size: 620 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/f71a2b74/attachment-0001.vcf>
On May 17, 2012, at 12:19 PM, Paynter, Richard wrote:> I have a customer who wants to set zfs_arc_max to 1G for a 16G system, and 2G for a 32G system. Both of these are SAP and/or Oracle db servers. They apparently want to maximize the amount of memory available for the applications.This is the best use case of the zfs_arc_max tunable.> > My question, now, is whether it would be advisable to do this, and ? if not ? what the impact might be.For this use case, it is a good idea. The impact of not setting it occurs when you later want to start a memory-hogging program, the request for arc to shrink can take a little while to satisfy, and the memory-hog might not be designed to cope with retrying large memory allocation requests.> > The 32G system is an M3000; the 16G system is a V490. Both are running Solaris 10 Update 8, patched to 147440-09. > > I have a feeling that setting zfs_arc_max this low is not a good idea, but would like to get some corroborating information/references, if possible.This use case is why the zfs_arc_max tunable exists. -- richard -- ZFS Performance and Training Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120517/d740a6ff/attachment.html>