We are having a server running zfs root with 64G RAM and the system has 3 zones running oracle fusion app and zfs cache is using 40G memory as per kstat zfs:0:arcstats:size. and system shows only 5G of memory is free rest is taken by kernel and 2 remaining zones. Now my problem is that fusion guys are getting not enough memory message while starting their application due to top and vmstat shows 5G as free memory. But i read ZFS cache releases memory as required by the application so why fusion application is not starting up. Is there some we can do to decrease the ARC Cache usage on the fly without rebooting the global zone ? -- This message posted from opensolaris.org
On Jun 3, 2010, at 10:33 AM, Ketan wrote:> We are having a server running zfs root with 64G RAM and the system has 3 zones running oracle fusion app and zfs cache is using 40G memory as per > > kstat zfs:0:arcstats:size. and system shows only 5G of memory is free rest is taken by kernel and 2 remaining zones. > > Now my problem is that fusion guys are getting not enough memory message while starting their application due to top and vmstat shows 5G as free memory. But i read ZFS cache releases memory as required by the application so why fusion application is not starting up.ZFS will release the memory, but sometimes this can take a moment and in the meantime, applications which are not written to retry allocations are likely to bail out.> Is there some we can do to decrease the ARC Cache usage on the fly without rebooting the global zone ?This is a case where limiting the ARC size can be useful. For more information see the ZFS Evil Tuning Guide http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache -- richard -- Richard Elling richard at nexenta.com +1-760-896-4422 ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/
Thanx Rick .. but this guide does not offer any method to reduce the ARC cache size on the fly without rebooting the system. And the system''s memory utilization is running very high since 2 weeks now and just 5G of memory is free. And the arc cache is showing 40G of usage. and its not decreasing its just increasing. -- This message posted from opensolaris.org
On Thu, 2010-06-03 at 11:36 -0700, Ketan wrote:> Thanx Rick .. but this guide does not offer any method to reduce the ARC cache size on the fly without rebooting the system. And the system''s memory utilization is running very high since 2 weeks now and just 5G of memory is free. And the arc cache is showing 40G of usage. and its not decreasing its just increasing.Write a program which does a bunch of mallocs. This should cause zfs to release data. Then when your program exits, you''ll be able to use the memory it left behind. For example: /* usage: a.out <size> <repeat> */ #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { int sz, cnt, fail, i; if (argc < 2) { printf("usage: %s <size> <count>\n", argv[0]); exit(1); } sz = atoi(argv[1]); cnt = atoi(argv[2]); fail = 0; printf("Performing %d allocations of %d bytes\n", cnt, sz); for (i = 0; i < cnt; i++) { if (malloc(sz) == NULL) { fail++; } } printf("Mallocs: %d succceeded, %d failed\n", cnt - fail, fail); exit(fail); } -- Garrett
So you want me to run this on production global zone running 3 other production applications .. :-) -- This message posted from opensolaris.org
On Thu, Jun 3, 2010 at 1:06 PM, Ketan <vibhuneb at gmail.com> wrote:> So you want me to run this on production global zone running 3 other production applications .. :-)It''s probably lower impact than a reboot... -B -- Brandon High : bhigh at freaks.com
On Thu, 3 Jun 2010, Garrett D''Amore wrote:> On Thu, 2010-06-03 at 11:36 -0700, Ketan wrote: >> Thanx Rick .. but this guide does not offer any method to reduce the ARC cache size on the fly without rebooting the system. And the system''s memory utilization is running very high since 2 weeks now and just 5G of memory is free. And the arc cache is showing 40G of usage. and its not decreasing its just increasing. > > Write a program which does a bunch of mallocs. This should cause zfs to > release data. Then when your program exits, you''ll be able to use the > memory it left behind.Your sample program makes assumptions which may not be true. For example, it assumes that allocating an address range will consume actual "memory" (rather than just a reservation in the huge 64-bit address space) from the system. This is not necessarily the case. If your program arranges to modify a byte in each page of allocated memory, then there is a better chance of success (but not assured). Expect system performance to suffer dramatically. SunOS 4 provided a program named "chill" which performed this function. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
When I looked for references on ARC freeing algo, I did find some lines of codes talking about freeing ARC when memory is under pressure. Nice...but what could be memory under pressure in the kernel syntax ? Jumping from C lines to blogs to docs....I went back to basics : - lotsfree - fastscan IMHO the lotsfree (The greater of 1/64th of physical memory or 512 Kbytes) is stupid when you''re using ZFS. Nicolas