Hello, Im about to migrate a 6Tb database from Veritas Volume Manager to ZFS, I want to set arc_max parameter so ZFS cant use all my system''s memory, but i dont know how much i should set, do you think 24Gb will be enough for a 6Tb database? obviously the more the better but i cant set too much memory. Have someone implemented succesfully something similar? We ran some test and the usage of memory was as follow: (With Arc_max at 30Gb) Kernel = 18Gb ZFS DATA = 55Gb Anon = 90Gb Page Cache = 10Gb Free = 25Gb My system have 192Gb RAM and the database SGA = 85Gb. I would appreciate if someone could tell me about their experience. Best Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121201/07dbd312/attachment.html>
On 2012-12-01 15:05, Fung Zheng wrote:> Hello, > > Im about to migrate a 6Tb database from Veritas Volume Manager to ZFS, I > want to set arc_max parameter so ZFS cant use all my system''s memory, > but i dont know how much i should set, do you think 24Gb will be enough > for a 6Tb database? obviously the more the better but i cant set too > much memory. Have someone implemented succesfully something similar?Not claiming to be an expert fully ready to (mis)lead you (and I haven''t done similar quests for databases), I might suggest that you set the ZFS dataset option "primarycache=metadata" on your dataset which holds the database. (PS: what OS version are you on?) The general consent is that serious apps like databases are better than generic OS/FS caches at caching what the DBMS deems fit (and the data blocks might get cached twice - in ARC and in app cache), however having ZFS *metadata* cached should speed up your HDD IO - the server might keep the {much of} needed block map in RAM and not have to start by fetching it from disks every time. Also make sure to set the "recordsize" attribute as appropriate for your DB software - to match the DB block size. Usually this ranges around 4, 8 or 16Kb (with zfs default being 128Kb for filesystem datasets). You might also want to put non-tablespace files (logs, indexes, etc.) into separate datasets with their appropriate record sizes - this would let you play with different caching and compression settings, if applicable (you might save some IOPS by reading and writing less mechanical data at a small hit to CPU horsepower by using LZJB). Also such systems tend to benefit from SSD L2ARC read-caches and SSD SLOG (ZIL) write-caches. These are different pieces of equipment with distinct characteristics (SLOG is mirrored, small, write-mostly, and should endure write-wear and survive sudden poweroffs; L2ARC is big, fast for small random reads, moderately reliable). If you do use a big L2ARC, you might indeed want to have both ZFS caches for frequently accessed datasets (i.e. index) to hold both the userdata and metadata (as is the default), while the randomly accessed tablespaces might be or not be good candidates for such caching - however you can test this setting change on the fly. I believe, you must allow caching userdata for a dataset in RAM if you want to let it spill over onto L2ARC. HTH, //Jim Klimov
Hello, Thanks for you reply, i forgot to mention that the doc "Configuring ZFS for an Oracle Database" was followed, this include primarycache, logbias, recordsize properties, all the best practices was followed and my only doubt is the arc_max parameter, i want to know if 24Gb is good enough for a 6Tb database, someone have had implemented something similar? which was the value used for arc_max? OS = Solaris 10 Regards On Sat, Dec 1, 2012 at 5:28 PM, Jim Klimov <jimklimov at cos.ru> wrote:> On 2012-12-01 15:05, Fung Zheng wrote: > >> Hello, >> >> Im about to migrate a 6Tb database from Veritas Volume Manager to ZFS, I >> want to set arc_max parameter so ZFS cant use all my system''s memory, >> but i dont know how much i should set, do you think 24Gb will be enough >> for a 6Tb database? obviously the more the better but i cant set too >> much memory. Have someone implemented succesfully something similar? >> > > Not claiming to be an expert fully ready to (mis)lead you (and I > haven''t done similar quests for databases), I might suggest that > you set the ZFS dataset option "primarycache=metadata" on your > dataset which holds the database. (PS: what OS version are you on?) > > The general consent is that serious apps like databases are better > than generic OS/FS caches at caching what the DBMS deems fit (and > the data blocks might get cached twice - in ARC and in app cache), > however having ZFS *metadata* cached should speed up your HDD IO - > the server might keep the {much of} needed block map in RAM and not > have to start by fetching it from disks every time. > > Also make sure to set the "recordsize" attribute as appropriate for > your DB software - to match the DB block size. Usually this ranges > around 4, 8 or 16Kb (with zfs default being 128Kb for filesystem > datasets). You might also want to put non-tablespace files (logs, > indexes, etc.) into separate datasets with their appropriate record > sizes - this would let you play with different caching and compression > settings, if applicable (you might save some IOPS by reading and > writing less mechanical data at a small hit to CPU horsepower by > using LZJB). > > Also such systems tend to benefit from SSD L2ARC read-caches and > SSD SLOG (ZIL) write-caches. These are different pieces of equipment > with distinct characteristics (SLOG is mirrored, small, write-mostly, > and should endure write-wear and survive sudden poweroffs; L2ARC is > big, fast for small random reads, moderately reliable). > > If you do use a big L2ARC, you might indeed want to have both ZFS > caches for frequently accessed datasets (i.e. index) to hold both > the userdata and metadata (as is the default), while the randomly > accessed tablespaces might be or not be good candidates for such > caching - however you can test this setting change on the fly. > I believe, you must allow caching userdata for a dataset in RAM > if you want to let it spill over onto L2ARC. > > HTH, > //Jim Klimov > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121201/685cc433/attachment.html>
On Sat, 1 Dec 2012, Fung Zheng wrote:> Hello, > > Thanks for you reply, i forgot to mention that the doc "Configuring ZFS for an Oracle Database" was followed, this > include primarycache, logbias, recordsize properties, all the best practices was followed and my only doubt is the > arc_max parameter, i want to know if 24Gb is good enough for a 6Tb database, someone have had implemented something > similar? which was the value used for arc_max?As I recall, you can tune zfs_arc_max while the system is running so you can easily adjust this while your database is running and observe behavior and without rebooting. It is possible that my recollection is wrong though. If my recollection is correct, then it is not so important to know what is "good enough" before starting to put your database in service. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/