Will Pressly
2013-Oct-07 23:44 UTC
[nsd-users] workload for testing a large number of zones?
Hi All, I apologize for the long note, but I would really appreciate the list's expertise on this matter. I am really interested in understanding the practical limits of how many zones NSD 4 can handle. When I mean large numbers of zones, I mean on the orders of millions. From all of the literature concerning NSD4 testing, I have only seen references to hundreds of thousands of zones. This leads me to some questions: 1. Is there a repository of a large number of zone files used in testing, large-scale, high performance authoritative-only name servers such as NSD/Knot/Yaddifa? If so, can someone offer a pointer to it, please? 2. Assuming there exists no such repository, if I have to generate 5 million random zones files, what are your thoughts on the distribution of size of zone files -- I guess in terms of record count for the most part. Also, the composition of the zone files -- meaning mostly A records, a few MX here and there, a few CNAMEs here and there, etc. EG: a normal distribution with a mean of 30 records and appropriate mins and maxes... including 90% A records, etc... 3. For this analysis, I want to leave DNSSEC out. I know DNSSEC can really bloat zone files -- so is this a horribly bad assumption? (Another assumption is that I do not have to mess with axfr at all.) 4. Where do we think NSD will break down, in terms of cardinality of the set of zone files? The internal data structures (rbtrees/radix tree) seem like they will hold performance-wise at large scale. Memory seems like it might be a concern. Thoughts? Thanks, Will Pressly -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.nlnetlabs.nl/pipermail/nsd-users/attachments/20131007/c755da6c/attachment.htm>
W.C.A. Wijngaards
2013-Oct-08 08:58 UTC
[nsd-users] workload for testing a large number of zones?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Will, On 10/08/2013 01:44 AM, Will Pressly wrote:> Hi All, > > I apologize for the long note, but I would really appreciate the > list's expertise on this matter. > > I am really interested in understanding the practical limits of how > many zones NSD 4 can handle. When I mean large numbers of zones, I > mean on the orders of millions. From all of the literature > concerning NSD4 testing, I have only seen references to hundreds of > thousands of zones.Experience is not what I can get you, but here's theoretical results. This is for 5M zones, all very small, no DNSSEC, configured as master zones, the nsd-mem tool (64bit machine) was used to estimate memory because it can give an estimate of memory without actually using the memory (unless it is one big zone). 27.801.003.701 ram usage (excl space for buffers) 13.760.000.000 disk usage (excl 12% space claimed for growth) Estimate: 41.561.003.701 data and big mmap 30.094.337.034 data and partial mmap The zonefiles for this test had SOA, 2xNS, one MX and an A record for 'www'. NSEC would be bigger, but NSEC3 would be disproportionally bigger, if you can choose your DNSSEC, NSEC will be nicer to your memory usage. nsd-mem parses about 88k-246k zones/minute on startup (from text zonefile, and writes them to nsd.db). For a slightly bigger zone, that you mention, with 30 records in the zone, mostly A, with 1 AAAA, 1 MX, 3 CNAMEs, here are some memory estimate results for that: 73.162.003.701 ram usage (excl space for buffers) 105.280.000.000 disk usage (excl 12% space claimed for growth) Estimate: 178.442.003.701 data and big mmap 90.708.670.367 data and partial mmap So, 30-40G for very small zone, 90-180G for 30name zone. Likely scales with the number of zones, so for 1 million zones that is 10G (tiny zone) and 20-40G (bigger zone) of memory. I do not really have any feedback on what zones are normal, or on other operational realities, but I thought a look at memory usage might make you pick a machine that can do it :-) The nsd.conf option zonefiles-check: no disables filetime checks, with could save you time on startup and "kill -HUP". Reread a zone's zonefile specifically by name with nsd-control reload <name> in this case. Best regards, Wouter> > This leads me to some questions: 1. Is there a repository of a > large number of zone files used in testing, large-scale, high > performance authoritative-only name servers such as > NSD/Knot/Yaddifa? If so, can someone offer a pointer to it, > please? 2. Assuming there exists no such repository, if I have to > generate 5 million random zones files, what are your thoughts on > the distribution of size of zone files -- I guess in terms of > record count for the most part. Also, the composition of the zone > files -- meaning mostly A records, a few MX here and there, a few > CNAMEs here and there, etc. EG: a normal distribution with a mean > of 30 records and appropriate mins and maxes... including 90% A > records, etc... 3. For this analysis, I want to leave DNSSEC out. I > know DNSSEC can really bloat zone files -- so is this a horribly > bad assumption? (Another assumption is that I do not have to mess > with axfr at all.) 4. Where do we think NSD will break down, in > terms of cardinality of the set of zone files? The internal data > structures (rbtrees/radix tree) seem like they will hold > performance-wise at large scale. Memory seems like it might be a > concern. Thoughts? > > Thanks, Will Pressly > > > _______________________________________________ nsd-users mailing > list nsd-users at NLnetLabs.nl > http://open.nlnetlabs.nl/mailman/listinfo/nsd-users >-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJSU8kvAAoJEJ9vHC1+BF+NM6MP/RSC9PGx8zc6xquyqsthCRU7 JOb+xY9yIVKMTFOO8QemwuBV67P3d2zRLauXpaLFSsPG/rPcHR6ArBhe7vB7Bw/g 3CQQtni66TofrLApTQ0zBXNEP9my4Wz8sMA2GCl6VVo86I5rrpN+0SDtR7BOxCH+ NmpxVOMR40QbLa16KId8khjM/XTlr/FioVEu87qYvetHkko+op/160l3eikviVam k6QXYm2J3ZPb0Q1zCGL1USmWbkBPhZLIB24kAdFDTYHByAv4Fco5+i9hX5pH+Dul wGjpHdUjaDZPZXiARfyokU3S8XPZUB6LGKyhYbliaf3ryOjcXSRIQT49wsWvWGUx 3WNjwu4806ItoXvrWbEJDzR0BiglarxG4cOI0QmGLBxdL4veaP9YS+cEF366bYTh V1vONPKCwqdmgkeqTaLBRRIpEEDrjaI23dsIirqWaZ1rvMTjO360TJRYFzRmi1d6 hQ7xoNtf0zH+AcMkvOnOj4KCXJLlu9iBEeZ98bHSt2G23Rjt3oK18DPoj76hlPxN g01vGh5u+NTwMNRE7tTa9rqDpBeuc+QozMn8hwfz2Mm9SXXaMbtJI6RRCKUocjrk PXdZIY1srTmO0xxHhlfnuiB+tiZbzanW0bZskWdL8q70JgeHBfyfj9OZSh6kky7J 4LGal70AHPUa465z+vZO =rJlN -----END PGP SIGNATURE-----