Trevor Vaughan
2010-Nov-03 17:02 UTC
[Puppet Users] Graph explosion and workaround technique with stages.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I''m cross-posting this to both the users and dev groups so that the users can use the workaround technique to reduce graph churn and the devs can hopefully fix the issue. So, I''m loving stages, but they appear, by default, to generate a many-to-many dependency graph. By adding ''pre'' and ''post'' stages that contain a single item, the graphs are vastly reduced in complexity and we''re seeing something like a 3X speedup in graph execution in simple test cases. I''ve attached a couple of puppet files illustrating the issue with their associated graphs. If the graphs could be auto-reduced to a single point prior to execution automatically, it would be quite helpful. Thanks, Trevor - -- Trevor Vaughan Vice President, Onyx Point, Inc. email: tvaughan@onyxpoint.com phone: 410-541-ONYX (6699) pgp: 0x6C701E94 - -- This account not approved for unencrypted sensitive information -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJM0ZWKAAoJECNCGV1OLcyprioH/0jeeUnc64Aa7dsEojgvfME0 zTIYOPKK6yWrO/01zQ+fjWk2YjEVI0G7rjwu8Oiqyw3Dx7uQjwbC2X+CHEtADXcQ Nd/+tFw4aEmIGeoUPpA57ieh+x3a5AmGkMk+eK6DU1u/ZicsgaK59mIU4lj76kL6 8C+nlPF/euDGni//u0mBtWd+zLJSUPhnDfE5Fz3szh8XqxWnzOjqSzELe8+dVWC4 rvdVf4ZJp2mDN4DKfK3qatXYbWClOo8RWBVB2aPMTtoKK8/KMFHMQYBA8zBbaiZZ Mli54ialu+2kPzS+OZFIWbSzh98uB9E/mwDFJEcY1m0EhUHsVMejrvcke/RpE6o=E6AG -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Kanies
2010-Nov-03 23:55 UTC
[Puppet Users] Re: [Puppet-dev] Graph explosion and workaround technique with stages.
On Nov 3, 2010, at 10:02 AM, Trevor Vaughan wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I''m cross-posting this to both the users and dev groups so that the > users can use the workaround technique to reduce graph churn and the > devs can hopefully fix the issue. > > So, I''m loving stages, but they appear, by default, to generate a > many-to-many dependency graph. > > By adding ''pre'' and ''post'' stages that contain a single item, the graphs > are vastly reduced in complexity and we''re seeing something like a 3X > speedup in graph execution in simple test cases. > > I''ve attached a couple of puppet files illustrating the issue with their > associated graphs. > > If the graphs could be auto-reduced to a single point prior to execution > automatically, it would be quite helpful.Thanks a ton for tracking this down. Unfortunately, I don''t think it''s as clear as your pictures make it seem. Which graphs are these pictures produced from? There are three graphs you might reasonably look at - the normal, pre-relationship graph which has just containment edges (I think this is normally called resources.dot), the graph with just the relationships (relationships.dot, I think), and the essentially multiplied graph of dependency times containment edges (not sure what this is called). The latter will generally be a many to many graph, and I think that''s what you''re using. The reason it''s not in the case of your transitions version is that we had to keep those class objects in there (we would normally remove them) because if we didn''t then certain kinds of dependencies got missed. So basically, this isn''t so much a feature as the side-effect of an ugly hack to a bug. I think the right answer is to get rid of the need for this graph at all by having a single graph contain both dependency and containment edges, thus not needing the many to many nature. I tried very hard to get this into 2.6 but couldn''t finish it in time. In seeing this multiplier, it''s probably justification to raise the priority on this, which doesn''t hurt my feelings at all. For those interested in the code, it''s in my refactor/master/3691-no_relationship_graph branch but is likely no longer in a mergeable state. -- The easiest way for your children to learn about money is for you not to have any. -- Katharine Whitehorn --------------------------------------------------------------------- Luke Kanies -|- http://puppetlabs.com -|- +1(615)594-8199 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Trevor Vaughan
2010-Nov-04 11:11 UTC
[Puppet Users] Re: [Puppet-dev] Graph explosion and workaround technique with stages.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 They were produced from the expanded-relationships.dot. Glad to hear that there''s a fix on the way, we''re really liking the potential of stages! Trevor On 11/03/2010 07:55 PM, Luke Kanies wrote:> On Nov 3, 2010, at 10:02 AM, Trevor Vaughan wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> I''m cross-posting this to both the users and dev groups so that the >> users can use the workaround technique to reduce graph churn and the >> devs can hopefully fix the issue. >> >> So, I''m loving stages, but they appear, by default, to generate a >> many-to-many dependency graph. >> >> By adding ''pre'' and ''post'' stages that contain a single item, the graphs >> are vastly reduced in complexity and we''re seeing something like a 3X >> speedup in graph execution in simple test cases. >> >> I''ve attached a couple of puppet files illustrating the issue with their >> associated graphs. >> >> If the graphs could be auto-reduced to a single point prior to execution >> automatically, it would be quite helpful. > > Thanks a ton for tracking this down. > > Unfortunately, I don''t think it''s as clear as your pictures make it seem. Which graphs are these pictures produced from? > > There are three graphs you might reasonably look at - the normal, pre-relationship graph which has just containment edges (I think this is normally called resources.dot), the graph with just the relationships (relationships.dot, I think), and the essentially multiplied graph of dependency times containment edges (not sure what this is called). > > The latter will generally be a many to many graph, and I think that''s what you''re using. The reason it''s not in the case of your transitions version is that we had to keep those class objects in there (we would normally remove them) because if we didn''t then certain kinds of dependencies got missed. > > So basically, this isn''t so much a feature as the side-effect of an ugly hack to a bug. > > I think the right answer is to get rid of the need for this graph at all by having a single graph contain both dependency and containment edges, thus not needing the many to many nature. I tried very hard to get this into 2.6 but couldn''t finish it in time. In seeing this multiplier, it''s probably justification to raise the priority on this, which doesn''t hurt my feelings at all. > > For those interested in the code, it''s in my refactor/master/3691-no_relationship_graph branch but is likely no longer in a mergeable state. >- -- Trevor Vaughan Vice President, Onyx Point, Inc. email: tvaughan@onyxpoint.com phone: 410-541-ONYX (6699) pgp: 0x6C701E94 - -- This account not approved for unencrypted sensitive information -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJM0pTcAAoJECNCGV1OLcypNbMH/jaq1VZQCML/RcCcN78ZXt71 b/asA0CqRNQUpnSgh1Ro7i6KjSEfyoa93EbMVsMhRcTyS9EravksN5ElfojEY0oX oE+vd5eu87N65Frvl0ZHvmkI0vBkUQU8Mu61+/lDkH1KCA+rT8FYcdkQIl9yNU03 Z7r9f/3dGbBnA25xeaiyu9uLqc/AuwKGR4ECMo2GpRyFAV4Xg4s/Wd3qC2mc3kiX VwW/1i40bXtG/X4jhjrLFw21ucRkfidYsEnszUkVDWRurBHCdPZm2QZYHBhpuY+u 55JzxSQw4V48MeHXHQ72FFPZQdZYFyRndbF38821gQ0YR5NzpXg7XQ9P9TzlJhM=jnAP -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.