Puppet users - May 2011 - Concat Module posted to Onyx Point Github.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

All,

I''d love to get feedback on a module that turns R.I. Pienaar''s
file concatenation idea into a native type.

The code is located at the Onyx Point Github:
https://github.com/onyxpoint/pupmod-concat

We found that using this instead of the purely puppet-based implementation we
saw vast improvements in speed, graph complexity, and memory usage.

We''ll be getting this to the module forge soon, but I thought that
I''d put it out for public perusal in the mean time.

Improvements welcome!

Thanks!

Trevor

- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaughan@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJN1E/nAAoJECNCGV1OLcypS/0H/AnsyhLfa+gL2tHQviNvEc7g
dQ72ewqH84UawOTZjjHvkLS7MbqiIQqvrPx5ULJTnN1wHn1wBtwXujKsaJivRkwn
8iE+uY0GxDzgFtid5Rm+QyuZWKr2sxddKhfgU0A2m99CjlvXyU7x6evYEa/W6A2t
2W8EdeYxBw2lqVGD1V1V00D279utpdtbMRukmgmrfRY/EzFvUV81Mad4FUe3i6YI
V3UFbpmTlMKenN9f6vaedQK3cDDddnUp14vVQe9iNOUNpiJq6CMWczwLv9vs0YS4
FfAr+30tCWk8vKkGp8Tx5foUMWjd/zPxpWHYuBh+/4Vv3iFRcQaaZeT5CQYUjTA=+o0H
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> All,
> 
> I''d love to get feedback on a module that turns R.I.
Pienaar''s file
> concatenation idea into a native type.
> 
> The code is located at the Onyx Point Github:
> https://github.com/onyxpoint/pupmod-concat
> 
> We found that using this instead of the purely puppet-based
> implementation we saw vast improvements in speed, graph complexity,
> and memory usage.
> 
> We''ll be getting this to the module forge soon, but I thought that
> I''d put it out for public perusal in the mean time.
> 
> Improvements welcome!

This is great thanks a lot, not tested it but I have hoped someone 
would take up the challenge for ages :)

Pretty interesting/odd ordering approach.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks, I hope you like it!

The ordering approach was a bit troublesome but we needed extreme flexibility
for various types of file targets so this is what we ended up with.

Trevor

On 05/18/2011 07:10 PM, R.I.Pienaar wrote:
> 
> 
> ----- Original Message -----
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> All,
>>
>> I''d love to get feedback on a module that turns R.I.
Pienaar''s file
>> concatenation idea into a native type.
>>
>> The code is located at the Onyx Point Github:
>> https://github.com/onyxpoint/pupmod-concat
>>
>> We found that using this instead of the purely puppet-based
>> implementation we saw vast improvements in speed, graph complexity,
>> and memory usage.
>>
>> We''ll be getting this to the module forge soon, but I thought
that
>> I''d put it out for public perusal in the mean time.
>>
>> Improvements welcome!
> 
> 
> This is great thanks a lot, not tested it but I have hoped someone 
> would take up the challenge for ages :)
> 
> Pretty interesting/odd ordering approach.
> 
- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaughan@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJN1Fp/AAoJECNCGV1OLcypQGQH/24wb/DbX7yx3o3yueulolK3
CT98w+lYgODzCE7szOMo04Shu8SE59FCfBukfH/FHxNvZNERa82VntyenkiCVkEn
EHa5r4JsyCN0hUHEnFr21jFwBvzjvUblmwolKVHw5FLyy/CbKyX2tI6USl0BWtEf
EvP055nQhYs2nOfbJ4tVF3u6c8Q2vO+xquQeYxqu2JDAw95Mz1FfqvSYu5y2MkV/
vUlS91yJYpzW5LSQSZmj+8Q0CZGgrIsy2XxOgXT/DPcvgcIxaaKja4+dW5zpmuc1
LEx0YAXCd9EAJWgSm52Yz9Po1uUeK9Jl/u40qm6oKPKOBhsH0TqXs18mycMA5rg=XtEH
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Trevor

Thanks for the module.

I''ve been testing out the concat module, thanks, but the only issue I
see is
files keep getting ''executed successfully'' each round.

Is there any way to not have it do this? I haven''t really looked that
your
ruby logic yet.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Larry,

Unfortunately, no.

We didn''t implement a insync? check since we just needed to overwrite
the target files anyway.

This is something that appears to be on the todo list as Puppet Labs
pulls this into the core.

We might beat them to it, but it''s doubtful at this time.

Thanks!

Trevor

On Wed, May 25, 2011 at 10:52 PM, Larry Ludwig <larrylud@gmail.com>
wrote:
> Hi Trevor
> Thanks for the module.
>
> I''ve been testing out the concat module, thanks, but the only
issue I see is
> files keep getting ''executed successfully'' each round.
> Is there any way to not have it do this? I haven''t really looked
that your
> ruby logic yet.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaughan@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Trevor,

I''ll add my thanks to the pile, this is pretty awesome stuff! I
compared the
--graph results with the old concat moule, and this one - impressive 
difference :)

As Larry says, the fact that it executes Concat_build on every run is a 
stopper for us, since the dependant services get restarted and we get report 
emails of any changes.

Also, I can''t see any way to hook into the filebucket for backups. I
managed
to hack in a "cp file file.bak" type functionality just before it
actually
cp''s the newly-built file. However, my Ruby-fu isn''t up to
pulling in the
filebucket stuff yet - might be worth adding to the todo list.

Thanks again for the great module though :)

Greg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/OXRxUlV4QnVxU3NK.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi again,

After an hour or three of hacking, I''ve managed to add the insync?
checks to
both the build and fragment types. It''s the first time I''ve
ever really
messed with types and providers, so it''s very ugly (and probably has
bugs),
however I''m happy to share my patches. Would that be best a) here b) in
puppet-dev, or c) direct to Onyx Point?

Cheers,
Greg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/bDJFMXd4Q0VFMG9K.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Greg Sutcliffe wrote:
> Hi again,
> 
> After an hour or three of hacking, I''ve managed to add the insync?
> checks to both the build and fragment types. It''s the first time
I''ve
> ever really messed with types and providers, so it''s very ugly
(and
> probably has bugs), however I''m happy to share my patches. Would
that be
> best a) here b) in puppet-dev, or c) direct to Onyx Point?
> 
Awesome stuff! The puppet-dev list is the best place.

Cheers

James

-- 
James Turnbull
Puppet Labs
1-503-734-8571

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for the patch set!

Feel free to add a pull request on GitHub, that way I can also just
pull into a branch.

Trevor

On Thu, Jun 2, 2011 at 11:25 AM, James Turnbull <james@puppetlabs.com>
wrote:
> Greg Sutcliffe wrote:
>> Hi again,
>>
>> After an hour or three of hacking, I''ve managed to add the
insync?
>> checks to both the build and fragment types. It''s the first
time I''ve
>> ever really messed with types and providers, so it''s very ugly
(and
>> probably has bugs), however I''m happy to share my patches.
Would that be
>> best a) here b) in puppet-dev, or c) direct to Onyx Point?
>>
>
> Awesome stuff! The puppet-dev list is the best place.
>
> Cheers
>
> James
>
> --
> James Turnbull
> Puppet Labs
> 1-503-734-8571
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaughan@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Currently, the way that I''m preventing this is by bookending the
concat_build with a file object.

Basically:

concat_build { ''foo'':  target =>
''/etc/foo'' }

file { ''/etc/foo'': checksum => md5, owner, mode, etc.... }

something { ''bar'': subscribe =>
File[''/etc/foo''] }

It''s not elegant, but, in most cases, I wanted to manage the file
anyway and didn''t relish trying to inherit the file type and do
anything useful.

Trevor

On Thu, Jun 2, 2011 at 9:02 AM, Greg Sutcliffe <greg.sutcliffe@gmail.com>
wrote:
> Hi Trevor,
> I''ll add my thanks to the pile, this is pretty awesome stuff! I
compared the
> --graph results with the old concat moule, and this one - impressive
> difference :)
> As Larry says, the fact that it executes Concat_build on every run is a
> stopper for us, since the dependant services get restarted and we get
report
> emails of any changes.
> Also, I can''t see any way to hook into the filebucket for backups.
I managed
> to hack in a "cp file file.bak" type functionality just before it
actually
> cp''s the newly-built file. However, my Ruby-fu isn''t up
to pulling in the
> filebucket stuff yet - might be worth adding to the todo list.
> Thanks again for the great module though :)
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/OXRxUlV4QnVxU3NK.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaughan@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi
> After an hour or three of hacking, I''ve managed to add the insync?
checks to
> both the build and fragment types. It''s the first time
I''ve ever really
> messed with types and providers, so it''s very ugly (and probably
has bugs),
> however I''m happy to share my patches. Would that be best a) here
b) in
> puppet-dev, or c) direct to Onyx Point?
awesome! I used your changes and besides some corner cases they looked
fine. I pushed my fixes for these corner cases and also some (initial)
general cleanup of the module on github [1].

Thanks!

~pete

[1] https://github.com/duritong/pupmod-concat/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk30nvcACgkQbwltcAfKi3+o2wCgnPLw0lnd6OBnEb91VazYXCGU
wTYAoJLGtze917x5pw5snvf5oHUNOSjl
=YuBj
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 19.5.2011 01:01, Trevor Vaughan wrote:
> All,
> 
> I''d love to get feedback on a module that turns R.I.
Pienaar''s file concatenation idea into a native type.
> 
> The code is located at the Onyx Point Github:
https://github.com/onyxpoint/pupmod-concat
> 
> We found that using this instead of the purely puppet-based implementation
we saw vast improvements in speed, graph complexity, and memory usage.
> 
> We''ll be getting this to the module forge soon, but I thought that
I''d put it out for public perusal in the mean time.
> 
> Improvements welcome!
I noticed that even with
file_delimiter => '''',
All fragments had an trailing newline.

Here is a simple patch which fixes this. With this you can do a single
line out of multiple fragments.

https://github.com/mafalb/pupmod-concat/tree/t_empty_file_delimiter
I also sent a pull request to onyxpoint

-- 
Kind Regards, Markus Falb

On 19.5.2011 01:01, Trevor Vaughan wrote:
> All,
> 
> I''d love to get feedback on a module that turns R.I.
Pienaar''s file concatenation idea into a native type.
> 
> The code is located at the Onyx Point Github:
https://github.com/onyxpoint/pupmod-concat
> 
> We found that using this instead of the purely puppet-based implementation
we saw vast improvements in speed, graph complexity, and memory usage.
> 
> We''ll be getting this to the module forge soon, but I thought that
I''d put it out for public perusal in the mean time.
> 
> Improvements welcome!
I also tried with 0.25.5 and noticed that your module does not work.
All autorequire stuff does not work. It seems like things like

if catalog.resources.find_all { ... }.empty?

is always true. I have no clue why.
2.6.8 (both master and client) is working fine, Thank you for that.

-- 
Kind Regards, Markus Falb

On 12 June 2011 20:21, Markus Falb <markus.falb@fasel.at> wrote:
> I also tried with 0.25.5 and noticed that your module does not work.
> All autorequire stuff does not work. It seems like things like
>
> if catalog.resources.find_all { ... }.empty?
>
> is always true. I have no clue why.

It''s because "catalog.resources()" only returns an array of
strings on
0.25.x

There might be another way to construct the autorequires. Not sure though.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.