Puppet users - Mar 2010 - large files hierarchy - per module or monolithic?

Hi All

I''ve done a small scale puppet (130 servers) implementation per the
book and
had my files hierarchy under each module:
    modules/module1/files
    modules/module2/files
    modules/module3/files
    etc

This worked fine because we only had a small number of people working on the
project and a small number of files to manage.

I am now looking at massively upscaling to have puppet run on thousands of
servers, with a global admin staff of about 50, and manage more than 6000
files (variants and unique instances).

In a previous life, I''ve used cfengine with a large file hierarchy
spread
over multiple modules and had to manage issues where differing modules
wanted to manage the same file. Identifying the issue, and refactoring
modules took a lot of time and effort - and that was with an admin team of 5
all in the same office.

I am thinking that a monolithic module (call it ALL_FILES) that managed all
files and templates will be the way to go, so that it is more visible if a
file is already being managed, given the size and spread of admins:
    modules/ALL_FILES/files

Does anyone have any opinion or experiences they wish to share?

Thanks

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Why not simply using inheritance? or alternativly, have a source search
order?
e.g. find the first files it find.
in moduleA, modubleB, common

if you really need to manage different parts of the file in different
modules, you might want to look for volcane concat module.

cheers,
Ohad

On Tue, Mar 23, 2010 at 8:23 AM, John Warburton
<jwarburton@gmail.com>wrote:
> Hi All
>
> I''ve done a small scale puppet (130 servers) implementation per
the book
> and had my files hierarchy under each module:
>     modules/module1/files
>     modules/module2/files
>     modules/module3/files
>     etc
>
> This worked fine because we only had a small number of people working on
> the project and a small number of files to manage.
>
> I am now looking at massively upscaling to have puppet run on thousands of
> servers, with a global admin staff of about 50, and manage more than 6000
> files (variants and unique instances).
>
> In a previous life, I''ve used cfengine with a large file hierarchy
spread
> over multiple modules and had to manage issues where differing modules
> wanted to manage the same file. Identifying the issue, and refactoring
> modules took a lot of time and effort - and that was with an admin team of
5
> all in the same office.
>
> I am thinking that a monolithic module (call it ALL_FILES) that managed all
> files and templates will be the way to go, so that it is more visible if a
> file is already being managed, given the size and spread of admins:
>     modules/ALL_FILES/files
>
> Does anyone have any opinion or experiences they wish to share?
>
> Thanks
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

The issue isn''t organising puppet, but dealing with a global team of 50
people who may or may not really understand what they are dealing with

I was thinking that rather than find out too late that someone has tried to
manage the same file in muliple modules, I was wondering if putting all file
sunder the same tree might make it a little easier for less adept puppet
users?

John

On 23 March 2010 13:17, Ohad Levy <ohadlevy@gmail.com> wrote:
> Why not simply using inheritance? or alternativly, have a source search
> order?
> e.g. find the first files it find.
> in moduleA, modubleB, common
>
> if you really need to manage different parts of the file in different
> modules, you might want to look for volcane concat module.
>
> cheers,
> Ohad
>
> On Tue, Mar 23, 2010 at 8:23 AM, John Warburton
<jwarburton@gmail.com>wrote:
>
>> Hi All
>>
>> I''ve done a small scale puppet (130 servers) implementation
per the book
>> and had my files hierarchy under each module:
>>     modules/module1/files
>>     modules/module2/files
>>     modules/module3/files
>>     etc
>>
>> This worked fine because we only had a small number of people working
on
>> the project and a small number of files to manage.
>>
>> I am now looking at massively upscaling to have puppet run on thousands
of
>> servers, with a global admin staff of about 50, and manage more than
6000
>> files (variants and unique instances).
>>
>> In a previous life, I''ve used cfengine with a large file
hierarchy spread
>> over multiple modules and had to manage issues where differing modules
>> wanted to manage the same file. Identifying the issue, and refactoring
>> modules took a lot of time and effort - and that was with an admin team
of 5
>> all in the same office.
>>
>> I am thinking that a monolithic module (call it ALL_FILES) that managed
>> all files and templates will be the way to go, so that it is more
visible if
>> a file is already being managed, given the size and spread of admins:
>>     modules/ALL_FILES/files
>>
>> Does anyone have any opinion or experiences they wish to share?
>>
>> Thanks
>>
>> John
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
John Warburton
Ph: 0417 299 600
Email: jwarburton@gmail.com

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I have such a setup :)

what we ended up using is some sort of a define to the file statement, which
search in different places based on some logic.

at the end of the day, it depends on the usage cases, based on your ACL
(e.g. who is allowed to change what in your manifest etc).

my 2 cents,
Ohad

On Tue, Mar 23, 2010 at 1:50 PM, John Warburton
<jwarburton@gmail.com>wrote:
> The issue isn''t organising puppet, but dealing with a global team
of 50
> people who may or may not really understand what they are dealing with
>
> I was thinking that rather than find out too late that someone has tried to
> manage the same file in muliple modules, I was wondering if putting all
file
> sunder the same tree might make it a little easier for less adept puppet
> users?
>
> John
>
>
> On 23 March 2010 13:17, Ohad Levy <ohadlevy@gmail.com> wrote:
>
>> Why not simply using inheritance? or alternativly, have a source search
>> order?
>> e.g. find the first files it find.
>> in moduleA, modubleB, common
>>
>> if you really need to manage different parts of the file in different
>> modules, you might want to look for volcane concat module.
>>
>> cheers,
>> Ohad
>>
>> On Tue, Mar 23, 2010 at 8:23 AM, John Warburton
<jwarburton@gmail.com>wrote:
>>
>>> Hi All
>>>
>>> I''ve done a small scale puppet (130 servers)
implementation per the book
>>> and had my files hierarchy under each module:
>>>     modules/module1/files
>>>     modules/module2/files
>>>     modules/module3/files
>>>     etc
>>>
>>> This worked fine because we only had a small number of people
working on
>>> the project and a small number of files to manage.
>>>
>>> I am now looking at massively upscaling to have puppet run on
thousands
>>> of servers, with a global admin staff of about 50, and manage more
than 6000
>>> files (variants and unique instances).
>>>
>>> In a previous life, I''ve used cfengine with a large file
hierarchy spread
>>> over multiple modules and had to manage issues where differing
modules
>>> wanted to manage the same file. Identifying the issue, and
refactoring
>>> modules took a lot of time and effort - and that was with an admin
team of 5
>>> all in the same office.
>>>
>>> I am thinking that a monolithic module (call it ALL_FILES) that
managed
>>> all files and templates will be the way to go, so that it is more
visible if
>>> a file is already being managed, given the size and spread of
admins:
>>>     modules/ALL_FILES/files
>>>
>>> Does anyone have any opinion or experiences they wish to share?
>>>
>>> Thanks
>>>
>>> John
>>>
>>> --
>>> You received this message because you are subscribed to the Google
Groups
>>> "Puppet Users" group.
>>> To post to this group, send email to puppet-users@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/puppet-users?hl=en.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
>
>
> --
> John Warburton
> Ph: 0417 299 600
> Email: jwarburton@gmail.com
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Mar 22, 2010 at 10:50 PM, John Warburton <jwarburton@gmail.com>
wrote:
> The issue isn''t organising puppet, but dealing with a global team
of 50
> people who may or may not really understand what they are dealing with
>
> I was thinking that rather than find out too late that someone has tried to
> manage the same file in muliple modules, I was wondering if putting all
file
> sunder the same tree might make it a little easier for less adept puppet
> users?
You shouldn''t find out until too late though? If you have duplicate
resources in your manifests, they should fail to compile.

Do you use VCS $Id$ tags? I find those make it reasonably clear when a
file has been pushed centrally.


>
> John
>
> On 23 March 2010 13:17, Ohad Levy <ohadlevy@gmail.com> wrote:
>>
>> Why not simply using inheritance? or alternativly, have a source search
>> order?
>> e.g. find the first files it find.
>> in moduleA, modubleB, common
>>
>> if you really need to manage different parts of the file in different
>> modules, you might want to look for volcane concat module.
>>
>> cheers,
>> Ohad
>>
>> On Tue, Mar 23, 2010 at 8:23 AM, John Warburton
<jwarburton@gmail.com>
>> wrote:
>>>
>>> Hi All
>>>
>>> I''ve done a small scale puppet (130 servers)
implementation per the book
>>> and had my files hierarchy under each module:
>>>     modules/module1/files
>>>     modules/module2/files
>>>     modules/module3/files
>>>     etc
>>>
>>> This worked fine because we only had a small number of people
working on
>>> the project and a small number of files to manage.
>>>
>>> I am now looking at massively upscaling to have puppet run on
thousands
>>> of servers, with a global admin staff of about 50, and manage more
than 6000
>>> files (variants and unique instances).
>>>
>>> In a previous life, I''ve used cfengine with a large file
hierarchy spread
>>> over multiple modules and had to manage issues where differing
modules
>>> wanted to manage the same file. Identifying the issue, and
refactoring
>>> modules took a lot of time and effort - and that was with an admin
team of 5
>>> all in the same office.
>>>
>>> I am thinking that a monolithic module (call it ALL_FILES) that
managed
>>> all files and templates will be the way to go, so that it is more
visible if
>>> a file is already being managed, given the size and spread of
admins:
>>>     modules/ALL_FILES/files
>>>
>>> Does anyone have any opinion or experiences they wish to share?
>>>
>>> Thanks
>>>
>>> John
>>>
>>> --
>>> You received this message because you are subscribed to the Google
Groups
>>> "Puppet Users" group.
>>> To post to this group, send email to puppet-users@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> puppet-users+unsubscribe@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/puppet-users?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>
>
>
> --
> John Warburton
> Ph: 0417 299 600
> Email: jwarburton@gmail.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Mar 23, 2010 at 2:40 PM, Nigel Kersten <nigelk@google.com> wrote:
>
> Do you use VCS $Id$ tags? I find those make it reasonably clear when a
> file has been pushed centrally.
>
> I find the $id$ tags sometimes to be anonying, as if someone is checking
out the files via https or ssh the file url changes and hence trigger puppet
to replace the file...

cheers,
Ohad

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.