Puppet users - Jul 2013 - Deploying system configurations with another tool?

Hi,

Let''s imagine there is a daemon named foobard, which has it''s
configuration snippets in /etc/foobar.d/*.conf

Applications are deployed via some tool other than puppet, for example
Jenkins or Capistrano, and developers want to be able to
also push system configs for specific deamon(s) via their deployment system.


Now, this poses a problem for me, because I''m used to manage
configuration files via puppet. How do you guys solve this kind of issues?

Also, CM aside, there is a standing security issue. Deployment is run
via unprivileged account, which now has to be able somehow inject
potentially malicious configuration files into /etc.


Any input is appreciated.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 05.07.2013 13:37, Jakov Sosic wrote:
> Hi,
>
> Let''s imagine there is a daemon named foobard, which has
it''s
> configuration snippets in /etc/foobar.d/*.conf
>
> Applications are deployed via some tool other than puppet, for example
> Jenkins or Capistrano, and developers want to be able to
> also push system configs for specific deamon(s) via their deployment
system.
>
> Now, this poses a problem for me, because I''m used to manage
> configuration files via puppet. How do you guys solve this kind of issues?
>
> Any input is appreciated.
One of the big arguments for puppet is the unifying aspect of devs and 
ops to use the same tool/language/process, which improves cooperation, 
agility and quality of the work. This indicates that your application 
deployment should be integrated into your puppet manifests and those 
manifests should be integrated into the application development/release 
cycle.

Another big point in puppet''s favor is that it doesn''t want to
be the
be-all-end-all. If there''s a tool that is better suited to a task (the 
prime example being package managers) then *please* use that. This 
indicates that if capistrano is a good match for your organization''s 
application deployment (especially in the area of orchestration across 
nodes and rollback it leaves puppet in the dust), then you should 
leverage those capabilities.

You write "this poses a problem for me, because I''m used to manage
configuration files via puppet." While that may just be lost in 
translation, it does sound like your problem is of a much more personal 
and a less technical level. In that case you really need to get rid of 
the us-against-them attitude and find solutions that enable the 
organization as a whole to repeatably deliver more value faster while 
reducing the associated risks.



> Also, CM aside, there is a standing security issue. Deployment is run
> via unprivileged account, which now has to be able somehow inject
> potentially malicious configuration files into /etc.
This is a non-issue as the deployment process will always be able to 
push the changes it needs into the system. So a subversion (no pun 
intended) of the deployment process will always be a death knell, 
independent of the used tool. So either the devs have the need and right 
to modify those configuration files, or they don''t. If they have the 
need and right, then they also share the responsibility for the system.


Regards, David

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 07/05/2013 02:46 PM, David Schmitt wrote:
> One of the big arguments for puppet is the unifying aspect of devs and
> ops to use the same tool/language/process, which improves cooperation,
> agility and quality of the work. This indicates that your application
> deployment should be integrated into your puppet manifests and those
> manifests should be integrated into the application development/release
> cycle.
But how are they integrated in your environment? What would you do in my
case?

> Another big point in puppet''s favor is that it doesn''t
want to be the
> be-all-end-all. If there''s a tool that is better suited to a task
(the
> prime example being package managers) then *please* use that. This
> indicates that if capistrano is a good match for your
organization''s
> application deployment (especially in the area of orchestration across
> nodes and rollback it leaves puppet in the dust), then you should
> leverage those capabilities.
And that''s exactly why are we using another tool for the job.

> You write "this poses a problem for me, because I''m used to
manage
> configuration files via puppet." While that may just be lost in
> translation, it does sound like your problem is of a much more personal
> and a less technical level.
> In that case you really need to get rid of
> the us-against-them attitude and find solutions that enable the
> organization as a whole to repeatably deliver more value faster while
> reducing the associated risks.
''you need to find solutions'' is a thing I already now. If I
didn''t know
that, I wouldn''t have asked the question in the first place. On the
other hand, one thing I didn''t ask about was online psychotherapy...

Did I offend you some how with my OP? If I did, I''m really sorry, that
was not my intention.

> This is a non-issue as the deployment process will always be able to
> push the changes it needs into the system. So a subversion (no pun
> intended) of the deployment process will always be a death knell,
> independent of the used tool. So either the devs have the need and right
> to modify those configuration files, or they don''t. If they have
the
> need and right, then they also share the responsibility for the system.
Yeah, but things have to stay pretty tight. For example: if you enable
some user account to push files into dot-d directory, not-if-but-when
that account gets broken into, you have a possibility of privilege
escalation.

So, allowing the write privilege for that directory obviously is not a
good choice.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 05.07.2013 15:13, Jakov Sosic wrote:
> On 07/05/2013 02:46 PM, David Schmitt wrote:
>> One of the big arguments for puppet is the unifying aspect of devs and
>> ops to use the same tool/language/process, which improves cooperation,
>> agility and quality of the work. This indicates that your application
>> deployment should be integrated into your puppet manifests and those
>> manifests should be integrated into the application development/release
>> cycle.
>
> But how are they integrated in your environment? What would you do in my
> case?
In the environments I support everything is deployed through puppet. 
This leads to a big unification in dev/test environments. Through 
vagrant the complete stack can be tested locally before pushing to code 
review. From there the code travels through gerrit and jenkins until it 
is deployed to the puppetmaster.
> Did I offend you some how with my OP? If I did, I''m really sorry,
that
> was not my intention.
At no point I was offended by your words. I noticed a weakness in your 
explanation and frankly (even ruthlessly) addressed it. Please accept my 
apology for my rudeness.
>> This is a non-issue as the deployment process will always be able to
>> push the changes it needs into the system. So a subversion (no pun
>> intended) of the deployment process will always be a death knell,
>> independent of the used tool. So either the devs have the need and
right
>> to modify those configuration files, or they don''t. If they
have the
>> need and right, then they also share the responsibility for the system.
>
> Yeah, but things have to stay pretty tight. For example: if you enable
> some user account to push files into dot-d directory, not-if-but-when
> that account gets broken into, you have a possibility of privilege
> escalation.
What is the risk of having an attacker who breaks into the deployment 
user (which should only do deployment and nothing else), but is not able 
to achieve root directly? A kernel exploit would yield root. Exploiting 
a running daemon over the network is out, since the deployment user 
doesn''t run daemons (use locked down sudo to trigger service restarts).
One possibility would be a symlink race against one of the deployment 
scripts to subvert the deployed code. Without knowing your actual threat 
profile, I''d say that that would be pretty unusual for an attacker to 
custom craft such a local cross-user exploit against a locally developed 
script set.
> So, allowing the write privilege for that directory obviously is not a
> good choice.
It''s a very fine line to walk. Perhaps an API (even a little script
that
does syntax checks and/or auditing) might suffice.


Regards, David

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

----- Original Message -----
| On 07/05/2013 02:46 PM, David Schmitt wrote:
| 
| > One of the big arguments for puppet is the unifying aspect of devs
| > and
| > ops to use the same tool/language/process, which improves
| > cooperation,
| > agility and quality of the work. This indicates that your
| > application
| > deployment should be integrated into your puppet manifests and
| > those
| > manifests should be integrated into the application
| > development/release
| > cycle.
| 
| But how are they integrated in your environment? What would you do in
| my
| case?

In my environment we choose the right tool for the job, point blank.  We put as
much into puppet as we can, but we understand that it isn''t the
be-all-end-all tool.  Understanding where the lines of delineation are is an
important aspect to configuration management.  It does require some extra work
to ensure that we''re not stepping over each others toes all the time,
but it also helps each group understand what each other is trying to achieve.

Where we can make use of puppet instead of some other tool we decide during a
meeting so everyone knows who''s responsible for what.

| > Another big point in puppet''s favor is that it doesn''t
want to be
| > the
| > be-all-end-all. If there''s a tool that is better suited to a
task
| > (the
| > prime example being package managers) then *please* use that. This
| > indicates that if capistrano is a good match for your
| > organization''s
| > application deployment (especially in the area of orchestration
| > across
| > nodes and rollback it leaves puppet in the dust), then you should
| > leverage those capabilities.
| 
| And that''s exactly why are we using another tool for the job.

Assuming there have been the proper discussions made this is not an issue
whatsoever.  In fact it''s good.
 

<snip>

| > This is a non-issue as the deployment process will always be able
| > to
| > push the changes it needs into the system. So a subversion (no pun
| > intended) of the deployment process will always be a death knell,
| > independent of the used tool. So either the devs have the need and
| > right
| > to modify those configuration files, or they don''t. If they have
| > the
| > need and right, then they also share the responsibility for the
| > system.
| 
| Yeah, but things have to stay pretty tight. For example: if you
| enable
| some user account to push files into dot-d directory, not-if-but-when
| that account gets broken into, you have a possibility of privilege
| escalation.
| 
| So, allowing the write privilege for that directory obviously is not
| a
| good choice.


Whether that push has been done via puppet or some other tool is irrelevant.  If
the account is compromised it''s been compromised.  You need to ask
yourself if using puppet vs some other tool changes this fact in *ANY* way.  If
something is wrong, it was wrong from the time of deployment and has therefore
*ALWAYS* been wrong.  I don''t know of any tool that can stop this. 
Mitigate maybe, but stop it entirely, not a chance.

Just my 2c.

-- 
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier@sfu.ca
Website : http://www.sfu.ca/itservices

“A successful person is one who can lay a solid foundation from the bricks
others have thrown at them.” -David Brinkley via Luke Shaw

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 07/05/2013 06:19 PM, David Schmitt wrote:
> In the environments I support everything is deployed through puppet.
> This leads to a big unification in dev/test environments. Through
> vagrant the complete stack can be tested locally before pushing to code
> review. From there the code travels through gerrit and jenkins until it
> is deployed to the puppetmaster.
Nice one, but currently not achievable in my case :( Yeah, social
problems are always thougher to surpass then the technological ones.

> At no point I was offended by your words. I noticed a weakness in your
> explanation and frankly (even ruthlessly) addressed it. Please accept my
> apology for my rudeness.
Explanation was definitely weak, but situation is really far from
''we''
vs ''them''... Both teams want the best possible solution.

> What is the risk of having an attacker who breaks into the deployment
> user (which should only do deployment and nothing else), but is not able
> to achieve root directly?
Because one of the daemons that is supposed to be controlled this was is
supervisor. And allowing unprivileged user to put stuff with no limits
at all into dot-d is really only a command away from privilege
escalation to root...

> It''s a very fine line to walk. Perhaps an API (even a little
script that
> does syntax checks and/or auditing) might suffice.
One thing that did cross my mind is to allow deployment process to push
specific files to specific locations on puppet master. That way, after
the files are injected into master, all the deployment tool has to do
afterwards is to initiate agent run on each node.

What do you think about that idea?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 2013-07-05 20:49, Jakov Sosic wrote:
>> It''s a very fine line to walk. Perhaps an API (even a little
script that
>> does syntax checks and/or auditing) might suffice.
>
> One thing that did cross my mind is to allow deployment process to push
> specific files to specific locations on puppet master. That way, after
> the files are injected into master, all the deployment tool has to do
> afterwards is to initiate agent run on each node.
>
> What do you think about that idea?
If the deployment user can push to the puppetmaster it only increases 
the attack surface.

Another idea: the configurations get deployed to a central (git) repo 
and puppet pulls the updates from there. That way the deployment process 
never touches the puppet master, the dot.d is still under developer 
control, but not reachable from the local deployment UID.


If that''s the level of paranoia you need to apply, don''t
forget to audit
other attack vectors like init scripts, setuid binaries and things that 
can be affected by subverting the application itself.


Regards, David

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.