Matthew Macdonald-Wallace
2010-Jul-13 10:23 UTC
[Puppet Users] Deploying Splunk using Puppet
Hi all, We''re looking at deploying Splunk using puppet - can anyone give any guidance on the best way to do this? We need to setup light-forwarders and an indexing service via puppet manifests however it looks like we''re going to have to maintain a custom yum repo - how do other people do it? Kind regards. Matt -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 13 July 2010 11:23, Matthew Macdonald-Wallace <mattmacwall@gmail.com>wrote:> We''re looking at deploying Splunk using puppet - can anyone give any > guidance on the best way to do this? > > We need to setup light-forwarders and an indexing service via puppet > manifests however it looks like we''re going to have to maintain a > custom yum repo - how do other people do it? >I opted for syslog-ng TCP/SSL forwards, instead of Splunk''s own light forwarders, because I wasn''t keen on adding the additional proprietary dependency on each frontend node. Each node configures itself to send the relevant logs back to a central host and Splunk indexes from a pattern of directories therein. I also wrapped the web service up with Nginx for some additional security. You can find my modules here: http://github.com/dcarley/puppet/tree/master/modules/service_splunk/ http://github.com/dcarley/puppet/tree/master/modules/service_syslog/ I suspect that you will almost certainly need to roll your own package if you want to play with the latest and greatest Splunk 4.1. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matthew Macdonald-Wallace
2010-Jul-13 11:05 UTC
Re: [Puppet Users] Deploying Splunk using Puppet
Thanks Dan, I''ll take a look at this and see if it needs adjusting. I''ve just been told that there is an RSS feed for versions that I can subscribe to so I can keep my own Yum Repo updated (we use Centos here... :( ) M. On 13 July 2010 11:48, Dan Carley <dan.carley@gmail.com> wrote:> On 13 July 2010 11:23, Matthew Macdonald-Wallace <mattmacwall@gmail.com> > wrote: >> >> We''re looking at deploying Splunk using puppet - can anyone give any >> guidance on the best way to do this? >> >> We need to setup light-forwarders and an indexing service via puppet >> manifests however it looks like we''re going to have to maintain a >> custom yum repo - how do other people do it? > > I opted for syslog-ng TCP/SSL forwards, instead of Splunk''s own light > forwarders, because I wasn''t keen on adding the > additional proprietary dependency on each frontend node. Each node > configures itself to send the relevant logs back to a central host and > Splunk indexes from a pattern of directories therein. I also wrapped the web > service up with Nginx for some additional security. You can find my modules > here: > http://github.com/dcarley/puppet/tree/master/modules/service_splunk/ > http://github.com/dcarley/puppet/tree/master/modules/service_syslog/ > I suspect that you will almost certainly need to roll your own package if > you want to play with the latest and greatest Splunk 4.1. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 13 July 2010 12:05, Matthew Macdonald-Wallace <mattmacwall@gmail.com> wrote:> I''ll take a look at this and see if it needs adjusting.Looks like his module is pretty specific to Gentoo Linux. Good effort though and should be easy to port over. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
We deploy splunk from our own repo and then use a combination of the below
stuff to get it working: (I chopped a few custom environment bits out but
you''ll get the idea from the stuff below) Obviously there''s a
bunch of
files and a few templates but they are almost all environment specific/the
stock files tweaked for hostnames. Hopefully this will be enough to get you
running!
===============init.pp:
===============
class splunk4 {
file { "/etc/init.d/splunk":
ensure => "present",
source => "puppet:///modules/splunk4/init-splunk",
}
file { "/opt/splunk/etc/apps/unix/local/":
ensure => "directory",
require => Package["splunk"],
}
file { "/opt/splunk/etc/apps/unix/local/app.conf":
source => "puppet:///modules/splunk4/enableunix.conf",
ensure => "present",
require => [ Package["splunk"],
File["/opt/splunk/etc/apps/unix/local/"] ],
notify => Service["splunk"],
}
file { "/opt/splunk/etc/apps/unix/local/inputs.conf":
content => template("splunk4/unix.conf.erb"),
ensure => "present",
require => [ Package["splunk"],
File["/opt/splunk/etc/apps/unix/local/"] ],
notify => Service["splunk"],
}
package {
"splunk":
ensure => "present",
}
service {
"splunk":
ensure => "running",
hasrestart => "true",
require => [ Service["auditd"],
File["/opt/splunk/etc/apps/unix/local/app.conf"],
File["/etc/init.d/splunk"], ];
"auditd":
ensure => "running",
enable => "true",
hasrestart => "true",
hasstatus => "true";
}
}
define splunk4::inputs($value) {
case $ipaddress {
/^x.x.x/: {
concat::fragment{"splunk4_${name}":
target =>
"/opt/splunk/etc/system/local/inputs.conf",
content => "[monitor://$value]\nindex = dev\ndisabled
false\n\n",
}
}
default: {
concat::fragment{"splunk4_${name}":
target =>
"/opt/splunk/etc/system/local/inputs.conf",
content => "[monitor://$value]\ndisabled =
false\n\n",
}
}
}
}
=============client.pp:
=============
class splunk4::client inherits splunk4 {
include concat::setup
$splunkpassword = "changeme"
file { "/opt/splunk/etc/apps/SplunkLightForwarder/local/":
ensure => "directory",
require => Package["splunk"],
}
file { "/opt/splunk/etc/apps/SplunkLightForwarder/local/app.conf":
source =>
"puppet:///modules/splunk4/client/enablelightforwarder.conf",
ensure => "present",
recurse => "true",
notify => Service["splunk"],
require => [ Package["splunk"],
File["/opt/splunk/etc/apps/SplunkLightForwarder/local/"] ],
}
file {
"/opt/splunk/etc/apps/SplunkLightForwarder/local/inputs.conf":
source =>
"puppet:///modules/splunk4/client/lightforwarder-inputs.conf",
ensure => "present",
recurse => "true",
notify => Service["splunk"],
require => [ Package["splunk"],
File["/opt/splunk/etc/apps/SplunkLightForwarder/local/"] ],
}
# Override the base class splunk4 service to make sure the forwarding is
# enabled and that the list of things splunk indexes are in place.
Service["splunk"] {
require +> [
File["/opt/splunk/etc/apps/SplunkLightForwarder/local/app.conf"],
File["/opt/splunk/etc/apps/SplunkLightForwarder/local/inputs.conf"] ],
}
# First we setup a concat{} resource on a file, and then we immediately
# place a fragment in place that consists of some static things we wish
# to filter out of splunk. The order => 10 ensures this is placed at
# the top of the file.
concat { "/opt/splunk/etc/system/local/inputs.conf":
notify => Service["splunk"],
require => Package["splunk"],
}
concat::fragment{"filters":
target => "/opt/splunk/etc/system/local/inputs.conf",
order => "10",
source =>
"puppet:///modules/splunk4/client/filters-input.conf"
}
# Each call to inputs{} adds an additional fragment into the inputs.conf
# and is defined in init.pp.
splunk4::inputs{"messages":
value => "/var/log/messages",
}
splunk4::inputs{"secure":
value => "/var/log/secure",
}
splunk4::inputs{"mail":
value => "/var/log/mail",
}
splunk4::inputs{"maillog":
value => "/var/log/maillog",
}
file { "/opt/splunk/etc/system/local/outputs.conf":
ensure => "present",
source => "puppet:///modules/splunk4/client/outputs.conf",
require => Package["splunk"],
notify => Service["splunk"];
}
}
=================server.pp
=================
class splunk4::server {
include splunk4
file {
"/data/splunk":
ensure => directory,
before => Package["splunk"];
"/opt/splunk/etc/system/local/authentication.conf":
source =>
"puppet:///modules/splunk4/server/authentication.conf",
ensure => "present",
notify => Service["splunk"],
require => Package["splunk"];
"/opt/splunk/etc/auth/splunk.secret":
source =>
"puppet:///modules/splunk4/server/splunk.secret",
ensure => "present",
notify => Service["splunk"],
require => Package["splunk"];
"/opt/splunk/etc/system/local/indexes.conf":
source => "puppet:///modules/splunk4/server/indexes.conf",
ensure => "present",
notify => Service["splunk"],
require => Package["splunk"];
"/opt/splunk/etc/splunk-launch.conf":
source =>
"puppet:///modules/splunk4/server/splunk-launch.conf",
ensure => "present",
notify => Service["splunk"],
require => Package["splunk"];
}
}
On Tue, Jul 13, 2010 at 6:23 AM, Matthew Macdonald-Wallace <
mattmacwall@gmail.com> wrote:
> Hi all,
>
> We''re looking at deploying Splunk using puppet - can anyone give
any
> guidance on the best way to do this?
>
> We need to setup light-forwarders and an indexing service via puppet
> manifests however it looks like we''re going to have to maintain a
> custom yum repo - how do other people do it?
>
> Kind regards.
>
> Matt
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.