Victor Hooi
2011-Jul-07 03:34 UTC
[Puppet Users] Checking for free disk space on mountpoint?
Hi, I''m currently looking at migrating from some home-rolled Python scripts to using Puppet =). One of the things we need to check for is that there is sufficient free disk space on a mountpoint (for logging etc.) this is currently quite easy in Python using statvfs(). Are there any resources in Puppet that might be able to do this? I checked the mount type (http://docs.puppetlabs.com/references/stable/type.html#mount), but it didn''t seem to have anything to do this. Cheers, Victor -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/lE1ahe-Z5KoJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-07 15:41 UTC
Re: [Puppet Users] Checking for free disk space on mountpoint?
Let me understand this correctly - you want to use the Puppet RAL layer to interrogate data such as free space on a mount? At the moment you can''t use RAL data elsewhere in Puppet content ... but you can use the RAL layer in your own code and in things like MCollective using the ralsh plugin ... Perhaps you''re after facts? http://docs.puppetlabs.com/guides/custom_facts.html Excuse me if my understanding of your needs is completely out of whack :-). Maybe you want to describe what you want to do with this data once you have gathered it. ken. On Thu, Jul 7, 2011 at 4:34 AM, Victor Hooi <victorhooi@gmail.com> wrote:> Hi, > I''m currently looking at migrating from some home-rolled Python scripts to > using Puppet =). > One of the things we need to check for is that there is sufficient free disk > space on a mountpoint (for logging etc.) this is currently quite easy in > Python using statvfs(). > Are there any resources in Puppet that might be able to do this? I checked > the mount type > (http://docs.puppetlabs.com/references/stable/type.html#mount), but it > didn''t seem to have anything to do this. > Cheers, > Victor > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/lE1ahe-Z5KoJ. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
HI Victor, You can write a simple custom Fact (leveraging the Facter library that Puppet uses) that will run something like ''df'' and parse the output. Your puppet agents will report their disk usage, along with all the other Facts, every time they make a Puppet run. -Kent On Jul 6, 8:34 pm, Victor Hooi <victorh...@gmail.com> wrote:> Hi, > > I''m currently looking at migrating from some home-rolled Python scripts to > using Puppet =). > > One of the things we need to check for is that there is sufficient free disk > space on a mountpoint (for logging etc.) this is currently quite easy in > Python using statvfs(). > > Are there any resources in Puppet that might be able to do this? I checked > the mount type > (http://docs.puppetlabs.com/references/stable/type.html#mount), but it > didn''t seem to have anything to do this. > > Cheers, > Victor-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Doug Warner
2011-Jul-07 18:38 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
Here''s my hack-y fact that adds facts for mounted partitions on linux w/ disk used, available, and total as well as a fact for all available mounts. -Doug --- begin --- mounts = [] mntpoints=`mount -t ext2,ext3,ext4,reiserfs,xfs` mntpoints.split(/\n/).each do |m| mount = m.split(/ /)[2] mounts << mount end Facter.add("mounts") do confine :kernel => :linux setcode do mounts.join('','') end end mounts.each do |mount| output = %x{df #{mount}} output.each do |str| dsk_size = nil dsk_used = nil dsk_avail = nil if str =~ /^\S+\s+([0-9]+)\s+([0-9]+)\s+([0-9]+)\s+/ dsk_size = $1 dsk_used = $2 dsk_avail = $3 Facter.add("mount_#{mount}_size") do setcode do dsk_size end end Facter.add("mount_#{mount}_used") do setcode do dsk_used end end Facter.add("mount_#{mount}_avail") do setcode do dsk_avail end end end end end --- end --- On 07/07/2011 01:53 PM, Kent wrote:> HI Victor, > > You can write a simple custom Fact (leveraging the Facter library that > Puppet uses) that will run something like ''df'' and parse the output. > Your puppet agents will report their disk usage, along with all the > other Facts, every time they make a Puppet run. > > -Kent > > On Jul 6, 8:34 pm, Victor Hooi <victorh...@gmail.com> wrote: >> Hi, >> >> I''m currently looking at migrating from some home-rolled Python scripts to >> using Puppet =). >> >> One of the things we need to check for is that there is sufficient free disk >> space on a mountpoint (for logging etc.) this is currently quite easy in >> Python using statvfs(). >> >> Are there any resources in Puppet that might be able to do this? I checked >> the mount type >> (http://docs.puppetlabs.com/references/stable/type.html#mount), but it >> didn''t seem to have anything to do this. >> >> Cheers, >> Victor >
Victor Hooi
2011-Jul-08 08:19 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
heya, Excuse me if my understanding of your needs is completely out of whack :-). Maybe you want to describe what you want to do with this data once you have gathered it. The reason we are doign this is to make sure that we have sufficient free disk space on various mountpoints to actually run the application on the server. So this is something I''d like to enforce (or "ensure") as part of the configuration. Which would best point to integrate this Facter-based check into a Puppet configuration for a node? @Doug: Thanks heaps for your Facter script =). Hmm, the only thing is, ideally I''d like it to be cross-platform (Linux, Solaris and Windows) - it seems there isn''t really a abstracted way of checking this in-built into Facter/Puppet, which is a shame. We also need to check the file permissions on certain files, to make sure the application will run correctly. I searched and found this: http://projects.puppetlabs.com/projects/1/wiki/File_Permission_Check_Patterns which seems to do the job. I would have thought these two things (checking free disk space on various network mountpoints, and checking file/directory permissions) would be two fairly common tasks when configuring out boxes. How do people normally tackle this with Puppet? Cheers, Victor -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/bd3NjsTM6XwJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-08 11:16 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
> The reason we are doign this is to make sure that we have sufficient free > disk space on various mountpoints to actually run the application on the > server.Oh - and your monitoring doesn''t do this already?> So this is something I''d like to enforce (or "ensure") as part of the > configuration. Which would best point to integrate this Facter-based check > into a Puppet configuration for a node?I guess so ... facter can be used as variables to decide if a configuration can be applied. You could use conditionals to decide if and service has ensure => running/stopped based on something from facter using conditional. But this is during Puppet runtime only ...> @Doug: Thanks heaps for your Facter script =). Hmm, the only thing is, > ideally I''d like it to be cross-platform (Linux, Solaris and Windows) - it > seems there isn''t really a abstracted way of checking this in-built into > Facter/Puppet, which is a shame.Not necessarily. You can make decisions in Facter based on other facts. ''operatingsystem'' can be one of those facts. For example: case Facter.value(:operatingsystem) when "Solaris" # ... Solaris related lookup code when "centos", "redhat" # ... centos/redhat related df lookup code else # ... some default behaviour end> We also need to check the file permissions on certain files, to make sure > the application will run correctly. I searched and found this: > http://projects.puppetlabs.com/projects/1/wiki/File_Permission_Check_Patterns > > which seems to do the job. > I would have thought these two things (checking free disk space on various > network mountpoints, and checking file/directory permissions) would be two > fairly common tasks when configuring out boxes. How do people normally > tackle this with Puppet?I don''t personally for application stop/start scripts. If I wanted to I would put it in the system V startup script or whatever mechanism is kicking around. That way it works for Puppet and human users who just jump on a box and want to use the systems standard stop/start mechanism. Problem is - you do it all in Puppet ... only Puppet can stop and start it. You could write start.pp and a stop.pp script if you wanted and write all the scripting logic in Puppet I guess. Not normally done though. I can''t say I have tested mount space before application start. I usually rely on monitoring to tell me there is a problem there. Is there a specific reason you need this, ie. avoid some odd corruption during disk fulls or something? I would imagine the chances of corruption during a "disk full" are much higher _after_ the application has started and been running for a while though :-). Hence why I guess I rely on monitoring more often ... monitoring will alert me to a disk full even when I''m not trying to start the application. ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Victor Hooi
2011-Jul-12 02:37 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
Ken, Currently, the existing processes we have are quite ad-hoc, and developers are often responsible for deployment, which is done by hand. I''m hoping Puppet will automate the process, provide better standardisation, and segregate development away from deployment/production. All the things I''m attempting to check for (e.g. disk space, file/directory permissions) are things that somebody currently does manually from the command-line (e.g. using "df", "touch x" in a directory). I need to to automate it all into Puppet to show that it''s a viable option and can replace the manual processes, and then in phase two look at getting things like Nagios (for disk monitoring, or whatever else we have used elsehwere in the organisation for monitoring) up and running in tandem as well. The disk space needs to be monitored because parts of the application don''t handle full disks well and will crash nastily. This is only an issue intra-day. Basically, we spin up the application in the morning, and we know with good certainty the maximum space they''ll consume, then the application is killed off at the end of the day. As long as we have more the <x> bytes available when we install things, we know we''ll be fine. The permissions is something I''d like to ensure when we setup a box - the directory/files are setup for us by sys-admins, and we just want to run some sanity checks to make sure they''ve set it up properly for us. Currently there''s a lot of to-ing and fro-ing to check everythign is setup by the sys-admins correctly, I''d like to simply run-up Puppet, and have it ensure for these things. So the best way is to write custom facts for all the things we''re sanity checking, then just do a ensure-> on them, right? I couldn''t find anything on http://forge.puppetlabs.com/ about permissions or disk space...lol, guess they''re too small/trivial to be there. There aren''t other sources of these things, are there? Cheers, Victor -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/ikRQqX6oZmkJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
vagn scott
2011-Jul-12 03:37 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
On 07/11/2011 10:37 PM, Victor Hooi wrote:> As long as we have more the <x> bytes available when we install > things, we know we''ll be fine.Here is a proof of concept. With a little tweaking you should be able to test for an absolute amount of free space on any mounted volume. Hope that helps, --vagn #! /usr/bin/puppet apply Exec { path => "/u0/home/vagn/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" } $script = inline_template("#! /bin/bash d=\$1 p=\$2 t=\$( df -h \$d | tail -n +2 | sed ''s/%//g'' | while read a b c d e f ; do echo \$e ; done ) if test \$t -gt \$p then echo \$t greater than \$p PASSED exit 0 else echo \$t not greater than \$p FAILED exit 1 fi )") define diskspace($dev, $percent) { exec { "diskspace $title $dev $percent": command => "/usr/local/bin/diskspace-script ${dev} ${percent}", logoutput => true, require => File [ "/usr/local/bin/diskspace-script" ], } } node default { file { "/usr/local/bin/diskspace-script": content => $script, mode => 775, } diskspace { "check 1": dev => "/dev/sda1", percent => "30", } diskspace { "check 2": dev => "/dev/sda1", percent => "40", } } -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jul-12 05:05 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
On Jul 11, 2011, at 7:37 PM, Victor Hooi wrote:> Ken, > > Currently, the existing processes we have are quite ad-hoc, and developers are often responsible for deployment, which is done by hand. I''m hoping Puppet will automate the process, provide better standardisation, and segregate development away from deployment/production. > > All the things I''m attempting to check for (e.g. disk space, file/directory permissions) are things that somebody currently does manually from the command-line (e.g. using "df", "touch x" in a directory). > > I need to to automate it all into Puppet to show that it''s a viable option and can replace the manual processes, and then in phase two look at getting things like Nagios (for disk monitoring, or whatever else we have used elsehwere in the organisation for monitoring) up and running in tandem as well. > > The disk space needs to be monitored because parts of the application don''t handle full disks well and will crash nastily. This is only an issue intra-day. Basically, we spin up the application in the morning, and we know with good certainty the maximum space they''ll consume, then the application is killed off at the end of the day. As long as we have more the <x> bytes available when we install things, we know we''ll be fine. > > The permissions is something I''d like to ensure when we setup a box - the directory/files are setup for us by sys-admins, and we just want to run some sanity checks to make sure they''ve set it up properly for us. Currently there''s a lot of to-ing and fro-ing to check everythign is setup by the sys-admins correctly, I''d like to simply run-up Puppet, and have it ensure for these things. > > So the best way is to write custom facts for all the things we''re sanity checking, then just do a ensure-> on them, right?So, here''s what I see you doing. Tell me if I''m confused. 1) You want to do an "ensure" on permissions to make sure "stuff" is set right. In general this isn''t done with facts. You just declare it, and puppet makes it so. 2) You have resources you don''t want to run if there''s not enough space available. This can be done by creating a "free_disk_space" fact and using that in an if statement that wraps resources. Keep in mind that, this being unix, there''s no "drives" so if you will probably be checking the amount of space on root if you don''t think about it. This might be what you want. If not you can create facts like free_space_var or whatever. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
vagn scott
2011-Jul-12 06:10 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
On 07/12/2011 01:05 AM, Patrick wrote:> Keep in mind that, this being unix, there''s no "drives" so if you will probably be checking the amount of space on root if you don''t think about it.There most certainly are drives in unix. Not drive letters, though. The drives are normally denoted either by their device special file, such as /dev/md127, or by their mount points, such as /u0. Here is a list of everything mounted on one particular system. vagn@nika:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 28G 9.3G 17G 36% / tmpfs 7.9G 0 7.9G 0% /lib/init/rw udev 7.9G 324K 7.9G 1% /dev tmpfs 7.9G 0 7.9G 0% /dev/shm /dev/md127 2.7T 846G 1.8T 33% /u0 /dev/mapper/udisks-luks-uuid-3bb2bdd5-1b84-49b4-abe8-4a107f174702-uid1000 15G 7.9G 6.2G 56% /u0/home/vagn/s5 /dev/mapper/udisks-luks-uuid-d5ff5dbd-053b-48a7-b3ee-4f71a2e5c2fb-uid1000 2.7T 855G 1.8T 33% /bu02 /dev/sr0 6.8G 6.8G 0 100% /media/cdrom0 And here is the listing of one drive by device: vagn@nika:~$ df -h /dev/md127 Filesystem Size Used Avail Use% Mounted on /dev/md127 2.7T 846G 1.8T 33% /u0 vagn@nika:~$ And the same by mount point: vagn@nika:~$ df -h /u0 Filesystem Size Used Avail Use% Mounted on /dev/md127 2.7T 846G 1.8T 33% /u0 See man pages for stat, statvfs or statfs for details. Also, the stat command is interesting: vagn@nika:~$ stat -f -c "freeblocks on /u0 = %a" /u0 freeblocks on /u0 = 462905880 vagn@nika:~$ bc <<. 462905880 * 4 . 1851623520 vagn@nika:~$ df /u0 Filesystem 1K-blocks Used Available Use% Mounted on /dev/md127 2884294872 886157536 1851623520 33% /u0 -- vagn -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jul-12 06:12 UTC
Re: [Puppet Users] Re: Checking for free disk space on mountpoint?
On Jul 11, 2011, at 11:10 PM, vagn scott wrote:> On 07/12/2011 01:05 AM, Patrick wrote: >> Keep in mind that, this being unix, there''s no "drives" so if you will probably be checking the amount of space on root if you don''t think about it.> There most certainly are drives in unix. Not drive letters, though.Ah. I misspoke. I meant that the path to where you put a file on the filesystem doesn''t explicitly mention a drive name. For instance, you can''t tell looking at the path if / and /var are on the same drive, where as you (usually) can tell with Windows. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.