Greetings!
I am having a problem with puppet mounting a device and am hoping someone
can help. Here is the short version, if you have questions or need more
detail, please feel free to ask.
I have a Panasas storage device on my network on which my home directory
resides. The Panasas device mounts the filesystem via a kernel module
(which they call DirectFlow). Thus, when I run as root ` mount /home` a
kernel module is loaded and then the filesystem is loaded. I have certain
restrictions in place on my servers which I have to keep in line (Puppet is
AMAZING for this!) and the mounting of /home is one of those restrictions.
I have in my puppet manifest this:
mount { "/home" :
ensure => mounted,
atboot => true,
device => "panfs://192.168.1.20/home",
fstype => "panfs",
options => "defaults,nodev",
remounts => true,
pass => 2,
dump => 1,
}
If I run, as root, `puppet agent --test` then the /home filesystem is
mounted and everything is wonderful. However, if I let the puppet agent
daemon try to mount /home I get errors in the log files without the mount
ever happening.
Apr 2 13:01:08 testnode puppet-agent[29955]:
(/Stage[main]/mount::Homefilesystem/Mount[/home]/ensure) ensure changed
''unmounted'' to ''mounted''
Apr 2 13:01:08 testnode puppet-agent[29955]:
(/Stage[main]/mount::Homefilesystem/Mount[/home]) Could not evaluate:
Execution of ''/bin/mount -o defaults,nodev /home'' returned 1:
mount.panfs
error: cannot init pan_sock_ping 0x239d (pan_sock: protected socket,
permission denied)
Neither Panasas representatives I talked to seemed to have any idea what
Puppet was before I spoke to them. My coworkers, the Panasas reps, and I
brainstormed a few ideas but only three seemed to "work":
* Have the puppet daemon run as root instead of the puppet user (which is
an obvious issue)
* Use auto-mount (which "works" but is causing some oddities in a few
of my
jobs which I am fairly sure is due to the latency of the mount)
* Have Puppet call a script with the setuid bit configured which can mount
/home (which doesn''t 100% address my needs of puppet being able to
remount
if one of those parameters is wrong/missing/changed/whatever without that
script getting complicated).
Before I commit towards one option, I thought I would ask the other Puppet
masters out there for ideas. Given the popularity of Puppet in datacenters
as well as the popularity of SAN devices in datacenters I figure someone
out there has probably solved this problem. I am hoping that their solution
is better then the ones we have come up with. :-D
Can anyone help me out with this?
Thank you in advance!!
Stack
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Steven Nemetz
2013-Apr-09 14:35 UTC
RE: [Puppet Users] Puppet mount fails due to kernel module
This is an OS privilege issue. So puppet has to working within what can be done
on the OS.
By default mount requires root privileges to make changes. This is why it will
work as root but not as puppet.
So you either run as root or set it up so a non-root user can mount
Some OS''s have options to allow mount to work for non-root users. Check
your mount man page and test.
Load a different app to handle user mounts. There are other apps available for
this type of thing. Fuse and apps based on fuse are the most common, but not the
only ones available.
Hope that helps a little,
Steven
Date: Mon, 8 Apr 2013 21:02:26 -0700
From: i.am.stack@gmail.com
To: puppet-users@googlegroups.com
Subject: [Puppet Users] Puppet mount fails due to kernel module
Greetings!
I am having a problem with puppet mounting a device and am hoping someone can
help. Here is the short version, if you have questions or need more detail,
please feel free to ask.
I have a Panasas storage device on my network on which my home directory
resides. The Panasas device mounts the filesystem via a kernel module (which
they call DirectFlow). Thus, when I run as root ` mount /home` a kernel module
is loaded and then the filesystem is loaded. I have certain restrictions in
place on my servers which I have to keep in line (Puppet is AMAZING for this!)
and the mounting of /home is one of those restrictions.
I have in my puppet manifest this:mount { "/home" :
ensure => mounted,
atboot => true,
device => "panfs://192.168.1.20/home",
fstype => "panfs",
options => "defaults,nodev",
remounts => true,
pass => 2,
dump => 1,
}
If I run, as root, `puppet agent --test` then the /home filesystem is mounted
and everything is wonderful. However, if I let the puppet agent daemon try to
mount /home I get errors in the log files without the mount ever happening.
Apr 2 13:01:08 testnode puppet-agent[29955]:
(/Stage[main]/mount::Homefilesystem/Mount[/home]/ensure) ensure changed
''unmounted'' to ''mounted''Apr 2 13:01:08
testnode puppet-agent[29955]: (/Stage[main]/mount::Homefilesystem/Mount[/home])
Could not evaluate: Execution of ''/bin/mount -o defaults,nodev
/home'' returned 1: mount.panfs error: cannot init pan_sock_ping 0x239d
(pan_sock: protected socket, permission denied)
Neither Panasas representatives I talked to seemed to have any idea what Puppet
was before I spoke to them. My coworkers, the Panasas reps, and I brainstormed a
few ideas but only three seemed to "work":
* Have the puppet daemon run as root instead of the puppet user (which is an
obvious issue)* Use auto-mount (which "works" but is causing some
oddities in a few of my jobs which I am fairly sure is due to the latency of the
mount)* Have Puppet call a script with the setuid bit configured which can mount
/home (which doesn''t 100% address my needs of puppet being able to
remount if one of those parameters is wrong/missing/changed/whatever without
that script getting complicated).
Before I commit towards one option, I thought I would ask the other Puppet
masters out there for ideas. Given the popularity of Puppet in datacenters as
well as the popularity of SAN devices in datacenters I figure someone out there
has probably solved this problem. I am hoping that their solution is better then
the ones we have come up with. :-D
Can anyone help me out with this?
Thank you in advance!!
Stack
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Jonathan Stanton
2013-Apr-09 14:56 UTC
Re: [Puppet Users] Puppet mount fails due to kernel module
Hello, On Apr 9, 2013, at 12:02 AM, Stack Kororā wrote:> Greetings! > > I am having a problem with puppet mounting a device and am hoping someone > can help. Here is the short version, if you have questions or need more > detail, please feel free to ask. > > I have a Panasas storage device on my network on which my home directory > resides. The Panasas device mounts the filesystem via a kernel module > (which they call DirectFlow). Thus, when I run as root ` mount /home` a > kernel module is loaded and then the filesystem is loaded. I have certain > restrictions in place on my servers which I have to keep in line (Puppet is > AMAZING for this!) and the mounting of /home is one of those restrictions. > > I have in my puppet manifest this: > mount { "/home" : > ensure => mounted, > atboot => true, > device => "panfs://192.168.1.20/home", > fstype => "panfs", > options => "defaults,nodev", > remounts => true, > pass => 2, > dump => 1, > } > > If I run, as root, `puppet agent --test` then the /home filesystem is > mounted and everything is wonderful. However, if I let the puppet agent > daemon try to mount /home I get errors in the log files without the mount > ever happening. >The puppet agent that runs on your server is normally running as root (it has to be to have privileges to make all of the change it can do). However a number of Linux operating systems have further security protection beyond just root which can restrict what daemons can do but does not restrict what a ''human'' logged in as root may be able to do. For example all of the Redhat Enterprise Linux (and Fedora) use Selinux which will normally limit what even root executing dameon processes can do in order to protect the system from attacks. Other OS''s like OpenBSD/FreeBSD or Linux distributions have similar capabilities that go by different names. If those are enabled, you may find that things you can do as a root user don''t work when run from cron or from a daemon process. If this is RHEL/Centos then try putting SElinux in permissive mode (as root user run ''setenforce Permissive'' ) and see if you have the same problem. If so then that identifies the issue and you can either generate a custom selinux policy for puppet, run in permissive, or change the way the mount happens. What to do depends on your organization security policy. Since it works when you run puppet agent --test the manifest itself is probably fine and the puppet mount code is able to load your module ok.> Apr 2 13:01:08 testnode puppet-agent[29955]: > (/Stage[main]/mount::Homefilesystem/Mount[/home]/ensure) ensure changed > ''unmounted'' to ''mounted'' > Apr 2 13:01:08 testnode puppet-agent[29955]: > (/Stage[main]/mount::Homefilesystem/Mount[/home]) Could not evaluate: > Execution of ''/bin/mount -o defaults,nodev /home'' returned 1: mount.panfs > error: cannot init pan_sock_ping 0x239d (pan_sock: protected socket, > permission denied) > > Neither Panasas representatives I talked to seemed to have any idea what > Puppet was before I spoke to them. My coworkers, the Panasas reps, and I > brainstormed a few ideas but only three seemed to "work": > * Have the puppet daemon run as root instead of the puppet user (which is > an obvious issue)This should already be happening. Only the ''puppet master'' runs as a regular ''puppet'' user, the agent normally runs as root.> * Use auto-mount (which "works" but is causing some oddities in a few of my > jobs which I am fairly sure is due to the latency of the mount) > * Have Puppet call a script with the setuid bit configured which can mount > /home (which doesn''t 100% address my needs of puppet being able to remount > if one of those parameters is wrong/missing/changed/whatever without that > script getting complicated).I would not recommend this. It is fragile and as you say loses much of the benefit of puppet.> > Before I commit towards one option, I thought I would ask the other Puppet > masters out there for ideas. Given the popularity of Puppet in datacenters > as well as the popularity of SAN devices in datacenters I figure someone > out there has probably solved this problem. I am hoping that their solution > is better then the ones we have come up with. :-D > > Can anyone help me out with this? > > Thank you in advance!! > > Stack >Hope this helps. Jonathan ------------------------------------------------------------------------------- Jonathan Stanton jonathan@spreadconcepts.com Spread Group Messaging www.spread.org Spread Concepts LLC www.spreadconcepts.com -------------------------------------------------------------------------------> -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Stack Kororā
2013-Apr-09 23:07 UTC
Re: [Puppet Users] Puppet mount fails due to kernel module
On Tuesday, April 9, 2013 9:56:11 AM UTC-5, Jonathan Stanton wrote:> > [snip]If those are enabled, you may find that things you can do as a root user> don''t work when run from cron or from a daemon process. > > If this is RHEL/Centos then try putting SElinux in permissive mode (as > root user run ''setenforce Permissive'' ) and see if you have the same > problem. If so then that identifies the issue and you can either generate a > custom selinux policy for puppet, run in permissive, or change the way the > mount happens. What to do depends on your organization security policy. >Thanks Jonathan! That was exactly the problem. SELinux runs on these systems and I was so focused on the puppet part that I missed the SELinux part. Doh!> [snip]This should already be happening. Only the ''puppet master'' runs as a> regular ''puppet'' user, the agent normally runs as root. >Thanks for letting me know. I thought both ran as puppet before now. I put SELinux into permissive mode and let the puppet agent do its thing successfully! Hooray!! But I really need SELinux...Any suggestions on getting this to work through SELinux* ? * I completely understand that this is not a Puppet problem anymore so a response of ''Go harass the SELinux list'' won''t hurt my feelings any. But it is worth it to ask as I am sure there are others who deal with Puppet and SELinux. :-) Since SELinux is in permissive mode, I piped the relevant information from audit.log into audit2allow. $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -m panfs module panfs 1.0; require { type node_t; type sysctl_vm_t; type mount_t; class capability net_raw; class dir search; class file read; class rawip_socket { ioctl shutdown bind create getattr node_bind }; } #============= mount_t ============== allow mount_t node_t:rawip_socket node_bind; allow mount_t self:capability net_raw; allow mount_t self:rawip_socket { bind create ioctl shutdown getattr }; allow mount_t sysctl_vm_t:dir search; allow mount_t sysctl_vm_t:file read; Since that looked good, I updated the module in SELinux $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -M panfs $ semodule -i panfs $semodule -l | grep panfs panfs 1.0 Then I turned SELinux back on with setenforce and reset puppet with `service puppet restart`. I didn''t get any SELinux audit messages, but it still doesn''t mount. It looks like (to me anyway) that the mounting process is still trying to get to resources that it can''t access because they are being blocked by SELinux. However, I was really hoping that it would put something in to the audit.log file, but nothing changed. Any ideas as to why it didn''t work? Apr 9 16:22:00 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]/ensure) ensure changed ''unmounted'' to ''mounted'' Apr 9 16:22:01 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) Could not evaluate: Execution of ''/bin/mount -o defaults,nodev /home'' returned 1: mount.panfs warning: couldn''t ping address 192.168.1.20:3095 using 192.168.1.11:1, 0x239d (pan_sock: protected socket, permission denied) Apr 9 16:22:01 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs warning: This mount still may succeed, but one or more local interfaces (listed below) failed communicate with the Panasas realm during mount. This suggests that a route cannot be established between these local interface(s) and the system. A client sends a list of IP addresses on which the Panasas storage system may establish a connection. If any one of these addresses should be excluded from the mount time check, use the ''callback-network-disallow'' or ''callback-address-disallow'' mount options. See ''man 8 mount.panfs'' for more details on PanFS mount options. Excluding the interface from the check at mount time will avoid long running mount commands. Apr 9 16:22:01 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs: failed local addresses: 192.168.1.11:1 Apr 9 16:22:01 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs: successful local addresses: Apr 9 16:22:01 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error: couldn''t ping realm servers for mount Apr 9 16:22:01 test puppet-agent[32086]: (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error: cannot process mount options in postinit step 0x7 (Invalid argument) Thanks again! I appreciate the help! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Sergio Jimenez
2013-Apr-10 13:35 UTC
Re: [Puppet Users] Puppet mount fails due to kernel module
Hi! just a thought, I don''t have any place to test it and it''s been a while since I had to deal with SELinux...but check SELinux booleans you might find something there that is not allowing puppet agent to use mount. IIRC you could list all of them with "semanage boolean -l". sjr On Wednesday, April 10, 2013 1:07:04 AM UTC+2, Stack Kororā wrote:> > > > On Tuesday, April 9, 2013 9:56:11 AM UTC-5, Jonathan Stanton wrote: >> >> [snip] > > If those are enabled, you may find that things you can do as a root user >> don''t work when run from cron or from a daemon process. >> >> If this is RHEL/Centos then try putting SElinux in permissive mode (as >> root user run ''setenforce Permissive'' ) and see if you have the same >> problem. If so then that identifies the issue and you can either generate a >> custom selinux policy for puppet, run in permissive, or change the way the >> mount happens. What to do depends on your organization security policy. >> > > Thanks Jonathan! > That was exactly the problem. SELinux runs on these systems and I was so > focused on the puppet part that I missed the SELinux part. Doh! > > >> [snip] > > This should already be happening. Only the ''puppet master'' runs as a >> regular ''puppet'' user, the agent normally runs as root. >> > > Thanks for letting me know. I thought both ran as puppet before now. > > > > I put SELinux into permissive mode and let the puppet agent do its thing > successfully! Hooray!! But I really need SELinux...Any suggestions on > getting this to work through SELinux* ? > > * I completely understand that this is not a Puppet problem anymore so a > response of ''Go harass the SELinux list'' won''t hurt my feelings any. But it > is worth it to ask as I am sure there are others who deal with Puppet and > SELinux. > :-) > > Since SELinux is in permissive mode, I piped the relevant information from > audit.log into audit2allow. > > $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -m panfs > module panfs 1.0; > > require { > type node_t; > type sysctl_vm_t; > type mount_t; > class capability net_raw; > class dir search; > class file read; > class rawip_socket { ioctl shutdown bind create getattr node_bind > }; > } > > #============= mount_t ============== > allow mount_t node_t:rawip_socket node_bind; > allow mount_t self:capability net_raw; > allow mount_t self:rawip_socket { bind create ioctl shutdown getattr }; > allow mount_t sysctl_vm_t:dir search; > allow mount_t sysctl_vm_t:file read; > > Since that looked good, I updated the module in SELinux > $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -M panfs > $ semodule -i panfs > $semodule -l | grep panfs > panfs 1.0 > > Then I turned SELinux back on with setenforce and reset puppet with > `service puppet restart`. I didn''t get any SELinux audit messages, but it > still doesn''t mount. It looks like (to me anyway) that the mounting process > is still trying to get to resources that it can''t access because they are > being blocked by SELinux. However, I was really hoping that it would put > something in to the audit.log file, but nothing changed. Any ideas as to > why it didn''t work? > > Apr 9 16:22:00 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]/ensure) ensure changed > ''unmounted'' to ''mounted'' > Apr 9 16:22:01 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) Could not evaluate: > Execution of ''/bin/mount -o defaults,nodev /home'' returned 1: mount.panfs > warning: couldn''t ping address 192.168.1.20:3095 using 192.168.1.11:1, > 0x239d (pan_sock: protected socket, permission denied) > Apr 9 16:22:01 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs warning: > This mount still may succeed, but one or more local interfaces (listed > below) failed communicate with the Panasas realm during mount. This > suggests that a route cannot be established between these local > interface(s) and the system. A client sends a list of IP addresses on > which the Panasas storage system may establish a connection. If any one of > these addresses should be excluded from the mount time check, use the > ''callback-network-disallow'' or ''callback-address-disallow'' mount options. > See ''man 8 mount.panfs'' for more details on PanFS mount options. > Excluding the interface from the check at mount time will avoid long > running mount commands. > Apr 9 16:22:01 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs: failed > local addresses: 192.168.1.11:1 > Apr 9 16:22:01 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs: > successful local addresses: > Apr 9 16:22:01 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error: > couldn''t ping realm servers for mount > Apr 9 16:22:01 test puppet-agent[32086]: > (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error: > cannot process mount options in postinit step 0x7 (Invalid argument) > > Thanks again! I appreciate the help! > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Stack Kororā
2013-Apr-12 22:22 UTC
[Puppet Users] Re: Puppet mount fails due to kernel module
Greetings!
Thanks for all the pointers in the right direction. I was finally able to
figure it out today. I figured I would post my solution in case someone
else finds this thread later on and might be able to use it. I am running
this on a few servers successfully, but again, I just figured it out today
so this is what I would call "beta" (at best). :-D
This is just the module needed in SELinux for puppet to mount a Panasas
device + the commands I used to create the module. After the module is
loaded either restart the puppet service or wait till its next go and it
should auto mount the device.
Hope it helps! And thanks again!
$ cat panfs.te
module panfs 1.4;
require {
type node_t;
type sysctl_vm_t;
type mount_t;
class capability net_raw;
class dir search;
class file read;
class rawip_socket { read bind create getattr write ioctl shutdown
node_bind };
}
#============= mount_t ==============
allow mount_t node_t:rawip_socket node_bind;
allow mount_t self:rawip_socket { bind create ioctl shutdown write read
getattr };
allow mount_t self:capability net_raw;
allow mount_t sysctl_vm_t:dir search;
allow mount_t sysctl_vm_t:file read;
$ checkmodule -M -m panfs.te -o panfs.mod
$ semodule_package -o panfs.pp -m panfs.mod
$ semodule -i panfs.pp
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.