Puppet users - Mar 2013 - Augeas lens dependency on valid config file parsing

Hi All,

I just ran into a frustrating Augeas problem, which I thought I''d share
with you all.

We run a cron every morning to fetch the latest RPMs (for the packages
we''re interested in) from various places such as rpmforge, EPEL, etc.

This morning I tried to build a VM and it failed. What happened was that
last night, a new version of nagios-nrpe was downloaded so the VM that was
built had this latest version. The RPM includes nrpe.cfg which had a new
option added to it:

allow_bash_command_substitution=0

In the manifest, I have an augeas block to modify the allowed_hosts line in
the nrpe.cfg file. Puppet was failing giving a "Save failed" error
message,
so I went in via augtool to see what''s going on.

I noticed that it could not parse nrpe.cfg (i.e. ls /files/etc/nagios
returned nothing). I had the previous config file saved for NRPE somewhere,
so I diff''ed them and noticed the only new option is the option I
mentioned
above. After removing this new option, augtool could parse the file again
and Puppet was happy.

After having a quick read of the nrpe.lns file inside augeas-libs, I
noticed it actually lists each of the options that it will recognise from
the config file, and evidently fails if it finds new options that it
doesn''t recognise.

Anyway, it hit me that there is quite a big dependency between config files
and augeas lenses. A change in one config file can affect Augeas which
affects Puppet runs. Or is this just a poorly written lens?

- Gonzalo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 22/03/13 04:02, Gonzalo Servat wrote:
> Hi All,
> 
> I just ran into a frustrating Augeas problem, which I thought I''d
share
> with you all.
> 
> We run a cron every morning to fetch the latest RPMs (for the packages
> we''re interested in) from various places such as rpmforge, EPEL,
etc.
> 
> This morning I tried to build a VM and it failed. What happened was that
> last night, a new version of nagios-nrpe was downloaded so the VM that
> was built had this latest version. The RPM includes nrpe.cfg which had a
> new option added to it:
> 
> allow_bash_command_substitution=0
> 
> In the manifest, I have an augeas block to modify the allowed_hosts line
> in the nrpe.cfg file. Puppet was failing giving a "Save failed"
error
> message, so I went in via augtool to see what''s going on.
> 
> I noticed that it could not parse nrpe.cfg (i.e. ls /files/etc/nagios
> returned nothing). I had the previous config file saved for NRPE
> somewhere, so I diff''ed them and noticed the only new option is
the
> option I mentioned above. After removing this new option, augtool could
> parse the file again and Puppet was happy.
> 
> After having a quick read of the nrpe.lns file inside augeas-libs, I
> noticed it actually lists each of the options that it will recognise
> from the config file, and evidently fails if it finds new options that
> it doesn''t recognise.
> 
> Anyway, it hit me that there is quite a big dependency between config
> files and augeas lenses. A change in one config file can affect Augeas
> which affects Puppet runs. Or is this just a poorly written lens?
You might better off directing this at augeas-devel@redhat.com (CCed).

Lenses can be written either to match a file format very specifically
(as in this case) or quite loosely.  There are pros and cons to each.

Clearly writing a fairly loose definition for the file which accepts
something like [a-z_]+ for the name of a setting is useful when new
settings are added, as then the lens doesn''t stop parsing when it comes
across the new name.

The downside is if the new setting has data that requires a different or
more complex representation in the Augeas tree.  A loose lens might
interpret a setting that''s made up of multiple space-separated values
as
a single value, meaning it''s not as useful to people as they
can''t alter
a single value using Augeas (which is the point of the library).  This
situation is worsened when somebody goes to fix it, as they will likely
to have to change the format of the tree for that entry, which is an
incompatible change between Augeas versions.

A recent example of this is the Postfix main.cf lens, which treats
everything as a simple value, but then don''t split up settings like
virtual_user_maps which are made up of multiple values:
https://www.redhat.com/archives/augeas-devel/2012-November/msg00049.html

Some lenses are of the former type and some of the latter, depending on
the original contributor''s viewpoint and the type of file
they''re
parsing, how the format is being developed etc.

I think on balance we''re preferring the former type in Augeas, so if
you
wanted to change this to a regex that matches all key names except for
include and command (which have special formats in the lens), a patch
would probably be accepted.  At the very least, a bug report would be
appreciated:

https://fedorahosted.org/augeas/

Perhaps ideally, lenses could be bundled with the software responsible
for the config file formats themselves so they''re more easily kept in
sync.  There are a small number of projects that do this (libvirt,
libguestfs, corosync), but it needs a supportive upstream.

Regards,

-- 
Dominic Cleal
Red Hat Engineering

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, Mar 22, 2013 at 9:11 PM, Dominic Cleal <dcleal@redhat.com> wrote:
> Perhaps ideally, lenses could be bundled with the software responsible
> for the config file formats themselves so they''re more easily kept
in
> sync.  There are a small number of projects that do this (libvirt,
> libguestfs, corosync), but it needs a supportive upstream.
>
Hi Dominic,

Thanks for replying. I haven''t got my head around the lens syntax just
yet
but I will have a go at it and see if I can submit a patch.

I think your last paragraph is spot on -- the lenses should be bundled with
the software responsible for the config file formats. Perhaps augeas should
look in /etc/augeas.d/ or something for the lenses and packages can include
a lens file that goes in that directory. It makes sense, as a change in a
config file has to include a review of the lens before releasing a new
package.

- Gonzalo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Sat, Mar 23, 2013 at 9:47 AM, Gonzalo Servat <gservat@gmail.com> wrote:
> On Fri, Mar 22, 2013 at 9:11 PM, Dominic Cleal <dcleal@redhat.com>
wrote:
>
>> Perhaps ideally, lenses could be bundled with the software responsible
>>  for the config file formats themselves so they''re more easily
kept in
>> sync.  There are a small number of projects that do this (libvirt,
>> libguestfs, corosync), but it needs a supportive upstream.
>>
>
> Hi Dominic,
>
> Thanks for replying. I haven''t got my head around the lens syntax
just yet
> but I will have a go at it and see if I can submit a patch.
>
> I think your last paragraph is spot on -- the lenses should be bundled
> with the software responsible for the config file formats. Perhaps augeas
> should look in /etc/augeas.d/ or something for the lenses and packages can
> include a lens file that goes in that directory. It makes sense, as a
> change in a config file has to include a review of the lens before
> releasing a new package.
>
I had a go at modifying the lens and got it working to accept any option.
For anyone that runs into the same problem, I''ve opened a ticket and
submitted a patch:

    https://fedorahosted.org/augeas/ticket/334

- Gonzalo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I think that this is mostly because of lens that are written this way.
We shouldn''t blame them that this is "poorly" written in my
opinion.
It is just more restrictive. Lens that I have written accept a
regular expression as an identifier instead of listing them.

Best, Nikola

On Fri, Mar 22, 2013 at 03:02:51PM +1100, Gonzalo Servat
wrote:
> Hi All,
> 
> I just ran into a frustrating Augeas problem, which I thought I''d
share
> with you all.
> 
> We run a cron every morning to fetch the latest RPMs (for the packages
> we''re interested in) from various places such as rpmforge, EPEL,
etc.
> 
> This morning I tried to build a VM and it failed. What happened was that
> last night, a new version of nagios-nrpe was downloaded so the VM that was
> built had this latest version. The RPM includes nrpe.cfg which had a new
> option added to it:
> 
> allow_bash_command_substitution=0
> 
> In the manifest, I have an augeas block to modify the allowed_hosts line in
> the nrpe.cfg file. Puppet was failing giving a "Save failed"
error message,
> so I went in via augtool to see what''s going on.
> 
> I noticed that it could not parse nrpe.cfg (i.e. ls /files/etc/nagios
> returned nothing). I had the previous config file saved for NRPE somewhere,
> so I diff''ed them and noticed the only new option is the option I
mentioned
> above. After removing this new option, augtool could parse the file again
> and Puppet was happy.
> 
> After having a quick read of the nrpe.lns file inside augeas-libs, I
> noticed it actually lists each of the options that it will recognise from
> the config file, and evidently fails if it finds new options that it
> doesn''t recognise.
> 
> Anyway, it hit me that there is quite a big dependency between config files
> and augeas lenses. A change in one config file can affect Augeas which
> affects Puppet runs. Or is this just a poorly written lens?
> 
> - Gonzalo
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 22/03/13 23:22, Gonzalo Servat wrote:
> On Sat, Mar 23, 2013 at 9:47 AM, Gonzalo Servat <gservat@gmail.com
> <mailto:gservat@gmail.com>> wrote:
> 
>     On Fri, Mar 22, 2013 at 9:11 PM, Dominic Cleal <dcleal@redhat.com
>     <mailto:dcleal@redhat.com>> wrote:
> 
>         Perhaps ideally, lenses could be bundled with the software
>         responsible
>         for the config file formats themselves so they''re more
easily
>         kept in
>         sync.  There are a small number of projects that do this (libvirt,
>         libguestfs, corosync), but it needs a supportive upstream.
> 
> 
>     Hi Dominic,
> 
>     Thanks for replying. I haven''t got my head around the lens
syntax
>     just yet but I will have a go at it and see if I can submit a patch.
> 
>     I think your last paragraph is spot on -- the lenses should be
>     bundled with the software responsible for the config file formats.
>     Perhaps augeas should look in /etc/augeas.d/ or something for the
>     lenses and packages can include a lens file that goes in that
>     directory. It makes sense, as a change in a config file has to
>     include a review of the lens before releasing a new package.
Yes, Augeas supports two directories it simply reads from to find all
lenses.  These are typically /usr/share/augeas/lenses and
/usr/share/augeas/lenses/dist.  Usually when software shipping lenses
itself is packaged, they get installed to the former directory while
Augeas'' own go in dist/.
> I had a go at modifying the lens and got it working to accept any
> option. For anyone that runs into the same problem, I''ve opened a
ticket
> and submitted a patch:
> 
>     https://fedorahosted.org/augeas/ticket/334 
Thank you very much, I''ll review it when I get the opportunity.

-- 
Dominic Cleal
Red Hat Engineering

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.