Puppet users - Oct 2013 - puppetdb, foreman, one particular node has a problem

I have about 75 nodes now and recently added 6 of them. They are all 
running Debian squeeze and all have exactly the same Puppet packages. One 
is giving an error when I try to enroll it:

Failed to import facts: PG::UndefinedColumn: ERROR:  column
"environment"
of relation "hosts" does not exist
LINE 1: ...d", "created_at", "disk",
"domain_id", "enabled", "environme...

: INSERT INTO "hosts" ("architecture_id", "build",
"certname", "comment",
"compute_resource_id", "created_at", "disk",
"domain_id", "enabled", "environment",
"environment_id", "hostgroup_id", "image_file",
"image_id", "installed_at",
"ip", "last_compile", "last_freshcheck",
"last_report", "location_id",
"mac", "managed", "medium_id",
"model_id", "name", "operatingsystem_id",
"organization_id", "owner_id", "owner_type",
"ptable_id", "puppet_ca_proxy_id",
"puppet_proxy_id","puppet_status", "root_pass",
"serial", "source_file_id",
"subnet_id", "type", "updated_at",
"use_image", "uuid") VALUES ($1, $2, $3,
$4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, 
$20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, 
$35, $36, $37, $38, $39, $40, $41) RETURNING "id"
  Rendered text template (0.0ms)

theres nothing special about this node and other enrollments I''ve done 
today have been fine. Exactly the same environments, same packages, same 
everything. Whats different?





-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

I just realised that between now and then there was a foreman upgrade which 
has basically broken my ability to enroll new nodes.

Great. Thanks package maintainer.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

It turned out that there was an update to the foreman package which 
completely broke Puppets ability to enroll new nodes.

Call me old fashioned, I''ve been a Debian sysadmin for over 10 years,
but
on a ''stable'' system an apt-get upgrade is not supposed to
break things.
Thats one of the things Debian is famous for. Just so happens my server is 
running Ubuntu server ''long term support'' version and I
expected something
similar there.

I''ve spent some time now ripping Foreman out. Its too unreliable and I 
don''t think the benefits outweigh the costs. Puppet *without* foreman
is
just fine.

Bye bye, foreman, I''ll hardly miss you.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, Oct 16, 2013 at 11:07 AM, Steve Wray <stevedwray@gmail.com> wrote:
> It turned out that there was an update to the foreman package which
> completely broke Puppets ability to enroll new nodes.
>
> Call me old fashioned, I''ve been a Debian sysadmin for over 10
years, but
> on a ''stable'' system an apt-get upgrade is not supposed
to break things.
> Thats one of the things Debian is famous for. Just so happens my server is
> running Ubuntu server ''long term support'' version and I
expected something
> similar there.
>
>
Would you mind sharing information about which version did you upgrade from
<-> to ?

we take releases very seriously (usually 4 RC with full packages) before
rolling out an upgrade, I''m sorry it didn''t work for you this
time.

Ohad

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 2013-10-16 10:07, Steve Wray wrote:
> It turned out that there was an update to the foreman package which
> completely broke Puppets ability to enroll new nodes.
> 
> Call me old fashioned, I''ve been a Debian sysadmin for over 10
years,
> but on a ''stable'' system an apt-get upgrade is not
supposed to break
> things. Thats one of the things Debian is famous for. Just so happens my
> server is running Ubuntu server ''long term support''
version and I
> expected something similar there.
> 
> I''ve spent some time now ripping Foreman out. Its too unreliable
and I
> don''t think the benefits outweigh the costs. Puppet *without*
foreman is
> just fine.
> 
> Bye bye, foreman, I''ll hardly miss you.
You are complaining on the wrong mailing list. If you have feedback for
the Foreman developers please head over to
https://groups.google.com/forum/#!forum/foreman-users and bring the
topic up there.

Andreas

Yes, I know. At first I didn''t anticipate that it was a foreman issue. 

To be honest I''ve found foremans usefulness marginal at best and its 
performance hit on the Puppet master server quite significant, so I''m
not
inclined to pursue it further.

On Wednesday, 16 October 2013 17:26:33 UTC+8, Andreas Ntaflos
wrote:
>
> On 2013-10-16 10:07, Steve Wray wrote: 
> > It turned out that there was an update to the foreman package which 
> > completely broke Puppets ability to enroll new nodes. 
> > 
> > Call me old fashioned, I''ve been a Debian sysadmin for over
10 years,
> > but on a ''stable'' system an apt-get upgrade is not
supposed to break
> > things. Thats one of the things Debian is famous for. Just so happens
my
> > server is running Ubuntu server ''long term support''
version and I
> > expected something similar there. 
> > 
> > I''ve spent some time now ripping Foreman out. Its too
unreliable and I
> > don''t think the benefits outweigh the costs. Puppet *without*
foreman is
> > just fine. 
> > 
> > Bye bye, foreman, I''ll hardly miss you. 
>
> You are complaining on the wrong mailing list. If you have feedback for 
> the Foreman developers please head over to 
> https://groups.google.com/forum/#!forum/foreman-users and bring the 
> topic up there. 
>
> Andreas 
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Sure, I''m using this repository

deb http://deb.theforeman.org/ precise stable

I upgraded from 
foreman-postgresql 1.2.3+debian1
foreman 1.2.3+debian1
foreman-proxy 1.2.1+ubuntu1
foreman-installer 1.2.1-debian1

to
foreman-postgresql 1.3.0-1
foreman 1.3.0-1
foreman-proxy 1.3.0-1
foreman-installer 1.3.0-1

it looks as if the upgrade didn''t make the required changes to the 
database, or something like that. I dropped the db and recreated it and the 
error about the missing column went away but still couldn''t enroll new 
nodes. 

I urgently needed to make this work so I just ripped out foreman. I
didn''t
have time to research the issue further or try to fix foreman.



On Wednesday, 16 October 2013 17:24:46 UTC+8, ohad
wrote:
>
>
>
>
> On Wed, Oct 16, 2013 at 11:07 AM, Steve Wray
<steve...@gmail.com<javascript:>
> > wrote:
>
>> It turned out that there was an update to the foreman package which 
>> completely broke Puppets ability to enroll new nodes.
>>
>> Call me old fashioned, I''ve been a Debian sysadmin for over 10
years, but
>> on a ''stable'' system an apt-get upgrade is not
supposed to break things.
>> Thats one of the things Debian is famous for. Just so happens my server
is
>> running Ubuntu server ''long term support'' version and
I expected something
>> similar there.
>>
>>
> Would you mind sharing information about which version did you upgrade 
> from <-> to ?
>
> we take releases very seriously (usually 4 RC with full packages) before 
> rolling out an upgrade, I''m sorry it didn''t work for you
this time.
>
> Ohad
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 16 October 2013 11:48, Steve Wray <stevedwray@gmail.com> wrote:
> Sure, I''m using this repository
>
> deb http://deb.theforeman.org/ precise stable
>
> it looks as if the upgrade didn''t make the required changes to the
> database, or something like that. I dropped the db and recreated it and the
> error about the missing column went away but still couldn''t enroll
new
> nodes.
>
Firstly, as the deb package maintainer for Foreman, I''m sorry you had
issues. Thanks also for the bug report - we''ve just tracked down and
fixed
an issue which specifically affects sqlite3 upgrades (I''m not sure if
you
are on sqlite3, but your symptoms fit, I think). 1.3.0-2 is now in the repo
with the fix.

With regards to upgrades not breaking things, I agree. We spent some time
discussing the state of the Debian Foreman repo a few weeks ago[1] and one
outcome of that was that we would create a repo for each major version (eg
http://deb.theforeman.org/ precise 1.2) and that ''stable''
would track the
latest release. This was done, and as 1.3 was released on Monday,
''stable''
now points to 1.3. The 1.2 repo allows people who wish to be conservative
to lock themselves to a particular release.

An unavoidable side-effect of this is that since we only had "stable"
before, everyone is currently using it. I accept we could have done a
better job of publicising the existence of the 1.2 repo before 1.3 came out
though. I''ve just sent a mail to our users and announce lists to make
people more aware of the new repos.

Once again, apologies for your troubles, and thanks for the feedback.

Greg
[1]https://groups.google.com/d/msg/foreman-dev/76A7LmuPzLM/O8ob4aEX3VAJ

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Your response is encouraging, thanks.

I wasn''t using sqlite, I was using postgresql. I have about 100 nodes
(and
growing) sqlite quickly became unusable.

One of the things I tried was aptitude install with =version but this
didn''t work, apparently it couldn''t find the old versions in
the repo. I
started downloading the .debs to install the old versions manually but
realised it would be quicker to just uninstall foreman, we had just
commissioned a bunch of servers, and more arriving all the time, so I
really had to just get a move on.

Honestly, if it hadn''t been a Debian-based system, say CentOS,
I''d have
been a LOT more cautious about the 8upgrade but I''ve got so used to
things
just not breaking in stable (last time anything like that bit me was when
someone decided to apply a ''security'' patch to sudo).

When I have some breathing room I''ll do some more tests with the newer
version of foreman.




On 16 October 2013 20:02, Greg Sutcliffe <greg.sutcliffe@gmail.com> wrote:
> On 16 October 2013 11:48, Steve Wray <stevedwray@gmail.com> wrote:
>
>> Sure, I''m using this repository
>>
>> deb http://deb.theforeman.org/ precise stable
>>
>> it looks as if the upgrade didn''t make the required changes to
the
>> database, or something like that. I dropped the db and recreated it and
the
>> error about the missing column went away but still couldn''t
enroll new
>> nodes.
>>
>
> Firstly, as the deb package maintainer for Foreman, I''m sorry you
had
> issues. Thanks also for the bug report - we''ve just tracked down
and fixed
> an issue which specifically affects sqlite3 upgrades (I''m not sure
if you
> are on sqlite3, but your symptoms fit, I think). 1.3.0-2 is now in the repo
> with the fix.
>
> With regards to upgrades not breaking things, I agree. We spent some time
> discussing the state of the Debian Foreman repo a few weeks ago[1] and one
> outcome of that was that we would create a repo for each major version (eg
> http://deb.theforeman.org/ precise 1.2) and that ''stable''
would track the
> latest release. This was done, and as 1.3 was released on Monday,
''stable''
> now points to 1.3. The 1.2 repo allows people who wish to be conservative
> to lock themselves to a particular release.
>
> An unavoidable side-effect of this is that since we only had
"stable"
> before, everyone is currently using it. I accept we could have done a
> better job of publicising the existence of the 1.2 repo before 1.3 came out
> though. I''ve just sent a mail to our users and announce lists to
make
> people more aware of the new repos.
>
> Once again, apologies for your troubles, and thanks for the feedback.
>
> Greg
> [1]https://groups.google.com/d/msg/foreman-dev/76A7LmuPzLM/O8ob4aEX3VAJ
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/Dfgwz8bPYvY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On 16 October 2013 14:39, Steve Wray <stevedwray@gmail.com> wrote:
> Your response is encouraging, thanks.
>
> I wasn''t using sqlite, I was using postgresql. I have about 100
nodes (and
> growing) sqlite quickly became unusable.
>
Ah ok. There are changes to the ENC script (which also registers new Hosts
in the Foreman interface) in 1.3, but due to a variety of reasons, the ENC
(/etc/puppet/node.rb) isn''t packaged. So potentially, that needed an
update
to fix your host registration problem.

One of the things I tried was aptitude install with =version but
this
> didn''t work, apparently it couldn''t find the old versions
in the repo. I
> started downloading the .debs to install the old versions manually but
> realised it would be quicker to just uninstall foreman, we had just
> commissioned a bunch of servers, and more arriving all the time, so I
> really had to just get a move on.
>
Understandable, I''ve been in those firefighting situations.
It''s
interesting that this didn''t work for you though - I was using that
syntax
this very morning to test the sqlite3 error fixes :)
>
> Honestly, if it hadn''t been a Debian-based system, say CentOS,
I''d have
> been a LOT more cautious about the 8upgrade but I''ve got so used
to things
> just not breaking in stable (last time anything like that bit me was when
> someone decided to apply a ''security'' patch to sudo).
>
> When I have some breathing room I''ll do some more tests with the
newer
> version of foreman.
>
Looking forward to your feedback. See you on foreman-users :)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.