Puppet users - Oct 2013 - Sync data between various PuppetDB instances

Dear all,

We have a few reasons to have various PuppetDBs (and Masters, one per
cluster) in our datacenter and, eventually, there may be a need to share
*only* a few exported resources between clusters, like a few ssh-keys or
nagios tests.

Is there a mechanism to pull and push some exported resources from one
PuppetDB to another? I guess this could be done with the API, but has
anybody tried it, or will it work at all?

Thanks a lot,
Pablo Fernandez

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

I have found a PuppetDB Hiera backend
(https://github.com/dalen/puppet-puppetdbquery) that could (potentially)
be used to query other PuppetDBs... but that would return hiera data,
and not exported resources like a normal PuppetDB does.

Did anybody try to push data to a PuppetDB by hand? The API does not
seem to have the PUT or POST methods.

Thanks!
BR/Pablo


On 10/07/2013 01:07 PM, Pablo Fernandez wrote:
> Dear all,
>
> We have a few reasons to have various PuppetDBs (and Masters, one per
> cluster) in our datacenter and, eventually, there may be a need to share
> *only* a few exported resources between clusters, like a few ssh-keys or
> nagios tests.
>
> Is there a mechanism to pull and push some exported resources from one
> PuppetDB to another? I guess this could be done with the API, but has
> anybody tried it, or will it work at all?
>
> Thanks a lot,
> Pablo Fernandez
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

>> Is there a mechanism to pull and push some exported resources from one
>> PuppetDB to another? I guess this could be done with the API, but has
>> anybody tried it, or will it work at all?
Not provided by us, yet. We''ve had a lot of discussions internally
about this kind of thing. How about you outline your requirements in
the form of a feature request here:

http://projects.puppetlabs.com/projects/puppetdb/issues
> Did anybody try to push data to a PuppetDB by hand? The API does not
> seem to have the PUT or POST methods.
The client obviously pushes uses POST :-). The documentation for the
mechanism is here:

http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

ken.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks Ken,

I took a look at the Command API and seems to be quite interesting, but
it triggers a few more questions.

As a reminder (also for others), what I wanted to do was to add exported
resources from Node_A (managed by PuppetMaster_A that has its own
PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
order for other Nodes_B (managed by PuppetMaster_B) be aware and
possibly realize those exported resources.

Taking a look the Commands API seem to capabilities to change a whole
catalog, or facts. Do they need to be existing catalogs, or could I add
new ones?
I am thinking: If I want to add Node_A''s exported resource(s) to
PuppetDB_B, I could "artificially" add its whole catalog to
PuppetDB_B... but the Command API is "replace catalog", no "add
catalog". Would it be rejected? How does the first catalog of a node get
introduced into the DB? And moreover, how do I query a whole catalog?

Another possibility would be to include the resource inside an existing
Node_B catalog on the PuppetDB_B (after all, exported resources don''t
really mater what node do they belong, as long as they have the right
attributes, right?), but the Catalog wire format seems to require the
whole set of resources, so, if I define the resources of Node_B with
some exported resources from Node_A, I believe I would lose the original
Node_B''s resources. The only possibility would be to query Node_B full
catalog, and *add also* those resources from Node_A before sending them
to the DB... and this seems to me like a bit dirty :)

Am I missing anything? Or is there another way to implemen what I need
with Puppet 3.2?

Thanks again,
BR/Pablo


On 10/10/2013 09:16 AM, Ken Barber wrote:
>>> Is there a mechanism to pull and push some exported resources from
one
>>> PuppetDB to another? I guess this could be done with the API, but
has
>>> anybody tried it, or will it work at all?
> Not provided by us, yet. We''ve had a lot of discussions internally
> about this kind of thing. How about you outline your requirements in
> the form of a feature request here:
>
> http://projects.puppetlabs.com/projects/puppetdb/issues
>
>> Did anybody try to push data to a PuppetDB by hand? The API does not
>> seem to have the PUT or POST methods.
> The client obviously pushes uses POST :-). The documentation for the
> mechanism is here:
>
> http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>
> ken.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

I''d be interested in accomplishing a similar thing. I want to have a
central reporting puppetdb which I can (outside of puppet) query for
multisite information without tying isolated environments together via a
single postgres db, or using multi-master replication.

my needs do not involve making each puppet site aware of each other''s
data,
but it would be nice to have one db which has all the data for my digging
pleasure.

from my understanding this is not currently supported.

I was thinking something could be done to ask puppetdb at site A (what
updates have you had in the past N minutes/ since the last time I checked
in with you/ since timestamp(N)), and then iterate through the collected
data and feed it into a reporting puppetdb instance, however I''ve not
gone
beyond thinking about it.





On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch>wrote:
> Thanks Ken,
>
> I took a look at the Command API and seems to be quite interesting, but
> it triggers a few more questions.
>
> As a reminder (also for others), what I wanted to do was to add exported
> resources from Node_A (managed by PuppetMaster_A that has its own
> PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
> order for other Nodes_B (managed by PuppetMaster_B) be aware and
> possibly realize those exported resources.
>
> Taking a look the Commands API seem to capabilities to change a whole
> catalog, or facts. Do they need to be existing catalogs, or could I add
> new ones?
> I am thinking: If I want to add Node_A''s exported resource(s) to
> PuppetDB_B, I could "artificially" add its whole catalog to
> PuppetDB_B... but the Command API is "replace catalog", no
"add
> catalog". Would it be rejected? How does the first catalog of a node
get
> introduced into the DB? And moreover, how do I query a whole catalog?
>
> Another possibility would be to include the resource inside an existing
> Node_B catalog on the PuppetDB_B (after all, exported resources
don''t
> really mater what node do they belong, as long as they have the right
> attributes, right?), but the Catalog wire format seems to require the
> whole set of resources, so, if I define the resources of Node_B with
> some exported resources from Node_A, I believe I would lose the original
> Node_B''s resources. The only possibility would be to query Node_B
full
> catalog, and *add also* those resources from Node_A before sending them
> to the DB... and this seems to me like a bit dirty :)
>
> Am I missing anything? Or is there another way to implemen what I need
> with Puppet 3.2?
>
> Thanks again,
> BR/Pablo
>
>
> On 10/10/2013 09:16 AM, Ken Barber wrote:
> >>> Is there a mechanism to pull and push some exported resources
from one
> >>> PuppetDB to another? I guess this could be done with the API,
but has
> >>> anybody tried it, or will it work at all?
> > Not provided by us, yet. We''ve had a lot of discussions
internally
> > about this kind of thing. How about you outline your requirements in
> > the form of a feature request here:
> >
> > http://projects.puppetlabs.com/projects/puppetdb/issues
> >
> >> Did anybody try to push data to a PuppetDB by hand? The API does
not
> >> seem to have the PUT or POST methods.
> > The client obviously pushes uses POST :-). The documentation for the
> > mechanism is here:
> >
> > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
> >
> > ken.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

It sounds like a shade of #19321 -
https://projects.puppetlabs.com/issues/19321


On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com> wrote:
> I''d be interested in accomplishing a similar thing. I want to have
a
> central reporting puppetdb which I can (outside of puppet) query for
> multisite information without tying isolated environments together via a
> single postgres db, or using multi-master replication.
>
> my needs do not involve making each puppet site aware of each
other''s
> data, but it would be nice to have one db which has all the data for my
> digging pleasure.
>
> from my understanding this is not currently supported.
>
> I was thinking something could be done to ask puppetdb at site A (what
> updates have you had in the past N minutes/ since the last time I checked
> in with you/ since timestamp(N)), and then iterate through the collected
> data and feed it into a reporting puppetdb instance, however I''ve
not gone
> beyond thinking about it.
>
>
>
>
>
> On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch>wrote:
>
>> Thanks Ken,
>>
>> I took a look at the Command API and seems to be quite interesting, but
>> it triggers a few more questions.
>>
>> As a reminder (also for others), what I wanted to do was to add
exported
>> resources from Node_A (managed by PuppetMaster_A that has its own
>> PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
>> order for other Nodes_B (managed by PuppetMaster_B) be aware and
>> possibly realize those exported resources.
>>
>> Taking a look the Commands API seem to capabilities to change a whole
>> catalog, or facts. Do they need to be existing catalogs, or could I add
>> new ones?
>> I am thinking: If I want to add Node_A''s exported resource(s)
to
>> PuppetDB_B, I could "artificially" add its whole catalog to
>> PuppetDB_B... but the Command API is "replace catalog", no
"add
>> catalog". Would it be rejected? How does the first catalog of a
node get
>> introduced into the DB? And moreover, how do I query a whole catalog?
>>
>> Another possibility would be to include the resource inside an existing
>> Node_B catalog on the PuppetDB_B (after all, exported resources
don''t
>> really mater what node do they belong, as long as they have the right
>> attributes, right?), but the Catalog wire format seems to require the
>> whole set of resources, so, if I define the resources of Node_B with
>> some exported resources from Node_A, I believe I would lose the
original
>> Node_B''s resources. The only possibility would be to query
Node_B full
>> catalog, and *add also* those resources from Node_A before sending them
>> to the DB... and this seems to me like a bit dirty :)
>>
>> Am I missing anything? Or is there another way to implemen what I need
>> with Puppet 3.2?
>>
>> Thanks again,
>> BR/Pablo
>>
>>
>> On 10/10/2013 09:16 AM, Ken Barber wrote:
>> >>> Is there a mechanism to pull and push some exported
resources from one
>> >>> PuppetDB to another? I guess this could be done with the
API, but has
>> >>> anybody tried it, or will it work at all?
>> > Not provided by us, yet. We''ve had a lot of discussions
internally
>> > about this kind of thing. How about you outline your requirements
in
>> > the form of a feature request here:
>> >
>> > http://projects.puppetlabs.com/projects/puppetdb/issues
>> >
>> >> Did anybody try to push data to a PuppetDB by hand? The API
does not
>> >> seem to have the PUT or POST methods.
>> > The client obviously pushes uses POST :-). The documentation for
the
>> > mechanism is here:
>> >
>> > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>> >
>> > ken.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
an
>> email to puppet-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Yes, I saw that... this may be a solution for replicating full PuppetDBs
(if you tackle the backlog during a long shortage), but that does not
scale very well if you have 5-10 PuppetDBs all being replicated to a
single, bigger one.

Besides, it would not cover cases when you want to have only some nodes
replicated, or even only some resources.


On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
> It sounds like a shade of #19321 -
> https://projects.puppetlabs.com/issues/19321
>
>
> On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com
> <mailto:wolf@wolfspyre.com>> wrote:
>
>     I''d be interested in accomplishing a similar thing. I want to
have
>     a central reporting puppetdb which I can (outside of puppet) query
>     for multisite information without tying isolated environments
>     together via a single postgres db, or using multi-master
>     replication. 
>
>     my needs do not involve making each puppet site aware of each
>     other''s data, but it would be nice to have one db which has
all
>     the data for my digging pleasure.
>
>     from my understanding this is not currently supported.
>
>     I was thinking something could be done to ask puppetdb at site A
>     (what updates have you had in the past N minutes/ since the last
>     time I checked in with you/ since timestamp(N)), and then iterate
>     through the collected data and feed it into a reporting puppetdb
>     instance, however I''ve not gone beyond thinking about it.
>
>
>
>
>
>     On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
>     <pablo.fernandez@cscs.ch <mailto:pablo.fernandez@cscs.ch>>
wrote:
>
>         Thanks Ken,
>
>         I took a look at the Command API and seems to be quite
>         interesting, but
>         it triggers a few more questions.
>
>         As a reminder (also for others), what I wanted to do was to
>         add exported
>         resources from Node_A (managed by PuppetMaster_A that has its own
>         PuppetDB_A) into a different PuppetDB_B, where Node_A is
>         unknown, in
>         order for other Nodes_B (managed by PuppetMaster_B) be aware and
>         possibly realize those exported resources.
>
>         Taking a look the Commands API seem to capabilities to change
>         a whole
>         catalog, or facts. Do they need to be existing catalogs, or
>         could I add
>         new ones?
>         I am thinking: If I want to add Node_A''s exported
resource(s) to
>         PuppetDB_B, I could "artificially" add its whole catalog
to
>         PuppetDB_B... but the Command API is "replace catalog",
no "add
>         catalog". Would it be rejected? How does the first catalog of
>         a node get
>         introduced into the DB? And moreover, how do I query a whole
>         catalog?
>
>         Another possibility would be to include the resource inside an
>         existing
>         Node_B catalog on the PuppetDB_B (after all, exported
>         resources don''t
>         really mater what node do they belong, as long as they have
>         the right
>         attributes, right?), but the Catalog wire format seems to
>         require the
>         whole set of resources, so, if I define the resources of
>         Node_B with
>         some exported resources from Node_A, I believe I would lose
>         the original
>         Node_B''s resources. The only possibility would be to query
>         Node_B full
>         catalog, and *add also* those resources from Node_A before
>         sending them
>         to the DB... and this seems to me like a bit dirty :)
>
>         Am I missing anything? Or is there another way to implemen
>         what I need
>         with Puppet 3.2?
>
>         Thanks again,
>         BR/Pablo
>
>
>         On 10/10/2013 09:16 AM, Ken Barber wrote:
>         >>> Is there a mechanism to pull and push some exported
>         resources from one
>         >>> PuppetDB to another? I guess this could be done with
the
>         API, but has
>         >>> anybody tried it, or will it work at all?
>         > Not provided by us, yet. We''ve had a lot of
discussions
>         internally
>         > about this kind of thing. How about you outline your
>         requirements in
>         > the form of a feature request here:
>         >
>         > http://projects.puppetlabs.com/projects/puppetdb/issues
>         >
>         >> Did anybody try to push data to a PuppetDB by hand? The
API
>         does not
>         >> seem to have the PUT or POST methods.
>         > The client obviously pushes uses POST :-). The documentation
>         for the
>         > mechanism is here:
>         >
>         > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>         >
>         > ken.
>         >
>
>         --
>         You received this message because you are subscribed to the
>         Google Groups "Puppet Users" group.
>         To unsubscribe from this group and stop receiving emails from
>         it, send an email to puppet-users+unsubscribe@googlegroups.com
>         <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>         To post to this group, send email to
>         puppet-users@googlegroups.com
>         <mailto:puppet-users@googlegroups.com>.
>         Visit this group at http://groups.google.com/group/puppet-users.
>         For more options, visit https://groups.google.com/groups/opt_out.
>
>
>     -- 
>     You received this message because you are subscribed to the Google
>     Groups "Puppet Users" group.
>     To unsubscribe from this group and stop receiving emails from it,
>     send an email to puppet-users+unsubscribe@googlegroups.com
>     <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>     To post to this group, send email to puppet-users@googlegroups.com
>     <mailto:puppet-users@googlegroups.com>.
>     Visit this group at http://groups.google.com/group/puppet-users.
>     For more options, visit https://groups.google.com/groups/opt_out.
>
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

That would really depend on the approach ultimately taken for the
multi-master. If the puppet master (or agent in masterless mode) forwards
the same fact/report/node/etc data to multiple puppetdb instances then yes
the scale breaks horribly and does not meet part of this request.  If there
is some kind of store-and-forward mechanism in puppetdb itself, it would be
possible to extend the forward to say ''only if it matches these
rules''.
 This could definitely be useful in a variety of ways beyond simple HA
purposes.



On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch>wrote:
>  Yes, I saw that... this may be a solution for replicating full PuppetDBs
> (if you tackle the backlog during a long shortage), but that does not scale
> very well if you have 5-10 PuppetDBs all being replicated to a single,
> bigger one.
>
> Besides, it would not cover cases when you want to have only some nodes
> replicated, or even only some resources.
>
>
>
> On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
>
> It sounds like a shade of #19321 -
> https://projects.puppetlabs.com/issues/19321
>
>
> On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com>
wrote:
>
>> I''d be interested in accomplishing a similar thing. I want to
have a
>> central reporting puppetdb which I can (outside of puppet) query for
>> multisite information without tying isolated environments together via
a
>> single postgres db, or using multi-master replication.
>>
>>  my needs do not involve making each puppet site aware of each
other''s
>> data, but it would be nice to have one db which has all the data for my
>> digging pleasure.
>>
>>  from my understanding this is not currently supported.
>>
>>  I was thinking something could be done to ask puppetdb at site A (what
>> updates have you had in the past N minutes/ since the last time I
checked
>> in with you/ since timestamp(N)), and then iterate through the
collected
>> data and feed it into a reporting puppetdb instance, however
I''ve not gone
>> beyond thinking about it.
>>
>>
>>
>>
>>
>> On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch
>> > wrote:
>>
>>> Thanks Ken,
>>>
>>> I took a look at the Command API and seems to be quite interesting,
but
>>> it triggers a few more questions.
>>>
>>> As a reminder (also for others), what I wanted to do was to add
exported
>>> resources from Node_A (managed by PuppetMaster_A that has its own
>>> PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown,
in
>>> order for other Nodes_B (managed by PuppetMaster_B) be aware and
>>> possibly realize those exported resources.
>>>
>>> Taking a look the Commands API seem to capabilities to change a
whole
>>> catalog, or facts. Do they need to be existing catalogs, or could I
add
>>> new ones?
>>> I am thinking: If I want to add Node_A''s exported
resource(s) to
>>> PuppetDB_B, I could "artificially" add its whole catalog
to
>>> PuppetDB_B... but the Command API is "replace catalog",
no "add
>>> catalog". Would it be rejected? How does the first catalog of
a node get
>>> introduced into the DB? And moreover, how do I query a whole
catalog?
>>>
>>> Another possibility would be to include the resource inside an
existing
>>> Node_B catalog on the PuppetDB_B (after all, exported resources
don''t
>>> really mater what node do they belong, as long as they have the
right
>>> attributes, right?), but the Catalog wire format seems to require
the
>>> whole set of resources, so, if I define the resources of Node_B
with
>>> some exported resources from Node_A, I believe I would lose the
original
>>> Node_B''s resources. The only possibility would be to query
Node_B full
>>> catalog, and *add also* those resources from Node_A before sending
them
>>> to the DB... and this seems to me like a bit dirty :)
>>>
>>> Am I missing anything? Or is there another way to implemen what I
need
>>> with Puppet 3.2?
>>>
>>> Thanks again,
>>> BR/Pablo
>>>
>>>
>>> On 10/10/2013 09:16 AM, Ken Barber wrote:
>>> >>> Is there a mechanism to pull and push some exported
resources from
>>> one
>>> >>> PuppetDB to another? I guess this could be done with
the API, but has
>>> >>> anybody tried it, or will it work at all?
>>> > Not provided by us, yet. We''ve had a lot of
discussions internally
>>> > about this kind of thing. How about you outline your
requirements in
>>> > the form of a feature request here:
>>> >
>>> > http://projects.puppetlabs.com/projects/puppetdb/issues
>>> >
>>> >> Did anybody try to push data to a PuppetDB by hand? The
API does not
>>> >> seem to have the PUT or POST methods.
>>> > The client obviously pushes uses POST :-). The documentation
for the
>>> > mechanism is here:
>>> >
>>> > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>>> >
>>> > ken.
>>> >
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
send
>>> an email to puppet-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to puppet-users@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/puppet-users.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
an
>> email to puppet-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Ok, so I finally opted for inserting a full node from one DB into
another, since getting the resources should be quite easy with a GET.

But then I run into a problem when running the replace catalog command
(with an empty catalog):

curl -vv -G -H "Accept: application/json"
''http://localhost:8080/v2/commands'' --data-urlencode
''payload={"command":"replace
catalog","version": 2,"payload":"{
\"metadata\": {\"api_version\": 1}, \"data\":  
{\"name\":
\"puppet09.cscs.ch\",\"version\": \"1111\",
\"edges\": [],
\"resources\": [] }    }"}''

Which returns ok. The puppetdb.log shows:

2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
[43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying after
attempt 8, due to: org.postgresql.util.PSQLException: ERROR: syntax
error at or near ")"
  Position: 66

I have looked again and again into the data, and syntax seems to be correct.

Anybody tried this? Hints?

Thanks a lot,
Pablo Fernandez




On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
> That would really depend on the approach ultimately taken for the
> multi-master. If the puppet master (or agent in masterless mode)
> forwards the same fact/report/node/etc data to multiple puppetdb
> instances then yes the scale breaks horribly and does not meet part of
> this request.  If there is some kind of store-and-forward mechanism in
> puppetdb itself, it would be possible to extend the forward to say
> ''only if it matches these rules''.  This could definitely
be useful in
> a variety of ways beyond simple HA purposes.
>
>
>
> On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
> <pablo.fernandez@cscs.ch <mailto:pablo.fernandez@cscs.ch>>
wrote:
>
>     Yes, I saw that... this may be a solution for replicating full
>     PuppetDBs (if you tackle the backlog during a long shortage), but
>     that does not scale very well if you have 5-10 PuppetDBs all being
>     replicated to a single, bigger one.
>
>     Besides, it would not cover cases when you want to have only some
>     nodes replicated, or even only some resources.
>
>
>
>     On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
>>     It sounds like a shade of #19321 -
>>     https://projects.puppetlabs.com/issues/19321
>>
>>
>>     On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com
>>     <mailto:wolf@wolfspyre.com>> wrote:
>>
>>         I''d be interested in accomplishing a similar thing. I
want to
>>         have a central reporting puppetdb which I can (outside of
>>         puppet) query for multisite information without tying
>>         isolated environments together via a single postgres db, or
>>         using multi-master replication. 
>>
>>         my needs do not involve making each puppet site aware of each
>>         other''s data, but it would be nice to have one db
which has
>>         all the data for my digging pleasure.
>>
>>         from my understanding this is not currently supported.
>>
>>         I was thinking something could be done to ask puppetdb at
>>         site A (what updates have you had in the past N minutes/
>>         since the last time I checked in with you/ since
>>         timestamp(N)), and then iterate through the collected data
>>         and feed it into a reporting puppetdb instance, however
I''ve
>>         not gone beyond thinking about it.
>>
>>
>>
>>
>>
>>         On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
>>         <pablo.fernandez@cscs.ch
<mailto:pablo.fernandez@cscs.ch>> wrote:
>>
>>             Thanks Ken,
>>
>>             I took a look at the Command API and seems to be quite
>>             interesting, but
>>             it triggers a few more questions.
>>
>>             As a reminder (also for others), what I wanted to do was
>>             to add exported
>>             resources from Node_A (managed by PuppetMaster_A that has
>>             its own
>>             PuppetDB_A) into a different PuppetDB_B, where Node_A is
>>             unknown, in
>>             order for other Nodes_B (managed by PuppetMaster_B) be
>>             aware and
>>             possibly realize those exported resources.
>>
>>             Taking a look the Commands API seem to capabilities to
>>             change a whole
>>             catalog, or facts. Do they need to be existing catalogs,
>>             or could I add
>>             new ones?
>>             I am thinking: If I want to add Node_A''s exported
>>             resource(s) to
>>             PuppetDB_B, I could "artificially" add its whole
catalog to
>>             PuppetDB_B... but the Command API is "replace
catalog",
>>             no "add
>>             catalog". Would it be rejected? How does the first
>>             catalog of a node get
>>             introduced into the DB? And moreover, how do I query a
>>             whole catalog?
>>
>>             Another possibility would be to include the resource
>>             inside an existing
>>             Node_B catalog on the PuppetDB_B (after all, exported
>>             resources don''t
>>             really mater what node do they belong, as long as they
>>             have the right
>>             attributes, right?), but the Catalog wire format seems to
>>             require the
>>             whole set of resources, so, if I define the resources of
>>             Node_B with
>>             some exported resources from Node_A, I believe I would
>>             lose the original
>>             Node_B''s resources. The only possibility would be
to
>>             query Node_B full
>>             catalog, and *add also* those resources from Node_A
>>             before sending them
>>             to the DB... and this seems to me like a bit dirty :)
>>
>>             Am I missing anything? Or is there another way to
>>             implemen what I need
>>             with Puppet 3.2?
>>
>>             Thanks again,
>>             BR/Pablo
>>
>>
>>             On 10/10/2013 09:16 AM, Ken Barber wrote:
>>             >>> Is there a mechanism to pull and push some
exported
>>             resources from one
>>             >>> PuppetDB to another? I guess this could be
done with
>>             the API, but has
>>             >>> anybody tried it, or will it work at all?
>>             > Not provided by us, yet. We''ve had a lot of
discussions
>>             internally
>>             > about this kind of thing. How about you outline your
>>             requirements in
>>             > the form of a feature request here:
>>             >
>>             >
http://projects.puppetlabs.com/projects/puppetdb/issues
>>             >
>>             >> Did anybody try to push data to a PuppetDB by
hand?
>>             The API does not
>>             >> seem to have the PUT or POST methods.
>>             > The client obviously pushes uses POST :-). The
>>             documentation for the
>>             > mechanism is here:
>>             >
>>             >
http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>>             >
>>             > ken.
>>             >
>>
>>             --
>>             You received this message because you are subscribed to
>>             the Google Groups "Puppet Users" group.
>>             To unsubscribe from this group and stop receiving emails
>>             from it, send an email to
>>             puppet-users+unsubscribe@googlegroups.com
>>             <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>             To post to this group, send email to
>>             puppet-users@googlegroups.com
>>             <mailto:puppet-users@googlegroups.com>.
>>             Visit this group at
>>             http://groups.google.com/group/puppet-users.
>>             For more options, visit
>>             https://groups.google.com/groups/opt_out.
>>
>>
>>         -- 
>>         You received this message because you are subscribed to the
>>         Google Groups "Puppet Users" group.
>>         To unsubscribe from this group and stop receiving emails from
>>         it, send an email to
>>         puppet-users+unsubscribe@googlegroups.com
>>         <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>         To post to this group, send email to
>>         puppet-users@googlegroups.com
>>         <mailto:puppet-users@googlegroups.com>.
>>         Visit this group at
http://groups.google.com/group/puppet-users.
>>         For more options, visit
https://groups.google.com/groups/opt_out.
>>
>>
>>     -- 
>>     You received this message because you are subscribed to the
>>     Google Groups "Puppet Users" group.
>>     To unsubscribe from this group and stop receiving emails from it,
>>     send an email to puppet-users+unsubscribe@googlegroups.com
>>     <mailto:puppet-users+unsubscribe@googlegroups.com>.
>>     To post to this group, send email to
>>     puppet-users@googlegroups.com
<mailto:puppet-users@googlegroups.com>.
>>     Visit this group at http://groups.google.com/group/puppet-users.
>>     For more options, visit https://groups.google.com/groups/opt_out.
>
>     -- 
>     You received this message because you are subscribed to the Google
>     Groups "Puppet Users" group.
>     To unsubscribe from this group and stop receiving emails from it,
>     send an email to puppet-users+unsubscribe@googlegroups.com
>     <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>     To post to this group, send email to puppet-users@googlegroups.com
>     <mailto:puppet-users@googlegroups.com>.
>     Visit this group at http://groups.google.com/group/puppet-users.
>     For more options, visit https://groups.google.com/groups/opt_out.
>
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

If I try to add a fictitious resource:

{\"type\": \"Service\",
 \"title\": \"iptables\",
 \"exported\": false,
 \"file\":
\"/etc/puppet/modules/puppetdb/manifests/server.pp\",
 \"line\": 121,
 \"tags\": [\"node\", \"puppetdb\"],
 \"parameters\": {
                  \"ensure\" : \"running\"}
}

I get:

2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command]
[15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying after
attempt 8, due to: org.postgresql.util.PSQLException: No value specified
for parameter 1.


This way of issuing commands is quite nasty...

BR/Pablo


On 10/15/2013 11:00 AM, Pablo Fernandez wrote:
> Ok, so I finally opted for inserting a full node from one DB into
> another, since getting the resources should be quite easy with a GET.
>
> But then I run into a problem when running the replace catalog command
> (with an empty catalog):
>
> curl -vv -G -H "Accept: application/json"
> ''http://localhost:8080/v2/commands'' --data-urlencode
> ''payload={"command":"replace
catalog","version": 2,"payload":"{
> \"metadata\": {\"api_version\": 1}, \"data\":
{\"name\":
> \"puppet09.cscs.ch\",\"version\": \"1111\",
\"edges\": [],
> \"resources\": [] }    }"}''
>
> Which returns ok. The puppetdb.log shows:
>
> 2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
> [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying
> after attempt 8, due to: org.postgresql.util.PSQLException: ERROR:
> syntax error at or near ")"
>   Position: 66
>
> I have looked again and again into the data, and syntax seems to be
> correct.
>
> Anybody tried this? Hints?
>
> Thanks a lot,
> Pablo Fernandez
>
>
>
>
> On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
>> That would really depend on the approach ultimately taken for the
>> multi-master. If the puppet master (or agent in masterless mode)
>> forwards the same fact/report/node/etc data to multiple puppetdb
>> instances then yes the scale breaks horribly and does not meet part
>> of this request.  If there is some kind of store-and-forward
>> mechanism in puppetdb itself, it would be possible to extend the
>> forward to say ''only if it matches these rules''. 
This could
>> definitely be useful in a variety of ways beyond simple HA purposes.
>>
>>
>>
>> On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
>> <pablo.fernandez@cscs.ch <mailto:pablo.fernandez@cscs.ch>>
wrote:
>>
>>     Yes, I saw that... this may be a solution for replicating full
>>     PuppetDBs (if you tackle the backlog during a long shortage), but
>>     that does not scale very well if you have 5-10 PuppetDBs all
>>     being replicated to a single, bigger one.
>>
>>     Besides, it would not cover cases when you want to have only some
>>     nodes replicated, or even only some resources.
>>
>>
>>
>>     On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
>>>     It sounds like a shade of #19321 -
>>>     https://projects.puppetlabs.com/issues/19321
>>>
>>>
>>>     On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble
<wolf@wolfspyre.com
>>>     <mailto:wolf@wolfspyre.com>> wrote:
>>>
>>>         I''d be interested in accomplishing a similar
thing. I want
>>>         to have a central reporting puppetdb which I can (outside
of
>>>         puppet) query for multisite information without tying
>>>         isolated environments together via a single postgres db, or
>>>         using multi-master replication. 
>>>
>>>         my needs do not involve making each puppet site aware of
>>>         each other''s data, but it would be nice to have
one db which
>>>         has all the data for my digging pleasure.
>>>
>>>         from my understanding this is not currently supported.
>>>
>>>         I was thinking something could be done to ask puppetdb at
>>>         site A (what updates have you had in the past N minutes/
>>>         since the last time I checked in with you/ since
>>>         timestamp(N)), and then iterate through the collected data
>>>         and feed it into a reporting puppetdb instance, however
I''ve
>>>         not gone beyond thinking about it.
>>>
>>>
>>>
>>>
>>>
>>>         On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
>>>         <pablo.fernandez@cscs.ch
<mailto:pablo.fernandez@cscs.ch>>
>>>         wrote:
>>>
>>>             Thanks Ken,
>>>
>>>             I took a look at the Command API and seems to be quite
>>>             interesting, but
>>>             it triggers a few more questions.
>>>
>>>             As a reminder (also for others), what I wanted to do
was
>>>             to add exported
>>>             resources from Node_A (managed by PuppetMaster_A that
>>>             has its own
>>>             PuppetDB_A) into a different PuppetDB_B, where Node_A
is
>>>             unknown, in
>>>             order for other Nodes_B (managed by PuppetMaster_B) be
>>>             aware and
>>>             possibly realize those exported resources.
>>>
>>>             Taking a look the Commands API seem to capabilities to
>>>             change a whole
>>>             catalog, or facts. Do they need to be existing
catalogs,
>>>             or could I add
>>>             new ones?
>>>             I am thinking: If I want to add Node_A''s
exported
>>>             resource(s) to
>>>             PuppetDB_B, I could "artificially" add its
whole catalog to
>>>             PuppetDB_B... but the Command API is "replace
catalog",
>>>             no "add
>>>             catalog". Would it be rejected? How does the first
>>>             catalog of a node get
>>>             introduced into the DB? And moreover, how do I query a
>>>             whole catalog?
>>>
>>>             Another possibility would be to include the resource
>>>             inside an existing
>>>             Node_B catalog on the PuppetDB_B (after all, exported
>>>             resources don''t
>>>             really mater what node do they belong, as long as they
>>>             have the right
>>>             attributes, right?), but the Catalog wire format seems
>>>             to require the
>>>             whole set of resources, so, if I define the resources
of
>>>             Node_B with
>>>             some exported resources from Node_A, I believe I would
>>>             lose the original
>>>             Node_B''s resources. The only possibility would
be to
>>>             query Node_B full
>>>             catalog, and *add also* those resources from Node_A
>>>             before sending them
>>>             to the DB... and this seems to me like a bit dirty :)
>>>
>>>             Am I missing anything? Or is there another way to
>>>             implemen what I need
>>>             with Puppet 3.2?
>>>
>>>             Thanks again,
>>>             BR/Pablo
>>>
>>>
>>>             On 10/10/2013 09:16 AM, Ken Barber wrote:
>>>             >>> Is there a mechanism to pull and push some
exported
>>>             resources from one
>>>             >>> PuppetDB to another? I guess this could be
done with
>>>             the API, but has
>>>             >>> anybody tried it, or will it work at all?
>>>             > Not provided by us, yet. We''ve had a lot
of
>>>             discussions internally
>>>             > about this kind of thing. How about you outline
your
>>>             requirements in
>>>             > the form of a feature request here:
>>>             >
>>>             >
http://projects.puppetlabs.com/projects/puppetdb/issues
>>>             >
>>>             >> Did anybody try to push data to a PuppetDB by
hand?
>>>             The API does not
>>>             >> seem to have the PUT or POST methods.
>>>             > The client obviously pushes uses POST :-). The
>>>             documentation for the
>>>             > mechanism is here:
>>>             >
>>>             >
http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>>>             >
>>>             > ken.
>>>             >
>>>
>>>             --
>>>             You received this message because you are subscribed to
>>>             the Google Groups "Puppet Users" group.
>>>             To unsubscribe from this group and stop receiving
emails
>>>             from it, send an email to
>>>             puppet-users+unsubscribe@googlegroups.com
>>>            
<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>>             To post to this group, send email to
>>>             puppet-users@googlegroups.com
>>>             <mailto:puppet-users@googlegroups.com>.
>>>             Visit this group at
>>>             http://groups.google.com/group/puppet-users.
>>>             For more options, visit
>>>             https://groups.google.com/groups/opt_out.
>>>
>>>
>>>         -- 
>>>         You received this message because you are subscribed to the
>>>         Google Groups "Puppet Users" group.
>>>         To unsubscribe from this group and stop receiving emails
>>>         from it, send an email to
>>>         puppet-users+unsubscribe@googlegroups.com
>>>         <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>>         To post to this group, send email to
>>>         puppet-users@googlegroups.com
>>>         <mailto:puppet-users@googlegroups.com>.
>>>         Visit this group at
http://groups.google.com/group/puppet-users.
>>>         For more options, visit
>>>         https://groups.google.com/groups/opt_out.
>>>
>>>
>>>     -- 
>>>     You received this message because you are subscribed to the
>>>     Google Groups "Puppet Users" group.
>>>     To unsubscribe from this group and stop receiving emails from
>>>     it, send an email to puppet-users+unsubscribe@googlegroups.com
>>>     <mailto:puppet-users+unsubscribe@googlegroups.com>.
>>>     To post to this group, send email to
>>>     puppet-users@googlegroups.com
>>>     <mailto:puppet-users@googlegroups.com>.
>>>     Visit this group at
http://groups.google.com/group/puppet-users.
>>>     For more options, visit
https://groups.google.com/groups/opt_out.
>>
>>     -- 
>>     You received this message because you are subscribed to the
>>     Google Groups "Puppet Users" group.
>>     To unsubscribe from this group and stop receiving emails from it,
>>     send an email to puppet-users+unsubscribe@googlegroups.com
>>     <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>     To post to this group, send email to
>>     puppet-users@googlegroups.com
<mailto:puppet-users@googlegroups.com>.
>>     Visit this group at http://groups.google.com/group/puppet-users.
>>     For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to puppet-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users.
>> For more options, visit https://groups.google.com/groups/opt_out.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Hehe, sorry for the auto-reply... but I made it work, and I wanted to
post it for reference.

The key was to add also the edges: it can''t be an empty list.

For the record, doing:

# curl -vv -G -H "Accept: application/json"
''http://localhost:8080/v2/commands'' --data-urlencode
''payload={
  "command": "replace catalog",
  "version": 2,
  "payload": "{
    \"metadata\": {
      \"api_version\": 1
    },
    \"data\":   {
      \"name\":      \"myhost.blabla.bla\",
      \"version\":   \"1111\",
      \"edges\":     [
        {\"source\": {\"type\": \"Service\",
\"title\": \"iptables\"},
         \"target\": {\"type\": \"Service\",
\"title\": \"iptables\"},
         \"relationship\": \"before\"}
      ],
      \"resources\": [
        {\"type\": \"Service\",
         \"title\": \"iptables\",
         \"exported\": false,
         \"file\":
\"/etc/puppet/modules/puppetdb/manifests/server.pp\",
         \"line\": 121,
         \"tags\": [\"node\", \"puppetdb\"],
         \"parameters\": {\"hasrestart\": true,
\"ensure\" : \"running\"}
        }
      ]
    }
  }"
}''

Works like a charm, and add a new host into the PuppetDB out of the blue.
Next: automatically serialize those escaped quotes and try to see if it
exported resources work as they should.

Thanks, and sorry again for the noise,
BR/Pablo



On 10/15/2013 11:25 AM, Pablo Fernandez wrote:
> If I try to add a fictitious resource:
>
> {\"type\": \"Service\",
>  \"title\": \"iptables\",
>  \"exported\": false,
>  \"file\":
\"/etc/puppet/modules/puppetdb/manifests/server.pp\",
>  \"line\": 121,
>  \"tags\": [\"node\", \"puppetdb\"],
>  \"parameters\": {
>                   \"ensure\" : \"running\"}
> }
>
> I get:
>
> 2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command]
> [15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying
> after attempt 8, due to: org.postgresql.util.PSQLException: No value
> specified for parameter 1.
>
>
> This way of issuing commands is quite nasty...
>
> BR/Pablo
>
>
> On 10/15/2013 11:00 AM, Pablo Fernandez wrote:
>> Ok, so I finally opted for inserting a full node from one DB into
>> another, since getting the resources should be quite easy with a GET.
>>
>> But then I run into a problem when running the replace catalog
>> command (with an empty catalog):
>>
>> curl -vv -G -H "Accept: application/json"
>> ''http://localhost:8080/v2/commands'' --data-urlencode
>> ''payload={"command":"replace
catalog","version": 2,"payload":"{
>> \"metadata\": {\"api_version\": 1},
\"data\":   {\"name\":
>> \"puppet09.cscs.ch\",\"version\":
\"1111\", \"edges\": [],
>> \"resources\": [] }    }"}''
>>
>> Which returns ok. The puppetdb.log shows:
>>
>> 2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
>> [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying
>> after attempt 8, due to: org.postgresql.util.PSQLException: ERROR:
>> syntax error at or near ")"
>>   Position: 66
>>
>> I have looked again and again into the data, and syntax seems to be
>> correct.
>>
>> Anybody tried this? Hints?
>>
>> Thanks a lot,
>> Pablo Fernandez
>>
>>
>>
>>
>> On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
>>> That would really depend on the approach ultimately taken for the
>>> multi-master. If the puppet master (or agent in masterless mode)
>>> forwards the same fact/report/node/etc data to multiple puppetdb
>>> instances then yes the scale breaks horribly and does not meet part
>>> of this request.  If there is some kind of store-and-forward
>>> mechanism in puppetdb itself, it would be possible to extend the
>>> forward to say ''only if it matches these rules''. 
This could
>>> definitely be useful in a variety of ways beyond simple HA
purposes.
>>>
>>>
>>>
>>> On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
>>> <pablo.fernandez@cscs.ch
<mailto:pablo.fernandez@cscs.ch>> wrote:
>>>
>>>     Yes, I saw that... this may be a solution for replicating full
>>>     PuppetDBs (if you tackle the backlog during a long shortage),
>>>     but that does not scale very well if you have 5-10 PuppetDBs
all
>>>     being replicated to a single, bigger one.
>>>
>>>     Besides, it would not cover cases when you want to have only
>>>     some nodes replicated, or even only some resources.
>>>
>>>
>>>
>>>     On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
>>>>     It sounds like a shade of #19321 -
>>>>     https://projects.puppetlabs.com/issues/19321
>>>>
>>>>
>>>>     On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble
>>>>     <wolf@wolfspyre.com
<mailto:wolf@wolfspyre.com>> wrote:
>>>>
>>>>         I''d be interested in accomplishing a similar
thing. I want
>>>>         to have a central reporting puppetdb which I can
(outside
>>>>         of puppet) query for multisite information without
tying
>>>>         isolated environments together via a single postgres
db, or
>>>>         using multi-master replication. 
>>>>
>>>>         my needs do not involve making each puppet site aware
of
>>>>         each other''s data, but it would be nice to
have one db
>>>>         which has all the data for my digging pleasure.
>>>>
>>>>         from my understanding this is not currently supported.
>>>>
>>>>         I was thinking something could be done to ask puppetdb
at
>>>>         site A (what updates have you had in the past N
minutes/
>>>>         since the last time I checked in with you/ since
>>>>         timestamp(N)), and then iterate through the collected
data
>>>>         and feed it into a reporting puppetdb instance, however
>>>>         I''ve not gone beyond thinking about it.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>         On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
>>>>         <pablo.fernandez@cscs.ch
<mailto:pablo.fernandez@cscs.ch>>
>>>>         wrote:
>>>>
>>>>             Thanks Ken,
>>>>
>>>>             I took a look at the Command API and seems to be
quite
>>>>             interesting, but
>>>>             it triggers a few more questions.
>>>>
>>>>             As a reminder (also for others), what I wanted to
do
>>>>             was to add exported
>>>>             resources from Node_A (managed by PuppetMaster_A
that
>>>>             has its own
>>>>             PuppetDB_A) into a different PuppetDB_B, where
Node_A
>>>>             is unknown, in
>>>>             order for other Nodes_B (managed by PuppetMaster_B)
be
>>>>             aware and
>>>>             possibly realize those exported resources.
>>>>
>>>>             Taking a look the Commands API seem to capabilities
to
>>>>             change a whole
>>>>             catalog, or facts. Do they need to be existing
>>>>             catalogs, or could I add
>>>>             new ones?
>>>>             I am thinking: If I want to add Node_A''s
exported
>>>>             resource(s) to
>>>>             PuppetDB_B, I could "artificially" add
its whole catalog to
>>>>             PuppetDB_B... but the Command API is "replace
catalog",
>>>>             no "add
>>>>             catalog". Would it be rejected? How does the
first
>>>>             catalog of a node get
>>>>             introduced into the DB? And moreover, how do I
query a
>>>>             whole catalog?
>>>>
>>>>             Another possibility would be to include the
resource
>>>>             inside an existing
>>>>             Node_B catalog on the PuppetDB_B (after all,
exported
>>>>             resources don''t
>>>>             really mater what node do they belong, as long as
they
>>>>             have the right
>>>>             attributes, right?), but the Catalog wire format
seems
>>>>             to require the
>>>>             whole set of resources, so, if I define the
resources
>>>>             of Node_B with
>>>>             some exported resources from Node_A, I believe I
would
>>>>             lose the original
>>>>             Node_B''s resources. The only possibility
would be to
>>>>             query Node_B full
>>>>             catalog, and *add also* those resources from Node_A
>>>>             before sending them
>>>>             to the DB... and this seems to me like a bit dirty
:)
>>>>
>>>>             Am I missing anything? Or is there another way to
>>>>             implemen what I need
>>>>             with Puppet 3.2?
>>>>
>>>>             Thanks again,
>>>>             BR/Pablo
>>>>
>>>>
>>>>             On 10/10/2013 09:16 AM, Ken Barber wrote:
>>>>             >>> Is there a mechanism to pull and push
some exported
>>>>             resources from one
>>>>             >>> PuppetDB to another? I guess this
could be done
>>>>             with the API, but has
>>>>             >>> anybody tried it, or will it work at
all?
>>>>             > Not provided by us, yet. We''ve had a
lot of
>>>>             discussions internally
>>>>             > about this kind of thing. How about you
outline your
>>>>             requirements in
>>>>             > the form of a feature request here:
>>>>             >
>>>>             >
http://projects.puppetlabs.com/projects/puppetdb/issues
>>>>             >
>>>>             >> Did anybody try to push data to a PuppetDB
by hand?
>>>>             The API does not
>>>>             >> seem to have the PUT or POST methods.
>>>>             > The client obviously pushes uses POST :-). The
>>>>             documentation for the
>>>>             > mechanism is here:
>>>>             >
>>>>             >
http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>>>>             >
>>>>             > ken.
>>>>             >
>>>>
>>>>             --
>>>>             You received this message because you are
subscribed to
>>>>             the Google Groups "Puppet Users" group.
>>>>             To unsubscribe from this group and stop receiving
>>>>             emails from it, send an email to
>>>>             puppet-users+unsubscribe@googlegroups.com
>>>>            
<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>>>             To post to this group, send email to
>>>>             puppet-users@googlegroups.com
>>>>             <mailto:puppet-users@googlegroups.com>.
>>>>             Visit this group at
>>>>             http://groups.google.com/group/puppet-users.
>>>>             For more options, visit
>>>>             https://groups.google.com/groups/opt_out.
>>>>
>>>>
>>>>         -- 
>>>>         You received this message because you are subscribed to
the
>>>>         Google Groups "Puppet Users" group.
>>>>         To unsubscribe from this group and stop receiving
emails
>>>>         from it, send an email to
>>>>         puppet-users+unsubscribe@googlegroups.com
>>>>        
<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>>>         To post to this group, send email to
>>>>         puppet-users@googlegroups.com
>>>>         <mailto:puppet-users@googlegroups.com>.
>>>>         Visit this group at
>>>>         http://groups.google.com/group/puppet-users.
>>>>         For more options, visit
>>>>         https://groups.google.com/groups/opt_out.
>>>>
>>>>
>>>>     -- 
>>>>     You received this message because you are subscribed to the
>>>>     Google Groups "Puppet Users" group.
>>>>     To unsubscribe from this group and stop receiving emails
from
>>>>     it, send an email to
puppet-users+unsubscribe@googlegroups.com
>>>>     <mailto:puppet-users+unsubscribe@googlegroups.com>.
>>>>     To post to this group, send email to
>>>>     puppet-users@googlegroups.com
>>>>     <mailto:puppet-users@googlegroups.com>.
>>>>     Visit this group at
http://groups.google.com/group/puppet-users.
>>>>     For more options, visit
https://groups.google.com/groups/opt_out.
>>>
>>>     -- 
>>>     You received this message because you are subscribed to the
>>>     Google Groups "Puppet Users" group.
>>>     To unsubscribe from this group and stop receiving emails from
>>>     it, send an email to puppet-users+unsubscribe@googlegroups.com
>>>     <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
>>>     To post to this group, send email to
>>>     puppet-users@googlegroups.com
>>>     <mailto:puppet-users@googlegroups.com>.
>>>     Visit this group at
http://groups.google.com/group/puppet-users.
>>>     For more options, visit
https://groups.google.com/groups/opt_out.
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to puppet-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to puppet-users@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/puppet-users.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to puppet-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users.
>> For more options, visit https://groups.google.com/groups/opt_out.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Tue, Oct 15, 2013 at 6:51 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch>wrote:
>  Hehe, sorry for the auto-reply... but I made it work, and I wanted to
> post it for reference.
>
> The key was to add also the edges: it can''t be an empty list.
>
> For the record, doing:
>
> # curl -vv -G -H "Accept: application/json" ''
> http://localhost:8080/v2/commands'' --data-urlencode
''payload={
>
>   "command": "replace catalog",
>   "version": 2,
>   "payload": "{
>     \"metadata\": {
>       \"api_version\": 1
>     },
>     \"data\":   {
>       \"name\":      \"myhost.blabla.bla\",
>       \"version\":   \"1111\",
>       \"edges\":     [
>         {\"source\": {\"type\": \"Service\",
\"title\": \"iptables\"},
>          \"target\": {\"type\": \"Service\",
\"title\": \"iptables\"},
>          \"relationship\": \"before\"}
>       ],
>       \"resources\": [
>
>         {\"type\": \"Service\",
>          \"title\": \"iptables\",
>          \"exported\": false,
>          \"file\":
\"/etc/puppet/modules/puppetdb/manifests/server.pp\",
>          \"line\": 121,
>          \"tags\": [\"node\", \"puppetdb\"],
>          \"parameters\": {\"hasrestart\": true,
\"ensure\" : \"running\"}
>         }
>       ]
>     }
>   }"
> }''
>
> Works like a charm, and add a new host into the PuppetDB out of the blue.
> Next: automatically serialize those escaped quotes and try to see if it
> exported resources work as they should.
>
> Thanks, and sorry again for the noise,
> BR/Pablo
>
Hi Pablo, I''m glad you got this working and I apologize it took so
long. A
few comments:

1) You don''t need to formulate the "payload" as a
pre-serialized JSON
string (with all the requisite escaping and whatnot) anymore. As of
"replace catalog" version 2, you can just inline the raw JSON object
itself:

"payload": {"data": ..., "metadata": ..., ...}

The entire command is a single JSON object that way, and the payload is
just a normal value in that hash.

2) You are correct that the edges key is required, and should contain every
relationship specified in the supplied catalog:
http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html#main-data-type-catalog

I do agree that the error message isn''t helpful at all, though, and we
should fix that. It''s likely that we aren''t doing proper
validation on that
and letting things get to the storage code, which throws an opaque,
low-level error that is difficult to debug. That said, as there aren''t
any
relationships specified in your above set of resource (no require/subscribe
params, for example) I''m curious why the empty list of edges fails.
That
smells like a bug to me; would you mind filing a ticket against PuppetDB
with the above command included? We can take a look and see what''s up.

3) With curl, you can supply an argument of "--data-urlencode
payload@mycatalog.json". Then you can just put the payload in a
mycatalog.json file, and curl should automatically load the content and
urlencode it for you. See
http://curl.haxx.se/docs/manpage.html#--data-urlencode

deepak


>
>
>
> On 10/15/2013 11:25 AM, Pablo Fernandez wrote:
>
> If I try to add a fictitious resource:
>
> {\"type\": \"Service\",
>  \"title\": \"iptables\",
>  \"exported\": false,
>  \"file\":
\"/etc/puppet/modules/puppetdb/manifests/server.pp\",
>  \"line\": 121,
>  \"tags\": [\"node\", \"puppetdb\"],
>  \"parameters\": {
>                   \"ensure\" : \"running\"}
> }
>
> I get:
>
> 2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command]
> [15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying after
> attempt 8, due to: org.postgresql.util.PSQLException: No value specified
> for parameter 1.
>
>
> This way of issuing commands is quite nasty...
>
> BR/Pablo
>
>
> On 10/15/2013 11:00 AM, Pablo Fernandez wrote:
>
> Ok, so I finally opted for inserting a full node from one DB into another,
> since getting the resources should be quite easy with a GET.
>
> But then I run into a problem when running the replace catalog command
> (with an empty catalog):
>
> curl -vv -G -H "Accept: application/json" ''
> http://localhost:8080/v2/commands'' --data-urlencode
> ''payload={"command":"replace
catalog","version": 2,"payload":"{
> \"metadata\": {\"api_version\": 1}, \"data\":
{\"name\": \"
> puppet09.cscs.ch\",\"version\": \"1111\",
\"edges\": [], \"resources\":
> [] }    }"}''
>
> Which returns ok. The puppetdb.log shows:
>
> 2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
> [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying after
> attempt 8, due to: org.postgresql.util.PSQLException: ERROR: syntax error
> at or near ")"
>   Position: 66
>
> I have looked again and again into the data, and syntax seems to be
> correct.
>
> Anybody tried this? Hints?
>
> Thanks a lot,
> Pablo Fernandez
>
>
>
>
> On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
>
> That would really depend on the approach ultimately taken for the
> multi-master. If the puppet master (or agent in masterless mode) forwards
> the same fact/report/node/etc data to multiple puppetdb instances then yes
> the scale breaks horribly and does not meet part of this request.  If there
> is some kind of store-and-forward mechanism in puppetdb itself, it would be
> possible to extend the forward to say ''only if it matches these
rules''.
>  This could definitely be useful in a variety of ways beyond simple HA
> purposes.
>
>
>
> On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch
> > wrote:
>
>>  Yes, I saw that... this may be a solution for replicating full
PuppetDBs
>> (if you tackle the backlog during a long shortage), but that does not
scale
>> very well if you have 5-10 PuppetDBs all being replicated to a single,
>> bigger one.
>>
>> Besides, it would not cover cases when you want to have only some nodes
>> replicated, or even only some resources.
>>
>>
>>
>> On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
>>
>> It sounds like a shade of #19321 -
>> https://projects.puppetlabs.com/issues/19321
>>
>>
>> On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com>
wrote:
>>
>>> I''d be interested in accomplishing a similar thing. I want
to have a
>>> central reporting puppetdb which I can (outside of puppet) query
for
>>> multisite information without tying isolated environments together
via a
>>> single postgres db, or using multi-master replication.
>>>
>>>  my needs do not involve making each puppet site aware of each
other''s
>>> data, but it would be nice to have one db which has all the data
for my
>>> digging pleasure.
>>>
>>>  from my understanding this is not currently supported.
>>>
>>>  I was thinking something could be done to ask puppetdb at site A
(what
>>> updates have you had in the past N minutes/ since the last time I
checked
>>> in with you/ since timestamp(N)), and then iterate through the
collected
>>> data and feed it into a reporting puppetdb instance, however
I''ve not gone
>>> beyond thinking about it.
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez <
>>> pablo.fernandez@cscs.ch> wrote:
>>>
>>>> Thanks Ken,
>>>>
>>>> I took a look at the Command API and seems to be quite
interesting, but
>>>> it triggers a few more questions.
>>>>
>>>> As a reminder (also for others), what I wanted to do was to add
exported
>>>> resources from Node_A (managed by PuppetMaster_A that has its
own
>>>> PuppetDB_A) into a different PuppetDB_B, where Node_A is
unknown, in
>>>> order for other Nodes_B (managed by PuppetMaster_B) be aware
and
>>>> possibly realize those exported resources.
>>>>
>>>> Taking a look the Commands API seem to capabilities to change a
whole
>>>> catalog, or facts. Do they need to be existing catalogs, or
could I add
>>>> new ones?
>>>> I am thinking: If I want to add Node_A''s exported
resource(s) to
>>>> PuppetDB_B, I could "artificially" add its whole
catalog to
>>>> PuppetDB_B... but the Command API is "replace
catalog", no "add
>>>> catalog". Would it be rejected? How does the first catalog
of a node get
>>>> introduced into the DB? And moreover, how do I query a whole
catalog?
>>>>
>>>> Another possibility would be to include the resource inside an
existing
>>>> Node_B catalog on the PuppetDB_B (after all, exported resources
don''t
>>>> really mater what node do they belong, as long as they have the
right
>>>> attributes, right?), but the Catalog wire format seems to
require the
>>>> whole set of resources, so, if I define the resources of Node_B
with
>>>> some exported resources from Node_A, I believe I would lose the
original
>>>> Node_B''s resources. The only possibility would be to
query Node_B full
>>>> catalog, and *add also* those resources from Node_A before
sending them
>>>> to the DB... and this seems to me like a bit dirty :)
>>>>
>>>> Am I missing anything? Or is there another way to implemen what
I need
>>>> with Puppet 3.2?
>>>>
>>>> Thanks again,
>>>> BR/Pablo
>>>>
>>>>
>>>> On 10/10/2013 09:16 AM, Ken Barber wrote:
>>>> >>> Is there a mechanism to pull and push some
exported resources from
>>>> one
>>>> >>> PuppetDB to another? I guess this could be done
with the API, but
>>>> has
>>>> >>> anybody tried it, or will it work at all?
>>>> > Not provided by us, yet. We''ve had a lot of
discussions internally
>>>> > about this kind of thing. How about you outline your
requirements in
>>>> > the form of a feature request here:
>>>> >
>>>> > http://projects.puppetlabs.com/projects/puppetdb/issues
>>>> >
>>>> >> Did anybody try to push data to a PuppetDB by hand?
The API does not
>>>> >> seem to have the PUT or POST methods.
>>>> > The client obviously pushes uses POST :-). The
documentation for the
>>>> > mechanism is here:
>>>> >
>>>> > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
>>>> >
>>>> > ken.
>>>> >
>>>>
>>>> --
>>>> You received this message because you are subscribed to the
Google
>>>> Groups "Puppet Users" group.
>>>> To unsubscribe from this group and stop receiving emails from
it, send
>>>> an email to puppet-users+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to
puppet-users@googlegroups.com.
>>>> Visit this group at
http://groups.google.com/group/puppet-users.
>>>> For more options, visit
https://groups.google.com/groups/opt_out.
>>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
send
>>> an email to puppet-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to puppet-users@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/puppet-users.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
an
>> email to puppet-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>    --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
an
>> email to puppet-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, Oct 10, 2013 at 6:12 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch>wrote:
> Thanks Ken,
>
> I took a look at the Command API and seems to be quite interesting, but
> it triggers a few more questions.
>
> As a reminder (also for others), what I wanted to do was to add exported
> resources from Node_A (managed by PuppetMaster_A that has its own
> PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
> order for other Nodes_B (managed by PuppetMaster_B) be aware and
> possibly realize those exported resources.
>
> Taking a look the Commands API seem to capabilities to change a whole
> catalog, or facts. Do they need to be existing catalogs, or could I add
> new ones?
> I am thinking: If I want to add Node_A''s exported resource(s) to
> PuppetDB_B, I could "artificially" add its whole catalog to
> PuppetDB_B... but the Command API is "replace catalog", no
"add
> catalog". Would it be rejected? How does the first catalog of a node
get
> introduced into the DB? And moreover, how do I query a whole catalog?
>
Lots of great questions here! :)

* You can certainly add the whole catalog for a node to a secondary PuppetDB

* The command is called "replace catalog" to indicate that
it''s, well, a
replacement operation. So if you already have a catalog for node
foo.comstored, issuing a "replace catalog" with a new catalog for
foo.com will cause PuppetDB to delete the old catalog and store the new one
for foo.com. It does this atomically, of course. This is "normal"
operation
for PuppetDB, as it only stores a single catalog for a given node.

* A "replace catalog" command won''t be rejected if
there''s already a
catalog for that node; it''ll just tell PuppetDB to store new data in
place
of the old data. In fact, this is how PuppetDB works with puppetmasters
today...an agent requests a catalog from the master, the master compiles it
for the agent, and the master issues a "replace catalog" command to
puppetdb accordingly.

* There is an endpoint to get the catalog for a node:
/v3/catalogs/foo.com. The wire format of the result should be the
same, documented catalog wire
format as during insertion:
http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html.
I do believe, though, that we overlooked adding documentation for this
new endpoint...apologies! A pull req adding those docs to the
documentation/ directory of puppetdb would be much appreciated. :)

>
> Another possibility would be to include the resource inside an existing
> Node_B catalog on the PuppetDB_B (after all, exported resources
don''t
> really mater what node do they belong, as long as they have the right
> attributes, right?), but the Catalog wire format seems to require the
> whole set of resources, so, if I define the resources of Node_B with
> some exported resources from Node_A, I believe I would lose the original
> Node_B''s resources. The only possibility would be to query Node_B
full
> catalog, and *add also* those resources from Node_A before sending them
> to the DB... and this seems to me like a bit dirty :)
>
Yes, you''d have to get the catalog for a node, modify it, and resubmit
it.
Remember that command submission is asynchronous, commands can be retried,
etc. If you have a command of the form "add some resources to the currently
stored catalog", the definition of "currently stored"
isn''t a static thing.
It''s thus subject to race conditions and other confusion. By requiring
an
entire catalog be submitted, and by making it clear through the name of the
command itself ("replace catalog") a large class of problems go away
while
still keeping throughput high and retry-handling manageable.

I''m not opposed to the idea of a "merge" type of command for
catalogs, of
course...but I think it''s something that should be approached
cautiously,
with a good understanding of how it will impact (and be impacted by) the
rest of the system. Maybe doing something like a command that merged a set
of resources into a known, specific catalog (identified by a stable hash or
id) would work?

>
> Am I missing anything? Or is there another way to implemen what I need
> with Puppet 3.2?
>
> Thanks again,
> BR/Pablo
>
>
> On 10/10/2013 09:16 AM, Ken Barber wrote:
> >>> Is there a mechanism to pull and push some exported resources
from one
> >>> PuppetDB to another? I guess this could be done with the API,
but has
> >>> anybody tried it, or will it work at all?
> > Not provided by us, yet. We''ve had a lot of discussions
internally
> > about this kind of thing. How about you outline your requirements in
> > the form of a feature request here:
> >
> > http://projects.puppetlabs.com/projects/puppetdb/issues
> >
> >> Did anybody try to push data to a PuppetDB by hand? The API does
not
> >> seem to have the PUT or POST methods.
> > The client obviously pushes uses POST :-). The documentation for the
> > mechanism is here:
> >
> > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html
> >
> > ken.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Dear Deepak,


On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:
>
> 1) You don''t need to formulate the "payload" as a
pre-serialized JSON
> string (with all the requisite escaping and whatnot) anymore. As of
> "replace catalog" version 2, you can just inline the raw JSON
object
> itself:
>
> "payload": {"data": ..., "metadata": ...,
...}
>
> The entire command is a single JSON object that way, and the payload
> is just a normal value in that hash
> 3) With curl, you can supply an argument of "--data-urlencode
> payload@mycatalog.json". Then you can just put the payload in a
> mycatalog.json file, and curl should automatically load the content
> and urlencode it for you.
> See http://curl.haxx.se/docs/manpage.html#--data-urlencode

This RAW version is quite interesting (escaping part of the thing is a
bit nasty), I will play with it :)

I have a question, though... if you can insert the RAW payload... what
happens with the command name?
I guess you mean:
"payload": { "command": "replace catalog", 
"version": 2,  "payload":
{"data": ..., "metadata": ..., ...} }
Without any escape sequence at all, right?

And then send it with the --data-urlencode


>
>  That said, as there aren''t any relationships specified in your
above
> set of resource (no require/subscribe params, for example) I''m
curious
> why the empty list of edges fails. That smells like a bug to me; would
> you mind filing a ticket against PuppetDB with the above command
> included? We can take a look and see what''s up.
Submitted!


Thanks a lot,
Pablo Fernandez

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks again for your answers, they are quite useful!


On 10/15/2013 09:39 PM, Deepak Giridharagopal wrote:
> * You can certainly add the whole catalog for a node to a secondary
> PuppetDB
I actually think this is the way to go, at least in my case.
>
> * There is an endpoint to get the catalog for a node:
> /v3/catalogs/foo.com <http://foo.com> . The wire format of the result
> should be the same, documented catalog wire format as during
> insertion:
http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html
> . I do believe, though, that we overlooked adding documentation for
> this new endpoint...apologies! A pull req adding those docs to the
> documentation/ directory of puppetdb would be much appreciated. :)
This is quite useful indeed! It even has the metadata part embedded.

It is quite strange that the wire format differs from the GET output (
/v3/nodes/<node>/resources ) in the sense that the replace catalog
command requires resources to have the "sourcefile" (from the GET)
named
as "file", and "sourceline" as "line". The rest of
the keys have the
same name and are compatible... it''s a pity this small asymmetry. I
submitted a refactor request for this.


Thanks again,
Pablo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

I tried to upload a full catalog with the command:
# curl -vv -G -H "Accept: application/json"
''http://localhost:8080/v2/commands'' --data-urlencode
payload@catalog.json

Where catalog.json would start with:
{"version": 2, "command": "replace catalog",
"payload": {"data":
{"version ...etc...

But I get a:
HTTP/1.1 413 FULL head

This seems to be the Jetty server not accepting big POST. This one is
112 KB, which is not really big for a catalog.
I probably hit a configuration issue, but I don''t see the option in the
[jetty] section... bug maybe?

Thanks!
BR/Pablo


On 10/16/2013 09:16 AM, Pablo Fernandez wrote:
> Dear Deepak,
>
>
> On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:
>>
>> 1) You don''t need to formulate the "payload" as a
pre-serialized JSON
>> string (with all the requisite escaping and whatnot) anymore. As of
>> "replace catalog" version 2, you can just inline the raw JSON
object
>> itself:
>>
>> "payload": {"data": ..., "metadata": ...,
...}
>>
>> The entire command is a single JSON object that way, and the payload
>> is just a normal value in that hash
>> 3) With curl, you can supply an argument of "--data-urlencode
>> payload@mycatalog.json". Then you can just put the payload in a
>> mycatalog.json file, and curl should automatically load the content
>> and urlencode it for you.
>> See http://curl.haxx.se/docs/manpage.html#--data-urlencode
>
>
> This RAW version is quite interesting (escaping part of the thing is a
> bit nasty), I will play with it :)
>
> I have a question, though... if you can insert the RAW payload... what
> happens with the command name?
> I guess you mean:
> "payload": { "command": "replace catalog", 
"version": 2,  "payload":
> {"data": ..., "metadata": ..., ...} }
> Without any escape sequence at all, right?
>
> And then send it with the --data-urlencode
>
>
>
>>
>>  That said, as there aren''t any relationships specified in
your above
>> set of resource (no require/subscribe params, for example) I''m
>> curious why the empty list of edges fails. That smells like a bug to
>> me; would you mind filing a ticket against PuppetDB with the above
>> command included? We can take a look and see what''s up.
>
> Submitted!
>
>
> Thanks a lot,
> Pablo Fernandez
> -- 
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, Oct 17, 2013 at 9:09 AM, Pablo Fernandez
<pablo.fernandez@cscs.ch>wrote:
>  Hi,
>
> I tried to upload a full catalog with the command:
> # curl -vv -G -H "Accept: application/json" ''
> http://localhost:8080/v2/commands'' --data-urlencode
payload@catalog.json
>
> Where catalog.json would start with:
> {"version": 2, "command": "replace catalog",
"payload": {"data": {"version
> ...etc...
>
> But I get a:
> HTTP/1.1 413 FULL head
>
> This seems to be the Jetty server not accepting big POST. This one is 112
> KB, which is not really big for a catalog.
> I probably hit a configuration issue, but I don''t see the option
in the
> [jetty] section... bug maybe?
>
(resending as I forgot to reply to the list)

The "-G" command you''re giving to curl forces it to use a GET
for the
data...can you try after removing that option? That should default it to
POST.

deepak


> Thanks!
> BR/Pablo
>
>
>
> On 10/16/2013 09:16 AM, Pablo Fernandez wrote:
>
> Dear Deepak,
>
>
> On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:
>
>
>  1) You don''t need to formulate the "payload" as a
pre-serialized JSON
> string (with all the requisite escaping and whatnot) anymore. As of
> "replace catalog" version 2, you can just inline the raw JSON
object
> itself:
>
>  "payload": {"data": ..., "metadata": ...,
...}
>
>  The entire command is a single JSON object that way, and the payload is
> just a normal value in that hash
>
>   3) With curl, you can supply an argument of "--data-urlencode
> payload@mycatalog.json". Then you can just put the payload in a
> mycatalog.json file, and curl should automatically load the content and
> urlencode it for you. See
> http://curl.haxx.se/docs/manpage.html#--data-urlencode
>
>
>
> This RAW version is quite interesting (escaping part of the thing is a bit
> nasty), I will play with it :)
>
> I have a question, though... if you can insert the RAW payload... what
> happens with the command name?
> I guess you mean:
> "payload": { "command": "replace catalog", 
"version": 2,  "payload":
> {"data": ..., "metadata": ..., ...} }
> Without any escape sequence at all, right?
>
> And then send it with the --data-urlencode
>
>
>
>
>   That said, as there aren''t any relationships specified in your
above set
> of resource (no require/subscribe params, for example) I''m curious
why the
> empty list of edges fails. That smells like a bug to me; would you mind
> filing a ticket against PuppetDB with the above command included? We can
> take a look and see what''s up.
>
>
> Submitted!
>
>
> Thanks a lot,
> Pablo Fernandez
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

That worked quite well, thanks!! That''s what happens when you
copy-paste
without understanding everything :)

Let me then share with the rest the little script to extract (and hint
to insert) a full catalog into another PuppetDB:

#!/usr/bin/python
# Echo the output to a file (e.g. catalog.json), and use this way:
# curl -vv -H "Accept: application/json"
''http://localhost:8080/v2/commands'' --data-urlencode
payload@catalog.json
import json
import urllib2
response =
urllib2.urlopen(''http://localhost:8080/v3/catalogs/<node_name>'')
data = json.load(response)  
object = { "command": "replace catalog",
"version": 2, "payload": data }
print json.dumps(object, indent=2)


I guess you could potentiate this script by running the curl itself, but
that would require permissions on both PuppetDBs.

BR/Pablo


On 10/17/2013 06:02 PM, Deepak Giridharagopal wrote:
> On Thu, Oct 17, 2013 at 9:09 AM, Pablo Fernandez
> <pablo.fernandez@cscs.ch <mailto:pablo.fernandez@cscs.ch>>
wrote:
>
>     Hi,
>
>     I tried to upload a full catalog with the command:
>     # curl -vv -G -H "Accept: application/json"
>     ''http://localhost:8080/v2/commands'' --data-urlencode
>     payload@catalog.json <mailto:payload@catalog.json>
>
>     Where catalog.json would start with:
>     {"version": 2, "command": "replace
catalog", "payload": {"data":
>     {"version ...etc...
>
>     But I get a:
>     HTTP/1.1 413 FULL head
>
>     This seems to be the Jetty server not accepting big POST. This one
>     is 112 KB, which is not really big for a catalog.
>     I probably hit a configuration issue, but I don''t see the
option
>     in the [jetty] section... bug maybe?
>
>
> (resending as I forgot to reply to the list)
>
> The "-G" command you''re giving to curl forces it to use
a GET for the
> data...can you try after removing that option? That should default it
> to POST.
>
> deepak 
>
>
>
>     Thanks!
>     BR/Pablo
>
>
>
>     On 10/16/2013 09:16 AM, Pablo Fernandez wrote:
>>     Dear Deepak,
>>
>>
>>     On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:
>>>
>>>     1) You don''t need to formulate the "payload"
as a pre-serialized
>>>     JSON string (with all the requisite escaping and whatnot)
>>>     anymore. As of "replace catalog" version 2, you can
just inline
>>>     the raw JSON object itself:
>>>
>>>     "payload": {"data": ...,
"metadata": ..., ...}
>>>
>>>     The entire command is a single JSON object that way, and the
>>>     payload is just a normal value in that hash
>>>     3) With curl, you can supply an argument of
"--data-urlencode
>>>     payload@mycatalog.json
<mailto:payload@mycatalog.json>". Then
>>>     you can just put the payload in a mycatalog.json file, and curl
>>>     should automatically load the content and urlencode it for you.
>>>     See http://curl.haxx.se/docs/manpage.html#--data-urlencode
>>
>>
>>     This RAW version is quite interesting (escaping part of the thing
>>     is a bit nasty), I will play with it :)
>>
>>     I have a question, though... if you can insert the RAW payload...
>>     what happens with the command name?
>>     I guess you mean:
>>     "payload": { "command": "replace
catalog",  "version": 2,
>>     "payload": {"data": ..., "metadata":
..., ...} }
>>     Without any escape sequence at all, right?
>>
>>     And then send it with the --data-urlencode
>>
>>
>>
>>>
>>>      That said, as there aren''t any relationships
specified in your
>>>     above set of resource (no require/subscribe params, for
example)
>>>     I''m curious why the empty list of edges fails. That
smells like
>>>     a bug to me; would you mind filing a ticket against PuppetDB
>>>     with the above command included? We can take a look and see
>>>     what''s up.
>>
>>     Submitted!
>>
>>
>>     Thanks a lot,
>>     Pablo Fernandez
>>     -- 
>>     You received this message because you are subscribed to the
>>     Google Groups "Puppet Users" group.
>>     To unsubscribe from this group and stop receiving emails from it,
>>     send an email to puppet-users+unsubscribe@googlegroups.com
>>     <mailto:puppet-users+unsubscribe@googlegroups.com>.
>>     To post to this group, send email to
>>     puppet-users@googlegroups.com
<mailto:puppet-users@googlegroups.com>.
>>     Visit this group at http://groups.google.com/group/puppet-users.
>>     For more options, visit https://groups.google.com/groups/opt_out.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.