Puppet users - Nov 2012 - parse puppet file url in customized functions?

Hi all, 
       I want to parse the puppet url in my customizes functions.  Eg, 
        

 module Puppet::Parser::Functions
      newfunction(:load_file, ) do |arg|
           path = get_real_path(arg[0])
           load_file_content(path)
      end
    end


suppose arg[0] is a puppet file url: puppt:///module/someModule/fileName.yaml,
how can I parse this url and get the real os path of that file?

BR/Woosley


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/SwFbxhgqfdgJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sat, Nov 10, 2012 at 9:22 AM, woosley. xu. <redicaps@gmail.com> wrote:
> Hi all,
>        I want to parse the puppet url in my customizes functions.  Eg,
>
>
>  module Puppet::Parser::Functions
>       newfunction(:load_file, ) do |arg|
>            path = get_real_path(arg[0])
>            load_file_content(path)
>       end
>     end
>
>
> suppose arg[0] is a puppet file url:
puppt:///module/someModule/fileName.yaml, how can I parse this url and get the
real os path of that file?
>
>
There isn''t really a good and reliable way I can think of to do this.
 Functions are executed when a catalog is being compiled, but there is no
guarantee that the host compiling a catalog is also the same host that will
serve up file content and metadata.  In multiple-master configurations the
fileserver puppet master processes are often running on different hosts
than the catalog compiler puppet master processes.

What problem are you trying to solve?  Perhaps there is another approach
that does not require translating Puppet URI''s to real paths.

-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I am thinking about loading some paramters from a external file for my 
souces.  Eg,

class test { 
         
        ## set parameters. 
        load_from_yaml("puppet:///modules/test/config.yaml")
}

I already have a simple function which can load from the abspath 


module Puppet::Parser::Functions
  
    newfunction(:load_from_yaml) do |args|

        unless args.length == 1
            raise Puppet::ParseError, ("loadyaml(): wrong number of 
arguments (#{args.length}; must be 1)")
        end 
        path = args[0]
        raise Puppet::ParseError, ("path must be absolute") unless 
Puppet::Util.absolute_path?(path)
        params = YAML.load_file(path)
        params.each do |param, value|
            setvar(param, value)
        end
    end 
end 


supporting  puppet url format would be a great help here. 


On Tuesday, November 13, 2012 4:48:15 AM UTC+8, Jeff McCune
wrote:
>
> On Sat, Nov 10, 2012 at 9:22 AM, woosley. xu.
<redi...@gmail.com<javascript:>
> > wrote:
>
>> Hi all, 
>>        I want to parse the puppet url in my customizes functions.  Eg, 
>>         
>>
>>  module Puppet::Parser::Functions
>>       newfunction(:load_file, ) do |arg|
>>            path = get_real_path(arg[0])
>>            load_file_content(path)
>>       end
>>     end
>>
>>
>> suppose arg[0] is a puppet file url:
puppt:///module/someModule/fileName.yaml, how can I parse this url and get the
real os path of that file?
>>
>>
> There isn''t really a good and reliable way I can think of to do
this.
>  Functions are executed when a catalog is being compiled, but there is no 
> guarantee that the host compiling a catalog is also the same host that will
> serve up file content and metadata.  In multiple-master configurations the 
> fileserver puppet master processes are often running on different hosts 
> than the catalog compiler puppet master processes.
>
> What problem are you trying to solve?  Perhaps there is another approach 
> that does not require translating Puppet URI''s to real paths.
>
> -Jeff
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/5wxc6-yDDJkJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Nov 13, 2012 at 5:25 AM, woosley. xu. <redicaps@gmail.com>
wrote:
> I am thinking about loading some paramters from a external file for my
> souces.  Eg,
>
> class test {
>
>         ## set parameters.
>         load_from_yaml("puppet:///modules/test/config.yaml")
> }
Ah, cool idea.  I think this would be a really valuable function to
include in the stdlib [1] module.

While it''s very difficult to get the filesystem path, it seems like
you really only need the contents of the file.  If you have a
puppet:// URI, you can speak to the Puppet fileserve using the REST
API to get the file contents.  This would avoid the need to read the
file from disk yourself and it will work well even in multiple-master
deployment scenarios since you''re speaking to the API rather than to
the underlying filesystem.

The static compiler does just this.  It reads the contents of the file
using the FileServer API in the store_content method:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/indirector/catalog/static_compiler.rb#L160-161

You might be able to do something similar in your parser function.
Given a URI, pass the string form to
Puppet::FileServing::Content.indirection.find and you should get back
a Ruby instance that models the file metadata and contains the
content.

Hope this helps,
-Jeff

[1] https://forge.puppetlabs.com/puppetlabs/stdlib

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks Jeff,
    I copy-pasted the code and it just worked.  Here is my function:




require ''uri''
module Puppet::Parser::Functions

    newfunction(:loadyaml, :doc => <<-''ENDHEREDOC'')
do |args|
        Load a YAML file containing a hash, set the hash values.
Support puppet urls

        For example:
        loadyaml(''/etc/puppet/data/myhash.yaml'')
        loadyaml(''puppet:///modules/name/myhash.yaml'')
ENDHEREDOC

        unless args.length == 1
            raise Puppet::ParseError, ("loadyaml(): wrong number of
arguments (#{args.length}; must be 1)")
        end

        params = nil
        path = args[0]
        unless Puppet::Util.absolute_path?(path)
            uri = URI.parse(URI.escape(path))
            raise Puppet::ParseError, ("Cannot use relative URLs
''#{path}''") unless uri.absolute?
            raise Puppet::ParseError, ("Cannot use opaque URLs
''#{path}''") unless uri.hierarchical?
            raise Puppet::ParseError, ("Cannot use URLs of type
''#{uri.scheme}'' as source for fileserving") unless
%w{puppet}.include?(uri.scheme)
            Puppet.info "loading parameters from #{path}"
            content = Puppet::FileServing::Content.indirection.find(path)
            params = YAML.load(content.content)
        else
            params = YAML.load_file(path)
        end

        params.each do |param, value|
            setvar(param, value)
        end
    end
end


2012/11/14 Jeff McCune <jeff@puppetlabs.com>:
> On Tue, Nov 13, 2012 at 5:25 AM, woosley. xu. <redicaps@gmail.com>
wrote:
>> I am thinking about loading some paramters from a external file for my
>> souces.  Eg,
>>
>> class test {
>>
>>         ## set parameters.
>>         load_from_yaml("puppet:///modules/test/config.yaml")
>> }
>
> Ah, cool idea.  I think this would be a really valuable function to
> include in the stdlib [1] module.
>
> While it''s very difficult to get the filesystem path, it seems
like
> you really only need the contents of the file.  If you have a
> puppet:// URI, you can speak to the Puppet fileserve using the REST
> API to get the file contents.  This would avoid the need to read the
> file from disk yourself and it will work well even in multiple-master
> deployment scenarios since you''re speaking to the API rather than
to
> the underlying filesystem.
>
> The static compiler does just this.  It reads the contents of the file
> using the FileServer API in the store_content method:
>
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/indirector/catalog/static_compiler.rb#L160-161
>
> You might be able to do something similar in your parser function.
> Given a URI, pass the string form to
> Puppet::FileServing::Content.indirection.find and you should get back
> a Ruby instance that models the file metadata and contains the
> content.
>
> Hope this helps,
> -Jeff
>
> [1] https://forge.puppetlabs.com/puppetlabs/stdlib
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>


-- 
woosley.xu.    http://twitter.com/redicaps

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Nov 13, 2012 at 10:19 PM, woosley. xu. <redicaps@gmail.com>
wrote:
> Thanks Jeff,
>     I copy-pasted the code and it just worked.
That''s great to hear!
> Here is my function:
It''d be wonderful if you contributed this function to the stdlib
module as a pull request.  There are hundreds (maybe thousands?) of
people and organizations who are using the stdlib module to add
functionality to Puppet.  If you''ve never submitted a pull request on
github before there''s some really good information about our
contributing process at
https://github.com/puppetlabs/puppet/blob/master/CONTRIBUTING.md

And there''s some general information about pull requests at:
https://help.github.com/articles/using-pull-requests

Hope this helps.  If you''d like to work with me on getting this
function into stdlib, I''m jmccune in #puppet-dev on freenode.

-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks Jeff, I will send a pull-request.


2012/11/15 Jeff McCune <jeff@puppetlabs.com>:
> On Tue, Nov 13, 2012 at 10:19 PM, woosley. xu. <redicaps@gmail.com>
wrote:
>> Thanks Jeff,
>>     I copy-pasted the code and it just worked.
>
> That''s great to hear!
>
>> Here is my function:
>
> It''d be wonderful if you contributed this function to the stdlib
> module as a pull request.  There are hundreds (maybe thousands?) of
> people and organizations who are using the stdlib module to add
> functionality to Puppet.  If you''ve never submitted a pull request
on
> github before there''s some really good information about our
> contributing process at
> https://github.com/puppetlabs/puppet/blob/master/CONTRIBUTING.md
>
> And there''s some general information about pull requests at:
> https://help.github.com/articles/using-pull-requests
>
> Hope this helps.  If you''d like to work with me on getting this
> function into stdlib, I''m jmccune in #puppet-dev on freenode.
>
> -Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>


-- 
woosley.xu.    http://twitter.com/redicaps

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.