Boy, am I frustrated. I''m about ready to throw puppet out the window here. I''m trying to configure glusterfs, and you know, it kinda made sense to separate the data from the manifests, so I went ahead and put this into a YAML file, which hiera loads... glusterfs_volumes: gfsvol01: volume_name: gfsvol01 master_node: gfs01.us1.xxx.com # Make sure only one node runs the gluster commands. nodes: - name: gfs01.us1.xxx.com bricks: - device: /dev/bcvg/disk1 brick_name: /var/bricks/gfsvol01-0 - device: /dev/bcvg/disk2 brick_name: /var/bricks/gfsvol01-1 - name: gfs02.us1.xxx.com bricks: - device: /dev/bcvg/disk3 brick_name: /var/bricks/gfsvol01-0 - device: /dev/bcvg/disk4 brick_name: /var/bricks/gfsvol01-1 For the last couple of days I have been dealing with the inadequacies of puppet in dealing with working with this kind of data structure. You can''t easily iterate through it, and every time you do, you have to write a new definition that takes an array. The whole thing ends up turning into a giant complicated mess. I tried writing some custom functions in ruby that do things like, return a list of nodes for a volume, or return a list of bricks for a node, but it really irks me that I have to keep writing ruby scripts for this (since ruby makes my eyes bleed). So... what are my options here? Aren''t we supposed to strive for separating the manifest from the data? I could probably get away with a few definitions that take a set of parameters. However, when the time comes to say, add a new node to the cluster, we have to modify the manifest. At one point, I had this working so that all you had to do was add a node to the yaml file, make ZERO changes to the manifest file, and after running puppet, it would add the node to the cluster. It may make my life easier if I flatten the yaml file, but then I''m changing the data to suit the limitations of the DSL. At this point, I''m very close to simply sticking with the yaml file, have puppet push that out to the clients, write some python scripts to do all the magic (reading the yaml file), and have puppet run those scripts with Exec {}. Is proper array/hash iteration ever going to be added to puppet? Doug. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Have you had a look at the functions in puppetlabs-stdlib? There''s a bunch of functions for dealing with hashes and iterating over them there, keys() for example could be helpful in this case. On 30 August 2012 07:50, Douglas Garstang <doug.garstang@gmail.com> wrote:> Boy, am I frustrated. I''m about ready to throw puppet out the window > here. I''m trying to configure glusterfs, and you know, it kinda made > sense to separate the data from the manifests, so I went ahead and put > this into a YAML file, which hiera loads... > > glusterfs_volumes: > gfsvol01: > volume_name: gfsvol01 > master_node: gfs01.us1.xxx.com # Make sure only one node > runs the gluster commands. > nodes: > - name: gfs01.us1.xxx.com > bricks: > - device: /dev/bcvg/disk1 > brick_name: /var/bricks/gfsvol01-0 > - device: /dev/bcvg/disk2 > brick_name: /var/bricks/gfsvol01-1 > - name: gfs02.us1.xxx.com > bricks: > - device: /dev/bcvg/disk3 > brick_name: /var/bricks/gfsvol01-0 > - device: /dev/bcvg/disk4 > brick_name: /var/bricks/gfsvol01-1 > > For the last couple of days I have been dealing with the inadequacies > of puppet in dealing with working with this kind of data structure. > You can''t easily iterate through it, and every time you do, you have > to write a new definition that takes an array. The whole thing ends up > turning into a giant complicated mess. > > I tried writing some custom functions in ruby that do things like, > return a list of nodes for a volume, or return a list of bricks for a > node, but it really irks me that I have to keep writing ruby scripts > for this (since ruby makes my eyes bleed). > > So... what are my options here? Aren''t we supposed to strive for > separating the manifest from the data? I could probably get away with > a few definitions that take a set of parameters. However, when the > time comes to say, add a new node to the cluster, we have to modify > the manifest. At one point, I had this working so that all you had to > do was add a node to the yaml file, make ZERO changes to the manifest > file, and after running puppet, it would add the node to the cluster. > > It may make my life easier if I flatten the yaml file, but then I''m > changing the data to suit the limitations of the DSL. > > At this point, I''m very close to simply sticking with the yaml file, > have puppet push that out to the clients, write some python scripts to > do all the magic (reading the yaml file), and have puppet run those > scripts with Exec {}. > > Is proper array/hash iteration ever going to be added to puppet? > > Doug. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Erik Dalén -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
David Schmitt
2012-Aug-30 11:46 UTC
GlusterFS module (was: Re: [Puppet Users] ... with Puppet.)
My limited understanding from what I''ve read about hiera, is that node-specific data should go to node-specific files, which are to be configured as first priority lookup. That way you can strip away most of the layers of your YAML. I know not enough about glusterfs or your setup to advise you on the internal structure, but from your YAML it seems as if you''re trying to configure a mapping between physical disks and "bricks". Maybe this can be calculated instead of hardcoded? That would take care of the final layer of your YAML. Finally, the config info exchange between your glusterfs-nodes and your glusterfs-master should almost certainly be done by exporting and collecting the necessary resources. Since the advent of puppetdb, there''s no reason at all anymore to avoid storedconfigs. D. PS: having verbs in a resource name (like "add_brick") indicates that you''re not yet fully grasping the declarative style of puppet. Or should the brick be added each time puppet runs? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thursday, August 30, 2012 6:40:11 AM UTC-5, Erik Dalén wrote:> > Have you had a look at the functions in puppetlabs-stdlib? > There''s a bunch of functions for dealing with hashes and iterating > over them there, keys() for example could be helpful in this case. >Also, when you have the choice, I think it usually works better to design your manifests first and then fit the data structure to them, as opposed to designing the data structure first and trying to fit manifests to that structure. Also, if you''re having trouble modeling a complex resource, then that may be a sign that what you really need is a custom type (I know, bleeding eyes and all that, but still...). John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/YNKPqj8UaCYJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Doug, I''m not terribly familiar with gluster, but that thing in your hand certainly looks square and you''re trying to fit it in that round hole. It looks to me like you''ve created an arbitrary service-centric data model, and are then attempting to smash on the tool (puppet) to make it work, and becoming frustrated with it. I liken this to me designing some data structure with pointers, then venting that BASIC can''t handle it [1]. As has been pointed out on several occasions on this list recently, most of us seem to structure our data in a host-centric model, not a service centric model. This would flatten out the below structure quite a bit. It also would likely lead to duplication of data, which is what I suspect you''re trying to avoid. If you''re keen on writing a python script, you could most likely write one to consume the below YAML file and spit out a node-centric YAML that could more easily be used by the puppet DSL. Something like this maybe (No attempt has been made to make this validate)? gluster_master_volumes: [gfsvol01] gluster_bricks: /var/bricks/gfsvol01-0: device: /dev/bcvg/disk1 /var/bricks/gfsvol01-1: device: /dev/bcvg/disk2 Then you should be able to use hiera_hash and creates_resources on gluster_bricks. Another option would be to write a custom type to handle the gluster components. Since types are written in Ruby, you should be able to use the data in a more native way, which would make looping over the hashes or arrays much easier. If you need this to work as is, I suggest contacting Puppet for professional services - those guys over there are really awesome, and I''m sure they can whip something up that can consume the below data structure and do what you want with it. Jason [1] Ok, I''m sure one of you will point out some basic implementation with pointers On 08/30/2012 01:50 AM, Douglas Garstang wrote:> Boy, am I frustrated. I''m about ready to throw puppet out the window > here. I''m trying to configure glusterfs, and you know, it kinda made > sense to separate the data from the manifests, so I went ahead and put > this into a YAML file, which hiera loads... > > glusterfs_volumes: > gfsvol01: > volume_name: gfsvol01 > master_node: gfs01.us1.xxx.com # Make sure only one node > runs the gluster commands. > nodes: > - name: gfs01.us1.xxx.com > bricks: > - device: /dev/bcvg/disk1 > brick_name: /var/bricks/gfsvol01-0 > - device: /dev/bcvg/disk2 > brick_name: /var/bricks/gfsvol01-1 > - name: gfs02.us1.xxx.com > bricks: > - device: /dev/bcvg/disk3 > brick_name: /var/bricks/gfsvol01-0 > - device: /dev/bcvg/disk4 > brick_name: /var/bricks/gfsvol01-1 > > For the last couple of days I have been dealing with the inadequacies > of puppet in dealing with working with this kind of data structure. > You can''t easily iterate through it, and every time you do, you have > to write a new definition that takes an array. The whole thing ends up > turning into a giant complicated mess. > > I tried writing some custom functions in ruby that do things like, > return a list of nodes for a volume, or return a list of bricks for a > node, but it really irks me that I have to keep writing ruby scripts > for this (since ruby makes my eyes bleed). > > So... what are my options here? Aren''t we supposed to strive for > separating the manifest from the data? I could probably get away with > a few definitions that take a set of parameters. However, when the > time comes to say, add a new node to the cluster, we have to modify > the manifest. At one point, I had this working so that all you had to > do was add a node to the yaml file, make ZERO changes to the manifest > file, and after running puppet, it would add the node to the cluster. > > It may make my life easier if I flatten the yaml file, but then I''m > changing the data to suit the limitations of the DSL. > > At this point, I''m very close to simply sticking with the yaml file, > have puppet push that out to the clients, write some python scripts to > do all the magic (reading the yaml file), and have puppet run those > scripts with Exec {}. > > Is proper array/hash iteration ever going to be added to puppet? > > Doug. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.