The 1st anniversary of the module team! Hello from the module team here at Puppet Labs! I’m starting this email with a lie because I’m not sure exactly when our first anniversary really is, but I started on the 1st of July and the team had only just gotten started, so that’s as good a date as any. For those readers who are unaware, the module team at Puppet Labs exists primarily to implement the supported modules initiative. For anyone that missed the announcement last year, the goal of supported modules is to help you more easily discover amazing modules and offer support for those modules to Puppet Enterprise customers. Over the last year we’ve been laying the foundations to make this sustainable (and making it up as we go along). In order to support modules across the diverse set of platforms PE runs on, we’ve had to experiment with and learn how to test modules in a sustainable, scalable way, and over the last year we’ve been trying to accomplish that. Members of the team Before we talk about what we’ve been doing over the last year, I thought it would be nice to briefly talk about who is in the team, our backgrounds, and where you can get hold of us. I’ll list everyone in the order they joined the team to make life easy for me. Hunter Haugen Hunter was the very first member of the team and many of you know him as “hunner” on IRC. Previously a member of the Professional Services team, Hunter spent his time traveling and visiting customers all over the world. His background, like mine, is mostly UNIX systems administration. He’s responsible for the huge refactoring of the apache module a while back, and is all over the popular puppetlabs modules we hope you’re all using. Ashley Penney This one is me. I’m “ashp” on IRC and hopefully I know many of you. I’ve been a Puppet user since the start of 2008, when I spent most of my time harassing Luke on IRC over “bugs” I found that turned out to be fundamental design decisions. I’ve been in operations for ~12 years, and this is the only job I’ve ever had where nobody will wake me up at 0300 to let me know everything has crashed and the world is on fire. It’s pretty awesome. Chris Hoge Anyone here who has used the openstack modules can thank Chris for putting in a ton of work to make them awesome. Just before I took this job, I tried to use the puppetlabs openstack modules and after a week I threw up my hands and gave up as nothing worked. Now they actually work and are awesome. Progress! Chris primarily focuses on openstack, but he sometimes has to wrestle modules that are dependencies into shape (like mongodb!). You can find him as “hogepodge” on IRC. Travis Fields Travis joined to help the module team build out and build up awesome modules specifically for Windows. The rest of us are Linux users, so we often just threw up our hands and said “I can’t fix that!” when modules had issues on Windows. Since joining the team, he’s taken over the reboot and registry modules, fixed vcsrepo to work on windows, taken on the new acl module, as well as fixed a number of issues throughout our tooling to make sure Windows is a true first class platform for modules instead of something we hide under the bed from. Travis goes by “cyberious” on IRC. Morgan Haskel Morgan previously worked with Onyxpoint (a long time Puppet community member!) on Puppet modules. Battle-scarred from forcing complex modules into behaving properly, she joined Puppet Labs to help us write amazing supported modules. She’s brought some adult supervision to the team and ensures we’re on a regular cadence for module releases. You can ask her questions about Hadoop (she’ll love it, I promise) on IRC as “_morgan”. AJ Johnson The almost-newest member of the team is our boss; he's in charge of ensuring we’re all pointing in the right direction and focused on actually building things the community benefits from. He escaped from IBM to come wrangle the team into a semblance of order and make sure we’re on track to deliver supported modules! Colleen Murphy The actual-newest member of the team comes to us for the summer as an intern from PSU (that’s the portland one, not the Pennsylvania one). She’s a Linux sysadmin, Puppet user, and developer, and she is already helping us tackle a project we’ve been putting off for months. You can find her on IRC as “crinkle”. If you’re igalic or blkperl then I preemptively ban you from asking her for PR merges! :) Other People This is already longer than an Oscar acceptance speech, so I want to wrap up by just saying that we have a bunch of other fantastic people that help us keep this show on the road. Lauren Rother helps ensure modules have documentation that makes sense, Heidi Pio shouts at us when we don’t close JIRA tickets, Justin Stoller makes the CI environment work, John Duarte shakes his head at our attempts at risk assessments and testing plans, Ryan Coleman helps us figure out what we’re even meant to be building in the first place, and I’m sure I’ve forgotten someone who is going to be so mad at me. What we’ve been doing Onwards to something more of you might care about: what is it we DO around here? Over the last year we’ve been focused on two main things: refactoring, rewriting, and testing existing modules, and helping shape the tools, documentation, and environment to make testing hundreds of modules feasible. The time has flown by and I think the entire team would like to have released more modules than we did, but we’ve managed to get a lot of the groundwork done. First four months In the first four months Hunter and I ran rampant over the modules trying to deal with some of the backlog of PRs, feature requests, and rewrites that were needed. We rewrote the mysql and rabbitmq modules, refactored a bunch of smaller helper modules (passenger, etc), and merged or closed hundreds (and hundreds) of PRs that had been outstanding (for years in some cases). We tried to aggressively manage the backlog of bug reports and just generally get the modules into some better shape. Alongside this work, we started talking over standards and what constituted a well-written module in our opinion. As Puppet users, obviously we all disagreed, but over time we’ve managed to wrestle consensus down to what we consider to be reasonably state of the art in modules. Next four months The story of the next four months was acceptance testing. We started out with a bunch of tests we had written in rspec-system during the initial refactoring efforts of the first four months, but we switched to beaker-rspec (https://github.com/puppetlabs/beaker-rspec) in order to handle our acceptance tests. This allows us to run `rspec spec/acceptance` and get virtual machines automatically spun up, manifests applied, the results checked through various shell commands, and then have the virtual machines killed off. This allowed us to test the true behaviors of modules for the first time rather than testing the contents of the puppet catalog. In these four months we added thousands of tests across the modules that were to be supported. These tests exposed issues on many platforms and over the months we worked to try and deal with those issues. Alongside this work, we continued trying to handle merging PRs in, as well as the odd bit of refactoring to types and providers as we went. We wrote the “Beginners Guide to Modules <http://docs.puppetlabs.com/guides/module_guides/bgtm.html>” as well as continued to work on making sure Beaker works well for community members. We started rebuilding the vagrant boxes and uploading those so everyone had recent operating systems to test against. These eventually transitioned off to https://vagrantcloud.com/puppetlabs/. Last four months Our most recent four months have also involved testing. As the number of acceptance tests grew and grew, it became harder and harder to get our full set of tests to pass. We continued work on managing the modules in general, including an almost full rewrite of the SOAP based f5 module. We also wrote the “Advanced Guide to Modules” which is still in the editing stage, but attempts to walk you through building a module from the ground up (with types and providers, facts, functions, unit tests, acceptance tests, all of it). Faced with the increasing amount of time we’re spending focused on testing we’ve put together a plan to fix things. I won’t give you all the tedious details but we’re going to change how we currently acceptance test, and it’ll affect anyone who contributes to our modules. Once we had our hammer of beaker, everything looked like a nail and we started doing unit level tests in the acceptance framework. Things like “checking the rpm is really installed” for postgres rather than just checking “postgres is running”. We’re going to rework these acceptance tests to have a much smaller number of end to end tests. We think this will snowball over the next four months and really help us increase our speed when it comes to development work. What we’re doing right now The good news is we’re writing modules! I’m writing a drastically improved REST based f5 module that will replace the SOAP module in time. Morgan has written a tomcat module that handles multiple installs of tomcat on a single server, compiled from source if needed, and we’re preparing to release that next week. Travis is working on a powershell module. Hunter is working on recovering from being sick, as well as improvements to the haproxy module. Alongside modules, we’re (I say 'we', I mean Colleen is) also working on a project to allow us to centrally manage files that are common across modules. Things like the .travis.yml or Gemfile. To do that we need to account for local module differences, however, and Colleen has written us a framework that allows us to manage these files properly. It’ll mean we can update .travis.yml in a single repository and then push it out automatically to all our modules. This has been something that’s made certain mass changes needed for testing really difficult. I’m also working on a fixtures module which is temporarily living at https://github.com/apenney/pe_fixtures. This will eventually contain a `facter -p` run for every single PE platform we have, and exists to be used with rspec-puppet. Right now, if you want to make sure your module fails gracefully on unsupported platforms you have to go mock up the facts for all those platforms by hand. When we finish this work, you’ll be able to easily iterate over all platforms and check how your module performs on them with rspec-puppet. Things we’d like to do Sadly, I can’t give away all our secrets here! We’re continuing to work on making it easier for community members to author modules. Our first year was spent focused on figuring out the answers to questions we know many of you have, and now we want to spend some time to make sure we fully document and expose those answers so everyone can benefit. We’ve got some other stuff in the pipeline to make it easier to find high quality modules, as well as to write high quality modules. We hope you bear with us as we continue to get all the pieces in place to allow us to scale to hundreds of support modules, modules hopefully for everything you need to do under the sun. I spent from 2008-2013 writing private modules for a variety of companies that I couldn’t release and every new job meant rewriting the same things again. I came to Puppet Labs (hell, I begged them over and over to create this team) so that we can write these modules once and have everyone contribute to making them amazing, so that you never have to write the same module twice in your career. We won’t stop until you’re never faced with the horror of writing a module for some boring piece of open source software, freeing you up to do things that you actually care about. :) -- Ashley Penney ashley.penney@puppetlabs.com Module Engineer *Join us at PuppetConf 2014**, September 23-24 in San Francisco - http://puppetconf.com <http://puppetconf.com/>* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAC9eg%2B%3DJh9J7yxN5O8nJDeCPb%3Do0NRcf7ZW%3D5rWozD6e7izH_g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.