I discovered Capistrano while I was trying to figure out what I wanted. See attached notes. http://www.genunix.org/wiki/index.php/GNOSIS/Kraken Puppet seems promising. Thanks, Brian -- - Brian Gupta http://opensolaris.org/os/project/nycosug/
On 10/8/07, Brandorr <brandorr@opensolaris.org> wrote:> I discovered Capistrano while I was trying to figure out what I > wanted. See attached notes. > http://www.genunix.org/wiki/index.php/GNOSIS/KrakenPuppet and Capistrano fill two very different niche''s. Capistrano is essentially a mashup of Rake and SSH. It allows you to develop recipes for dealing with executing commands on servers in parallel, the canonical example being the deployment of a Rails application. Capistrano *can* be used to automate standard systems administration tasks (and some people do use it that way) but it''s really no different than what many systems administrators have been doing for years with shell scripts and for loops.* Puppet is a language for expressing your infrastructure in code. Using Puppet''s language, you define all the resources needed in your infrastructure, and then apply those resources to individual nodes (or servers.) Puppet makes your infrastructure more reliable, repeatable, and documented. We use Puppet and Capistrano both extensively in our consulting practice. Puppet builds and manages the infrastructure (including the things done when you can cap deploy:setup,) while Capistrano handles the deployment of new code. (which it is pretty good at.) When you add a tool like the soon-to-get-a-real-release iClassify, you can integrate Puppet and Capistrano directly. It lets you organize your systems with tags (that map to Puppet Classes), and Capistrano queries it to set up the server roles. For example, you might have a "database" puppet class that configures a mysql server. It further might be a "master" or "slave", which maps to the Capistrano :db role. We query iClassify with Capistrano, asking for the database servers, and making the one that is your master the :primary one. Capistrano and Puppet go together like cheese and crackers. Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com
On 10/8/07, Brandorr <brandorr@opensolaris.org> wrote:> I discovered Capistrano while I was trying to figure out what I > wanted. See attached notes. > http://www.genunix.org/wiki/index.php/GNOSIS/Kraken > > Puppet seems promising.I just read your wiki entry. You should consider contributing to Factor, for your system inventory needs. Puppet is a significant part of an open source data center management tool, and is a much better integration choice than cfengine (imnsho.) Having written two or three systems like the one you are describing in the past (including iClassify today) I suggest you do a few things: 1. You need to keep the schema flexible, since you never know what sort of thing people need to track. If you find yourself writing a new class for each entity you need to track in your datacenter (server, router, switch, rack, pdu, et al) then you''re off track. 2. You need to allow for the expression of relationships between entities. For example, a server is in a rack. This is best done fairly loosely. iClassify doesn''t do this at all right now, although I have some ideas about how it might. 3. Use a full text search engine like Xapian, Solr, Ferret or Lucene. One of these models used Xapian entirely, and had no relational database engine at all. Thus far, it remains my favorite of my implementations of this problem for flexibility. In the end, you can''t beat saying "tags:database AND mysql-master:true", or "ipaddress:[192.168.1.0-192.168.10.0]". 4. As much as possible, entities need to be responsible for reporting in to the management system. They should push data to it, it should not be polling the entities for updates. (Where possible. Sometimes, you have to poll.) I could go on. :) There is lots of work going on in this space right now in the puppet community at large, and if would be excellent if it would include the work you are already planning for. Regards, Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com
On 10/8/07, Adam Jacob <adam@hjksolutions.com> wrote:> On 10/8/07, Brandorr <brandorr@opensolaris.org> wrote: > > I discovered Capistrano while I was trying to figure out what I > > wanted. See attached notes. > > http://www.genunix.org/wiki/index.php/GNOSIS/Kraken > > > > Puppet seems promising. > > I just read your wiki entry. You should consider contributing to > Factor, for your system inventory needs. Puppet is a significant part > of an open source data center management tool, and is a much better > integration choice than cfengine (imnsho.)Factor? (A quick Google search did not help).> Having written two or three systems like the one you are describing in > the past (including iClassify today) I suggest you do a few things: > > 1. You need to keep the schema flexible, since you never know what > sort of thing people need to track. If you find yourself writing a > new class for each entity you need to track in your datacenter > (server, router, switch, rack, pdu, et al) then you''re off track.What about new subclasses, that can be dynamically defined by the user? Can you talk about this more?> 2. You need to allow for the expression of relationships between > entities. For example, a server is in a rack. This is best done > fairly loosely. iClassify doesn''t do this at all right now, although > I have some ideas about how it might.I have thought about this. Any idea which Object it will be a property of? IE: Is the server a property of the rack, or is the rack location, a property of the server? Or is it something different altogether? (Maybe, both)> 3. Use a full text search engine like Xapian, Solr, Ferret or Lucene. > One of these models used Xapian entirely, and had no relational > database engine at all. Thus far, it remains my favorite of my > implementations of this problem for flexibility. In the end, you can''t > beat saying "tags:database AND mysql-master:true", or > "ipaddress:[192.168.1.0-192.168.10.0]".I think I''d want both.. (search index for speed, actual data in database.)> 4. As much as possible, entities need to be responsible for reporting > in to the management system. They should push data to it, it should > not be polling the entities for updates. (Where possible. Sometimes, > you have to poll.)Yes, I agree, but for the initial implementation, I thought it would be easier to implement a polling based systems. (And we are going to need it anyway for initialization).> I could go on. :) > > There is lots of work going on in this space right now in the puppet > community at large, and if would be excellent if it would include the > work you are already planning for.No need to reinvent the wheel. If there is a system close enough to what I want, then contributing to it, seems the wiser choice.> Regards, > Adam > -- > HJK Solutions - We Launch Startups - http://www.hjksolutions.com > Adam Jacob, Senior Partner > T: (206) 508-4759 E: adam@hjksolutions.com > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >-- - Brian Gupta http://opensolaris.org/os/project/nycosug/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday 09 October 2007, Brandorr wrote:> On 10/8/07, Adam Jacob <adam@hjksolutions.com> wrote: > > On 10/8/07, Brandorr <brandorr@opensolaris.org> wrote: > > > I discovered Capistrano while I was trying to figure out what I > > > wanted. See attached notes. > > > http://www.genunix.org/wiki/index.php/GNOSIS/Kraken > > > > > > Puppet seems promising. > > > > I just read your wiki entry. You should consider contributing to > > Factor, for your system inventory needs. Puppet is a significant part > > of an open source data center management tool, and is a much better > > integration choice than cfengine (imnsho.) > > Factor? (A quick Google search did not help).Facter: http://reductivelabs.com/projects/facter/> > Having written two or three systems like the one you are describing in > > the past (including iClassify today) I suggest you do a few things: > > > > 1. You need to keep the schema flexible, since you never know what > > sort of thing people need to track. If you find yourself writing a > > new class for each entity you need to track in your datacenter > > (server, router, switch, rack, pdu, et al) then you''re off track. > > What about new subclasses, that can be dynamically defined by the > user? Can you talk about this more?When you start with a flexible tool like facter, then you should _only_ make classes a-posteriori (that is after receiving the data). i.e. "(lsbdistcodename == Debian) => Debian Server" and "(hwvendor == cisco && ciscoclass == router) => big_iron_router". Since it is very easy for a client to add facts to facter, it has to be very easy to add attributes to your schema. For the client!> > 2. You need to allow for the expression of relationships between > > entities. For example, a server is in a rack. This is best done > > fairly loosely. iClassify doesn''t do this at all right now, although > > I have some ideas about how it might. > > I have thought about this. Any idea which Object it will be a property > of? IE: Is the server a property of the rack, or is the rack location, > a property of the server? Or is it something different altogether? > (Maybe, both)In a relational system I''d say the rack is just (one of) the locations. Everything else can be queried (e.g. "all servers in this rack")> > 4. As much as possible, entities need to be responsible for reporting > > in to the management system. They should push data to it, it should > > not be polling the entities for updates. (Where possible. Sometimes, > > you have to poll.) > > Yes, I agree, but for the initial implementation, I thought it would > be easier to implement a polling based systems. (And we are going to > need it anyway for initialization).With polling you have to maintain a list of clients. Using a push model, this maintenance goes (mostly) away. Regards, David - -- The primary freedom of open source is not the freedom from cost, but the free- dom to shape software to do what you want. This freedom is /never/ exercised without cost, but is available /at all/ only by accepting the very different costs associated with open source, costs not in money, but in time and effort. - -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHDINv/Pp1N6Uzh0URAjY7AKCL0Z9kbcxLdIEMsOCGx5YuJ9X0sACggEJw yrUIWO7N9LyRPKQgB4cnZ1Y=uEnR -----END PGP SIGNATURE-----
On 10/9/07, Brandorr <brandorr@opensolaris.org> wrote:> > 1. You need to keep the schema flexible, since you never know what > > sort of thing people need to track. If you find yourself writing a > > new class for each entity you need to track in your datacenter > > (server, router, switch, rack, pdu, et al) then you''re off track. > > What about new subclasses, that can be dynamically defined by the > user? Can you talk about this more?I feel like the right answer to this problems lies in a middle ground between LDAP and a full relational database. Basically, you need to model what sort of information you need to track, instead of the specific entities you need. For example, you need to track "things" (racks, servers, software, people,) which we''ll call "entities". Those entities have attributes (location, ipaddress, username.) The schema for each entity is entirely user defined (although you''ll probably supply an application that has lots of them included through a discovery process.) The reason servers have IP Addresses isn''t because you defined a schema somewhere that says it, it''s because something you called a "server" has an attribute named that (and the final resulting schema is the aggregate of all the attributes in existence for a given set of like entities.) This approach has one big downside, which is that it''s difficult to create a manual entry user-interface. (Since, without an object, you don''t have a schema yet.) You can get past it by looking at the data storage layer as a separate thing, which your web-app uses (and it''s fine for your web-app to know about the schema.)> > 2. You need to allow for the expression of relationships between > > entities. For example, a server is in a rack. This is best done > > fairly loosely. iClassify doesn''t do this at all right now, although > > I have some ideas about how it might. > > I have thought about this. Any idea which Object it will be a property > of? IE: Is the server a property of the rack, or is the rack location, > a property of the server? Or is it something different altogether? > (Maybe, both)Both, I think, when it comes time to ask questions about things. The tricky place for this is in the user interface, where you want the relationships to be updated properly (and easily.)> > 3. Use a full text search engine like Xapian, Solr, Ferret or Lucene. > > One of these models used Xapian entirely, and had no relational > > database engine at all. Thus far, it remains my favorite of my > > implementations of this problem for flexibility. In the end, you can''t > > beat saying "tags:database AND mysql-master:true", or > > "ipaddress:[192.168.1.0-192.168.10.0]". > > I think I''d want both.. (search index for speed, actual data in database.)Beware how this scales. If you get millions of objects at once, you''ll need a queue. :)> > 4. As much as possible, entities need to be responsible for reporting > > in to the management system. They should push data to it, it should > > not be polling the entities for updates. (Where possible. Sometimes, > > you have to poll.) > > Yes, I agree, but for the initial implementation, I thought it would > be easier to implement a polling based systems. (And we are going to > need it anyway for initialization).I disagree. Polling means knowing in advance what exists in your infrastructure, which you might not know. For initialization, just have the agent be installed on each host (and configured,) which is trivial to do with a tool like puppet.> No need to reinvent the wheel. If there is a system close enough to > what I want, then contributing to it, seems the wiser choice.You aren''t the only one who needs it, certainly. Collaborating on solving this problem well (and with some degree of finality, at least for the 80% use case) would make me happy. :) Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com