Hello camping people I''ve written a Ruby library for working with CouchDB. It''s a pretty thin API - it provides a database object, a document object (a glorified hash) and a design object. In case anyone''s not familiar with CouchDB, it''s a schema-less JSON document database with a HTTP interface. You store documents in the database, then query the database with MapReduce views. Your map and reduce queries are just JavaScript, and they''re stored as fields of design documents. Generally, each design document corresponds to an ''app''. So it''s all lovely and simple, and I''ve been working on writing a module that plugs my API into Camping. Models look like this:> module MyFood::Models > class Recipe < CouchDocument > needs :name > needs :ingredients > needs :instructions > suggests :dates_cooked, :array > suggests :cost > end > endCouchDocument is a hash with some methods for pushing to and pulling from the document database, and some validation functions. "needs" and "suggests" both just refer to keys. I haven''t decided exactly how to implement "suggests" yet; "needs" enforces the existence of particular keys before a document can be pushed. It also looks at the name of the class and uses it for a "kind" or "type" key. I''ve written a small library that uses S Expressions to parse Ruby into JavaScript. It''s limited, but for the moment it''s adequate to my purposes: writing JavaScript MapReduce functions as Ruby blocks. This means you can do this:> module MyFood::MapReduce > view :untried_recipes do > map do > emit(doc.name, doc) if doc.kind == "Recipe" and doc.dates_cooked.length == 0 > end > end > > > view :cost_of_crazy_huge_indian_meal > map do > emit(doc.name, doc.cost) if doc.kind == "Recipe" and doc.cuisine == "South Asian" > end > reduce do > return sum(values) > end > end > endAnd you get a design document with two entries in the "views" field:> "map"=> > "function ( doc ) { > if( doc.kind == "Recipe" && doc.dates_cooked.length == 0 ) { > emit(doc.name, doc) > } > }"And:> "map" => > "function ( doc ) { > if( doc.kind == "Recipe" && doc.cuisine == "South Asian" ) { > emit(doc.name, doc.cost) > } > }", "reduce"=> > "function ( key, values, rereduce ) { > return sum(values) > }"So, yeah, whatever, that''s all well and good, you can write your queries and it''ll update the design document on the server so you can just do a HTTP GET request to the view''s URL and CouchDB will return your rows or whatever. Cool. The thing is, I have no idea how to unify this with Camping. I feel like a "relaxing" schema-less thing like this is the perfect match for really small web projects, and there''s the added bonus that since the CouchDB settings are just defined in a global variable at the top of your app, you can have different databases - or even different database servers - for different Camping apps running on the same web server if you like. There''s been a bit of discussion on the list lately about how great it is that Ruby''s controllers belong to classes - there''s an intrinsic logic to it. I want something as intuitive as that for this query language, but I don''t know how to do it. The earliest implementation of this I did just had a Design class, and when you wrote a view it''d define a method on the Design class with that view''s name - so, above, you''d call Design.untried_recipes, or Design.cost_of_blahblah, and it''d query Couch for the result. That''s okay until you have more than a small handful of views - more than that, and there''s too much to keep track of. The views could just belong to models, so you could call Recipe.untried, or something, but the thing about MapReduce is that so many of your queries don''t naturally belong to a particular "model", and I don''t really want to built an architecture that makes people tend to write their models and queries as if it''s SQL and you need to normalise and get all heavy with joins and generally ruin your life with boredom. If you want a view with particular data, you write a view that gives you exactly that, you save it, CouchDB automatically caches it in your fancy B-trees, and you''re rolling. It''s beautiful, but I don''t know how to make it user-friendly. So uh does anyone have any advice? -- Cerales (@lodoicea) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/camping-list/attachments/20110828/231c49df/attachment.html>
On Sun, Aug 28, 2011 at 04:58, Daniel Bryan <danbryan at gmail.com> wrote:> Hello camping people > > I''ve written a Ruby library for working with CouchDB. It''s a pretty thin > API - it provides a database object, a document object (a glorified hash) > and a design object. > > In case anyone''s not familiar with CouchDB, it''s a schema-less JSON > document database with a HTTP interface. You store documents in the > database, then query the database with MapReduce views. Your map and reduce > queries are just JavaScript, and they''re stored as fields of design > documents. Generally, each design document corresponds to an ''app''. > > So it''s all lovely and simple, and I''ve been working on writing a module > that plugs my API into Camping. > > ? > > I''ve written a small library that uses S Expressions to parse Ruby into > JavaScript. It''s limited, but for the moment it''s adequate to my purposes: > writing JavaScript MapReduce functions as Ruby blocks. This means you can do > this: > > module MyFood::MapReduce > view :untried_recipes do > map do > emit(doc.name, doc) if doc.kind == "Recipe" and > doc.dates_cooked.length == 0 > end > end > > view :cost_of_crazy_huge_indian_meal > map do > emit(doc.name, doc.cost) if doc.kind == "Recipe" and doc.cuisine => "South Asian" > end > reduce do > return sum(values) > end > end > end > > And you get a design document with two entries in the "views" field: > > "map"=> > "function ( doc ) { > if( doc.kind == "Recipe" && doc.dates_cooked.length == 0 ) { > emit(doc.name, doc) > } > }" > > > And: > > "map" => > "function ( doc ) { > if( doc.kind == "Recipe" && doc.cuisine == "South Asian" ) { > emit(doc.name, doc.cost) > } > }", "reduce"=> > "function ( key, values, rereduce ) { > return sum(values) > }" > > >Woah, that sounds pretty cool. Are you using RubyParser or Ripper? So, yeah, whatever, that''s all well and good, you can write your queries and> it''ll update the design document on the server so you can just do a HTTP GET > request to the view''s URL and CouchDB will return your rows or whatever. > Cool. The thing is, I have no idea how to unify this with Camping. I feel > like a "relaxing" schema-less thing like this is the perfect match for > really small web projects, and there''s the added bonus that since the > CouchDB settings are just defined in a global variable at the top of your > app, you can have different databases - or even different database servers - > for different Camping apps running on the same web server if you like. > > There''s been a bit of discussion on the list lately about how great it is > that Ruby''s controllers belong to classes - there''s an intrinsic logic to > it. I want something as intuitive as that for this query language, but I > don''t know how to do it. The earliest implementation of this I did just had > a Design class, and when you wrote a view it''d define a method on the Design > class with that view''s name - so, above, you''d call Design.untried_recipes, > or Design.cost_of_blahblah, and it''d query Couch for the result. That''s okay > until you have more than a small handful of views - more than that, and > there''s too much to keep track of. >I agree. What''s the point of namespaces (modules/classes) if you''re going to store everything in the same place?> The views could just belong to models, so you could call Recipe.untried, or > something, but the thing about MapReduce is that so many of your queries > don''t naturally belong to a particular "model", and I don''t really want to > built an architecture that makes people tend to write their models and > queries as if it''s SQL and you need to normalise and get all heavy with > joins and generally ruin your life with boredom. If you want a view with > particular data, you write a view that gives you exactly that, you save it, > CouchDB automatically caches it in your fancy B-trees, and you''re rolling. > It''s beautiful, but I don''t know how to make it user-friendly. >I actually have the same problem with ActiveRecord: There''s so many things that''s working across tables, but it forces you to put it in one model. I don''t think that even makes sense in a SQL-world. It doesn''t make sense in a relational world, but it seems people like objects so well that they try to put the (quite sensible) relational model in a object-oriented world?> So uh > > does anyone > > have any advice? > > -- Cerales (@lodoicea) >It''s a hard problem that you''ve stumbled upon; trying to integrate two quite different point-of-views (Ruby vs CouchDb) in an elegant way. Okay, let''s see? First of all: Does all MapReduce-queries work on different types of documents; i.e. does it make sense to run MapReduce on just one type of documents? I guess so? In that cases, it would be quite useful to say `Receipes.tagged_with("Bacon")`? What if you introduce the concept of *contexts*? So all CouchDocument are automatically one context (that only include one type of document), but you can also introduce your own contexts. So you can have a CookingUser-context, which describes a User and what he can do with Receipes. In your controller you can say `CookingUser.new(current_user).untried_recepies`. Or maybe a UserReceipes which has an "untried"-view. So contexts are just a lightweight class where you can place your views. The developer has to decide what the context should be called and how they should work. The point is that the contexts should mirror the use-cases of the data, and as there might be many ways to look at the same data, you can''t have an automatic way of saying where the view belongs. It all depends on the point-of-view. I guess what I''m really trying to say: Don''t have one class where you put everything. Don''t force the views to belong to one set of classes. Do something in-between: Let the views live where they make sense, sometimes on a CouchDocument-class (when the views only work on that class), sometimes on a custom class (where it''s the developer who decides what that class *means*). I''ll have to admit that these are mainly my thoughts on the general "It sucks to place all queries in only ActiveRecord-classes"-problem I mentioned above, but I hope this might be applicable in other cases too? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/camping-list/attachments/20110828/47778d95/attachment-0001.html>
> Woah, that sounds pretty cool. Are you using RubyParser or Ripper?Neither. I''m using sourcify ( http://rubydoc.info/gems/sourcify/0.5.0/frames ) to convert blocks into an S Expression, and then my own library to parse that and spit out equivalent JavaScript syntax. I''ve made no attempt to deal with Ruby syntax that has no direct JS equivalent (like block arguments to function calls), nor to capture operations that would be illegal in JavaScript - https://github.com/cerales/ShyRubyJS> Okay, let''s see? First of all: Does all MapReduce-queries work on different types of documents; i.e. does it make sense to run MapReduce on just one type of documents? I guess so? In that cases, it would be quite useful to say `Receipes.tagged_with("Bacon")`?Yeah, it does make sense, and a lot of the time you might want to do queries that just get a particular "kind" of document and filter it in some way. Regardless of the final solution I think that should be supported, so I guess I''ll try to think of a robust query language for that kind of thing.> What if you introduce the concept of *contexts*? So all CouchDocument are automatically one context (that only include one type of document), but you can also introduce your own contexts. So you can have a CookingUser-context, which describes a User and what he can do with Receipes. In your controller you can say `CookingUser.new(current_user).untried_recepies`. Or maybe a UserReceipes which has an "untried"-view. > > So contexts are just a lightweight class where you can place your views. The developer has to decide what the context should be called and how they should work. The point is that the contexts should mirror the use-cases of the data, and as there might be many ways to look at the same data, you can''t have an automatic way of saying where the view belongs. It all depends on the point-of-view.I like this! The way I''m interpreting this is that contexts could be a really light layer on top of the anonymous views. So it''d have your abstract view specification - a map block and a view block - and then a couple of little links specifying, for example, which model classes / "kinds" it''s associated with, for the sake of the developer''s sanity. When I started development on this concept, I put a lot of thought into how models would be written, to the point that they became effectively a schema. I realised that that was absurd - anything heavier than the optional "needs" and "suggests" concepts is throwing away the agility, convenience and forgiveness of a NoSQL approach. So that''s out the window, but there''s still a challenge here - model schemas, as well as being a concession to the brutality of SQL and its greedy demands, are a very useful tool for keeping track of and self-documenting your application. It seems there needs to be some kind of tool to replace that if your''e doing something like this. So what I''m thinking might be useful is some kind of optional admin interface that will look at your Models and your Views and your Contexts, and try to visualise or contextualise them in some way. Like the django admin interface, with all the power of introspection, but somehow groovy. Of course, it''s so _easy_ to work with data when it''s just JSON documents that there''s no reason you couldn''t, with a little thought, build a super user-friendly interface where you build your MapReduce views just by being given a graphical view of your document Kinds and dragging and dropping things into the fields for what you want to emit, what you want to filter, etc. It''s probably not the best idea philosophically, but for giving people the tools to really easily write a dynamic web app it seems like it could be useful. brain racing aaaaaaa Thanks for the help, Judofyr - Daniel / Cerales On Monday, 29 August 2011 at 1:54 AM, Magnus Holm wrote:> On Sun, Aug 28, 2011 at 04:58, Daniel Bryan <danbryan at gmail.com (mailto:danbryan at gmail.com)> wrote: > > Hello camping people > > > > I''ve written a Ruby library for working with CouchDB. It''s a pretty thin API - it provides a database object, a document object (a glorified hash) and a design object. > > > > In case anyone''s not familiar with CouchDB, it''s a schema-less JSON document database with a HTTP interface. You store documents in the database, then query the database with MapReduce views. Your map and reduce queries are just JavaScript, and they''re stored as fields of design documents. Generally, each design document corresponds to an ''app''. > > > > So it''s all lovely and simple, and I''ve been working on writing a module that plugs my API into Camping. > > > > ? > > > > I''ve written a small library that uses S Expressions to parse Ruby into JavaScript. It''s limited, but for the moment it''s adequate to my purposes: writing JavaScript MapReduce functions as Ruby blocks. This means you can do this: > > > > > module MyFood::MapReduce > > > view :untried_recipes do > > > map do > > > emit(doc.name (http://doc.name), doc) if doc.kind == "Recipe" and doc.dates_cooked.length == 0 > > > end > > > end > > > > > > > > > view :cost_of_crazy_huge_indian_meal > > > map do > > > emit(doc.name (http://doc.name), doc.cost) if doc.kind == "Recipe" and doc.cuisine == "South Asian" > > > end > > > reduce do > > > return sum(values) > > > end > > > end > > > end > > > > > > > And you get a design document with two entries in the "views" field: > > > > > "map"=> > > > "function ( doc ) { > > > if( doc.kind == "Recipe" && doc.dates_cooked.length == 0 ) { > > > emit(doc.name (http://doc.name), doc) > > > } > > > }" > > > > > > > > > And: > > > > > "map" => > > > "function ( doc ) { > > > if( doc.kind == "Recipe" && doc.cuisine == "South Asian" ) { > > > emit(doc.name (http://doc.name), doc.cost) > > > } > > > }", "reduce"=> > > > "function ( key, values, rereduce ) { > > > return sum(values) > > > }" > > > > > > > > > Woah, that sounds pretty cool. Are you using RubyParser or Ripper? > > > So, yeah, whatever, that''s all well and good, you can write your queries and it''ll update the design document on the server so you can just do a HTTP GET request to the view''s URL and CouchDB will return your rows or whatever. Cool. The thing is, I have no idea how to unify this with Camping. I feel like a "relaxing" schema-less thing like this is the perfect match for really small web projects, and there''s the added bonus that since the CouchDB settings are just defined in a global variable at the top of your app, you can have different databases - or even different database servers - for different Camping apps running on the same web server if you like. > > > > There''s been a bit of discussion on the list lately about how great it is that Ruby''s controllers belong to classes - there''s an intrinsic logic to it. I want something as intuitive as that for this query language, but I don''t know how to do it. The earliest implementation of this I did just had a Design class, and when you wrote a view it''d define a method on the Design class with that view''s name - so, above, you''d call Design.untried_recipes, or Design.cost_of_blahblah, and it''d query Couch for the result. That''s okay until you have more than a small handful of views - more than that, and there''s too much to keep track of. > > I agree. What''s the point of namespaces (modules/classes) if you''re going to store everything in the same place? > > > The views could just belong to models, so you could call Recipe.untried, or something, but the thing about MapReduce is that so many of your queries don''t naturally belong to a particular "model", and I don''t really want to built an architecture that makes people tend to write their models and queries as if it''s SQL and you need to normalise and get all heavy with joins and generally ruin your life with boredom. If you want a view with particular data, you write a view that gives you exactly that, you save it, CouchDB automatically caches it in your fancy B-trees, and you''re rolling. It''s beautiful, but I don''t know how to make it user-friendly. > > > > > I actually have the same problem with ActiveRecord: There''s so many things that''s working across tables, but it forces you to put it in one model. I don''t think that even makes sense in a SQL-world. It doesn''t make sense in a relational world, but it seems people like objects so well that they try to put the (quite sensible) relational model in a object-oriented world? > > > So uh > > > > does anyone > > > > have any advice? > > > > -- Cerales (@lodoicea) > > It''s a hard problem that you''ve stumbled upon; trying to integrate two quite different point-of-views (Ruby vs CouchDb) in an elegant way. > Okay, let''s see? First of all: Does all MapReduce-queries work on different types of documents; i.e. does it make sense to run MapReduce on just one type of documents? I guess so? In that cases, it would be quite useful to say `Receipes.tagged_with("Bacon")`? > > What if you introduce the concept of *contexts*? So all CouchDocument are automatically one context (that only include one type of document), but you can also introduce your own contexts. So you can have a CookingUser-context, which describes a User and what he can do with Receipes. In your controller you can say `CookingUser.new(current_user).untried_recepies`. Or maybe a UserReceipes which has an "untried"-view. > > So contexts are just a lightweight class where you can place your views. The developer has to decide what the context should be called and how they should work. The point is that the contexts should mirror the use-cases of the data, and as there might be many ways to look at the same data, you can''t have an automatic way of saying where the view belongs. It all depends on the point-of-view. > > I guess what I''m really trying to say: Don''t have one class where you put everything. Don''t force the views to belong to one set of classes. Do something in-between: Let the views live where they make sense, sometimes on a CouchDocument-class (when the views only work on that class), sometimes on a custom class (where it''s the developer who decides what that class *means*). > > I''ll have to admit that these are mainly my thoughts on the general "It sucks to place all queries in only ActiveRecord-classes"-problem I mentioned above, but I hope this might be applicable in other cases too? > _______________________________________________ > Camping-list mailing list > Camping-list at rubyforge.org (mailto:Camping-list at rubyforge.org) > http://rubyforge.org/mailman/listinfo/camping-list-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/camping-list/attachments/20110829/836ae778/attachment.html>
> So what I''m thinking might be useful is some kind of optional admin > interface that will look at your Models and your Views and your > Contexts, and try to visualise or contextualise them in some way. > Like the django admin interface, with all the power of > introspection, but somehow groovy. > > Of course, it''s so _easy_ to work with data when it''s just JSON > documents that there''s no reason you couldn''t, with a little > thought, build a super user-friendly interface where you build your > MapReduce views just by being given a graphical view of your > document Kinds and dragging and dropping things into the fields for > what you want to emit, what you want to filter, etc. It''s probably > not the best idea philosophically, but for giving people the tools > to really easily write a dynamic web app it seems like it could be > useful.I have to say that sounds *really* interesting - DaveE -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/camping-list/attachments/20110829/a9836c1a/attachment-0001.html>
On Mon, Aug 29, 2011 at 05:39, Daniel Bryan <danbryan at gmail.com> wrote:> Woah, that sounds pretty cool. Are you using RubyParser or Ripper? > > Neither. I''m using sourcify ( > http://rubydoc.info/gems/sourcify/0.5.0/frames ) to convert blocks into an > S Expression, and then my own library to parse that and spit out equivalent > JavaScript syntax. I''ve made no attempt to deal with Ruby syntax that has no > direct JS equivalent (like block arguments to function calls), nor to > capture operations that would be illegal in JavaScript - > https://github.com/cerales/ShyRubyJS >Very interesting!> > What if you introduce the concept of *contexts*? So all CouchDocument are > automatically one context (that only include one type of document), but you > can also introduce your own contexts. So you can have a CookingUser-context, > which describes a User and what he can do with Receipes. In your controller > you can say `CookingUser.new(current_user).untried_recepies`. Or maybe a > UserReceipes which has an "untried"-view. > > So contexts are just a lightweight class where you can place your views. > The developer has to decide what the context should be called and how they > should work. The point is that the contexts should mirror the use-cases of > the data, and as there might be many ways to look at the same data, you > can''t have an automatic way of saying where the view belongs. It all depends > on the point-of-view. > > > I like this! > > The way I''m interpreting this is that contexts could be a really light > layer on top of the anonymous views. So it''d have your abstract view > specification - a map block and a view block - and then a couple of little > links specifying, for example, which model classes / "kinds" it''s associated > with, for the sake of the developer''s sanity. > > When I started development on this concept, I put a lot of thought into how > models would be written, to the point that they became effectively a schema. > I realised that that was absurd - anything heavier than the optional "needs" > and "suggests" concepts is throwing away the agility, convenience and > forgiveness of a NoSQL approach. > > So that''s out the window, but there''s still a challenge here - model > schemas, as well as being a concession to the brutality of SQL and its > greedy demands, are a very useful tool for keeping track of and > self-documenting your application. It seems there needs to be some kind of > tool to replace that if your''e doing something like this. >The truth is that there is no such thing as "schema-less". There will always be a schema. The question is if it will be enforced in the database, at insert-time in code (each document has a strict schema) or at runtime in code (it works just as a Hash).> > So what I''m thinking might be useful is some kind of optional admin > interface that will look at your Models and your Views and your Contexts, > and try to visualise or contextualise them in some way. Like the django > admin interface, with all the power of introspection, but somehow groovy. > > Of course, it''s so _easy_ to work with data when it''s just JSON documents > that there''s no reason you couldn''t, with a little thought, build a super > user-friendly interface where you build your MapReduce views just by being > given a graphical view of your document Kinds and dragging and dropping > things into the fields for what you want to emit, what you want to filter, > etc. It''s probably not the best idea philosophically, but for giving people > the tools to really easily write a dynamic web app it seems like it could be > useful. >I''ve been thinking about the exact same concept, but a more general solution based on relational algebra. My idea was that you defined your schema, relationships and queries in an admin-panel thingie (inspired from Yahoo! Pipes) in a database-agnostic way (relational model), and then later defined the mapping from the database (which could be CouchDB, MongoDB, MySQL, Redis) to the conceptual schema. I''d say your project is more limited, but way more feasible than my "big plan". So I think you should definitely go for it and make a kick-ass CouchDB platform :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/camping-list/attachments/20110830/49dd9bc3/attachment.html>