Hello, I am building a web application that will require a lot of statistic logging. Each request could generate between 2 and 6 database inserts. It seems a bit of an overhead to have this running in the main application. Is something like this worth creating a stats logging process (with backgroundrb for example). Also, is this something that could be saved in a separate database since it''s pretty irrelevant from the main collection of data (just to run marketing statistics on). Or am I just overthinking all of this? Any thoughts on this? Have you dealt with something similar before? Thanks, -carl -- EPA Rating: 3000 Lines of Code / Gallon (of coffee) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Friday 09 February 2007, Carl Lerche wrote:> I am building a web application that will require a lot of statistic > logging. Each request could generate between 2 and 6 database > inserts. It seems a bit of an overhead to have this running in the > main application.I think your options mainly depend on exactly how the inserts are triggered. If there''s any request-related processing involved you may have to do it in the controller(s), hopefully in a filter. If you can handle the statistic gathering at the model level, then observers may be for you. If the statistics are related purely to operations on the data, you may even get away with a database trigger. Michael -- Michael Schuerig mailto:michael-q5aiKMLteq4b1SvskN2V4Q@public.gmane.org http://www.schuerig.de/michael/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
apsoto-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
2007-Feb-09 18:46 UTC
Re: Statistics logging
I''d also add that you batch the inserts instead of inserting into the database on every logged event. On Feb 8, 5:03 pm, Michael Schuerig <mich...-q5aiKMLteq4b1SvskN2V4Q@public.gmane.org> wrote:> On Friday 09 February 2007, Carl Lerche wrote: > > > I am building a web application that will require a lot of statistic > > logging. Each request could generate between 2 and 6 database > > inserts. It seems a bit of an overhead to have this running in the > > main application. > > I think your options mainly depend on exactly how the inserts are > triggered. If there''s any request-related processing involved you may > have to do it in the controller(s), hopefully in a filter. If you can > handle the statistic gathering at the model level, then observers may > be for you. If the statistics are related purely to operations on the > data, you may even get away with a database trigger. > > Michael > > -- > Michael Schuerig > mailto:mich...-q5aiKMLteq5BV9CJdY2HSA@public.gmane.org://www.schuerig.de/michael/--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
apsoto-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:> I''d also add that you batch the inserts instead of inserting into the > database on every logged event.Yes, especially if you''re expecting high traffic. You could store as yaml and possibly avoid the later inserts all together. Or store as yaml and have a nice hash ready for your batch. -- Chris Martin Web Developer http://chriscodes.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Could you elaborate on this please? I would do all my logging throughout each page into some stats collector object, then the stats collector object has a write method of some sort called in an after_filter that sends all the inserts at once? I would have to write SQL directly I"m guessing or is there a way to do batch inserts into multiple tables with AR? Thanks, -carl On 2/9/07, Chris Martin <chriscodes-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > apsoto-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote: > > I''d also add that you batch the inserts instead of inserting into the > > database on every logged event. > > Yes, especially if you''re expecting high traffic. > > You could store as yaml and possibly avoid the later inserts all > together. Or store as yaml and have a nice hash ready for your batch. > > -- > Chris Martin > Web Developer > http://chriscodes.com > > > >-- EPA Rating: 3000 Lines of Code / Gallon (of coffee) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Carl Lerche wrote:> Could you elaborate on this please? > > I would do all my logging throughout each page into some stats > collector object, then the stats collector object has a write method > of some sort called in an after_filter that sends all the inserts at > once? I would have to write SQL directly I"m guessing or is there a > way to do batch inserts into multiple tables with AR? > > Thanks, > -carl >Sounds good. Although you lost me at insert to multiple tables at once, don''t know enough about what you''re trying to do. It really depends on how many stats you''re logging, and the requirements for reading them, whether it''s "overthinking" or if my suggestion would be to "little". Since you said it was irrelevant to your app data, I was thinking just to store it in a yaml file, and load/display it from your "marketing app". That way it''s separated. You could do something like stats = { ''user'' => ''Tom'', ''time'' => Time.now ''uri'' => request[''REQUEST_URI''], ''browser'' => request[''HTTP_USER_AGENT''] } stat_file = File.new("#{RAILS_ROOT}/stats/stats.yml", "w") stat_file << stats.to_yaml stat_file.close Now you''ve got your stats in a simple yaml file that can be read like yaml_stats = YAML.load(File.open("#{RAILS_ROOT}/stats/stats.yml")) Then displayed, or manipulated as you wish, and when you wish. That''s obviously a really basic example. You could do something more interesting like run a BackgrounDRB process that loads the yaml file, does your inserts, removes the file, does a backflip with a twist, then starts over, etc. If the stats must go in a separate table, or separate database all together, I''d keep the logic to do that within BackgroundRB, or even a separate "stats-for-marketing" rails app. Your main app writes the files, and backgrounDRB or other app reads them and writes to the database. Depending on your actual requirements, and if writing to a file is an option, you could simply write to a csv file that marketing could open in Excel. Then no extra code is needed to read the stats, and you''re on your way to finishing early! HTH -- Chris Martin Web Developer http://chriscodes.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---