Dear All, Fairly new to rails and Heroku, so could be doing something wacky - do let me know if you think my code practice is off, even if unrelated to this error, I''d like to learn! I''m using Rails 3.0.0, ruby 1.8.7, and ''sqlite3-ruby'', ''1.2.5'', :require => ''sqlite3''. I''ve got an application that goes off and scrapes websites for event information, to do this I''m using: nokogiri - for scraping date - to help with parsing chronic - to help recognise dates open-uri - to open the url all of these are just gem ''gem-name'', so latest unspecified versions. The code that does this scraping is in the ApplicationController class - a bad place for it? I''ve just got it working as a background process using delayed_job, I have a class called EventSearcher with the required ''perform'' method. My error occurs when I upload to Heroku, if I make changes to the app, but NOT to ApplicationController, do $git add . / git commit .... / git push heroku master / heroku workers 1 / heroku open then the process runs, appears successful in the logs, but returns no events The error can be solved I''ve discovered by making small changes to Application controller, such as adding the line logger.debug"Changing AppController again" then doing $git add . / git commit .... / git push heroku master / heroku workers 1 / heroku open this time the process runs, appears in logs to take longer, and returns the scraped info as required. Any ideas why this should be?? I have tried solving this myself by putting in ''logger.debug" commands, but these don''t show on my heroku logs, any idea how to get them to show up? I''ve put the code below, for ApplicationController, EventSearcher, and part of SuggestedEvents it''s a bit of a mess I know, but runs fine on local machine. -------------- Application Controller class ApplicationController < ActionController::Base require ''date'' require ''nokogiri'' require ''open-uri'' require ''chronic'' protect_from_forgery include SessionsHelper def authenticate deny_access unless signed_in? end def find_all_events @event_searches = EventSearch.all @event_searches.each do |es| do_search(es) end end def find_one_band_events(event_id) es = EventSearch.where(:id => event_id)[0] do_search(es) end def do_search(event_search) logger.debug"Changing AppController again" logger.debug"in do search" url = event_search.urlOne if validate(url) logger.debug"Passed Validate URL" doc = Nokogiri::HTML(open(url)) #search the doc with the saved css search strings if !(event_search.eventDateCSS.empty?) startDate = doc.css(event_search.eventDateCSS) else eventDate = 0 end if !(event_search.eventNameCSS.empty?) eventName = doc.css(event_search.eventNameCSS) else eventName = 0 end if !(event_search.eventLocationCSS.empty?) location = doc.css(event_search.eventLocationCSS) else location = 0 end if !(event_search.eventTimeCSS.empty?) time = doc.css(event_search.eventTimeCSS) else time = 0 end if !(event_search.priceCSS.empty?) price = doc.css(event_search.priceCSS) else price = 0 end if !(event_search.descriptionCSS.empty?) description = doc.css(event_search.descriptionCSS) else description = 0 end i=0 if eventDate != 0 while i < startDate.length # find all events on that page going through each case of start date SuggestedEvent.new do |@savedEvent| if eventDate != 0 @savedEvent.eventDate = date_from_string(startDate[i].content) logger.debug(startDate[i].content) end test = SuggestedEvent.where(:bandName => event_search.bandName, :eventDate => @savedEvent.eventDate) if !test[0] @savedEvent.bandName = event_search.bandName if eventName != 0 @savedEvent.eventName = remove_whitespace(eventName[i].content) end if time != 0 @savedEvent.doorsOpen = Chronic.parse(time[i].content) end if price != 0 @savedEvent.price = price[i].content end if description !=0 @savedEvent.description remove_whitespace(description[i].content) end if location != 0 savedVenueName = paramMatcher(''Venue'',''venueName'', remove_whitespace(location[i].content)) if savedVenueName @savedEvent.venue = savedVenueName[0] @savedEvent.city_location = savedVenueName[1] @savedEvent.longitude = savedVenueName[2] @savedEvent.latitude = savedVenueName[3] else @newVenue = Venue.new @newVenue.venueName = remove_whitespace(location[i].content) @newVenue.old_venue_name = @newVenue.venueName @newVenue.save @savedEvent.venue = remove_whitespace(location[i].content) end #of if else @savedEvent.venue = ''No saved venue found'' end #of location ! @savedEvent.save end # of if !test end # of .new do i += 1 end # of while end #of if eventDate !=0 end # of if url passes end # of def # get the date from whatever string gets thrown def date_from_string(date) # should remove st rd nd and th firstParse = Chronic.parse(date) r1 = /[a-zA-Z]/ #daY Less than 12, assume chronic wrong for Britain, as long as also no characters such as December, #where it would be right if firstParse.day <= 12 and !r1.match(date) #swap month with day firstParse = firstParse.change(:day => firstParse.month, :month => firstParse.day) end #of if return firstParse end #of def # remove whitespace and other characters such as newline def remove_whitespace(dirty_name) return dirty_name.split('' '').join(" ") end #find names in strings from the net def paramMatcher(modelString, param, cssScrapedString) case modelString when ''Venue'' venues = Venue.all i=0 venues.each do |v| if v.venueName.scan(cssScrapedString)[0] or cssScrapedString.scan(v.venueName)[0] return [v.venueName, v.city_location, v.longitude, v.latitude] end end # of .each end # of case return nil end # of def param Matcher def validate(url) begin uri = URI.parse(url) if uri.class != URI::HTTP return false end rescue URI::InvalidURIError return false end return true end end # of class ----------------- Event Searcher class EventSearcher < ApplicationController attr_accessor :all_events attr_accessor :one_event_id def perform logger.debug"in perform" if all_events == 1 # searches all band names logger.debug"in perform, all_events == 1" find_all_events end if one_event_id != nil #searches just one band name using id logger.debug"in perform, one_event != nil" logger.debug(one_event_id) find_one_band_events(one_event_id) end end # Of Perform end #of class -------------- Suggested Events class SuggestedEventsController < ApplicationController #.... other functions removed for brevity.... def search_all_events logger.debug"Sugg E search all events" #the instance of the object that gets put on the queue @event_searcher = EventSearcher.new @event_searcher.one_event_id = nil @event_searcher.all_events = 1 Delayed::Job.enqueue @event_searcher flash[:notice] = "Finding All events, this takes a while, refresh after 10 mins" redirect_to suggested_events_path end end -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
On Jan 3, 10:37 am, Michael Baldock <li...-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> Dear All, > > Fairly new to rails and Heroku, so could be doing something wacky - do > let me know if you think my code practice is off, even if unrelated to > this error, I''d like to learn! > > I''m using Rails 3.0.0, ruby 1.8.7, and ''sqlite3-ruby'', ''1.2.5'', :require > => ''sqlite3''. > > I''ve got an application that goes off and scrapes websites for event > information, to do this I''m using: > > nokogiri - for scraping > date - to help with parsing > chronic - to help recognise dates > open-uri - to open the url > > all of these are just gem ''gem-name'', so latest unspecified versions. > > The code that does this scraping is in the ApplicationController class - > a bad place for it?I don''t know if it is related to your problem, but it does seem a bizarre place to put that code - I''d stick it all in the EventSearcher class and have EventSearcher inherit from object directy) Fred Fred> I''ve just got it working as a background process using delayed_job, I > have a class called EventSearcher with the required ''perform'' method. > > My error occurs when I upload to Heroku, if I make changes to the app, > but NOT to ApplicationController, > do $git add . / git commit .... / git push heroku master / heroku > workers 1 / heroku open > then the process runs, appears successful in the logs, but returns no > events > > The error can be solved I''ve discovered by making small changes to > Application controller, such as adding the line > logger.debug"Changing AppController again" > then doing $git add . / git commit .... / git push heroku master / > heroku workers 1 / heroku open > this time the process runs, appears in logs to take longer, and returns > the scraped info as required. > > Any ideas why this should be?? > > I have tried solving this myself by putting in ''logger.debug" commands, > but these don''t show on my heroku logs, any idea how to get them to show > up? > > I''ve put the code below, for ApplicationController, EventSearcher, and > part of SuggestedEvents it''s a bit of a mess I know, but runs fine on > local machine. > > -------------- Application Controller > > class ApplicationController < ActionController::Base > > require ''date'' > require ''nokogiri'' > require ''open-uri'' > require ''chronic'' > > protect_from_forgery > include SessionsHelper > > def authenticate > deny_access unless signed_in? > end > > def find_all_events > > @event_searches = EventSearch.all > > @event_searches.each do |es| > > do_search(es) > > end > end > > def find_one_band_events(event_id) > > es = EventSearch.where(:id => event_id)[0] > do_search(es) > end > > def do_search(event_search) > > logger.debug"Changing AppController again" > logger.debug"in do search" > url = event_search.urlOne > > if validate(url) > > logger.debug"Passed Validate URL" > > doc = Nokogiri::HTML(open(url)) > > #search the doc with the saved css search strings > if !(event_search.eventDateCSS.empty?) > startDate = doc.css(event_search.eventDateCSS) else eventDate = 0 > end > if !(event_search.eventNameCSS.empty?) > eventName = doc.css(event_search.eventNameCSS) else eventName = 0 > end > if !(event_search.eventLocationCSS.empty?) > location = doc.css(event_search.eventLocationCSS) else location = 0 > end > if !(event_search.eventTimeCSS.empty?) > time = doc.css(event_search.eventTimeCSS) else time = 0 end > if !(event_search.priceCSS.empty?) > price = doc.css(event_search.priceCSS) else price = 0 end > if !(event_search.descriptionCSS.empty?) > description = doc.css(event_search.descriptionCSS) else description > = 0 end > i=0 > > if eventDate != 0 > while i < startDate.length # find all events on that page going > through each case of start date > > SuggestedEvent.new do |@savedEvent| > > if eventDate != 0 > @savedEvent.eventDate = date_from_string(startDate[i].content) > logger.debug(startDate[i].content) > end > > test = SuggestedEvent.where(:bandName => event_search.bandName, > :eventDate => @savedEvent.eventDate) > > if !test[0] > > @savedEvent.bandName = event_search.bandName > if eventName != 0 > @savedEvent.eventName = remove_whitespace(eventName[i].content) > end > if time != 0 > @savedEvent.doorsOpen = Chronic.parse(time[i].content) end > if price != 0 > @savedEvent.price = price[i].content end > if description !=0 > @savedEvent.description > remove_whitespace(description[i].content) end > > if location != 0 > savedVenueName = paramMatcher(''Venue'',''venueName'', > remove_whitespace(location[i].content)) > if savedVenueName > @savedEvent.venue = savedVenueName[0] > @savedEvent.city_location = savedVenueName[1] > @savedEvent.longitude = savedVenueName[2] > @savedEvent.latitude = savedVenueName[3] > else > @newVenue = Venue.new > @newVenue.venueName = remove_whitespace(location[i].content) > @newVenue.old_venue_name = @newVenue.venueName > @newVenue.save > @savedEvent.venue = remove_whitespace(location[i].content) > end #of if > else > @savedEvent.venue = ''No saved venue found'' > end #of location !> @savedEvent.save > end # of if !test > end # of .new do > i += 1 > end # of while > > end #of if eventDate !=0 > > end # of if url passes > > end # of def > > # get the date from whatever string gets thrown > def date_from_string(date) > > # should remove st rd nd and th > > firstParse = Chronic.parse(date) > r1 = /[a-zA-Z]/ > > #daY Less than 12, assume chronic wrong for Britain, as long as also no > characters such as December, > #where it would be right > if firstParse.day <= 12 and !r1.match(date) > > #swap month with day > firstParse = firstParse.change(:day => firstParse.month, :month => > firstParse.day) > > end #of if > > return firstParse > > end #of def > > # remove whitespace and other characters such as newline > > def remove_whitespace(dirty_name) > > return dirty_name.split('' '').join(" ") > > end > > #find names in strings from the net > def paramMatcher(modelString, param, cssScrapedString) > > case modelString > when ''Venue'' > venues = Venue.all > i=0 > venues.each do |v| > > if v.venueName.scan(cssScrapedString)[0] or > cssScrapedString.scan(v.venueName)[0] > > return [v.venueName, v.city_location, v.longitude, v.latitude] > end > > end # of .each > end # of case > > return nil > end # of def param Matcher > > def validate(url) > begin > uri = URI.parse(url) > if uri.class != URI::HTTP > return false > end > rescue URI::InvalidURIError > return false > end > return true > end > > end # of class > > ----------------- Event Searcher > > class EventSearcher < ApplicationController > > attr_accessor :all_events > attr_accessor :one_event_id > > def perform > > logger.debug"in perform" > > if all_events == 1 # searches all band names > > logger.debug"in perform, all_events == 1" > > find_all_events > > end > > if one_event_id != nil #searches just one band name using id > > logger.debug"in perform, one_event != nil" > logger.debug(one_event_id) > find_one_band_events(one_event_id) > > end > > end # Of Perform > > end #of class > > -------------- Suggested Events > > class SuggestedEventsController < ApplicationController > > #.... other functions removed for brevity.... > > def search_all_events > > logger.debug"Sugg E search all events" > #the instance of the object that gets put on the queue > @event_searcher = EventSearcher.new > @event_searcher.one_event_id = nil > @event_searcher.all_events = 1 > Delayed::Job.enqueue @event_searcher > > flash[:notice] = "Finding All events, this takes a while, refresh > after 10 mins" > redirect_to suggested_events_path > > end > > end > > -- > Posted viahttp://www.ruby-forum.com/.-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
Thanks for the advice Fred, I''ve moved the search logic / parsing across to the EventSearcher class now as you suggested, and it seems to be working. I had it in there to begin with, and moved it out because it couldn''t find other classes it needed. But I now think this was caused because I didn''t restart the local server before retrying. When you say "and have EventSearcher inherit from object directy)" could you explain how to do this, currently I have EventSearcher inherit from Application controller with class EventSearcher < ApplicationController is this what you mean? thanks, Mike -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
Frederick Cheung
2011-Jan-03 13:22 UTC
Re: Re: Heroku, needs constant AppController updates?
On 3 Jan 2011, at 13:04, Michael Baldock <lists-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> Thanks for the advice Fred, > > I''ve moved the search logic / parsing across to the EventSearcher class > now as you suggested, and it seems to be working. I had it in there to > begin with, and moved it out because it couldn''t find other classes it > needed. But I now think this was caused because I didn''t restart the > local server before retrying. > > When you say "and have EventSearcher inherit from object directy)" could > you explain how to do this, currently I have EventSearcher inherit from > Application controller with >I meant just class EventSearcher ... end (which is the same as < Object) Fred> class EventSearcher < ApplicationController > > is this what you mean? > > thanks, > > Mike > > -- > Posted via http://www.ruby-forum.com/. > > -- > You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. > To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en. >-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
I tried making EventSearcher just inherit from object, ie: class EventSearcher but then I get errors from almost the first line, eg. failed with NameError: undefined local variable or method `logger'' for my debug statements, and I don''t suppose it''ll know about my model Classes either. Having it inherit from ApplicationController solves these problems, but is there a better way of doing this? cheers, Mike -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
Frederick Cheung
2011-Jan-03 14:40 UTC
Re: Re: Heroku, needs constant AppController updates?
On 3 Jan 2011, at 14:05, Michael Baldock <lists-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> I tried making EventSearcher just inherit from object, ie: > > class EventSearcher > > but then I get errors from almost the first line, eg. > > failed with NameError: undefined local variable or method `logger'' > > for my debug statements, and I don''t suppose it''ll know about my model > Classes either. Having it inherit from ApplicationController solves > these problems, but is there a better way of doing this?Finding your model classes should not be a problem. logger is an instance method on controllers, so you don''t get it here. Rails.logger will get you the same logger though (and you could define a logger method yourself to save some typing. Fred> > cheers, > > Mike > > -- > Posted via http://www.ruby-forum.com/. > > -- > You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. > To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en. >-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
Cool, learning plenty here, cheers! I''ve got a logger method in the class now, and so taken away the unnecessary inheritance. Cheers! My heroku app is working for now, although I''m not sure it''ll stay working, it seems to stop fetching events after a day, and needs a new git push, with changes to specifically the "def do_search(event_search)" method which is the one that does the work of getting the url, and parsing the html. A bit weird, as it works every time on local machine. I''ve asked them about this on the heroku mailing list as well. Thanks for your help! M -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
On Jan 3, 3:59 pm, Michael Baldock <li...-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> Cool, learning plenty here, cheers! > > I''ve got a logger method in the class now, and so taken away the > unnecessary inheritance. Cheers! > > My heroku app is working for now, although I''m not sure it''ll stay > working, it seems to stop fetching events after a day, and needs a new > git push, with changes to specifically the "def do_search(event_search)" > method which is the one that does the work of getting the url, and > parsing the html. A bit weird, as it works every time on local machine. > I''ve asked them about this on the heroku mailing list as well.If your daemon is idle for extended periods of time it might be your mysql connection timing out. You could try tossing in a call to ActiveRecord::Base.connection_pool.verify_active_connections! to weed out dead connections Fred> > Thanks for your help! > > M > > -- > Posted viahttp://www.ruby-forum.com/.-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
Marnen Laibow-Koser
2011-Jan-03 19:26 UTC
Re: Heroku, needs constant AppController updates?
Michael Baldock wrote in post #972049:> Cool, learning plenty here, cheers! > > I''ve got a logger method in the class now, and so taken away the > unnecessary inheritance. Cheers! > > My heroku app is working for now, although I''m not sure it''ll stay > working, it seems to stop fetching events after a day, and needs a new > git push, with changes to specifically the "def do_search(event_search)" > method which is the one that does the work of getting the url, and > parsing the html. A bit weird, as it works every time on local machine. > I''ve asked them about this on the heroku mailing list as well.Why don''t you set up a cron job or use Resque or delayed_job to run the job periodically?> > Thanks for your help! > > MBest, -- Marnen Laibow-Koser http://www.marnen.org marnen-sbuyVjPbboAdnm+yROfE0A@public.gmane.org -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.