search for: conditional_requests

...ize this is kind of a fundamental change for such a simple feature, but it looks like there are some API changes planned for 0.8.0 anyway, so I figured I''d throw this out there and start a conversation about it. Also, (because someone will probably point it out) I''m aware of the @conditional_requests attribute, which uses If-Modified-Since if the page is in history. However, this isn''t practical for the classic If-Modified-Since case: when you''re scraping/downloading a large file. In this case, it''s not optimal to keep the page in history (using memory) until I make m...

...; type in the Model. irb(main):039:0> agent.user = WWW::Mechanize.new #<WWW::Mechanize:0xb71295f0 @follow_meta_refresh=false, @key=nil, @user_agent="WWW-Mechanize/0.6.10 (http://rubyforge.org/projects/mechanize/)", @log=nil, @proxy_addr=nil, @keep_alive=true, @user=nil, @pass=nil, @conditional_requests=true, @proxy_port=nil, @pluggable_parser=#<WWW::Mechanize::PluggableParser:0xb712953c @parsers={"text/html"=>WWW::Mechanize::Page}, defaultWWW::Mechanize::File, read_timeoutnil, keep_alive_time300, ca_filenil, watch_for_setnil, proxy_passnil, auth_hash{}, certnil, history[], open_ti...

...here. In my research for my project, recipe 11.16 in O''Reilly''s Ruby Cookbook references a website[1] discussing the importance of the If-Modified-Since header in polite RSS readers. It mentions that the Etag header is also important. I see in Mechanize''s code that if conditional_requests is set, it''ll add the If-Modified-Since header. But this requires that the page is already in the history, and there''s currently no provision for caching the history. Since RSS readers (and most scrapers in general) are likely to be run periodically, mechanize should try to main...

...mit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. Changes: = Mechanize CHANGELOG == 0.6.8 * Keep alive can be shut off now with WWW::Mechanize#keep_alive * Conditional requests can be shut off with WWW::Mechanize#conditional_requests * Monkey patched Net::HTTP#keep_alive? * [#9877] Moved last request time. Thanks Max Stepanov * Added WWW::Mechanize::File#save * Defaulting file name to URI or Content-Disposition * Updating compatability with hpricot * Added more unit tests http://mechanize.rubyforge.org/ -- Aaron Patterson...