Is there a way to request only the http headers? I have a bot that connects to sites and spits out the html title but for binary files I like it to just read the http headers to get the file size etc... and not read in the entire binary! Thanks!
Seems like WWW::Mechanize#head would work: http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 -Mat On Mar 30, 2009, at 8:05 PM, Daniel Aquino wrote:> Is there a way to request only the http headers? > > I have a bot that connects to sites and spits out the html title but > for binary files I like it to just read the http headers to get the > file size etc... and not read in the entire binary! > > Thanks! > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users
On Mon, Mar 30, 2009 at 5:58 PM, Mat Schaffer <mat.schaffer at gmail.com> wrote:> Seems like WWW::Mechanize#head would work: > > http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183Yes. A head request sounds appropriate. -- Aaron Patterson http://tenderlovemaking.com/
Yea I really searched around for this and couldn''t figure out how to do it... Thanks so much... Also I think I remember reading something that the http server has to support a head request. is this true? On Mon, Mar 30, 2009 at 11:18 PM, Aaron Patterson <aaron.patterson at gmail.com> wrote:> On Mon, Mar 30, 2009 at 5:58 PM, Mat Schaffer <mat.schaffer at gmail.com> wrote: >> Seems like WWW::Mechanize#head would work: >> >> http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 > > Yes. ?A head request sounds appropriate. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users >
Is there anyway to limit the amount of data to read from any link? Perhaps use a filter to detect the <title/> tag and abort connection? Or set a timeout on how long data should be read from the link? I''m sure a malicious person could easily still feed in a massively large file and cause the daemon to stick around reading it all... And the only thing I''m interested in is the <title/> Thanks! On Tue, Mar 31, 2009 at 1:17 AM, Daniel Aquino <mr.danielaquino at gmail.com> wrote:> Yea I really searched around for this and couldn''t figure out how to do it... > > Thanks so much... > > Also I think I remember reading something that the http server has to > support a head request. > > is this true? > > On Mon, Mar 30, 2009 at 11:18 PM, Aaron Patterson > <aaron.patterson at gmail.com> wrote: >> On Mon, Mar 30, 2009 at 5:58 PM, Mat Schaffer <mat.schaffer at gmail.com> wrote: >>> Seems like WWW::Mechanize#head would work: >>> >>> http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 >> >> Yes. ?A head request sounds appropriate. >> >> -- >> Aaron Patterson >> http://tenderlovemaking.com/ >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users >> >
On Mon, Mar 30, 2009 at 10:17 PM, Daniel Aquino <mr.danielaquino at gmail.com> wrote:> Yea I really searched around for this and couldn''t figure out how to do it... > > Thanks so much... > > Also I think I remember reading something that the http server has to > support a head request. > > is this true?Yes, but most do. I don''t think I''ve run in to one that doesn''t. -- Aaron Patterson http://tenderlovemaking.com/
if I call agent.head and then call agent.get I end up with only head... On Tue, Mar 31, 2009 at 1:50 AM, Aaron Patterson <aaron.patterson at gmail.com> wrote:> On Mon, Mar 30, 2009 at 10:17 PM, Daniel Aquino > <mr.danielaquino at gmail.com> wrote: >> Yea I really searched around for this and couldn''t figure out how to do it... >> >> Thanks so much... >> >> Also I think I remember reading something that the http server has to >> support a head request. >> >> is this true? > > Yes, but most do. ?I don''t think I''ve run in to one that doesn''t. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users >