Mechanize users - Apr 2007 - Scraping and saving.

Hi,

I''m working to scrape and save some ebooks.  Mechanize has been
wonderful so far. The link I''m having trouble with is this one.
http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H

When I click that in the browser it saves it to a file named
H_1632.zip. How do I get that name from the page.  I suspect to save
this to a file I would just do (if lnk is the lnk)
lnk.click() but how do I save it and how do I get the name that the
browser gets?

Thanks,

-Joe

Hi Joe!

On Tue, Apr 03, 2007 at 10:07:03AM +0900, Joe Duhamel
wrote:
> Hi,
> 
> I''m working to scrape and save some ebooks.  Mechanize has been
> wonderful so far. The link I''m having trouble with is this one.
>
http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H
> 
> When I click that in the browser it saves it to a file named
> H_1632.zip. How do I get that name from the page.  I suspect to save
> this to a file I would just do (if lnk is the lnk)
> lnk.click() but how do I save it and how do I get the name that the
> browser gets?
I have to admit I haven''t tried requests like that with mechanize yet.
I tried it in Firefox using LiveHTTPHeaders to see how the browser gets
the filename, then just reproduced it with mechanize:

  agent = WWW::Mechanize.new
  page = agent.get(
   
''http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H'')
  page.save_as(
   
page.header[''content-disposition''].split(/=/)[1].gsub(/"/,
'''')
  )

I''ll change mechanize to default the filename if it sees this header in
the future.  Then you should be able to do ''pave.save'', and it
will do
the right thing.  Unfortunately you''ll have to use the listed technique
for now....

Hope that helps!

-- 
Aaron Patterson
http://tenderlovemaking.com/

Alan,

Thanks for the quick reply. Other than your web site getting me in
trouble with the Banks''s Web Proxy Police. This has rocked. First
thanks for the headsup on LiveHTTPHeaders and second. yes that was
exactly was I was looking to do.

-Joe

On 4/3/07, Aaron Patterson <aaron at tenderlovemaking.com>
wrote:
> Hi Joe!
>
> On Tue, Apr 03, 2007 at 10:07:03AM +0900, Joe Duhamel wrote:
> > Hi,
> >
> > I''m working to scrape and save some ebooks.  Mechanize has
been
> > wonderful so far. The link I''m having trouble with is this
one.
> >
http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H
> >
> > When I click that in the browser it saves it to a file named
> > H_1632.zip. How do I get that name from the page.  I suspect to save
> > this to a file I would just do (if lnk is the lnk)
> > lnk.click() but how do I save it and how do I get the name that the
> > browser gets?
>
> I have to admit I haven''t tried requests like that with mechanize
yet.
> I tried it in Firefox using LiveHTTPHeaders to see how the browser gets
> the filename, then just reproduced it with mechanize:
>
>   agent = WWW::Mechanize.new
>   page = agent.get(
>    
''http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H'')
>   page.save_as(
>    
page.header[''content-disposition''].split(/=/)[1].gsub(/"/,
'''')
>   )
>
> I''ll change mechanize to default the filename if it sees this
header in
> the future.  Then you should be able to do ''pave.save'',
and it will do
> the right thing.  Unfortunately you''ll have to use the listed
technique
> for now....
>
> Hope that helps!
>
> --
> Aaron Patterson
> http://tenderlovemaking.com/
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>