R help - Feb 2009 - R as a web scraping tool using RCurl

Hi List,
I am trying to leverage my knowledge of R in trying to use it for tasks that
may not make R the best choice for these tasks.
I wish to automate a web scraping task, which requires a multi-step
procedure:
1) log in to a website
2) Go to a particular page
3) From the drop down menu, click on a particular link
4) From the tabulated data presented, choose relevant information based on a
filter on the date column.

I am not highly acquainted with RCurl or CURL for that matter. I've used
Perl extensively and know that such tasks are more suitable for such
scripting tools as Perl which have an efficient regex engine and a great
number of modules/packages for such web scraping tasks.

I am investigating RCurl's capabilities since I wish to use R, assuming no
knowledge of Perl or other more suitable web-scraping tools.

I would greatly appreciate any links/information/tutorials/book suggestions
that will allow me to use RCurl to get the above tasks accomplished. I have
looked at the Omegahat RCurl links and the manuals present there but would
like R users to share their personal experiences and resources they may have
used to use and implement RCurl.


Thanks
Harsh Singhal

"Senior Jedi General"
Decision Systems
Mu Sigma Inc.
Chicago, IL

	[[alternative HTML version deleted]]

Try Firefox and an add in called I Macros from www.iopus.com as an simpler
alternative

read some stuff here

http://www.decisionstats.com/2009/01/web-crawling-automation/

regards,

Ajay


On Wed, Feb 18, 2009 at 2:06 PM, Harsh <singhalblr@gmail.com> wrote:
> Hi List,
> I am trying to leverage my knowledge of R in trying to use it for tasks
> that
> may not make R the best choice for these tasks.
> I wish to automate a web scraping task, which requires a multi-step
> procedure:
> 1) log in to a website
> 2) Go to a particular page
> 3) From the drop down menu, click on a particular link
> 4) From the tabulated data presented, choose relevant information based on
> a
> filter on the date column.
>
> I am not highly acquainted with RCurl or CURL for that matter. I've
used
> Perl extensively and know that such tasks are more suitable for such
> scripting tools as Perl which have an efficient regex engine and a great
> number of modules/packages for such web scraping tasks.
>
> I am investigating RCurl's capabilities since I wish to use R, assuming
no
> knowledge of Perl or other more suitable web-scraping tools.
>
> I would greatly appreciate any links/information/tutorials/book suggestions
> that will allow me to use RCurl to get the above tasks accomplished. I have
> looked at the Omegahat RCurl links and the manuals present there but would
> like R users to share their personal experiences and resources they may
> have
> used to use and implement RCurl.
>
>
> Thanks
> Harsh Singhal
>
> "Senior Jedi General"
> Decision Systems
> Mu Sigma Inc.
> Chicago, IL
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]