dear jeff,
restarting the storing of webscraped data should be easy, but
what if it takes more than a few tries? I am using a very costly AWS EC2
instance and am hard pressed for funds. You should know that it is very
difficult for me to afford even one break. I think the execution should take
about two hours, but even one break could make it 5 - 6 hours (including fixing
of the bug). What if I use save () in the body of the loop? It might make it run
for about three hours, but it is better than 5 -6 hours right?
THanking you,
Yours sincerely,
AKSHAY M KULKARNI
________________________________
From: Jeff Newmiller <jdnewmil at dcn.davis.ca.us>
Sent: Monday, February 7, 2022 12:57 AM
To: r-help at r-project.org <r-help at r-project.org>; Ebert,Timothy Aaron
<tebert at ufl.edu>; akshay kulkarni <akshay_e4 at hotmail.com>; R
help Mailing list <r-help at r-project.org>
Subject: Re: [R] handling breaks in a for loop
Perhaps a more conventional division of functional concerns should be applied?
Separate collection of data from computation. Input functions should do input,
analysis functions should do analysis, and output functions should do output...
effectively lowering the cost of failure. If your first stage is simply storing
the webscraped data, restarting should be easy.
On February 6, 2022 11:03:19 AM PST, "Ebert,Timothy Aaron" <tebert
at ufl.edu> wrote:>Is the bug in the data or in the code?
>Can you identify the cause? Can you get a subset of the data to find the bug
using fewer resources?
>If you can identify the root cause, then you can set an error trap.
>If the error is a memory problem, they you either need to save, reallocate
memory, or get more memory.
>
>Tim
>
>-----Original Message-----
>From: R-help <r-help-bounces at r-project.org> On Behalf Of akshay
kulkarni
>Sent: Sunday, February 6, 2022 1:56 PM
>To: R help Mailing list <r-help at r-project.org>
>Subject: [R] handling breaks in a for loop
>
>[External Email]
>
>dear members,
> I have a very large for loop (basically a web
scraping loop) with 500 iterations(over different stocks). I am concerned about
the loop breaking midway.
>
>My question is: is there any method to store the succesful iterations and
continue with the iteration (ofcourse, after fixing the bug) where the exception
occured? I can insert a save () function in the body of the loop and save the
succesful iterations but this would slow down the execution speed. Any other
method? I want to avoid trycatch as much as possible because an NA in the place
of a succesful iteration would cost me much.
>
>Thanking you,
>Yours sincerely,
>AKSHAY M KULKARNI
>
> [[alternative HTML version deleted]]
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dhelp&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=9PEhQh2kVeAsRzsn7AkP-g&m=F0cPNGUlUW-Cd9RJzYladHn1oypv0cPLA-d9dEO42hExw8rxis6aw3FPV6DtBQU2&s=jmZ4HPdnRH6ive_u-90RBNQQtFrnMey9AZv8MRZGQbc&e>PLEASE
do read the posting guide
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.R-2Dproject.org_posting-2Dguide.html&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=9PEhQh2kVeAsRzsn7AkP-g&m=F0cPNGUlUW-Cd9RJzYladHn1oypv0cPLA-d9dEO42hExw8rxis6aw3FPV6DtBQU2&s=Tpxb0boNRBRe0_7_FZteYKLLal9zfDXAXImIExuV35o&e>and
provide commented, minimal, self-contained, reproducible code.
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
--
Sent from my phone. Please excuse my brevity.
[[alternative HTML version deleted]]