If your inout loop is ONLY storing data, then failures are either in the
retrieval or storing actions. If you have sufficient storage space then the
latter should be reliable. Network failures are however inevitable... you have
no choice but to deal with them somehow... the internet is intrinsically
unreliable. If missing one stock invalidates your analysis then you have set
yourself up for failure and no amount of tricky coding will save you.
On February 6, 2022 11:41:54 AM PST, akshay kulkarni <akshay_e4 at
hotmail.com> wrote:>dear jeff,
> restarting the storing of webscraped data should be easy,
but what if it takes more than a few tries? I am using a very costly AWS EC2
instance and am hard pressed for funds. You should know that it is very
difficult for me to afford even one break. I think the execution should take
about two hours, but even one break could make it 5 - 6 hours (including fixing
of the bug). What if I use save () in the body of the loop? It might make it run
for about three hours, but it is better than 5 -6 hours right?
>
>THanking you,
>Yours sincerely,
>AKSHAY M KULKARNI
>________________________________
>From: Jeff Newmiller <jdnewmil at dcn.davis.ca.us>
>Sent: Monday, February 7, 2022 12:57 AM
>To: r-help at r-project.org <r-help at r-project.org>; Ebert,Timothy
Aaron <tebert at ufl.edu>; akshay kulkarni <akshay_e4 at
hotmail.com>; R help Mailing list <r-help at r-project.org>
>Subject: Re: [R] handling breaks in a for loop
>
>Perhaps a more conventional division of functional concerns should be
applied? Separate collection of data from computation. Input functions should do
input, analysis functions should do analysis, and output functions should do
output... effectively lowering the cost of failure. If your first stage is
simply storing the webscraped data, restarting should be easy.
>
>On February 6, 2022 11:03:19 AM PST, "Ebert,Timothy Aaron"
<tebert at ufl.edu> wrote:
>>Is the bug in the data or in the code?
>>Can you identify the cause? Can you get a subset of the data to find the
bug using fewer resources?
>>If you can identify the root cause, then you can set an error trap.
>>If the error is a memory problem, they you either need to save,
reallocate memory, or get more memory.
>>
>>Tim
>>
>>-----Original Message-----
>>From: R-help <r-help-bounces at r-project.org> On Behalf Of akshay
kulkarni
>>Sent: Sunday, February 6, 2022 1:56 PM
>>To: R help Mailing list <r-help at r-project.org>
>>Subject: [R] handling breaks in a for loop
>>
>>[External Email]
>>
>>dear members,
>> I have a very large for loop (basically a web
scraping loop) with 500 iterations(over different stocks). I am concerned about
the loop breaking midway.
>>
>>My question is: is there any method to store the succesful iterations
and continue with the iteration (ofcourse, after fixing the bug) where the
exception occured? I can insert a save () function in the body of the loop and
save the succesful iterations but this would slow down the execution speed. Any
other method? I want to avoid trycatch as much as possible because an NA in the
place of a succesful iteration would cost me much.
>>
>>Thanking you,
>>Yours sincerely,
>>AKSHAY M KULKARNI
>>
>> [[alternative HTML version deleted]]
>>
>>______________________________________________
>>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dhelp&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=9PEhQh2kVeAsRzsn7AkP-g&m=F0cPNGUlUW-Cd9RJzYladHn1oypv0cPLA-d9dEO42hExw8rxis6aw3FPV6DtBQU2&s=jmZ4HPdnRH6ive_u-90RBNQQtFrnMey9AZv8MRZGQbc&e>>PLEASE
do read the posting guide
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.R-2Dproject.org_posting-2Dguide.html&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=9PEhQh2kVeAsRzsn7AkP-g&m=F0cPNGUlUW-Cd9RJzYladHn1oypv0cPLA-d9dEO42hExw8rxis6aw3FPV6DtBQU2&s=Tpxb0boNRBRe0_7_FZteYKLLal9zfDXAXImIExuV35o&e>>and
provide commented, minimal, self-contained, reproducible code.
>>
>>______________________________________________
>>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>https://stat.ethz.ch/mailman/listinfo/r-help
>>PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>>and provide commented, minimal, self-contained, reproducible code.
>
>--
>Sent from my phone. Please excuse my brevity.
--
Sent from my phone. Please excuse my brevity.