similar to: R 3.0.1 : parallel collection triggers "long memory not supported yet"

I have two data sets about lots of companies' stock and fiscal data. One is monthly data with about 144,000 lines, and the other is quaterly with about 56,000. Each data set takes different company code. I need to merge these two together. I read both ask cvs. And the other file with corresponding firm code. Now I have three data sets. return$PERMNO, account$GVKEY. id is the data frames

Hi, I am facing a very weird problem with parallel::mclapply. I have a script which does some data wrangling on an input dataset in parallel and then writes the results to disk. I have been using this script daily for more than one year always on an EC2 instance launched from the same AMI (no updates installed after launch) and processed thousands of different input data sets successfully. I now

Hi Andreas, the error is reported when some child process cannot send results to the master process, which originates from an error returned by write() - when write() returns -1 or 0. The logic around the writing has not changed since R 3.5.2. It should not be related to the printing in the child, only to returning the value. The problem may be originating from the execution environment,

Hi Andreas, thank you very much, good job finding it was EBADF. Now the question is why the pipe has been closed prematurely; it could be accidentally by R (a race condition in the cleanup code in fork.c) or possibly by some other code running in the same process (maybe the R program itself or some other code it runs). Maybe we can take this off the list and come back when we know the cause

I've been hacking with parallel package for some time and built a parallel processing framework with it. However, although very rarely, I did notice "ignoring SIGPIPE signal" error every now and then. After a deep dig into the source code, I think I found something worth noticing. In short, wring to pipe in the C function mc_exit(SEXP sRes) may cause a SIGPIPE. Code from

Hello everyone, I'm trying to parallelize an R script I have written. To do this, I am first trying to use the multicore package, because I've had some previous success with that. The function I'm trying to parallelize is illumqc. I'd like to create a separate process for each of 8 files, contained in the vector "files". Below is my code: for(i in

I have an array called "stocks" which contains numeric dates, ticker symbols,prices, etc. > stocks[1:3,] DATE TICKER PERMNO EXCHCD TSYMBOL TRDSTAT SHROUT PRC RET 1 19950131 EWST 10001 3 EWST A 2224 -7.75000 -0.031250 2 19950228 EWST 10001 3 EWST A 2224 7.54688 -0.026210 3 19950331 EWST

Have read the latest code, but I still don't understand why mc_exit needs to write zero on exit. If a child closes its pipe, parent will know that on next select. Best, Yijiang Tomas Kalibera <tomas.kalibera at gmail.com> ?2019?5?20??? ??10:52??? > > This issue has already been addressed in 76462 (R-devel) and also ported > to R-patched. In fact rmChild() is used in

Hi everyone! I have a question about data processing efficiency. My data are as follows: I have a data set on quarterly institutional ownership of equities; some of them have had recent IPOs, some have not (I have a binary flag set). The total dataset size is 700k+ rows. My goal is this: For every quarter since issue for each IPO, I need to find a "matched" firm in the same

Dear all, I want to do piecewise CAPM linear regression in R: RRiskArb−Rf = (1−δ)[αMktLow+βMktLow(RMkt−Rf)] + δ[αMkt High +βMkt High(RMkt −Rf )] where δ is a dummy variable if the excess return on the value-weighted CRSP index is above a threshold level and zero otherwise. and at the same time add the restriction: αMkt Low + βMkt Low · Threshold = αMkt High + βMkt High · Threshold to ensure

Inthe code below I was trying to to obtain the GMM estimates for CAPM (REGRESSION) for 36 stocks each have 180 observations,however it only gives me one output rather than 36. In SAS i would just put in a *By statement*. I have a variable TICKER that categorize them into 36 groups. *How can I obtain all 36 output instead of just one.* **

Hi, I'm slowly working through Tsay's "Analysis of Financial Time Series" 3rd ed. ?I'm trying to replicate Table 2.1 on p.47, which gives PACF, AIC, and BIC for the monthly simple returns of the CRSP value-weighted index. The data: http://faculty.chicagobooth.edu/ruey.tsay/teaching/fts3/m-ibm3dx2608.txt > da <-

Hi again, One important correction of my first message: I misinterpreted the output. Actually in that R session 2 input files were processed one after the other in a loop. The first (with 88 parts went fine). The second (with 85 parts) produced the sendMaster errors and failed. If (in a new session via Rscript) I only process the second input file it will work. The other observations on R vs

Hi, I am experiencing a problem related to serialization behavior in 2.15.0 beta (binary installed from Debian unstable) and 2.16.0 (from svn) that is not present in 2.14.2 (binary from Debian testing). I don't fully understand the problem. Also, I tried but have not yet been able to create a small, self-contained example that reproduces the problem. However, I do have a large, not

Hi Tomas, Thanks for your prompt reply and your offer to help. I might need to get back to this since I am not too experienced in debugging these kinds of issues. Anyway, I gave it a try and I think I have found the immediate cause: I installed the debug symbols (r-base-core-dbg), placed https://github.com/wch/r-source/blob/tags/R-3-5-2/src/library/parallel/src/fork.c in cwd and changed the

Hi all, With the help of Tomas, I was able to track the issue down: Prior to R v3.6.0 the parallel package passes an uninitialized variable as the file descriptor argument to the close system call. In my particular R session this uninitialized variable (reproducibly) was holding the value 7, which corresponded to the file descriptor of the write end of the pipe the second child would use to

Hi all, With the help of Tomas, I was able to track the issue down: Prior to R v3.6.0 the parallel package passes an uninitialized variable as the file descriptor argument to the close system call. In my particular R session this uninitialized variable (reproducibly) was holding the value 7, which corresponded to the file descriptor of the write end of the pipe the second child would use to

I have an lapply function call that I want to parallelize. Below is a very simplified version of the code: url_base <- "https://cloud.r-project.org/src/contrib/" files <- c("A3_1.0.0.tar.gz", "ABC.RAP_0.9.0.tar.gz") res <- parallel::mclapply(files, function(s) download.file(paste0(url_base, s), s)) Instead of download a couple of files in parallel, I get a

Hi there, I had an error when using RSQLite. My code and the error message are the following: > dbGetQuery(sql.dispersion, + "create table monthly_data_temp as + select a.*, b.industry + from monthly_data as a left join + siccd_industry_ff49 as b + on a.siccd = b.siccd + order by permno, date") Error

I've been hacking with parallel package for some time and built a parallel processing framework with it. However, although very rarely, I did notice "ignoring SIGPIPE signal" error every now and then. After a deep dig into the source code, I think I found something worth noticing. In short, wring to pipe in the C function mc_exit(SEXP sRes) may cause a SIGPIPE. Code from