Displaying 20 results from an estimated 500 matches similar to: "bigmemory package woes"
2009 Jun 02
2
bigmemory - extracting submatrix from big.matrix object
I am using the library(bigmemory) to handle large datasets, say 1 GB,
and facing following problems. Any hints from anybody can be helpful.
_Problem-1:
_
I am using "read.big.matrix" function to create a filebacked big matrix
of my data and get the following warning:
> x =
read.big.matrix("/home/utkarsh.s/data.csv",header=T,type="double",shared=T,backingfile
2010 Aug 11
1
Bigmemory: Error Running Example
Hi,
I am trying to run the bigmemory example provided on the
http://www.bigmemory.org/
The example runs on the "airline data" and generates summary of the csv
files:-
library(bigmemory)
library(biganalytics)
x <- read.big.matrix("2005.csv", type="integer", header=TRUE,
backingfile="airline.bin",
descriptorfile="airline.desc",
2010 Dec 17
1
[Fwd: adding more columns in big.matrix object of bigmemory package]
Hi,
With reference to the mail below, I have large datasets, coming from various
different sources, which I can read into filebacked big.matrix using library
bigmemory. I want to merge them all into one 'big.matrix' object. (Later, I
want to run regression using library 'biglm').
I am unsuccessfully trying to do this from quite some time now. Can you
please
2012 May 05
2
looking for adice on bigmemory framework with C++ and java interoperability
I work with problems that have rather large data requirements -- typically
a bunch of multigig arrays. Given how generous R is with using memory, the
only way for me to work with R has been to use bigmatrices from bigmemory
package. One thing that is missing a bit is interoperability of bigmatrices
with C++ and possibly java. What i mean by that is API that would allow
read and write filebacked
2011 Sep 29
1
efficient coding with foreach and bigmemory
I recently learned about the bigmemory and foreach packages and am trying
to use them to help me create a very large matrix. Without those
packages, I can create the type of matrix that I want with 10 columns and
5e6 rows. I would like to be able to scale up to 5e9 rows, or more, if
possible.
I have created a simplified example of what I'm trying to do, below. The
first part of the
2019 Jul 19
1
difficulty with sanitizer using bigmemory
Dear all,
bigKRLS, which has been on CRAN for a couple of years, had to be pulled
recently due to what seems to be a sanitizer issue stemming from its use of
bigmemory. bigKRLS works fine (we?ve used it ourselves on many different
platforms and have had over 15,000 downloads without an end user reporting
difficulties because of this issue). Unfortunately, we have been unable to
reproduce the
2011 Nov 30
1
Error adding Bigmemory package
I am trying to add the bigmemory packages but I get the following error
message:
In file included from bigmemory.cpp:14:0:
../inst/include/bigmemory/isna.hpp: In function 'bool neginf(double)':
../inst/include/bigmemory/isna.hpp:22:57: error: 'isinf' was not declared in
this scope
make: *** [bigmemory.o] Error 1
ERROR: compilation failed for package 'bigmemory'
* removing
2011 Jun 24
1
Installation of bigmemory fails
Hello All,
I tried to intall the bigmemory package from a CRAN mirror site and
received the following output while installing. Any idea what's going
on and how to fix it? The system details are provided below.
--------------------- begin error messages -----------------------
* installing *source* package 'bigmemory' ...
checking for Sun Studio compiler...no
checking for
2013 Apr 29
2
bigmemory and R 3.0
Dear helpers,
Does anyone have information on the status of bigmemory and R3.0? Will it
just take time for the devs to re-code for the new environment? Or is there
an alternative for this new version?
Thanks
Ben Caldwell
[[alternative HTML version deleted]]
2010 Sep 08
1
bigmemory doubt
Hi,
Is it possible for me to read data from shared memory created by a vc++ program into R using bigmemory?
[[alternative HTML version deleted]]
2010 Feb 22
1
big panel: filehash, bigmemory or other
Dear R-list
I'm on my way to start a new project on a rather big panel, consisting
of approximately 8 million observations in 30 waves of data and about
15 variables. I have a similar data set that is approximately 7
gigabytes in size.
Until now I have done my data management in SAS, and Stata, mostly
identifying spells, counting events in intervals, and a like, but I
would like to
2012 May 11
1
bigmemory
To answer your first question about read.big.matrix(), we don't know what
your acc3.dat file is, but it doesn't appear to have been detected as a
standard file (like a CSV file) or -- perhaps -- doesn't even exist (or
doesn't exist in your current directory)?
Next:
> In addition, I am planning to do a multiple imputation with MICE package
> using the data read by bigmemory
2012 Oct 18
3
bigmemory for dataframes?
Hi Folks,
I've been bumping my head against the 4GB limit for 32-bit R. I can't
go to 64-bit R due to package compatibility issues (ROBDC - possible but
painful, xlsReadWrite - not possible, and others). I have a number of
big dataframes whose columns all sorts of data types - factor,
character, integer, etc. I run and save models that keep copies of the
modeled data inside the model
2011 Dec 01
1
bigmemory on Solaris
At one point we might have gotten something working (older version?) on
Solaris x86, but were never successful on Solaris sparc that I remember --
it isn't a platform we can test and support. We believe there are problems
with BOOST library compatibilities.
We'll try (again) to clear up the other warnings in the logs, though. !-)
We should also revisit the possibility of a CRAN BOOST
2010 Feb 06
2
question about bigmemory: releasing RAM from a big.matrix that isn't used anymore
Hi all,
I'm on a Linux server with 48Gb RAM. I did the following:
x <- big.matrix(nrow=20000,ncol=500000,type='short',init=0,dimnames=list(1:20000,1:500000))
#Gets around the 2^31 issue - yeah!
in Unix, when I hit the "top" command, I see R is taking up about 18Gb
RAM, even though the object x is 0 bytes in R. That's fine: that's how
bigmemory is supposed to
2013 Mar 20
1
bigmemory: Using backing file as alternate to write.big.matrix
Hi,
Does the backing file of a big.matrix store the contents of entire matrix?
Or does it store the portion of it that is not stored in RAM? In other
words, can the backing file be treated as a file containing the matrix's
full data?
I have been writing my big.matrix objects to disk (write.big.matrix), and
other programs that want to access this matrix then just read it in
(read.big.matrix).
2012 May 04
2
Can't import this 4GB DATASET
Dear Experienced R Practitioners,
I have 4GB .txt data called "dataset.txt" and have attempted to use *ff,
bigmemory, filehash and sqldf *packages to import it, but have had no
success. The readLines output of this data is:
readLines("dataset.txt",n=20)
[1] " "
2009 Jul 20
2
kmeans.big.matrix
Hi,
I'm playing around with the 'bigmemory' package, and I have finally
managed to create some really big matrices. However, only now I
realize that there may not be functions made for what I want to do
with the matrices...
I would like to perform a cluster analysis based on a big.matrix.
Googling around I have found indications that a certain
kmeans.big.matrix() function should
2009 Feb 25
3
Using very large matrix
Dear friends,
I have to use a very large matrix. Something of the sort of
matrix(80000,80000,n) .... where n is something numeric of the sort 0.xxxxxx
I have not found a way of doing it. I keep getting the error
Error in matrix(nrow = 80000, ncol = 80000, 0.2) : too many elements specified
Any suggestions? I have searched the mailing list, but to no avail.
Best,
--
Corrado Topi
Global
2012 May 09
2
ergm model, nodematch with diff=T
Dear all,
I am new to network analysis, but since I have good data I started to read
about it and learned how to use the ergm and related packages. I generally
get interesting results, but when I run a model including sociality and
selective mixing effects for different groups, the model runs (and
converges) but I get a warning as follows:
mod <- ergm(network ~ edges + gwesp(0, fixed=T) +