search for: biganalyt

Hello colleagues, I have tried to use the package bigmemory, biganalytics and biglm. I want to specify a multivariate regression with a weight. I have imported a large dataset with the library(bigmemory). I load the library (biglm) and specified a regression with a weight. But I get everytime an error message like "object not found" or "`weights'...

...em that I run into is when I start to try and run large datasets through them.  The data sets are 10-15 million in record quantity and usually have 15-30 fields (both numerical and categorical). I saw that there were some packages that could deal with large datasets in R - bigmemory, ff, ffdf, biganalytics.  My problem is that I am not much of a coder (and the reason I use the above mentioned GUIs).  These GUIs do show the executable R code in the background - my thought was to run a small sample through the GUI, copy the code, and then incorporate some of the large data packages mentioned abo...

...ain me what this means. I have read the documentation. I don?t understand the documentation for biglm.big.matrix. I found a sentence to weights that is as follow: A one-sided, ?weight = single term formula specifying weights?. Who knows what that means? My Coding is: library(bigmemory) library(biganalytics) library(biglm) help(read.big.matrix) NIKA <- read.big.matrix("G:\\VAR.dat", header=TRUE, type="double", sep="\t") Reg <- biglm.big.matrix(formula = LAannualisiert ~ 0 + VAR01_2008 + VAR02_2008 + VAR03_2008 + VAR04_2008 + VAR05_2008 + VAR06_2008 + VAR07_20...

Hi, I am trying to run the bigmemory example provided on the http://www.bigmemory.org/ The example runs on the "airline data" and generates summary of the csv files:- library(bigmemory) library(biganalytics) x <- read.big.matrix("2005.csv", type="integer", header=TRUE, backingfile="airline.bin", descriptorfile="airline.desc", extraCols="Age") summary(x) This runs fine for the provided csv for year 1987 (size=121MB). However, for big files like...

...on part of the data, to make sure the vector length do not exceed 2^32-1, there's no problem at all. Meaning that the kmeans in R could recognize this type of matrix. If I run the entire matrix, R says "too many elements specified." I have considered the 'bigmemory' and 'biganalytics' packages. But to save the sparse matrix as common CSV file would take approx 70G and 99% being 0. I just don't think it's necessary or efficient to treat it as a dense matrix. It there anyway to deal with the vector length limit? Can I split the whole matrix into small ones and the...

...?q=kmeans+not+give+same+output+different+runs) es pasarle los centroides al algoritmo, pero esto no siempre -mejor dicho, raras veces- es factible. Otra cosa es el algoritmo kmeans que elijas, y aquí lanzo también una pregunta por si alguien lo conoce: probando el kmeans estándar y el bigkmeans de biganalytics (creedme que es una pequeña maravilla que divide el tiempo de ejecución por 2 ó 3 al menos, y permite manejar objetos que no caben en memoria -usando bigmemory), dan resultados muy diferentes. Lo que más me preocupa es el tamaño de los clusters, pero no veo por ninguna parte que eso se pueda con...

The long-promised revision to bigmemory has arrived, with package 4.2.3 now on CRAN. The mutexes (locks) have been extracted and will be available through package synchronicity (on R-Forge, soon to appear on CRAN). Initial versions of packages biganalytics and bigtabulate are on CRAN, and new versions which resolve the warnings and have streamlined CRAN-friendly configurations will appear shortly. Package bigalgebra will remain on R-Forge for the time being as the user-interface is developed and the configuration possibilities expand. For more i...

The long-promised revision to bigmemory has arrived, with package 4.2.3 now on CRAN. The mutexes (locks) have been extracted and will be available through package synchronicity (on R-Forge, soon to appear on CRAN). Initial versions of packages biganalytics and bigtabulate are on CRAN, and new versions which resolve the warnings and have streamlined CRAN-friendly configurations will appear shortly. Package bigalgebra will remain on R-Forge for the time being as the user-interface is developed and the configuration possibilities expand. For more i...

I'm using bigkmeans in 'biganalytics' to cluster my 60,000 by 600,000 matrix. I'm using a 8 core Linux VM. I have register parallel backend with >registerDoMC() And I checked how many cores registered with >getDoParWorkers() It returns 8, which is the number of cores I have on my machine. And I run the test below,...

Dear helpers, Does anyone have information on the status of bigmemory and R3.0? Will it just take time for the devs to re-code for the new environment? Or is there an alternative for this new version? Thanks Ben Caldwell [[alternative HTML version deleted]]

Hi, I am working with a file (900MB in size) that has around 10 million records (in particular FASTQ records). I am able to read in the file as an object of BStringSet. When I start to manipulate the data, after almost 4 hours, I get the error message as Error: cannot allocate vector of size X.0 Mb (where X was once 160MB and then 180MB). The R version used is 2.11.1. I am not sure how to check

...?q=kmeans+not+give+same+output+different+runs) es pasarle los centroides al algoritmo, pero esto no siempre -mejor dicho, raras veces- es factible. Otra cosa es el algoritmo kmeans que elijas, y aquí lanzo también una pregunta por si alguien lo conoce: probando el kmeans estándar y el bigkmeans de biganalytics (creedme que es una pequeña maravilla que divide el tiempo de ejecución por 2 ó 3 al menos, y permite manejar objetos que no caben en memoria -usando bigmemory), dan resultados muy diferentes. Lo que más me preocupa es el tamaño de los clusters, pero no veo por ninguna parte que eso se pueda con...

I work with problems that have rather large data requirements -- typically a bunch of multigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked

I'm in the process of converting some S3 methods to S4 methods. I have this function : setGeneric("enrichmentCalc", function(rs, organism, seqLen, ...){standardGeneric("enrichmentCalc")}) setMethod("enrichmentCalc", c("GenomeDataList", "BSgenome"), function(rs, organism, seqLen, ...) { ... ... ... })

hola a todos, Me gustaria saber qué haceis cuando el dataset para trabsjar es tan grande que R no lo soporta. Lo troceo con alguna otra herramienta? Cual? Gracias y feliz año! Victoria > _______________________________________________ > R-help-es mailing list > R-help-es en r-project.org <javascript:;> > https://stat.ethz.ch/mailman/listinfo/r-help-es > -- Enviado con

...s/benchmark The benchmark package provides a toolbox for setup, execution and analysis of benchmark experiments. Main focus is the analysis of data accumulating during the execution -- one primary objective is the statistical correct computation of the candidate algorithms' order. * biganalytics (1.0.14) Jay and Mike http://crantastic.org/packages/biganalytics This package extends the bigmemory package with various analytics. Functions bigkmeans and binit may also be used with native R objects. For tapply-like functions, the bigtabulate package may also be helpful. For line...