similar to: bigglm Memory Issues

Hi, Until now, I thought that the results of glm() and bigglm() would coincide. Probably a naive assumption? Anyways, I've been using bigglm() on some datasets I have available. One of the sets has >15M observations. I have 3 continuous predictors (A, B, C) and a binary outcome (Y). And tried the following: m1 <- bigglm(Y~A+B+C, family=binomial(), data=dataset1, chunksize=10e6)

Hi Sir, Thanks for making package available to us. I am facing few problems if you can give some hints: Problem-1: The model summary and residual deviance matched (in the mail below) but I didn't understand why AIC is still different. > AIC(m1) [1] 532965 > AIC(m1big_longer) [1] 101442.9 Problem-2: chunksize argument is there in bigglm but not in biglm, consequently,

Hi, since bigglm doesn't have update, I was wondering how to achieve something like (similar to the example in ff package manual using biglm): first <- TRUE ffrowapply ({ if (first) { first <- FALSE fit <- bigglm(eqn, as.data.frame(bigdata[i1:i2,,drop=FALSE]), chunksize = 10000, family = binomial()) } else { fit <- update(fit,

I am using an example posted in this help forum to work with a file. the head of the file looks like: 988887 2007-03-05 2007-06-01 90 3 5.450 205500.00 999.00 999.000 0.000 0 0 988887 2007-03-06 2007-06-01 90 3 5.450 205500.00 999.00 999.000 0.000 1 0 988887 2007-03-07 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100 2 0 988887 2007-03-08 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100

Hi, there Since glm cannot handle factors very well. I try to use bigglm like this: logit_model <- bigglm(responser~var1+var2+var3, data, chunksize=1000, family=binomial(), weights=~trial, sandwich=FALSE) fitted <- predict(logit_model, data) only var2 is factor, var1 and var3 are numeric. I expect fitted should be a vector of value falls in (0,1) However, I get something like this:

Dear all, I am using the bigglm package to fit a few GLM's to a large dataset (3 million rows, 6 columns). While trying to fit a Poisson GLM I noticed that the coefficient estimates were very different from what I obtained when estimating the model on a smaller dataset using glm(), I wrote a very basic toy example to compare the results of bigglm() against a glm() call. Consider the

This is to submit a commented example function for use in the data argument to the bigglm(biglm) function, when you want to read the data from a file (instead of a URL), or rescale or modify the data before fitting the model. In the hope that this may be of help to someone out there. make.data <- function (filename, chunksize, ...) { conn<-NULL; function (reset=FALSE) { if

Dear all, I have a large data set and would like to fit a logistic regression model using the bigglm function. I need to include an offset in the model but when I do this the bigglm function seems to ignore it. For example, running the two models below produces the same model and the offset is ignored bigglm(y~x,offset=z,data=Test,family=binomial(link = "logit"))

G'morning What does the error message "Error in x %*% coef(object) : non- conformable arguments" indicate when calculating the response values for newdata with a model from bigglm (in package biglm), and how can I debug it? I am attempting to do Monte Carlo simulations, which may explain the loop in the code that follows. After the code I have included the output, which shows that

If a package defined an S3 generic and an S4 generic for the same function (so as to add methods for S4 classes to the existing code), how do I set up the namespace to have them exported? With import(stats) exportMethods(bigglm) importClassesFrom(DBI) useDynLib(biglm) export(biglm) export(bigglm) in NAMESPACE, the S3 generic is not exported. > methods("bigglm") [1] bigglm.RODBC*

Hola, estoy intentando hacer una regresión logística entre la primera columna de mi data.table (In.hospital_death) y otras dos (GSV y BUN) , me da el error de abajo, he intentado eliminar las filas con valor NA por si esta función no lo admite, pero sigue dando el mismo error. ¿Alguien sabe porqué ocurre? (probé previamente a usar la función glm pero obtenía out of memory) library(XLConnect)

Hi, I have a large table in Postgresql (result of an MCMC simulation, with 1 million rows) and I would like to retrive colums (correspond to variables) using RODBC. I have a column called "index" which is used to order rows. Unfortunately, sqlQuery can't return all the values from a column at once (RODBC complains about lack of memory). So I am using the following code:

Gracias! El 15 de junio de 2015, 16:54, Freddy Omar López Quintero < freddy.vate01 en gmail.com> escribió: > ?Holap.? > > ran out of iterations and failed to converge > > > ?Prueba aumentando el número de iteraciones, con el argumento maxit: > > ?GLM <- bigglm(In.hospital_death ~ GCS + BUN, data = DatosGLM, family = >> binomial(logit), maxit=1000)? >

I am trying to write a customized app using C that would fetch voice file from SQL Server 2000 using ODBC and FREETDS. Currently I am only able to fetch first 63 KB chunk from the DB, and not able to fetch the rest of the file, below is the code that i am using to do so, fd = open(fullpath, O_RDWR | O_CREAT | O_TRUNC, 0770); if (fd < 0) { ast_log(LOG_WARNING, "Failed to write

In file vorbisfile.c, function int ov_pcm_seek_page(OggVorbis_File *vf,ogg_int64_t pos) the calculation of the approximate seek point causes an overflow for very large files and on pure 32 bit architectures. /* take a (pretty decent) guess. */ bisect=begin + (target-begintime)*(end-begin)/(endtime-begintime) - CHUNKSIZE; The temporary expression (target-begintime)*(end-begin)

Hi, I'm new to R and would like to know, how one can populate the list with array data. I'm reading a tab separated table in R. The data in the table looks something like this. #Table Data Comp    A    B    C Extracellular    103    268    535759 Nucleus    45603    47783    442744 #R code myData <- read.table("table.data",                 header=T,                

Hi Everybody, Thanks in advance for your help. This is my first time using the foreach statement and I cant get it to work properly so here is what i have test<-function(){ repeat { cand2[l-1]<-rinvgamma(1,phi,lambda[l-1]) q2<-dinvgamma(cand2[l-1],phi,lambda[l-1]) p2<-cand2[l-1]^-1.5*exp(-y[l]^2/(2*cand2[l-1]))*exp(-((log(cand2[l-1])-mu_t_cand[l-1])^2)/2*sigmasq)

Hola R, tengo las siguientes preguntas: Pregunta 1: Cargar las tablas de los datos de peliculas en R usando `ff`. Cómo se construye una columna nueva que de, para cada cliente y cada evaluación, de el número de días que han pasado desde la primera evaluación del cliente? Qué función se utiliza para verifica el consumo de memoria en las operaciones? Pregunta 2: Cómo se corre un modelo de

The new code is dedicated to calculate free inodes number of all inode_allocs, then return the info to userpace in terms of an array. Specially, flag 'OCFS2_INFO_FL_NON_COHERENT', manipulated by '--cluster-coherent' from userspace, is now going to be involved. setting the flag on means no cluster coherency considered, usually, userspace tools choose none-coherency strategy by

Hello list, I am trying to avoid "Rifying" some of my SAS code to generate ROC plots, and the logistic.display() and lroc() functions in the epicalc package do what I want. However, I must generate my logistic model with bigglm because I have 1) limited hardware, 2) ~2.5 million rows, and 4 categorical and 2 continuous independent variables. When I attempt to invoke epicalc's