similar to: bigglm "update" with ff

Hi, Until now, I thought that the results of glm() and bigglm() would coincide. Probably a naive assumption? Anyways, I've been using bigglm() on some datasets I have available. One of the sets has >15M observations. I have 3 continuous predictors (A, B, C) and a binary outcome (Y). And tried the following: m1 <- bigglm(Y~A+B+C, family=binomial(), data=dataset1, chunksize=10e6)

Hi Sir, Thanks for making package available to us. I am facing few problems if you can give some hints: Problem-1: The model summary and residual deviance matched (in the mail below) but I didn't understand why AIC is still different. > AIC(m1) [1] 532965 > AIC(m1big_longer) [1] 101442.9 Problem-2: chunksize argument is there in bigglm but not in biglm, consequently,

Hi, there Since glm cannot handle factors very well. I try to use bigglm like this: logit_model <- bigglm(responser~var1+var2+var3, data, chunksize=1000, family=binomial(), weights=~trial, sandwich=FALSE) fitted <- predict(logit_model, data) only var2 is factor, var1 and var3 are numeric. I expect fitted should be a vector of value falls in (0,1) However, I get something like this:

Hi all, I'm somewhat of a novice in terms of programming, so I thought I'd come here to seek some help with an issue I'm having. I'm trying to model a glm using bigglm, but in spite of my best efforts, I cannot get it to work! Here is the particular line of code that is giving me trouble: >mod = bigglm(Pres/wt ~ Xdes, data=dat, family=poisson(), weights = ~wt, maxit=100,

Dear all, I am using the bigglm package to fit a few GLM's to a large dataset (3 million rows, 6 columns). While trying to fit a Poisson GLM I noticed that the coefficient estimates were very different from what I obtained when estimating the model on a smaller dataset using glm(), I wrote a very basic toy example to compare the results of bigglm() against a glm() call. Consider the

I am using an example posted in this help forum to work with a file. the head of the file looks like: 988887 2007-03-05 2007-06-01 90 3 5.450 205500.00 999.00 999.000 0.000 0 0 988887 2007-03-06 2007-06-01 90 3 5.450 205500.00 999.00 999.000 0.000 1 0 988887 2007-03-07 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100 2 0 988887 2007-03-08 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100

This is to submit a commented example function for use in the data argument to the bigglm(biglm) function, when you want to read the data from a file (instead of a URL), or rescale or modify the data before fitting the model. In the hope that this may be of help to someone out there. make.data <- function (filename, chunksize, ...) { conn<-NULL; function (reset=FALSE) { if

Dear all, I have a large data set and would like to fit a logistic regression model using the bigglm function. I need to include an offset in the model but when I do this the bigglm function seems to ignore it. For example, running the two models below produces the same model and the offset is ignored bigglm(y~x,offset=z,data=Test,family=binomial(link = "logit"))

G'morning What does the error message "Error in x %*% coef(object) : non- conformable arguments" indicate when calculating the response values for newdata with a model from bigglm (in package biglm), and how can I debug it? I am attempting to do Monte Carlo simulations, which may explain the loop in the code that follows. After the code I have included the output, which shows that

If a package defined an S3 generic and an S4 generic for the same function (so as to add methods for S4 classes to the existing code), how do I set up the namespace to have them exported? With import(stats) exportMethods(bigglm) importClassesFrom(DBI) useDynLib(biglm) export(biglm) export(bigglm) in NAMESPACE, the S3 generic is not exported. > methods("bigglm") [1] bigglm.RODBC*

I started rebalancing my 25x2 distributed-replicate volume two days ago. Since then, the memory usage of the rebalance processes has been steadily climbing by 1-2 megabytes per minute. Following http://gluster.org/community/documentation/index.php/High_Memory_Usage, I tried "echo 2 > /proc/sys/vm/drop_caches". This had no effect on the processes' memory usage. Some of the

Hi, I am using version 2.1-1 of the ff package. I have a data set with 80 million rows and I need to create a new ffdf object, subseting by values in one of the original ffdf's columns. Here is my code: bigData <- read.table.ffdf(file="/data/demodata/data/smallData.txt", next.rows=1e5, head=TRUE, sep="|") dim(bigData) N <- nrow(bigData);N select <- ff(

I started rebalancing my volume after updating from 3.2.7 to 3.3.1. After a few hours, I noticed a large number of failures in the rebalance status: > Node Rebalanced-files size scanned failures > status > --------- ----------- ----------- ----------- ----------- > ------------ > localhost 0 0Bytes 4288805

I have always assumed that having a package in the 'Depends' field would automatically also?import?the namespace. However, it seems that in R 2.15, dependencies do not become available until the package is actually?attached?to the searchpath. Is this intended behavior? The problem appears as follows: Suppose there is a package 'Child' which?Depends, but does not explicitly

Si, Bueno en realidad, estamos en un curso de bigdata y estan haciendo mucha scosas con Java, pero yo intento pasarlo a Python tood, ya uqe me parece un lenguaje mucho mas simple y más parecido a R Y si, lo que quiero principalmente es aplicar machine learning sobre conjuntos de datos enormes. Alguna recomendación? ________________________________ De: Carlos Ortega <cof en

Hola, estoy intentando hacer una regresión logística entre la primera columna de mi data.table (In.hospital_death) y otras dos (GSV y BUN) , me da el error de abajo, he intentado eliminar las filas con valor NA por si esta función no lo admite, pero sigue dando el mismo error. ¿Alguien sabe porqué ocurre? (probé previamente a usar la función glm pero obtenía out of memory) library(XLConnect)

Aunque es un poco offtopic, creeis necesario aprender java para temas de bigdata o con python es suficiente Gracias Jesús [[alternative HTML version deleted]]

Hola R, tengo las siguientes preguntas: Pregunta 1: Cargar las tablas de los datos de peliculas en R usando `ff`. Cómo se construye una columna nueva que de, para cada cliente y cada evaluación, de el número de días que han pasado desde la primera evaluación del cliente? Qué función se utiliza para verifica el consumo de memoria en las operaciones? Pregunta 2: Cómo se corre un modelo de

Hello, Is it possible to execute functions (outside the ui and server shiny environments) after reading data using reactiveFileReader() ? For example, I'd like to fit a linear model on data read using reactiveFileReader() outside ui/server. library(shiny) library(dplyr) bigData <- reactiveFileReader(1000, NULL, 'data.csv', read.csv) fit <- lm(y ~., data = bigData()) ui

Gracias! El 15 de junio de 2015, 16:54, Freddy Omar López Quintero < freddy.vate01 en gmail.com> escribió: > ?Holap.? > > ran out of iterations and failed to converge > > > ?Prueba aumentando el número de iteraciones, con el argumento maxit: > > ?GLM <- bigglm(In.hospital_death ~ GCS + BUN, data = DatosGLM, family = >> binomial(logit), maxit=1000)? >