search for: chunksiz

The new code is dedicated to calculate free inodes number of all inode_allocs, then return the info to userpace in terms of an array. Specially, flag 'OCFS2_INFO_FL_NON_COHERENT', manipulated by '--cluster-coherent' from userspace, is now going to be involved. setting the flag on means no cluster coherency considered, usually, userspace tools choose none-coherency strategy by

...t;index" which is used to order rows. Unfortunately, sqlQuery can't return all the values from a column at once (RODBC complains about lack of memory). So I am using the following code: getcolumns <- function(channel, tablename, colnames, totalrows, ordered=TRUE,chunksize=1e5) { r <- matrix(double(0),totalrows,length(colnames)) for (i in 1:ceiling(totalrows/chunksize)) { cat(".") r[((i-1)*chunksize+1):(i*chunksize)] <- as.matrix( sqlQuery(channel, paste("SELECT", paste(colnames,collapse=", "),...

Linux 3.11 added respective support, so I think we should follow suit. 1: xen: add LZ4 decompression support 2: libxc: add LZ4 decompression support Signed-off-by: Jan Beulich <jbeulich@suse.com>

...imple application to decompress a set of Ogg files to raw PCM format, using vorbisfile, because that is the simplest way of doing it. These Ogg files are sounds for a game, and thus sometimes very short (about .25 s). However, vorbisfile has severe problems with Ogg files that are smaller than the CHUNKSIZE which is #defined at the top of vorbisfile.c (8150 bytes at the moment). In that case, the ov_open() call takes an unusually long amount of time (10-30 s on a P3-450), and the first ov_read() call after that will _always_ return 0, indicating EOF. The result is that decoding with vorbisfile takes...

...", colsize); //PRINTING COLSIZE ON CLI if (fd > -1) { char tmp[1]=""; lseek(fd, fdlen - 1, SEEK_SET); if (write(fd, tmp, 1) != 1) { close(fd); res = -1; goto free_res; } } if (fd > -1){ //Trying to fetch data in chunks for (offset = 0; offset < colsize; offset += CHUNKSIZE) { if ((fdm = mmap(NULL, CHUNKSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset)) == MAP_FAILED) { ast_log(LOG_WARNING, "Could not mmap the output file: %s (%d)\n", strerror(errno), errno); goto free_res; } else { res = SQLGetData(stmt, x + 1, SQL_BINARY, fdm, CHUNKSIZE, NULL);...

...obably a naive assumption? Anyways, I've been using bigglm() on some datasets I have available. One of the sets has >15M observations. I have 3 continuous predictors (A, B, C) and a binary outcome (Y). And tried the following: m1 <- bigglm(Y~A+B+C, family=binomial(), data=dataset1, chunksize=10e6) m2 <- bigglm(Y~A*B+C, family=binomial(), data=dataset1, chunksize=10e6) imp <- m1$deviance-m2$deviance For my surprise "imp" was negative. I then tried the same models, using glm() instead... and as I expected, "imp" was positive. I also noticed differences on...

Hi, since bigglm doesn't have update, I was wondering how to achieve something like (similar to the example in ff package manual using biglm): first <- TRUE ffrowapply ({ if (first) { first <- FALSE fit <- bigglm(eqn, as.data.frame(bigdata[i1:i2,,drop=FALSE]), chunksize = 10000, family = binomial()) } else { fit <- update(fit, as.data.frame(bigdata[i1:i2,,drop=FALSE]), chunksize = 10000, family = binomial()) } }, X=bigdata, VERBOSE = TRUE, BATCHSIZE = nmax) Many thanks. Yuesheng [[alternative HTML version deleted]]

...lp with an issue I'm having. I'm trying to model a glm using bigglm, but in spite of my best efforts, I cannot get it to work! Here is the particular line of code that is giving me trouble: >mod = bigglm(Pres/wt ~ Xdes, data=dat, family=poisson(), weights = ~wt, maxit=100, tol=1.e-10, chunksize=(dim(Xdes)[1] + 1)) When I attempt to run this, it gives me the following error: >Error: cannot allocate vector of size 255.9 Mb However, I had set the maximum memory to 4095 Mb and had used only 926 Mb to that point. If I try to use a smaller chunksize, I get this error: >Error in mod...

...n int ov_pcm_seek_page(OggVorbis_File *vf,ogg_int64_t pos) the calculation of the approximate seek point causes an overflow for very large files and on pure 32 bit architectures. /* take a (pretty decent) guess. */ bisect=begin + (target-begintime)*(end-begin)/(endtime-begintime) - CHUNKSIZE; The temporary expression (target-begintime)*(end-begin) grows with the square of the file size. This effectively limits the file size to the square root of the maximum value of ogg_int64_t. In consequence the file is scanned from the very beginning which is pretty slow over a network. FIX:...

...999.000 -0.100 8 0 988887 2007-03-14 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100 9 0 988887 2007-03-15 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100 10 0 988887 2007-03-16 2007-06-01 90 3 5.450 205500.00 999.00 999.000 -0.100 11 0 the code is: make.data <- function (filename, chunksize, ...) { conn<-NULL; function (reset=FALSE) { if (reset) { if (!is.null(conn)) { close(conn); }; conn <<- file (description=filename, open="r"); } else { rval <- read.table (conn, nrows=chunksize,sep=' ', sk...

...les. I have a file called "silence" that's just a silent wav file. It compresses down to barely more than the header, so it's _really_ small. When I try to play it, it goes into an infinite loop in vorbisfile.c, line 159. In RC3, that code was: while(offset==-1){ begin-=CHUNKSIZE; if(begin<0) begin=0; _seek_helper(vf,begin); while(vf->offset<begin+CHUNKSIZE){ ret=_get_next_page(vf,og,begin+CHUNKSIZE-vf->offset); if(ret==OV_EREAD)return(OV_EREAD); if(ret<0){ break; }else{ offset=ret; } } }...

...making package available to us. I am facing few problems if you can give some hints: Problem-1: The model summary and residual deviance matched (in the mail below) but I didn't understand why AIC is still different. > AIC(m1) [1] 532965 > AIC(m1big_longer) [1] 101442.9 Problem-2: chunksize argument is there in bigglm but not in biglm, consequently, udate.biglm is there, but not update.bigglm Is my observation correct? If yes, why is this difference? Regards Utkarsh / / From: Thomas Lumley <tlumley_at_u.washington.edu <mailto:tlumley_at_u.washington.edu?Subject=Re:%20%5BR...

Hi, there Since glm cannot handle factors very well. I try to use bigglm like this: logit_model <- bigglm(responser~var1+var2+var3, data, chunksize=1000, family=binomial(), weights=~trial, sandwich=FALSE) fitted <- predict(logit_model, data) only var2 is factor, var1 and var3 are numeric. I expect fitted should be a vector of value falls in (0,1) However, I get something like this: str(fitted) num [1:260617, 1] -0.0564 -0.0564 -0.1817...

...ented example function for use in the data argument to the bigglm(biglm) function, when you want to read the data from a file (instead of a URL), or rescale or modify the data before fitting the model. In the hope that this may be of help to someone out there. make.data <- function (filename, chunksize, ...) { conn<-NULL; function (reset=FALSE) { if (reset) { if (!is.null(conn)) { close(conn); }; # This is for a file. # For other methods, see: help("connections") # and replace the following definition of conn # (and possibly the r...

...and2[l-1],phi,lambda[l-1]) p2<-cand2[l-1]^-1.5*exp(-y[l]^2/(2*cand2[l-1]))*exp(-((log(cand2[l-1])-mu_t_cand[l-1])^2)/2*sigmasq) ratio<-p2/(c[l-1]*q2) if (runif(1)< ratio) {break} } f2[l-1]<-min(p2,c[l-1]*q2) }} foreach(l=2:(n-1),.combine=c,.packages='MCMCpack',.options.nws=li st(chunkSize=250)) %dopar% test() even though i've created the array earlier called cand2 when i run this code i get the error object "cand2" not found. What I'm trying to do is the rejection method for my cand2. i need cand2 and f2 as vectors as output both of length l-2 in the correct ord...

Hi all, we got a new test system here and I just also tested btrfs raid6 on that. Write performance is slightly lower than hw-raid (LSI megasas) and md-raid6, but it probably would be much better than any of these two, if it wouldn''t read all the during the writes. Is this a known issue? This is with linux-3.9.2. Thanks, Bernd -- To unsubscribe from this list: send the line

...ar_ptr = arena_for_chunk(p); (gdb) 3445 (void)mutex_lock(&ar_ptr->mutex); (gdb) 3447 _int_free(ar_ptr, mem); (gdb) Couldn't get registers: No such process. (gdb) [Switching to Thread -1216079680 (LWP 4132)] 4291 p = mem2chunk(mem); (gdb) 4278 { (gdb) 4292 size = chunksize(p); (gdb) 4298 if (__builtin_expect ((uintptr_t) p > (uintptr_t) -size, 0) (gdb) 4307 if (__builtin_expect (size < MINSIZE, 0)) (gdb) 4320 if ((unsigned long)(size) <= (unsigned long)(get_max_fast ()) (gdb) 4360 else if (!chunk_is_mmapped(p)) { (gdb) 4365 if (__...

...numeric"), contains="numeric" #so that it inherits from ‘vector’ without assuming logical data ) setGeneric("set.chunk", function(x,...) standardGeneric("set.chunk")) setMethod("set.chunk", signature(x = "range.vec"), function (x, chunksize=100, chunkpos=1) { #This function extracts a chunk of data from the range.vec object. begin <- x@start + (chunkpos - 1)*x@step end <- x@start + (chunkpos + chunksize - 2)*x@step data <- seq(begin, end, x@step) #calculate values in data chunk #get r...

Dear R people, I have a very big tab-delim txt file with header and I only want to import several columns into R. I checked the options for "read.table" and only found "nrows" which lets you specify the maximum number of rows to read in. Although I can use some text editors (e.g., wordpad) to edit the txt file first before running R, I feel it?s not very convenient. The

So I have been playing with a RAID 10 f2 ( 2 disks far layout) setup...thanks for all of the advice..Now I am playing with the format and want to make sure I have it setup the best that I can, my raid was built using the raid 10 option with 2 disks with the layout=far, chunk size 512....now I read all of the docs I could find about format and stride and stripe size and this is what i came up