similar to: Reduce memory peak when serializing to raw vectors

Hi, I've been doing some tests using serialize() to a raw vector: df <- data.frame(runif(50e6,1,10)) ser <- serialize(df,NULL) In this example the data frame and the serialized raw vector occupy ~400MB each, for a total of ~800M. However the memory peak during serialize() is ~1.2GB: $ cat /proc/15155/status |grep Vm ... VmHWM: 1207792 kB VmRSS: 817272 kB We work with very

Jorge, what you propose is not possible because the size of the output is unknown, that's why a dynamically growing PStream buffer is used - it cannot be pre-allocated. Cheers, Simon > On Mar 17, 2015, at 1:37 PM, Martinez de Salinas, Jorge <jorge.martinez-de-salinas at hp.com> wrote: > > Hi, > > I've been doing some tests using serialize() to a raw vector: >

In principle, yes (that's what Rserve serialization does), but AFAIR we don't have the infrastructure in place for that. But then you may as well serialize to a connection instead. To be honest I don't see why you would serialize anything big to a vector - you can't really do anything useful with that ... (what you couldn't do with the streaming version). Sent from my iPhone

Presumably one could stream over the data twice, the first to get the size, without storing the data. Slower but more memory efficient, unless I'm missing something. Michael On Tue, Mar 17, 2015 at 2:03 PM, Simon Urbanek <simon.urbanek at r-project.org> wrote: > Jorge, > > what you propose is not possible because the size of the output is > unknown, that's why a

Hi All, I am using RHEL 6.4 on a Dell Server with 32 cores. But in the Cpus_allowed_list only 30 cores are available. Why is that? See the snippet from cat /proc/self/status: Cpus_allowed: 7fff7fff Cpus_allowed_list: 0-14,16-30 The allowed list is same for the init process as well (see below). I didn't see this issue, in RHEL 6.2. Why is that? Is there a way to change it? -miraj

Hi, sorry if this question is trivial or unclear, this is my first venture into mixed C/R programming (I am reasonably experienced in each separately). I am trying to write a serialization function for a format called typedbytes, which is used as an interchange format in Hadoop circles. Since I would need to serialize according to the internal R format many small R objects I looked at the c

I'm trying to efficiently allow conversion of R objects to PostgreSQL bytea (raw binary) datatype within PL/R for persistent storage in Postgres tables. I have found R_serialize() which looks like what I need, -- e.g. R_serialize(object, NULL, FALSE, NULL) -- except that it is marked attribute_hidden. Is there some other externally available interface that I can use? Thanks, Joe

I'm using Wine 1.0 on Puppy Linux 3.01 I wrote an application with a http-server and a socket-server for windows and I like to use this on linux. (Delphi with Indy components) My application works fine on windows without any memory leak. But running with wine the VmRSS-value is permanently increased. I use following bash-script to check this: Code: while true; do cat

I'm using Wine 1.1.18 on Puppy Linux 3.01 I wrote an application with a http-server and a socket-server for windows and I like to use this on linux. (Delphi with Indy components) My application works fine on windows without any memory leak. But running with wine the VmRSS-value is continual increased. I use following bash-script to check this: Code: while true; do cat

Hello all, I'd like to utilize the R_Sock* functions from R_ext/R-ftp-http.h in my R package. The intent is to use these in conjunction with R_serialize() to store R objects in a remote data store. I'm aware that version 2.2.1 of "Writing R extensions" explains that these may be undocumented and unstable, but I have a couple of questions: 1) are they platform independent?

Dear R developers, While improving duplicated.array() and friends and developing equivalents for the new ff package for large datasets I came across two questions: 1) is it safe to use duplicated.default(), unique.default() and match() on arbitrary lists? If so, we can speed up duplicated.array and friends considerably by using list() instead of paste(collapse="\r") 2) while

Have an interesting one here, originally found on xen 4.1.0 but just upgraded to xen 4.1.1 and it''s still here. Creating a VM with a tap2 device pointed at /dev/mapper/something, when that device doesn''t exist, causes the tapdisk2 process to go into D mode and also manages to take out any process that queries it. For example, I have /dev/mapper/nathanxenuk1 as a valid disk

Hello List, I have a question about dovecot-1.1.0-beta3 but first I'll ramble a little bit: Recently I upgraded a 0.99 installation of dovecot to 1.0.5. All went relatively smoothly until the dovecot-auth process began returning "out of memory" after a day. There is a rather large user base here. The current setup is using passdb pam with blocking=yes, with pam_ldap.so handling

On Sat, 1 Jul 2017 16:30:25 +0100, Rowland Penny via samba wrote: > On Sat, 01 Jul 2017 11:48:21 -0300 > Guido Lorenzutti via samba wrote: > >> Hi there! I been using samba3 with ldap for years, and now im about to move to samba4 to leave the slapd. > > I take it you mean that you use Samba as an AD DC Exactly. >> I didnt try yet to migrate the directory from

Folks, i thought it should be straightforward but after a few hours poking around, I decided it's best to post my question on this list. I have a data frame consisting of a (large) number of date columns, which are read in from a csv file as character string. I want to convert them to Date type. Following is an example, where the first column is of integer type, while the rest are type

Given the below info: what signal is SigBlk indicating? THanks, Jerry ------------ State: S (sleeping) Tgid: 11542 Pid: 11542 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 256 Groups: 0 1 2 3 4 6 10 VmSize: 4888 kB VmLck: 0 kB VmRSS: 1944 kB VmData: 692 kB VmStk: 112 kB VmExe: 1220 kB VmLib:

http://r.789695.n4.nabble.com/file/n3312061/x_and_y_values.txt x_and_y_values.txt I have the absorbance values form HPLC Chromatogram. I need to find the peaks in datapoints and area under each peak. I am not sure how how to find the peaks, I tried couple of libraries and peak function but they return me a list of values which when computed from area turns out to be huge nnumbers. In the

Hi, I've been trying to find a way to label the the peak or mean of a densityplot for a while but haven't been successful. Does anyone know how to accomplish this? Thank you for your help in advance. Joon [[alternative HTML version deleted]]

While reviewing Greg Woods' PATCH 28/36, I realized that our namespace use "peakcurrent". A more coherent name would be "current.peak", as used in Greg's patch (well, in the metasys driver to be precise). For the time being, only the upscode2 driver is using "peakcurrent". But as told, metasys is already using "current.peak". So I'm going to

I have two questions... When I disconnect from my Icecast mount (when broadcasting), I want the following to happen... 1. The "Peak Listeners" count to not be reset to 0 the next time I connect 2. For the mount to remain present (so that my listeners don't get a 404 error and remain listening to silence until I reconnect. Is this possible? Thanks in advance, -- Ben