R devel - Mar 2017 - Error: memory exhausted (limit reached?)

Hi again,

Sorry, I messed up, the link to the actual file is:
https://mega.nz/#!ZMs0TSRJ!47DCZCnE6_FnICUp8MVS2R9eY_GdVIyGZ5O9TiejHfc

FYI, it loads perfectly on 2 machines with Ubuntu 16.04 and R version 3.2.3
(2015-12-10) -- "Wooden Christmas-Tree"

But exceededs stack mem under Ubuntu 16.10's R version 3.3.1 (2016-06-21) --
"Bug in Your Hair"


-------- Forwarded Message --------
Subject: Error: memory exhausted (limit reached?)
Date: Mon, 13 Mar 2017 16:51:15 +0100
From: Guillaume MULLER <gm at presans.com>
To: r-help at r-project.org

Hello everyone,

I was working on a DeepNet project at home, with an old PC with 4Gb RAM, running
Ubuntu 16.04.

For efficiency reason, I stored my dataset as a csv file with write.csv and
reloaded it at will with read.csv. I did it several time, everything was working
fine.

This morning, at work, I wanted to use a more powerful machine with 8Gb of RAM
running under Ubuntu 16.10, but I ran into a strange error:

$ R
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)
Error: C stack usage  7970548 is too close to the limit


I read a few fora on the Internet and found a potential workaround, consisting
in increasing the stack size using ulimit. Unfortunately, it doesn't work
for me:
$ ulimit -s
8192
$ ulimit -s $((100*$(ulimit -s)))
$ R --vanilla
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)

How is it possible that a machine with twice the RAM as the previous one cannot
load my file? Also how is it possible that a 513MB file cannot be read on a
machine with 8GB RAM?

Since the only other difference is the Ubuntu version (thus R version), I assume
there's a bug, but I don't know where to look for...

-----------------
FYI, my R version is:


16:05:12 R > version
               _
platform       x86_64-pc-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          3
minor          3.1
year           2016
month          06
day            21
svn rev        70800
language       R
version.string R version 3.3.1 (2016-06-21)
nickname       Bug in Your Hair

-----------------
For the sake of completeness, my Trainset is available at the following link,
looks perfectly correct and loads perfectly @home but not @work...

https://mega.nz/#!1E9VHYKZ!isyoJEe6zoLdtC3efoZQreyxKP8sVJyvaiiLSEg4sBY

$ ls trainSetWhole.csv
-rw------- 1 guize guize 513M Mar  1 18:14 trainSetWhole.csv
$ head -n2 trainSetWhole.csv
"","i1","i2","i3","i4","i5","i6","i7","i8","i9","i10","i11","i12","i13","i14","i15","i16","i17","i18","i19","i20","i21","i22","i23","i24","i25","i26","i27","i28","i29","i30","i31","i32","i33","i34","i35","i36","i37","i38","i39","i40","i41","i42","i43","i44","i45","i46","i47","i48","i49","i50","i51","i52","i53","i54","i55","i56","i57","i58","i59","i60","i61","i62","i63","i64","i65","i66","i67","i68","i69","i70","i71","i72","i73","i74","i75","i76","i77","i78","i79","i80","i81","i82","i83","i84","i85","i86","i87","i88","i89","i90","i91","i92","i93","i94","i95","i96","i97","i98","i99","i100","i101","i102","i103","i104","i105","i106","i107","i108","i109","i110","i111","i112","i113","i114","i115","i116","i117","i118","i119","i120","i121","i122","i123","i124","i125","i126","i127","i128","i129","i130","i131","i132","i133","i134","i135","i136","i137","i138","i139","i140","c0","c1","c2","c3","c4","c5","c6","c7","c8","c9","cE","cH"
"1",-1.37446682201354e-17,2.26278846107692e-17,-1.27974510570192e-17,0.350805746453904,0.0689877720064749,3.96782166297516e-17,8.3747934622645e-17,-3.10693828550825e-17,-6.20727662108592e-17,3.50781139850846e-17,-7.43520083579608e-18,-2.01937105782818e-17,7.90527666844222e-17,0.00144287421147209,-1.88026404914112e-17,7.50553543898089e-17,-3.1156305544842e-17,3.33522787760855e-17,-8.2365725541805e-17,5.58356055978426e-17,-1.0212113811016e-17,2.03979863594455e-17,4.09717807934463e-17,0.0216467781249343,-3.44153670483002e-17,1.63751844321319e-17,0.0370002406507207,6.07678690807364e-18,-8.21680252405594e-17,4.44832722506604e-17,6.4069604073711e-17,-2.25481427040111e-17,4.03090616886684e-17,0.0326641123208996,-7.66887661628525e-17,2.74073501177539e-19,2.16820561358683e-17,4.48746042635995e-17,-1.09133528709867e-16,7.27192988615857e-17,3.12504939940947e-17,1.00863400203073e-17,-1.16417253776312e-17,0.000827131700928926,2.15591878205102e-17,-3.55400897372337e-17,-5.31549197577629e-17,6.36151166067713e-17,-3.98986076789542e-17,8.923132020282e-17,2.25307349483604e-17,1.12105833438192e-17,6.11627660475528e-17,0.00457304914240373,-4.40071366400508e-17,-2.44832622161246e-17,0.0421621636081912,2.46626049639688e-17,-9.62286119146992e-18,-1.88510143728912e-17,-1.04308238059585e-17,-2.93643045718137e-17,2.20818728169428e-17,-4.44567220452202e-17,-8.94603852191165e-17,-9.94874448913085e-18,0.0114804462249642,-9.35428456683928e-18,-1.80861616310379e-17,3.92469708878433e-17,-4.74071809520974e-17,-9.12321682896516e-18,-3.69120306784775e-17,0.13411638484663,-1.15868506960443e-16,-7.46389966104001e-17,0.015082205973855,6.53743919930211e-18,2.67372672961429e-17,-1.91214022579557e-17,-2.63284305395001e-18,4.36662713526659e-17,-1.32415731622945e-17,0.0917760638890958,-5.1463921187737e-17,-3.06212083550592e-17,0.020354034217623,1.42110160178925e-17,8.78127934370195e-18,1.10296552176504e-17,6.04875461249026e-18,-1.12907950679241e-16,6.41934790659652e-17,0.0635906532396394,-5.80721386923163e-17,2.60888831084575e-17,0.318234894813079,-2.64405232759225e-17,-3.83483045548365e-17,6.03901669028453e-17,-8.12491526717919e-17,-2.31961766473706e-17,2.49851938854807e-17,0.0666134385177111,0.277994494373684,0.757276941042661,-6.17404037289403e-18,7.89863674358793e-17,-7.67835382820615e-17,1.02140113616298e-17,1.52661784797379e-17,-9.56651701308766e-17,-3.02020715735595e-17,0.388119618133219,0.407682070148667,2.2374985383075e-18,1.11172424441548e-16,5.834694549909e-18,-6.98992457434969e-17,2.58373931894799e-17,1.77047896799026e-17,-1.32411436947046e-17,2.48508545648632e-17,0.00261698558797486,-4.74012929746027e-18,-6.88567774177344e-18,-1.42368729782461e-17,3.86907870454172e-18,-1.18115468072844e-16,5.8232650855203e-17,-6.19221175303288e-18,1.66357706932964e-17,6.39953697342706e-17,2.06819618922065e-17,1.12542559622211e-17,1.61613122517958e-17,-8.54903096963431e-18,-2.91678323773648e-17,-1.24859483206721e-16,7.55251863839618e-18,1,0,0,0,0,0,0,0,0,0,0,0


If anyone has an idea...

Hi,

I first posted this message on r-help, but got redirected here.

I encounter a strange memory error, and I'd like some help to determine if
I'm doing something wrong or if there's a bug in recent R versions...

I'm currently working on a DeepNet project @home, with an old PC with 4Gb
RAM, running Ubuntu 16.04.

For efficiency reason, I preprocessed my dataset and stored it as a csv file
with write.csv() so that I can reload it at will with read.csv(). I did it
several time, everything was working fine.

A few days ago, I tried to pursue my work on anther machine @work, I wanted to
use a more recent & powerful machine with 8Gb of RAM running under Ubuntu
16.10, but I ran into a strange error:

$ R
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)
Error: C stack usage  7970548 is too close to the limit


I read a few fora on the Internet and found a potential workaround, consisting
in increasing the stack size using ulimit. Unfortunately, it doesn't work
for me:

$ ulimit -s
8192
$ ulimit -s $((100*$(ulimit -s)))
$ R --vanilla
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)


This was under Ubuntu 16.10 with R version 3.3.1 (2016-06-21) "Bug in Your
Hair"

Yesterday, I upgraded my Ubuntu to 17.04 (R version 3.3.2 (2016-10-31)
"Sincere Pumpkin Patch") and tried again. This resulted in the exact
same error.

How is it possible that a 513MB file cannot be read on a machine with 8GB RAM?
Also, how is it possible that a machine with twice the RAM as the previous one
cannot load the same file?

Since the only other difference is the Ubuntu version (thus R version), I assume
there's a bug in R/csv loader, but I don't know where to look for...

If anyone has an idea I would be glad to hear it...


GM
--------

For the sake of completeness, I share my Trainset at the following
link:
> https://mega.nz/#!ZMs0TSRJ!47DCZCnE6_FnICUp8MVS2R9eY_GdVIyGZ5O9TiejHfc
FYI, it loads perfectly on 2 machines with Ubuntu 16.04 and R version 3.2.3
(2015-12-10) -- "Wooden Christmas-Tree". But exceededs stack mem under
Ubuntu 16.10's R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
and Ubuntu to 17.04's R version 3.3.2 (2016-10-31) "Sincere Pumpkin
Patch".