Hello everyone,
I was working on a DeepNet project at home, with an old PC with 4Gb RAM, running
Ubuntu 16.04.
For efficiency reason, I stored my dataset as a csv file with write.csv and
reloaded it at will with read.csv. I did it several time, everything was working
fine.
This morning, at work, I wanted to use a more powerful machine with 8Gb of RAM
running under Ubuntu 16.10, but I ran into a strange error:
$ R
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)
Error: C stack usage 7970548 is too close to the limit
I read a few fora on the Internet and found a potential workaround, consisting
in increasing the stack size using ulimit. Unfortunately, it doesn't work
for me:
$ ulimit -s
8192
$ ulimit -s $((100*$(ulimit -s)))
$ R --vanilla
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)
How is it possible that a machine with twice the RAM as the previous one cannot
load my file? Also how is it possible that a 513MB file cannot be read on a
machine with 8GB RAM?
Since the only other difference is the Ubuntu version (thus R version), I assume
there's a bug, but I don't know where to look for...
-----------------
FYI, my R version is:
16:05:12 R > version
_
platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 3.1
year 2016
month 06
day 21
svn rev 70800
language R
version.string R version 3.3.1 (2016-06-21)
nickname Bug in Your Hair
-----------------
For the sake of completeness, my Trainset is available at the following link,
looks perfectly correct and loads perfectly @home but not @work...
https://mega.nz/#!1E9VHYKZ!isyoJEe6zoLdtC3efoZQreyxKP8sVJyvaiiLSEg4sBY
$ ls trainSetWhole.csv
-rw------- 1 guize guize 513M Mar 1 18:14 trainSetWhole.csv
$ head -n2 trainSetWhole.csv
"","i1","i2","i3","i4","i5","i6","i7","i8","i9","i10","i11","i12","i13","i14","i15","i16","i17","i18","i19","i20","i21","i22","i23","i24","i25","i26","i27","i28","i29","i30","i31","i32","i33","i34","i35","i36","i37","i38","i39","i40","i41","i42","i43","i44","i45","i46","i47","i48","i49","i50","i51","i52","i53","i54","i55","i56","i57","i58","i59","i60","i61","i62","i63","i64","i65","i66","i67","i68","i69","i70","i71","i72","i73","i74","i75","i76","i77","i78","i79","i80","i81","i82","i83","i84","i85","i86","i87","i88","i89","i90","i91","i92","i93","i94","i95","i96","i97","i98","i99","i100","i101","i102","i103","i104","i105","i106","i107","i108","i109","i110","i111","i112","i113","i114","i115","i116","i117","i118","i119","i120","i121","i122","i123","i124","i125","i126","i127","i128","i129","i130","i131","i132","i133","i134","i135","i136","i137","i138","i139","i140","c0","c1","c2","c3","c4","c5","c6","c7","c8","c9","cE","cH"
"1",-1.37446682201354e-17,2.26278846107692e-17,-1.27974510570192e-17,0.350805746453904,0.0689877720064749,3.96782166297516e-17,8.3747934622645e-17,-3.10693828550825e-17,-6.20727662108592e-17,3.50781139850846e-17,-7.43520083579608e-18,-2.01937105782818e-17,7.90527666844222e-17,0.00144287421147209,-1.88026404914112e-17,7.50553543898089e-17,-3.1156305544842e-17,3.33522787760855e-17,-8.2365725541805e-17,5.58356055978426e-17,-1.0212113811016e-17,2.03979863594455e-17,4.09717807934463e-17,0.0216467781249343,-3.44153670483002e-17,1.63751844321319e-17,0.0370002406507207,6.07678690807364e-18,-8.21680252405594e-17,4.44832722506604e-17,6.4069604073711e-17,-2.25481427040111e-17,4.03090616886684e-17,0.0326641123208996,-7.66887661628525e-17,2.74073501177539e-19,2.16820561358683e-17,4.48746042635995e-17,-1.09133528709867e-16,7.27192988615857e-17,3.12504939940947e-17,1.00863400203073e-17,-1.16417253776312e-17,0.000827131700928926,2.15591878205102e-17,-3.55400897372337e-17,-5.31549197577629e-17,6.36151166067713e-17,-3.98986076789542e-17,8.923132020282e-17,2.25307349483604e-17,1.12105833438192e-17,6.11627660475528e-17,0.00457304914240373,-4.40071366400508e-17,-2.44832622161246e-17,0.0421621636081912,2.46626049639688e-17,-9.62286119146992e-18,-1.88510143728912e-17,-1.04308238059585e-17,-2.93643045718137e-17,2.20818728169428e-17,-4.44567220452202e-17,-8.94603852191165e-17,-9.94874448913085e-18,0.0114804462249642,-9.35428456683928e-18,-1.80861616310379e-17,3.92469708878433e-17,-4.74071809520974e-17,-9.12321682896516e-18,-3.69120306784775e-17,0.13411638484663,-1.15868506960443e-16,-7.46389966104001e-17,0.015082205973855,6.53743919930211e-18,2.67372672961429e-17,-1.91214022579557e-17,-2.63284305395001e-18,4.36662713526659e-17,-1.32415731622945e-17,0.0917760638890958,-5.1463921187737e-17,-3.06212083550592e-17,0.020354034217623,1.42110160178925e-17,8.78127934370195e-18,1.10296552176504e-17,6.04875461249026e-18,-1.12907950679241e-16,6.41934790659652e-17,0.0635906532396394,-5.80721386923163e-17,2.60888831084575e-17,0.318234894813079,-2.64405232759225e-17,-3.83483045548365e-17,6.03901669028453e-17,-8.12491526717919e-17,-2.31961766473706e-17,2.49851938854807e-17,0.0666134385177111,0.277994494373684,0.757276941042661,-6.17404037289403e-18,7.89863674358793e-17,-7.67835382820615e-17,1.02140113616298e-17,1.52661784797379e-17,-9.56651701308766e-17,-3.02020715735595e-17,0.388119618133219,0.407682070148667,2.2374985383075e-18,1.11172424441548e-16,5.834694549909e-18,-6.98992457434969e-17,2.58373931894799e-17,1.77047896799026e-17,-1.32411436947046e-17,2.48508545648632e-17,0.00261698558797486,-4.74012929746027e-18,-6.88567774177344e-18,-1.42368729782461e-17,3.86907870454172e-18,-1.18115468072844e-16,5.8232650855203e-17,-6.19221175303288e-18,1.66357706932964e-17,6.39953697342706e-17,2.06819618922065e-17,1.12542559622211e-17,1.61613122517958e-17,-8.54903096963431e-18,-2.91678323773648e-17,-1.24859483206721e-16,7.55251863839618e-18,1,0,0,0,0,0,0,0,0,0,0,0
If anyone has an idea...
Hi again,
Sorry, I messed up, the link to the actual file is:
https://mega.nz/#!ZMs0TSRJ!47DCZCnE6_FnICUp8MVS2R9eY_GdVIyGZ5O9TiejHfc
FYI, it loads perfectly on 2 machines with Ubuntu 16.04 and R version 3.2.3
(2015-12-10) -- "Wooden Christmas-Tree"
But exceededs stack mem under Ubuntu 16.10's R version 3.3.1 (2016-06-21) --
"Bug in Your Hair"
-------- Forwarded Message --------
Subject: Error: memory exhausted (limit reached?)
Date: Mon, 13 Mar 2017 16:51:15 +0100
From: Guillaume MULLER <gm at presans.com>
To: r-help at r-project.org
Hello everyone,
I was working on a DeepNet project at home, with an old PC with 4Gb RAM, running
Ubuntu 16.04.
For efficiency reason, I stored my dataset as a csv file with write.csv and
reloaded it at will with read.csv. I did it several time, everything was working
fine.
This morning, at work, I wanted to use a more powerful machine with 8Gb of RAM
running under Ubuntu 16.10, but I ran into a strange error:
$ R
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)
Error: C stack usage 7970548 is too close to the limit
I read a few fora on the Internet and found a potential workaround, consisting
in increasing the stack size using ulimit. Unfortunately, it doesn't work
for me:
$ ulimit -s
8192
$ ulimit -s $((100*$(ulimit -s)))
$ R --vanilla
16:05:12 R > trainSet <- read.csv("trainSetWhole.csv")
Error: memory exhausted (limit reached?)
How is it possible that a machine with twice the RAM as the previous one cannot
load my file? Also how is it possible that a 513MB file cannot be read on a
machine with 8GB RAM?
Since the only other difference is the Ubuntu version (thus R version), I assume
there's a bug, but I don't know where to look for...
-----------------
FYI, my R version is:
16:05:12 R > version
_
platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 3.1
year 2016
month 06
day 21
svn rev 70800
language R
version.string R version 3.3.1 (2016-06-21)
nickname Bug in Your Hair
-----------------
For the sake of completeness, my Trainset is available at the following link,
looks perfectly correct and loads perfectly @home but not @work...
https://mega.nz/#!1E9VHYKZ!isyoJEe6zoLdtC3efoZQreyxKP8sVJyvaiiLSEg4sBY
$ ls trainSetWhole.csv
-rw------- 1 guize guize 513M Mar 1 18:14 trainSetWhole.csv
$ head -n2 trainSetWhole.csv
"","i1","i2","i3","i4","i5","i6","i7","i8","i9","i10","i11","i12","i13","i14","i15","i16","i17","i18","i19","i20","i21","i22","i23","i24","i25","i26","i27","i28","i29","i30","i31","i32","i33","i34","i35","i36","i37","i38","i39","i40","i41","i42","i43","i44","i45","i46","i47","i48","i49","i50","i51","i52","i53","i54","i55","i56","i57","i58","i59","i60","i61","i62","i63","i64","i65","i66","i67","i68","i69","i70","i71","i72","i73","i74","i75","i76","i77","i78","i79","i80","i81","i82","i83","i84","i85","i86","i87","i88","i89","i90","i91","i92","i93","i94","i95","i96","i97","i98","i99","i100","i101","i102","i103","i104","i105","i106","i107","i108","i109","i110","i111","i112","i113","i114","i115","i116","i117","i118","i119","i120","i121","i122","i123","i124","i125","i126","i127","i128","i129","i130","i131","i132","i133","i134","i135","i136","i137","i138","i139","i140","c0","c1","c2","c3","c4","c5","c6","c7","c8","c9","cE","cH"
"1",-1.37446682201354e-17,2.26278846107692e-17,-1.27974510570192e-17,0.350805746453904,0.0689877720064749,3.96782166297516e-17,8.3747934622645e-17,-3.10693828550825e-17,-6.20727662108592e-17,3.50781139850846e-17,-7.43520083579608e-18,-2.01937105782818e-17,7.90527666844222e-17,0.00144287421147209,-1.88026404914112e-17,7.50553543898089e-17,-3.1156305544842e-17,3.33522787760855e-17,-8.2365725541805e-17,5.58356055978426e-17,-1.0212113811016e-17,2.03979863594455e-17,4.09717807934463e-17,0.0216467781249343,-3.44153670483002e-17,1.63751844321319e-17,0.0370002406507207,6.07678690807364e-18,-8.21680252405594e-17,4.44832722506604e-17,6.4069604073711e-17,-2.25481427040111e-17,4.03090616886684e-17,0.0326641123208996,-7.66887661628525e-17,2.74073501177539e-19,2.16820561358683e-17,4.48746042635995e-17,-1.09133528709867e-16,7.27192988615857e-17,3.12504939940947e-17,1.00863400203073e-17,-1.16417253776312e-17,0.000827131700928926,2.15591878205102e-17,-3.55400897372337e-17,-5.31549197577629e-17,6.36151166067713e-17,-3.98986076789542e-17,8.923132020282e-17,2.25307349483604e-17,1.12105833438192e-17,6.11627660475528e-17,0.00457304914240373,-4.40071366400508e-17,-2.44832622161246e-17,0.0421621636081912,2.46626049639688e-17,-9.62286119146992e-18,-1.88510143728912e-17,-1.04308238059585e-17,-2.93643045718137e-17,2.20818728169428e-17,-4.44567220452202e-17,-8.94603852191165e-17,-9.94874448913085e-18,0.0114804462249642,-9.35428456683928e-18,-1.80861616310379e-17,3.92469708878433e-17,-4.74071809520974e-17,-9.12321682896516e-18,-3.69120306784775e-17,0.13411638484663,-1.15868506960443e-16,-7.46389966104001e-17,0.015082205973855,6.53743919930211e-18,2.67372672961429e-17,-1.91214022579557e-17,-2.63284305395001e-18,4.36662713526659e-17,-1.32415731622945e-17,0.0917760638890958,-5.1463921187737e-17,-3.06212083550592e-17,0.020354034217623,1.42110160178925e-17,8.78127934370195e-18,1.10296552176504e-17,6.04875461249026e-18,-1.12907950679241e-16,6.41934790659652e-17,0.0635906532396394,-5.80721386923163e-17,2.60888831084575e-17,0.318234894813079,-2.64405232759225e-17,-3.83483045548365e-17,6.03901669028453e-17,-8.12491526717919e-17,-2.31961766473706e-17,2.49851938854807e-17,0.0666134385177111,0.277994494373684,0.757276941042661,-6.17404037289403e-18,7.89863674358793e-17,-7.67835382820615e-17,1.02140113616298e-17,1.52661784797379e-17,-9.56651701308766e-17,-3.02020715735595e-17,0.388119618133219,0.407682070148667,2.2374985383075e-18,1.11172424441548e-16,5.834694549909e-18,-6.98992457434969e-17,2.58373931894799e-17,1.77047896799026e-17,-1.32411436947046e-17,2.48508545648632e-17,0.00261698558797486,-4.74012929746027e-18,-6.88567774177344e-18,-1.42368729782461e-17,3.86907870454172e-18,-1.18115468072844e-16,5.8232650855203e-17,-6.19221175303288e-18,1.66357706932964e-17,6.39953697342706e-17,2.06819618922065e-17,1.12542559622211e-17,1.61613122517958e-17,-8.54903096963431e-18,-2.91678323773648e-17,-1.24859483206721e-16,7.55251863839618e-18,1,0,0,0,0,0,0,0,0,0,0,0
If anyone has an idea...
I suspect this is beyond the expertise of most participants of this list. You might wish to post this on r-devel instead,where such expertise is more likely to be found. However, that also may not be the right place either; it's just the best I could think of. Cheers, Bert Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Tue, Mar 14, 2017 at 2:42 AM, Guillaume MULLER <gm at presans.com> wrote:> Hi again, > > Sorry, I messed up, the link to the actual file is: > https://mega.nz/#!ZMs0TSRJ!47DCZCnE6_FnICUp8MVS2R9eY_GdVIyGZ5O9TiejHfc > > FYI, it loads perfectly on 2 machines with Ubuntu 16.04 and R version 3.2.3 (2015-12-10) -- "Wooden Christmas-Tree" > > But exceededs stack mem under Ubuntu 16.10's R version 3.3.1 (2016-06-21) -- "Bug in Your Hair" > > > -------- Forwarded Message -------- > Subject: Error: memory exhausted (limit reached?) > Date: Mon, 13 Mar 2017 16:51:15 +0100 > From: Guillaume MULLER <gm at presans.com> > To: r-help at r-project.org > > Hello everyone, > > I was working on a DeepNet project at home, with an old PC with 4Gb RAM, running Ubuntu 16.04. > > For efficiency reason, I stored my dataset as a csv file with write.csv and reloaded it at will with read.csv. I did it several time, everything was working fine. > > This morning, at work, I wanted to use a more powerful machine with 8Gb of RAM running under Ubuntu 16.10, but I ran into a strange error: > > $ R > 16:05:12 R > trainSet <- read.csv("trainSetWhole.csv") > Error: memory exhausted (limit reached?) > Error: C stack usage 7970548 is too close to the limit > > > I read a few fora on the Internet and found a potential workaround, consisting in increasing the stack size using ulimit. Unfortunately, it doesn't work for me: > $ ulimit -s > 8192 > $ ulimit -s $((100*$(ulimit -s))) > $ R --vanilla > 16:05:12 R > trainSet <- read.csv("trainSetWhole.csv") > Error: memory exhausted (limit reached?) > > How is it possible that a machine with twice the RAM as the previous one cannot load my file? Also how is it possible that a 513MB file cannot be read on a machine with 8GB RAM? > > Since the only other difference is the Ubuntu version (thus R version), I assume there's a bug, but I don't know where to look for... > > ----------------- > FYI, my R version is: > > > 16:05:12 R > version > _ > platform x86_64-pc-linux-gnu > arch x86_64 > os linux-gnu > system x86_64, linux-gnu > status > major 3 > minor 3.1 > year 2016 > month 06 > day 21 > svn rev 70800 > language R > version.string R version 3.3.1 (2016-06-21) > nickname Bug in Your Hair > > ----------------- > For the sake of completeness, my Trainset is available at the following link, looks perfectly correct and loads perfectly @home but not @work... > > https://mega.nz/#!1E9VHYKZ!isyoJEe6zoLdtC3efoZQreyxKP8sVJyvaiiLSEg4sBY > > $ ls trainSetWhole.csv > -rw------- 1 guize guize 513M Mar 1 18:14 trainSetWhole.csv > $ head -n2 trainSetWhole.csv > "","i1","i2","i3","i4","i5","i6","i7","i8","i9","i10","i11","i12","i13","i14","i15","i16","i17","i18","i19","i20","i21","i22","i23","i24","i25","i26","i27","i28","i29","i30","i31","i32","i33","i34","i35","i36","i37","i38","i39","i40","i41","i42","i43","i44","i45","i46","i47","i48","i49","i50","i51","i52","i53","i54","i55","i56","i57","i58","i59","i60","i61","i62","i63","i64","i65","i66","i67","i68","i69","i70","i71","i72","i73","i74","i75","i76","i77","i78","i79","i80","i81","i82","i83","i84","i85","i86","i87","i88","i89","i90","i91","i92","i93","i94","i95","i96","i97","i98","i99","i100","i101","i102","i103","i104","i105","i106","i107","i108","i109","i110","i111","i112","i113","i114","i115","i116","i117","i118","i119","i120","i121","i122","i123","i124","i125","i126","i127","i128","i129","i130","i131","i132","i133","i134","i135","i136","i137","i138","i139","i140","c0","c1","c2","c3","c4","c5","c6","c7","c8","c9","cE","cH" > "1",-1.37446682201354e-17,2.26278846107692e-17,-1.27974510570192e-17,0.350805746453904,0.0689877720064749,3.96782166297516e-17,8.3747934622645e-17,-3.10693828550825e-17,-6.20727662108592e-17,3.50781139850846e-17,-7.43520083579608e-18,-2.01937105782818e-17,7.90527666844222e-17,0.00144287421147209,-1.88026404914112e-17,7.50553543898089e-17,-3.1156305544842e-17,3.33522787760855e-17,-8.2365725541805e-17,5.58356055978426e-17,-1.0212113811016e-17,2.03979863594455e-17,4.09717807934463e-17,0.0216467781249343,-3.44153670483002e-17,1.63751844321319e-17,0.0370002406507207,6.07678690807364e-18,-8.21680252405594e-17,4.44832722506604e-17,6.4069604073711e-17,-2.25481427040111e-17,4.03090616886684e-17,0.0326641123208996,-7.66887661628525e-17,2.74073501177539e-19,2.16820561358683e-17,4.48746042635995e-17,-1.09133528709867e-16,7.27192988615857e-17,3.12504939940947e-17,1.00863400203073e-17,-1.16417253776312e-17,0.000827131700928926,2.15591878205102e-17,-3.55400897372337e-17,-5.31549197577629e-! > 17,6.36151166067713e-17,-3.98986076789542e-17,8.923132020282e-17,2.25307349483604e-17,1.12105833438192e-17,6.11627660475528e-17,0.00457304914240373,-4.40071366400508e-17,-2.44832622161246e-17,0.0421621636081912,2.46626049639688e-17,-9.62286119146992e-18,-1.88510143728912e-17,-1.04308238059585e-17,-2.93643045718137e-17,2.20818728169428e-17,-4.44567220452202e-17,-8.94603852191165e-17,-9.94874448913085e-18,0.0114804462249642,-9.35428456683928e-18,-1.80861616310379e-17,3.92469708878433e-17,-4.74071809520974e-17,-9.12321682896516e-18,-3.69120306784775e-17,0.13411638484663,-1.15868506960443e-16,-7.46389966104001e-17,0.015082205973855,6.53743919930211e-18,2.67372672961429e-17,-1.91214022579557e-17,-2.63284305395001e-18,4.36662713526659e-17,-1.32415731622945e-17,0.0917760638890958,-5.1463921187737e-17,-3.06212083550592e-17,0.020354034217623,1.42110160178925e-17,8.78127934370195e-18,1.10296552176504e-17,6.04875461249026e-18,-1.12907950679241e-16,6.41934790659652e-17,0.06359065323963! > 94,-5.80721386923163e-17,2.60888831084575e-17,0.318234894813079,-2.644 > 05232759225e-17,-3.83483045548365e-17,6.03901669028453e-17,-8.12491526717919e-17,-2.31961766473706e-17,2.49851938854807e-17,0.0666134385177111,0.277994494373684,0.757276941042661,-6.17404037289403e-18,7.89863674358793e-17,-7.67835382820615e-17,1.02140113616298e-17,1.52661784797379e-17,-9.56651701308766e-17,-3.02020715735595e-17,0.388119618133219,0.407682070148667,2.2374985383075e-18,1.11172424441548e-16,5.834694549909e-18,-6.98992457434969e-17,2.58373931894799e-17,1.77047896799026e-17,-1.32411436947046e-17,2.48508545648632e-17,0.00261698558797486,-4.74012929746027e-18,-6.88567774177344e-18,-1.42368729782461e-17,3.86907870454172e-18,-1.18115468072844e-16,5.8232650855203e-17,-6.19221175303288e-18,1.66357706932964e-17,6.39953697342706e-17,2.06819618922065e-17,1.12542559622211e-17,1.61613122517958e-17,-8.54903096963431e-18,-2.91678323773648e-17,-1.24859483206721e-16,7.55251863839618e-18,1,0,0,0,0,0,0,0,0,0,0,0 > > > If anyone has an idea... > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.