PIKAL Petr
2021-Apr-20 07:02 UTC
[R] What is an alternative to expand.grid if create a long vector?
Hi Keep your mails on the list. Actually you did not say much about your data and the way how do you want to model them. There are plenty of modelling functions in R starting with e.g. lm but I am not aware of a procedure in which you just design your explanatory variables to set plausible model. But I am not expert in statistics and this list is not ment for solving statistical problems. Cheers Petr From: Shah Alam <dr.alamsolangi at gmail.com> Sent: Monday, April 19, 2021 5:20 PM To: PIKAL Petr <petr.pikal at precheza.cz> Subject: Re: [R] What is an alternative to expand.grid if create a long vector? Dear Petr, Thanks for your response. I am designing a model with 10 unknown parameters. generating the combination of unknown parameters will be used in the model to estimate the set of vectors that fits well to actual data. Is there any other was to do it? I also used randomLHS function from lhs package. But, it did not serve the purpose. Best regards, Shah Alam On Mon, 19 Apr 2021 at 16:07, PIKAL Petr <petr.pikal at precheza.cz <mailto:petr.pikal at precheza.cz> > wrote: Hi Actually expand.grid produces data frame and not vector. And dimension of the data frame is "big"> dim(A)[1] 100000000 4> str(A)'data.frame': 100000000 obs. of 4 variables: $ Var1: num 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01 ... $ Var2: num 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 ... $ Var3: num 0.38 0.38 0.38 0.38 0.38 0.38 0.38 0.38 0.38 0.38 ... $ Var4: num 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 ... - attr(*, "out.attrs")=List of 2 ..$ dim : int [1:4] 100 100 100 100 ..$ dimnames:List of 4 .. ..$ Var1: chr [1:100] "Var1=0.001" "Var1=0.002" "Var1=0.003" "Var1=0.004" ... .. ..$ Var2: chr [1:100] "Var2=0.0001000000" "Var2=0.0001090909" "Var2=0.0001181818" "Var2=0.0001272727" ... .. ..$ Var3: chr [1:100] "Var3=0.3800000" "Var3=0.3804040" "Var3=0.3808081" "Var3=0.3812121" ... .. ..$ Var4: chr [1:100] "Var4=0.1200000" "Var4=0.1206061" "Var4=0.1212121" "Var4=0.1218182" ...>in case of 4 sequences 1e8 rows, 4 columns in case of 10 sequences 1e20 rows and 10 columns in your last example 1.4e8 rows and 10 columns which probably cross the memory capacity of your PC. Maybe you could increase memory of you PC. If I am correct to store the first you need about 3.2GB, to strore the last 11.2 GB. May I ask what you want to do with such a big object? Cheers Petr> -----Original Message----- > From: R-help <r-help-bounces at r-project.org > <mailto:r-help-bounces at r-project.org> > On Behalf Of Shah Alam > Sent: Monday, April 19, 2021 2:36 PM > To: r-help mailing list <r-help at r-project.org <mailto:r-help at r-project.org> > > > Subject: [R] What is an alternative to expand.grid if create a longvector?> > Dear All, > > I would like to know that is there any problem in *expand.grid* functionor it> is a limitation of this function. > > I am trying to create a combination of elements using expand.gridfunction.> > A <- expand.grid( > c(seq(0.001, 0.1, length.out = 100)), > c(seq(0.0001, 0.001, length.out = 100)), c(seq(0.38, 0.42, length.out 100)), > c(seq(0.12, 0.18, length.out = 100))) > > Four combinations work fine. However, If I increase the combinations up to > ten. The following error appears. > > A <- expand.grid( > c(seq(0.001, 1, length.out = 100)), > c(seq(0.0001, 0.001, length.out = 100)), c(seq(0.38, 0.42, length.out 100)), > c(seq(0.12, 0.18, length.out = 100)), c(seq(0.01, 0.04, length.out 100)), > c(seq(0.0001, 0.001, length.out = 100)), c(seq(0.0001, 0.001, length.out > 100)), c(seq(0.001, 0.01, length.out = 100)), c(seq(0.01, 0.3, length.out= 100))> ) > > *Error in rep.int <http://rep.int> <http://rep.int>(rep.int > <http://rep.int> <http://rep.int>(seq_len(nx), > rep.int <http://rep.int> <http://rep.int>(rep.fac, nx)), orep) : invalid > 'times' value* > > After reducing the length to 10. It produced a different type of error > > A <- expand.grid( > c(seq(0.001, 0.005, length.out = 10)), > c(seq(0.0001, 0.0005, length.out = 10)), c(seq(0.38, 0.42, length.out 5)), > c(seq(0.12, 0.18, length.out = 7)), c(seq(0.01, 0.04, length.out = 5)), > c(seq(0.0001, 0.001, length.out = 10)), c(seq(0.0001, 0.001, length.out 10)), > c(seq(0.001, 0.01, length.out = 10)), c(seq(0.1, 0.8, length.out = 8)) > ) > > *Error: cannot allocate vector of size 1.0 Gb* > > What is an alternative to expand.grid if create a long vector based on 10 > elements? > > With kind regards, > Shah Alam > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org <mailto:R-help at r-project.org> mailing list -- To > UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting- > guide.html > and provide commented, minimal, self-contained, reproducible code.
Jan van der Laan
2021-Apr-20 07:42 UTC
[R] What is an alternative to expand.grid if create a long vector?
This is an optimisation problem that you are trying to solve using a grid search. There are numerous methods for optimisation, see https://cran.r-project.org/web/views/Optimization.html for and overview for R. It really depends on the exact problem what method is appropriate. As Petr said helping you decide which method to use does not fit on this list. Perhaps de overview linked to above (and the terms 'grid search' and 'optimization') can help you find an appropriate method. HTH, Jan On 20-04-2021 09:02, PIKAL Petr wrote:> Hi > > > > Keep your mails on the list. Actually you did not say much about your data and > the way how do you want to model them. There are plenty of modelling functions > in R starting with e.g. lm but I am not aware of a procedure in which you just > design your explanatory variables to set plausible model. But I am not expert > in statistics and this list is not ment for solving statistical problems. > > > > Cheers > > Petr > > > > > > From: Shah Alam <dr.alamsolangi at gmail.com> > Sent: Monday, April 19, 2021 5:20 PM > To: PIKAL Petr <petr.pikal at precheza.cz> > Subject: Re: [R] What is an alternative to expand.grid if create a long > vector? > > > > Dear Petr, > > > > Thanks for your response. I am designing a model with 10 unknown parameters. > generating the combination of unknown parameters will be used in the model to > estimate the set of vectors that fits well to actual data. Is there any other > was to do it? I also used randomLHS function from lhs package. But, it did not > serve the purpose. > > > > Best regards, > > Shah Alam > > > > > > On Mon, 19 Apr 2021 at 16:07, PIKAL Petr <petr.pikal at precheza.cz > <mailto:petr.pikal at precheza.cz> > wrote: > > Hi > > Actually expand.grid produces data frame and not vector. And dimension of > the data frame is "big" > >> dim(A) > [1] 100000000 4 >> str(A) > 'data.frame': 100000000 obs. of 4 variables: > $ Var1: num 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01 ... > $ Var2: num 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 1e-04 > ... > $ Var3: num 0.38 0.38 0.38 0.38 0.38 0.38 0.38 0.38 0.38 0.38 ... > $ Var4: num 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12 ... > - attr(*, "out.attrs")=List of 2 > ..$ dim : int [1:4] 100 100 100 100 > ..$ dimnames:List of 4 > .. ..$ Var1: chr [1:100] "Var1=0.001" "Var1=0.002" "Var1=0.003" > "Var1=0.004" ... > .. ..$ Var2: chr [1:100] "Var2=0.0001000000" "Var2=0.0001090909" > "Var2=0.0001181818" "Var2=0.0001272727" ... > .. ..$ Var3: chr [1:100] "Var3=0.3800000" "Var3=0.3804040" > "Var3=0.3808081" "Var3=0.3812121" ... > .. ..$ Var4: chr [1:100] "Var4=0.1200000" "Var4=0.1206061" > "Var4=0.1212121" "Var4=0.1218182" ... >> > > in case of 4 sequences 1e8 rows, 4 columns > in case of 10 sequences 1e20 rows and 10 columns > in your last example 1.4e8 rows and 10 columns which probably cross the > memory capacity of your PC. > > Maybe you could increase memory of you PC. If I am correct to store the > first you need about 3.2GB, to strore the last 11.2 GB. > > May I ask what you want to do with such a big object? > > Cheers > Petr > >> -----Original Message----- >> From: R-help <r-help-bounces at r-project.org >> <mailto:r-help-bounces at r-project.org> > On Behalf Of Shah Alam >> Sent: Monday, April 19, 2021 2:36 PM >> To: r-help mailing list <r-help at r-project.org <mailto:r-help at r-project.org> >> > >> Subject: [R] What is an alternative to expand.grid if create a long > vector? >> >> Dear All, >> >> I would like to know that is there any problem in *expand.grid* function > or it >> is a limitation of this function. >> >> I am trying to create a combination of elements using expand.grid > function. >> >> A <- expand.grid( >> c(seq(0.001, 0.1, length.out = 100)), >> c(seq(0.0001, 0.001, length.out = 100)), c(seq(0.38, 0.42, length.out > 100)), >> c(seq(0.12, 0.18, length.out = 100))) >> >> Four combinations work fine. However, If I increase the combinations up to >> ten. The following error appears. >> >> A <- expand.grid( >> c(seq(0.001, 1, length.out = 100)), >> c(seq(0.0001, 0.001, length.out = 100)), c(seq(0.38, 0.42, length.out > 100)), >> c(seq(0.12, 0.18, length.out = 100)), c(seq(0.01, 0.04, length.out > 100)), >> c(seq(0.0001, 0.001, length.out = 100)), c(seq(0.0001, 0.001, length.out >> 100)), c(seq(0.001, 0.01, length.out = 100)), c(seq(0.01, 0.3, length.out > = 100)) >> ) >> >> *Error in rep.int <http://rep.int> <http://rep.int>(rep.int >> <http://rep.int> <http://rep.int>(seq_len(nx), >> rep.int <http://rep.int> <http://rep.int>(rep.fac, nx)), orep) : invalid >> 'times' value* >> >> After reducing the length to 10. It produced a different type of error >> >> A <- expand.grid( >> c(seq(0.001, 0.005, length.out = 10)), >> c(seq(0.0001, 0.0005, length.out = 10)), c(seq(0.38, 0.42, length.out > 5)), >> c(seq(0.12, 0.18, length.out = 7)), c(seq(0.01, 0.04, length.out = 5)), >> c(seq(0.0001, 0.001, length.out = 10)), c(seq(0.0001, 0.001, length.out > 10)), >> c(seq(0.001, 0.01, length.out = 10)), c(seq(0.1, 0.8, length.out = 8)) >> ) >> >> *Error: cannot allocate vector of size 1.0 Gb* >> >> What is an alternative to expand.grid if create a long vector based on 10 >> elements? >> >> With kind regards, >> Shah Alam >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-help at r-project.org <mailto:R-help at r-project.org> mailing list -- To >> UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting- >> guide.html >> and provide commented, minimal, self-contained, reproducible code. > > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >