thr3ads.net - R help - [R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. " [Mar 2020]

If this information is useful, please help other people find it:
Share via:

javed khan

2020-Mar-03 11:29 UTC

[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

Hi, I am using different validation methods for random search and grid
search. The validation methods are 10 fold CV, bootstrap and LOOCV but for
LOOCV, I get the error message when I draw boxplots for all the results.

Error is , LOOCV is not compatible with `resamples()` since only one
resampling estimate is available.

The code is below.

d=readARFF("china.arff")
index <- createDataPartition(d$Effort, p = .70,list = FALSE)
tr <- d[index, ]
ts <- d[-index, ]
index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list = TRUE)




ct_rand <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
= index_2, search="random")
ct_grid <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
= index_2, search="grid")


ct_boot1 <- trainControl(method = "boot", number=100,  index =
index_2,
search="random")
ct_boot2 <- trainControl(method = "boot", number=100,  index =
index_2,
search="grid")

ct_locv <- trainControl(method = "LOOCV", 
search="random")
ct_locv2 <- trainControl(method = "LOOCV",  
search="grid")

set.seed(30218)
ran_CV <- train(Effort ~ ., data = tr,
                    method = "pls",
                    tuneLength = 15,
                    metric = "MAE",
                    preProc = c("center", "scale",
"zv"),
                    trControl = ct_rand)
getTrainPerf(ran_CV)
rn <- predict(ran_CV, newdata = ts)

## ## ## ## ##grid search CV

set.seed(30218)
grid_CV <- train(Effort ~ ., data = tr,
                     method = "pls",
                     metric = "MAE",
                     preProc = c("center", "scale",
"zv"),
                     trControl = ct_grid)

getTrainPerf(grid_CV)

set.seed(30218)
ran_boot <- train(Effort ~ ., data = tr,
                method = "pls",
                tuneLength = 15,
                metric = "MAE",
                preProc = c("center", "scale",
"zv"),
                trControl = ct_boot1)
getTrainPerf(ran_boot)
rn <- predict(ran_search, newdata = ts)
##MAE(rn, ts$Effort)


## ## ## ## ##grid search boot

set.seed(30218)
grid_boot <- train(Effort ~ ., data = tr,
                 method = "pls",
                 metric = "MAE",
                 preProc = c("center", "scale",
"zv"),
                 trControl = ct_boot2)

getTrainPerf(grid_boot)


set.seed(30218)
ran_locv <- train(Effort ~ ., data = tr,
                method = "pls",
                tuneLength = 15,
                metric = "MAE",
                preProc = c("center", "scale",
"zv"),
                trControl = ct_locv)
getTrainPerf(ran_locv)
rn <- predict(ran_search, newdata = ts)
##MAE(rn, ts$Effort)


## ## ## ## ##grid search CV

set.seed(30218)
grid_locv <- train(Effort ~ ., data = tr,
                 method = "pls",
                 metric = "MAE",
                 preProc = c("center", "scale",
"zv"),
                 trControl = ct_locv2)

getTrainPerf(grid_locv)


rValues <- resamples(list(Random_Search_CV=ran_CV, Grid_Search_CV=grid_CV,
Random_Search_Boot=ran_boot, Grid_Search_Boot=grid_boot ,
                          Random_Search_LOOCV=ran_locv,
Grid_Search_LOOCV=grid_locv))

bwplot(rValues,metric="MAE", scales=list(cex=1),
col="Green")

	[[alternative HTML version deleted]]

Bert Gunter

2020-Mar-03 16:07 UTC

head link

[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

2 1/2 suggestions:

1. Provide a small reproducible example with **minimal code** . It can be
difficult to sort through dozens of lines of code, and I, anyway, would be
unwilling to spend time trying to debug/isolate the problem when you have
apparently not made much of an effort to do so yourself. Others may well be
both more knowledgeable and more tolerant, of course.

2. If, **after a suitable wait ** you have not received useful answers,
contact the package maintainer of the package you used **which you have
again failed to identify** (the caret package?) . Also check to see whether
the package has its own user support structure. Some do, and this should be
the first point of contact anyway if so.

2 1/2 . Post in **plain text** not html, though I don't think it mattered
here.


Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Tue, Mar 3, 2020 at 3:30 AM javed khan <javedbtk111 at gmail.com>
wrote:
> Hi, I am using different validation methods for random search and grid
> search. The validation methods are 10 fold CV, bootstrap and LOOCV but for
> LOOCV, I get the error message when I draw boxplots for all the results.
>
> Error is , LOOCV is not compatible with `resamples()` since only one
> resampling estimate is available.
>
> The code is below.
>
> d=readARFF("china.arff")
> index <- createDataPartition(d$Effort, p = .70,list = FALSE)
> tr <- d[index, ]
> ts <- d[-index, ]
> index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list = TRUE)
>
>
>
>
> ct_rand <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
> = index_2, search="random")
> ct_grid <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
> = index_2, search="grid")
>
>
> ct_boot1 <- trainControl(method = "boot", number=100,  index =
index_2,
> search="random")
> ct_boot2 <- trainControl(method = "boot", number=100,  index =
index_2,
> search="grid")
>
> ct_locv <- trainControl(method = "LOOCV", 
search="random")
> ct_locv2 <- trainControl(method = "LOOCV",  
search="grid")
>
> set.seed(30218)
> ran_CV <- train(Effort ~ ., data = tr,
>                     method = "pls",
>                     tuneLength = 15,
>                     metric = "MAE",
>                     preProc = c("center", "scale",
"zv"),
>                     trControl = ct_rand)
> getTrainPerf(ran_CV)
> rn <- predict(ran_CV, newdata = ts)
>
> ## ## ## ## ##grid search CV
>
> set.seed(30218)
> grid_CV <- train(Effort ~ ., data = tr,
>                      method = "pls",
>                      metric = "MAE",
>                      preProc = c("center", "scale",
"zv"),
>                      trControl = ct_grid)
>
> getTrainPerf(grid_CV)
>
> set.seed(30218)
> ran_boot <- train(Effort ~ ., data = tr,
>                 method = "pls",
>                 tuneLength = 15,
>                 metric = "MAE",
>                 preProc = c("center", "scale",
"zv"),
>                 trControl = ct_boot1)
> getTrainPerf(ran_boot)
> rn <- predict(ran_search, newdata = ts)
> ##MAE(rn, ts$Effort)
>
>
> ## ## ## ## ##grid search boot
>
> set.seed(30218)
> grid_boot <- train(Effort ~ ., data = tr,
>                  method = "pls",
>                  metric = "MAE",
>                  preProc = c("center", "scale",
"zv"),
>                  trControl = ct_boot2)
>
> getTrainPerf(grid_boot)
>
>
> set.seed(30218)
> ran_locv <- train(Effort ~ ., data = tr,
>                 method = "pls",
>                 tuneLength = 15,
>                 metric = "MAE",
>                 preProc = c("center", "scale",
"zv"),
>                 trControl = ct_locv)
> getTrainPerf(ran_locv)
> rn <- predict(ran_search, newdata = ts)
> ##MAE(rn, ts$Effort)
>
>
> ## ## ## ## ##grid search CV
>
> set.seed(30218)
> grid_locv <- train(Effort ~ ., data = tr,
>                  method = "pls",
>                  metric = "MAE",
>                  preProc = c("center", "scale",
"zv"),
>                  trControl = ct_locv2)
>
> getTrainPerf(grid_locv)
>
>
> rValues <- resamples(list(Random_Search_CV=ran_CV,
Grid_Search_CV=grid_CV,
> Random_Search_Boot=ran_boot, Grid_Search_Boot=grid_boot ,
>                           Random_Search_LOOCV=ran_locv,
> Grid_Search_LOOCV=grid_locv))
>
> bwplot(rValues,metric="MAE", scales=list(cex=1),
col="Green")
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]

javed khan

2020-Mar-03 20:27 UTC

head link

[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

The data is as follows:  I included the code for 10 fold CV and LOOCV

structure(list(ID = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15), Language = c(1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1,
1, 1, 3), Hardware = c(1, 2, 3, 1, 2, 4, 4, 2, 1, 1, 1, 5, 6,
1, 1), Duration = c(17, 7, 15, 18, 13, 5, 5, 11, 14, 5, 13, 31,
20, 26, 14), KSLOC = c(253.6, 40.5, 450, 214.4, 449.9, 50, 43,
200, 289, 39, 254.2, 128.6, 161.4, 164.8, 60.2), AdjFP = c(1217.1,
507.3, 2306.8, 788.5, 1337.6, 421.3, 99.9, 993, 1592.9, 240,
1611, 789, 690.9, 1347.5, 1044.3), RAWFP = c(1010, 457, 2284,
881, 1583, 411, 97, 998, 1554, 250, 1603, 724, 705, 1375, 976
), EffortMM = c(287, 82.5, 1107.31, 86.9, 336.3, 84, 23.2, 130.3,
116, 72, 258.7, 230.7, 157, 246.9, 69.9)), class = "data.frame",
row.names
= c(NA,
-15L))


index <- createDataPartition(d$Effort, p = .70,list = FALSE)
tr <- d[index, ]
ts <- d[-index, ]
index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list = TRUE)

ct_rand <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
= index_2, search="random")
ct_grid <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
= index_2, search="grid")

ct_locv <- trainControl(method = "LOOCV", 
search="random")
ct_locv2 <- trainControl(method = "LOOCV",  
search="grid")

 ## ## ## ## ##Random Search for for 10 fold CV

set.seed(30218)
ran_CV <- train(Effort ~ ., data = tr,
                    method = "pls",
                    tuneLength = 15,
                    metric = "MAE",
                    preProc = c("center", "scale",
"zv"),
                    trControl = ct_rand)
getTrainPerf(ran_CV)
rn <- predict(ran_CV, newdata = ts)
##MAE(rn, ts$Effort)


## ## ## ## ##grid search  for 10 fold CV

set.seed(30218)
grid_CV <- train(Effort ~ ., data = tr,
                     method = "pls",
                     metric = "MAE",
                     preProc = c("center", "scale",
"zv"),
                     trControl = ct_grid)

getTrainPerf(grid_CV)

  ## ## ## ## ##Random Search for LOOCV

set.seed(30218)
ran_locv <- train(Effort ~ ., data = tr,
                method = "pls",
                tuneLength = 15,
                metric = "MAE",
                preProc = c("center", "scale",
"zv"),
                trControl = ct_locv)
getTrainPerf(ran_locv)
rn <- predict(ran_search, newdata = ts)
##MAE(rn, ts$Effort)


 ## ## ## ## ##Grid Search for LOOCV

set.seed(30218)
grid_locv <- train(Effort ~ ., data = tr,
                 method = "pls",
                 metric = "MAE",
                 preProc = c("center", "scale",
"zv"),
                 trControl = ct_locv2)

getTrainPerf(grid_locv)

rValues <- resamples(list(Random_Search_CV=ran_CV, Grid_Search_CV=grid_CV,

                          Random_Search_LOOCV=ran_locv,
Grid_Search_LOOCV=grid_locv))

bwplot(rValues,metric="MAE", scales=list(cex=1),
col="Green")




On Tue, Mar 3, 2020 at 5:07 PM Bert Gunter <bgunter.4567 at gmail.com>
wrote:
> 2 1/2 suggestions:
>
> 1. Provide a small reproducible example with **minimal code** . It can be
> difficult to sort through dozens of lines of code, and I, anyway, would be
> unwilling to spend time trying to debug/isolate the problem when you have
> apparently not made much of an effort to do so yourself. Others may well be
> both more knowledgeable and more tolerant, of course.
>
> 2. If, **after a suitable wait ** you have not received useful answers,
> contact the package maintainer of the package you used **which you have
> again failed to identify** (the caret package?) . Also check to see whether
> the package has its own user support structure. Some do, and this should be
> the first point of contact anyway if so.
>
> 2 1/2 . Post in **plain text** not html, though I don't think it
mattered
> here.
>
>
> Bert Gunter
>
> "The trouble with having an open mind is that people keep coming along
and
> sticking things into it."
> -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip
)
>
>
> On Tue, Mar 3, 2020 at 3:30 AM javed khan <javedbtk111 at gmail.com>
wrote:
>
>> Hi, I am using different validation methods for random search and grid
>> search. The validation methods are 10 fold CV, bootstrap and LOOCV but
for
>> LOOCV, I get the error message when I draw boxplots for all the
results.
>>
>> Error is , LOOCV is not compatible with `resamples()` since only one
>> resampling estimate is available.
>>
>> The code is below.
>>
>> d=readARFF("china.arff")
>> index <- createDataPartition(d$Effort, p = .70,list = FALSE)
>> tr <- d[index, ]
>> ts <- d[-index, ]
>> index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list = TRUE)
>>
>>
>>
>>
>> ct_rand <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
>> = index_2, search="random")
>> ct_grid <- trainControl(method = "repeatedcv", number=10,
repeats=10,index
>> = index_2, search="grid")
>>
>>
>> ct_boot1 <- trainControl(method = "boot", number=100, 
index = index_2,
>> search="random")
>> ct_boot2 <- trainControl(method = "boot", number=100, 
index = index_2,
>> search="grid")
>>
>> ct_locv <- trainControl(method = "LOOCV", 
search="random")
>> ct_locv2 <- trainControl(method = "LOOCV",  
search="grid")
>>
>> set.seed(30218)
>> ran_CV <- train(Effort ~ ., data = tr,
>>                     method = "pls",
>>                     tuneLength = 15,
>>                     metric = "MAE",
>>                     preProc = c("center", "scale",
"zv"),
>>                     trControl = ct_rand)
>> getTrainPerf(ran_CV)
>> rn <- predict(ran_CV, newdata = ts)
>>
>> ## ## ## ## ##grid search CV
>>
>> set.seed(30218)
>> grid_CV <- train(Effort ~ ., data = tr,
>>                      method = "pls",
>>                      metric = "MAE",
>>                      preProc = c("center", "scale",
"zv"),
>>                      trControl = ct_grid)
>>
>> getTrainPerf(grid_CV)
>>
>> set.seed(30218)
>> ran_boot <- train(Effort ~ ., data = tr,
>>                 method = "pls",
>>                 tuneLength = 15,
>>                 metric = "MAE",
>>                 preProc = c("center", "scale",
"zv"),
>>                 trControl = ct_boot1)
>> getTrainPerf(ran_boot)
>> rn <- predict(ran_search, newdata = ts)
>> ##MAE(rn, ts$Effort)
>>
>>
>> ## ## ## ## ##grid search boot
>>
>> set.seed(30218)
>> grid_boot <- train(Effort ~ ., data = tr,
>>                  method = "pls",
>>                  metric = "MAE",
>>                  preProc = c("center", "scale",
"zv"),
>>                  trControl = ct_boot2)
>>
>> getTrainPerf(grid_boot)
>>
>>
>> set.seed(30218)
>> ran_locv <- train(Effort ~ ., data = tr,
>>                 method = "pls",
>>                 tuneLength = 15,
>>                 metric = "MAE",
>>                 preProc = c("center", "scale",
"zv"),
>>                 trControl = ct_locv)
>> getTrainPerf(ran_locv)
>> rn <- predict(ran_search, newdata = ts)
>> ##MAE(rn, ts$Effort)
>>
>>
>> ## ## ## ## ##grid search CV
>>
>> set.seed(30218)
>> grid_locv <- train(Effort ~ ., data = tr,
>>                  method = "pls",
>>                  metric = "MAE",
>>                  preProc = c("center", "scale",
"zv"),
>>                  trControl = ct_locv2)
>>
>> getTrainPerf(grid_locv)
>>
>>
>> rValues <- resamples(list(Random_Search_CV=ran_CV,
Grid_Search_CV=grid_CV,
>> Random_Search_Boot=ran_boot, Grid_Search_Boot=grid_boot ,
>>                           Random_Search_LOOCV=ran_locv,
>> Grid_Search_LOOCV=grid_locv))
>>
>> bwplot(rValues,metric="MAE", scales=list(cex=1),
col="Green")
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
	[[alternative HTML version deleted]]

R help - Mar 2020 - Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "