[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

javed khan
2020-03-03T21:57:17Z

I am sorry for that... I am just using caret package.

Thanks

On Tue, Mar 3, 2020 at 9:57 PM Bert Gunter <bgunter.4567 at gmail.com>
wrote:

> ... and you **still** have not told us what package(s)...
>
> Bert Gunter
>
> "The trouble with having an open mind is that people keep coming along
and
> sticking things into it."
> -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip
)
>
>
> On Tue, Mar 3, 2020 at 12:28 PM javed khan <javedbtk111 at gmail.com>
wrote:
>
>> The data is as follows:  I included the code for 10 fold CV and LOOCV
>>
>> structure(list(ID = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
>> 13, 14, 15), Language = c(1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1,
>> 1, 1, 3), Hardware = c(1, 2, 3, 1, 2, 4, 4, 2, 1, 1, 1, 5, 6,
>> 1, 1), Duration = c(17, 7, 15, 18, 13, 5, 5, 11, 14, 5, 13, 31,
>> 20, 26, 14), KSLOC = c(253.6, 40.5, 450, 214.4, 449.9, 50, 43,
>> 200, 289, 39, 254.2, 128.6, 161.4, 164.8, 60.2), AdjFP = c(1217.1,
>> 507.3, 2306.8, 788.5, 1337.6, 421.3, 99.9, 993, 1592.9, 240,
>> 1611, 789, 690.9, 1347.5, 1044.3), RAWFP = c(1010, 457, 2284,
>> 881, 1583, 411, 97, 998, 1554, 250, 1603, 724, 705, 1375, 976
>> ), EffortMM = c(287, 82.5, 1107.31, 86.9, 336.3, 84, 23.2, 130.3,
>> 116, 72, 258.7, 230.7, 157, 246.9, 69.9)), class =
"data.frame",
>> row.names = c(NA,
>> -15L))
>>
>>
>> index <- createDataPartition(d$Effort, p = .70,list = FALSE)
>> tr <- d[index, ]
>> ts <- d[-index, ]
>> index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list = TRUE)
>>
>> ct_rand <- trainControl(method = "repeatedcv", number=10,
>> repeats=10,index = index_2, search="random")
>> ct_grid <- trainControl(method = "repeatedcv", number=10,
>> repeats=10,index = index_2, search="grid")
>>
>> ct_locv <- trainControl(method = "LOOCV",
search="random")
>> ct_locv2 <- trainControl(method = "LOOCV",
search="grid")
>>
>>  ## ## ## ## ##Random Search for for 10 fold CV
>>
>> set.seed(30218)
>> ran_CV <- train(Effort ~ ., data = tr,
>>                     method = "pls",
>>                     tuneLength = 15,
>>                     metric = "MAE",
>>                     preProc = c("center", "scale",
"zv"),
>>                     trControl = ct_rand)
>> getTrainPerf(ran_CV)
>> rn <- predict(ran_CV, newdata = ts)
>> ##MAE(rn, ts$Effort)
>>
>>
>> ## ## ## ## ##grid search  for 10 fold CV
>>
>> set.seed(30218)
>> grid_CV <- train(Effort ~ ., data = tr,
>>                      method = "pls",
>>                      metric = "MAE",
>>                      preProc = c("center", "scale",
"zv"),
>>                      trControl = ct_grid)
>>
>> getTrainPerf(grid_CV)
>>
>>   ## ## ## ## ##Random Search for LOOCV
>>
>> set.seed(30218)
>> ran_locv <- train(Effort ~ ., data = tr,
>>                 method = "pls",
>>                 tuneLength = 15,
>>                 metric = "MAE",
>>                 preProc = c("center", "scale",
"zv"),
>>                 trControl = ct_locv)
>> getTrainPerf(ran_locv)
>> rn <- predict(ran_search, newdata = ts)
>> ##MAE(rn, ts$Effort)
>>
>>
>>  ## ## ## ## ##Grid Search for LOOCV
>>
>> set.seed(30218)
>> grid_locv <- train(Effort ~ ., data = tr,
>>                  method = "pls",
>>                  metric = "MAE",
>>                  preProc = c("center", "scale",
"zv"),
>>                  trControl = ct_locv2)
>>
>> getTrainPerf(grid_locv)
>>
>> rValues <- resamples(list(Random_Search_CV=ran_CV,
>> Grid_Search_CV=grid_CV,
>>
>>                           Random_Search_LOOCV=ran_locv,
>> Grid_Search_LOOCV=grid_locv))
>>
>> bwplot(rValues,metric="MAE", scales=list(cex=1),
col="Green")
>>
>>
>>
>>
>> On Tue, Mar 3, 2020 at 5:07 PM Bert Gunter <bgunter.4567 at
gmail.com>
>> wrote:
>>
>>> 2 1/2 suggestions:
>>>
>>> 1. Provide a small reproducible example with **minimal code** . It
can
>>> be difficult to sort through dozens of lines of code, and I,
anyway, would
>>> be unwilling to spend time trying to debug/isolate the problem when
you
>>> have apparently not made much of an effort to do so yourself.
Others may
>>> well be both more knowledgeable and more tolerant, of course.
>>>
>>> 2. If, **after a suitable wait ** you have not received useful
answers,
>>> contact the package maintainer of the package you used **which you
have
>>> again failed to identify** (the caret package?) . Also check to see
whether
>>> the package has its own user support structure. Some do, and this
should be
>>> the first point of contact anyway if so.
>>>
>>> 2 1/2 . Post in **plain text** not html, though I don't think
it
>>> mattered here.
>>>
>>>
>>> Bert Gunter
>>>
>>> "The trouble with having an open mind is that people keep
coming along
>>> and sticking things into it."
>>> -- Opus (aka Berkeley Breathed in his "Bloom County"
comic strip )
>>>
>>>
>>> On Tue, Mar 3, 2020 at 3:30 AM javed khan <javedbtk111 at
gmail.com> wrote:
>>>
>>>> Hi, I am using different validation methods for random search
and grid
>>>> search. The validation methods are 10 fold CV, bootstrap and
LOOCV but
>>>> for
>>>> LOOCV, I get the error message when I draw boxplots for all the
results.
>>>>
>>>> Error is , LOOCV is not compatible with `resamples()` since
only one
>>>> resampling estimate is available.
>>>>
>>>> The code is below.
>>>>
>>>> d=readARFF("china.arff")
>>>> index <- createDataPartition(d$Effort, p = .70,list = FALSE)
>>>> tr <- d[index, ]
>>>> ts <- d[-index, ]
>>>> index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list =
TRUE)
>>>>
>>>>
>>>>
>>>>
>>>> ct_rand <- trainControl(method = "repeatedcv",
number=10,
>>>> repeats=10,index
>>>> = index_2, search="random")
>>>> ct_grid <- trainControl(method = "repeatedcv",
number=10,
>>>> repeats=10,index
>>>> = index_2, search="grid")
>>>>
>>>>
>>>> ct_boot1 <- trainControl(method = "boot",
number=100,  index = index_2,
>>>> search="random")
>>>> ct_boot2 <- trainControl(method = "boot",
number=100,  index = index_2,
>>>> search="grid")
>>>>
>>>> ct_locv <- trainControl(method = "LOOCV",
search="random")
>>>> ct_locv2 <- trainControl(method = "LOOCV",
search="grid")
>>>>
>>>> set.seed(30218)
>>>> ran_CV <- train(Effort ~ ., data = tr,
>>>>                     method = "pls",
>>>>                     tuneLength = 15,
>>>>                     metric = "MAE",
>>>>                     preProc = c("center",
"scale", "zv"),
>>>>                     trControl = ct_rand)
>>>> getTrainPerf(ran_CV)
>>>> rn <- predict(ran_CV, newdata = ts)
>>>>
>>>> ## ## ## ## ##grid search CV
>>>>
>>>> set.seed(30218)
>>>> grid_CV <- train(Effort ~ ., data = tr,
>>>>                      method = "pls",
>>>>                      metric = "MAE",
>>>>                      preProc = c("center",
"scale", "zv"),
>>>>                      trControl = ct_grid)
>>>>
>>>> getTrainPerf(grid_CV)
>>>>
>>>> set.seed(30218)
>>>> ran_boot <- train(Effort ~ ., data = tr,
>>>>                 method = "pls",
>>>>                 tuneLength = 15,
>>>>                 metric = "MAE",
>>>>                 preProc = c("center",
"scale", "zv"),
>>>>                 trControl = ct_boot1)
>>>> getTrainPerf(ran_boot)
>>>> rn <- predict(ran_search, newdata = ts)
>>>> ##MAE(rn, ts$Effort)
>>>>
>>>>
>>>> ## ## ## ## ##grid search boot
>>>>
>>>> set.seed(30218)
>>>> grid_boot <- train(Effort ~ ., data = tr,
>>>>                  method = "pls",
>>>>                  metric = "MAE",
>>>>                  preProc = c("center",
"scale", "zv"),
>>>>                  trControl = ct_boot2)
>>>>
>>>> getTrainPerf(grid_boot)
>>>>
>>>>
>>>> set.seed(30218)
>>>> ran_locv <- train(Effort ~ ., data = tr,
>>>>                 method = "pls",
>>>>                 tuneLength = 15,
>>>>                 metric = "MAE",
>>>>                 preProc = c("center",
"scale", "zv"),
>>>>                 trControl = ct_locv)
>>>> getTrainPerf(ran_locv)
>>>> rn <- predict(ran_search, newdata = ts)
>>>> ##MAE(rn, ts$Effort)
>>>>
>>>>
>>>> ## ## ## ## ##grid search CV
>>>>
>>>> set.seed(30218)
>>>> grid_locv <- train(Effort ~ ., data = tr,
>>>>                  method = "pls",
>>>>                  metric = "MAE",
>>>>                  preProc = c("center",
"scale", "zv"),
>>>>                  trControl = ct_locv2)
>>>>
>>>> getTrainPerf(grid_locv)
>>>>
>>>>
>>>> rValues <- resamples(list(Random_Search_CV=ran_CV,
>>>> Grid_Search_CV=grid_CV,
>>>> Random_Search_Boot=ran_boot, Grid_Search_Boot=grid_boot ,
>>>>                           Random_Search_LOOCV=ran_locv,
>>>> Grid_Search_LOOCV=grid_locv))
>>>>
>>>> bwplot(rValues,metric="MAE", scales=list(cex=1),
col="Green")
>>>>
>>>>         [[alternative HTML version deleted]]
>>>>
>>>> ______________________________________________
>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and
more, see
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide
>>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible
code.
>>>>
>>>