Hello Jack,
1 ) why do you thought that " larger C is prone to overfitting than smaller
C" ?
2 ) if you look at the formulation of the quadratic program problem you will
see that C rules the error of the "cutting plane " ( and overfitting
).
Therfore for hight C you allow that the "cutting plane" cuts worse the
set,
so SVM needs less points to build it. a proper explanation is in Kristin P.
Bennett and Colin Campbell, "Support Vector Machines: Hype or
Hallelujah?",
SIGKDD Explorations, 2,2, 2000, 1-13.
http://www.idi.ntnu.no/emner/it3704/lectures/papers/Bennett_2000_Support.pdf
3) you might find usefull this plots:
library(e1071)
m1 <- matrix( c(
0, 0, 0, 1, 1, 2, 1, 2, 3, 2, 3, 3, 0, 1,2,3,
0, 1, 2, 3,
1, 2, 3, 2, 3, 3, 0, 0, 0, 1, 1, 2, 4, 4,4,4, 0,
1, 2, 3,
1, 1, 1, 1, 1, 1, -1,-1, -1,-1,-1,-1, 1 ,1,1,1, 1,
1,-1,-1
), ncol = 3 )
Y = m1[,3]
X = m1[,1:2]
df = data.frame( X , Y )
par(mfcol=c(4,2))
for( cost in c( 1e-3 ,1e-2 ,1e-1, 1e0, 1e+1, 1e+2 ,1e+3)) {
#cost <- 1
model.svm <- svm( Y ~ . , data = df , type = "C-classification" ,
kernel "linear", cost = cost,
scale =FALSE )
#print(model.svm$SV)
plot(x=0,ylim=c(0,5), xlim=c(0,3),main= paste( "cost: ",cost,
"#SV: ",
nrow(model.svm$SV) ))
points(m1[m1[,3]>0,1], m1[m1[,3]>0,2], pch=3, col="green")
points(m1[m1[,3]<0,1], m1[m1[,3]<0,2], pch=4, col="blue")
points(model.svm$SV[,1],model.svm$SV[,2], pch=18 , col = "red")
}
Regards
Pau
2010/7/14 Jack Luo <jluo.rhelp@gmail.com>
> Hi,
>
> I have a question about the parameter C (cost) in svm function in e1071. I
> thought larger C is prone to overfitting than smaller C, and hence leads to
> more support vectors. However, using the Wisconsin breast cancer example on
> the link:
> http://planatscher.net/svmtut/svmtut.html
> I found that the largest cost have fewest support vectors, which is
> contrary
> to what I think. please see the scripts below:
> Am I misunderstanding something here?
>
> Thanks a bunch,
>
> -Jack
>
> > model1 <- svm(databctrain, classesbctrain, kernel =
"linear", cost > 0.01)
> > model2 <- svm(databctrain, classesbctrain, kernel =
"linear", cost = 1)
> > model3 <- svm(databctrain, classesbctrain, kernel =
"linear", cost = 100)
> > model1
>
> Call:
> svm.default(x = databctrain, y = classesbctrain, kernel =
"linear",
> cost = 0.01)
>
>
> Parameters:
> SVM-Type: C-classification
> SVM-Kernel: linear
> cost: 0.01
> gamma: 0.1111111
>
> Number of Support Vectors: 99
>
> > model2
>
> Call:
> svm.default(x = databctrain, y = classesbctrain, kernel =
"linear",
> cost = 1)
>
>
> Parameters:
> SVM-Type: C-classification
> SVM-Kernel: linear
> cost: 1
> gamma: 0.1111111
>
> Number of Support Vectors: 46
>
> > model3
>
> Call:
> svm.default(x = databctrain, y = classesbctrain, kernel =
"linear",
> cost = 100)
>
>
> Parameters:
> SVM-Type: C-classification
> SVM-Kernel: linear
> cost: 100
> gamma: 0.1111111
>
> Number of Support Vectors: 44
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]