Hi there, I am trying to write a tool which involves implementing logistic regression. With the batch gradient descent method, the convergence is guaranteed as it is a convex problem. However, I find that with the stochastic gradient decent method, it typically converges to some random points (i.e., not very close to the minimum point resulted from the batch method). I have tried different ways of decreasing the learning rate, and different starting points of weights. However, the performance (e.g., accuracy, precision/recall, ...) are comparable (to the batch method). I understand that this is possible, since SGD(stochastic gradient descent) uses an approximation to the real cost each step. Does it matter? I guess it does since otherwise the interpretation of the weights would not make much sense even the accuracy is comparable. If it matters, I wonder if you have some suggestions on how to make it converge or getting close to the global optimal point. Thanks! [[alternative HTML version deleted]]
I may be missing something, but what does this have to do specifically with R? I believe this is OT here and you need to post elsewhere, e.g. perhaps on stats.stackexchange.com. -- Bert On Tue, Mar 5, 2013 at 1:36 PM, Ivan Li <machinelearning2010 at gmail.com> wrote:> Hi there, > > I am trying to write a tool which involves implementing logistic > regression. With the batch gradient descent method, the convergence is > guaranteed as it is a convex problem. However, I find that with the > stochastic gradient decent method, it typically converges to some random > points (i.e., not very close to the minimum point resulted from the batch > method). I have tried different ways of decreasing the learning rate, and > different starting points of weights. However, the performance (e.g., > accuracy, precision/recall, ...) are comparable (to the batch method). > > I understand that this is possible, since SGD(stochastic gradient descent) > uses an approximation to the real cost each step. Does it matter? I guess > it does since otherwise the interpretation of the weights would not make > much sense even the accuracy is comparable. If it matters, I wonder if you > have some suggestions on how to make it converge or getting close to the > global optimal point. > > > > Thanks! > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm
Perhaps I should have added and FWIW, R, like essentially all statistical software, has "logistic regression" already built in, if I understand what you mean by the term (which I may not), via glm's. -- Bert On Tue, Mar 5, 2013 at 1:36 PM, Ivan Li <machinelearning2010 at gmail.com> wrote:> Hi there, > > I am trying to write a tool which involves implementing logistic > regression. With the batch gradient descent method, the convergence is > guaranteed as it is a convex problem. However, I find that with the > stochastic gradient decent method, it typically converges to some random > points (i.e., not very close to the minimum point resulted from the batch > method). I have tried different ways of decreasing the learning rate, and > different starting points of weights. However, the performance (e.g., > accuracy, precision/recall, ...) are comparable (to the batch method). > > I understand that this is possible, since SGD(stochastic gradient descent) > uses an approximation to the real cost each step. Does it matter? I guess > it does since otherwise the interpretation of the weights would not make > much sense even the accuracy is comparable. If it matters, I wonder if you > have some suggestions on how to make it converge or getting close to the > global optimal point. > > > > Thanks! > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm