I am looking for a function that can fit a smooth function to a vector of estimated proportions, such that the smoothed value is within specified confidence bounds of each proportion. In other words, given a small number of trials and large confidence intervals, I would prefer the function to vary smoothly, but given a large number of trials and small confidence intervals, I would prefer the function to lie within the confidence intervals, even if it is not smooth. I have attached a postscript file illustrating a data set I would like to smooth. As the figure shows, for large values of x, I have few data points, and so the ML estimate of the proportion varies widely, and the confidence intervals are very large. When I use the smooth.spline function with a large value of spar (the red line), the function is not as smooth as desired for large values of x. When I use a smaller value of spar (the green line), the function fails to stay within the confidence bounds of the proportions. Is there a smoothing function for which I can specify upper and lower limits for the y value for specific values of x? Thanks for any suggestions, Rose
Sorry, forgot to attach the graph. On 8/10/07, Rose Hoberman <roseh at cs.cmu.edu> wrote:> I am looking for a function that can fit a smooth function to a vector > of estimated proportions, such that the smoothed value is within > specified confidence bounds of each proportion. In other words, given > a small number of trials and large confidence intervals, I would > prefer the function to vary smoothly, but given a large number of > trials and small confidence intervals, I would prefer the function to > lie within the confidence intervals, even if it is not smooth. > > I have attached a postscript file illustrating a data set I would like > to smooth. As the figure shows, for large values of x, I have few > data points, and so the ML estimate of the proportion varies widely, > and the confidence intervals are very large. When I use the > smooth.spline function with a large value of spar (the red line), the > function is not as smooth as desired for large values of x. When I > use a smaller value of spar (the green line), the function fails to > stay within the confidence bounds of the proportions. Is there a > smoothing function for which I can specify upper and lower limits for > the y value for specific values of x? > > Thanks for any suggestions, > > Rose >-------------- next part -------------- A non-text attachment was scrubbed... Name: smoothProportions.ps Type: application/postscript Size: 11419 bytes Desc: not available Url : https://stat.ethz.ch/pipermail/r-help/attachments/20070810/09031ae4/attachment.ps
It is not entirely clear what you are using for y values in smooth.spline, but it would appear that it is just the point estimates. I would suggest using instead -- at each x value -- a few equally spaced quantiles of the estimated proportions. Implicitly, smooth.spline expects to be fitting a mean curve to data that has constant variance, so you might also consider reweighting to approximate this, as well. url: www.econ.uiuc.edu/~roger Roger Koenker email rkoenker at uiuc.edu Department of Economics vox: 217-333-4558 University of Illinois fax: 217-244-6678 Champaign, IL 61820 On Aug 10, 2007, at 10:23 AM, Rose Hoberman wrote:> Sorry, forgot to attach the graph. > > On 8/10/07, Rose Hoberman <roseh at cs.cmu.edu> wrote: >> I am looking for a function that can fit a smooth function to a >> vector >> of estimated proportions, such that the smoothed value is within >> specified confidence bounds of each proportion. In other words, >> given >> a small number of trials and large confidence intervals, I would >> prefer the function to vary smoothly, but given a large number of >> trials and small confidence intervals, I would prefer the function to >> lie within the confidence intervals, even if it is not smooth. >> >> I have attached a postscript file illustrating a data set I would >> like >> to smooth. As the figure shows, for large values of x, I have few >> data points, and so the ML estimate of the proportion varies widely, >> and the confidence intervals are very large. When I use the >> smooth.spline function with a large value of spar (the red line), the >> function is not as smooth as desired for large values of x. When I >> use a smaller value of spar (the green line), the function fails to >> stay within the confidence bounds of the proportions. Is there a >> smoothing function for which I can specify upper and lower limits for >> the y value for specific values of x? >> >> Thanks for any suggestions, >> >> Rose >> >> <smoothProportions.ps> > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting- > guide.html > and provide commented, minimal, self-contained, reproducible code.