I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
Thomas, Have a look at the source code for the webpage (ctrl-u in firefox, don't know in internet explorer, etc.). That is what you'd have to parse in order to get the forecast from this page. Typically when I parse webpages such as this I use regular expressions to do so (and I would never downplay the usefulness of regular expressions, but they take a little getting used to). There are two parts to the task: find patterns that allow you to pull out the datum/data you're after; and then write a program to pull it/them out. Also, of course, download the webpage (but that's no issue). I bet you'd be able to find a comma separated value (CSV) file containing the weather report somewhere, which would probably involve a little less labor in order to produce your automatic wardrobe advice. James On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:> I'm writing a program that will tell me whether I should wear a coat, > so I'd like to be able to download daily weather forecasts and daily > reports of recent past weather conditions. > > The NOAA has very promising tabular forecasts > (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), > but I can't figure out how to import them. > > Someone must have needed to do this before. Suggestions? > > Thomas Levine! > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >
2009/2/26 Thomas Levine <thomas.levine at gmail.com>:> I'm writing a program that will tell me whether I should wear a coat, > so I'd like to be able to download daily weather forecasts and daily > reports of recent past weather conditions. > > The NOAA has very promising tabular forecasts > (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), > but I can't figure out how to import them. > > Someone must have needed to do this before. Suggestions?You could use my geonames package that uses the GeoNames query service. There's a sample queries here: http://geonames.r-forge.r-project.org/ Easiest is probably to use GNfindNearByWeather: > as.data.frame(GNfindNearByWeather(57,-2)) clouds weatherCondition 1 broken clouds n/a observation windDirection ICAO 1 EGPD 262120Z 25003KT 9000 -RA BKN018 06/05 Q1012 NOSIG 250 EGPD elevation countryCode lng temperature dewPoint windSpeed humidity 1 65 GB -2.216667 6 5 03 93 stationName datetime lat hectoPascAltimeter 1 Aberdeen / Dyce 2009-02-26 21:20:00 57.2 1012 The package is on CRAN. There is of course an easier way to decide if you need to wear a coat, and that is to look out the window :) Barry
See also http://umbrellatoday.com/ Hadley On Thu, Feb 26, 2009 at 2:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:> I'm writing a program that will tell me whether I should wear a coat, > so I'd like to be able to download daily weather forecasts and daily > reports of recent past weather conditions. > > The NOAA has very promising tabular forecasts > (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), > but I can't figure out how to import them. > > Someone must have needed to do this before. Suggestions? > > Thomas Levine! > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- http://had.co.nz/
Can I just say, it's great to see the R community really come out in support of such a noble and worthy cause as this :). Downfall of civilization, all that. Not here, no! James On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:> I'm writing a program that will tell me whether I should wear a coat, > so I'd like to be able to download daily weather forecasts and daily > reports of recent past weather conditions. > > The NOAA has very promising tabular forecasts > (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), > but I can't figure out how to import them. > > Someone must have needed to do this before. Suggestions? > > Thomas Levine! > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >
tlevine wrote:> > The NOAA has very promising tabular forecasts > (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), > but I can't figure out how to import them. >Sometimes you can just use gsub to get html into R. Try this. ## read html wf<-readLines("http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital", warn=FALSE) ## you do need to find the rows with data (rows 18 and 19), grep may help here ## mark end of lines x<-gsub("</tr>", "\n", wf[18:19]) ## remove white space... x<-gsub(" ", "_", x) ## remove degree symbol or add comment.char="" to read.table below x<-gsub("°", "", x) ## remove html tags x<-gsub("<[^>]*>", " ", x) y<-read.table(con<-textConnection(x), fill=TRUE) close(con) #now just reshape this mess # join rows 1,16 and 17-32 (skip 1st two rows with day and hour) z<-data.frame(rbind( t(y[3:16,-1]), t(y[19:32,-1]) ), row.names=NULL, stringsAsFactors=FALSE) names(z)<-substr(y[3:16,1], 1,4) head(z) Temp Dewp Wind Wind.1 Wind.2 Gust Sky_ Pcpn Rel. Thun Rain Snow Free Slee 1 49 43 43 18 S 99 40 80 -- Chc -- -- -- 2 50 43 44 18 SSW 99 40 77 -- Chc -- -- -- 3 50 43 44 17 SW 99 40 77 -- Chc -- -- -- 4 48 43 42 16 SW 99 40 83 -- Chc -- -- -- 5 45 43 38 15 WSW 100 40 93 -- Chc -- -- -- 6 42 42 35 14 W 100 35 100 -- Chc -- -- -- # plot plot(z["Temp"]) Chris Stubben -- View this message in context: http://www.nabble.com/Download-daily-weather-data-tp22233373p22252517.html Sent from the R help mailing list archive at Nabble.com.