R help - Jul 2010 - XML and RCurl: problem with encoding (htmlTreeParse)

Hi All,

First method:-
>library(XML)
>theurl <- "http://home.sina.com"
>download.file(theurl, "tmp.html")
>txt <- readLines("tmp.html")
>txt <- htmlTreeParse(txt, error=function(...){}, useInternalNodes = 
TRUE)
>g <- xpathSApply(txt, "//p", function(x) xmlValue(x))
>head(grep(" ", g, value=T))

[1] " |  | ENGLISH"                               " "
[3] "  ()"              " "
[5] " "                " !  ! !"  



SecondMethod:-
>library(RCurl)
>theurl <-
getURL("http://home.sina.com",encoding='GB2312')
>Encoding(theurl)
[1]"unknown"
>txt <- readLines(con=textConnection(theurl),encoding='GB2312')
>txt[5:10] #show the lines which occurred encoding problem.
[1] "<meta http-equiv=\"Content-Type\"
content=\"text/html; charset=utf-8\" />"
[2] "<title>SINA.com US  -</title>"
[3] "<meta name=\"Keywords\" content=\", , , ,, SINA, US,
News, Chinese, Asia\" />"
[4] "<meta name=\"Description\" content=\", 24, , , , ,
BBS, .\" />"
[5]""
[6] "<link rel=\"stylesheet\" type=\"text/css\"
href=\"http://ui.sina.com/assets/css/style_home.css\" />"

i am trying to read data from a Chinese language website, but the Chinese
characters always unreadable, may I know if any good idea to cope such encoding
problem in RCurl and XML?


Regards,
Ryusuke
 		 	   		  
_________________________________________________________________


	[[alternative HTML version deleted]]

Hi Ryusuke

 I would use the encoding parameter of htmlParse() and 
 download and parse the content in one operation:

     htmlParse("http://home.sina.com", encoding = "UTF-8")

 If you want to use getURL() in RCurl, use the .encoding parameter

  You didn't tell us the output of Sys.getlocale()
  or how your terminal/console is configured, so the above
  may vary under your configuration, but works on various
  machines for me with different settings.

    D.


Ryusuke Kenji wrote:
> 
> Hi All,
> 
> First method:-
> >library(XML)
> 
> >theurl <- "http://home.sina.com"
> >download.file(theurl, "tmp.html")
> 
> >txt <- readLines("tmp.html")
> 
> >txt <- htmlTreeParse(txt, error=function(...){}, useInternalNodes = 
> TRUE)
> 
> >g <- xpathSApply(txt, "//p", function(x) xmlValue(x))
> 
> >head(grep(" ", g, value=T))
> 
> 
> [1] "?????? | ?????? | ENGLISH"                              
"??????????????? ???????????????"
> [3] "??????? ?????????? ??????????????????(???)"             
"?????????????????????????????? ????????????????????????"
> [5] " ???????????????????????????????????????"               
"? ??????????! ????? ??????! ????????????????????????!"
> 
> 
> 
> SecondMethod:-
> >library(RCurl)
> 
> >theurl <-
getURL("http://home.sina.com",encoding='GB2312')
> 
> >Encoding(theurl)
> 
> [1]"unknown"
> 
> >txt <-
readLines(con=textConnection(theurl),encoding='GB2312')
> >txt[5:10] #show the lines which occurred encoding problem.
> [1] "<meta http-equiv=\"Content-Type\"
content=\"text/html; charset=utf-8\" />"
> [2] "<title>SINA.com US ????????? -??????</title>"
> [3] "<meta name=\"Keywords\" content=\"????????????,
???????????????, ???????????????, ??????????????????,????????????, SINA, US,
News, Chinese, Asia\" />"
> [4] "<meta name=\"Description\"
content=\"???????????????????????????????????????,
???????????????24????????????????????????????????, ????????????????????????,
????????????, ??????????????????, ????????????????????????, ?????????BBS,
???????????????????????????????????.\" />"
> [5]""
> [6] "<link rel=\"stylesheet\" type=\"text/css\"
href=\"http://ui.sina.com/assets/css/style_home.css\" />"
> 
> i am trying to read data from a Chinese language website, but the Chinese
characters always unreadable, may I know if any good idea to cope such encoding
problem in RCurl and XML?
> 
> 
> Regards,
> Ryusuke
>  		 	   		  
> _________________________________________________________________
> 
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

-- 
"There are men who can think no deeper than a fact" - Voltaire


Duncan Temple Lang                duncan at wald.ucdavis.edu
Department of Statistics          work:  (530) 752-4782
4210 Mathematical Sciences Bldg.  fax:   (530) 752-7099
One Shields Ave.
University of California at Davis
Davis, CA 95616, USA



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-help/attachments/20100703/3120b3e5/attachment.bin>