search for: toraw

Dear R experts, It seems that Rprintf has to be used to print from a C routine to guarantee to write to R?s output according to https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Printing. However if a string is UTF-8 encoded, non-ASCII characters (e.g., the infinity symbol http://www.fileformat.info/info/unicode/char/221e/index.htm) are misprinted. Is this an unsupported feature or is

..."numeric", "numeric", "numeric", "numeric", "array")) But the 5th column is treated as an array of characters, and when trying to convert it to another class of data, I either get two strings "c(1,2,3)" and "c(1,1,4)" or using a toRaw converter, I get the corresponding ASCII ?? values. Should the input be modified in order to be able to read it as an array of integers? Thank you for your help. Balpo

R-devel community: I have encountered some unexpected behavior using iconv, which may be the source of errors I am getting when connecting to a UTF-16 -encoded SQL Server database. A simple example is below. When researching this problem, I found r-devel reports of the same problem in threads from June 2010 and February, 2016, and that bug #16738 was posted to Bugzilla as a result. However, I

On 30/04/2017 12:23 PM, Duncan Murdoch wrote: > No, I don't think anyone is working on this. > > There's a fairly simple workaround for the UTF-16 and UTF-32 iconv > issues: don't attempt to produce character vectors, produce raw vectors > instead. (The "toRaw" argument to iconv() asks for this.) Raw vectors > can contain embedded nulls. Character vectors can't, because > internally, R is using 8 bit C strings, and the nulls are string > terminators. > > I don't know how difficult it would be to fix the write.table problems....

...v("foo", to="UTF-16") >> >> produces >> >> Error in iconv("foo", to = "UTF-16"): >> embedded nul in string: '\xff\xfef\0o\0o\0' but this works > iconv("foo", to="UTF-16", toRaw=TRUE) [[1]] [1] ff fe 66 00 6f 00 6f 00 (indeed showing the embedded '\0's) >> In 2010 a (partial) patch for this problem was submitted: >> http://tolstoy.newcastle.edu.au/R/e10/devel/10/06/0648.html the patch only related to the iconv() problem not allowing &...

...roduces > > >> > > >> Error in iconv("foo", to = "UTF-16"): > > >> embedded nul in string: '\xff\xfef\0o\0o\0' > > > > but this works > > > > > iconv("foo", to="UTF-16", toRaw=TRUE) > > [[1]] > > [1] ff fe 66 00 6f 00 6f 00 > > > > (indeed showing the embedded '\0's) > > > > >> In 2010 a (partial) patch for this problem was submitted: > > >> http://tolstoy.newcastle.edu.au/R/e10/devel/10/06/0...

.../2017 12:23 PM, Duncan Murdoch wrote: >> No, I don't think anyone is working on this. >> >> There's a fairly simple workaround for the UTF-16 and UTF-32 iconv >> issues: don't attempt to produce character vectors, produce raw vectors >> instead. (The "toRaw" argument to iconv() asks for this.) Raw vectors >> can contain embedded nulls. Character vectors can't, because >> internally, R is using 8 bit C strings, and the nulls are string >> terminators. >> >> I don't know how difficult it would be to fix the wr...

This caught us yesterday when a string that we assumed to be in UTF-8 was actually using CP1252. (This came from an internal web based service, so the root cause is not R's fault.) The help page for iconv states that the result of an invalid conversion is NA only when the toRaw argument is TRUE, but this appears to be true in general. Example: test1 <- "M?ni?re's disease" # the offending string (it was buried in a 13000 character result string) test2 <- iconv(test1, to="CP1252") # create a version of the string that is in Window-1...

...d UTF-32 ? On 30/04/2017 12:23 PM, Duncan Murdoch wrote: > No, I don't think anyone is working on this. > > There's a fairly simple workaround for the UTF-16 and UTF-32 iconv > issues: don't attempt to produce character vectors, produce raw vectors > instead. (The "toRaw" argument to iconv() asks for this.) Raw vectors > can contain embedded nulls. Character vectors can't, because > internally, R is using 8 bit C strings, and the nulls are string > terminators. > > I don't know how difficult it would be to fix the write.table problems....

"R version 3.4.0 (2017-04-21)" on "x86_64-w64-mingw32" platform I am using CSVs and other text tables, and text in general (including regular expressions), on Windows 10. For me, that means dealing with Windows-1252 and UTF-8 encoding, with UTF-16 and UTF-32 as helpful curiosities. Something as simple as iconv ("\n", to = "UTF-16") causes an error, due to

...; >> > >> produces > >> > >> Error in iconv("foo", to = "UTF-16"): > >> embedded nul in string: '\xff\xfef\0o\0o\0' > > but this works > > > iconv("foo", to="UTF-16", toRaw=TRUE) > [[1]] > [1] ff fe 66 00 6f 00 6f 00 > > (indeed showing the embedded '\0's) > > >> In 2010 a (partial) patch for this problem was submitted: > >> http://tolstoy.newcastle.edu.au/R/e10/devel/10/06/0648.html > > the patch only...

No, I don't think anyone is working on this. There's a fairly simple workaround for the UTF-16 and UTF-32 iconv issues: don't attempt to produce character vectors, produce raw vectors instead. (The "toRaw" argument to iconv() asks for this.) Raw vectors can contain embedded nulls. Character vectors can't, because internally, R is using 8 bit C strings, and the nulls are string terminators. I don't know how difficult it would be to fix the write.table problems. Duncan Murdoch On 2...

If I execute the code from the "?write.table" examples section x <- data.frame(a = I("a \" quote"), b = pi) # (ommited code) write.csv(x, file = "foo.csv", fileEncoding = "UTF-16LE") the resulting CSV file has a size of 6 bytes which is too short (truncated): """,3 The problem seems to be the iconv function:

...> produces >> >> >> >> Error in iconv("foo", to = "UTF-16"): >> >> embedded nul in string: '\xff\xfef\0o\0o\0' >> >> but this works >> >> > iconv("foo", to="UTF-16", toRaw=TRUE) >> [[1]] >> [1] ff fe 66 00 6f 00 6f 00 >> >> (indeed showing the embedded '\0's) >> >> >> In 2010 a (partial) patch for this problem was submitted: >> >> http://tolstoy.newcastle.edu.au/R/e10/devel/10/06/0648.h...

...ne will seem better aligned (but not exactly aligned, since ordinary text has descenders below the baseline and plotmath places them on the baseline). (Related to PR#14537.) o sunflowerplot() now has a formula interface. (Wish of PR#14541.) o iconv() has a new argument toRaw to handle encodings such as UTF-16 with embedded nuls (as was possible before the CHARSXP cache was introduced). It will also accept as input the type of list generated with toRaw = TRUE. o Garbage-collecting an unused input text connection no longer gives a warn...

...ne will seem better aligned (but not exactly aligned, since ordinary text has descenders below the baseline and plotmath places them on the baseline). (Related to PR#14537.) o sunflowerplot() now has a formula interface. (Wish of PR#14541.) o iconv() has a new argument toRaw to handle encodings such as UTF-16 with embedded nuls (as was possible before the CHARSXP cache was introduced). It will also accept as input the type of list generated with toRaw = TRUE. o Garbage-collecting an unused input text connection no longer gives a warn...