search for: enc2nat

When I send some outlandish characters through enc2native (or format) in R 3.1.2 on Ubuntu trusty it works quite well: > "?????" [1] "?????" > enc2native("?????") [1] "?????" > Encoding(enc2native("?????")) [1] "UTF-8" In Windows the result is different: > "?????"...

Am 27.02.2015 um 03:13 schrieb Duncan Murdoch: > On 26/02/2015 6:34 PM, maillist at tlink.de wrote: >>> On 26/02/2015 3:09 PM, maillist at tlink.de wrote: >>>> When I send some outlandish characters through enc2native (or format) in >>>> R 3.1.2 on Ubuntu trusty it works quite well: >>>> >>>> > "?????" >>>> [1] "?????" >>>> > enc2native("?????") >>>> [1] "?????" >>>> >...

> On 26/02/2015 3:09 PM, maillist at tlink.de wrote: >> When I send some outlandish characters through enc2native (or format) in >> R 3.1.2 on Ubuntu trusty it works quite well: >> >> > "?????" >> [1] "?????" >> > enc2native("?????") >> [1] "?????" >> > Encoding(enc2native("?????")) >> [1] "...

On 27/02/2015 2:31 AM, maillist at tlink.de wrote: > Am 27.02.2015 um 03:13 schrieb Duncan Murdoch: >> On 26/02/2015 6:34 PM, maillist at tlink.de wrote: >>>> On 26/02/2015 3:09 PM, maillist at tlink.de wrote: >>>>> When I send some outlandish characters through enc2native (or format) in >>>>> R 3.1.2 on Ubuntu trusty it works quite well: >>>>> >>>>> > "?????" >>>>> [1] "?????" >>>>> > enc2native("?????") >>>>> [1] "?????" &...

On Thu, Feb 26, 2015 at 2:09 PM, maillist at tlink.de <maillist at tlink.de> wrote: > > When I send some outlandish characters through enc2native (or format) in R > 3.1.2 on Ubuntu trusty it works quite well: > > > "?????" > [1] "?????" > > enc2native("?????") > [1] "?????" > > Encoding(enc2native("?????")) > [1] "UTF-8" > > In Windows the r...

On 26/02/2015 6:34 PM, maillist at tlink.de wrote: >> On 26/02/2015 3:09 PM, maillist at tlink.de wrote: >>> When I send some outlandish characters through enc2native (or format) in >>> R 3.1.2 on Ubuntu trusty it works quite well: >>> >>> > "?????" >>> [1] "?????" >>> > enc2native("?????") >>> [1] "?????" >>> > Encoding(enc2native("?????...

...52, "2013" in Unicode. The same error seems to happen with enc2utf8(x) Now with iconv() the result is as expected. iconv(x, to = "UTF-8") The second problem IMO is that encoding markers get lost with the enc2* functions x_utf8 <- enc2utf8(x) Encoding(x_utf8) x_nat <- enc2native(x_utf8) Encoding(x_nat) Again, this is not the case with iconv() x_iutf8 <- iconv(x, to = "UTF-8") Encoding(x_iutf8) x_inat <- iconv(x_iutf8, from = "UTF-8") Encoding(x_inat) [[alternative HTML version deleted]]

...ho character is invalid in latin1, it renders as if it were the valid windows-1252 character [1] "’" > enc2utf8(x) # Encoding as UTF-8 gives us, not a valid UTF-8 'right quote' (/u2019), but the undefined unicode character 'PRIVATE USE TWO' [1] "\u0092" > enc2native(enc2utf8(x)) # Moving the UTF-8 to back to the native encoding correctly shows that it can't render the 'PRIVATE USE TWO' character in windows-1252 [1] "<U+0092>" --------------------- I think the problem occurs when R decides that the valid 1252 character should be...

...s expected. > > iconv(x, to = "UTF-8") > > > The second problem IMO is that encoding markers get lost with the enc2* > functions As you are changing encodings, you do not want to preserve encoding! > x_utf8 <- enc2utf8(x) > Encoding(x_utf8) > x_nat <- enc2native(x_utf8) > Encoding(x_nat) In an actual Latin-1 locale on Linux > x_utf8 <- c("??", "\u20ac", "\u2013") > Encoding(x_utf8) [1] "latin1" "UTF-8" "UTF-8" > enc2native(x_utf8) [1] "??" "<U+20AC>...

...returns "unknown". In cp1252 encoding on Windows (note that I am using the cp1252 escape "\x80" and not the Unicode "\u20AC") > x_utf8 <- enc2utf8(c("?", "\x80")) > Encoding(x_utf8) [1] "UTF-8" "UTF-8" > x_nat <- enc2native(x_utf8) > Encoding(x_nat) [1] "unknown" "unknown" See also Kirill's message to this list: "ASCII strings are marked as ASCII internally, but this information doesn't seem to be available, e.g., Encoding() returns "unknown" for such strings " ht...

...C locale but CP936 as the system code page (see the sessionInfo() output). On Windows, unfortunately, there are two different "current locales" at a time. With your settings (CP1252 as C locale and CP936 as system code page), I get the same results as you, file.exists() returns FALSE. enc2native(z) works fine and returns a valid Latin-1 string, but that is because here "native" is CP1252. Windows API functions and consequently some C library functions that return strings from the OS, however, convert to the encoding from the system code page, which is CP936 and it cannot r...

Hi Tomas, I received a report about R 4.0.0 in the knitr package (https://github.com/yihui/knitr/issues/1840), and I think it is related to the issue here. I created a minimal reproducible example below: owd = setwd(tempdir()) z = 'K\u00e4sch.txt' file.create(z) list.files() file.exists(list.files()) setwd(owd) Output: > owd = setwd(tempdir()) > z = 'K\u00e4sch.txt' >

...esting non-generic 'attributes' testing non-generic 'attributes<-' testing non-generic 'baseenv' testing non-generic 'call' testing non-generic 'class' testing non-generic 'class<-' testing non-generic 'emptyenv' testing non-generic 'enc2native' testing non-generic 'enc2utf8' testing non-generic 'environment<-' testing non-generic 'expression' testing non-generic 'gc.time' testing non-generic 'globalenv' testing non-generic 'interactive' testing non-generic 'invisible' test...

...36 as the system code > page (see the sessionInfo() output). On Windows, unfortunately, there > are two different "current locales" at a time. With your settings > (CP1252 as C locale and CP936 as system code page), I get the same > results as you, file.exists() returns FALSE. enc2native(z) works fine > and returns a valid Latin-1 string, but that is because here "native" is > CP1252. Windows API functions and consequently some C library functions > that return strings from the OS, however, convert to the encoding from > the system code page, which is CP936...

...hanks to suggestions from > David Winsenius. > > o Having formal arguments of an S4 method in a different order > from the generic is now an error (the warning having been > ignored by some package maintainers for a long time). > > o New functions enc2native() and enc2utf8() convert character > vectors with possibly marked encodings to the current locale and > UTF-8 respectively. > > o Unrecognized escapes and embedded nuls in character strings are > now an error, not just a warning. Thus option "warnEs...

...list() items (per level), thanks to suggestions from David Winsenius. o Having formal arguments of an S4 method in a different order from the generic is now an error (the warning having been ignored by some package maintainers for a long time). o New functions enc2native() and enc2utf8() convert character vectors with possibly marked encodings to the current locale and UTF-8 respectively. o Unrecognized escapes and embedded nuls in character strings are now an error, not just a warning. Thus option "warnEscapes"...

...list() items (per level), thanks to suggestions from David Winsenius. o Having formal arguments of an S4 method in a different order from the generic is now an error (the warning having been ignored by some package maintainers for a long time). o New functions enc2native() and enc2utf8() convert character vectors with possibly marked encodings to the current locale and UTF-8 respectively. o Unrecognized escapes and embedded nuls in character strings are now an error, not just a warning. Thus option "warnEscapes"...

...esting non-generic 'attributes' testing non-generic 'attributes<-' testing non-generic 'baseenv' testing non-generic 'call' testing non-generic 'class' testing non-generic 'class<-' testing non-generic 'emptyenv' testing non-generic 'enc2native' testing non-generic 'enc2utf8' testing non-generic 'environment<-' testing non-generic 'expression' testing non-generic 'gc.time' testing non-generic 'globalenv' testing non-generic 'interactive' testing non-generic 'invisible' test...