thr3ads.net - R devel - [Rd] readlines() truncates text file with Codepage 437 encoding [Jun 2016]

If this information is useful, please help other people find it:
Share via:

Adam Obeng

2016-Jun-06 10:11 UTC

[Rd] readlines() truncates text file with Codepage 437 encoding

Hello r-devel,


The attached Code page 437-encoded file contains 245 characters
(including the final newline), but readLines only reads 242 of them:
> test_text <- readLines(file('437__characters.txt',
encoding='437'))Warning message:
In readLines(file("437__characters.txt",  :
  incomplete final line found on
'437__characters.txt'> test_text[1]
"\v\f\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037
!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\177
?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????"> nchar(test_text)[1] 242

You'll note that readLines hasn't read the final characters
"??\n".

# Diagnostics

My best guess is that this is something to do with how readLines()
determines when it has reached EOF, because of the following:

- The file is terminated with an ASCII LF (0x0a), but R gives an
'incomplete final line found' warning. Note that in some implementations
of Code page 437, 0x0a is interpreted as a graphical character rather
than a control character, but this does not seem to be the problem here.
The same problem occurs if the file ends with 0x0d or 0x0d 0x0a.
- Adding seven or more characters to the end of the file makes it read
correctly
- Similarly, the file is read correctly if you remove three characters
from anywhere in the file
- The same issue seems to occur with reading files encoded in other DOS
code pages

# Additional information
> sessionInfo()R version 3.2.3 (2015-12-10)
Platform: x86_64-apple-darwin14.5.0 (64-bit)
Running under: OS X 10.10.5 (Yosemite)

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

The same behaviour occurs under R 2.15.1 on a Linux server.

In case the attached file is somehow corrupted, here is a hexdump:

00000000: 0b0c 0e0f 1011 1213 1415 1617 1819 1a1b  ................
00000010: 1c1d 1e1f 2021 2223 2425 2627 2829 2a2b  .... !"#$%&'()*+
00000020: 2c2d 2e2f 3031 3233 3435 3637 3839 3a3b  ,-./0123456789:;
00000030: 3c3d 3e3f 4041 4243 4445 4647 4849 4a4b  <=>?@ABCDEFGHIJK
00000040: 4c4d 4e4f 5051 5253 5455 5657 5859 5a5b  LMNOPQRSTUVWXYZ[
00000050: 5c5d 5e5f 6061 6263 6465 6667 6869 6a6b  \]^_`abcdefghijk
00000060: 6c6d 6e6f 7071 7273 7475 7677 7879 7a7b  lmnopqrstuvwxyz{
00000070: 7c7d 7e7f ffad 9b9c 9da6 aeaa f8f1 fde6  |}~.............
00000080: faa7 afac aba8 8e8f 9280 90a5 999a e185  ................
00000090: a083 8486 9187 8a82 8889 8da1 8c8b a495  ................
000000a0: a293 94f6 97a3 9681 989f e2e9 e4e8 eae0  ................
000000b0: ebee e3e5 e7ed fc9e f9fb ecef f7f0 f3f2  ................
000000c0: a9f4 f5c4 b3da bfc0 d9c3 b4c2 c1c5 cdba  ................
000000d0: d5d6 c9b8 b7bb d4d3 c8be bdbc c6c7 ccb5  ................
000000e0: b6b9 d1d2 cbcf d0ca d8d7 cedf dcdb ddde  ................
000000f0: b0b1 b2fe 0a                             .....


Has anyone encountered something similar?


Kind regards,

Adam Obeng

Martin Maechler

2016-Jun-08 08:50 UTC

head link

[Rd] readlines() truncates text file with Codepage 437 encoding

Appended is the file -- you need to tell your e-mail software to use
one of the MIME types that R-devel does accept;  text/plain
is what I chose

((Yes, as R mailing list server "operator", with a bit of detective
work,
  I was able to find the "uncleaned" e-mail and extract the
  attachment from it))

Martin Maechler
ETH Zurich


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 437__characters.txt
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20160608/d0b88325/attachment.txt>
-------------- next part --------------
>>>>> Adam Obeng <adam.obeng at columbia.edu>
>>>>>     on Mon, 6 Jun 2016 11:11:21 +0100 writes:
    > Hello r-devel, The attached Code page 437-encoded file
    > contains 245 characters (including the final newline), but
    > readLines only reads 242 of them:

    >> test_text <- readLines(file('437__characters.txt',
    >> encoding='437'))
    > Warning message: In readLines(file("437__characters.txt",
    > : incomplete final line found on '437__characters.txt'
    >> test_text
    > [1]
    >
"\v\f\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037
    >
!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\177
    >
?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????"
    >> nchar(test_text)
    > [1] 242

    > You'll note that readLines hasn't read the final
    > characters "??\n".

    > # Diagnostics

    > My best guess is that this is something to do with how
    > readLines() determines when it has reached EOF, because of
    > the following:

    > - The file is terminated with an ASCII LF (0x0a), but R
    > gives an 'incomplete final line found' warning. Note that
    > in some implementations of Code page 437, 0x0a is
    > interpreted as a graphical character rather than a control
    > character, but this does not seem to be the problem here.
    > The same problem occurs if the file ends with 0x0d or 0x0d
    > 0x0a.  - Adding seven or more characters to the end of the
    > file makes it read correctly - Similarly, the file is read
    > correctly if you remove three characters from anywhere in
    > the file - The same issue seems to occur with reading
    > files encoded in other DOS code pages

    > # Additional information

    >> sessionInfo()
    > R version 3.2.3 (2015-12-10) Platform:
    > x86_64-apple-darwin14.5.0 (64-bit) Running under: OS X
    > 10.10.5 (Yosemite)

    > locale: [1]
    > en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

    > attached base packages: [1] stats graphics grDevices utils
    > datasets methods base

    > The same behaviour occurs under R 2.15.1 on a Linux
    > server.

    > In case the attached file is somehow corrupted, here is a
    > hexdump:

    > 00000000: 0b0c 0e0f 1011 1213 1415 1617 1819 1a1b
    > ................  00000010: 1c1d 1e1f 2021 2223 2425 2627
    > 2829 2a2b .... !"#$%&'()*+ 00000020: 2c2d 2e2f 3031 3233
    > 3435 3637 3839 3a3b ,-./0123456789:; 00000030: 3c3d 3e3f
    > 4041 4243 4445 4647 4849 4a4b <=>?@ABCDEFGHIJK 00000040:
    > 4c4d 4e4f 5051 5253 5455 5657 5859 5a5b LMNOPQRSTUVWXYZ[
    > 00000050: 5c5d 5e5f 6061 6263 6465 6667 6869 6a6b
    > \]^_`abcdefghijk 00000060: 6c6d 6e6f 7071 7273 7475 7677
    > 7879 7a7b lmnopqrstuvwxyz{ 00000070: 7c7d 7e7f ffad 9b9c
    > 9da6 aeaa f8f1 fde6 |}~.............  00000080: faa7 afac
    > aba8 8e8f 9280 90a5 999a e185 ................  00000090:
    > a083 8486 9187 8a82 8889 8da1 8c8b a495 ................
    > 000000a0: a293 94f6 97a3 9681 989f e2e9 e4e8 eae0
    > ................  000000b0: ebee e3e5 e7ed fc9e f9fb ecef
    > f7f0 f3f2 ................  000000c0: a9f4 f5c4 b3da bfc0
    > d9c3 b4c2 c1c5 cdba ................  000000d0: d5d6 c9b8
    > b7bb d4d3 c8be bdbc c6c7 ccb5 ................  000000e0:
    > b6b9 d1d2 cbcf d0ca d8d7 cedf dcdb ddde ................
    > 000000f0: b0b1 b2fe 0a .....


    > Has anyone encountered something similar?


    > Kind regards,

    > Adam Obeng ______________________________________________
    > R-devel at r-project.org mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-devel

Martin Maechler

2016-Jun-09 14:40 UTC

head link

[Rd] readlines() truncates text file with Codepage 437 encoding

I can reproduce the issue on Linux (Fedora F22),
R 3.3.0 patched of today.

Here's code for experimenting which allows to reproduce the
issue without the need for an attached file (there's a temporary
file created and removed as part of the function below) :

##---------------------------------------------------------------------------

##' @title write-binary-readLines testing
##' @param i  vector of integers in 0:255 to be used as character codes
##' @param file.name optional
##' @param encoding "437" is the one where the problem has been
reported
##' @return the readLines() resulting character string with attributes
##' @author Martin Maechler
wb.readL <- function(i, file.name = tempfile("bin"), encoding =
"437") {
    stopifnot(is.integer(i), 0 <= i, i <= 255,
              is.character(file.name))
    ff <- file(file.name, "wb")
    writeBin(as.raw(i), ff)
    close(ff) ; on.exit(unlink(file.name))
    ## Now read "as codepage" :
    ch <- readLines(file(file.name, encoding = encoding))
    ##    ---------                 -------------------  typically gives warning
    structure(ch,
              fSize = file.size(file.name),
              nchars = c(b = nchar(ch, "b"),
                         c = nchar(ch, "c"),
                         w = nchar(ch, "w")))
}

ii <- c(11:12, 14:255, 10L)
(cc <- wb.readL(ii))

##---------------------------------------------------------------------------


gives
> (cc <- wb.readL(ii))[1]
"\v\f\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037
!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\177?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????"
attr(,"fSize")
[1] 245
attr(,"nchars")
  b   c   w
427 241 241
Warning message:
In readLines(file(file.name, encoding = encoding)) :
  incomplete final line found on
'/tmp/RtmpaPyDyp/bin65842896d5f1'>

Reasonably Related Threads

Search for more reasonably related threads

R devel - Jun 2016 - readlines() truncates text file with Codepage 437 encoding

[Rd] readlines() truncates text file with Codepage 437 encoding

[Rd] readlines() truncates text file with Codepage 437 encoding

[Rd] readlines() truncates text file with Codepage 437 encoding

Reasonably Related Threads