R devel - Aug 2025 - Non-ASCII citation keys prevent compiling with LC_ALL=C

Hello R-devel,

I've been watching the development of automatic Rd bibliography
generation with great interest and I'm looking forward to using
\bibcitet{...} and \bibshow{*} in my packages. Currently, non-ASCII
characters used in the citation keys prevent R from successfully
compiling when the current locale encoding is unable to represent them:

% touch src/library/stats/man/factanal.Rd && LC_ALL=C make
...
installing parsed Rd
make[3]: Entering directory '.../src/library'
  base
Error: factanal.Rd:99: (converted from warning) Could not find
bibentries for the following keys: %s
  'R:J<U+00F6>reskog:1963'
Execution halted
make[3]: *** [Makefile:76: stats.Rdts] Error 1

But as long as the locale encoding can represent the key, it's fine:

% touch src/library/stats/man/factanal.Rd && \
 LC_ALL=en_GB.iso885915 luit make
(works well without a UTF-8 locale)

I think this can be made to work by telling tools:::process_Rd() ->
tools:::processRdChunk() to parse character strings in R code as UTF-8:

Index: src/library/tools/R/RdConv2.R
==================================================================---
src/library/tools/R/RdConv2.R	(revision 88617)
+++ src/library/tools/R/RdConv2.R	(working copy)
@@ -229,8 +229,8 @@
 	code <- structure(code[tags != "COMMENT"],
 	                  srcref = codesrcref) # retain for error locations
 	chunkexps <- tryCatch(
-	    parse(text = sub("\n$", "", as.character(code)),
-	          keep.source = options$keep.source),
+	    parse(text = sub("\n$", "",
enc2utf8(as.character(code))),
+	          keep.source = options$keep.source, encoding = "UTF-8"),
 	    error = function (e) stopRd(code, Rdfile, conditionMessage(e))
 	)
 
That enc2utf8() may be extraneous, since tools::parse_Rd() is
documented to convert text to UTF-8 while parsing. The downsides are,
of course, parse(encoding=...) not working with MBCS locales and the
ever-present danger of breaking some user code that depends on the
current behaviour (this was tested using 'make check-devel', not on
CRAN packages).

Should R compile under LC_ALL=C? Maybe it's time for people whose
builds are failing to switch the continuous integration containers from
C to C.UTF-8?

-- 
Best regards,
Ivan

>>>>> Ivan Krylov via R-devel writes:
> Hello R-devel,
> I've been watching the development of automatic Rd bibliography
> generation with great interest and I'm looking forward to using
> \bibcitet{...} and \bibshow{*} in my packages. 
Thanks! :-)
> Currently, non-ASCII characters used in the citation keys prevent R
> from successfully compiling when the current locale encoding is unable
> to represent them:
> % touch src/library/stats/man/factanal.Rd && LC_ALL=C make
> ...
> installing parsed Rd
> make[3]: Entering directory '.../src/library'
>   base
> Error: factanal.Rd:99: (converted from warning) Could not find
> bibentries for the following keys: %s
>   'R:J<U+00F6>reskog:1963'
> Execution halted
> make[3]: *** [Makefile:76: stats.Rdts] Error 1
> But as long as the locale encoding can represent the key, it's fine:
> % touch src/library/stats/man/factanal.Rd && \
>  LC_ALL=en_GB.iso885915 luit make
> (works well without a UTF-8 locale)
Oh dear.  I thought we have coverage for this from building daily
snapshots with LC_ALL=C, but apparently not.  There were 10 non-ASCII
keys so far: I have for now changed them to all ASCII.

But clearly, when a package declares its Rd files to be in UTF-8 one
would expect that Sexpr macros can also take UTF-8, but that's not so
simple given that it involves calling the R parser.  Your suggested
change looks good to me: non-UTF-8 MBCS locales have a problem with
parse(encoding = "UTF-8"), but I don't think we have real coverage
for
these.

(Afaic, in principle, it might be nice to make these "work" via
writing
to a tempfile, parsing from their with re-encoding, and at the end run
enc2utf8() on all strings obtained, but that's not so simple ...)

Anyway, need to discuss this a bit more within R Core.  For now, things
"work" again with LC_ALL=C.  

(My regular checks use C.UTF-8, but I am not sure how universally
available this is?)

Best

> I think this can be made to work by telling tools:::process_Rd() ->
> tools:::processRdChunk() to parse character strings in R code as UTF-8:
> Index: src/library/tools/R/RdConv2.R
> ==================================================================> ---
src/library/tools/R/RdConv2.R	(revision 88617)
> +++ src/library/tools/R/RdConv2.R	(working copy)
> @@ -229,8 +229,8 @@
>  	code <- structure(code[tags != "COMMENT"],
>  	                  srcref = codesrcref) # retain for error locations
>  	chunkexps <- tryCatch(
> -	    parse(text = sub("\n$", "", as.character(code)),
> -	          keep.source = options$keep.source),
> +	    parse(text = sub("\n$", "",
enc2utf8(as.character(code))),
> +	          keep.source = options$keep.source, encoding =
"UTF-8"),
>  	    error = function (e) stopRd(code, Rdfile, conditionMessage(e))
>  	)
 
> That enc2utf8() may be extraneous, since tools::parse_Rd() is
> documented to convert text to UTF-8 while parsing. The downsides are,
> of course, parse(encoding=...) not working with MBCS locales and the
> ever-present danger of breaking some user code that depends on the
> current behaviour (this was tested using 'make check-devel', not on
> CRAN packages).
> Should R compile under LC_ALL=C? Maybe it's time for people whose
> builds are failing to switch the continuous integration containers from
> C to C.UTF-8?
> -- 
> Best regards,
> Ivan
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel