R devel - Apr 2020 - Plotmath on Fedora 31 broken with with pango >= 1.44 - workarounds?

Hi

Thanks for your input on this I?aki and Nicolas.

I am starting testing an R fix for this problem today.

As suggested, the plan is to allow the R user to specify a font family 
other than "symbol" for plotmath output (or, more generally, in R 
parlance, for 'font=5' or 'fontface=5') on a Cairo-based
graphics device.

Paul


On 27/03/20 11:30 pm, I?aki Ucar wrote:
> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
> <nicolas.mailhot at laposte.net> wrote:
>>
>> <snip>
>>
>> R brought this all on itself by hardcoding a Windows-only ?Symbol? font
>> family name in its default conf. Linux systems are UTF-8 by default for
>> ~20 years now, they don?t need the forcing of magic font families to
>> handle symbols not present in the 8-bit legacy Windows encodings.
>>
>> The actual effect of this conf is not the selection of font files with
>> special and unusual symbols. It is to priorize fonts that match the
>> "Symbol" magic name. And those fonts are few and crumbling on
Linux
>> systems, because no one has needed to bother with them since Linux
>> switched to UTF-8 last millenium.
>>
>> Just stop using ?Symbol? in R and things will work a lot better.
>> Alternatively, prepare to maintain the ?Symbol? aliasing stack in
>> fontconfig (and fight with wine for it), because *no* *one* *else*
>> *cares* about this legacy Windows-specific stuff.
> 
> So, in the light of Nicolas' input (thanks!), I think that font
> selection should be fixed upstream in R. I'd be happy to put all this
> together in R's bugzilla, but I don't have an account. Could
someone
> please invite me?
> 
> I?aki
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

Thanks, Paul. I've created a bug report to keep track of this
(https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17748), and taken
the liberty of adding you to CC. We'll need to cherry-pick the fix for
R 3.6.3 in Fedora 31.

I?aki

On Sun, 29 Mar 2020 at 21:15, Paul Murrell <paul at stat.auckland.ac.nz>
wrote:
>
> Hi
>
> Thanks for your input on this I?aki and Nicolas.
>
> I am starting testing an R fix for this problem today.
>
> As suggested, the plan is to allow the R user to specify a font family
> other than "symbol" for plotmath output (or, more generally, in R
> parlance, for 'font=5' or 'fontface=5') on a Cairo-based
graphics device.
>
> Paul
>
>
> On 27/03/20 11:30 pm, I?aki Ucar wrote:
> > On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
> > <nicolas.mailhot at laposte.net> wrote:
> >>
> >> <snip>
> >>
> >> R brought this all on itself by hardcoding a Windows-only ?Symbol?
font
> >> family name in its default conf. Linux systems are UTF-8 by
default for
> >> ~20 years now, they don?t need the forcing of magic font families
to
> >> handle symbols not present in the 8-bit legacy Windows encodings.
> >>
> >> The actual effect of this conf is not the selection of font files
with
> >> special and unusual symbols. It is to priorize fonts that match
the
> >> "Symbol" magic name. And those fonts are few and
crumbling on Linux
> >> systems, because no one has needed to bother with them since Linux
> >> switched to UTF-8 last millenium.
> >>
> >> Just stop using ?Symbol? in R and things will work a lot better.
> >> Alternatively, prepare to maintain the ?Symbol? aliasing stack in
> >> fontconfig (and fight with wine for it), because *no* *one* *else*
> >> *cares* about this legacy Windows-specific stuff.
> >
> > So, in the light of Nicolas' input (thanks!), I think that font
> > selection should be fixed upstream in R. I'd be happy to put all
this
> > together in R's bugzilla, but I don't have an account. Could
someone
> > please invite me?
> >
> > I?aki
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >


-- 
I?aki ?car

Hi

I have created an R branch that contains a potential fix ...

https://svn.r-project.org/R/branches/R-symfam/

This allows, for example, ...

cairo_pdf(symbolfamily="OpenSymbol")

... to specify that the OpenSymbol family should be used as the
"symbol"
font (e.g., for "plotmath") in R.

This is just a separate branch for now because, while I have tested it 
under Unbuntu 18.04 and Fedora 31, I cannot even build R for Windows 
(right now) or Mac (ever) and I do not want to drop a bomb on R-devel at 
this stage of the release process for R 4.0.0.

The attached file contains at least an outline of steps required to do a 
minimal test if anyone wants to try the fix on Linux.

cc'ing Simon and Jeroen in case they are able to help with checking that 
this builds and works on Mac and/or Windows.

NOTEs:
- 'symbolfamily' can only be specified when a graphics device is opened,
and it is then fixed for that device.
- on Windows, for cairo-based devices, the "symbol" font is still 
hard-coded as "Standard Symbols L"

Paul

On 30/03/20 8:15 am, Paul Murrell wrote:
> Hi
> 
> Thanks for your input on this I?aki and Nicolas.
> 
> I am starting testing an R fix for this problem today.
> 
> As suggested, the plan is to allow the R user to specify a font family 
> other than "symbol" for plotmath output (or, more generally, in R
> parlance, for 'font=5' or 'fontface=5') on a Cairo-based
graphics device.
> 
> Paul
> 
> 
> On 27/03/20 11:30 pm, I?aki Ucar wrote:
>> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
>> <nicolas.mailhot at laposte.net> wrote:
>>>
>>> <snip>
>>>
>>> R brought this all on itself by hardcoding a Windows-only ?Symbol?
font
>>> family name in its default conf. Linux systems are UTF-8 by default
for
>>> ~20 years now, they don?t need the forcing of magic font families
to
>>> handle symbols not present in the 8-bit legacy Windows encodings.
>>>
>>> The actual effect of this conf is not the selection of font files
with
>>> special and unusual symbols. It is to priorize fonts that match the
>>> "Symbol" magic name. And those fonts are few and
crumbling on Linux
>>> systems, because no one has needed to bother with them since Linux
>>> switched to UTF-8 last millenium.
>>>
>>> Just stop using ?Symbol? in R and things will work a lot better.
>>> Alternatively, prepare to maintain the ?Symbol? aliasing stack in
>>> fontconfig (and fight with wine for it), because *no* *one* *else*
>>> *cares* about this legacy Windows-specific stuff.
>>
>> So, in the light of Nicolas' input (thanks!), I think that font
>> selection should be fixed upstream in R. I'd be happy to put all
this
>> together in R's bugzilla, but I don't have an account. Could
someone
>> please invite me?
>>
>> I?aki
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
-- 
Dr Paul Murrell
Department of Statistics
The University of Auckland
Private Bag 92019
Auckland
New Zealand
64 9 3737599 x85392
paul at stat.auckland.ac.nz
http://www.stat.auckland.ac.nz/~paul/

I do my devel/patch work on Mac so I can take a shot at testing your branch
in the next couple days.

~G

On Sun, Mar 29, 2020 at 7:24 PM Paul Murrell <paul at stat.auckland.ac.nz>
wrote:
> Hi
>
> I have created an R branch that contains a potential fix ...
>
> https://svn.r-project.org/R/branches/R-symfam/
>
> This allows, for example, ...
>
> cairo_pdf(symbolfamily="OpenSymbol")
>
> ... to specify that the OpenSymbol family should be used as the
"symbol"
> font (e.g., for "plotmath") in R.
>
> This is just a separate branch for now because, while I have tested it
> under Unbuntu 18.04 and Fedora 31, I cannot even build R for Windows
> (right now) or Mac (ever) and I do not want to drop a bomb on R-devel at
> this stage of the release process for R 4.0.0.
>
> The attached file contains at least an outline of steps required to do a
> minimal test if anyone wants to try the fix on Linux.
>
> cc'ing Simon and Jeroen in case they are able to help with checking
that
> this builds and works on Mac and/or Windows.
>
> NOTEs:
> - 'symbolfamily' can only be specified when a graphics device is
opened,
> and it is then fixed for that device.
> - on Windows, for cairo-based devices, the "symbol" font is still
> hard-coded as "Standard Symbols L"
>
> Paul
>
> On 30/03/20 8:15 am, Paul Murrell wrote:
> > Hi
> >
> > Thanks for your input on this I?aki and Nicolas.
> >
> > I am starting testing an R fix for this problem today.
> >
> > As suggested, the plan is to allow the R user to specify a font family
> > other than "symbol" for plotmath output (or, more generally,
in R
> > parlance, for 'font=5' or 'fontface=5') on a
Cairo-based graphics device.
> >
> > Paul
> >
> >
> > On 27/03/20 11:30 pm, I?aki Ucar wrote:
> >> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
> >> <nicolas.mailhot at laposte.net> wrote:
> >>>
> >>> <snip>
> >>>
> >>> R brought this all on itself by hardcoding a Windows-only
?Symbol? font
> >>> family name in its default conf. Linux systems are UTF-8 by
default for
> >>> ~20 years now, they don?t need the forcing of magic font
families to
> >>> handle symbols not present in the 8-bit legacy Windows
encodings.
> >>>
> >>> The actual effect of this conf is not the selection of font
files with
> >>> special and unusual symbols. It is to priorize fonts that
match the
> >>> "Symbol" magic name. And those fonts are few and
crumbling on Linux
> >>> systems, because no one has needed to bother with them since
Linux
> >>> switched to UTF-8 last millenium.
> >>>
> >>> Just stop using ?Symbol? in R and things will work a lot
better.
> >>> Alternatively, prepare to maintain the ?Symbol? aliasing stack
in
> >>> fontconfig (and fight with wine for it), because *no* *one*
*else*
> >>> *cares* about this legacy Windows-specific stuff.
> >>
> >> So, in the light of Nicolas' input (thanks!), I think that
font
> >> selection should be fixed upstream in R. I'd be happy to put
all this
> >> together in R's bugzilla, but I don't have an account.
Could someone
> >> please invite me?
> >>
> >> I?aki
> >>
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>
>
> --
> Dr Paul Murrell
> Department of Statistics
> The University of Auckland
> Private Bag 92019
> Auckland
> New Zealand
> 64 9 3737599 x85392
> paul at stat.auckland.ac.nz
> http://www.stat.auckland.ac.nz/~paul/
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]

On Mon, 30 Mar 2020 at 04:24, Paul Murrell <paul at stat.auckland.ac.nz>
wrote:
>
> Hi
>
> I have created an R branch that contains a potential fix ...
>
> https://svn.r-project.org/R/branches/R-symfam/
>
> This allows, for example, ...
>
> cairo_pdf(symbolfamily="OpenSymbol")
>
> ... to specify that the OpenSymbol family should be used as the
"symbol"
> font (e.g., for "plotmath") in R.
Will this be a default on Linux? Or are you planning any mechanism
(env variable, option...) to make it the default? Because, otherwise,
as pango is updated across distributions, R graphics will be "broken"
by default unless the user explicitly calls the graphics device in
that way to set that option, which I would say is uncommon.

I?aki
> This is just a separate branch for now because, while I have tested it
> under Unbuntu 18.04 and Fedora 31, I cannot even build R for Windows
> (right now) or Mac (ever) and I do not want to drop a bomb on R-devel at
> this stage of the release process for R 4.0.0.
>
> The attached file contains at least an outline of steps required to do a
> minimal test if anyone wants to try the fix on Linux.
>
> cc'ing Simon and Jeroen in case they are able to help with checking
that
> this builds and works on Mac and/or Windows.
>
> NOTEs:
> - 'symbolfamily' can only be specified when a graphics device is
opened,
> and it is then fixed for that device.
> - on Windows, for cairo-based devices, the "symbol" font is still
> hard-coded as "Standard Symbols L"
>
>
> Paul
>
> On 30/03/20 8:15 am, Paul Murrell wrote:
> > Hi
> >
> > Thanks for your input on this I?aki and Nicolas.
> >
> > I am starting testing an R fix for this problem today.
> >
> > As suggested, the plan is to allow the R user to specify a font family
> > other than "symbol" for plotmath output (or, more generally,
in R
> > parlance, for 'font=5' or 'fontface=5') on a
Cairo-based graphics device.
> >
> > Paul
> >
> >
> > On 27/03/20 11:30 pm, I?aki Ucar wrote:
> >> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
> >> <nicolas.mailhot at laposte.net> wrote:
> >>>
> >>> <snip>
> >>>
> >>> R brought this all on itself by hardcoding a Windows-only
?Symbol? font
> >>> family name in its default conf. Linux systems are UTF-8 by
default for
> >>> ~20 years now, they don?t need the forcing of magic font
families to
> >>> handle symbols not present in the 8-bit legacy Windows
encodings.
> >>>
> >>> The actual effect of this conf is not the selection of font
files with
> >>> special and unusual symbols. It is to priorize fonts that
match the
> >>> "Symbol" magic name. And those fonts are few and
crumbling on Linux
> >>> systems, because no one has needed to bother with them since
Linux
> >>> switched to UTF-8 last millenium.
> >>>
> >>> Just stop using ?Symbol? in R and things will work a lot
better.
> >>> Alternatively, prepare to maintain the ?Symbol? aliasing stack
in
> >>> fontconfig (and fight with wine for it), because *no* *one*
*else*
> >>> *cares* about this legacy Windows-specific stuff.
> >>
> >> So, in the light of Nicolas' input (thanks!), I think that
font
> >> selection should be fixed upstream in R. I'd be happy to put
all this
> >> together in R's bugzilla, but I don't have an account.
Could someone
> >> please invite me?
> >>
> >> I?aki
> >>
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>
>
> --
> Dr Paul Murrell
> Department of Statistics
> The University of Auckland
> Private Bag 92019
> Auckland
> New Zealand
> 64 9 3737599 x85392
> paul at stat.auckland.ac.nz
> http://www.stat.auckland.ac.nz/~paul/


-- 
I?aki ?car

On Mon, 30 Mar 2020 at 04:24, Paul Murrell <paul at stat.auckland.ac.nz>
wrote:
>
> Hi
>
> I have created an R branch that contains a potential fix ...
>
> https://svn.r-project.org/R/branches/R-symfam/
>
> This allows, for example, ...
>
> cairo_pdf(symbolfamily="OpenSymbol")
>
> ... to specify that the OpenSymbol family should be used as the
"symbol"
> font (e.g., for "plotmath") in R.
>
> This is just a separate branch for now because, while I have tested it
> under Unbuntu 18.04 and Fedora 31, I cannot even build R for Windows
> (right now) or Mac (ever) and I do not want to drop a bomb on R-devel at
> this stage of the release process for R 4.0.0.
>
> The attached file contains at least an outline of steps required to do a
> minimal test if anyone wants to try the fix on Linux.
>
> cc'ing Simon and Jeroen in case they are able to help with checking
that
> this builds and works on Mac and/or Windows.
>
> NOTEs:
> - 'symbolfamily' can only be specified when a graphics device is
opened,
> and it is then fixed for that device.
> - on Windows, for cairo-based devices, the "symbol" font is still
> hard-coded as "Standard Symbols L"
>
> Paul
>
> On 30/03/20 8:15 am, Paul Murrell wrote:
> > Hi
> >
> > Thanks for your input on this I?aki and Nicolas.
> >
> > I am starting testing an R fix for this problem today.
> >
> > As suggested, the plan is to allow the R user to specify a font family
> > other than "symbol" for plotmath output (or, more generally,
in R
> > parlance, for 'font=5' or 'fontface=5') on a
Cairo-based graphics device.
> >
> > Paul
> >
> >
> > On 27/03/20 11:30 pm, I?aki Ucar wrote:
> >> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
> >> <nicolas.mailhot at laposte.net> wrote:
> >>>
> >>> <snip>
> >>>
> >>> R brought this all on itself by hardcoding a Windows-only
?Symbol? font
> >>> family name in its default conf. Linux systems are UTF-8 by
default for
> >>> ~20 years now, they don?t need the forcing of magic font
families to
> >>> handle symbols not present in the 8-bit legacy Windows
encodings.
> >>>
> >>> The actual effect of this conf is not the selection of font
files with
> >>> special and unusual symbols. It is to priorize fonts that
match the
> >>> "Symbol" magic name. And those fonts are few and
crumbling on Linux
> >>> systems, because no one has needed to bother with them since
Linux
> >>> switched to UTF-8 last millenium.
> >>>
> >>> Just stop using ?Symbol? in R and things will work a lot
better.
> >>> Alternatively, prepare to maintain the ?Symbol? aliasing stack
in
> >>> fontconfig (and fight with wine for it), because *no* *one*
*else*
> >>> *cares* about this legacy Windows-specific stuff.
> >>
> >> So, in the light of Nicolas' input (thanks!), I think that
font
> >> selection should be fixed upstream in R. I'd be happy to put
all this
> >> together in R's bugzilla, but I don't have an account.
Could someone
> >> please invite me?
> >>
> >> I?aki
> >>
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>
>
> --
> Dr Paul Murrell
> Department of Statistics
> The University of Auckland
> Private Bag 92019
> Auckland
> New Zealand
> 64 9 3737599 x85392
> paul at stat.auckland.ac.nz
> http://www.stat.auckland.ac.nz/~paul/


-- 
I?aki ?car

Le lundi 30 mars 2020 ? 15:24 +1300, Paul Murrell a ?crit
:
> Hi
> 
> I have created an R branch that contains a potential fix ...
> 
> https://svn.r-project.org/R/branches/R-symfam/
> 
> This allows, for example, ...
> 
> cairo_pdf(symbolfamily="OpenSymbol")
> 
> ... to specify that the OpenSymbol family should be used as the
> "symbol" font (e.g., for "plotmath") in R.
Thanks for looking at it!

But, really, there is no such thing as a Symbol font on Linux anymore.
Symbol is pre-unicode thinking. Most modern general-purpose unicode
fonts will include every codepoint Symbol ever shipped, and fontconfig
will fallback gracefully when that?s not the case (unless your
fontconfig integration is broken).

Just use the sans-serif or monospace fontconfig defaults. You don?t
need Symbol, or OpenSymbol, or any special font setup.

Symbol?s codepoint coverage is laughable by 2020?s UTF-8 standards.

Symbol << Normal Unicode font (DejaVu*) << Special math fonts
(STIX2)

I you do advanced math stuff, you may need a special math font like
STIX, but that?s lights years more advanced than Symbol, and a general
purpose font like DejaVu has been shipping a MATH block for several
years now

Regards,

-- 
Nicolas Mailhot

Hi

The R branch ...

https://svn.r-project.org/R/branches/R-symfam/

... is now set up so that it works "out of the box" on Fedora by
setting
the default to be 'symbolfamily=cairoSymbolFont(family, usePUA=FALSE)' 
when grSoftVersion()["pango"] is greater than "1.44".

This means that on Fedora 31 (at least on the Docker container I am 
testing on) "sans"->"NimubusSans" is used as the symbol
font by default
and R converts Adobe Symbol Encoding code points to non-PUA UTF8 code 
points.  This is not the prettiest result, but it is a lot better than 
the page full of missing glyphs that we had.

The default on less "bleeding edge" systems, e.g., my Ubuntu 18.04, 
remains 'symbolfamily="Symbol"'.

The default on other platforms is supposed to be the same as it was, but 
I need help to confirm that.  I have set up a github repo ...

https://github.com/pmur002/R-symfam-testing

... that describes how to test this on macOS and Windows if anyone has 
time to do so.

I will start trying to set up a Windows test unless someone beats me to it.

Paul

On 30/03/20 3:24 pm, Paul Murrell wrote:
> Hi
> 
> I have created an R branch that contains a potential fix ...
> 
> https://svn.r-project.org/R/branches/R-symfam/
> 
> This allows, for example, ...
> 
> cairo_pdf(symbolfamily="OpenSymbol")
> 
> ... to specify that the OpenSymbol family should be used as the
"symbol"
> font (e.g., for "plotmath") in R.
> 
> This is just a separate branch for now because, while I have tested it 
> under Unbuntu 18.04 and Fedora 31, I cannot even build R for Windows 
> (right now) or Mac (ever) and I do not want to drop a bomb on R-devel at 
> this stage of the release process for R 4.0.0.
> 
> The attached file contains at least an outline of steps required to do a 
> minimal test if anyone wants to try the fix on Linux.
> 
> cc'ing Simon and Jeroen in case they are able to help with checking
that
> this builds and works on Mac and/or Windows.
> 
> NOTEs:
> - 'symbolfamily' can only be specified when a graphics device is
opened,
> and it is then fixed for that device.
> - on Windows, for cairo-based devices, the "symbol" font is still
> hard-coded as "Standard Symbols L"
> 
> Paul
> 
> On 30/03/20 8:15 am, Paul Murrell wrote:
>> Hi
>>
>> Thanks for your input on this I?aki and Nicolas.
>>
>> I am starting testing an R fix for this problem today.
>>
>> As suggested, the plan is to allow the R user to specify a font family 
>> other than "symbol" for plotmath output (or, more generally,
in R
>> parlance, for 'font=5' or 'fontface=5') on a
Cairo-based graphics device.
>>
>> Paul
>>
>>
>> On 27/03/20 11:30 pm, I?aki Ucar wrote:
>>> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
>>> <nicolas.mailhot at laposte.net> wrote:
>>>>
>>>> <snip>
>>>>
>>>> R brought this all on itself by hardcoding a Windows-only
?Symbol? font
>>>> family name in its default conf. Linux systems are UTF-8 by
default for
>>>> ~20 years now, they don?t need the forcing of magic font
families to
>>>> handle symbols not present in the 8-bit legacy Windows
encodings.
>>>>
>>>> The actual effect of this conf is not the selection of font
files with
>>>> special and unusual symbols. It is to priorize fonts that match
the
>>>> "Symbol" magic name. And those fonts are few and
crumbling on Linux
>>>> systems, because no one has needed to bother with them since
Linux
>>>> switched to UTF-8 last millenium.
>>>>
>>>> Just stop using ?Symbol? in R and things will work a lot
better.
>>>> Alternatively, prepare to maintain the ?Symbol? aliasing stack
in
>>>> fontconfig (and fight with wine for it), because *no* *one*
*else*
>>>> *cares* about this legacy Windows-specific stuff.
>>>
>>> So, in the light of Nicolas' input (thanks!), I think that font
>>> selection should be fixed upstream in R. I'd be happy to put
all this
>>> together in R's bugzilla, but I don't have an account.
Could someone
>>> please invite me?
>>>
>>> I?aki
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
> 
-- 
Dr Paul Murrell
Department of Statistics
The University of Auckland
Private Bag 92019
Auckland
New Zealand
64 9 3737599 x85392
paul at stat.auckland.ac.nz
http://www.stat.auckland.ac.nz/~paul/