R devel - Apr 2020 - Plotmath on Fedora 31 broken with with pango >= 1.44 - workarounds?

Paul et al,

I will try to do this tonight or tomorrow, though it will not be built with
th system tools because I have yet to get that tto work locally (spent a
good chunk of this morning trying).

I will send a separate messaage regarding those difficulties as well so
that we can at least confirm that they are due to a malconfiguration on my
part.

Best,
~G

On Tue, Apr 7, 2020 at 7:25 PM Paul Murrell <paul at stat.auckland.ac.nz>
wrote:
>
> The R-symfam branch (r78176) is now working, for my basic tests, on ...
>
> Ubuntu (pango < 1.44)
> Ubuntu (no pango)
> Fedora (pango > 1.44)
> Windows
>
> I need help to confirm that this builds on macOS and that the basic
> tests work ...
>
> https://github.com/pmur002/R-symfam-testing
>
> Brian has been helping with the build, but I am still looking for
> someone who can run the tests please.  Happy to be fed PDF files to
> scrutinize myself;  it's generating the PDF files on macOS that I need
> help with.
>
> Paul
>
> On 6/04/20 2:59 pm, Paul Murrell wrote:
> > Hi
> >
> > The R branch ...
> >
> > https://svn.r-project.org/R/branches/R-symfam/
> >
> > ... is now set up so that it works "out of the box" on
Fedora by setting
> > the default to be 'symbolfamily=cairoSymbolFont(family,
usePUA=FALSE)'
> > when grSoftVersion()["pango"] is greater than
"1.44".
> >
> > This means that on Fedora 31 (at least on the Docker container I am
> > testing on) "sans"->"NimubusSans" is used as
the symbol font by default
> > and R converts Adobe Symbol Encoding code points to non-PUA UTF8 code
> > points.  This is not the prettiest result, but it is a lot better than
> > the page full of missing glyphs that we had.
> >
> > The default on less "bleeding edge" systems, e.g., my Ubuntu
18.04,
> > remains 'symbolfamily="Symbol"'.
> >
> > The default on other platforms is supposed to be the same as it was,
but
> > I need help to confirm that.  I have set up a github repo ...
> >
> > https://github.com/pmur002/R-symfam-testing
> >
> > ... that describes how to test this on macOS and Windows if anyone has
> > time to do so.
> >
> > I will start trying to set up a Windows test unless someone beats me
to
> it.
> >
> > Paul
> >
> > On 30/03/20 3:24 pm, Paul Murrell wrote:
> >> Hi
> >>
> >> I have created an R branch that contains a potential fix ...
> >>
> >> https://svn.r-project.org/R/branches/R-symfam/
> >>
> >> This allows, for example, ...
> >>
> >> cairo_pdf(symbolfamily="OpenSymbol")
> >>
> >> ... to specify that the OpenSymbol family should be used as the
> >> "symbol" font (e.g., for "plotmath") in R.
> >>
> >> This is just a separate branch for now because, while I have
tested it
> >> under Unbuntu 18.04 and Fedora 31, I cannot even build R for
Windows
> >> (right now) or Mac (ever) and I do not want to drop a bomb on
R-devel
> >> at this stage of the release process for R 4.0.0.
> >>
> >> The attached file contains at least an outline of steps required
to do
> >> a minimal test if anyone wants to try the fix on Linux.
> >>
> >> cc'ing Simon and Jeroen in case they are able to help with
checking
> >> that this builds and works on Mac and/or Windows.
> >>
> >> NOTEs:
> >> - 'symbolfamily' can only be specified when a graphics
device is
> >> opened, and it is then fixed for that device.
> >> - on Windows, for cairo-based devices, the "symbol" font
is still
> >> hard-coded as "Standard Symbols L"
> >>
> >> Paul
> >>
> >> On 30/03/20 8:15 am, Paul Murrell wrote:
> >>> Hi
> >>>
> >>> Thanks for your input on this I?aki and Nicolas.
> >>>
> >>> I am starting testing an R fix for this problem today.
> >>>
> >>> As suggested, the plan is to allow the R user to specify a
font
> >>> family other than "symbol" for plotmath output (or,
more generally,
> >>> in R parlance, for 'font=5' or 'fontface=5')
on a Cairo-based
> >>> graphics device.
> >>>
> >>> Paul
> >>>
> >>>
> >>> On 27/03/20 11:30 pm, I?aki Ucar wrote:
> >>>> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
> >>>> <nicolas.mailhot at laposte.net> wrote:
> >>>>>
> >>>>> <snip>
> >>>>>
> >>>>> R brought this all on itself by hardcoding a
Windows-only ?Symbol?
> >>>>> font
> >>>>> family name in its default conf. Linux systems are
UTF-8 by default
> >>>>> for
> >>>>> ~20 years now, they don?t need the forcing of magic
font families to
> >>>>> handle symbols not present in the 8-bit legacy Windows
encodings.
> >>>>>
> >>>>> The actual effect of this conf is not the selection of
font files
> with
> >>>>> special and unusual symbols. It is to priorize fonts
that match the
> >>>>> "Symbol" magic name. And those fonts are few
and crumbling on Linux
> >>>>> systems, because no one has needed to bother with them
since Linux
> >>>>> switched to UTF-8 last millenium.
> >>>>>
> >>>>> Just stop using ?Symbol? in R and things will work a
lot better.
> >>>>> Alternatively, prepare to maintain the ?Symbol?
aliasing stack in
> >>>>> fontconfig (and fight with wine for it), because *no*
*one* *else*
> >>>>> *cares* about this legacy Windows-specific stuff.
> >>>>
> >>>> So, in the light of Nicolas' input (thanks!), I think
that font
> >>>> selection should be fixed upstream in R. I'd be happy
to put all this
> >>>> together in R's bugzilla, but I don't have an
account. Could someone
> >>>> please invite me?
> >>>>
> >>>> I?aki
> >>>>
> >>>> ______________________________________________
> >>>> R-devel at r-project.org mailing list
> >>>> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>>>
> >>
> >
>
> --
> Dr Paul Murrell
> Department of Statistics
> The University of Auckland
> Private Bag 92019
> Auckland
> New Zealand
> 64 9 3737599 x85392
> paul at stat.auckland.ac.nz
> http://www.stat.auckland.ac.nz/~paul/
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]

Hi Paul,

So I've run the tests (and am  attaching all numerous pdfs here) but the
takeaway from what I can see is as follows:

raw, without specifying font family, "look good" (see plot*raw.pdf) 
from
your branch (my eye is upset by the positioning of the phi symbol in plot2,
but I've confirmed that it looks the same generated from 3.5.1, so
that isn't your branch).

The various font family settings seem to work too, from what I can tell.
Both font families you suggested, however, Helvetica and Apple Symbols (the
s is important) have significantly incomplete coverage with PUA on. Apple
Symbols does have nearly complete coverage (though  to my eye the symbols
are all smaller...) with PUA turned off, but Helvetica remains very spotty,
with disabling PUA only modestly increasing it's coverage, and not in the
places that seem likely to matter.

I hope that helps,
~G

On Tue, Apr 7, 2020 at 8:59 PM Gabriel Becker <gabembecker at gmail.com>
wrote:
> Paul et al,
>
> I will try to do this tonight or tomorrow, though it will not be built
> with th system tools because I have yet to get that tto work locally (spent
> a good chunk of this morning trying).
>
> I will send a separate messaage regarding those difficulties as well so
> that we can at least confirm that they are due to a malconfiguration on my
> part.
>
> Best,
> ~G
>
> On Tue, Apr 7, 2020 at 7:25 PM Paul Murrell <paul at
stat.auckland.ac.nz>
> wrote:
>
>>
>> The R-symfam branch (r78176) is now working, for my basic tests, on ...
>>
>> Ubuntu (pango < 1.44)
>> Ubuntu (no pango)
>> Fedora (pango > 1.44)
>> Windows
>>
>> I need help to confirm that this builds on macOS and that the basic
>> tests work ...
>>
>> https://github.com/pmur002/R-symfam-testing
>>
>> Brian has been helping with the build, but I am still looking for
>> someone who can run the tests please.  Happy to be fed PDF files to
>> scrutinize myself;  it's generating the PDF files on macOS that I
need
>> help with.
>>
>> Paul
>>
>> On 6/04/20 2:59 pm, Paul Murrell wrote:
>> > Hi
>> >
>> > The R branch ...
>> >
>> > https://svn.r-project.org/R/branches/R-symfam/
>> >
>> > ... is now set up so that it works "out of the box" on
Fedora by
>> setting
>> > the default to be 'symbolfamily=cairoSymbolFont(family,
usePUA=FALSE)'
>> > when grSoftVersion()["pango"] is greater than
"1.44".
>> >
>> > This means that on Fedora 31 (at least on the Docker container I
am
>> > testing on) "sans"->"NimubusSans" is used
as the symbol font by default
>> > and R converts Adobe Symbol Encoding code points to non-PUA UTF8
code
>> > points.  This is not the prettiest result, but it is a lot better
than
>> > the page full of missing glyphs that we had.
>> >
>> > The default on less "bleeding edge" systems, e.g., my
Ubuntu 18.04,
>> > remains 'symbolfamily="Symbol"'.
>> >
>> > The default on other platforms is supposed to be the same as it
was,
>> but
>> > I need help to confirm that.  I have set up a github repo ...
>> >
>> > https://github.com/pmur002/R-symfam-testing
>> >
>> > ... that describes how to test this on macOS and Windows if anyone
has
>> > time to do so.
>> >
>> > I will start trying to set up a Windows test unless someone beats
me to
>> it.
>> >
>> > Paul
>> >
>> > On 30/03/20 3:24 pm, Paul Murrell wrote:
>> >> Hi
>> >>
>> >> I have created an R branch that contains a potential fix ...
>> >>
>> >> https://svn.r-project.org/R/branches/R-symfam/
>> >>
>> >> This allows, for example, ...
>> >>
>> >> cairo_pdf(symbolfamily="OpenSymbol")
>> >>
>> >> ... to specify that the OpenSymbol family should be used as
the
>> >> "symbol" font (e.g., for "plotmath") in R.
>> >>
>> >> This is just a separate branch for now because, while I have
tested it
>> >> under Unbuntu 18.04 and Fedora 31, I cannot even build R for
Windows
>> >> (right now) or Mac (ever) and I do not want to drop a bomb on
R-devel
>> >> at this stage of the release process for R 4.0.0.
>> >>
>> >> The attached file contains at least an outline of steps
required to do
>> >> a minimal test if anyone wants to try the fix on Linux.
>> >>
>> >> cc'ing Simon and Jeroen in case they are able to help with
checking
>> >> that this builds and works on Mac and/or Windows.
>> >>
>> >> NOTEs:
>> >> - 'symbolfamily' can only be specified when a graphics
device is
>> >> opened, and it is then fixed for that device.
>> >> - on Windows, for cairo-based devices, the "symbol"
font is still
>> >> hard-coded as "Standard Symbols L"
>> >>
>> >> Paul
>> >>
>> >> On 30/03/20 8:15 am, Paul Murrell wrote:
>> >>> Hi
>> >>>
>> >>> Thanks for your input on this I?aki and Nicolas.
>> >>>
>> >>> I am starting testing an R fix for this problem today.
>> >>>
>> >>> As suggested, the plan is to allow the R user to specify a
font
>> >>> family other than "symbol" for plotmath output
(or, more generally,
>> >>> in R parlance, for 'font=5' or
'fontface=5') on a Cairo-based
>> >>> graphics device.
>> >>>
>> >>> Paul
>> >>>
>> >>>
>> >>> On 27/03/20 11:30 pm, I?aki Ucar wrote:
>> >>>> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
>> >>>> <nicolas.mailhot at laposte.net> wrote:
>> >>>>>
>> >>>>> <snip>
>> >>>>>
>> >>>>> R brought this all on itself by hardcoding a
Windows-only ?Symbol?
>> >>>>> font
>> >>>>> family name in its default conf. Linux systems are
UTF-8 by default
>> >>>>> for
>> >>>>> ~20 years now, they don?t need the forcing of
magic font families to
>> >>>>> handle symbols not present in the 8-bit legacy
Windows encodings.
>> >>>>>
>> >>>>> The actual effect of this conf is not the
selection of font files
>> with
>> >>>>> special and unusual symbols. It is to priorize
fonts that match the
>> >>>>> "Symbol" magic name. And those fonts are
few and crumbling on Linux
>> >>>>> systems, because no one has needed to bother with
them since Linux
>> >>>>> switched to UTF-8 last millenium.
>> >>>>>
>> >>>>> Just stop using ?Symbol? in R and things will work
a lot better.
>> >>>>> Alternatively, prepare to maintain the ?Symbol?
aliasing stack in
>> >>>>> fontconfig (and fight with wine for it), because
*no* *one* *else*
>> >>>>> *cares* about this legacy Windows-specific stuff.
>> >>>>
>> >>>> So, in the light of Nicolas' input (thanks!), I
think that font
>> >>>> selection should be fixed upstream in R. I'd be
happy to put all this
>> >>>> together in R's bugzilla, but I don't have an
account. Could someone
>> >>>> please invite me?
>> >>>>
>> >>>> I?aki
>> >>>>
>> >>>> ______________________________________________
>> >>>> R-devel at r-project.org mailing list
>> >>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>> >>>>
>> >>
>> >
>>
>> --
>> Dr Paul Murrell
>> Department of Statistics
>> The University of Auckland
>> Private Bag 92019
>> Auckland
>> New Zealand
>> 64 9 3737599 x85392
>> paul at stat.auckland.ac.nz
>> http://www.stat.auckland.ac.nz/~paul/
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot001apsymNoPUA.pdf
Type: application/pdf
Size: 29162 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot001raw.pdf
Type: application/pdf
Size: 44262 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0001.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot001helvNoPUA.pdf
Type: application/pdf
Size: 36124 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0002.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot001apsym.pdf
Type: application/pdf
Size: 27796 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0003.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot001helv.pdf
Type: application/pdf
Size: 35341 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0004.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot002apsymNoPUA.pdf
Type: application/pdf
Size: 12278 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0005.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot002apsym.pdf
Type: application/pdf
Size: 12278 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0006.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot002helvNoPUA.pdf
Type: application/pdf
Size: 14845 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0007.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot002helv.pdf
Type: application/pdf
Size: 14845 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0008.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot002raw.pdf
Type: application/pdf
Size: 14349 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0009.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot003apsym.pdf
Type: application/pdf
Size: 22879 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0010.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot003apsymNoPUA.pdf
Type: application/pdf
Size: 22879 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0011.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot003helv.pdf
Type: application/pdf
Size: 27939 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0012.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot003helvNoPUA.pdf
Type: application/pdf
Size: 27939 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0013.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot003raw.pdf
Type: application/pdf
Size: 26890 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0014.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot004apsym.pdf
Type: application/pdf
Size: 16220 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0015.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot004apsymNoPUA.pdf
Type: application/pdf
Size: 16220 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0016.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot004helv.pdf
Type: application/pdf
Size: 21668 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0017.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot004raw.pdf
Type: application/pdf
Size: 20411 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0018.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot004helvNoPUA.pdf
Type: application/pdf
Size: 21668 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0019.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot005apsym.pdf
Type: application/pdf
Size: 13454 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0020.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot005apsymNoPUA.pdf
Type: application/pdf
Size: 13454 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0021.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot005helv.pdf
Type: application/pdf
Size: 14750 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0022.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot005helvNoPUA.pdf
Type: application/pdf
Size: 14750 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0023.pdf>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot005raw.pdf
Type: application/pdf
Size: 15651 bytes
Desc: not available
URL:
<https://stat.ethz.ch/pipermail/r-devel/attachments/20200408/2ddbb91f/attachment-0024.pdf>

Thanks Gabriel.  Simon and Brian are also helping out.  I will start a 
speparate thread for the four of us to coordinate.

Paul

On 8/04/20 3:59 pm, Gabriel Becker wrote:
> Paul et al,
> 
> I will try to do this tonight or tomorrow, though it will not be built 
> with th system tools because I have yet to get that tto work locally 
> (spent a good chunk of this morning trying).
> 
> I will send a separate messaage?regarding those difficulties as well so 
> that we can at least confirm that they are due to a malconfiguration on 
> my part.
> 
> Best,
> ~G
> 
> On Tue, Apr 7, 2020 at 7:25 PM Paul Murrell <paul at stat.auckland.ac.nz
> <mailto:paul at stat.auckland.ac.nz>> wrote:
> 
> 
>     The R-symfam branch (r78176) is now working, for my basic tests, on ...
> 
>     Ubuntu (pango < 1.44)
>     Ubuntu (no pango)
>     Fedora (pango > 1.44)
>     Windows
> 
>     I need help to confirm that this builds on macOS and that the basic
>     tests work ...
> 
>     https://github.com/pmur002/R-symfam-testing
> 
>     Brian has been helping with the build, but I am still looking for
>     someone who can run the tests please.? Happy to be fed PDF files to
>     scrutinize myself;? it's generating the PDF files on macOS that I
need
>     help with.
> 
>     Paul
> 
>     On 6/04/20 2:59 pm, Paul Murrell wrote:
>      > Hi
>      >
>      > The R branch ...
>      >
>      > https://svn.r-project.org/R/branches/R-symfam/
>      >
>      > ... is now set up so that it works "out of the box" on
Fedora by
>     setting
>      > the default to be 'symbolfamily=cairoSymbolFont(family,
>     usePUA=FALSE)'
>      > when grSoftVersion()["pango"] is greater than
"1.44".
>      >
>      > This means that on Fedora 31 (at least on the Docker container I
am
>      > testing on) "sans"->"NimubusSans" is used
as the symbol font by
>     default
>      > and R converts Adobe Symbol Encoding code points to non-PUA UTF8
>     code
>      > points.? This is not the prettiest result, but it is a lot better
>     than
>      > the page full of missing glyphs that we had.
>      >
>      > The default on less "bleeding edge" systems, e.g., my
Ubuntu 18.04,
>      > remains 'symbolfamily="Symbol"'.
>      >
>      > The default on other platforms is supposed to be the same as it
>     was, but
>      > I need help to confirm that.? I have set up a github repo ...
>      >
>      > https://github.com/pmur002/R-symfam-testing
>      >
>      > ... that describes how to test this on macOS and Windows if
>     anyone has
>      > time to do so.
>      >
>      > I will start trying to set up a Windows test unless someone beats
>     me to it.
>      >
>      > Paul
>      >
>      > On 30/03/20 3:24 pm, Paul Murrell wrote:
>      >> Hi
>      >>
>      >> I have created an R branch that contains a potential fix ...
>      >>
>      >> https://svn.r-project.org/R/branches/R-symfam/
>      >>
>      >> This allows, for example, ...
>      >>
>      >> cairo_pdf(symbolfamily="OpenSymbol")
>      >>
>      >> ... to specify that the OpenSymbol family should be used as
the
>      >> "symbol" font (e.g., for "plotmath") in
R.
>      >>
>      >> This is just a separate branch for now because, while I have
>     tested it
>      >> under Unbuntu 18.04 and Fedora 31, I cannot even build R for
>     Windows
>      >> (right now) or Mac (ever) and I do not want to drop a bomb on
>     R-devel
>      >> at this stage of the release process for R 4.0.0.
>      >>
>      >> The attached file contains at least an outline of steps
required
>     to do
>      >> a minimal test if anyone wants to try the fix on Linux.
>      >>
>      >> cc'ing Simon and Jeroen in case they are able to help
with checking
>      >> that this builds and works on Mac and/or Windows.
>      >>
>      >> NOTEs:
>      >> - 'symbolfamily' can only be specified when a
graphics device is
>      >> opened, and it is then fixed for that device.
>      >> - on Windows, for cairo-based devices, the "symbol"
font is still
>      >> hard-coded as "Standard Symbols L"
>      >>
>      >> Paul
>      >>
>      >> On 30/03/20 8:15 am, Paul Murrell wrote:
>      >>> Hi
>      >>>
>      >>> Thanks for your input on this I?aki and Nicolas.
>      >>>
>      >>> I am starting testing an R fix for this problem today.
>      >>>
>      >>> As suggested, the plan is to allow the R user to specify
a font
>      >>> family other than "symbol" for plotmath output
(or, more
>     generally,
>      >>> in R parlance, for 'font=5' or
'fontface=5') on a Cairo-based
>      >>> graphics device.
>      >>>
>      >>> Paul
>      >>>
>      >>>
>      >>> On 27/03/20 11:30 pm, I?aki Ucar wrote:
>      >>>> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
>      >>>> <nicolas.mailhot at laposte.net
>     <mailto:nicolas.mailhot at laposte.net>> wrote:
>      >>>>>
>      >>>>> <snip>
>      >>>>>
>      >>>>> R brought this all on itself by hardcoding a
Windows-only
>     ?Symbol?
>      >>>>> font
>      >>>>> family name in its default conf. Linux systems
are UTF-8 by
>     default
>      >>>>> for
>      >>>>> ~20 years now, they don?t need the forcing of
magic font
>     families to
>      >>>>> handle symbols not present in the 8-bit legacy
Windows encodings.
>      >>>>>
>      >>>>> The actual effect of this conf is not the
selection of font
>     files with
>      >>>>> special and unusual symbols. It is to priorize
fonts that
>     match the
>      >>>>> "Symbol" magic name. And those fonts
are few and crumbling on
>     Linux
>      >>>>> systems, because no one has needed to bother with
them since
>     Linux
>      >>>>> switched to UTF-8 last millenium.
>      >>>>>
>      >>>>> Just stop using ?Symbol? in R and things will
work a lot better.
>      >>>>> Alternatively, prepare to maintain the ?Symbol?
aliasing stack in
>      >>>>> fontconfig (and fight with wine for it), because
*no* *one*
>     *else*
>      >>>>> *cares* about this legacy Windows-specific stuff.
>      >>>>
>      >>>> So, in the light of Nicolas' input (thanks!), I
think that font
>      >>>> selection should be fixed upstream in R. I'd be
happy to put
>     all this
>      >>>> together in R's bugzilla, but I don't have an
account. Could
>     someone
>      >>>> please invite me?
>      >>>>
>      >>>> I?aki
>      >>>>
>      >>>> ______________________________________________
>      >>>> R-devel at r-project.org <mailto:R-devel at
r-project.org> mailing list
>      >>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>      >>>>
>      >>
>      >
> 
>     -- 
>     Dr Paul Murrell
>     Department of Statistics
>     The University of Auckland
>     Private Bag 92019
>     Auckland
>     New Zealand
>     64 9 3737599 x85392
>     paul at stat.auckland.ac.nz <mailto:paul at stat.auckland.ac.nz>
>     http://www.stat.auckland.ac.nz/~paul/
> 
>     ______________________________________________
>     R-devel at r-project.org <mailto:R-devel at r-project.org>
mailing list
>     https://stat.ethz.ch/mailman/listinfo/r-devel
> 
-- 
Dr Paul Murrell
Department of Statistics
The University of Auckland
Private Bag 92019
Auckland
New Zealand
64 9 3737599 x85392
paul at stat.auckland.ac.nz
http://www.stat.auckland.ac.nz/~paul/

Thanks for this Gabriel - extremely useful.
Those results look pretty good.

Paul

On 8/04/20 9:55 pm, Gabriel Becker wrote:
> Hi Paul,
> 
> So I've run the tests?(and am? attaching all numerous pdfs here) but
the
> takeaway from what I can see is as follows:
> 
> raw, without specifying font family, "look good" (see
plot*raw.pdf)
> from your branch (my eye is upset by the positioning of the phi symbol 
> in plot2, but I've confirmed that it looks the same generated from 
> 3.5.1, so that?isn't?your branch).
> 
> The various font family settings seem to work too, from what I can tell. 
> Both font families?you suggested, however, Helvetica and Apple Symbols 
> (the s is important) have significantly incomplete coverage with PUA on. 
> Apple Symbols does have nearly complete coverage (though? to my eye the 
> symbols are all smaller...) with PUA turned off, but Helvetica remains 
> very spotty, with disabling PUA only modestly increasing it's coverage,
> and not in the places that seem likely to matter.
> 
> I hope that helps,
> ~G
> 
> On Tue, Apr 7, 2020 at 8:59 PM Gabriel Becker <gabembecker at gmail.com 
> <mailto:gabembecker at gmail.com>> wrote:
> 
>     Paul et al,
> 
>     I will try to do this tonight or tomorrow, though it will not be
>     built with th system tools because I have yet to get that tto work
>     locally (spent a good chunk of this morning trying).
> 
>     I will send a separate messaage?regarding those difficulties as well
>     so that we can at least confirm that they are due to a
>     malconfiguration on my part.
> 
>     Best,
>     ~G
> 
>     On Tue, Apr 7, 2020 at 7:25 PM Paul Murrell
>     <paul at stat.auckland.ac.nz <mailto:paul at
stat.auckland.ac.nz>> wrote:
> 
> 
>         The R-symfam branch (r78176) is now working, for my basic tests,
>         on ...
> 
>         Ubuntu (pango < 1.44)
>         Ubuntu (no pango)
>         Fedora (pango > 1.44)
>         Windows
> 
>         I need help to confirm that this builds on macOS and that the basic
>         tests work ...
> 
>         https://github.com/pmur002/R-symfam-testing
> 
>         Brian has been helping with the build, but I am still looking for
>         someone who can run the tests please.? Happy to be fed PDF files to
>         scrutinize myself;? it's generating the PDF files on macOS that
>         I need
>         help with.
> 
>         Paul
> 
>         On 6/04/20 2:59 pm, Paul Murrell wrote:
>          > Hi
>          >
>          > The R branch ...
>          >
>          > https://svn.r-project.org/R/branches/R-symfam/
>          >
>          > ... is now set up so that it works "out of the box"
on Fedora
>         by setting
>          > the default to be 'symbolfamily=cairoSymbolFont(family,
>         usePUA=FALSE)'
>          > when grSoftVersion()["pango"] is greater than
"1.44".
>          >
>          > This means that on Fedora 31 (at least on the Docker
>         container I am
>          > testing on) "sans"->"NimubusSans" is
used as the symbol font
>         by default
>          > and R converts Adobe Symbol Encoding code points to non-PUA
>         UTF8 code
>          > points.? This is not the prettiest result, but it is a lot
>         better than
>          > the page full of missing glyphs that we had.
>          >
>          > The default on less "bleeding edge" systems, e.g.,
my Ubuntu
>         18.04,
>          > remains 'symbolfamily="Symbol"'.
>          >
>          > The default on other platforms is supposed to be the same as
>         it was, but
>          > I need help to confirm that.? I have set up a github repo ...
>          >
>          > https://github.com/pmur002/R-symfam-testing
>          >
>          > ... that describes how to test this on macOS and Windows if
>         anyone has
>          > time to do so.
>          >
>          > I will start trying to set up a Windows test unless someone
>         beats me to it.
>          >
>          > Paul
>          >
>          > On 30/03/20 3:24 pm, Paul Murrell wrote:
>          >> Hi
>          >>
>          >> I have created an R branch that contains a potential fix
...
>          >>
>          >> https://svn.r-project.org/R/branches/R-symfam/
>          >>
>          >> This allows, for example, ...
>          >>
>          >> cairo_pdf(symbolfamily="OpenSymbol")
>          >>
>          >> ... to specify that the OpenSymbol family should be used
as the
>          >> "symbol" font (e.g., for "plotmath")
in R.
>          >>
>          >> This is just a separate branch for now because, while I
have
>         tested it
>          >> under Unbuntu 18.04 and Fedora 31, I cannot even build R
for
>         Windows
>          >> (right now) or Mac (ever) and I do not want to drop a
bomb
>         on R-devel
>          >> at this stage of the release process for R 4.0.0.
>          >>
>          >> The attached file contains at least an outline of steps
>         required to do
>          >> a minimal test if anyone wants to try the fix on Linux.
>          >>
>          >> cc'ing Simon and Jeroen in case they are able to help
with
>         checking
>          >> that this builds and works on Mac and/or Windows.
>          >>
>          >> NOTEs:
>          >> - 'symbolfamily' can only be specified when a
graphics
>         device is
>          >> opened, and it is then fixed for that device.
>          >> - on Windows, for cairo-based devices, the
"symbol" font is
>         still
>          >> hard-coded as "Standard Symbols L"
>          >>
>          >> Paul
>          >>
>          >> On 30/03/20 8:15 am, Paul Murrell wrote:
>          >>> Hi
>          >>>
>          >>> Thanks for your input on this I?aki and Nicolas.
>          >>>
>          >>> I am starting testing an R fix for this problem
today.
>          >>>
>          >>> As suggested, the plan is to allow the R user to
specify a
>         font
>          >>> family other than "symbol" for plotmath
output (or, more
>         generally,
>          >>> in R parlance, for 'font=5' or
'fontface=5') on a Cairo-based
>          >>> graphics device.
>          >>>
>          >>> Paul
>          >>>
>          >>>
>          >>> On 27/03/20 11:30 pm, I?aki Ucar wrote:
>          >>>> On Wed, 25 Mar 2020 at 12:25, Nicolas Mailhot
>          >>>> <nicolas.mailhot at laposte.net
>         <mailto:nicolas.mailhot at laposte.net>> wrote:
>          >>>>>
>          >>>>> <snip>
>          >>>>>
>          >>>>> R brought this all on itself by hardcoding a
Windows-only
>         ?Symbol?
>          >>>>> font
>          >>>>> family name in its default conf. Linux
systems are UTF-8
>         by default
>          >>>>> for
>          >>>>> ~20 years now, they don?t need the forcing of
magic font
>         families to
>          >>>>> handle symbols not present in the 8-bit
legacy Windows
>         encodings.
>          >>>>>
>          >>>>> The actual effect of this conf is not the
selection of
>         font files with
>          >>>>> special and unusual symbols. It is to
priorize fonts that
>         match the
>          >>>>> "Symbol" magic name. And those
fonts are few and
>         crumbling on Linux
>          >>>>> systems, because no one has needed to bother
with them
>         since Linux
>          >>>>> switched to UTF-8 last millenium.
>          >>>>>
>          >>>>> Just stop using ?Symbol? in R and things will
work a lot
>         better.
>          >>>>> Alternatively, prepare to maintain the
?Symbol? aliasing
>         stack in
>          >>>>> fontconfig (and fight with wine for it),
because *no*
>         *one* *else*
>          >>>>> *cares* about this legacy Windows-specific
stuff.
>          >>>>
>          >>>> So, in the light of Nicolas' input (thanks!),
I think that
>         font
>          >>>> selection should be fixed upstream in R. I'd
be happy to
>         put all this
>          >>>> together in R's bugzilla, but I don't
have an account.
>         Could someone
>          >>>> please invite me?
>          >>>>
>          >>>> I?aki
>          >>>>
>          >>>> ______________________________________________
>          >>>> R-devel at r-project.org <mailto:R-devel at
r-project.org>
>         mailing list
>          >>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>          >>>>
>          >>
>          >
> 
>         -- 
>         Dr Paul Murrell
>         Department of Statistics
>         The University of Auckland
>         Private Bag 92019
>         Auckland
>         New Zealand
>         64 9 3737599 x85392
>         paul at stat.auckland.ac.nz <mailto:paul at
stat.auckland.ac.nz>
>         http://www.stat.auckland.ac.nz/~paul/
> 
>         ______________________________________________
>         R-devel at r-project.org <mailto:R-devel at r-project.org>
mailing list
>         https://stat.ethz.ch/mailman/listinfo/r-devel
> 
-- 
Dr Paul Murrell
Department of Statistics
The University of Auckland
Private Bag 92019
Auckland
New Zealand
64 9 3737599 x85392
paul at stat.auckland.ac.nz
http://www.stat.auckland.ac.nz/~paul/

Le mercredi 08 avril 2020 ? 02:55 -0700, Gabriel Becker a ?crit
:
> Hi Paul,
Hi Gabriel,

Thanks a lot for the testing.
> The various font family settings seem to work too, from what I can
> tell. Both font families you suggested, however, Helvetica and Apple
> Symbols (the s is important) have significantly incomplete coverage
> with PUA on.
That is to be expected, the AMS symbol dump in PUA space was a quick
hack to make pre-unicode symbols available in an unicode world, pending
their normalisation.

That standardisation is long past (IIRC it occured by unicode 3.2
released in March 2002), so no newly created/updated font family is
going to place those symbols in PUA anymore.

Now adding the AMS symbols to new fonts has been slow, due to the large
amount of software hardcoding Symbol (and equivallent) and masking the
actual glyph userbase to font makers. It will accelerate with more apps
expecting plain unicode by default.

Thanks for the testing!

Regards,

-- 
Nicolas Mailhot