Fontconfig - Nov 2009 - Marking glyphs as deliberately blank, per font

I have been attempting to create some fonts that reproduce the look of old
text terminals. One of the features of terminals like the VT100 was
double-height, double-width mode on certain lines. In order to do this, I
have created two fonts, one for the upper half of characters, and one for
the bottom half. However, for characters where all the marks would appear
in just one half of the glyph, say quote marks or the underscore, I am
left with blank glyphs in places where fontconfig doesn''t expect them,
and
those glyphs are marked as broken by fontconfig, and substituted.

On 11/25/2009 07:01 AM, Paul Flo Williams wrote:
> I have been attempting to create some fonts that reproduce the look of old
> text terminals. One of the features of terminals like the VT100 was
> double-height, double-width mode on certain lines. In order to do this, I
> have created two fonts, one for the upper half of characters, and one for
> the bottom half. However, for characters where all the marks would appear
> in just one half of the glyph, say quote marks or the underscore, I am
> left with blank glyphs in places where fontconfig doesn''t expect
them, and
> those glyphs are marked as broken by fontconfig, and substituted.
>
>> From my reading of the fonts.conf page, I think that the<blank> 
element
> is a global configuration, and I don''t believe I can say for a
given font,
> "these glyphs are intentionally blank", which leaves me with
these two
> questions:
>
> 1. Is my understanding correct, that I can''t override<blank>
per font?
I believe so.
> 2. The fonts.conf page says that ''fonts often include
"broken" glyphs which
>     appear in the encoding but are drawn as blanks on the screen.''
A trawl of
>     the mailing list suggests that this configuration option is ancient --
at
>     least I can''t find any discussion of its introduction -- so is
this form
>     of breakage common in fonts we use today?
Donno.

behdad

> I am aware that I''m attempting to do something rather odd here, so
I think
> I''ll have to do my own substituting with spaces when producing
"screen
> shots" of these old terminal displays, but I wanted to check my
> understanding of fontconfig''s blank mechanism.

Le Mer 25 novembre 2009 23:54, Behdad Esfahbod a ?crit :
>
> On 11/25/2009 07:01 AM, Paul Flo Williams wrote:
>> 2. The fonts.conf page says that ''fonts often include
"broken" glyphs which
>>     appear in the encoding but are drawn as blanks on the
screen.'' A trawl
>> of
>>     the mailing list suggests that this configuration option is ancient
--
>> at
>>     least I can''t find any discussion of its introduction --
so is this form
>>     of breakage common in fonts we use today?
>
> Donno.
I don''t know if it''s common, but it definitely exists today.
We''ve seen it
just a few months ago in Fedora (in smc fonts IIRC). In their case it was not
even a blank but a buggy glyph meant to indicate ''reserved''

IMHO now there are efforts in big apps such as Firefox to do more CSS3,
including the CSS3 "select unicode subrange in font" operator, it is
becoming
even more urgent for fontconfig to stop considering a font is a basic atomic
unit, and allow fontconfig users to write rules that only apply to part of a
font. It is a huge problem if the font selection model used by fontconfig does
not permit direct mapping from the font selection model used by web-oriented
apps (which can be basically any app now more and more GUI toolkits move to
use of CSS to specify styling)

http://people.mozilla.org/~jdaggett/AdvancingWebTypography.pdf page 42

The "fonts should only include one script with consistent quality so
fontconfig does not need to split them" is a lost cause now, the font
community chose, and not the way fontconfig authors wished.

All the conflicts between scripts and weird inverted logic overrides almost no
one really understands are caused by the "font is an atomic fontconfig
unit"
postulate. All past attempts to mitigate those problems without changing the
"font is an atomic fontconfig unit" postulate have failed miserably.
Humans
just do not think fonts are an atomic unit, and can not wrap their minds
around a system where they are forced to think this way.

-- 
Nicolas Mailhot

Nicolas, all that talk makes sense, but unless a concrete proposal for how to 
change fontconfig comes in, I don''t know what to do about it.

behdad

On 11/26/2009 04:35 AM, Nicolas Mailhot wrote:
>
>
> Le Mer 25 novembre 2009 23:54, Behdad Esfahbod a ?crit :
>>
>> On 11/25/2009 07:01 AM, Paul Flo Williams wrote:
>
>>> 2. The fonts.conf page says that ''fonts often include
"broken" glyphs which
>>>      appear in the encoding but are drawn as blanks on the
screen.'' A trawl
>>> of
>>>      the mailing list suggests that this configuration option is
ancient --
>>> at
>>>      least I can''t find any discussion of its introduction
-- so is this form
>>>      of breakage common in fonts we use today?
>>
>> Donno.
>
> I don''t know if it''s common, but it definitely exists
today. We''ve seen it
> just a few months ago in Fedora (in smc fonts IIRC). In their case it was
not
> even a blank but a buggy glyph meant to indicate
''reserved''
>
> IMHO now there are efforts in big apps such as Firefox to do more CSS3,
> including the CSS3 "select unicode subrange in font" operator, it
is becoming
> even more urgent for fontconfig to stop considering a font is a basic
atomic
> unit, and allow fontconfig users to write rules that only apply to part of
a
> font. It is a huge problem if the font selection model used by fontconfig
does
> not permit direct mapping from the font selection model used by
web-oriented
> apps (which can be basically any app now more and more GUI toolkits move to
> use of CSS to specify styling)
>
> http://people.mozilla.org/~jdaggett/AdvancingWebTypography.pdf page 42
>
> The "fonts should only include one script with consistent quality so
> fontconfig does not need to split them" is a lost cause now, the font
> community chose, and not the way fontconfig authors wished.
>
> All the conflicts between scripts and weird inverted logic overrides almost
no
> one really understands are caused by the "font is an atomic fontconfig
unit"
> postulate. All past attempts to mitigate those problems without changing
the
> "font is an atomic fontconfig unit" postulate have failed
miserably. Humans
> just do not think fonts are an atomic unit, and can not wrap their minds
> around a system where they are forced to think this way.
>

Le jeudi 26 novembre 2009 ? 13:14 -0500, Behdad Esfahbod a ?crit
:
> Nicolas, all that talk makes sense, but unless a concrete proposal for
> how to 
> change fontconfig comes in, I don''t know what to do about it.
To sum up, and from a pure fontconfig user POW (I don''t want to know
how
the code looks like), I think you could solve many user pains by making
it possible to use

<font>
 <family>some family name</family>
 <style>some style name</style>
 <format>some font format name</format>
 <unicode>
   <min>some-unicode-value</min>
   <max>some-greater-unicode-value</max>
 <unicode>
 <lang>some fontconfig orth id</lang>
</font>

instead of plain
<family>some family name</family>

in all the places <family/> is exposed in fontconfig files today

In other words, permit users to write rules that apply to just a
specific sub-element of a font family

In this notation :

1. family is an optional element. It can occur several times (if you
want a rule to apply to multiple font families). When family is absent
the rule applies to all font families

2. style is an optional element. It can occur several times (if you want
a rule to apply to multiple styles). When style is absent the rule
applies to all styles

3. format is an optional element. It can occur several times (if you
want a rule to apply to multiple font formats). When format is absent
the rule applies to all formats

4. unicode  is an optional element. It can occur several times (if you
want a rule to apply to multiple unicode blocks). When unicode is absent
the rule applies to the complete encoding range

5. lang  is an optional element. It can occur several times (if you want
a rule to apply to multiple lang blocks). When lang is absent the rule
applies to all langs encoding range. Lang means whatever glyphs are
associated to a particular lang in an .orth file.

And it would also be useful to define a <not/> operator, for example
<not>
 <lang>ja_Ja</lang>
</not>

apply the rule to all the elements defined before, except for the glyphs
associated with ja_Ja

Of course this is only my personal opinion, if anyone on the list
strongly disagrees with this design please speak up now

Regards,

-- 
Nicolas Mailhot

Le jeudi 26 novembre 2009 ? 19:44 +0100, Nicolas Mailhot a ?crit :
> To sum up, and from a pure fontconfig user POW (I don''t want to
know
> how the code looks like), I think you could solve many user pains by
> making it possible to use
> 
> <font>
>  <family>some family name</family>
>  <style>some style name</style>
>  <format>some font format name</format>
>  <unicode>
>    <min>some-unicode-value</min>
>    <max>some-greater-unicode-value</max>
>  <unicode>
>  <lang>some fontconfig orth id</lang>
> </font>
And it would probably useful to add a <file/> selector that permits to
specify a specific file on the filesystem. Unfortunately sometimes you
get font files that claim the wrong font family, and it impossible right
now to write fontconfig rules that apply to those particular files
without having side-effects on other files that rightly claim this font
family.

-- 
Nicolas Mailhot

Le jeudi 26 novembre 2009 ? 19:44 +0100, Nicolas Mailhot a ?crit :
> I think you could solve many user pains by
> making it possible to use
> 
> <font>
>  <family>some family name</family>
>  <style>some style name</style>
>  <format>some font format name</format>
>  <unicode>
>    <min>some-unicode-value</min>
>    <max>some-greater-unicode-value</max>
>  <unicode>
>  <lang>some fontconfig orth id</lang>
> </font>
?
> And it would also be useful to define a <not/> operator, for example
> <not>
>  <lang>ja_Ja</lang>
> </not>
And just to be 100% clear, in my mind

1. sub-elements of fonts are implicitely OR-ed if they are of the same
kind: 
<font>
<family>foo</family>
<family>bar</family>
<font>

applies to all fonts with a family name of foo or bar

2. sub-elements of fonts are implicitely AND-ed if they are of different
kinds

<font>
<family>foo</family>
<style>bar</style>
<font>

applies to fonts only if they have the foo family and the bar style

3. lang and unicode are of the same kind: they are two different ways to
specify an unicode range. Maybe it''s better to make it explicit and
define a

<glyph type="unicode-range"/> and <glyph
type="lang"/> selector?

-- 
Nicolas Mailhot

Nicolas Mailhot wrote:
 
> Le jeudi 26 novembre 2009 ? 13:14 -0500, Behdad Esfahbod a ?crit :
>> Nicolas, all that talk makes sense, but unless a concrete proposal
>> for how to change fontconfig comes in, I don''t know what to do
>> about it.
> 
> To sum up, and from a pure fontconfig user POW (I don''t want to
know
> how the code looks like), I think you could solve many user pains by
> making it possible to use
It doesn''t solve any of my pains :)
 
> <font>
>  <family>some family name</family>
>  <style>some style name</style>
>  <format>some font format name</format>
>  <unicode>
>    <min>some-unicode-value</min>
>    <max>some-greater-unicode-value</max>
>  <unicode>
>  <lang>some fontconfig orth id</lang>
> </font>
It looks like a reinvention of what is already there.

<match target="font">
  <test
name="family"><string>...</string></test>
  <test
name="style"><string>...</string></test>
  <test name="whatever">...</test>
  <edit name="whatever">...</edit>
</match>
> And it would also be useful to define a <not/> operator, for example
> <not>
>  <lang>ja_Ja</lang>
> </not>
<test compare="not_eq">
?

-- 
The fact is, most software is crap, and most software developers
are lazy and stupid. Same as most customers are stupid too.
                         -- Hua Zhong, Linux Kernel Mailing List

Le jeudi 26 novembre 2009 ? 22:06 +0100, Krzysztof Kotlenga a ?crit
:
> Nicolas Mailhot wrote:
>  
> > Le jeudi 26 novembre 2009 ? 13:14 -0500, Behdad Esfahbod a ?crit :
> >> Nicolas, all that talk makes sense, but unless a concrete proposal
> >> for how to change fontconfig comes in, I don''t know what
to do
> >> about it.
> > 
> > To sum up, and from a pure fontconfig user POW (I don''t want
to know
> > how the code looks like), I think you could solve many user pains by
> > making it possible to use
> 
> It doesn''t solve any of my pains :)
It solves the "part of the font file has bogus blank glyphs"
>  
> > <font>
> >  <family>some family name</family>
> >  <style>some style name</style>
> >  <format>some font format name</format>
> >  <unicode>
> >    <min>some-unicode-value</min>
> >    <max>some-greater-unicode-value</max>
> >  <unicode>
> >  <lang>some fontconfig orth id</lang>
> > </font>
> 
> It looks like a reinvention of what is already there.
> 
> <match target="font">
>   <test
name="family"><string>...</string></test>
>   <test
name="style"><string>...</string></test>
>   <test name="whatever">...</test>
>   <edit name="whatever">...</edit>
> </match>
What is already there is so verbose very few people use it, and when
they do use it they usually mix up things and it does not work the way
they expect it to work due to all the layers of verbosity that get in
the way of human parsing

I''ve never seen problems with simple succinct statements like :
        <alias>
                <family>serif</family>
                <prefer>
                        <family>Bitstream Vera Serif</family>
                        <family>DejaVu Serif</family>
                        <family>Times New Roman</family>
                        <family>Thorndale AMT</family>
                        <family>Luxi Serif</family>
                        <family>Nimbus Roman No9 L</family>
                        <family>Times</family>
                </prefer>
        </alias>

OTOH the match target/test is a reinvention of emacs'' let''s
use lisp
syntax as configuration syntax hell
> > And it would also be useful to define a <not/> operator, for
example
> > <not>
> >  <lang>ja_Ja</lang>
> > </not>
> 
> <test compare="not_eq">
> ?
Another oververbose statement


-- 
Nicolas Mailhot

Le jeudi 26 novembre 2009 ? 23:07 +0100, Nicolas Mailhot a ?crit
:
> Le jeudi 26 novembre 2009 ? 22:06 +0100, Krzysztof Kotlenga a ?crit
> :
> > Nicolas Mailhot wrote:
> > > And it would also be useful to define a <not/> operator,
for
> example
> > > <not>
> > >  <lang>ja_Ja</lang>
> > > </not>
> > 
> > <test compare="not_eq">
> > ?
> 
> Another oververbose statement
Also one needs to distinguish between lang, as the lang resolution is
running for and lang, as the glyphs in a font file that correspond to a
particular .orth lang. Current fontconfig is geared for the first case.

But usually people who package fonts are interested by the second case
too (do something with Greek glyphs in font foo, not do something with
font foo when resolving for Greek). Of course both are often combined,
but just the first part is insufficient ("when resolving for Greek, use
Greek glyphs from font foo, then Greek Glyphs from font bar", is not the
same as "when resolving for Greek, use font foo, then font bar".

The second statement leads to Chinese users trying to override their own
overrides since a simple "when resolving for Chinese use font foo" has
the side effect of making base latin and common glyphs in font foo
override the default latin font if the lang is set to Chinese. So they
use weird constructs like "when the lang is Chinese, forget about
default font settings and put my chinese font first". Except oops,
I''ve
just broken latin, so do also "when using my chinese font put this latin
font first". Therefore they need to know the right latin font fontconfig
would have chosen without their Chinese override (they do not)

-- 
Nicolas Mailhot

Le jeudi 26 novembre 2009 ? 23:07 +0100, Nicolas Mailhot a ?crit
:
> Le jeudi 26 novembre 2009 ? 22:06 +0100, Krzysztof Kotlenga a ?crit
>  
> > It doesn''t solve any of my pains :)
> 
> It solves the "part of the font file has bogus blank glyphs"
To expand yet more (thinking loud), the core fontconfig syntax no one
ever had any problem with is 

<alias>
  <family>serif</family>
  <prefer>
    <family>Bitstream Vera Serif</family>
    <family>DejaVu Serif</family>
  </prefer>
</alias>

It is easy to understand because is clearly states
1. the question: what fontconfig is resolving (serif)
2. the priorized list of possible answer
3. that it constructs a virtual alias

in a compact declarative syntax and not an intermingling of XML and
functional code-like statements

Looking back at the most common fontconfig statements we use in Fedora
you find:

A. Declaring which generic a font contributes to:

<alias>
  <family>[genericname]</family>
  <prefer>
    <family>[fontname]</family>
  </prefer>
</alias>

(basic rule already discussed)


B. Declaring where to take elements to complete a font (usually a
generic)

<alias>
  <family>[fontname]</family>
  <default>
    <family>[genericname]</family>
  </default>
</alias>

Except things already start going downhill, what was prefer before is
default now for no clear reason. Why can''t prefer be used in both
cases?

<alias>
  <family>[fontname]</family>
  <prefer>
    <family>[genericname]</family>
  </prefer>
</alias>

C. Declaring a font can be used as replacement for another font if this
other font is absent

<alias binding="same">
  <family>[otherfontname]</family>
  <accept>
    <family>[fontname]</family>
  </accept>
</alias>

Now it''s not default or prefer but accept. And there is a weird binding
thinguie. To keep syntax consistent, it should have been

<alias>
  <family>[otherfontname]</family>
  <prefer>
    <family>[otherfontname]</family>
    <family>[fontname]</family>
  </prefer>
</alias>

Which is a little more verbose, but uses the same verbs as A.

D. Then you have locale overrides (the most painful part right now)

<match>
  <test name="lang">
    <string>[loc]</string>
  </test>
  <test name="family">
    <string>[genericname]</string>
  </test>
  <edit name="family" mode="prepend"
binding="same">
    <string>[fontname]</string>
  </edit>
</match>

Things continue to devolve in special syntax. And this block BTW is very
confusing to packagers : since it uses prepend, the last such block will
win, when it''s the first one for other rules. Also it will slap a whole
font at the top of the stack when the packager really only wanted to
change glyphs associated with loc (not all glyphs when lang is loc)

To keep syntax consistent with A. and have a reliable rule the packager
understands, it should have been

<alias>
  <font>
    <family>[genericname]</family>
    <lang>[loc]</lang>
  </font>
  <prefer>
    <font>
      <family>[fontname]</family>
      <lang>[loc]</lang>
    </font>
  </prefer>
</alias>

And here we can even forget unicode han unification, we''ve just stated
succintly that when fontconfig needs the loc lang bits in genericname,
it should take the loc lang bits in fontname (and only them). As a bonus
there is no need of evil prepending black magic anymore to fight for
priority with other CJK fonts

E. And font rescaling

<match target="font">
  <test name="family" mode="eq">
    <string>[fontname]</string>
  </test>
  <edit name="matrix" mode="assign">
    <times>
      <name>matrix</name>
      <matrix>
        <double>[factor]</double>
        <double>0</double>
        <double>0</double>
        <double>[factor]</double>
      </matrix>
    </times>
  </edit>
</match>

should have been

<alias>
  <family>[fontname]</family>
  <prefer>
    <font>
      <family>[fontname]</family>
      <matrix>...</matrix>
    </font>
  </prefer>
</alias>

F. Font merging

  <match target="scan">
    <test name="family">
      <string>[fontname]</string>
    </test>
    <edit name="fontversion">
      <int>2</int>
    </edit>
  </match>
  <match target="scan">
    <test name="family">
      <string>[mergedfontname]</string>
    </test>
    <edit name="family">
      <string>[fontname]</string>
    </edit>
    <edit name="fullname">
      <string>[fontname] [fontstyle]</string>
    </edit>
    <edit name="fontversion">
      <int>1</int>
    </edit>
  </match>

should have been

<alias>
  <font>
    <family>[mergedfontname]</family>
    <style>[mergedfontstyle]</style>
  <font>
  <font>
    <fullname>[mergedfullname]</fullname>
  </font>
  <prefer>
    <font>
      <family>[fontname1]</family>
      <style>[fontstyle1]</style>
    <font>
    <font>
      <family>[fontname2]</style>
      <style>[fontstyle2]</style>
    <font>
  </prefer>
</alias>

In this example we alias a font both when it is accessed trough
(mergedfontname and mergedfontstyle) and when it is accessed through
mergedfullname

(I admit that there alias alone is probably insufficient as you want to
tell fontconfig to only display mergedfontname, mergedfontstyle in
lists. But that''s the only elements that can not be declared directly
using the same syntax as other examples)

G. Font remapping

<match target="scan">
  <test name="family">
    <string>[problemfontname]</string>
  </test>
  <test name="style">
    <string>[problemfontstyle]</string>
  </test>
  <edit name="family">
    <string>[correctedfontname]</string>
  </edit>
  <edit name="style">
    <string>[correctedfontstyle]</string>
  </edit>
  <edit name="fullname">
    <string>[correctedfontname] [correctedfontstyle]</string>
  </edit>
</match>

Should have been

<alias>
  <font>
    <family>[correctedfontname]</family>
    <style>[correctedfontstyle]</style>
  <font>
  <font>
    <fullname>[correctedfontname] [correctedfontstyle]</fullname>
  </font>  
  <prefer>
    <font>
      <file>location-of-the-problem-file</file>     
    <font>
  </prefer>
</alias>

(like in the previous example, needs a twist to hide the metadata de
problem file declares)

H. When a font has some rotten parts it could be declared with something
like

<alias>
  <family>[fontname]</family>
  <prefer>
    <font>
      <family>[fontname]</family>
      <!-- list of the good parts in the font, or bad parts not-ed -->
      <lang>[loc1]</lang>
      <lang>[loc2]</lang>
    </font>
    <family>[genericname]</family>
  </prefer>
</alias>

ie when requested fontname, use the loc1 and loc2 parts of fontname
only, and fallback to genericname for the rest

I hope those examples make my thoughts clearer. Basically, there are too
many different verbs and special syntax in current fontconfig files,
when the bulk of what people need to declare is always blocks like

1. list of font pattern to resolve (or fake)
2. list of font patterns to try as answer in priority order
3. what the apps should see (actual metadata of the files used as answer
or something else)

I suppose the current system is friendly to people who know fontconfig
code innards, but others do not want to know so much about them.

I''ve used (and abused) of the A. pattern here, it''s probably
not the
best syntax to use as a general case, and one could probably do
something cleaner like

<!-- A unit block -->
<rule>
  <!-- (optional) list of font blocks we can resolve, if we need a
context (remapping, merging and creating fake generics does not need a
particular one -->
  <match>
    <font>...</font>
  </match>
  <!-- list of font blocks used to compose the result, by preference
order -->
  <prefer>
    <font>...<font>
    <font>...<font>
  </prefer>
  <!-- (optional) what the apps should display: only a font with those
fake characteristics, only the bits we used to compose it, or both. This
part is useful for remapping, merging and creating fake generics -->
  <fake display="fake/constituants/both">
    <font>...</font>
  </fake>
</rule>

And then you define that a <font> element that only includes one child
can be expressed directly as this child. For example

<font>
 <family>foo</family>
</font>

and

<family>foo</family>

Are the same thing, <font> is only a container to specify multiple
concurrent characteristics.

So, I''m not asking for current fontconfig config syntax to be replaced
with this one, but it would be mighty nice if it could evolve towards
something as simple and generic as this example, with less black magic
sprinkled everywhere.

-- 
Nicolas Mailhot

Le vendredi 27 novembre 2009 ? 01:10 +0100, Nicolas Mailhot a ?crit :
>   <!-- (optional) what the apps should display: only a font with those
> fake characteristics, only the bits we used to compose it, or both.
> This
> part is useful for remapping, merging and creating fake generics -->
>   <fake display="fake/constituants/both">
>     <font>...</font>
>   </fake>
The "display" attribute is a bad idea (should have known it''s
the only
attribute in this mail)

1. if you create a fake you always want to display it
2. if you want to hide a font from applications it''s probably better to
declare it explicitely

<!-- hidden font or font sub-part, only accessible by apps if it''s
been
used in a prefer list, never displayed on font lists, never resolved
directly -->
<hide>
  <font>...</font>
</hide>

-- 
Nicolas Mailhot

Nicolas Mailhot wrote:
> Le jeudi 26 novembre 2009 ? 22:06 +0100, Krzysztof Kotlenga a ?crit :
>> Nicolas Mailhot wrote:
>>> Le jeudi 26 novembre 2009 ? 13:14 -0500, Behdad Esfahbod a ?crit :
>>> 
>>> <font>
>>>  <family>some family name</family>
>>>  <style>some style name</style>
>>>  <format>some font format name</format>
>>>  <unicode>
>>>    <min>some-unicode-value</min>
>>>    <max>some-greater-unicode-value</max>
>>>  <unicode>
>>>  <lang>some fontconfig orth id</lang>
>>> </font>
>> 
>> It looks like a reinvention of what is already there.
>> 
>> <match target="font">
>>   <test
name="family"><string>...</string></test>
>>   <test
name="style"><string>...</string></test>
>>   <test name="whatever">...</test>
>>   <edit name="whatever">...</edit>
>> </match>
> 
> What is already there is so verbose very few people use it
                                      ^^^^^^^^
You never know...
> and when they do use it they usually mix up things and it does not
> work the way they expect it to work due to all the layers of
> verbosity that get in the way of human parsing
I agree it''s verbose, but not without a reason. You seem to miss the
fact that fontconfig allows arbitrary properties. This is something
that can''t be done with tags.
> I''ve never seen problems with simple succinct statements like :
>         <alias>
>         [snip]
>         </alias>
Well, to quote fonts-conf(5), "alias elements provide a shorthand
notation (...)". So, you''d like to see introduced another form of
shorthand notation. Another change causing user visible complexity to
grow (!). As a user, I wish it won''t happen.
> OTOH the match target/test is a reinvention of emacs''
let''s use lisp
> syntax as configuration syntax hell
This doesn''t sound like technical merit.
>>> And it would also be useful to define a <not/> operator, for
example
>>> <not>
>>>  <lang>ja_Ja</lang>
>>> </not>
>> 
>> <test compare="not_eq">
>> ?
> 
> Another oververbose statement
You can''t please everyone. :)

PS. Isn''t this horribly OT?
PPS. I''m sorry for sending this twice.

-- 
The fact is, most software is crap, and most software developers
are lazy and stupid. Same as most customers are stupid too.
                         -- Hua Zhong, Linux Kernel Mailing List

Le vendredi 27 novembre 2009 ? 01:45 +0100, Krzysztof Kotlenga a ?crit
:
> Nicolas Mailhot wrote:
> 
> > Le jeudi 26 novembre 2009 ? 22:06 +0100, Krzysztof Kotlenga a
> > What is already there is so verbose very few people use it
>                                       ^^^^^^^^
> You never know...
Fact: I check every single font package that ends up in Fedora. On my
system I have

find /etc/fonts/conf.d/ | wc -l
239

Is that a good enough sample for you?
> 
> > OTOH the match target/test is a reinvention of emacs''
let''s use lisp
> > syntax as configuration syntax hell
> 
> This doesn''t sound like technical merit.
This is not about "technical merit" as in "what fontconfig can
do".
Everyone agrees that fontconfig is a very powerful tool. This is about
the usability of fontconfig config syntax. Currently the power of
fontconfig is not exercised because its config language is
over-convoluted and too low-level (either people do not try to use it at
all, or when they shoot themselves in the foot because of the syntax)

This is about things like
http://bugs.freedesktop.org/show_bug.cgi?id=20911
that should not exist at all if the config syntax was clearer for human
beings ("with X, ... I don''t think it can do any harm at
all." famous
last words of one of the commenters, if you knew how often I read
something like that from a CJK packager word to see a new bug opened a
month later)

PS. Why did you remove the list CC?

-- 
Nicolas Mailhot

Le Ven 27 novembre 2009 01:10, Nicolas Mailhot a ?crit :
> To keep syntax consistent with A. and have a reliable rule the packager
> understands, it should have been
>
> <alias>
>   <font>
>     <family>[genericname]</family>
>     <lang>[loc]</lang>
>   </font>
>   <prefer>
>     <font>
>       <family>[fontname]</family>
>       <lang>[loc]</lang>
>     </font>
>   </prefer>
> </alias>
Actually, it should have been something like:

 <alias>
   <font>
     <family>[genericname]</family>
     <langrequest>[loc]</langrequest>
     <langcover>[loc]</langcover>
   </font>
   <prefer>
     <font>
       <family>[fontname]</family>
     </font>
   </prefer>
 </alias>

to distinguish between lang as "the language I''m rendering
for" and lang as
"the glyphs associated with the foo .orth"

Many many CJK fontconfig configuration bugs occur because lang (as a rendering
context) is confused with lang (as specific glyphs in a font)

-- 
Nicolas Mailhot

Nicolas Mailhot wrote:
>>> What is already there is so verbose very few people use it
>>                                      ^^^^^^^^
>> You never know...
> 
> Fact: I check every single font package that ends up in Fedora. On my
> system I have
> 
> find /etc/fonts/conf.d/ | wc -l
> 239
> 
> Is that a good enough sample for you?
No. It doesn''t allow you to take an assumption that very few people use
(and actually understand and are happy with) fonts.conf. There''s a
world beyond Fedora, too.
>>> OTOH the match target/test is a reinvention of emacs''
let''s use
>>> lisp syntax as configuration syntax hell
>> 
>> This doesn''t sound like technical merit.
> 
> This is not about "technical merit" as in "what fontconfig
can do".
> Everyone agrees that fontconfig is a very powerful tool. This is about
> the usability of fontconfig config syntax. Currently the power of
> fontconfig is not exercised because its config language is
> over-convoluted and too low-level (either people do not try to use it
> at all, or when they shoot themselves in the foot because of the
> syntax)
> 
> This is about things like
> http://bugs.freedesktop.org/show_bug.cgi?id=20911
One of the best answers for such statements is: patches welcome. And it
usually ends the discussion. Otherwise someone has to step up and do
the work, preferably breaking as little as possible and not forcing his
own "syntax" preferences (it''s not even a syntax, because
syntax is
XML). Interestingly, comments on this bug mention that Ubuntu does a
good job?
> (...) CJK packager
So perhaps we should invite them here. But this topic deserves a new
thread, or two:
- about the CJK issues
- about breaking well-established configuration format.
> PS. Why did you remove the list CC?
My mistake. I posted to the list afterward.

-- 
The fact is, most software is crap, and most software developers
are lazy and stupid. Same as most customers are stupid too.
                         -- Hua Zhong, Linux Kernel Mailing List

Le Ven 27 novembre 2009 12:30, Krzysztof Kotlenga a ?crit
:
>
> Nicolas Mailhot wrote:
>
>>>> What is already there is so verbose very few people use it
>>>                                      ^^^^^^^^
>>> You never know...
>>
>> Fact: I check every single font package that ends up in Fedora. On my
>> system I have
>>
>> find /etc/fonts/conf.d/ | wc -l
>> 239
>>
>> Is that a good enough sample for you?
>
> No. It doesn''t allow you to take an assumption that very few
people use
> (and actually understand and are happy with) fonts.conf. There''s a
> world beyond Fedora, too.
Ok, then let''s just say that among the nitwits that contribute to
Fedora, this
syntax is not used widely (said nitwits include the person that did the last
official fontconfig release)

Is that good enough for you and can we go back to fixing stuff please ?
>> This is about things like
>> http://bugs.freedesktop.org/show_bug.cgi?id=20911
>
> One of the best answers for such statements is: patches welcome. And it
> usually ends the discussion.
Then: patches welcome. If the current syntax is so great please fix all those
~ 200 fontconfig files using syntax you so like so we have less problems
downstream.

As long as we end up writing a lot of those files in Fedora (upstreamed when
possible for example the fontconfig files dejavu ships were written by me
before upstreaming) I don''t think it''s unreasonable to ask for
a syntax that
makes our work easier.
> Otherwise someone has to step up and do
> the work, preferably breaking as little as possible and not forcing his
> own "syntax" preferences (it''s not even a syntax,
because syntax is
> XML). Interestingly, comments on this bug mention that Ubuntu does a
> good job?
I refrained from the obvious snarky answer to this comment so far,
don''t push
me. Looking at the mess all the Debian-imported rules caused in Fedora, I
don''t share their assessment.

-- 
Nicolas Mailhot

Nicolas Mailhot wrote:
> This is about things like
> http://bugs.freedesktop.org/show_bug.cgi?id=20911
> that should not exist at all if the config syntax was clearer for human
> beings ("with X, ... I don''t think it can do any harm at
all." famous
> last words of one of the commenters, if you knew how often I read
> something like that from a CJK packager word to see a new bug opened a
> month later)
>   
I see you are teasing me here. I thought on this issue, we
are on the same side.
> Actually, it should have been something like:
>
>  <alias>
>    <font>
>      <family>[genericname]</family>
>      <langrequest>[loc]</langrequest>
>      <langcover>[loc]</langcover>
>    </font>
>    <prefer>
>      <font>
>        <family>[fontname]</family>
>      </font>
>    </prefer>
>  </alias>
>
> ...
>
> Many many CJK fontconfig configuration bugs occur because lang (as a
rendering
> context) is confused with lang (as specific glyphs in a font)
look, isn''t this the same (sort of) as I proposed in the
Comment#16 in the above bug?

I apologize for causing issues when using the fontconfig
files I wrote for several CJK fonts. But I do want to remind you
that this is largely due to the complex nature of CJK, not by
others. Most non-CJK languages don''t have the z(shape)-variant
problems as in CJK; and they also don''t need to embed bitmaps
to make their glyph readable. So, the reason you see more
complains about CJK is rather a reflection to the insufficiency
to handle these unique problems in the current font rendering
pipeline (I know most of you hate bitmaps).

Talking about syntax, I am sort of agree on both sides.
Yes, using prepend/prepend_first/postpend..., you can do most
things a specific language requires. Indeed, that''s what
I used to do, but being blamed badly in the past [1]. In
Fedora''s fontconfig guideline, the only "clean" syntax
for setting font orders is
"<alias><prefer></prefer></alias>",
anything else, prepend/prepend_first with/wo strong binding,
will be treated as monsters.
And yes, if <alias><prefer> support language tag inside,
it would make these rules looks a lot nicer, and of course,
help me get away from a lot of blames by using the
"fragile" constructs.

So, in summary, I support to add the proposed syntax to
fontconfig; but if you do not get enough support to add this
to fontconfig, then please don''t blame cjk people in the
future by using the low-level operations.

Now back to the original thread, I personally don''t
think it is much of a CJK issue. I think what was originally
asked is about the granularity of fontconfig: fontconfig
has font-level fallback and glyph-block (in terms of lang)
level fallback, but it does not seem to have glyph-level
fallback. CJK issue can be pretty much solved by the second
level, but glyph-level is a even finer one. I too think
this is something fontconfig developers need to think
of in the future.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=381311#c0

Le vendredi 27 novembre 2009 ? 13:46 -0500, Qianqian Fang a ?crit
:
> Nicolas Mailhot wrote:
> > This is about things like
> > http://bugs.freedesktop.org/show_bug.cgi?id=20911
> > that should not exist at all if the config syntax was clearer for
> human
> > beings ("with X, ... I don''t think it can do any harm at
all."
> famous
> > last words of one of the commenters, if you knew how often I read
> > something like that from a CJK packager word to see a new bug opened
> a
> > month later)
> >   
> I see you are teasing me here. I thought on this issue, we
> are on the same side.
I hope we are :) As far as I''m concerned, it''s not a Fedora vs
Debian,
there are only two distributions that try to push the fontconfig
enveloppe right now, our approaches differ but not the objectives.
> > Many many CJK fontconfig configuration bugs occur because lang (as a
> rendering
> > context) is confused with lang (as specific glyphs in a font)
> look, isn''t this the same (sort of) as I proposed in the
> Comment#16 in the above bug?
> 
> I apologize for causing issues when using the fontconfig
> files I wrote for several CJK fonts. But I do want to remind you
> that this is largely due to the complex nature of CJK, not by
> others.
I that was not clear I''m not blaming you or other CJK users.
They''re not
worse than other font users, they just inherited more difficult
requirements. I''m fairly sure that as Indic and Arabic fonts become
more
numerous, those other regions will experiment the same pains (since they
also have the same glyph sharing problems)

If I wrote this suite of messages on the fontconfig list it''s also
because we were talking past one another in the bug tracker, and if we
both, after spending a fairly long time working with fontconfig, can not
explain to one another what the correct sane syntax to use should be, we
have a huge problem.

I don''t want a fontconfig syntax I barely understand for basic stuff
like declaring a CJK font. I want a syntax which is simple enough I can
understand it *and* explain to someone who never packaged a font for
fontconfig before. Not a "use this bit of vodoo here just because" but
"use this here because here is the general syntax model and see how
we''re just declining it for your use case". I need to be able to
coach
font packagers with little technical experience, very basic English
skills, from cultures where you don''t ask questions publicly when you
don''t understand, and besides which may be in a different timezone and
never available for an online chat to clarify matters.

Current fontconfig syntax, is not helping a lot.
> Talking about syntax, I am sort of agree on both sides.
> Yes, using prepend/prepend_first/postpend..., you can do most
> things a specific language requires. Indeed, that''s what
> I used to do, but being blamed badly in the past [1]. In
> Fedora''s fontconfig guideline, the only "clean" syntax
> for setting font orders is
"<alias><prefer></prefer></alias>",
> anything else, prepend/prepend_first with/wo strong binding,
> will be treated as monsters.
Note that they won''t be treated as monsters, because they
don''t get the
job done, they''re treated as monsters, because they make it unsafe for
other packagers to write other fontconfig rules. Without them you can
assume a rule declared in a file with 64 prefix will take precedence
over rules declares in a file > 64, with them all bets are of and rules
priority does not map to fontconfig file prefix number anymore.
> And yes, if <alias><prefer> support language tag inside,
> it would make these rules looks a lot nicer, and of course,
> help me get away from a lot of blames by using the
> "fragile" constructs.
> 
> So, in summary, I support to add the proposed syntax to
> fontconfig; but if you do not get enough support to add this
> to fontconfig, then please don''t blame cjk people in the
> future by using the low-level operations.
I''m fairly sure you could do most of it with low-level operations
today.
I''m not sure the complexity of the low-level syntax makes it possible
for people to easily agree on what the correct pattern should be. From
my POW, getting fontconfig to work for CJK should be a priority, as
that''s unfortunately where a large proportion of current fontconfig
problems lay today. And not because of a specific flaw in CJK packagers,
but because of the inadequacy of the tools that exist to their needs.
> Now back to the original thread, I personally don''t
> think it is much of a CJK issue. I think what was originally
> asked is about the granularity of fontconfig: fontconfig
> has font-level fallback and glyph-block (in terms of lang)
> level fallback, but it does not seem to have glyph-level
> fallback. CJK issue can be pretty much solved by the second
> level, but glyph-level is a even finer one. I too think
> this is something fontconfig developers need to think
> of in the future.
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=381311#c0
Regards,

-- 
Nicolas Mailhot

On 11/27/2009 06:30 AM, Krzysztof Kotlenga wrote:
>
> One of the best answers for such statements is: patches welcome. And it
> usually ends the discussion.
That''s usually true.  However, in this case, as the Fedora font special
interest group lead, Nicolas has great insight into the current configuration 
syntax.  And I asked for his input.

Unfortunately though the problems seem to be much more fundamental than I 
thought.  Nicolas is right to the point with problems he brings up. 
I''m not
sure I agree with solutions he proposes though.

Alternative solutions, for example, include:

   - Write XSL converters from his proposed simple syntax to fontconfig format,

   - Write a GUI / TUI tool to generate fontconfig confs.

The point being: we all know XML is too verbose to write manually. 
That''s by
design though.  The solution is to not write it manually...

Anyway, I''m afraid I can''t pay too much attention to this
right away as I have
to get back to finishing HarfBuzz and other long overdue projects.

Cheers,
behdad

PS. Krzysztof, thanks for contributing to the thread.  You said many things 
that I wanted to say.

Behdad Esfahbod wrote:
> Unfortunately though the problems seem to be much more fundamental than I 
> thought.  Nicolas is right to the point with problems he brings up. 
I''m not
> sure I agree with solutions he proposes though.
>   
can you be more specific? what''s wrong with the syntax Nicolas
proposed? redundancy?
> Alternative solutions, for example, include:
>
>    - Write XSL converters from his proposed simple syntax to fontconfig
format,
>    - Write a GUI / TUI tool to generate fontconfig confs.
>   
talking about GUI, I do have something to share. I wrote a
javascript-based program to let users set font orders and
get instant feedbacks. You can browse the page at

http://wenq.org/cloud/fcdesigner.html#en

a variant of the code enumerating local fonts using
a flash object can be browsed at

http://wenq.org/cloud/fcdesigner_local.html#en

hit the [create] button on the top to produce the output,
which is the content of ~/.fonts.conf. It does use "invasive"
syntax such as prepend/binding=strong, but this can be
easily changed. I included default options for Chinese,
Japanese and Latin; for the first two, bitmap-priority
and vector-priority settings can be chosen from the top menu.

Any feedback is welcome (maybe start a new thread?)
> The point being: we all know XML is too verbose to write manually. 
That''s by
> design though.  The solution is to not write it manually...
>
> Anyway, I''m afraid I can''t pay too much attention to this
right away as I have
> to get back to finishing HarfBuzz and other long overdue projects.
>
> Cheers,
> behdad
>
> PS. Krzysztof, thanks for contributing to the thread.  You said many things
> that I wanted to say.
> _______________________________________________
> Fontconfig mailing list
> Fontconfig at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/fontconfig
>
>
>
>

Le vendredi 27 novembre 2009 ? 16:16 -0500, Behdad Esfahbod a ?crit :
> And I asked for his input.
It was a good time for me to do so, as I spend all the year looking into
font packages, and don''t always take the time to provide feedback on
what I see.
> Unfortunately though the problems seem to be much more fundamental
> than I 
> thought.  Nicolas is right to the point with problems he brings up. 
> I''m not 
> sure I agree with solutions he proposes though.
I''m sorry if I was not convincing enough
> Alternative solutions, for example, include:
> 
>    - Write XSL converters from his proposed simple syntax to
> fontconfig format,
I was doing xslt before the fontconfig master conf file was split, it
works for advanced packagers, everyone else refuses to touch it. So if
there needs to be xslt processing it should be hidden from fontconfig
users
>    - Write a GUI / TUI tool to generate fontconfig confs.
A gui/tui would be great but is that something that much easier to do
than cleaning up the config syntax? The history of such config file
generators is not very good, and usually you need to clean up the config
syntax anyway so humans have a chance to understand problems when the
generated config is not working.
> The point being: we all know XML is too verbose to write manually. 
> That''s by 
> design though.  The solution is to not write it manually...
I disagree there. XML is too verbose to get it right or human friendly
is a urban myth which is resurrected every time a developer does not
want to make this effort. But the web was largely created using manual
XML-y syntax (before editors were available) and is a clear proof
human-friendly syntax is possible (there are other examples such as
docbook). However such human-friendliness can only be achieved if a lot
of time is spent trying to understand how humans will need to use the
file, while developers like to map their application quirks directly to
XML.

I''ve always considered the original xml fontconfig syntax brilliant,
you
could point someone to the sans preference list for example and he would
have no problem editing it without reading any manual file.

The problem is that we''ve started to do more and more with fontconfig
since then, and all the small additions that happened over the years,
while making it possible to do more, progressively transformed a
fontconfig file from something consistent and simple and easy to read in
something a lot less obvious. And that is easy to write now, with
hindsight, seeing what the use-cases in the field are, and were config
writers stumble. It is not a criticism of the people that did those
additions, hindsight his wonderful but you only get it after mistakes
are made.

So fontconfig syntax is IMHO ripe for a re-factoring, to simplify it,
improve consistency, and hide all the low-level stuff that seeped in
through and that fontconfig users should really not have to care about.

(it is not different from what you''ve done with HarfBuzz-ng, a config
file is a software ? human API, and APIs need reworking from time to
time)
> Anyway, I''m afraid I can''t pay too much attention to this
right away
> as I have 
> to get back to finishing HarfBuzz and other long overdue projects.
I hope that whenever you, or another fontconfig developer, has the time
to work on the subject, he''ll take some time to reflect on what I wrote
this week.

-- 
Nicolas Mailhot

Hi,

Anyway since I need to explain this stuff all the time to new font
packagers I''ve written a clean fontconfig presentation support

http://nim.fedorapeople.org/Understanding%20fonts%20and%20fontconfig.tar.gz

I''d appreciate if people on this list could check it for accuracy and
verify if there are no obvious holes in the fontconfig pseudo-syntax
used in the use case examples 

Also, I''d appreciate if people that know fontconfig innards could
provide the cleanest safe way to do the same thing as the pseudo-syntax
examples. I gleaned some patterns right and left over time, but if I
have to discuss CJK with Qianqian Fang, I''d rather start from something
more authoritative.

The use case examples are not random, they correspond to actual needs we
had to tackle in the Fedora Fonts SIG since it was created two years ago
(astute readers will recognize the Droid use case).

Regards,

-- 
Nicolas Mailhot
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Ceci est une partie de message
 =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20091129/37961327/attachment.pgp

Hi,

Nicolas Mailhot wrote:
> Anyway since I need to explain this stuff all the time to new font
> packagers I''ve written a clean fontconfig presentation support
> 
> http://nim.fedorapeople.org/Understanding%20fonts%20and%20fontconfig.tar.gz
> 
> I''d appreciate if people on this list could check it for accuracy
and
> verify if there are no obvious holes in the fontconfig pseudo-syntax
> used in the use case examples 
I don''t have much to say in terms of the correctness of the 
pseudo-syntax, but the presentation as a whole was very interesting.

My only comment is that I''m not crazy about the usage of
"<fake>" in the
examples.

--Pat