Fontconfig - Feb 2009 - Trouble forcing rescanning of font dirs

Hello,

 

We are using fontconfig in our project for font matching and it works
well. However at certain times we need to check if the user has
installed new fonts or removed others and make sure that fontconfig
stays current with those changes.

 

Thus we try to force fontconfig to validate all its caches and we do
this by doing something like

 

if ( ! FcConfigUptoDate (0) )

        FcInitReinitialize ();

 

at appropriate times (i.e. just before reloading content).

 

Although FcConfigUptoDate() correctly reports a change when something is
added or removed from the font directories FcInitReinitialize() does not
produce the intended effect (revalidating the font directories).

 

We tracked a bit the code in fontconfig  and when the list of fonts is
being rebuilt via FcConfigBuildFonts() the function FcDirCacheRead() is
called but with its force argument to false (which appears correct to
me).  Thus the call chain to load the cache file is FcDirCacheRead() ->
FcDirCacheLoad() -> FcDirCacheProcess() -> FcDirCacheMapHelper() ->
FcDirCacheMapFd(). But when FcDirCacheMapFd() finds the cache file
already loaded through FcCacheFindByStat() it does not verify that the
font directory  stat data matches the one in the cache file, happily
returning an outdated cache.

 

I think that FcDirCacheMapFd() should call FcCacheTimeValid() if
FcCacheFindByStat() returns a cache. Would this be  the correct fix or
should I go through some other mechanism to force fontconfig to validate
all its caches? 

 

I would greatly appreciate some feedback on this.

 

Sincerely,

 

Diego

--
Diego Santa Cruz, PhD
Technology Architect
_________________________________
SpinetiX S.A.
Rue des Terreaux 17
1003, Lausanne, Switzerland
T +41 21 341 15 50
F +41 21 311 19 56
diego.santacruz at spinetix.com
http://www.spinetix.com
_________________________________

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.freedesktop.org/archives/fontconfig/attachments/20090202/f66af5d3/attachment.html

"Diego Santa Cruz" <Diego.SantaCruz at spinetix.com> ????????:
> Thus we try to force fontconfig to validate all its caches and we do
> this by doing something like
>
>  
>
> if ( ! FcConfigUptoDate (0) )
>
>         FcInitReinitialize ();
>
>  
>
> at appropriate times (i.e. just before reloading content).
I also think this is a bug, see 

https://bugzilla.novell.com/show_bug.cgi?id=438855#c23

Comment #24 of this bug has a patch which I attach here as well
for convenience.

The line

 if (dir_stat.st_mtime <= file_stat.st_mtime) {

was removed recently from fccache.c, I think this was probably removed
by accident and was a mistake.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

-------------- next part --------------
A non-text attachment was scrubbed...
Name: bnc438855-make-FcInitReinitialize-work-again.patch
Type: text/x-patch
Size: 715 bytes
Desc: not available
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20090202/ae3179ef/attachment.bin

Mike FABIAN wrote:
> I also think this is a bug, see 
> 
> https://bugzilla.novell.com/show_bug.cgi?id=438855#c23
> 
> Comment #24 of this bug has a patch which I attach here as well
> for convenience.
> 
> The line
> 
>  if (dir_stat.st_mtime <= file_stat.st_mtime) {
> 
> was removed recently from fccache.c, I think this was probably removed
> by accident and was a mistake.
I doubt you really mean "recently".  Anyway, that was not a mistake. 
That
line will cause lots of problems with clock skews and changes (which are not
uncommon at new installations).  Instead, the cache stores the dir timestamp,
and the two are compared for equality.

I think the approach Diego proposes may be real the fix.  However, I''m
puzzled
as live font installation and uninstallation both work for me correctly. 
I''ll
check it out again.

Diego, which fontconfig version are you using?

Makes me wonder, what patches do you carry around in Novell that should be
upstream?  Please file them in bugzilla and I''ll take a look. 
We''re getting
2.7 out really soon.

behdad

Behdad Esfahbod <behdad at behdad.org> ????????:
> Makes me wonder, what patches do you carry around in Novell that should be
> upstream?
The .src.rpm we currently use is here:
> Please file them in bugzilla and I''ll take a look.
I?ll do that today, I think there were no important changes except the
FcInitReinitialize() problem which caused considerable headache.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> I doubt you really mean "recently".  Anyway, that was not a
mistake.  That
> line will cause lots of problems with clock skews and changes (which are
not
> uncommon at new installations).  Instead, the cache stores the dir
timestamp,
> and the two are compared for equality.
OK, but it is very important to fix this, currently it really doesn?t
work.  I just forgot to post our patch because of the chaos when
releasing openSUSE 11.1 but this is really important.
> I think the approach Diego proposes may be real the fix.  However,
I''m
> puzzled as live font installation and uninstallation both work for me
> correctly.  I''ll check it out again.
Does the test I did in 

https://bugzilla.novell.com/show_bug.cgi?id=438855#c25
https://bugzilla.novell.com/show_bug.cgi?id=438855#c26

work for you?

For me it does not work without my patch.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

> -----Original Message-----
> From: Behdad Esfahbod [mailto:behdad.esfahbod at gmail.com] On Behalf Of
> Behdad Esfahbod
> Sent: 02 February 2009 13:06
> To: mfabian at suse.de
> Cc: Diego Santa Cruz; fontconfig at lists.freedesktop.org
> Subject: Re: [Fontconfig] Trouble forcing rescanning of font dirs
> 
> Mike FABIAN wrote:
> > I also think this is a bug, see
> >
> > https://bugzilla.novell.com/show_bug.cgi?id=438855#c23
> >
> > Comment #24 of this bug has a patch which I attach here as well
> > for convenience.
> >
> > The line
> >
> >  if (dir_stat.st_mtime <= file_stat.st_mtime) {
> >
> > was removed recently from fccache.c, I think this was probably
> removed
> > by accident and was a mistake.
> 
> I doubt you really mean "recently".  Anyway, that was not a
mistake.
> That
> line will cause lots of problems with clock skews and changes (which
> are not
> uncommon at new installations).  Instead, the cache stores the dir
> timestamp,
> and the two are compared for equality.
I am also afraid from comparing mtime''s like above, so I do not think
it is the correct fix.
> 
> I think the approach Diego proposes may be real the fix.  However,
I''m
> puzzled
> as live font installation and uninstallation both work for me
> correctly.  I''ll
> check it out again.
> 
> Diego, which fontconfig version are you using?
I am running 2.6.0.

I guess the reason why it works for you and not us is that we are only running
one application that is using fontconfig and that we do not run fc-cache right
after font install for a variety of reasons. In this case nobody is recreating
the cache file and thus fontconfig finds that the copy it has already in memory
is the same as the one on disk and in that case it does not bother to check for
the cache being up to date.

I am testing my proposed fix and will let you know how it goes shortly.

Diego

--
Diego Santa Cruz, PhD
Technology Architect
_________________________________
SpinetiX S.A.
Rue des Terreaux 17
1003, Lausanne, Switzerland
T +41 21 341 15 50
F +41 21 311 19 56
diego.santacruz at spinetix.com
http://www.spinetix.com
_________________________________

http://bugs.freedesktop.org/show_bug.cgi?id=19904

For details see:

https://bugzilla.novell.com/show_bug.cgi?id=158573

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Mike FABIAN wrote:
> Behdad Esfahbod <behdad at behdad.org> ????????:
> 
>> Makes me wonder, what patches do you carry around in Novell that should
be
>> upstream?
> 
> The .src.rpm we currently use is here:
Is it just me or you forgot to paste the URL?

>> Please file them in bugzilla and I''ll take a look.
> 
> I?ll do that today, I think there were no important changes except the
> FcInitReinitialize() problem which caused considerable headache.
Thanks.

behdad

Behdad Esfahbod <behdad at behdad.org> ????????:
> Mike FABIAN wrote:
>> Behdad Esfahbod <behdad at behdad.org> ????????:
>> 
>>> Makes me wonder, what patches do you carry around in Novell that
should be
>>> upstream?
>> 
>> The .src.rpm we currently use is here:
>
> Is it just me or you forgot to paste the URL?
Oh, sorry, here it is:

http://download.opensuse.org/repositories/M17N/SUSE_Factory/src/fontconfig-2.6.0-10.2.src.rpm

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

> -----Original Message-----
> From: fontconfig-bounces at lists.freedesktop.org [mailto:fontconfig-
> bounces at lists.freedesktop.org] On Behalf Of Diego Santa Cruz
> 
> I am testing my proposed fix and will let you know how it goes
shortly.
> 
Please find attached my proposed fix. As far as I could test it does
work as intended.

However, we have hit two more problems once fontconfig realizes that the
cache file is outdated and builds a new one.

1. When fontconfig builds a cache it inserts it into the cache chain
with cache_dev, cache_ino and cache_mtime all set to zero, since the
cache is not yet on disk. However, when the cache is finally written to
disk by FcDirCacheWrite() the fields in the cache chain are not updated.
The effect is that any later FcCacheFindByStat() on the same font
directory will fail because the cache_{dev,ino,mtime} cannot match,
resulting in reloading yet again the cache files. Would doing a stat
just before FcAtomicReplaceOrig() and inserting these fields into the
cache chain if FcAtomicReplaceOrig() succeeds be OK?

2. Under Windows the FcAtomicReplaceOrig() always fails if the cache
file is loaded somewhere. In fact, unlink and rename fail if the (old)
file is mmaped by someone (which often is the same app). I think that,
under Windows, if unlink() fails we should try to rename the old file to
some other name, and then rename the new cache file to the correct cache
name. Note that under Windows an open or mmaped file can be renamed, at
least within the same directory. This however runs the risk of leaving
files behind, but well its Windows... Any other ideas?

Sincerely,

Diego


--
Diego Santa Cruz, PhD
Technology Architect
_________________________________
SpinetiX S.A.
Rue des Terreaux 17
1003, Lausanne, Switzerland
T +41 21 341 15 50
F +41 21 311 19 56
diego.santacruz at spinetix.com
http://www.spinetix.com
_________________________________

-------------- next part --------------
A non-text attachment was scrubbed...
Name: fccache-valid-mtime.patch
Type: application/octet-stream
Size: 578 bytes
Desc: fccache-valid-mtime.patch
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20090202/33e140fc/attachment.obj

Hi Diego,

I committed a modified version of your patch to my tree.

Thanks,
behdad

PS. Here''s the patch:

diff --git a/src/fccache.c b/src/fccache.c
index 53df860..f35304a 100644
--- a/src/fccache.c
+++ b/src/fccache.c
@@ -538,7 +538,13 @@ FcDirCacheMapFd (int fd, struct stat *fd_stat, struct stat
        return NULL;
     cache = FcCacheFindByStat (fd_stat);
     if (cache)
-       return cache;
+    {
+       if (FcCacheTimeValid (cache, dir_stat))
+           return cache;
+       FcDirCacheUnload (cache);
+       cache = NULL;
+    }
+
     /*
      * Lage cache files are mmap''ed, smaller cache files are read.
This
      * balances the system cost of mmap against per-process memory usage.



Diego Santa Cruz wrote:
> Hello,
> 
>  
> 
> We are using fontconfig in our project for font matching and it works
> well. However at certain times we need to check if the user has
> installed new fonts or removed others and make sure that fontconfig
> stays current with those changes.
> 
>  
> 
> Thus we try to force fontconfig to validate all its caches and we do
> this by doing something like
> 
>  
> 
> if ( ! FcConfigUptoDate (0) )
> 
>         FcInitReinitialize ();
> 
>  
> 
> at appropriate times (i.e. just before reloading content).
> 
>  
> 
> Although FcConfigUptoDate() correctly reports a change when something is
> added or removed from the font directories FcInitReinitialize() does not
> produce the intended effect (revalidating the font directories).
> 
>  
> 
> We tracked a bit the code in fontconfig  and when the list of fonts is
> being rebuilt via FcConfigBuildFonts() the function FcDirCacheRead() is
> called but with its force argument to false (which appears correct to
> me).  Thus the call chain to load the cache file is FcDirCacheRead() ->
> FcDirCacheLoad() -> FcDirCacheProcess() -> FcDirCacheMapHelper()
->
> FcDirCacheMapFd(). But when FcDirCacheMapFd() finds the cache file
> already loaded through FcCacheFindByStat() it does not verify that the
> font directory  stat data matches the one in the cache file, happily
> returning an outdated cache.
> 
>  
> 
> I think that FcDirCacheMapFd() should call FcCacheTimeValid() if
> FcCacheFindByStat() returns a cache. Would this be  the correct fix or
> should I go through some other mechanism to force fontconfig to validate
> all its caches?
> 
>  
> 
> I would greatly appreciate some feedback on this.
> 
>  
> 
> Sincerely,
> 
>  
> 
> Diego
> 
> --
> Diego Santa Cruz, PhD
> Technology Architect
> _________________________________
> SpinetiX S.A.
> Rue des Terreaux 17
> 1003, Lausanne, Switzerland
> T +41 21 341 15 50
> F +41 21 311 19 56
> diego.santacruz at spinetix.com
> http://www.spinetix.com
> _________________________________
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Fontconfig mailing list
> Fontconfig at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/fontconfig

Hi Behdad,

Thanks for your feedback, in fact the second version of my patch is much closer
to yours :-) I was only missing the cache = NULL part.

As you may have noticed I have posted all the patches we are currently using on
top of 2.6.0 to the mailing list. Some of them are already in your tree but not
all.

BTW, is there any particular reason why the "official" fontconfig tree
is inactive since the release of 2.6.0 and yours is getting all the commits? Is
your tree to become the new "official" tree?

Best,

Diego
> -----Original Message-----
> From: Behdad Esfahbod [mailto:behdad.esfahbod at gmail.com] On Behalf Of
> Behdad Esfahbod
> Sent: 04 February 2009 03:38
> To: Diego Santa Cruz
> Cc: fontconfig at lists.freedesktop.org
> Subject: Re: [Fontconfig] Trouble forcing rescanning of font dirs
> 
> Hi Diego,
> 
> I committed a modified version of your patch to my tree.
> 
> Thanks,
> behdad
> 
> PS. Here''s the patch:
> 
> diff --git a/src/fccache.c b/src/fccache.c
> index 53df860..f35304a 100644
> --- a/src/fccache.c
> +++ b/src/fccache.c
> @@ -538,7 +538,13 @@ FcDirCacheMapFd (int fd, struct stat *fd_stat,
> struct stat
>         return NULL;
>      cache = FcCacheFindByStat (fd_stat);
>      if (cache)
> -       return cache;
> +    {
> +       if (FcCacheTimeValid (cache, dir_stat))
> +           return cache;
> +       FcDirCacheUnload (cache);
> +       cache = NULL;
> +    }
> +
>      /*
>       * Lage cache files are mmap''ed, smaller cache files are
read.
> This
>       * balances the system cost of mmap against per-process memory
> usage.
> 
> 
> 
> Diego Santa Cruz wrote:
> > Hello,
> >
> >
> >
> > We are using fontconfig in our project for font matching and it works
> > well. However at certain times we need to check if the user has
> > installed new fonts or removed others and make sure that fontconfig
> > stays current with those changes.
> >
> >
> >
> > Thus we try to force fontconfig to validate all its caches and we do
> > this by doing something like
> >
> >
> >
> > if ( ! FcConfigUptoDate (0) )
> >
> >         FcInitReinitialize ();
> >
> >
> >
> > at appropriate times (i.e. just before reloading content).
> >
> >
> >
> > Although FcConfigUptoDate() correctly reports a change when something
> is
> > added or removed from the font directories FcInitReinitialize() does
> not
> > produce the intended effect (revalidating the font directories).
> >
> >
> >
> > We tracked a bit the code in fontconfig  and when the list of fonts
> is
> > being rebuilt via FcConfigBuildFonts() the function FcDirCacheRead()
> is
> > called but with its force argument to false (which appears correct to
> > me).  Thus the call chain to load the cache file is FcDirCacheRead()
> ->
> > FcDirCacheLoad() -> FcDirCacheProcess() -> FcDirCacheMapHelper()
->
> > FcDirCacheMapFd(). But when FcDirCacheMapFd() finds the cache file
> > already loaded through FcCacheFindByStat() it does not verify that
> the
> > font directory  stat data matches the one in the cache file, happily
> > returning an outdated cache.
> >
> >
> >
> > I think that FcDirCacheMapFd() should call FcCacheTimeValid() if
> > FcCacheFindByStat() returns a cache. Would this be  the correct fix
> or
> > should I go through some other mechanism to force fontconfig to
> validate
> > all its caches?
> >
> >
> >
> > I would greatly appreciate some feedback on this.
> >
> >
> >
> > Sincerely,
> >
> >
> >
> > Diego
> >
> > --
> > Diego Santa Cruz, PhD
> > Technology Architect
> > _________________________________
> > SpinetiX S.A.
> > Rue des Terreaux 17
> > 1003, Lausanne, Switzerland
> > T +41 21 341 15 50
> > F +41 21 311 19 56
> > diego.santacruz at spinetix.com
> > http://www.spinetix.com
> > _________________________________
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> ---
> >
> > _______________________________________________
> > Fontconfig mailing list
> > Fontconfig at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/fontconfig

Diego Santa Cruz wrote:
> Hi Behdad,
Hi,
> As you may have noticed I have posted all the patches we are currently
using on top of 2.6.0 to the mailing list. Some of them are already in your tree
but not all.
I''m going through the rest today.
> BTW, is there any particular reason why the "official" fontconfig
tree is inactive since the release of 2.6.0 and yours is getting all the
commits? Is your tree to become the new "official" tree?
I''m preparing patches.  Keith is supposed to review them and turn it
into 2.7.

Cheers,
behdad
> Best,
> 
> Diego
> 
>> -----Original Message-----
>> From: Behdad Esfahbod [mailto:behdad.esfahbod at gmail.com] On Behalf
Of
>> Behdad Esfahbod
>> Sent: 04 February 2009 03:38
>> To: Diego Santa Cruz
>> Cc: fontconfig at lists.freedesktop.org
>> Subject: Re: [Fontconfig] Trouble forcing rescanning of font dirs
>>
>> Hi Diego,
>>
>> I committed a modified version of your patch to my tree.
>>
>> Thanks,
>> behdad
>>
>> PS. Here''s the patch:
>>
>> diff --git a/src/fccache.c b/src/fccache.c
>> index 53df860..f35304a 100644
>> --- a/src/fccache.c
>> +++ b/src/fccache.c
>> @@ -538,7 +538,13 @@ FcDirCacheMapFd (int fd, struct stat *fd_stat,
>> struct stat
>>         return NULL;
>>      cache = FcCacheFindByStat (fd_stat);
>>      if (cache)
>> -       return cache;
>> +    {
>> +       if (FcCacheTimeValid (cache, dir_stat))
>> +           return cache;
>> +       FcDirCacheUnload (cache);
>> +       cache = NULL;
>> +    }
>> +
>>      /*
>>       * Lage cache files are mmap''ed, smaller cache files are
read.
>> This
>>       * balances the system cost of mmap against per-process memory
>> usage.
>>
>>
>>
>> Diego Santa Cruz wrote:
>>> Hello,
>>>
>>>
>>>
>>> We are using fontconfig in our project for font matching and it
works
>>> well. However at certain times we need to check if the user has
>>> installed new fonts or removed others and make sure that fontconfig
>>> stays current with those changes.
>>>
>>>
>>>
>>> Thus we try to force fontconfig to validate all its caches and we
do
>>> this by doing something like
>>>
>>>
>>>
>>> if ( ! FcConfigUptoDate (0) )
>>>
>>>         FcInitReinitialize ();
>>>
>>>
>>>
>>> at appropriate times (i.e. just before reloading content).
>>>
>>>
>>>
>>> Although FcConfigUptoDate() correctly reports a change when
something
>> is
>>> added or removed from the font directories FcInitReinitialize()
does
>> not
>>> produce the intended effect (revalidating the font directories).
>>>
>>>
>>>
>>> We tracked a bit the code in fontconfig  and when the list of fonts
>> is
>>> being rebuilt via FcConfigBuildFonts() the function
FcDirCacheRead()
>> is
>>> called but with its force argument to false (which appears correct
to
>>> me).  Thus the call chain to load the cache file is
FcDirCacheRead()
>> ->
>>> FcDirCacheLoad() -> FcDirCacheProcess() ->
FcDirCacheMapHelper() ->
>>> FcDirCacheMapFd(). But when FcDirCacheMapFd() finds the cache file
>>> already loaded through FcCacheFindByStat() it does not verify that
>> the
>>> font directory  stat data matches the one in the cache file,
happily
>>> returning an outdated cache.
>>>
>>>
>>>
>>> I think that FcDirCacheMapFd() should call FcCacheTimeValid() if
>>> FcCacheFindByStat() returns a cache. Would this be  the correct fix
>> or
>>> should I go through some other mechanism to force fontconfig to
>> validate
>>> all its caches?
>>>
>>>
>>>
>>> I would greatly appreciate some feedback on this.
>>>
>>>
>>>
>>> Sincerely,
>>>
>>>
>>>
>>> Diego
>>>
>>> --
>>> Diego Santa Cruz, PhD
>>> Technology Architect
>>> _________________________________
>>> SpinetiX S.A.
>>> Rue des Terreaux 17
>>> 1003, Lausanne, Switzerland
>>> T +41 21 341 15 50
>>> F +41 21 311 19 56
>>> diego.santacruz at spinetix.com
>>> http://www.spinetix.com
>>> _________________________________
>>>
>>>
>>>
>>>
>>>
---------------------------------------------------------------------
>> ---
>>> _______________________________________________
>>> Fontconfig mailing list
>>> Fontconfig at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/fontconfig

On Wed, 2009-02-04 at 10:32 -0500, Behdad Esfahbod wrote:
> I''m preparing patches.  Keith is supposed to review them and turn
it
> into 2.7.
And I''m busy getting X server 1.6 released, and so I haven''t
had enough
time to finish this review. Carl Worth reviewed a pile of obvious
patches and pushed those to upstream at least.

-- 
keith.packard at intel.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20090204/b2556f19/attachment.pgp

Mike FABIAN wrote:
>
http://download.opensuse.org/repositories/M17N/SUSE_Factory/src/fontconfig-2.6.0-10.2.src.rpm
Lots of stuff here, and not enough documentation.  I''m going through
the
patches right now (conf files are the other useful set).  Many of the patches
look like are not actually used by the RPM anymore and are from 2.3.94 days.
Certainly obsolete.


AppleRoman-DynaFont.patch:  Any reason why those entries are being removed
from the AppleRoman encoding?

28937-turn-off-embedded-bitmaps-by-default.patch:  Any reason why
you''re not
doing this in config files instead?

fc-cache.patch:  I don''t think this one is needed anymore.  My testing
suggests so at least.  Then again, there is no documentation as to what
it''s
trying to fix.

fc-match.patch:  Some real crack here.  That patch is *very* wrong.  Can you
dig what problem it''s *fixing*?

fontconfig-minmaxabs.patch:  Doesn''t seem to be needed.  Unless another
patch
you have uses those macros.

fc-app-dir-fix-new-new.diff:  I''m guessing that this is supposed to fix
the
issue with recursive directory structures.  I believe this has been fixed
since 2.4.

fc-validatecache-option.diff:  Any idea why you needed this?


behdad

Behdad Esfahbod <behdad at behdad.org> ????????:
> Mike FABIAN wrote:
>>
http://download.opensuse.org/repositories/M17N/SUSE_Factory/src/fontconfig-2.6.0-10.2.src.rpm
>
> Lots of stuff here, and not enough documentation.  I''m going
through the
> patches right now (conf files are the other useful set).  Many of the
patches
> look like are not actually used by the RPM anymore and are from 2.3.94
days.
> Certainly obsolete.
Yes, I will delete them soon. I kept some old patches for reference for
a while but I will really delete the very old ones now.

[...]

I try to answer each question in a separate mail.

First about the embedded bitmaps:
> 28937-turn-off-embedded-bitmaps-by-default.patch:  Any reason why
you''re not
> doing this in config files instead?
Yes, this patch is obsolete and it is not applied anymore. We are doing
this embedded bitmap enabling/disabling in the config files for a long
time already, see suse-hinting.conf in the source.rpm. Therefore this
patch isn''t applied for a long time. For the original reason of the
patch see

https://bugzilla.novell.com/show_bug.cgi?id=128937


Zhe Su wanted to have embedded bitmaps off by default. But he wanted
them turned on for Chinese. He is Chinese.  There are very few non CJK
fonts with embedded bitmaps, the only one I currently know seems to be
Microsoft?s "Courier New". But the bitmaps in "Courier New"
seem to be
few and not really important. Maybe Zhe Su wanted to have them
turned of by default because there are some non-CJK fonts
where they cause problems? I don?t know and the original bug
doesn?t say either. 

Currently we do this:

        <match target="font">
                <edit name="embeddedbitmap"
mode="assign">
                        <bool>false</bool>
                </edit>
        </match>
        <match target="font">
                <test name="lang" compare="contains">
                        <string>ja</string>
                        <string>ko</string>
                        <string>zh-CN</string>
                        <string>zh-TW</string>
                        <string>zh-HK</string>
                        <string>zh-SG</string>
                </test>
                <edit name="embeddedbitmap"
mode="assign">
                        <bool>true</bool>
                </edit>
        </match>

which achieves the same what Zhe Su wanted but in the config files.

And, because of user request we disable embedded bitmaps for the IPA
fonts because it doesn?t really make sense to get a bitmap at 12 pixels,
an anti-aliased outline at 13 and a bitmap again at 14:

<!--
    The IPA fonts have bitmaps only in 12, 14, and 16 pixels.
    That is rather useless, it is not nice to see a bitmaps at 12
    pixels, antialiased glyphs at 13 pixels and bitmaps at 14
    pixels again.
-->
       <match target="font">
               <test name="family">
                       <string>IPAMincho</string>
                       <string>IPAGothic</string>
                       <string>IPAPMincho</string>
                       <string>IPAPGothic</string>
                       <string>IPAUIGothic</string>
               </test>
               <edit name="autohint">
                       <bool>false</bool>
               </edit>
               <edit name="embeddedbitmap">
                       <bool>false</bool>
               </edit>
       </match>


-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> AppleRoman-DynaFont.patch:  Any reason why those entries are being removed
> from the AppleRoman encoding?
Very old bug, unfortunately not publicly visible I think because
our bugzilla was not yet public at that time. I have moved
the bug to openSUSE 11.2 Alpha0 and hope it is visible now:

https://bugzilla.novell.com/show_bug.cgi?id=53243

As you can see in the bug report, the submitter of this patch, Zhe Su
couldn''t tell me either what this patch does.  Apparently it had
something to do with the commercial Japanese Dynafont font series,
therefore I tried to get hold of these fonts and do some testing and
found that weird glyphs at completely wrong positions were displayed
without this patch. The patch seemed to have no effect whatsoever on
other fonts.

I?ll try now if the situation is still the same and the wrong
glyphs reappear if I disable the patch.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Mike FABIAN <mfabian at suse.de> ????????:
> Behdad Esfahbod <behdad at behdad.org> ????????:
>
>> AppleRoman-DynaFont.patch:  Any reason why those entries are being
removed
>> from the AppleRoman encoding?
>
> Very old bug, unfortunately not publicly visible I think because
> our bugzilla was not yet public at that time. I have moved
> the bug to openSUSE 11.2 Alpha0 and hope it is visible now:
>
> https://bugzilla.novell.com/show_bug.cgi?id=53243
>
> As you can see in the bug report, the submitter of this patch, Zhe Su
> couldn''t tell me either what this patch does.  Apparently it had
> something to do with the commercial Japanese Dynafont font series,
> therefore I tried to get hold of these fonts and do some testing and
> found that weird glyphs at completely wrong positions were displayed
> without this patch. The patch seemed to have no effect whatsoever on
> other fonts.
>
> I?ll try now if the situation is still the same and the wrong
> glyphs reappear if I disable the patch.
The situation is unchanged, the dynafonts still display the same
wrong glyphs if I disable the patch.


-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> fc-match.patch:  Some real crack here.  That patch is *very* wrong.  Can
you
> dig what problem it''s *fixing*?
No, the patch is not applied, it is commented out in the .spec file.
I think this was only experimental and never really applied
in a fontconfig.rpm on openSUSE.

You are right that it looks wrong, I?ll remove it, it is only confusing.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> fc-cache.patch:  I don''t think this one is needed anymore.
Right. Obsolete. Was commented out in our the fontconfig.spec file
and not applied anymore.
> My testing suggests so at least.  Then again, there is no
> documentation as to what it''s trying to fix.
Originally this patch was from my colleague Takashi Iwai (CC:)
his comment was:

iwai> Hi,
iwai> 
iwai> the below patch fixes the segfault of fc-cache invoked with arguments
iwai> of non-existing directory name.

I remember that we submitted this upstream but it was rejected because
of the use of realpath(). The problem seems to have been solved in a
different way now in fontconfig, fc-cache doesn''t crash on non-existing
directories. Therefore the patch was commented out in our .spec file.


-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Mike FABIAN wrote:
> Mike FABIAN <mfabian at suse.de> ????????:
> 
>> Behdad Esfahbod <behdad at behdad.org> ????????:
>>
>>> AppleRoman-DynaFont.patch:  Any reason why those entries are being
removed
>>> from the AppleRoman encoding?
>> Very old bug, unfortunately not publicly visible I think because
>> our bugzilla was not yet public at that time. I have moved
>> the bug to openSUSE 11.2 Alpha0 and hope it is visible now:
>>
>> https://bugzilla.novell.com/show_bug.cgi?id=53243
>>
>> As you can see in the bug report, the submitter of this patch, Zhe Su
>> couldn''t tell me either what this patch does.  Apparently it
had
>> something to do with the commercial Japanese Dynafont font series,
>> therefore I tried to get hold of these fonts and do some testing and
>> found that weird glyphs at completely wrong positions were displayed
>> without this patch. The patch seemed to have no effect whatsoever on
>> other fonts.
>>
>> I?ll try now if the situation is still the same and the wrong
>> glyphs reappear if I disable the patch.
> 
> The situation is unchanged, the dynafonts still display the same
> wrong glyphs if I disable the patch.
So, it''s the case of a buggy fonts.  When we have support for removing
certain
glyphs from a font pattern you can use that to blacklist these glyphs.

As for whether fontconfig is right or wrong, I crosschecked the table with:

  http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT

and the fontconfig version is correct, but missing the entries 0xE0 to 0xFF.
Maybe there''s good reason for not including those, and that reason may
extend
to the ones you are removing.  I have no idea.  Keith?

behdad

Behdad Esfahbod wrote:
> So, it''s the case of a buggy fonts.  When we have support for
removing certain
> glyphs from a font pattern you can use that to blacklist these glyphs.
> 
> As for whether fontconfig is right or wrong, I crosschecked the table with:
> 
>   http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT
> 
> and the fontconfig version is correct, but missing the entries 0xE0 to
0xFF.
> Maybe there''s good reason for not including those, and that reason
may extend
> to the ones you are removing.  I have no idea.  Keith?
So, I checked again, and you are commenting out the everything but the ASCII
range, essentially making the MacRoman encoding equivalent to ASCII.  This
plain means that the font in question has the wrong encoding tag. 
That''s all.

behdad

Behdad Esfahbod <behdad at behdad.org> ????????:
> Behdad Esfahbod wrote:
>
>> So, it''s the case of a buggy fonts.  When we have support for
removing certain
>> glyphs from a font pattern you can use that to blacklist these glyphs.
>> 
>> As for whether fontconfig is right or wrong, I crosschecked the table
with:
>> 
>>   http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT
>> 
>> and the fontconfig version is correct, but missing the entries 0xE0 to
0xFF.
>> Maybe there''s good reason for not including those, and that
reason may extend
>> to the ones you are removing.  I have no idea.  Keith?
>
> So, I checked again, and you are commenting out the everything but the
ASCII
> range, essentially making the MacRoman encoding equivalent to ASCII.  This
> plain means that the font in question has the wrong encoding tag. 
That''s all.
OK, so I should blacklist the wrong glyphs in these fonts when this
becomes possible with fontconfig. Fixing the encoding tag in the
fonts is not possible in case of commercial fonts.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Mike FABIAN wrote:
> Behdad Esfahbod <behdad at behdad.org> ????????:
> 
>> Behdad Esfahbod wrote:
>>
>>> So, it''s the case of a buggy fonts.  When we have support
for removing certain
>>> glyphs from a font pattern you can use that to blacklist these
glyphs.
>>>
>>> As for whether fontconfig is right or wrong, I crosschecked the
table with:
>>>
>>>   http://unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT
>>>
>>> and the fontconfig version is correct, but missing the entries 0xE0
to 0xFF.
>>> Maybe there''s good reason for not including those, and
that reason may extend
>>> to the ones you are removing.  I have no idea.  Keith?
>> So, I checked again, and you are commenting out the everything but the
ASCII
>> range, essentially making the MacRoman encoding equivalent to ASCII. 
This
>> plain means that the font in question has the wrong encoding tag. 
That''s all.
> 
> OK, so I should blacklist the wrong glyphs in these fonts when this
> becomes possible with fontconfig. Fixing the encoding tag in the
> fonts is not possible in case of commercial fonts.
Right.

I was actually wrong about fontconfig''s table being incomplete. 
It''s as
complete as the ROMAN.TXT table.  The sort order tricked me.

behdad

Mike FABIAN <mfabian at suse.de> ????????:
> Behdad Esfahbod <behdad at behdad.org> ????????:
>
>> fc-cache.patch:  I don''t think this one is needed anymore.
>
> Right. Obsolete. Was commented out in our the fontconfig.spec file
> and not applied anymore.
>
>> My testing suggests so at least.  Then again, there is no
>> documentation as to what it''s trying to fix.
>
> Originally this patch was from my colleague Takashi Iwai (CC:)
> his comment was:
>
> iwai> Hi,
> iwai> 
> iwai> the below patch fixes the segfault of fc-cache invoked with
arguments
> iwai> of non-existing directory name.
>
> I remember that we submitted this upstream but it was rejected because
> of the use of realpath(). The problem seems to have been solved in a
> different way now in fontconfig, fc-cache doesn''t crash on
non-existing
> directories. Therefore the patch was commented out in our .spec file.
Something which canonicalizes pathnames would be nice though, if
realpath() cannot be used, is there another way to do it?
Currently fontconfig distinguishes even between directory names
like /foo/bar and /foo/bar/ and /foo//bar although they
are all the same:

mfabian at magellan:~$ rm  .fontconfig/*
mfabian at magellan:~$ fc-cache -v  /usr/share/fonts/truetype
/usr/share/fonts/truetype: skipping, existing cache is valid: 972 fonts, 0 dirs
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/mfabian/.fontconfig: cleaning cache directory
fc-cache: succeeded
mfabian at magellan:~$ fc-cache -v  /usr/share/fonts/truetype/
/usr/share/fonts/truetype/: caching, new cache contents: 972 fonts, 0 dirs
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/mfabian/.fontconfig: cleaning cache directory
fc-cache: succeeded
mfabian at magellan:~$ fc-cache -v  /usr/share/fonts//truetype
/usr/share/fonts//truetype: caching, new cache contents: 972 fonts, 0 dirs
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/mfabian/.fontconfig: cleaning cache directory
fc-cache: succeeded
mfabian at magellan:~$ fc-cache -v  /usr/share/fonts//truetype
/usr/share/fonts//truetype: skipping, existing cache is valid: 972 fonts, 0 dirs
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/mfabian/.fontconfig: cleaning cache directory
fc-cache: succeeded
mfabian at magellan:~$

Probably I kept this fc-cache.patch (but commented out
in the fontconfig.spec file) to remind me that it would
be nice to canonicalize pathnames.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> fontconfig-minmaxabs.patch:  Doesn''t seem to be needed.  Unless
another patch
> you have uses those macros.
Yes, obsolete. Was needed to fix the build long ago but has been forgotten.
I?ll remove it.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> fc-app-dir-fix-new-new.diff: I''m guessing that this is supposed to
fix
> the issue with recursive directory structures.  I believe this has
> been fixed since 2.4.
yes, therefore this patch was commented out. I remove it.
For details about this patch see 

https://bugzilla.novell.com/show_bug.cgi?id=148693#c6

and the following comments.

This was discussed with Patrick Lam and has then been
solved upstream.

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Behdad Esfahbod <behdad at behdad.org> ????????:
> fc-validatecache-option.diff:  Any idea why you needed this?
commented out in our fontconfig.spec.

Experimental patch by Takashi Iwai to achieve some speed up when one is
really, really sure that all caches are OK and do not need to be
checked.

Has been discussed on the fontconfig mailing list.

For details see 

https://bugzilla.novell.com/show_bug.cgi?id=148361

-- 
Mike FABIAN   <mfabian at suse.de>   http://www.suse.de/~mfabian
?????????????
I ? Unicode

Mike FABIAN wrote:
> Something which canonicalizes pathnames would be nice though, if
> realpath() cannot be used, is there another way to do it?
> Currently fontconfig distinguishes even between directory names
> like /foo/bar and /foo/bar/ and /foo//bar although they
> are all the same:
In my tree the are all considered the same already:

commit 26a496c95a72930b122c544a25af0c33de041c68
Author: Behdad Esfahbod <behdad at behdad.org>
Date:   Tue Aug 12 15:10:04 2008 -0400

    When canonizing filenames, squash // and remove final / (#bug 16286)

    The fact that we now drop final slashes from all filenames without
    checking that the file name represents a directory may surprise some,
    but it doesn''t bother me really.
> Probably I kept this fc-cache.patch (but commented out
> in the fontconfig.spec file) to remind me that it would
> be nice to canonicalize pathnames.
Well, just file a bug right away and you don''t have to keep reminder
files
around :).

behdad