Fontconfig - Nov 2005 - commit (branch): read and write cache files.

On Mon, 2005-07-25 at 14:39 -0400, Patrick Lam wrote:
> Keith Packard wrote:
> > I was thinking we''d continue to use per-directory caches
unless you
> > don''t think that will work. The reason here is that each user
can add
> > directories to the set, and having only a global cache means that
> > everyone would have a ~/.fonts.cache-2 file. Right now, the local
caches
> > are significantly more expensive to use as many additional stat(2)
calls
> > are needed to validate the cache.
> 
> I think that per-directory caches will work once I implement banked
> references to mmaped objects, e.g. right now we have
> 
> typedef struct _FcValueListPtr {
>     FcStorage               storage;
>     union {
>         int                 stat;
>         struct _FcValueList *dyn;
>     } u;
> } FcValueListPtr;
> 
> and after my next change we''ll have
> 
> typedef struct _FcStaticPtr {
> 	int bank;
> 	int offset;
> } FcStaticPtr;
> 
> typedef struct _FcValueListPtr {
>     FcStorage               storage;
>     union {
>         FcStaticPtr         stat;
>         struct _FcValueList *dyn;
>     } u;
> } FcValueListPtr;
> 
> I wonder if we should manually pack the FcStaticPtr into, say, 24 bits
> for offset and 8 bits for bank.  Or if we should wonder about that for
> now.  The banked data structures enable per-directory caches.
If that would limit the number of directories to 256, that would be
bad...

However, what you might consider is making the special bank value of
''0''
mean a pointer with other bank values being offsets. That would reduce
the size of the FcValueListPtr back to 8 bytes on 32-bit machines:

typedef struct _FcValueListPtr {
	int	bank;
	union {
		int stat;
		struct _FcValueList *dyn;
	} u;
} FcValueListPtr;

#define FC_BANK_DYNAMIC		0
> 
> The only remaining issue is that FcObjectPtr is now an int, and we want
> to keep it that way.  We can probably do something clever there, maybe
> again the 24/8 split or something.
I fear weird structures like this. If there aren''t a lot of them in
writable memory, then a simpler representation will cause fewer problems
in the future.
> 
> > If we need a single global cache, it would need to live in /var
> > somewhere, which is something of a pain as there isn''t strong
concensus
> > on what /var should look like these days. A quick read through the LSB
> > might provide some suggestions.
> 
> Hopefully we can avoid the single global cache.
Right. That leaves us with the same file structure we have today. There
have been requests for us to *permit* externally stored cache files for
read-only directories (generally on shared Windows partitions). We can
consider how to manage that after we have per-directory caches working
again.

Thanks for all of your hard work.

-keith

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20050725/bd94b930/attachment.pgp

Keith Packard wrote:
> I was thinking we''d continue to use per-directory caches unless
you
> don''t think that will work. The reason here is that each user can
add
> directories to the set, and having only a global cache means that
> everyone would have a ~/.fonts.cache-2 file. Right now, the local caches
> are significantly more expensive to use as many additional stat(2) calls
> are needed to validate the cache.
I think that per-directory caches will work once I implement banked
references to mmaped objects, e.g. right now we have

typedef struct _FcValueListPtr {
    FcStorage               storage;
    union {
        int                 stat;
        struct _FcValueList *dyn;
    } u;
} FcValueListPtr;

and after my next change we''ll have

typedef struct _FcStaticPtr {
	int bank;
	int offset;
} FcStaticPtr;

typedef struct _FcValueListPtr {
    FcStorage               storage;
    union {
        FcStaticPtr         stat;
        struct _FcValueList *dyn;
    } u;
} FcValueListPtr;

I wonder if we should manually pack the FcStaticPtr into, say, 24 bits
for offset and 8 bits for bank.  Or if we should wonder about that for
now.  The banked data structures enable per-directory caches.

The only remaining issue is that FcObjectPtr is now an int, and we want
to keep it that way.  We can probably do something clever there, maybe
again the 24/8 split or something.
> If we need a single global cache, it would need to live in /var
> somewhere, which is something of a pain as there isn''t strong
concensus
> on what /var should look like these days. A quick read through the LSB
> might provide some suggestions.
Hopefully we can avoid the single global cache.

pat

On Mon, 2005-07-25 at 00:15 -0400, Patrick Lam wrote:
> Currently it writes the mmap file to /tmp, but this will change soon.  I
> noticed that currently fontconfig writes cache files to directories
> containing fonts, plus a global one on a per-user basis in
> ~/.fonts.cache.  Is there an appropriate place to put a global cache file?
I was thinking we''d continue to use per-directory caches unless you
don''t think that will work. The reason here is that each user can add
directories to the set, and having only a global cache means that
everyone would have a ~/.fonts.cache-2 file. Right now, the local caches
are significantly more expensive to use as many additional stat(2) calls
are needed to validate the cache.

If we need a single global cache, it would need to live in /var
somewhere, which is something of a pain as there isn''t strong concensus
on what /var should look like these days. A quick read through the LSB
might provide some suggestions.

-keith

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20050725/88e2b3d1/attachment.pgp

>>>>> "Patrick" == Patrick Lam <plam@MIT.EDU>
writes:
Patrick> I wonder if we should manually pack the FcStaticPtr into,
Patrick> say, 24 bits for offset and 8 bits for bank. ...  The banked
Patrick> data structures enable per-directory caches.

Patrick> The only remaining issue is that FcObjectPtr is now an int,
Patrick> and we want to keep it that way.  We can probably do
Patrick> something clever there, maybe again the 24/8 split or
Patrick> something.

If either of those limit the number of fonts.cache-2 files to 256 it
will break.  Some users will tend to dump all of their fonts into a
single directory, but others will group them one dir per family or
similar.  Some will have CDs with hundreds of fonts and will copy
the CD wholesale and include the whole thing with a single entry
in fonts.conf.  Others will pick and choose.  Any split between
the number of cache files and the number of font files in each
cache file will be a problem if you want to pack it into only
32 bits -- even though 32 bits can enumerate many times more
fonts than anyone would have installed.

OTOH, if bank and dir are not 1::1, then:

      <gilda>Nevermind.</gilda>

:-)

-JimC
-- 
James H. Cloos, Jr. <cloos@jhcloos.com> <http://jhcloos.com>

Hi all,

I''ve just committed another revision to my branch.  This revision
includes support for reading and writing to an mmap file: if you run
fc-cache, then it creates this mmap file, and running fontconfig-using
programs uses the mmap file.  Here''s what valgrind fc-match --sort
tells me:

==22708== LEAK SUMMARY:
==22708==    definitely lost: 0 bytes in 0 blocks.
==22708==      possibly lost: 0 bytes in 0 blocks.
==22708==    still reachable: 17719 bytes in 904 blocks.
==22708==         suppressed: 0 bytes in 0 blocks.
==22708== Reachable blocks (those to which a pointer was found) are not
shown.
==22708== To see them, rerun with: --show-reachable=yes

(fc-list is even better, it loses 3200 bytes and has 2017 bytes still
reachable).

The normal one, by contrast, does this:

==22748===22748== LEAK SUMMARY:
==22748==    definitely lost: 0 bytes in 0 blocks.
==22748==      possibly lost: 0 bytes in 0 blocks.
==22748==    still reachable: 115649 bytes in 1525 blocks.
==22748==         suppressed: 0 bytes in 0 blocks.
==22748== Reachable blocks (those to which a pointer was found) are not
shown.
==22748== To see them, rerun with: --show-reachable=yes

Currently it writes the mmap file to /tmp, but this will change soon.  I
noticed that currently fontconfig writes cache files to directories
containing fonts, plus a global one on a per-user basis in
~/.fonts.cache.  Is there an appropriate place to put a global cache file?

pat