Fontconfig - Nov 2005 - [patch]: mmapping FontConfig data structures

This patch:

http://plam.csail.mit.edu/~plam/tmp/fontconfig-mmap-050609.diff

modifies fontconfig''s CVS HEAD to use an mmap()able file
containing internal fontconfig data structures.  The new binary
fc-mmap creates this map file (in /tmp, for now[1]) and fontconfig
clients use this map file instead of loading font data themselves.

On my system, my modified version of fontconfig mmaps 147786 bytes
into memory (instead of creating around 100k of dynamic data
structures).  I believe that this patch is stable; it produces no
differences in behaviour as compared to a stock fontconfig under
my tests.

The public API modified the patch is a pair of functions,
FcMmapForce() and FcMmapSave().  FcMmapForce(true) forces fontconfig
to ignore any mmaping data which might exist, and FcMmapSave() is used
in fc-mmap to write the data structures to disk.  Unlike an earlier version
of the patch, no other API modifications are necessary.

The structure of the patch is as follows.

1. Loading: I include a new file, src/fcmmap.c, which contains the
bulk of the logic for reading and writing the mmap file; some logic
lives in src/fclang.c (to serialize a data structure private to that
file).  During fontconfig''s initialization, it first calls
FcMmapLoadObjects immediately after opening the configuration file,
but before processing it, to load string constants into memory.  Later
on, fontconfig calls FcMmapLoadFonts to mmap the rest of the mmapped
data into memory, instead of loading the data from the font files.

2. Saving: I use a three-phase algorithm to save the data to disk.  In
the first phase (initiated by FcMmapSave''s call to
FcFontSetSerialize(config)), I move all font information reachable
from currentConfig.fonts to a set of static arrays (with a first
relocation from the dynamic addresses to the static arrays).  Next, I
write the data to the candidate mmap file.  The final phase is the
fixup; I use mmap() to read the data back into memory, recording the
addresses given to me by mmap() to, again, relocate internal pointers
as appropriate.  Once fixup is complete, the mmap file is ready to be
loaded with no further transformations.

Thanks to Sylvain Fourmanoit for hints on how to improve an earlier
version of this patch.

I would like this patch to be considered for inclusion in the fontconfig
CVS.

pat

[1] I would appreciate any suggestions on where the map file should be
created.  It is difficult to put the map file''s location in the
configuration file, because the map file is needed before fontconfig
loads the configuration information.

On Fri, 2005-06-10 at 13:50 -0400, Patrick Lam wrote:
> It so happens that I have also modified fontconfig to make most internal
> data structures offset-based rather than address-based, and it would be
> relatively straightforward.  However, this causes a large diff file, and
> can productively be done in another diff if you''d be willing to
commit
> this one, so that we separate the two sets of major changes from each
other.
I think separating these two changes is a good idea.
> I''m not currently familiar with the existing
''fc-cache'' program, but
> will look into it.  I also need to think a bit about how to do automatic
> discovery of new fonts, but it ought to be possible with a bit of work.
fc-cache constructs the fonts.cache-1 files from the internal
representation.  Note that fontconfig also stores cache information in
~/.fonts.cache-1 for fonts which aren''t stored in the per-directory
fonts.cache-1 files.  The ''-1'' in both
of these names is designed to version the cache information, when we
change the representation of the cache files, we''ll want to bump that
version number.

The ~/.fonts.cache file will also want to use this new mmap-able format.
> Going to an offset-based representation would make these files
> address-independent.  This is straightforward but tedious.
With your plan to do these changes in two stages, we will (at least) be
able to test this more invasive change separately.
> As with my earlier diff from two months back, the only
> externally-visible change would be FcPattern* -> FcPatternIdx.
We can''t have any externally visible changes as that would require
recoding every fontconfig using application...

I suggest that either we permit two separate representations within the
FcPattern datatype or we figure out how to use an offset-based
representation for dynamically created patterns.
> How would you like to proceed?  I could equally well cook up a patch
> which changes FcPattern* -> FcPatternIdx, etc and you could commit that
> first.  
I think this is a good first step
> This would be a large but straightforward patch, easy to review;
> from that version, I could fix the current patch to use offsets instead.
I''m not sure we want this intermediate state at all.
> Once we have the functionality in the current patch, then I could look
> into automatic updating and subsuming the current cache files.
The automatic updating of the cache files is already managed by
fontconfig, so all that we need to is change the representation of those
cache files.

-keith

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20050610/f24125dd/attachment.pgp

On Mon, 2005-06-13 at 23:26 -0400, Patrick Lam wrote:
> One thing that I might want to look into is eliminating duplication e.g.
> of FcLangSet and FcCharSet objects, for instance by canonicalizing them
> in a hash table or something.  Such a change would save space (which is
> now mmapped, but still.)
FcLangSet is a bitmap with one bit per supported language, or 183 bits
per font.  This may not be worth the effort to share.  FcCharSets were
shared in the past, but that sharing is global across all fonts read
into the system.  The data structures shared individual
''pages'' and then
(occasionally) entire collections of pages.  As identifying the
codepoints supported by every font is a common part of the matching
operation, having fewer pages referenced by the fonts would be a really
good thing.
> Another note is that the current version of fontconfig uses string
> addresses to compare and sort strings.  My patch computes hash codes
> instead of using string addresses, which are hard to use as a portable
> hash code
Hmm.  The use of string addresses was a result of some extensive
profiling done for fontconfig that showed string compares taking an
inordinate amount of time.  Many of these are for the element names, for
which a pointer comparison is awfully nice.  Given that all of the
elements in the cache files must be generated by fontconfig, we should
consider replacing the ''char *'' in the elements with an
integer indexing
a static table of names.  Sort that table and we can even do binary
searches within the patterns on just the integers.  Element names which
aren''t one of the standard names can be allocated from a separate pool
of values and used to index a table and compared using regular string
comparison functions.

For other strings, most comparisons must be done in a case-insensitive
fashion, and aren''t ''ordered'', so a hash function
which was computed in
a case insensitive fashion would actually be a huge advantage (hash not
match => strings not match).
> The remaining work is pretty simple; I just need to write fc-mmap.c using
> this new API, satisfying all of keithp''s constraints.
I don''t think we need another application; fc-cache should be able to
create the cache files using the existing library code.  We will then
deprecate the -1 version cache files and use only the -2 versions.

Are there any architecture dependencies in the cache file organization?
If so, we''ll want to provide for multiple architectures in the same
cache file.
> Please commit this patch.
I''m excited to see this come together so quickly; I''d like to
see some
code review before the patch lands, if there are others on the list with
the ability to help out there, it would be greatly appreciated.  I''ll
review the actual code soon, but more eyes are always better.

-keith

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20050613/b09deb53/attachment.pgp

Keith Packard wrote:
> Ok, now that I''m a bit more awake, here''s what
I''d *like* to see in an
> mmap-based patch for fontconfig:
> 
>  1)  The mmap-able files are a *replacement* for the existing
>      cache files.
> 
>  2)  Automatic discovery of new fonts still works, which
>      means timestamp checking of font directories at
>      application startup.
> 
>  3)  mmap''able files are address-independent
> 
>  4)  mmap''able files are architecture-independent
It so happens that I have also modified fontconfig to make most internal
data structures offset-based rather than address-based, and it would be
relatively straightforward.  However, this causes a large diff file, and
can productively be done in another diff if you''d be willing to commit
this one, so that we separate the two sets of major changes from each other.

I''m not currently familiar with the existing
''fc-cache'' program, but
will look into it.  I also need to think a bit about how to do automatic
discovery of new fonts, but it ought to be possible with a bit of work.
> 3) is necessary if these files are to replace the existing cache as
> being unable to map the cache files would be
> catastrophic in the absense of another level of caching.
Going to an offset-based representation would make these files
address-independent.  This is straightforward but tedious.
> 4) could be solved by making fc-cache store multiple
> versions of the cache information in the same file with a small header
> that provides a mapping into that data.
> Applications would read the header and mmap the appropriate section of
> the file.
Yes, this is quite easy to do with that hint.
> I assume that these changes would require a redesign of most of the
> fontconfig internal data structures to switch from pointers to offsets.
> I believe the necessary data structures are all hidden from the
> application so that this should be possible.
As with my earlier diff from two months back, the only
externally-visible change would be FcPattern* -> FcPatternIdx.

How would you like to proceed?  I could equally well cook up a patch
which changes FcPattern* -> FcPatternIdx, etc and you could commit that
first.  This would be a large but straightforward patch, easy to review;
from that version, I could fix the current patch to use offsets instead.
Once we have the functionality in the current patch, then I could look
into automatic updating and subsuming the current cache files.

pat

> 1)	What happens if you can''t mmap the cache files at the
> 	necessary address?
It just get ignored.
> 2)	Different users have different sets of font
> 	directories. Does this patch support that?
Yes, indirectly. What it does is that it takes a snapshot of the FcFontSet 
* fonts from the config; via the fc-mmap program, one can pass the -s flag 
to specify if one wants to take a snapshot excluding the user-part of 
the configuration or not. No definitive choice have been made up to now 
by Patrick on how the mmap files should be named or placed on disk, 
depending of their content.
> 3)	What about the new <selectfont> elements? These may
> 	also vary for different users.
I think answer for #2 holds, but I should check it out. I am writing a 
very similar patch, fixing a few issues I still have... I will post it 
here too. Regards,

-- 
Sylvain <syfou@users.sourceforge.net>

Never underestimate the bandwidth of a station wagon full of tapes.
 		-- Dr. Warren Jackson, Director, UTCS

Keith Packard wrote:
> I haven''t looked at the patch in detail, so some of these
questions
> may actually make incorrect assumptions.
> 
>  1)	What happens if you can''t mmap the cache files at the 
> 	necessary address?
If a mmap doesn''t go through (or it can''t find the file),
fontconfig
reverts to its default behaviour of loading the files directly.
>  2)	Different users have different sets of font
> 	directories. Does this patch support that?
The patch currently creates one mmap file per user, depending on the
user''s configuration.  I''ve just tested this, and it works
here.
>  3)	What about the new <selectfont> elements? These may
> 	also vary for different users.
I don''t know anything about <selectfont>, but since everything is
on a
per-user basis, I don''t see any problems.

It might be worthwhile to split the mmapped cache into a system-global
part and a per-user part; it shouldn''t even be very hard.  But I think
that even if we have to have one mmapped cache per user, it''s still a
win over the current status.

pat

James Cloos wrote:
> I''ve grabbed the (two) new versions and will try them.
> 
> But with the original version fc-mmap unlink(2)s the mmap file after
> generating it.  I upgraaded fc via gentoo''s emerge, so the
binaries
> are strip(1)ed, preventing real debugging for now, but this is the
> end of a trace:
Thanks for ths strace.  fc-mmap/fc-mmap unlinks in self-defense when it
gets an error code from a syscall, e.g. on your system, mmap() with a
requested length of 0 just fails.  I''ve wrapped that particular mmap in
a conditional in:

http://plam.csail.mit.edu/~plam/tmp/fontconfig-mmap-050610.diff

This version should work on your system.  Let me know what happens.

(I also made one other change, which prevents some unnecessary mmapping.
 Surprisingly, mmap can validly return 0x0 and it isn''t a failure.)

pat

James Cloos wrote:
> Another possibility is to replicate the idea of ~/.fonts.conf
> and ~/.fonts by using ~/.fonts.mmap.  
But only if fonts.mmap is platform independent.  Many places
use home directories NFS shared across different types of machines.
(big & little endian, 32 & 64 bit, etc.)

-- 
	-Alan Coopersmith-           alan.coopersmith@sun.com
	 Sun Microsystems, Inc. - X Window System Engineering

On Fri, 10 Jun 2005, Jeremy C. Reed wrote:
> Please make sure this doesn''t have race conditions when using /tmp
or
> TMPDIR.
>
> Maybe use O_EXCL (with open(2)) when creating, but that may not be good
> enough for some network-based filesystems.
/tmp is local on all known systems.
> And don''t use 666 for creating the file. If I overlooked or
misunderstood
> something, please let us know.
>
>  Jeremy C. Reed
--behdad
http://behdad.org/

On Fri, 10 Jun 2005, Patrick Lam wrote:
> Jeremy C. Reed wrote:
> > Please make sure this doesn''t have race conditions when using
/tmp or
> > TMPDIR.
> >
> > Maybe use O_EXCL (with open(2)) when creating, but that may not be
good
> > enough for some network-based filesystems.
> >
> > And don''t use 666 for creating the file. If I overlooked or
misunderstood
> > something, please let us know.
>
> I will use 600 rather than 666. Since this file is just hanging around
> anyway, it seems to be insufficient to avoid race conditions.  I think
> that if I could create a checksum (e.g. CRC32) of the expected state, as
> written, and verified the checksum before read, that ought to be
> sufficient; what do you think?
You better use O_EXCL anyway.  Just in case two copies of fc-mmap
happen to run in parallel, or something.

How is fc-mmap supposed to be called BTW?  Why not dump the cache
in the library if it doesn''t exist?

> Alan Coopersmith wrote:
> > But only if fonts.mmap is platform independent.  Many places
> > use home directories NFS shared across different types of machines.
> > (big & little endian, 32 & 64 bit, etc.)
>
> It''s hard to imagine how one could possibly make fonts.mmap
platform
> independent.  However, I have included the output of uname''s
sysname and
> machine fields in the filename to prevent horrible outcomes, and as an
> extra check, I verify endianness by writing 0x12345678 to the file and
> reading it again.
You should be fine as long as you do it in /tmp.
> pat
--behdad
http://behdad.org/

On Fri, 10 Jun 2005, Behdad Esfahbod wrote:
> On Fri, 10 Jun 2005, Patrick Lam wrote:
>
> > Jeremy C. Reed wrote:
> > > Please make sure this doesn''t have race conditions when
using /tmp or
> > > TMPDIR.
> > >
> > > Maybe use O_EXCL (with open(2)) when creating, but that may not
be good
> > > enough for some network-based filesystems.
> > >
> > > And don''t use 666 for creating the file. If I overlooked
or misunderstood
> > > something, please let us know.
> >
> > I will use 600 rather than 666. Since this file is just hanging around
> > anyway, it seems to be insufficient to avoid race conditions.  I think
> > that if I could create a checksum (e.g. CRC32) of the expected state,
as
> > written, and verified the checksum before read, that ought to be
> > sufficient; what do you think?
What do you mean by "just hanging around"?

I don''t know the code myself. And have not tested. Try manually making
a
symlink from the name of your file to be created and have it point to
another existing file ... and see what your code does to it. (Imagine
someone malicious making a symlink pointing to your password database and
waiting for someone to run this routine as root to cause a simple denial
of service.)
> You better use O_EXCL anyway.  Just in case two copies of fc-mmap
> happen to run in parallel, or something.
 Jeremy C. Reed

 	  	 	 BSD News, BSD tutorials, BSD links
	  	 	 http://www.bsdnewsletter.com/

On Fri, 10 Jun 2005, Patrick Lam wrote:
> Behdad Esfahbod wrote:
>
> > How is fc-mmap supposed to be called BTW?  Why not dump the cache
> > in the library if it doesn''t exist?
>
> Just run fc-mmap/fc-mmap (it has a few options, but you don''t need
any
> to successfully run.)  What do you mean by the library?
I mean, in a realworld scenario, the user is not supposed to run
fc-mmap manually, right?  Neither the admin should be required to
do that.  So my feeling is that although having fc-mmap around is
good, but there should be a configuration option in the
fonts.conf, that allow automatically creation of the cache.  Then
whenever the library is used (by an arbitrary application) and it
cannot find the mmapable cache, it tries to build it right then
and there.   There can be mechanisms to prevent it from trying
everytime, if for some reason, creation keeps failing though.

--behdad
http://behdad.org/

Patrick Lam wrote:
> This patch:
> 
> http://plam.csail.mit.edu/~plam/tmp/fontconfig-mmap-050609.diff
I''ve also uploaded

http://plam.csail.mit.edu/~plam/tmp/fontconfig-mmap-050609-efficient.diff

which is much more efficient at saving the mmap file to disk, in that
fc-mmap takes much less peak memory now.  I believe that it consumes the
same order of magnitude of memory as fontconfig does in normal operation.

pat

Patrick Lam wrote:
> Alan Coopersmith wrote:
> 
>>But only if fonts.mmap is platform independent.  Many places
>>use home directories NFS shared across different types of machines.
>>(big & little endian, 32 & 64 bit, etc.)
> 
> 
> It''s hard to imagine how one could possibly make fonts.mmap
platform
> independent.
Right, which is why a machine specific location such as /tmp or /var/tmp
is better than a user''s home directory.   (And even if you made it
platform independent, much of the fonts.mmap still depends on what
fonts/packages you''ve got installed on each machine, so is only really
sharable between truly identical machines.)

-- 
	-Alan Coopersmith-           alan.coopersmith@sun.com
	 Sun Microsystems, Inc. - X Window System Engineering

I''ve incorporated TMPDIR.  ~/.fonts.conf and ~/.fonts seem to be
defined
in the global fonts.conf file, not hardcoded, so I''ve avoided that for
now.  The new patch is at

http://plam.csail.mit.edu/~plam/tmp/fontconfig-050609-tmpdir.diff

pat

James Cloos wrote:
> Cool idea.
> 
> [SIGH.  I had my reply all ready to send, and then skimmed through the
> patch and discovered that much of it was unnecessary -- the code
> already does that.... :) ]
> 
> The environmental TMPDIR is often used to suggest a tmp directory,
> but fontconfig should probably also look for its own env.  Perhaps
> FC_TMPDIR first, then TMPDIR, then /tmp.  It already has at least
> FC_DEBUG, so FC_TMPDIR -- or FC_TMP -- is a reasonable extension.
> 
> Another possibility is to replicate the idea of ~/.fonts.conf
> and ~/.fonts by using ~/.fonts.mmap.  If that is not writeable
> then you''ll either have to fall back to TMPDIR and /tmp, or
> fall back to not using an mmap file.
> 
> -JimC

On Thu, 2005-06-09 at 22:51 -0400, Patrick Lam wrote:
> On my system, my modified version of fontconfig mmaps 147786 bytes
> into memory (instead of creating around 100k of dynamic data
> structures).  I believe that this patch is stable; it produces no
> differences in behaviour as compared to a stock fontconfig under
> my tests.
I haven''t looked at the patch in detail, so some of these questions
may actually make incorrect assumptions.

 1)	What happens if you can''t mmap the cache files at the 
	necessary address?
 2)	Different users have different sets of font
	directories. Does this patch support that?
 3)	What about the new <selectfont> elements? These may
	also vary for different users.

-keith

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20050609/cb526961/attachment.pgp

Please make sure this doesn''t have race conditions when using /tmp or
TMPDIR.

Maybe use O_EXCL (with open(2)) when creating, but that may not be good
enough for some network-based filesystems.

And don''t use 666 for creating the file. If I overlooked or
misunderstood
something, please let us know.

 Jeremy C. Reed

 	  	 	 technical support & remote administration
	  	 	 http://www.pugetsoundtechnology.com/

Jeremy C. Reed wrote:
> Please make sure this doesn''t have race conditions when using /tmp
or
> TMPDIR.
> 
> Maybe use O_EXCL (with open(2)) when creating, but that may not be good
> enough for some network-based filesystems.
> 
> And don''t use 666 for creating the file. If I overlooked or
misunderstood
> something, please let us know.
I will use 600 rather than 666. Since this file is just hanging around
anyway, it seems to be insufficient to avoid race conditions.  I think
that if I could create a checksum (e.g. CRC32) of the expected state, as
written, and verified the checksum before read, that ought to be
sufficient; what do you think?

Alan Coopersmith wrote:
> But only if fonts.mmap is platform independent.  Many places
> use home directories NFS shared across different types of machines.
> (big & little endian, 32 & 64 bit, etc.)
It''s hard to imagine how one could possibly make fonts.mmap platform
independent.  However, I have included the output of uname''s sysname
and
machine fields in the filename to prevent horrible outcomes, and as an
extra check, I verify endianness by writing 0x12345678 to the file and
reading it again.

pat

Ok, now that I''m a bit more awake, here''s what I''d
*like* to see in an
mmap-based patch for fontconfig:


 1)  The mmap-able files are a *replacement* for the existing
     cache files.

 2)  Automatic discovery of new fonts still works, which
     means timestamp checking of font directories at
     application startup.

 3)  mmap''able files are address-independent

 4)  mmap''able files are architecture-independent

1) means that the existing ''fc-cache'' program will generate
these files
and no new per-user action is required.

2) provides the same guarantee that fontconfig does today --
the user need simply drop new fonts in ~/.fonts and things "just
work".
The old X mkfontdir mess was a bad idea from
the very start.

3) is necessary if these files are to replace the existing cache as
being unable to map the cache files would be
catastrophic in the absense of another level of caching.

4) could be solved by making fc-cache store multiple
versions of the cache information in the same file with a small header
that provides a mapping into that data.
Applications would read the header and mmap the appropriate section of
the file.

I assume that these changes would require a redesign of most of the
fontconfig internal data structures to switch from pointers to offsets.
I believe the necessary data structures are all hidden from the
application so that this should be possible.

-keith

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20050610/8ab5eae8/attachment.pgp

Behdad Esfahbod wrote:
> I mean, in a realworld scenario, the user is not supposed to run
> fc-mmap manually, right?  Neither the admin should be required to
> do that.  So my feeling is that although having fc-mmap around is
> good, but there should be a configuration option in the
> fonts.conf, that allow automatically creation of the cache.  Then
> whenever the library is used (by an arbitrary application) and it
> cannot find the mmapable cache, it tries to build it right then
> and there.   There can be mechanisms to prevent it from trying
> everytime, if for some reason, creation keeps failing though.
Yes, it would indeed be better if the mmap file was created
automatically, and there could be a configuration option which could
prevent the mmap from being created.  Unfortunately, I can''t think of a
good way to prevent fontconfig from trying and failing, so I''d rather
not create the mmap file automatically until someone (possibly me) comes
up with a way to avoid continuous failure.  Ideally it would put
something in the local fonts.conf file, but fontconfig doesn''t have
infrastructure to modify config files itself.

Alternately, the user could run fc-mmap as part of the windowing
environment login script, which would probably work.

pat

On Fri, 10 Jun 2005, Patrick Lam wrote:
> Behdad Esfahbod wrote:
>
> Yes, it would indeed be better if the mmap file was created
> automatically, and there could be a configuration option which could
> prevent the mmap from being created.  Unfortunately, I can''t think
of a
> good way to prevent fontconfig from trying and failing, so I''d
rather
> not create the mmap file automatically until someone (possibly me) comes
> up with a way to avoid continuous failure.  Ideally it would put
> something in the local fonts.conf file, but fontconfig doesn''t
have
> infrastructure to modify config files itself.
1. If cache file exists and is not an empty file and is up to
   date, use it.  Otherwise:

2. Check whether user has write permission on the target cache
   file.  Give up if not.

3. If the cache file exists, but is an empty file, give up if the
last modification time of the emtpy file is less than 24 hours
ago.

4. Create an empty file as the cache.

5. Create the cache in a temporary file.

6. If successful, rename the cache to the final name.

> Alternately, the user could run fc-mmap as part of the windowing
> environment login script, which would probably work.
That works too, yes, but not on older systems that you be simply
running modern apps, not dekstop.
> pat
--behdad
http://behdad.org/

BTW, here''s something I found that appears to be obviously wrong.

pat

Index: src/fccfg.c
==================================================================RCS file:
/cvs/fontconfig/fontconfig/src/fccfg.c,v
retrieving revision 1.48
diff -u -r1.48 fccfg.c
--- src/fccfg.c 9 Mar 2005 00:47:11 -0000       1.48
+++ src/fccfg.c 11 Jun 2005 03:12:39 -0000
@@ -629,9 +629,11 @@
                ret = FcStrStrIgnoreCase (left.u.s, right.u.s) != 0;
                break;
            case FcOpNotEqual:
-           case FcOpNotContains:
                ret = FcStrCmpIgnoreCase (left.u.s, right.u.s) != 0;
                break;
+           case FcOpNotContains:
+               ret = FcStrStrIgnoreCase (left.u.s, right.u.s) == 0;
+               break;
            default:
                break;
            }

My new patch is at,

http://plam.csail.mit.edu/~plam/tmp/fontconfig-serialize-diffs-050613.diff

This patch allows the fundamental fontconfig data structures to be
serialized.  I''ve converted everything from FcPattern down to be able
to use *Ptr objects, which can be either static or dynamic (using a
union which either contains a pointer or an index) and replaced
storage of pointers in the heap with the appropriate *Ptr object.  I
then changed all writes of pointers to the heap with a *CreateDynamic
call, which creates a dynamic Ptr object pointing to the same object
as before.  This way, the fundamental fontconfig semantics should be
unchanged; I did not have to change external signatures this way,
although I did change some internal signatures.  When given a *Ptr
object, just run *U to get back to a normal pointer; it gives the
right answer regardless of whether we''re using static or dynamic
storage.

I''ve also implemented a Fc*Serialize call.  Calling FcFontSetSerialize
converts the dynamic FcFontSets contained in the config object to
static FcFontSets and also converts its dependencies (e.g. everything
you''d need to write to disk) to static objects.  Note that you have
to call Fc*PrepareSerialize first; this call will count the number
of objects that actually needs to be allocated, so that we can avoid
realloc.  The Fc*Serialize calls then check the static pointers for
nullness, and allocate the buffers if necessary.  I''ve tested the
execution of fc-list and fc-match after Fc*Serialize and they appear
to work the same way.

One thing that I might want to look into is eliminating duplication e.g.
of FcLangSet and FcCharSet objects, for instance by canonicalizing them
in a hash table or something.  Such a change would save space (which is
now mmapped, but still.)

Another note is that the current version of fontconfig uses string
addresses to compare and sort strings.  My patch computes hash codes
instead of using string addresses, which are hard to use as a portable
hash code.

The remaining work is pretty simple; I just need to write fc-mmap.c using
this new API, satisfying all of keithp''s constraints.

Please commit this patch.

pat

I''ve grabbed the (two) new versions and will try them.

But with the original version fc-mmap unlink(2)s the mmap file after
generating it.  I upgraaded fc via gentoo''s emerge, so the binaries
are strip(1)ed, preventing real debugging for now, but this is the
end of a trace:

close(3)                                = 0
open("/tmp/fontconfig-mmap-Linux-i686-cloos-:0.0", O_RDWR) = 3
lseek(3, 0, SEEK_SET)                   = 0
read(3, "xV4\22\0 \0\0\0\200\0\0\245\23\0\0\0\0\0\0\0\300\1\0L`"...,
136) = 136
mmap2(NULL, 450589, PROT_READ, MAP_SHARED, 3, 0x550) = 0xb78d9000
mmap2(0xb78d9000, 450589, PROT_READ, MAP_SHARED|MAP_FIXED, 3, 0x550) =
0xb78d9000
mmap2(NULL, 80480, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x2f0) = 0xb7f47000
mmap2(NULL, 2003328, PROT_READ, MAP_SHARED, 3, 0x304) = 0xb72a6000
mmap2(NULL, 250416, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x4ee) = 0xb789b000
mmap2(NULL, 125208, PROT_READ, MAP_SHARED, 3, 0x52c) = 0xb787c000
mmap2(NULL, 0, PROT_READ, MAP_SHARED, 3, 0x54c) = -1 EINVAL (Invalid argument)
unlink("/tmp/fontconfig-mmap-Linux-i686-cloos-:0.0") = 0
close(3)                                = 0
exit_group(0)                           = ?

Better info to follow soon.

-JimC

Cool idea.

[SIGH.  I had my reply all ready to send, and then skimmed through the
patch and discovered that much of it was unnecessary -- the code
already does that.... :) ]

The environmental TMPDIR is often used to suggest a tmp directory,
but fontconfig should probably also look for its own env.  Perhaps
FC_TMPDIR first, then TMPDIR, then /tmp.  It already has at least
FC_DEBUG, so FC_TMPDIR -- or FC_TMP -- is a reasonable extension.

Another possibility is to replicate the idea of ~/.fonts.conf
and ~/.fonts by using ~/.fonts.mmap.  If that is not writeable
then you''ll either have to fall back to TMPDIR and /tmp, or
fall back to not using an mmap file.

-JimC
-- 
James H. Cloos, Jr. <cloos@jhcloos.com>