Fontconfig - Mar 2006 - Format of cache file are different on 32bit and 64bit system.

Hi,
  Recently I found that format of fonts.cache-2 file are different on
32bit and 64bit system. It leads big performance issue for 32bit
applications on 32bit/64bit mixed system, like x86_64. Because 32bit
applications must re-create local cache file on startup, almost every
time.
  For more information, see https://bugzilla.novell.com/show_bug.cgi?id=160226

  Is it possible to unify the cache file format for both 32bit and 64bit system?

Regards
James Su

Zhe Su wrote:
> Hi,
>   Recently I found that format of fonts.cache-2 file are different on
> 32bit and 64bit system. It leads big performance issue for 32bit
> applications on 32bit/64bit mixed system, like x86_64. Because 32bit
> applications must re-create local cache file on startup, almost every
> time.
>   For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
> 
>   Is it possible to unify the cache file format for both 32bit and 64bit
system?
That bug is not world readable.

I designed the cache format so that it would contain caches for both 
32bit and 64bit systems in the same file.  It may not work; I''ll look 
into it tomorrow.

Fundamentally it''s impossible to use the same mmapped structures for 
both the 32bit and 64bit architectures, so in principle there should be 
two copies of the data in the cache file.

pat

Patrick Lam <plam@MIT.EDU> ????????:
> Zhe Su wrote:
>> Hi,
>>   Recently I found that format of fonts.cache-2 file are different on
>> 32bit and 64bit system. It leads big performance issue for 32bit
>> applications on 32bit/64bit mixed system, like x86_64. Because 32bit
>> applications must re-create local cache file on startup, almost every
>> time.
>>   For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
>>   Is it possible to unify the cache file format for both 32bit and
>> 64bit system?
>
> That bug is not world readable.
Really? I think it is world readable. Can you please try again?

-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
?????????????

On Thu, 23 Mar 2006, Patrick Lam wrote:
> Zhe Su wrote:
> > Hi,
> >   Recently I found that format of fonts.cache-2 file are different on
> > 32bit and 64bit system. It leads big performance issue for 32bit
> > applications on 32bit/64bit mixed system, like x86_64. Because 32bit
> > applications must re-create local cache file on startup, almost every
> > time.
> >   For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
> >
> >   Is it possible to unify the cache file format for both 32bit and
64bit system?
>
> That bug is not world readable.
Well, when Novell bugzilla says Access Denied, it means you need
to create an account and login!  And man, you have to fill each
and every field in their account creation form.


> I designed the cache format so that it would contain caches for both
> 32bit and 64bit systems in the same file.  It may not work; I''ll
look
> into it tomorrow.
>
> Fundamentally it''s impossible to use the same mmapped structures
for
> both the 32bit and 64bit architectures, so in principle there should be
> two copies of the data in the cache file.
>
> pat
--behdad
http://behdad.org/

"Commandment Three says Do Not Kill, Amendment Two says Blood Will
Spill"
	-- Dan Bern, "New American Language"

Zhe Su wrote:
> Hi,
>   I don''t know how to make it publicly accessible. You need novell
> bugzilla account to view the bug report.
>   I''m wondering how to generate both 32bit and 64bit cache in one
file
> by a 64bit fc-cache. Because we only ship 64bit fc-cache on such
> system, though both 32bit and 64bit libfontconfig are available.
That bug is now world readable.

I don''t know how the 64 bit fc-cache could possibly generate a 32 bit 
.fonts.cache, since it just doesn''t know what the memory layout is
going
to be like on the 32 bit arch.  Because of this, it seems like the best 
solution is what used to work and regressed: if you have an exclusively 
64 bit fonts.cache-2, then you generate a 32-bit .fonts.cache-2.  That 
should be a simple bug that I should be able to fix.

Stanislav Brabec is not well-informed.  Having a local .fonts.cache-2 in 
no way prevents fontconfig from using ~/.fonts.conf and from caching 
local fonts.  Also, please do not patch your fontconfig to generate 
.fonts.cache-2-64 and .fonts.cache-2-32.  That would be completely wrong.


pat

On Thursday, 23. March 2006 15:03, Patrick Lam wrote:
> local fonts.  Also, please do not patch your fontconfig to generate
> .fonts.cache-2-64 and .fonts.cache-2-32.  That would be completely wrong.
Don''t worry, thats not gonna happen :)


Dirk

Behdad Esfahbod <behdad@cs.toronto.edu> ????????:
> On Thu, 23 Mar 2006, Patrick Lam wrote:
>
>> Zhe Su wrote:
>> > Hi,
>> >   Recently I found that format of fonts.cache-2 file are different
on
>> > 32bit and 64bit system. It leads big performance issue for 32bit
>> > applications on 32bit/64bit mixed system, like x86_64. Because
32bit
>> > applications must re-create local cache file on startup, almost
every
>> > time.
>> >   For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
>> >
>> >   Is it possible to unify the cache file format for both 32bit and
64bit system?
>>
>> That bug is not world readable.
>
> Well, when Novell bugzilla says Access Denied, it means you need
> to create an account and login!
You should be able to view most bugs *readonly* even without creating
an account.
> And man, you have to fill each
> and every field in their account creation form.
It''s easier if you create your account on http://www.opensuse.org.

There is no difference between an account created on
http://bugzilla.novell.com
and on http://www.opensuse.org but if you create the account
on the latter site, it is easier and you need to fill out
less fields.

-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
?????????????

Patrick Lam <plam@MIT.EDU> ????????:
> Zhe Su wrote:
>> Hi,
>>   Recently I found that format of fonts.cache-2 file are different on
>> 32bit and 64bit system. It leads big performance issue for 32bit
>> applications on 32bit/64bit mixed system, like x86_64. Because 32bit
>> applications must re-create local cache file on startup, almost every
>> time.
>>   For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
>>   Is it possible to unify the cache file format for both 32bit and
>> 64bit system?
>
> That bug is not world readable.
>
> I designed the cache format so that it would contain caches for both
> 32bit and 64bit systems in the same file.  It may not work; I''ll
look
> into it tomorrow.
I think I have fixed it, see also

    http://bugzilla.novell.com/show_bug.cgi?id=160226

-------------- next part --------------
A non-text attachment was scrubbed...
Name: bugzilla-160226-multiarch-cache-file-problem.patch
Type: text/x-patch
Size: 762 bytes
Desc: not available
Url :
http://lists.freedesktop.org/archives/fontconfig/attachments/20060324/18e1cc65/bugzilla-160226-multiarch-cache-file-problem.bin
-------------- next part --------------


-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
?????????????

Mike FABIAN wrote:
> Patrick Lam <plam@MIT.EDU> ????????:
> 
> 
>>Zhe Su wrote:
>>
>>>Hi,
>>>  Recently I found that format of fonts.cache-2 file are different
on
>>>32bit and 64bit system. It leads big performance issue for 32bit
>>>applications on 32bit/64bit mixed system, like x86_64. Because 32bit
>>>applications must re-create local cache file on startup, almost
every
>>>time.
>>>  For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
>>>  Is it possible to unify the cache file format for both 32bit and
>>>64bit system?
>>
>>That bug is not world readable.
>>
>>I designed the cache format so that it would contain caches for both
>>32bit and 64bit systems in the same file.  It may not work;
I''ll look
>>into it tomorrow.
> 
> 
> I think I have fixed it, see also
> 
>     http://bugzilla.novell.com/show_bug.cgi?id=160226
fc-cache32 would be a fine solution to the dir caches problem, if you 
can generate that binary.

Your patch makes things better but doesn''t completely fix the problem 
yet.  I''m looking into it.

The system I''m on here is amd64 with an ia32 chroot.  /home is shared 
between the amd64 install and the chroot.

1. remove a fonts.cache-2 file (dir cache) from both the install and the 
chroot
2. fc-list in the install
3. fc-list in the chroot
4. fc-cat ~/.fonts.cache-2 in the install

You''ll see that the install''s .fonts.cache-2 gets trampled by
the
chroot''s .fonts.cache-2.

Note also that the line you patched was a fix to ''severe perf
regression
when global fonts.cache-2 files are missing'', which you and I tested 
back in October 2005.  I''m going to test that behaviour as well (put 
junk in .fonts.cache-2, see what happens).

pat

Patrick Lam <plam@MIT.EDU> ????????:
> Mike FABIAN wrote:
>> Patrick Lam <plam@MIT.EDU> ????????:
>>
>>>Zhe Su wrote:
>>>
>>>>Hi,
>>>>  Recently I found that format of fonts.cache-2 file are
different on
>>>>32bit and 64bit system. It leads big performance issue for 32bit
>>>>applications on 32bit/64bit mixed system, like x86_64. Because
32bit
>>>>applications must re-create local cache file on startup, almost
every
>>>>time.
>>>>  For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
>>>>  Is it possible to unify the cache file format for both 32bit
and
>>>>64bit system?
>>>
>>>That bug is not world readable.
>>>
>>>I designed the cache format so that it would contain caches for both
>>>32bit and 64bit systems in the same file.  It may not work;
I''ll look
>>>into it tomorrow.
>> I think I have fixed it, see also
>>     http://bugzilla.novell.com/show_bug.cgi?id=160226
>
> fc-cache32 would be a fine solution to the dir caches problem, if you
> can generate that binary.
>
> Your patch makes things better but doesn''t completely fix the
problem
> yet.  I''m looking into it.
>
> The system I''m on here is amd64 with an ia32 chroot.  /home is
shared
> between the amd64 install and the chroot.
>
> 1. remove a fonts.cache-2 file (dir cache) from both the install and
> the chroot
> 2. fc-list in the install
> 3. fc-list in the chroot
> 4. fc-cat ~/.fonts.cache-2 in the install
>
> You''ll see that the install''s .fonts.cache-2 gets
trampled by the
> chroot''s .fonts.cache-2.
I cannot reproduce this. It works fine for me.  I am running 32bit and
64bit applications using fontconfig alternatetly and they don''t
trample each others data in ~/.fonts.cache-2.
> Note also that the line you patched was a fix to ''severe perf
> regression when global fonts.cache-2 files are missing'', which you
and
> I tested back in October 2005.  I''m going to test that behaviour
as
> well (put junk in .fonts.cache-2, see what happens).
If ~/.fonts.cache-2 is "missing", it doesn''t have to be
deleted.

Deleting it when the current architecture cannot be found
cannot be correct in my opinion.

-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
?????????????

Patrick Lam <plam@MIT.EDU> ????????:
> Mike FABIAN wrote:
>> Patrick Lam <plam@MIT.EDU> ????????:
>>
>>>Zhe Su wrote:
>>>
>>>>Hi,
>>>>  Recently I found that format of fonts.cache-2 file are
different on
>>>>32bit and 64bit system. It leads big performance issue for 32bit
>>>>applications on 32bit/64bit mixed system, like x86_64. Because
32bit
>>>>applications must re-create local cache file on startup, almost
every
>>>>time.
>>>>  For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
>>>>  Is it possible to unify the cache file format for both 32bit
and
>>>>64bit system?
>>>
>>>That bug is not world readable.
>>>
>>>I designed the cache format so that it would contain caches for both
>>>32bit and 64bit systems in the same file.  It may not work;
I''ll look
>>>into it tomorrow.
>> I think I have fixed it, see also
>>     http://bugzilla.novell.com/show_bug.cgi?id=160226
>
> fc-cache32 would be a fine solution to the dir caches problem, if you
> can generate that binary.
Yes, that is rather easy.

-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
?????????????

Mike FABIAN wrote:
>>The system I''m on here is amd64 with an ia32 chroot.  /home is
shared
>>between the amd64 install and the chroot.
>>
>>1. remove a fonts.cache-2 file (dir cache) from both the install and
>>the chroot
>>2. fc-list in the install
>>3. fc-list in the chroot
>>4. fc-cat ~/.fonts.cache-2 in the install
>>
>>You''ll see that the install''s .fonts.cache-2 gets
trampled by the
>>chroot''s .fonts.cache-2.
> 
> 
> I cannot reproduce this. It works fine for me.  I am running 32bit and
> 64bit applications using fontconfig alternatetly and they don''t
> trample each others data in ~/.fonts.cache-2.
In that case, it could just be the computer I was using or some aspect
of its configuration.  I''ll try to look into it a bit more.
>>Note also that the line you patched was a fix to ''severe perf
>>regression when global fonts.cache-2 files are missing'', which
you and
>>I tested back in October 2005.  I''m going to test that
behaviour as
>>well (put junk in .fonts.cache-2, see what happens).
> 
> If ~/.fonts.cache-2 is "missing", it doesn''t have to be
deleted.
> 
> Deleting it when the current architecture cannot be found
> cannot be correct in my opinion.
Yes, that''s clearly wrong.  The problem was a related case, when it
couldn''t make sense of the cache file and just did weird things.

pat

On Thu, 2006-03-23 at 01:24 -0500, Patrick Lam wrote:
> Zhe Su wrote:
> > Hi,
> >   Recently I found that format of fonts.cache-2 file are different on
> > 32bit and 64bit system. It leads big performance issue for 32bit
> > applications on 32bit/64bit mixed system, like x86_64. Because 32bit
> > applications must re-create local cache file on startup, almost every
> > time.
> >   For more information, see
https://bugzilla.novell.com/show_bug.cgi?id=160226
> > 
> >   Is it possible to unify the cache file format for both 32bit and
64bit system?
> 
> That bug is not world readable.
> 
> I designed the cache format so that it would contain caches for both 
> 32bit and 64bit systems in the same file.  It may not work; I''ll
look
> into it tomorrow.
> 
> Fundamentally it''s impossible to use the same mmapped structures
for
> both the 32bit and 64bit architectures, so in principle there should be 
> two copies of the data in the cache file.
While I''m not going to encourage changing the file format, at this
point :-), the "fundamentally it''s impossible" statement
strikes me as
incorrect ... the GTK+ memmapped cache file formats, are, for example,
architecture independent.

If you rely on simply dumping native structures onto disk, how do you
deal with differences in architecture and endianess for, say, cache
files in a NFS mounted home directory? Or are there copies for 
every architecture in the file, not just 32/64 bit?

						Owen

Owen Taylor wrote:
>> Fundamentally it''s impossible to use the same mmapped
structures for
>> both the 32bit and 64bit architectures, so in principle there should be
>> two copies of the data in the cache file.
> 
> While I''m not going to encourage changing the file format, at this
> point :-), the "fundamentally it''s impossible" statement
strikes me as
> incorrect ... the GTK+ memmapped cache file formats, are, for example,
> architecture independent.
It might not, strictly speaking, be completely impossible, but it''s
pretty difficult to do so, since there are a lot more interwoven
pointers within the fontconfig data structures than within the GTK+ data
structures.
> If you rely on simply dumping native structures onto disk, how do you
> deal with differences in architecture and endianess for, say, cache
> files in a NFS mounted home directory? Or are there copies for 
> every architecture in the file, not just 32/64 bit?
I generate a signature of each arch, which contains the sizes of all of
the relevant data structures and the arch''s endianness.  When
fontconfig
is run on a new arch, it generates caches for that arch; if you run
fc-cache, then it generates per-directory caches in
/var/cache/fontconfig, and if you just run a fontconfig client, it
updates ~/.fonts.cache.  Both the global and per-directory cache can
contain arbitrary numbers of arches.

pat