Keith Packard
2005-Nov-21 08:50 UTC
[Fontconfig] fontconfig performance problems + FcConfigUptoDate() bug
Around 18 o''clock on Jun 30, Lubos Lunak wrote:> A problem with this approach is that fontconfig doesn''t seem to notice font > changes while the app is already running - is this a bug or feature?The basic plan is to update the database whenever the application is discovering available fonts through the listing API. However, Qt (and Mozilla, at least) pre-caches the set of available fonts at startup, so these applications never see font changes after startup. Fixing those applications to check for font changes with FcConfigUptoDate is a reasonable solution here.> However FcConfigUptoDate() doesn''t seem to work. See the attached patch. > First there''s an obvious misplaced parenthesis making it return always false, > and second, even this call fails to detect font changes (e.g. adding a new > font to /usr/X11R6/lib/X11/fonts/truetype).Thanks! Not surprising that this code is horribly broken; it''s never been used...> May I suggest this information is simply dumped to one file and read-only > mmap()-ed into memory,Yeah, I''m thinking we''ll have to restructure the fonts.cache file format to be mmap''able. I don''t see a huge issue here; the cache file names are nicely versioned giving us cross-compatibility. I''ll release an updated 2.2 version shortly with your patch, as well as a new 2.3 development snapshot. -keith -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 228 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20040630/7c3a699d/attachment.pgp
Lubos Lunak
2005-Nov-21 08:50 UTC
[Fontconfig] fontconfig performance problems + FcConfigUptoDate() bug
Hello, the fontconfig.org front page says Fontconfig can ... efficiently and quickly find the fonts you need among the set of fonts you have installed, even if you have installed thousands of fonts, while minimzing memory usage. (Hmm, there''s a typo in minimizing BTW :) ). However, fontconfig on my machine is responsible for a significant portion of memory used by my KDE desktop. Have a look at the memory usage table attached to http://lists.kde.org/?l=kde-devel&m=108696118031357&w=2 (hint: it''s readable after you click on the tr3.txt link), left number is bytes allocated, right number is number of allocations (i.e. malloc() calls). Font are responsible for about 1/5 of memory allocated by a KDE application. For whole KDE desktop, this can easily be 10MiB for the plain desktop. This is SUSE LINUX 9.1, with: $ rpm -qa | grep font XFree86-fonts-75dpi-4.3.99.902-40 ghostscript-fonts-std-7.07.1rc1-190 free-ttf-fonts-1.0-156 ghostscript-fonts-other-7.07.1rc1-190 XFree86-fonts-scalable-4.3.99.902-40 $ fc-list | wc -l 562 $ find /usr/X11R6/lib/X11/fonts/ | wc -l 3092 (that''s not all font files indeed, but most of it is) Most of the fonts (~400) and files (~1900) come from the 75dpi package, which I think is installed by default. Let me first start with the FcConfigUptoDate() bug. We (=KDE) are considering adding a hack to kdeinit that will reduce the startup time and memory impact caused by fontconfig (kdeinit is a process which launches KDE apps by fork()-ing and dlopen()-ing the apps). The hack will be calling XftInit() + XftInitFtLibrary() before the fork(), so it will do fonts initialization including fontconfig in the parent, and because of copy-on-write kdeinit-launched apps should have smaller memory usage. A problem with this approach is that fontconfig doesn''t seem to notice font changes while the app is already running - is this a bug or feature? Either way, the hack includes a call to FcConfigUptoDate() and possible reinitialization before launching apps. For details see http://lists.kde.org/?l=kde-optimize&m=108861099022617&w=2 . However FcConfigUptoDate() doesn''t seem to work. See the attached patch. First there''s an obvious misplaced parenthesis making it return always false, and second, even this call fails to detect font changes (e.g. adding a new font to /usr/X11R6/lib/X11/fonts/truetype). The patch should fix that as well. The problem seems to be triggered by my fonts.conf specifying only /usr/X11R6/lib/X11/fonts , and therefore config->configDirs doesn''t include subdirs, unlike config->fontDirs. But that''s only a workaround of course. It''d be nice to have this fixed (or at least improved) for real in the lib. I have basically no knowledge about fontconfig or even fonts, so I''m not sure what exactly is causing this high memory usage, but I assume it''s fontconfig creating information about all fonts it finds. May I suggest this information is simply dumped to one file and read-only mmap()-ed into memory, just like KDE''s ksysoca? Changing fonts should be a rare case, so this should in general make both startup faster and use less memory, and if the fonts change, than the cache file can be regenerated and mmap()-ed again. Thank you PS: Attached is also $FC_DEBUG=512 kwrite >/tmp/fc.txt , that might be useful as well I suppose. -- Lubos Lunak KDE developer --------------------------------------------------------------------- SuSE CR, s.r.o. e-mail: l.lunak@suse.cz , l.lunak@kde.org Drahobejlova 27 tel: +420 2 9654 2373 190 00 Praha 9 fax: +420 2 9654 2374 Czech Republic http://www.suse.cz/ -------------- next part -------------- A non-text attachment was scrubbed... Name: fccfg.c.patch Type: text/x-diff Size: 591 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20040630/c2fcf013/fccfg.c.bin -------------- next part -------------- FC_DEBUG=512 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 2015 26554 2006 25804 9 750 charleaf 1 4080 0 0 1 4080 fontset 1 12 0 0 1 12 fontptr 9 5760 8 4608 1 1152 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 856 129826 285 4560 571 125266 patelt 572 109824 571 109568 1 256 vallist 5269 127518 4910 98200 359 29318 substate 0 0 0 0 0 0 string 1940 33090 1515 27826 425 5264 listbuck 0 0 0 0 0 0 strset 45 1872 0 0 45 1872 strlist 1 8 0 0 1 8 config 1 56 0 0 1 56 langset 577 16156 571 15988 6 168 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 0 0 1 164 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 14015 1048757 12020 872003 1995 176754 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 5 200 Double 12 480 String 311 27718 Bool 1 40 Matrix 0 0 CharSet 9 360 FTFace 0 0 LangSet 5 200 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 7669 804946 7628 798174 41 6772 charleaf 5 20400 0 0 5 20400 fontset 1 12 0 0 1 12 fontptr 15 15360 14 13440 1 1920 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 1369 208456 456 7296 913 201160 patelt 913 175232 912 175104 1 128 vallist 8490 204864 7887 157740 603 47124 substate 0 0 0 0 0 0 string 2626 45988 2201 40730 425 5258 listbuck 0 0 0 0 0 0 strset 45 1872 0 0 45 1872 strlist 1 8 0 0 1 8 config 1 56 0 0 1 56 langset 941 26348 912 25536 29 812 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 0 0 1 164 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 24804 2097543 22164 1803469 2640 294074 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 6 240 Double 20 800 String 495 43084 Bool 1 40 Matrix 0 0 CharSet 38 1520 FTFace 0 0 LangSet 29 1160 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 11528 1833602 11484 1823772 44 9830 charleaf 6 24480 0 0 6 24480 fontset 1 12 0 0 1 12 fontptr 15 15360 14 13440 1 1920 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 1402 213670 467 7472 935 206198 patelt 935 179456 934 179328 1 128 vallist 8705 209973 8085 161700 620 48273 substate 0 0 0 0 0 0 string 2670 46766 2245 41503 425 5263 listbuck 0 0 0 0 0 0 strset 45 1872 0 0 45 1872 strlist 1 8 0 0 1 8 config 1 56 0 0 1 56 langset 966 27048 934 26152 32 896 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 0 0 1 164 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 29003 3146304 26317 2838816 2686 307488 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 6 240 Double 20 800 String 506 43993 Bool 1 40 Matrix 0 0 CharSet 41 1640 FTFace 0 0 LangSet 32 1280 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 15586 2863474 15537 2849528 49 13946 charleaf 6 24480 0 0 6 24480 fontset 1 12 0 0 1 12 fontptr 15 15360 14 13440 1 1920 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 1441 219808 480 7680 961 212128 patelt 961 184448 960 184320 1 128 vallist 8963 216231 8318 166360 645 49871 substate 0 0 0 0 0 0 string 2722 47725 2297 42466 425 5259 listbuck 0 0 0 0 0 0 strset 45 1872 0 0 45 1872 strlist 1 8 0 0 1 8 config 1 56 0 0 1 56 langset 996 27888 960 26880 36 1008 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 0 0 1 164 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 33466 4195363 30720 3876123 2746 319240 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 6 240 Double 20 800 String 522 45231 Bool 1 40 Matrix 0 0 CharSet 46 1840 FTFace 0 0 LangSet 36 1440 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 20356 3831402 20298 3814180 58 17222 charleaf 8 32640 0 0 8 32640 fontset 1 12 0 0 1 12 fontptr 17 19584 16 17408 1 2176 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 1582 242086 527 8432 1055 233654 patelt 1055 202496 1054 202368 1 128 vallist 9878 238066 9164 183280 714 54786 substate 0 0 0 0 0 0 string 2910 51038 2485 45783 425 5255 listbuck 0 0 0 0 0 0 strset 45 1872 0 0 45 1872 strlist 1 8 0 0 1 8 config 1 56 0 0 1 56 langset 1096 30688 1054 29512 42 1176 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 0 0 1 164 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 39678 5243949 36752 4886412 2926 357537 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 6 240 Double 21 840 String 575 49506 Bool 1 40 Matrix 0 0 CharSet 55 2200 FTFace 0 0 LangSet 42 1680 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 29237 4250620 28863 4212182 374 38438 charleaf 15 61200 0 0 15 61200 fontset 1 12 0 0 1 12 fontptr 33 71808 32 67584 1 4224 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 3151 430147 1050 16800 2101 413347 patelt 1584 270976 1583 270848 1 128 vallist 18048 456718 16020 320400 2028 136318 substate 0 0 0 0 0 0 string 5061 92318 4633 86988 428 5330 listbuck 0 0 0 0 0 0 strset 121 2988 72 720 49 2268 strlist 20 160 19 152 1 8 config 1 56 0 0 1 56 langset 2207 61796 2100 58800 107 2996 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 0 0 1 164 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 62207 6292800 56526 5619923 5681 672877 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 29 1160 Double 21 840 String 1485 114858 Bool 2 80 Matrix 0 0 CharSet 371 14840 FTFace 0 0 LangSet 107 4280 Fc Memory Usage: Which Alloc Free Active count bytes count bytes count bytes charset 30349 4528868 29977 4490918 372 37950 charleaf 15 61200 0 0 15 61200 fontset 1 12 0 0 1 12 fontptr 33 71808 32 67584 1 4224 objectse 0 0 0 0 0 0 objectpt 0 0 0 0 0 0 matrix 2 64 1 32 1 32 pattern 3165 432288 1055 16880 2110 415408 patelt 1592 272512 1592 272512 0 0 vallist 18125 458709 16101 322020 2024 136689 substate 0 0 0 0 0 0 string 5096 93488 4672 88222 424 5266 listbuck 0 0 0 0 0 0 strset 125 3036 76 760 49 2276 strlist 21 168 21 168 0 0 config 1 56 0 0 1 56 langset 2217 62076 2110 59080 107 2996 atomic 0 0 0 0 0 0 blanks 3 396 1 128 2 268 cache 1 164 1 164 0 0 strbuf 544 557056 544 557056 0 0 subst 52 624 0 0 52 624 objectty 0 0 0 0 0 0 constant 0 0 0 0 0 0 test 104 2496 0 0 104 2496 expr 414 4968 0 0 414 4968 vstack 486 9720 486 9720 0 0 attr 561 561 561 561 0 0 pstack 561 17952 561 17952 0 0 Total 63468 6578222 57791 5903757 5677 674465 Fc Frozen Values: Type Count Bytes Void 0 0 Integer 30 1200 Double 21 840 String 1492 115409 Bool 2 80 Matrix 0 0 CharSet 372 14880 FTFace 0 0 LangSet 107 4280
Lubos Lunak
2005-Nov-21 08:50 UTC
[Fontconfig] fontconfig performance problems + FcConfigUptoDate() bug
On Wednesday 30 of June 2004 20:36, Keith Packard wrote:> Around 18 o''clock on Jun 30, Lubos Lunak wrote: > > May I suggest this information is simply dumped to one file and read-only > > mmap()-ed into memory, > > Yeah, I''m thinking we''ll have to restructure the fonts.cache file format > to be mmap''able. I don''t see a huge issue here; the cache file names are > nicely versioned giving us cross-compatibility. > > I''ll release an updated 2.2 version shortly with your patch, as well as a > new 2.3 development snapshot.Hello, I''d like to ask if there are any plans to do the mmap() thing in fontconfig in the foreseeable future. I just posted a patch[*] to xdg@ mailing list solving the XIM problem in a similar way, so I thought I''d check the status. I first thought I could do the same with fontconfig, but I don''t know fontconfig internals, and it seems to me this would be more difficult than XIM :( , so I''m unsure if I could get bored enough to handle this. Thanks [*] http://freedesktop.org/pipermail/xorg/2004-August/002235.html -- Lubos Lunak KDE developer --------------------------------------------------------------------- SuSE CR, s.r.o. e-mail: l.lunak@suse.cz , l.lunak@kde.org Drahobejlova 27 tel: +420 2 9654 2373 190 00 Praha 9 fax: +420 2 9654 2374 Czech Republic http://www.suse.cz/
Lubos Lunak
2005-Nov-21 08:50 UTC
[Fontconfig] fontconfig performance problems + FcConfigUptoDate() bug
On Monday 09 of August 2004 19:49, Keith Packard wrote:> Around 19 o''clock on Aug 9, Lubos Lunak wrote: > > I''d like to ask if there are any plans to do the mmap() thing in > > fontconfig in the foreseeable future. I just posted a patch[*] to xdg@ > > mailing list solving the XIM problem in a similar way, so I thought I''d > > check the status. > > There is a desire, but not any implementation underway. > > We''ll have to completely replace the existing cache file structure to > usefully benefit from an mmap-based system. Fortunately, this is possible > as fontconfig version-numbers the cache files. > > If you have an interest, I can point you in suitable directions.Honestly I''m not sure if I do :-/ , I have already enough KDE work. The data structures of fontconfig seem more complicated than the XIM data, and moreover if I comprehend it correctly this data can be modified at runtime in the app. That looks more like a complete rewrite of a part of fontconfig to me, and that''d be probably better be done by somebody who''s not completely clueless about it (hmm, does that include also somebody else than you?). Can you estimate the extent of work this would require? -- Lubos Lunak KDE developer --------------------------------------------------------------------- SuSE CR, s.r.o. e-mail: l.lunak@suse.cz , l.lunak@kde.org Drahobejlova 27 tel: +420 2 9654 2373 190 00 Praha 9 fax: +420 2 9654 2374 Czech Republic http://www.suse.cz/
Keith Packard
2005-Nov-21 08:50 UTC
[Fontconfig] fontconfig performance problems + FcConfigUptoDate() bug
Around 19 o''clock on Aug 9, Lubos Lunak wrote:> I''d like to ask if there are any plans to do the mmap() thing in fontconfig > in the foreseeable future. I just posted a patch[*] to xdg@ mailing list > solving the XIM problem in a similar way, so I thought I''d check the status.There is a desire, but not any implementation underway. We''ll have to completely replace the existing cache file structure to usefully benefit from an mmap-based system. Fortunately, this is possible as fontconfig version-numbers the cache files. If you have an interest, I can point you in suitable directions. -keith -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 228 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20040809/61039f0d/attachment.pgp
Keith Packard
2005-Nov-21 08:50 UTC
[Fontconfig] fontconfig performance problems + FcConfigUptoDate() bug
Around 9 o''clock on Aug 10, Lubos Lunak wrote:> and moreover if I comprehend it correctly this data can be modified atruntime in the app. No, the cache data is loaded at startup time and (for each directory at least) cannot change during the program run, except when reloading the cache to discover changes in the system.> That looks more like a complete rewrite of a part of > fontconfig to me, and that''d be probably better be done by somebody who''s > not completely clueless about it (hmm, does that include also somebody else > than you?). Can you estimate the extent of work this would require?Yes, it''s certainly a significant reworking of the fontconfig internals. I suspect it will require the ability for internal functions to handle two separate representations for the basic FcPattern datatype; one which contains a maze of pointers (the current structure) and one which uses only offsets or somesuch. That datatype is hidden from applications, so it should be possible to completely replace it if necessary, or at least extend it significantly without ABI/API changes. I suspect this will take a couple of weeks of occasional tinkering to figure out how the data structures should work and then probably a week or so of solid coding to get it implemented, followed by a couple of weeks of testing and fixing. But, these are just WAGs; I haven''t even started the design process... -keith -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 228 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20040810/c7b71618/attachment.pgp