Hi. I''ve recently stumbled upon problem with FAT filesystem updated by Windowoze (SD card in my case) and fontconfig cache. Problem boils down to the Windows not updating mtime of directory when file is added or removed from directory, so fontconfig does not rebuild cache as it checks only mtime and size of directory. I''d like to fix this problem and submit patch upstream, but first I''d like to ask for a correct way to fix it. Ideas came to my mind so far are: 1) scan directory and subdirectories and use max(mtimes, ctimes) of all files as mtime of directory. 2) add "hash" field to directory cache which is hash of all subdirectories'' hashes and directory contents. Also this check should be conditional to avoid any overhead on more conventional filesystems. is it a good idea to add boolean attribute for <dir> entry in config? -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091207/b20c3891/attachment.pgp
On 12/06/2009 07:34 PM, Mikhail Gusarov wrote:> Hi. > > I''ve recently stumbled upon problem with FAT filesystem updated by > Windowoze (SD card in my case) and fontconfig cache. Problem boils down > to the Windows not updating mtime of directory when file is added or > removed from directory, so fontconfig does not rebuild cache as it > checks only mtime and size of directory. > > I''d like to fix this problem and submit patch upstream, but first I''d > like to ask for a correct way to fix it.I let our win32 hackers respond before making a decision either way. Tor?> Ideas came to my mind so far are: > > 1) scan directory and subdirectories and use max(mtimes, ctimes) of all > files as mtime of directory.Would be a PITA.> 2) add "hash" field to directory cache which is hash of all > subdirectories'' hashes and directory contents.Isn''t option #1 superior to this? And doesn''t need a change to the cache format.> Also this check should be conditional to avoid any overhead on more > conventional filesystems.Sure.> is it a good idea to add boolean attribute for <dir> entry in config?No. Why? Please file a bug so we can track this. behdad
Twas brillig at 21:18:18 08.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: >> 1) scan directory and subdirectories and use max(mtimes, ctimes) of >> all files as mtime of directory. BE> Would be a PITA. >> 2) add "hash" field to directory cache which is hash of all >> subdirectories'' hashes and directory contents. BE> Isn''t option #1 superior to this? And doesn''t need a change to the BE> cache format. I''ve came up with the following scenario: dir/ foo.ttf (mtime=5,ctime=5) otherdir/ bar.ttf (mtime=4,ctime=4) mv otherdir/bar.ttf dir => dir/ foo.ttf (mtime=5,ctime=5) bar.ttf (mtime=4,ctime=4) First variant will fail to detect new font in dir/ in this case, as Windows does not change any timestamp on moved file. Also, it should be possible to reuse mtime field in cache file for a hash (32-bit one, as cryptographical sthrength is not necessary here). >> is it a good idea to add boolean attribute for <dir> entry in config? BE> No. Why? Something like <dir broken_mtime="true">/foo/bar</dir> Or it could be detected runtime, with something platform-dependent, like statfs(2). BE> Please file a bug so we can track this. Sure. #25535 -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091209/519df624/attachment.pgp
> I let our win32 hackers respond before making a decision either way. ?Tor?I just vaguely remember noticing before that looking at the mtime of a directory indeed is not necessarily a reliable way to see if its contents has changed on Windows. Unfortunately. So far I haven''t bothered doing anything to it, and just adviced people to run fc-cache manually if it seems that they are missing fonts.>> 1) scan directory and subdirectories and use max(mtimes, ctimes) of all >> ? ? files as mtime of directory. > > Would be a PITA.But might still be the only (?) solution. Would be inside #ifdef _WIN32 obviously. --tml
Twas brillig at 12:18:57 09.12.2009 UTC+02 when tml at iki.fi did gyre and gimble: >> Would be a PITA. TL> But might still be the only (?) solution. Would be inside #ifdef TL> _WIN32 obviously. Why??! 1) User puts fonts on SD card 2) User inserts SD card into Linux-based device 3) Fontconfig caches on device are screwed due to screwed timestamps -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091209/910e8a7d/attachment.pgp
> 1) User puts fonts on SD card > 2) User inserts SD card into Linux-based device > 3) Fontconfig caches on device are screwed due to screwed timestampsYeah, good point. So some platform-specific way to find out the file system type should then be used on non-Windows to determine whether to do it (and do it always on Windows)? --tml
Twas brillig at 13:37:36 09.12.2009 UTC+02 when tml at iki.fi did gyre and gimble: >> 1) User puts fonts on SD card >> 2) User inserts SD card into Linux-based device >> 3) Fontconfig caches on device are screwed due to screwed timestamps TL> Yeah, good point. So some platform-specific way to find out the TL> file system type should then be used on non-Windows to determine TL> whether to do it (and do it always on Windows)? Yep, like statfs(2) on Linux or BSD. Windows updates mtime timestamps on NTFS directories, so this workaround is not necessary there. -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091209/fa1f658b/attachment.pgp
On 12/09/2009 06:18 AM, Mikhail Gusarov wrote:> > Twas brillig at 12:18:57 09.12.2009 UTC+02 when tml at iki.fi did gyre and gimble: > > >> Would be a PITA. > > TL> But might still be the only (?) solution. Would be inside #ifdef > TL> _WIN32 obviously. > > Why??! > > 1) User puts fonts on SD card > 2) User inserts SD card into Linux-based device > 3) Fontconfig caches on device are screwed due to screwed timestampsBut why would someone cache fonts on an SD card from a Linux system? If the whole system is on the SD card, it''s definitely not FAT. I don''t think we have to handle that case. behdad
On 12/09/2009 04:57 AM, Mikhail Gusarov wrote:> > Twas brillig at 21:18:18 08.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: > > >> 1) scan directory and subdirectories and use max(mtimes, ctimes) of > >> all files as mtime of directory. > > BE> Would be a PITA. > > >> 2) add "hash" field to directory cache which is hash of all > >> subdirectories'' hashes and directory contents. > > BE> Isn''t option #1 superior to this? And doesn''t need a change to the > BE> cache format. > > I''ve came up with the following scenario: > > dir/ > foo.ttf (mtime=5,ctime=5) > > otherdir/ > bar.ttf (mtime=4,ctime=4) > > mv otherdir/bar.ttf dir => > > dir/ > foo.ttf (mtime=5,ctime=5) > bar.ttf (mtime=4,ctime=4) > > First variant will fail to detect new font in dir/ in this case, as > Windows does not change any timestamp on moved file. > > Also, it should be possible to reuse mtime field in cache file for a > hash (32-bit one, as cryptographical sthrength is not necessary here).Right. Since we only compare mtime for equality, we can simply use it as the directory hash. Thanks, behdad
Twas brillig at 13:12:44 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: BE> But why would someone cache fonts on an SD card from a Linux BE> system? Due to precisely same reason why there is ~/.fonts in typical fontconfig setup. BE> If the whole system is on the SD card, it''s definitely not FAT. No, it''s not. It''s the user-accessible part of device, like $HOME on PC. Thanks to one OS the only sane choice FS for SD cards is FAT. BE> I don''t think we have to handle that case. Then I will have to keep my patch outside of main tree. What a wonderful idea! -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091210/d3830cdc/attachment.pgp
On 12/09/2009 01:16 PM, Mikhail Gusarov wrote:> > Twas brillig at 13:12:44 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: > > BE> But why would someone cache fonts on an SD card from a Linux > BE> system? > > Due to precisely same reason why there is ~/.fonts in typical fontconfig > setup. > > BE> If the whole system is on the SD card, it''s definitely not FAT. > > No, it''s not. It''s the user-accessible part of device, like $HOME on > PC. Thanks to one OS the only sane choice FS for SD cards is FAT.Still not convinced.> BE> I don''t think we have to handle that case. > > Then I will have to keep my patch outside of main tree. What a wonderful > idea!I said I don''t think we *have* to. Not that I''m opposing it if it''s implemented generically using statfs for example. behdad
Twas brillig at 13:59:49 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: >> No, it''s not. It''s the user-accessible part of device, like $HOME on >> PC. Thanks to one OS the only sane choice FS for SD cards is FAT. BE> Still not convinced. Well, let me provide the specific use-case then. Dedicated e-book reader. Users want to use the fonts they posess (and they do!) to read books. There is SD card slot, USB connection and no network access (pretty typical setup for e-ink-based e-book readers). What are the options for installing fonts on such device? - SD card - USB mass storage Anything else? No, nothing beyond it is supported by Windowoze out-of-the-box. Sane filesystem on SD/USB mass storage? No, VFAT only [*] Is it possible to "install" fonts from SD somehow? Well, maybe. But then there should be dedicated space on device. Ugly and inefficient, as fonts may be quite huge, so there should be a lot of space reserved, so better avoid it. Just dropping a couple of .ttf to ".fonts" directory on SD card is easy to use and easy to explain. [*] Even Nokia in N900 went this way with their internal storage. As I was told by one of engineers they researched the various options and none was satisfactory. -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091210/b2f27a96/attachment.pgp
Thanks for the detailed info. I understand how it *may* be useful. My question was more about whether any system is using fontconfig in that way already? Cheers, behdad On 12/09/2009 02:16 PM, Mikhail Gusarov wrote:> > Twas brillig at 13:59:49 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: > > >> No, it''s not. It''s the user-accessible part of device, like $HOME on > >> PC. Thanks to one OS the only sane choice FS for SD cards is FAT. > > BE> Still not convinced. > > Well, let me provide the specific use-case then. > > Dedicated e-book reader. Users want to use the fonts they posess (and > they do!) to read books. There is SD card slot, USB connection and no > network access (pretty typical setup for e-ink-based e-book readers). > > What are the options for installing fonts on such device? > > - SD card > - USB mass storage > > Anything else? No, nothing beyond it is supported by Windowoze > out-of-the-box. Sane filesystem on SD/USB mass storage? No, VFAT only [*] > > Is it possible to "install" fonts from SD somehow? Well, maybe. But then > there should be dedicated space on device. Ugly and inefficient, as > fonts may be quite huge, so there should be a lot of space reserved, so > better avoid it. > > Just dropping a couple of .ttf to ".fonts" directory on SD card is easy > to use and easy to explain. > > [*] Even Nokia in N900 went this way with their internal storage. As I > was told by one of engineers they researched the various options and > none was satisfactory. > > > > > _______________________________________________ > Fontconfig mailing list > Fontconfig at lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/fontconfig
Twas brillig at 14:26:39 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: BE> Thanks for the detailed info. I understand how it *may* be useful. BE> My question was more about whether any system is using fontconfig BE> in that way already? Yes, the one I develop. That''s why I stumbled upon this bug in first place :) One of the targets of http://openinkpot.org/ Hanvon N516 to be precise. -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091210/6534487a/attachment.pgp
On 12/09/2009 02:29 PM, Mikhail Gusarov wrote:> > Twas brillig at 14:26:39 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: > > BE> Thanks for the detailed info. I understand how it *may* be useful. > BE> My question was more about whether any system is using fontconfig > BE> in that way already? > > Yes, the one I develop. That''s why I stumbled upon this bug in first > place :)Ok, now everything makes more sense :). Please submit the patch on bugzilla whenever you have it. behdad> One of the targets of http://openinkpot.org/ Hanvon N516 to be precise.
Twas brillig at 14:31:18 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: BE> Please submit the patch on bugzilla whenever you have it. Done. Comments are welcome. I have renamed ''mtime'' field in cache into ''checksum'' as it is no longer always ''mtime'', but reused struct stat''s st_mtime field to pass checksum around as it would be too much burden to restructure stuff. -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available Url : http://lists.freedesktop.org/archives/fontconfig/attachments/20091221/791c5d6f/attachment.pgp
On 12/20/2009 07:13 PM, Mikhail Gusarov wrote:> > Twas brillig at 14:31:18 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: > > BE> Please submit the patch on bugzilla whenever you have it. > > Done. Comments are welcome.Will check. Thanks. behdad> I have renamed ''mtime'' field in cache into ''checksum'' as it is no longer > always ''mtime'', but reused struct stat''s st_mtime field to pass checksum > around as it would be too much burden to restructure stuff. >
Twas brillig at 14:31:18 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: >> Yes, the one I develop. That''s why I stumbled upon this bug in first >> place :) BE> Ok, now everything makes more sense :). Please submit the patch on BE> bugzilla whenever you have it. *ping* Patch sits in Bugzilla, awaiting review. -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available URL: <http://lists.freedesktop.org/archives/fontconfig/attachments/20100301/9f387542/attachment.pgp>
On 03/01/2010 01:18 AM, Mikhail Gusarov wrote:> > Twas brillig at 14:31:18 09.12.2009 UTC-05 when behdad at behdad.org did gyre and gimble: > > >> Yes, the one I develop. That''s why I stumbled upon this bug in first > >> place :) > > BE> Ok, now everything makes more sense :). Please submit the patch on > BE> bugzilla whenever you have it. > > *ping* > > Patch sits in Bugzilla, awaiting review.Did you see the comment I left on the bug last week? http://bugs.freedesktop.org/show_bug.cgi?id=25535 behdad
Twas brillig at 17:31:12 03.03.2010 UTC-05 when behdad at behdad.org did gyre and gimble: BE> Did you see the comment I left on the bug last week? BE> http://bugs.freedesktop.org/show_bug.cgi?id=25535 Ah, no, missed it. As it is a discussion item, let''s discuss here. "This changes the cache format on FAT filesystems. Not a huge deal, but I''d rather you just emulate correct mtime instead of using a hash. That would require stating all the files in the directory and taking the max mtime I guess?" Nope. Imagine the following directory: foo/ a.txt (mtime 12345) b.txt (mtime 6789) Adding file c.txt with mtime 4444 will not change max(mtime). Deleting file b.txt will not change max(mtime). Windows does not change mtime while copying or moving files. I gave it a thought before writing a patch - directory mtime changes every time contents of directory changes, so the only substitute for mtime if it is not available is a checksum of directory contents, it can''t be reliable emulated by just looking at files inside. -- http://fossarchy.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 834 bytes Desc: not available URL: <http://lists.freedesktop.org/archives/fontconfig/attachments/20100304/481d1c9a/attachment.pgp>
On 03/03/2010 08:34 PM, Mikhail Gusarov wrote:> > Twas brillig at 17:31:12 03.03.2010 UTC-05 when behdad at behdad.org did gyre and gimble: > > BE> Did you see the comment I left on the bug last week? > BE> http://bugs.freedesktop.org/show_bug.cgi?id=25535 > > Ah, no, missed it. As it is a discussion item, let''s discuss here. > > "This changes the cache format on FAT filesystems. Not a huge deal, but > I''d rather you just emulate correct mtime instead of using a hash. That > would require stating all the files in the directory and taking the max > mtime I guess?" > > Nope. Imagine the following directory: > > foo/ > a.txt (mtime 12345) > b.txt (mtime 6789) > > Adding file c.txt with mtime 4444 will not change max(mtime). Deleting > file b.txt will not change max(mtime). Windows does not change mtime > while copying or moving files. > > I gave it a thought before writing a patch - directory mtime changes > every time contents of directory changes, so the only substitute for > mtime if it is not available is a checksum of directory contents, it > can''t be reliable emulated by just looking at files inside.I see. I''ll review and commit the patch as is then. Thanks, behdad