On Wed, Aug 12, 2009 at 11:53 AM, Damjan Perenic<damjan.perenic at guest.arnes.si> wrote:> On Tue, Aug 11, 2009 at 11:04 PM, Richard > Elling<richard.elling at gmail.com> wrote: >> On Aug 11, 2009, at 7:39 AM, Ed Spencer wrote: >> >>> I suspect that if we ''rsync'' one of these filesystems to a second >>> server/pool ?that we would also see a performance increase equal to what >>> we see on the development server. (I don''t know how zfs send a receive >>> work so I don''t know if it would address this "Filesystem Entropy" or >>> specifically reorganize the files and directories). However, when we >>> created a testfs filesystem in the zfs pool on the production server, >>> and copied data to it, we saw the same performance as the other >>> filesystems, in the same pool. >> >> Directory walkers, like NetBackup or rsync, will not scale well as >> the number of files increases. ?It doesn''t matter what file system you >> use, the scalability will look more-or-less similar. For millions of files, >> ZFS send/receive works much better. ?More details are in my paper. > > It would be nice if ZFS had something similar to VxFS File Change Log. > This feature is very useful for incremental backups and other > directory walkers, providing they support FCL.I think this tangent deserves its own thread. :) To save a trip to google... http://sfdoccentral.symantec.com/sf/5.0MP3/linux/manpages/vxfs/man1m/fcladm.html This functionality would come in very handy. It would seem that it isn''t too big of a deal to identify the files that changed, as this type of data is already presented via "zpool status -v" when corruption is detected. http://docs.sun.com/app/docs/doc/819-5461/gbctx?a=view -- Mike Gerdts http://mgerdts.blogspot.com/
On Wed, Aug 12, 2009 at 2:15 PM, Mike Gerdts<mgerdts at gmail.com> wrote:> On Wed, Aug 12, 2009 at 11:53 AM, Damjan > Perenic<damjan.perenic at guest.arnes.si> wrote: >> On Tue, Aug 11, 2009 at 11:04 PM, Richard >> Elling<richard.elling at gmail.com> wrote: >>> On Aug 11, 2009, at 7:39 AM, Ed Spencer wrote: >>> >>>> I suspect that if we ''rsync'' one of these filesystems to a second >>>> server/pool ?that we would also see a performance increase equal to what >>>> we see on the development server. (I don''t know how zfs send a receive >>>> work so I don''t know if it would address this "Filesystem Entropy" or >>>> specifically reorganize the files and directories). However, when we >>>> created a testfs filesystem in the zfs pool on the production server, >>>> and copied data to it, we saw the same performance as the other >>>> filesystems, in the same pool. >>> >>> Directory walkers, like NetBackup or rsync, will not scale well as >>> the number of files increases. ?It doesn''t matter what file system you >>> use, the scalability will look more-or-less similar. For millions of files, >>> ZFS send/receive works much better. ?More details are in my paper. >> >> It would be nice if ZFS had something similar to VxFS File Change Log. >> This feature is very useful for incremental backups and other >> directory walkers, providing they support FCL. > > I think this tangent deserves its own thread. ?:) > > To save a trip to google... > > http://sfdoccentral.symantec.com/sf/5.0MP3/linux/manpages/vxfs/man1m/fcladm.html > > This functionality would come in very handy. ?It would seem that it > isn''t too big of a deal to identify the files that changed, as this > type of data is already presented via "zpool status -v" when > corruption is detected. > > http://docs.sun.com/app/docs/doc/819-5461/gbctx?a=viewIn fact ZFS has a good transaction log, maybe the issue is there isn''t software out there yet that uses it. -Ross
Mattias Pantzare
2009-Aug-12 22:49 UTC
[zfs-discuss] file change long - was zfs fragmentation
>>> It would be nice if ZFS had something similar to VxFS File Change Log. >>> This feature is very useful for incremental backups and other >>> directory walkers, providing they support FCL. >> >> I think this tangent deserves its own thread. ?:) >> >> To save a trip to google... >> >> http://sfdoccentral.symantec.com/sf/5.0MP3/linux/manpages/vxfs/man1m/fcladm.html >> >> This functionality would come in very handy. ?It would seem that it >> isn''t too big of a deal to identify the files that changed, as this >> type of data is already presented via "zpool status -v" when >> corruption is detected. >> >> http://docs.sun.com/app/docs/doc/819-5461/gbctx?a=view > > In fact ZFS has a good transaction log, maybe the issue is there isn''t > software out there yet that uses it.Where is that log? ZIL does not log all transactions and is cleared very quickly.
On Wed, Aug 12, 2009 at 6:49 PM, Mattias Pantzare<pantzer at ludd.ltu.se> wrote:>>>> It would be nice if ZFS had something similar to VxFS File Change Log. >>>> This feature is very useful for incremental backups and other >>>> directory walkers, providing they support FCL. >>> >>> I think this tangent deserves its own thread. ?:) >>> >>> To save a trip to google... >>> >>> http://sfdoccentral.symantec.com/sf/5.0MP3/linux/manpages/vxfs/man1m/fcladm.html >>> >>> This functionality would come in very handy. ?It would seem that it >>> isn''t too big of a deal to identify the files that changed, as this >>> type of data is already presented via "zpool status -v" when >>> corruption is detected. >>> >>> http://docs.sun.com/app/docs/doc/819-5461/gbctx?a=view >> >> In fact ZFS has a good transaction log, maybe the issue is there isn''t >> software out there yet that uses it. > > Where is that log? ZIL does not log all transactions and is cleared > very quickly.Isn''t the txg tags recorded during the block updates in ZFS? -Ross