I am testing btrfs for long-term storage and backup, and i would like
to know more about "autodefrag" option:
1. Will "autodefrag" option benefit ssd?
My understanding is:
autodrag -> number of extent decrease -> metadata decrease -> a
"healthier" filesystem in the long run
(P.S. I am aware that autodefrag will introduce extra write I/O)
2. AFAIK, "autodefrag" detects small random writes into files and
queues them up for an automatic defrag process, so the filesystem will
defragment itself while it''s used.
If the system reboot/crash/remount-ro, will the autodefrag process
continue after resume?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
can anybody helps? On 09/17/2012 07:15 PM, ching wrote:> I am testing btrfs for long-term storage and backup, and i would like > to know more about "autodefrag" option: > > 1. Will "autodefrag" option benefit ssd? > > My understanding is: > > autodrag -> number of extent decrease -> metadata decrease -> a > "healthier" filesystem in the long run > > (P.S. I am aware that autodefrag will introduce extra write I/O) > > 2. AFAIK, "autodefrag" detects small random writes into files and > queues them up for an automatic defrag process, so the filesystem will > defragment itself while it''s used. > > If the system reboot/crash/remount-ro, will the autodefrag process > continue after resume? > >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 09/19/2012 07:28 PM, ching wrote:> can anybody helps? > > On 09/17/2012 07:15 PM, ching wrote: >> I am testing btrfs for long-term storage and backup, and i would like >> to know more about "autodefrag" option: >> >> 1. Will "autodefrag" option benefit ssd? >> >> My understanding is: >> >> autodrag -> number of extent decrease -> metadata decrease -> a >> "healthier" filesystem in the long run >> >> (P.S. I am aware that autodefrag will introduce extra write I/O) >>Yes, your understanding is right, random write workloads will benefit from it.>> 2. AFAIK, "autodefrag" detects small random writes into files and >> queues them up for an automatic defrag process, so the filesystem will >> defragment itself while it''s used. >> >> If the system reboot/crash/remount-ro, will the autodefrag process >> continue after resume? >>For reboot, autodefrag will be waited to finish during umounting btrfs. For crash and remount-ro, it won''t resume since it is not that necessary and we''re all COWed so that the data is ok. And autodefrag will only take effect when taking the ''autodefrag'' mount option. thanks, liubo>> > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Am Mittwoch, 19. September 2012 schrieb Liu Bo:> On 09/19/2012 07:28 PM, ching wrote:[…]> > On 09/17/2012 07:15 PM, ching wrote: > >> I am testing btrfs for long-term storage and backup, and i would > >> like to know more about "autodefrag" option: > >> > >> 1. Will "autodefrag" option benefit ssd? > >> > >> My understanding is: > >> > >> autodrag -> number of extent decrease -> metadata decrease -> > >>a > >> > >> "healthier" filesystem in the long run > >> > >> (P.S. I am aware that autodefrag will introduce extra write I/O) > > Yes, your understanding is right, random write workloads will benefit > from it.What about the extra I/O? And the greatly reduced seek times on SSDs? Upto now I kept away from defragmenting on SSD. I wonder about a good way to decide whether autodefrag makes things better or worse for a specific workload. What are the criteria on rotating media and what are they on SSD? [… informational part about a BTRFS on SSD that should have an age of at least 8 months with almost daily upgrades …] I only have / running on SSD, but this since quite some time. And it does not seem to have gotten much worse – however this is only subjective feeling of performance. Except for fstrim times. fstrim take way more time than in the beginning¹. So there seems to be free space fragmentation. Which makes sense for a root filesystem on a Debian Sid machine with lots of upgrade activity and way over 50% usage. merkaba:~> df -hT / Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf /dev/dm-0 btrfs 19G 13G 3,6G 79% / merkaba:~> btrfs fi sh failed to read /dev/sr0 Label: ''debian'' uuid: […] Total devices 1 FS bytes used 12.25GB devid 1 size 18.62GB used 18.62GB path /dev/dm-0 merkaba:~> btrfs fi df / Data: total=15.10GB, used=11.58GB System, DUP: total=8.00MB, used=4.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=1.75GB, used=688.96MB Metadata: total=8.00MB, used=0.00 I think I get rid of that duplicate metadata once I redo the fs with 8 or 16 KiB metadata blocks. I thought about rebalancing it, but last time boot time doubled after a complete rebalance. The effect of a rebalance of just the metadata tree to one instead of two might be different tough. Intel SSD 320 300GB on Kernel 3.6-rc5. [1] It has been a second or two in the beginning I think. Then it grew over time. merkaba:~> time fstrim -v / /: 5877809152 bytes were trimmed fstrim -v / 0,00s user 5,74s system 14% cpu 39,920 total merkaba:~> time fstrim -v / /: 5875712000 bytes were trimmed fstrim -v / 0,00s user 5,55s system 14% cpu 39,095 total merkaba:~> time fstrim -v / /: 5875712000 bytes were trimmed fstrim -v / 0,00s user 5,62s system 14% cpu 38,538 total -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> 2. AFAIK, "autodefrag" detects small random writes into files and >>> queues them up for an automatic defrag process, so the filesystem will >>> defragment itself while it''s used. >>> >>> If the system reboot/crash/remount-ro, will the autodefrag process >>> continue after resume? >>> > For reboot, autodefrag will be waited to finish during umounting btrfs. > > For crash and remount-ro, it won''t resume since it is not that necessary and we''re all > COWed so that the data is ok. > > And autodefrag will only take effect when taking the ''autodefrag'' mount option. > > thanks, > liubo > >Can the loophole be fixed by tuning the autodefrag algorithm? e.g. also queue up small extents detected during read operation for the autodefrag process. Is there any io niceness control for autodefrag process too? it will be nice if the idle class is used. Actually, i try to write shell script to defrag entire FS monthy. but i soon find out it is hard to keep track of progress across reboot and avoid redefragging defragged file For me, autodefrag is a killer feature. :) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Sep 20, 2012 at 07:36:53AM +0800, ching wrote:> > >>> 2. AFAIK, "autodefrag" detects small random writes into files and > >>> queues them up for an automatic defrag process, so the filesystem will > >>> defragment itself while it''s used. > >>> > >>> If the system reboot/crash/remount-ro, will the autodefrag process > >>> continue after resume? > >>> > > For reboot, autodefrag will be waited to finish during umounting btrfs. > > > > For crash and remount-ro, it won''t resume since it is not that necessary and we''re all > > COWed so that the data is ok. > > > > And autodefrag will only take effect when taking the ''autodefrag'' mount option. > > > > thanks, > > liubo > > > > > > Can the loophole be fixed by tuning the autodefrag algorithm? e.g. > also queue up small extents detected during read operation for the > autodefrag process.Sounds interesting, though this would increase write load for a pure read but may improve fragmentation situation and is obviously done only on the first read. The write side of autodefrag could be extended to defrag also small extents that would fit into some threshold, not only newly written data.> Is there any io niceness control for autodefrag process too? it will > be nice if the idle class is used.No. Autodefrag will mark file data dirty and they''ll be written back to the storage in the same way as any other write through the worker threads.> Actually, i try to write shell script to defrag entire FS monthy. but > i soon find out it is hard to keep track of progress across reboot and > avoid redefragging defragged fileThe defrag process will skip already deframented files (in the meaning of "defragmented" as it sees fit, implementation dependent currently, not exactly that the file has 1 extent).> For me, autodefrag is a killer feature. :)Yeah, it improves the inherent problem of fragmentation for a COW based filesystem. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 19, 2012 at 07:39:42PM +0200, Martin Steigerwald wrote:> > >> (P.S. I am aware that autodefrag will introduce extra write I/O) > > > > Yes, your understanding is right, random write workloads will benefit > > from it. > > What about the extra I/O? And the greatly reduced seek times on SSDs? > > Upto now I kept away from defragmenting on SSD. > > I wonder about a good way to decide whether autodefrag makes things better > or worse for a specific workload. What are the criteria on rotating media > and what are they on SSD?Reducing data block fragmentation is accompanied by decreased metadata usage that''s needed to maintan the data blocks, and this means less cpu processing and less memory usage. Blocks in larger contiguous chunks are also more friendly to the SSD garbage collection. A full filesystem defrag & fstrim once in a while may improve performance and device lifetime.> [??? informational part about a BTRFS on SSD that should have an age of at > least 8 months with almost daily upgrades ???] > > I only have / running on SSD, but this since quite some time. And it does > not seem to have gotten much worse ??? however this is only subjective > feeling of performance. > > Except for fstrim times. fstrim take way more time than in the beginning¹. > So there seems to be free space fragmentation. Which makes sense for a > root filesystem on a Debian Sid machine with lots of upgrade activity and > way over 50% usage.I''m roughly counting: used: 11 + 2*0.688 = 12 touched blocks: 15 + 2* 1.75 = 18.5 of 19, ~90%, this may be problematic even on a SSD regarding the erase block related specifics.> merkaba:~> df -hT / > Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf > /dev/dm-0 btrfs 19G 13G 3,6G 79% / > > merkaba:~> btrfs fi sh > failed to read /dev/sr0 > Label: ''debian'' uuid: [???] > Total devices 1 FS bytes used 12.25GB > devid 1 size 18.62GB used 18.62GB path /dev/dm-0 > > merkaba:~> btrfs fi df / > Data: total=15.10GB, used=11.58GB > System, DUP: total=8.00MB, used=4.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=1.75GB, used=688.96MB > Metadata: total=8.00MB, used=0.00 > > I think I get rid of that duplicate metadata once I redo the fs with 8 or > 16 KiB metadata blocks. > > I thought about rebalancing it, but last time boot time doubled after a > complete rebalance. The effect of a rebalance of just the metadata tree to > one instead of two might be different tough. > > Intel SSD 320 300GB on Kernel 3.6-rc5. > > > [1] It has been a second or two in the beginning I think. Then it grew > over time. > > merkaba:~> time fstrim -v / > /: 5877809152 bytes were trimmed > fstrim -v / 0,00s user 5,74s system 14% cpu 39,920 total > merkaba:~> time fstrim -v / > /: 5875712000 bytes were trimmed > fstrim -v / 0,00s user 5,55s system 14% cpu 39,095 total > merkaba:~> time fstrim -v / > /: 5875712000 bytes were trimmed > fstrim -v / 0,00s user 5,62s system 14% cpu 38,538 totalLooks like the issued TRIM commands actually did no work. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Is there any io niceness control for autodefrag process too? it will >> be nice if the idle class is used. > No. Autodefrag will mark file data dirty and they''ll be written back to > the storage in the same way as any other write through the worker > threads. > >AFAIK, the autodefrag will read related extents and mark them dirty, io niceness should be applicable to the read operation -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Sep 21, 2012 at 05:03:41AM +0800, ching wrote:> AFAIK, the autodefrag will read related extents and mark them dirty, > io niceness should be applicable to the read operationI was hesitant to believe that, but tried it and it''s right: $ dd if=/dev/zero bs=4k count=1 seek=15 of=file $ dd if=/dev/zero bs=4k count=1 seek=13 of=file conv=notrunc $ dd if=/dev/zero bs=4k count=1 seek=14 of=file conv=notrunc Filesystem type is: 9123683e File size of file is 65536 (16 blocks, blocksize 4096) ext logical physical expected length flags 0 13 1027 1 1 14 1050 1027 1 2 15 1025 1050 1 eof file: 4 extents found remount with autodefrag: $ dd if=/dev/zero bs=4k count=1 seek=16 of=file conv=notrunc Filesystem type is: 9123683e File size of file is 69632 (17 blocks, blocksize 4096) ext logical physical expected length flags 0 13 1033 4 eof file: 2 extents found david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html