Today I booted my linux-mint 14 into the latest 3.10.1 kernel to
defragment the root btrfs filesystem on my ASUS N56VZ laptop with hybrid
Seagate Momentus XT disk. I did something like
find / -exec btrfs filesystem defrag {}
To my amazement it didn''t really made a lot of I/O (the HDD LED
wasn''t
blinking). Confident that at the beginning the defragmentation is
CPU-bound (like it sometimes is for instance in case of O&O Defrag on
Windows world), I let it go on, and I switched myself to my other
chores. After a few minutes my system froze. I was barely able to switch
to text terminal and login, but I couldn''t reach bash prompt; it looked
like the OS couln''t perform any I/O on the disk. At the end, the system
responded to SysRq REISUB combination though; Unfortunately I can''t
reproduce exactly what I typed in, because the command was not logged to
the bash history...).
It clearly showed me, that defragmenting the filesystem is not that
trivial, as it is for ext4. So I have quesions:
* Is the defragmentation of the whole filesystem supported at all? I
can''t find a single reference that it is, and a syntax of btrfs-progs
suggest that it isn''t. If supported, under what conditions? Like what %
of free space should be available?
* How to check the level of defragmentation, and what are the reasonable
threshold values, that should indicate the desktop filesystem needs
defragmenting? I know, that everyone''s millage my vary; I just want to
know some values as a point-of-reference.
* What is the recommended command, that would efficiently defragment the
whole file system, preferably with some sort of progress indication?
Does this command
find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs
filesystem defragment -cv
look like a reasonable idiom for defragmenting the whole filesystem?
Thank you for your help,
Adam Ryczkowski
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello Adam, I routinely defrag my filesystems and here is how I do it:
find /home -type d -mtime -3 -o -type f -mtime -3 | egrep -v
"Cache|cache" | while read file; do /usr/sbin/btrfs filesystem defrag
-f
-v "${file}"; done
The above is what I use to defrag my data (non-OS) files. I use a while
loop because it relieves pressure on the system by guaranteeing that
each file and directory are processed one at a time. I use "-mtime
-3"
because I run this script daily via anacron and I see no reason to
defragment user files over and over again. I specifically filter out
"cache" directories because they take forever to defragment and are
refragmented in no time anyway and they are simply low priority for me
in terms of performance. I may revisit that policy again sometime in the
future, but for now I have been skipping them.
The following is what I use for system files:
find / -type d -mtime -3 -o -type f | while read file; do
/usr/sbin/btrfs filesystem defrag -f -v "${file}"; done
This is very similar to what I use for user files except I go back only
for 3 days on directories (meta data), but run defrag on every file in
the system every day just in case. This approach has worked very well
for me, but your mileage may vary (no guarantees). Also, in the case of
the system files noted above, I defrag daily and ALWAYS OFFLINE, because
at this point as I understand things, btrfs cannot defragment files when
they are currently in use. I get around this issue by doing system (/)
defrags only offline.
That''s how I do it.
- George
On 07/17/2013 03:18 PM, Adam Ryczkowski wrote:> Today I booted my linux-mint 14 into the latest 3.10.1 kernel to
> defragment the root btrfs filesystem on my ASUS N56VZ laptop with hybrid
> Seagate Momentus XT disk. I did something like
>
> find / -exec btrfs filesystem defrag {}
>
> To my amazement it didn''t really made a lot of I/O (the HDD LED
wasn''t
> blinking). Confident that at the beginning the defragmentation is
> CPU-bound (like it sometimes is for instance in case of O&O Defrag on
> Windows world), I let it go on, and I switched myself to my other
> chores. After a few minutes my system froze. I was barely able to switch
> to text terminal and login, but I couldn''t reach bash prompt; it
looked
> like the OS couln''t perform any I/O on the disk. At the end, the
system
> responded to SysRq REISUB combination though; Unfortunately I
can''t
> reproduce exactly what I typed in, because the command was not logged to
> the bash history...).
>
> It clearly showed me, that defragmenting the filesystem is not that
> trivial, as it is for ext4. So I have quesions:
>
> * Is the defragmentation of the whole filesystem supported at all? I
> can''t find a single reference that it is, and a syntax of
btrfs-progs
> suggest that it isn''t. If supported, under what conditions? Like
what %
> of free space should be available?
>
> * How to check the level of defragmentation, and what are the reasonable
> threshold values, that should indicate the desktop filesystem needs
> defragmenting? I know, that everyone''s millage my vary; I just
want to
> know some values as a point-of-reference.
>
> * What is the recommended command, that would efficiently defragment the
> whole file system, preferably with some sort of progress indication?
> Does this command
>
> find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs
> filesystem defragment -cv
>
> look like a reasonable idiom for defragmenting the whole filesystem?
>
> Thank you for your help,
>
> Adam Ryczkowski
>
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Adam Ryczkowski posted on Thu, 18 Jul 2013 00:18:23 +0200 as excerpted:> Today I booted my linux-mint 14 into the latest 3.10.1 kernel to > defragment the root btrfs filesystem on my ASUS N56VZ laptop with hybrid > Seagate Momentus XT disk. I did something like > > find / -exec btrfs filesystem defrag {}> [description of experience snipped] It clearly showed me, that > defragmenting the filesystem is not that trivial, as it is for ext4. > So I have quesions: [snipped]> Does this command > > find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs > filesystem defragment -cv > > look like a reasonable idiom for defragmenting the whole filesystem?Given that you don''t mention the btrfs wiki, and to a large extent your questions are covered there, I''ll assume you''re not familiar with it. The short answer is that yes, a find of that nature, piped to xargs to run btrfs filesystem defrag, is the suggested solution. However, you really need to read up on the wiki, and then ask any remaining questions you may have. (I still had some questions myself.) Bookmarking link: https://btrfs.wiki.kernel.org/ Defrag is covered there in some detail, including a recommended find- piped defrag command. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Shridhar Daithankar
2013-Jul-18 02:59 UTC
Re: What is the current status of defragmentation?
On Thursday, July 18, 2013 12:18:23 AM you wrote:> * Is the defragmentation of the whole filesystem supported at all? I > can''t find a single reference that it is, and a syntax of btrfs-progs > suggest that it isn''t. If supported, under what conditions? Like what % > of free space should be available?It works for me, I have done it few times on 3.9.9/arch. But I have kept plenty of free space handy to avoid any corner cases.> * How to check the level of defragmentation, and what are the reasonable > threshold values, that should indicate the desktop filesystem needs > defragmenting? I know, that everyone''s millage my vary; I just want to > know some values as a point-of-reference.Thats a hard question to answer but after you fully defrag the system(every btrfs mount/partition/filesystem), rebooting immediately with compress/autodefrag should do it automatically, since then. Are you mounting with noatime? storing access time could lead to massive direcory level fragmentation which is hard to measure. filefrag can help you but its per file and does not exactly give the level of fragmentation.> > * What is the recommended command, that would efficiently defragment the > whole file system, preferably with some sort of progress indication? > Does this command > > find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs > filesystem defragment -cv > > look like a reasonable idiom for defragmenting the whole filesystem?again, I use this. no progress indicator but it works ---------- for dir in / /home ; do find $dir -mount -type d -exec btrfs fi defrag ''{}'' \; ; find $dir -mount -type f -exec btrfs fi defrag ''{}'' \; ; done ---------- HTH -- Regards Shridhar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Shridhar Daithankar posted on Thu, 18 Jul 2013 08:29:22 +0530 as excerpted:> [A]fter you fully defrag the system(every btrfs > mount/partition/filesystem), rebooting immediately with > compress/autodefrag should do it automatically, since then.Indeed. It''s worth noting that the automated install system used by many distros leaves system files (obviously, no user files at that point) rather fragmented. Since the install process may leave no opportunity to mount with autodefrag during the install, it''s worth either pre-creating/pre- mounting the system partitions and skipping the part of the install process that would create and mount them, or installing to a temporary location and copying the whole install to a final location once you have the final destinations created/mounted with autodefrag and optionally compress, etc. That''ll save quite some time of the initial defrag slowing down the system, as well as allow you to defrag otherwise in-use files that aren''t "online defragged" otherwise.> Are you mounting with noatime? storing access time could lead to massive > direcory level fragmentation which is hard to measure.The kernel''s relatime default helps, but unless you''re running something like mutt that really depends on atime, it''s worth using noatime as a rule. (The only volumes I don''t use noatime on are virtual filesystems such as tmpfs and sysfs, where it''s simply a memory access in any case and I''ve seen no qualified opinion or benchmarks arguing either way.)> filefrag can help you but its per file and does not exactly give the > level of fragmentation.Someone did mention that btrfs compressed files larger than some size (the btrfs leaf size?) will always look fragmented to filefrag. I do not personally understand enough about it to verify, however. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Indeed! That is a VERY good question and one I have not seen a good answer to thus far. A traditional defragger in the Windows realm, which defrags on a global basis rather than directory by directory, file by file, also "restacks" the file and directory structure in a way that minimizes free space and the result is also close to a maximum amount of contiguous free space somewhere within the filesystem after the defrag process finishes. As you correctly point out, no one has indicated a method as to how that can be accomplished with btrfs at this point. I am not even sure that rebuilding the partition from scratch would do the job. My solution so far is just to always make sure to have a ridiculous amount of free space available, but that cannot go on forever (and is very inefficient). So it would be nice to know what the solution to this conundrum would be. Thanks for pointing this out. On 07/18/2013 12:12 AM, Adam Ryczkowski wrote:> On 07/18/2013 02:17 AM, George Mitchell wrote: >> find /home -type d -mtime -3 -o -type f -mtime -3 | egrep -v >> "Cache|cache" | while read file; do /usr/sbin/btrfs filesystem defrag >> -f -v "${file}"; done > Thank you for your answer. > > This still defragments file-after-file. But what about a little > consolidation of free space? As I understand, if there is no consecutive > free space for a file''s extents, it is impossible to defragment it, no > matter how many times I run the above command, am I right? > > Adam Ryczkowski > > <http://www.google.com/> > > >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html