Hello everybody, reading about the speed improvements possible with (on boot) preloaded files (which should be continuous on disk) I searched for a ext3 defrag program. I found an ext2 defrag program (http://www.ibiblio.org/pub/Linux/system/filesystems/defrag-0.70.tar.gz, available in debian as defrag) which would have an optimal feature (moving files by a list) but refuses to work on ext3. Is there a version which does ext3? Or has somebody a program which allows me to move files on the disk? And BTW, is there an easy way (no kernel patching if possible) to determine which files are used in which order during boot? Regards, Phil
I'm on the e2fsprogs team. I worked on a defrag program for EXT3 for about six months. In the end, we abandoned the program as unfeasible. I have to go to work now, but I can send you more details later. Joseph D. Wagner
On Wed, Mar 02, 2005 at 01:14:01PM +0100, Ph. Marek wrote:> Hello everybody, > > reading about the speed improvements possible with (on boot) preloaded files > (which should be continuous on disk) I searched for a ext3 defrag program. > > I found an ext2 defrag program > (http://www.ibiblio.org/pub/Linux/system/filesystems/defrag-0.70.tar.gz, > available in debian as defrag) which would have an optimal feature (moving > files by a list) but refuses to work on ext3. > > Is there a version which does ext3? Or has somebody a program which allows me > to move files on the disk?The e2defrag program had some problems where it only worked on 1k blocksizes, if I remember correctly. It was also extremely dangerous in that if it crashed or you had a system crash/powerfailure in middle of the operation, your filesystem would be totally scrambled. Therefore, the only safe way to use it was to do a full backup, and if the system crashed, restore from the backup. Given that the filesystem had to be unmounted during the e2defrag process, and combined with the fact that if you wanted to be safe, you had to do a full backup of the data *anyway*, the time difference between doing "backup; e2defrag; mke2fs and restore if your system crashed" and "backup; mke2fs; restore" was such that it really wasn't worth it. Mainly, there hasn't been sufficient interest to write a (safe, effective) ext2 defragger. (There was one crazy person who didn't believe me when I told him that an ext3 defragger couldn't be done purely in userspace, until he banged his head against the wall enough times, but that doesn't count. :-) Instead there has been more interest in tweaking algorithsm that try to avoid the fragmentation problem in the first place --- for example, such as the Orlov allocator that got introduced during Linux 2.5. Another example is the delayed allocation code plus the extent mapping extension that has been currently discussed on ext2-devel. There has been talk about writing a kernel extension which implements a few safe, atomic operations, such as relocating a logical block #w in inode #x from block #y to #z, and "here are all the pathnames that point at inode #x, relocate that inode to be stored at location #y", and then implementing the rest in userspace. But it just hasn't risen to the top of anyone's todo lists yet. - Ted
I approached the problem from several different angles.
Approach #1 - Partition Must Be Unmounted to Defrag
Advantages:
* Minimize chance of file system corruption because
no other processes could be modifying files.
* Can manipulate the file system in ways that would
otherwise be unsafe if the file system were mounted.
Disadvantages:
* Must directly manipulate the journal (i.e. a lot of
tedious programming) so that the file system can
remain consistent in the event of a power failure
* The partition would be completely unavailable during
the entire defrag process resulting in significant
down time.
Ultimately rejected because of the last disadvantage.
Approach #2 - On-the-fly Defragmentation using the
EXT2FS library for Block Allocation
Advantages:
* No down time.
* Utilizes existing EXT3 journal programming by
performing the defragmentation using normal
read-write operations.
Disadvantages:
* Without mandatory locking, there would be no way
to ensure that another process has not modified the
file.
* Because of the way the kernel buffers I/O data,
another process could be attempting to utilize the
same blocks as the defrag program.
Attempted, but ultimately rejected because of the last disadvantage.
Approach #3 - On-the-fly Defragmentation without a
Block Allocation Policy
Advantages:
* This is a close as we can get to a "safe" on-the-fly
defragmentation program.
* No down time.
* Utilizes existing EXT3 journal programming by
performing the defragmentation using normal
read-write operations.
Disadvantages:
* Without a block allocation policy, defrag would be
nothing more than a sophisticated copy program.
* While it is likely that the copy would be less
fragmented than the original, without a block
allocation policy there would be no guarantee. The
copy would have to be checked against the original
before overwriting it.
* Without mandatory locking, there would be no way
to ensure that another process has not modified the
file.
Attempted, but ultimately rejected because of the last disadvantage.
In order to make a defrag for EXT3 safe, you would need to do one of
two things:
1) Develop a Defrag API or some sort of File System Maintenance API
which included Defrag support and successfully integrate it into
the kernel.
- OR -
2) Extend Mandatory Locking to every file the system opens and closes
and integrate such a patch into the kernel.
Either task is uphill sledding.
Joseph D. Wagner