Ext3 users - Oct 2006 - e2defrag - Unable to allocate buffer for inode priorities

Hi, first of all, apologies if this isn't the right mailing list but it was
the best I could find. If you know a better mailing list, please tell me.

Today I tried to defrag one of my filesystems. It's a 3.5T large filesystem
that has 6 software-raids in the bottom and then merged together using lvm. I
was running ext3 but removed the journal flag with
thor:~# tune2fs -O ^has_journal /dev/vgraid/data

After that I fsckd just to be sure I wouldnt meet any unexpected problems.

So now it was time to defrag, I used this command:
thor:~# e2defrag -r /dev/vgraid/data

After about 15 seconds (after it ate all my 1.5G of RAM) I got this answer:
e2defrag (/dev/vgraid/data): Unable to allocate buffer for inode priorities

I am using Debian unstable and here is the version information from e2defrag:
thor:~# e2defrag -V
e2defrag 0.73pjm1
RCS version $Id: defrag.c,v 1.4 1997/08/17 14:23:57 linux Exp $

I also tried to use -p 256, -p 128, -p 64 to see if it used less memory then, it
didn't seem like that to me, took the same time for the program to abort.


Is there any way to get around this problem? The answer might be to get 10G of
RAM, but that's not very realistic, 2G sure, but I think that's the
limit on my motherboard. A huge amount of swapfiles may solve it, and that's
probably doable, but it will be enormous slow I guess?

Why do I want to defrag? Well, fsck gives this nice info to me:
/dev/vgraid/data: 227652/475987968 files (41.2% non-contiguous),
847539147/951975936 blocks

41% sounds like a lot in my ears and I am having a constant read of files on the
drives, it's to slow already.


Very thankful for ideas or others experiences, maybe it's just not possible
with such large partition with todays tools, hey ext[23] only supports 4T.
Let's hope ext4 comes within a year in the mainstream kernels.


PS! Please CC me since I am not on the list so I dont have to wait for
marc's archive to get the mails.

--
Magnus M?nsson
Systems administrator
Massive Entertainment AB
Malm?, Sweden
Office: +46-40-6001000

On Oct 13, 2006  14:13 +0200, Magnus M?nsson wrote:
> Today I tried to defrag one of my filesystems. It's a 3.5T large
> filesystem that has 6 software-raids in the bottom and then merged
> together using lvm. I was running ext3 but removed the journal flag with
> Why do I want to defrag? Well, fsck gives this nice info to me:
> /dev/vgraid/data: 227652/475987968 files (41.2% non-contiguous),
847539147/951975936 blocks
> 
> 41% sounds like a lot in my ears and I am having a constant read of files
> on the drives, it's to slow already.
The 41% isn't necessarily bad if the files are very large.  For large
files it is inevitable that there will be fragmentation after 125MB or so.

What is a bigger problem is if the filesystem is constantly very nearly
full, or if your applications are appending a lot (e.g. mailspool).
> So now it was time to defrag, I used this command:
> thor:~# e2defrag -r /dev/vgraid/data
This program is dangerous to use and any attempts to use it should be
stopped.  It hasn't been updated in such a long time that it doesn't
even KNOW that it is dangerous (i.e. it doesn't check the filesystem
version number or feature flags).

What I would suggest in the meantime is to make as much free space in the 
filesystem as you can, find files that are very fragmented (via the
filefrag program) and then copy these files to a new temp file, and rename
it over the old file.  It should help for files that are very fragmented.

There is also a discussion about implementing online defragmentation, but
that is still a ways away.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.