I'm trying to do an fsck on an ext3 partition but I'm seeing abysmally slow disk throughput; monitoring with "dstat" (like vmstat) shows ~1200-1500KB/s throughput to the disks. Even with 24hrs of fsck-ing I only get ~3% (still in pass1). The filesystem is ext3 running "e2fsck -C0 /dev/sda3" and about 3.7TB on an x86_64-based system with 4GB RAM. e2fsprogs is 1.41.9. During initial periods of the fsck I see throughput in the 30-70MB/s range (ie, <0.4% complete). Shortly after that throughput tanks and stays there. Just a rough extrapolation of the size of my filesystem (3.4TB used; ~95%) makes it look like this will take ~28 days to complete. I'm using approx 12M inodes out of my 243M available. -Doug -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: OpenPGP digital signature URL: <http://listman.redhat.com/archives/ext3-users/attachments/20100120/3395d5fd/attachment.sig>
>>> On Wed, 20 Jan 2010 12:30:35 -0500, Doug Warner >>> <doug at warner.fm> said:doug> I'm trying to do an fsck on an ext3 partition but I'm doug> seeing abysmally slow disk throughput; monitoring with doug> "dstat" (like vmstat) shows ~1200-1500KB/s throughput to doug> the disks. That seems pretty good to me. Perhaps the impression of slowness is motivated by insufficient understanding of how 'fsck' works and the IOP limitations of small rotating mass storage arrays. doug> Even with 24hrs of fsck-ing I only get ~3% (still in doug> pass1). That's pretty good too. doug> The filesystem is ext3 running "e2fsck -C0 /dev/sda3" and doug> about 3.7TB on an x86_64-based system with 4GB RAM. doug> e2fsprogs is 1.41.9. There have been reports of a 1.5TB 'ext3' filesystem taking over a month: http://www.sabi.co.uk/blog/anno05-4th.html#051009 even if in optimal cases it can be better: http://www.sabi.co.uk/blog/0802feb.html#080210 doug> Just a rough extrapolation of the size of my filesystem doug> (3.4TB used; ~95%) makes it look like this will take ~28 doug> days to complete. I'm using approx 12M inodes out of my doug> 243M available. That sounds about right for lots of small files (~280KB average) in a very large number of directories (or in directories that are really very long), and which is 95% used. You have designed that filesystem that way, and it is performing as expected or better. People who use filesystems as databases deserve whatn they get.
Doug Warner wrote:> I'm trying to do an fsck on an ext3 partition but I'm seeing abysmally slow > disk throughput; monitoring with "dstat" (like vmstat) shows ~1200-1500KB/s > throughput to the disks. Even with 24hrs of fsck-ing I only get ~3% (still in > pass1). > > The filesystem is ext3 running "e2fsck -C0 /dev/sda3" and about 3.7TB on an > x86_64-based system with 4GB RAM. e2fsprogs is 1.41.9. > > During initial periods of the fsck I see throughput in the 30-70MB/s range > (ie, <0.4% complete). Shortly after that throughput tanks and stays there. > Just a rough extrapolation of the size of my filesystem (3.4TB used; ~95%) > makes it look like this will take ~28 days to complete. I'm using approx 12M > inodes out of my 243M available.It'd be interesting to use blktrace and/or seekwatcher to see if you are seeking madly all over the disk, that would certainly clobber perf. -Eric> -Doug
On Wed, Jan 20, 2010 at 12:30:35PM -0500, Doug Warner wrote:> I'm trying to do an fsck on an ext3 partition but I'm seeing abysmally slow > disk throughput; monitoring with "dstat" (like vmstat) shows ~1200-1500KB/s > throughput to the disks. Even with 24hrs of fsck-ing I only get ~3% (still in > pass1). > > The filesystem is ext3 running "e2fsck -C0 /dev/sda3" and about 3.7TB on an > x86_64-based system with 4GB RAM. e2fsprogs is 1.41.9. > > During initial periods of the fsck I see throughput in the 30-70MB/s range > (ie, <0.4% complete). Shortly after that throughput tanks and stays there. > Just a rough extrapolation of the size of my filesystem (3.4TB used; ~95%) > makes it look like this will take ~28 days to complete. I'm using approx 12M > inodes out of my 243M available.Are you using some kind of backup scheme that creates huge number of hard links to files? E2fsck could be thrashing due to lack of memory space. - Ted