Hi, I have a system that''s using a dracut-generated initramfs to mount a btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve noticed that the process of mounting the root filesystem takes much longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! Does anyone have any ideas? I''m going to try bisecting the issue to see if I can narrow down the cause. I''ve included excerpts from dmesg of the bad and good kernels here, and attached the complete dmesg from the bad kernel, in case it has anything interesting that I''ve trimmed out here. slow: [ 0.000000] Linux version 3.4.0-rc2 (cwalton@ayu) (gcc version 4.6.3 (Exherbo gcc-4.6.3) ) #57 SMP PREEMPT Mon Apr 9 11:19:43 EDT 2012 [ 0.000000] Command line: root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime mtrr_chunk_size=512M quiet <snip> [ 1.058257] udevd[701]: starting version 182 [ 1.389606] device label Linux data devid 1 transid 611923 /dev/sda4 [ 1.498659] dracut: Checking, if btrfs device complete [ 1.644808] device label Linux data devid 1 transid 611923 /dev/sda4 [ 1.647993] btrfs: disk space caching is enabled [ 2.180836] device label Linux data devid 1 transid 611923 /dev/sda4 [ 2.181663] btrfs: use lzo compression [ 2.181667] btrfs: enabling auto defrag [ 2.181670] btrfs: enabling inode map caching [ 2.181672] btrfs: disk space caching is enabled [ 2.697845] dracut: Checking btrfs: /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 [ 2.699999] dracut: trying to mount /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 [ 2.702637] device label Linux data devid 1 transid 611923 /dev/sda4 [ 2.704376] btrfs: disk space caching is enabled [ 3.081720] dracut: btrfs: /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 is clean [ 29.934639] dracut: Remounting /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 with -o subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime,ro [ 29.936810] device label Linux data devid 1 transid 611926 /dev/sda4 [ 29.937720] btrfs: use lzo compression [ 29.937726] btrfs: enabling auto defrag [ 29.937733] btrfs: enabling inode map caching [ 29.937735] btrfs: disk space caching is enabled [ 30.388066] dracut: Mounted root filesystem /dev/sda4 [ 30.461884] dracut: Switching root [ 31.241729] udevd[1322]: starting version 182 [ 31.422905] btrfs: use lzo compression [ 31.422909] btrfs: enabling auto defrag [ 31.422913] btrfs: enabling inode map caching [ 31.422915] btrfs: disk space caching is enabled vs fast: [ 0.000000] Linux version 3.3.1 (cwalton@ayu) (gcc version 4.6.3 (Exherbo gcc-4.6.3) ) #53 SMP PREEMPT Wed Apr 4 01:20:47 EDT 2012 [ 0.000000] Command line: root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime mtrr_chunk_size=512M quiet <snip> [ 1.060117] udevd[695]: starting version 182 [ 1.344328] device label Linux data devid 1 transid 611917 /dev/sda4 [ 1.384888] dracut: Checking, if btrfs device complete [ 1.405121] device label Linux data devid 1 transid 611917 /dev/sda4 [ 1.405905] btrfs: disk space caching is enabled [ 1.971953] device label Linux data devid 1 transid 611917 /dev/sda4 [ 1.972519] btrfs: use lzo compression [ 1.972523] btrfs: enabling auto defrag [ 1.972526] btrfs: enabling inode map caching [ 1.972528] btrfs: disk space caching is enabled [ 2.528540] dracut: Checking btrfs: /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 [ 2.530725] dracut: trying to mount /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 [ 2.558285] device label Linux data devid 1 transid 611917 /dev/sda4 [ 2.559087] btrfs: disk space caching is enabled [ 3.013103] dracut: btrfs: /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 is clean [ 3.518807] dracut: Remounting /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 with -o subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime,ro [ 3.520031] device label Linux data devid 1 transid 611920 /dev/sda4 [ 3.520680] btrfs: use lzo compression [ 3.520684] btrfs: enabling auto defrag [ 3.520688] btrfs: enabling inode map caching [ 3.520689] btrfs: disk space caching is enabled [ 4.066285] dracut: Mounted root filesystem /dev/sda4 [ 4.120861] dracut: Switching root [ 4.995934] btrfs: use lzo compression [ 4.995938] btrfs: enabling auto defrag [ 4.995942] btrfs: enabling inode map caching [ 4.995943] btrfs: disk space caching is enabled [ 5.010473] udevd[1330]: starting version 182 -- Calvin Walton <calvin.walton@kepstin.ca>
On Mon, Apr 9, 2012 at 9:53 AM, Calvin Walton <calvin.walton@kepstin.ca> wrote:> Hi, > > I have a system that''s using a dracut-generated initramfs to mount a > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > noticed that the process of mounting the root filesystem takes much > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > Does anyone have any ideas? I''m going to try bisecting the issue to see > if I can narrow down the cause. I''ve included excerpts from dmesg of the > bad and good kernels here, and attached the complete dmesg from the bad > kernel, in case it has anything interesting that I''ve trimmed out here.That sounds suspiciously like a symlink from btrfsck is to fsck.btrfs was added at about the same time as the update (old initrd maybe?). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote:> Hi, > > I have a system that''s using a dracut-generated initramfs to mount a > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > noticed that the process of mounting the root filesystem takes much > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > Does anyone have any ideas? I''m going to try bisecting the issue to see > if I can narrow down the cause. I''ve included excerpts from dmesg of the > bad and good kernels here, and attached the complete dmesg from the bad > kernel, in case it has anything interesting that I''ve trimmed out here.And the bisect results are in: 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff Author: Josef Bacik <josef@redhat.com> Date: Fri Jan 13 15:27:45 2012 -0500 Btrfs: remove the ideal caching code This is a relic from before we had the disk space cache and it was to make bootup times when you had btrfs as root not be so damned slow. Now that we have the disk space cache this isn''t a problem anymore and really having this code casues uneeded fragmentation and complexity, so just remove it. Thanks, Signed-off-by: Josef Bacik <josef@redhat.com> The commit doesn''t revert cleanly on top of 3.4.0-rc2, so I haven''t tested that; but it looks like that caching code is in fact still useful to make "btrfs as root not be so damned slow."> slow: > [ 0.000000] Linux version 3.4.0-rc2 (cwalton@ayu) (gcc version 4.6.3 (Exherbo gcc-4.6.3) ) #57 SMP PREEMPT Mon Apr 9 11:19:43 EDT 2012 > [ 0.000000] Command line: root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime mtrr_chunk_size=512M quiet > <snip> > [ 1.058257] udevd[701]: starting version 182 > [ 1.389606] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 1.498659] dracut: Checking, if btrfs device complete > [ 1.644808] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 1.647993] btrfs: disk space caching is enabled > [ 2.180836] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 2.181663] btrfs: use lzo compression > [ 2.181667] btrfs: enabling auto defrag > [ 2.181670] btrfs: enabling inode map caching > [ 2.181672] btrfs: disk space caching is enabled > [ 2.697845] dracut: Checking btrfs: /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > [ 2.699999] dracut: trying to mount /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > [ 2.702637] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 2.704376] btrfs: disk space caching is enabled > [ 3.081720] dracut: btrfs: /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 is clean > [ 29.934639] dracut: Remounting /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 with -o subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime,ro > [ 29.936810] device label Linux data devid 1 transid 611926 /dev/sda4 > [ 29.937720] btrfs: use lzo compression > [ 29.937726] btrfs: enabling auto defrag > [ 29.937733] btrfs: enabling inode map caching > [ 29.937735] btrfs: disk space caching is enabled > [ 30.388066] dracut: Mounted root filesystem /dev/sda4 > [ 30.461884] dracut: Switching root > [ 31.241729] udevd[1322]: starting version 182 > [ 31.422905] btrfs: use lzo compression > [ 31.422909] btrfs: enabling auto defrag > [ 31.422913] btrfs: enabling inode map caching > [ 31.422915] btrfs: disk space caching is enabled-- Calvin Walton <calvin.walton@kepstin.ca> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote:> On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > Hi, > > > > I have a system that''s using a dracut-generated initramfs to mount a > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > noticed that the process of mounting the root filesystem takes much > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > > > Does anyone have any ideas? I''m going to try bisecting the issue to see > > if I can narrow down the cause. I''ve included excerpts from dmesg of the > > bad and good kernels here, and attached the complete dmesg from the bad > > kernel, in case it has anything interesting that I''ve trimmed out here. > > And the bisect results are in: > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > Author: Josef Bacik <josef@redhat.com> > Date: Fri Jan 13 15:27:45 2012 -0500 > > Btrfs: remove the ideal caching code > > This is a relic from before we had the disk space cache and it was to make > bootup times when you had btrfs as root not be so damned slow. Now that we have > the disk space cache this isn''t a problem anymore and really having this code > casues uneeded fragmentation and complexity, so just remove it. Thanks, > > Signed-off-by: Josef Bacik <josef@redhat.com> > > The commit doesn''t revert cleanly on top of 3.4.0-rc2, so I haven''t > tested that; but it looks like that caching code is in fact still useful > to make "btrfs as root not be so damned slow."Hrm well you should have disk space cache which is 10x faster, if it''s falling back to the old slow way we should probably figure out why that is happening. Let me run some tests and see how often I''m getting no disk cache written out. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote:> On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > Hi, > > > > I have a system that''s using a dracut-generated initramfs to mount a > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > noticed that the process of mounting the root filesystem takes much > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > > > Does anyone have any ideas? I''m going to try bisecting the issue to see > > if I can narrow down the cause. I''ve included excerpts from dmesg of the > > bad and good kernels here, and attached the complete dmesg from the bad > > kernel, in case it has anything interesting that I''ve trimmed out here. > > And the bisect results are in: > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > Author: Josef Bacik <josef@redhat.com> > Date: Fri Jan 13 15:27:45 2012 -0500 > > Btrfs: remove the ideal caching code > > This is a relic from before we had the disk space cache and it was to make > bootup times when you had btrfs as root not be so damned slow. Now that we have > the disk space cache this isn''t a problem anymore and really having this code > casues uneeded fragmentation and complexity, so just remove it. Thanks, > > Signed-off-by: Josef Bacik <josef@redhat.com> > > The commit doesn''t revert cleanly on top of 3.4.0-rc2, so I haven''t > tested that; but it looks like that caching code is in fact still useful > to make "btrfs as root not be so damned slow." >Ok can you give this a whirl? You are going to have to boot/reboot a few times to let the cache get re-generated again to make sure it''s taken effect, but hopefully this will help out. Thanks, Josef diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index a844204..7a703d2 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -530,8 +530,8 @@ static int cache_block_group(struct btrfs_block_group_cache *cache, * we likely hold important locks. */ if (trans && (!trans->transaction->in_commit) && - (root && root != root->fs_info->tree_root) && - btrfs_test_opt(root, SPACE_CACHE)) { + root != fs_info->tree_root && + fs_info->mount_opt & BTRFS_MOUNT_SPACE_CACHE) { ret = load_free_space_cache(fs_info, cache); spin_lock(&cache->lock); -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote:> On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > > Hi, > > > > > > I have a system that''s using a dracut-generated initramfs to mount a > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > > noticed that the process of mounting the root filesystem takes much > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower!> > And the bisect results are in: > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > > Author: Josef Bacik <josef@redhat.com> > > Date: Fri Jan 13 15:27:45 2012 -0500 > > > > Btrfs: remove the ideal caching code> > > Ok can you give this a whirl? You are going to have to boot/reboot a few times > to let the cache get re-generated again to make sure it''s taken effect, but > hopefully this will help out. Thanks,Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with this patch applied I''m still seeing the same delay. -- Calvin Walton <calvin.walton@kepstin.ca> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote:> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: > > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > > > Hi, > > > > > > > > I have a system that''s using a dracut-generated initramfs to mount a > > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > > > noticed that the process of mounting the root filesystem takes much > > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > > > And the bisect results are in: > > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > > > Author: Josef Bacik <josef@redhat.com> > > > Date: Fri Jan 13 15:27:45 2012 -0500 > > > > > > Btrfs: remove the ideal caching code> > > > > Ok can you give this a whirl? You are going to have to boot/reboot a few times > > to let the cache get re-generated again to make sure it''s taken effect, but > > hopefully this will help out. Thanks, > > Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with > this patch applied I''m still seeing the same delay. >Alright my laptop is on btrfs, I''ll build my tree and see if I can see the same thing and then narrow down whats going on. If not you will be seeing many more patches from me ;). Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote:> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: > > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > > > Hi, > > > > > > > > I have a system that''s using a dracut-generated initramfs to mount a > > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > > > noticed that the process of mounting the root filesystem takes much > > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > > > And the bisect results are in: > > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > > > Author: Josef Bacik <josef@redhat.com> > > > Date: Fri Jan 13 15:27:45 2012 -0500 > > > > > > Btrfs: remove the ideal caching code> > > > > Ok can you give this a whirl? You are going to have to boot/reboot a few times > > to let the cache get re-generated again to make sure it''s taken effect, but > > hopefully this will help out. Thanks, > > Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with > this patch applied I''m still seeing the same delay. >Ok drop that previous patch and give this one a whirl, it helped on my laptop. This is only half of the problem AFAICS, but it''s the easier half to fix, in the meantime I need to lock down why we''re not writing out cache for a bunch of block groups, but thats trickier since the messages I need are spit out while I''m shutting down, so I need to get creative. Let me know if/how much this helps. Thanks, Josef diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index a844204..4a871f0 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -529,9 +529,7 @@ static int cache_block_group(struct btrfs_block_group_cache *cache, * allocate blocks for the tree root we can''t do the fast caching since * we likely hold important locks. */ - if (trans && (!trans->transaction->in_commit) && - (root && root != root->fs_info->tree_root) && - btrfs_test_opt(root, SPACE_CACHE)) { + if (fs_info->mount_opt & BTRFS_MOUNT_SPACE_CACHE) { ret = load_free_space_cache(fs_info, cache); spin_lock(&cache->lock); diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c index 054707e..baaa518 100644 --- a/fs/btrfs/free-space-cache.c +++ b/fs/btrfs/free-space-cache.c @@ -748,13 +748,6 @@ int load_free_space_cache(struct btrfs_fs_info *fs_info, u64 used = btrfs_block_group_used(&block_group->item); /* - * If we''re unmounting then just return, since this does a search on the - * normal root and not the commit root and we could deadlock. - */ - if (btrfs_fs_closing(fs_info)) - return 0; - - /* * If this block group has been marked to be cleared for one reason or * another then we can''t trust the on disk cache, so just return. */ @@ -768,6 +761,8 @@ int load_free_space_cache(struct btrfs_fs_info *fs_info, path = btrfs_alloc_path(); if (!path) return 0; + path->search_commit_root = 1; + path->skip_locking = 1; inode = lookup_free_space_inode(root, block_group, path); if (IS_ERR(inode)) { -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2012-04-10 at 11:16 -0400, Josef Bacik wrote:> On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: > > On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: > > > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > > > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > > > > Hi, > > > > > > > > > > I have a system that''s using a dracut-generated initramfs to mount a > > > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > > > > noticed that the process of mounting the root filesystem takes much > > > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > Ok drop that previous patch and give this one a whirl, it helped on my laptop. > This is only half of the problem AFAICS, but it''s the easier half to fix, in > the meantime I need to lock down why we''re not writing out cache for a bunch of > block groups, but thats trickier since the messages I need are spit out while > I''m shutting down, so I need to get creative. Let me know if/how much this > helps. Thanks,This one brings the mount time right down on my laptop, it''s back to around 0.5 seconds, same as 3.3.x. Tested-by: Calvin Walton <calvin.walton@kepstin.ca> I''ll keep following this email thread; let me know if you have any other patches that you want me to try out. -- Calvin Walton <calvin.walton@kepstin.ca> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Apr 10, 2012 at 12:04:00PM -0400, Calvin Walton wrote:> On Tue, 2012-04-10 at 11:16 -0400, Josef Bacik wrote: > This one brings the mount time right down on my laptop, it''s back to > around 0.5 seconds, same as 3.3.x.Did you notice any change during umount? david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2012-04-10 at 18:29 +0200, David Sterba wrote:> On Tue, Apr 10, 2012 at 12:04:00PM -0400, Calvin Walton wrote: > > On Tue, 2012-04-10 at 11:16 -0400, Josef Bacik wrote: > > This one brings the mount time right down on my laptop, it''s back to > > around 0.5 seconds, same as 3.3.x. > > Did you notice any change during umount?That''s hard to tell, given that it''s the root filesystem on my laptop (and the only btrfs filesystem on the computer). I don''t think it added very much, if any time to rebooting my computer - but that''s a remount read-only, not an unmount. -- Calvin Walton <calvin.walton@kepstin.ca> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Apr 10, 2012 at 5:16 PM, Josef Bacik <josef@redhat.com> wrote:> On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: >> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: >> > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: >> > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: >> > > > Hi, >> > > > >> > > > I have a system that''s using a dracut-generated initramfs to mount a >> > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve >> > > > noticed that the process of mounting the root filesystem takes much >> > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! >> >> > > And the bisect results are in: >> > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit >> > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff >> > > Author: Josef Bacik <josef@redhat.com> >> > > Date: Fri Jan 13 15:27:45 2012 -0500 >> > > >> > > Btrfs: remove the ideal caching code> >> > >> > Ok can you give this a whirl? You are going to have to boot/reboot a few times >> > to let the cache get re-generated again to make sure it''s taken effect, but >> > hopefully this will help out. Thanks, >> >> Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with >> this patch applied I''m still seeing the same delay. >> > > Ok drop that previous patch and give this one a whirl, it helped on my laptop. > This is only half of the problem AFAICS, but it''s the easier half to fix, in > the meantime I need to lock down why we''re not writing out cache for a bunch of > block groups, but thats trickier since the messages I need are spit out while > I''m shutting down, so I need to get creative. Let me know if/how much this > helps. Thanks,i have tried your patch and my system still needs several minutes to boot until it can be used. Also tried to reboot several times - it doesn''t look like its getting better. The last thing the system does when its shutting down is a read-only remount of "/" so no umount. Booting was much faster before i pulled for-linus a few weeks ago but i couldn''t find the time to bisect it yet .. please also look at the attached dmesg.txt. this is an core i3 system with 2x2TB BTRFS RAID1 and lots of home directories and snapshots. I''m going to test this patch on twenty more computers but with smaller HDDs and less files and see if it helps to speed up their boot times. Ahmet
On Mon, Apr 9, 2012 at 10:53 AM, Calvin Walton <calvin.walton@kepstin.ca> wrote:> > Hi, > > I have a system that''s using a dracut-generated initramfs to mount a > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > noticed that the process of mounting the root filesystem takes much > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > Does anyone have any ideas? I''m going to try bisecting the issue to see > if I can narrow down the cause. I''ve included excerpts from dmesg of the > bad and good kernels here, and attached the complete dmesg from the bad > kernel, in case it has anything interesting that I''ve trimmed out here. > > slow: > [ 0.000000] Linux version 3.4.0-rc2 (cwalton@ayu) (gcc version 4.6.3 > (Exherbo gcc-4.6.3) ) #57 SMP PREEMPT Mon Apr 9 11:19:43 EDT 2012 > [ 0.000000] Command line: > root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 > rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime > mtrr_chunk_size=512M quiet > <snip> > [ 1.058257] udevd[701]: starting version 182 > [ 1.389606] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 1.498659] dracut: Checking, if btrfs device complete > [ 1.644808] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 1.647993] btrfs: disk space caching is enabled > [ 2.180836] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 2.181663] btrfs: use lzo compression > [ 2.181667] btrfs: enabling auto defrag > [ 2.181670] btrfs: enabling inode map caching > [ 2.181672] btrfs: disk space caching is enabled > [ 2.697845] dracut: Checking btrfs: > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > [ 2.699999] dracut: trying to mount > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > [ 2.702637] device label Linux data devid 1 transid 611923 /dev/sda4 > [ 2.704376] btrfs: disk space caching is enabled > [ 3.081720] dracut: btrfs: > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 is clean > [ 29.934639] dracut: Remounting > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 with -o > subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime,ro > [ 29.936810] device label Linux data devid 1 transid 611926 /dev/sda4 > [ 29.937720] btrfs: use lzo compression > [ 29.937726] btrfs: enabling auto defrag > [ 29.937733] btrfs: enabling inode map caching > [ 29.937735] btrfs: disk space caching is enabled > [ 30.388066] dracut: Mounted root filesystem /dev/sda4 > [ 30.461884] dracut: Switching root > [ 31.241729] udevd[1322]: starting version 182 > [ 31.422905] btrfs: use lzo compression > [ 31.422909] btrfs: enabling auto defrag > [ 31.422913] btrfs: enabling inode map caching > [ 31.422915] btrfs: disk space caching is enabled > > vs fast: > [ 0.000000] Linux version 3.3.1 (cwalton@ayu) (gcc version 4.6.3 > (Exherbo gcc-4.6.3) ) #53 SMP PREEMPT Wed Apr 4 01:20:47 EDT 2012 > [ 0.000000] Command line: > root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 > rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatimeCalvin, I noticed that you have the space_cache mount option set. I remember josef saying that it is a one-time option where it''s a permanent change after mounting for the first time, therefore each subsequent mount will not require the option. In fact, if the option is specified again, the space_cache is reset. Is this still up-to-date info?> mtrr_chunk_size=512M quiet > <snip> > [ 1.060117] udevd[695]: starting version 182 > [ 1.344328] device label Linux data devid 1 transid 611917 /dev/sda4 > [ 1.384888] dracut: Checking, if btrfs device complete > [ 1.405121] device label Linux data devid 1 transid 611917 /dev/sda4 > [ 1.405905] btrfs: disk space caching is enabled > [ 1.971953] device label Linux data devid 1 transid 611917 /dev/sda4 > [ 1.972519] btrfs: use lzo compression > [ 1.972523] btrfs: enabling auto defrag > [ 1.972526] btrfs: enabling inode map caching > [ 1.972528] btrfs: disk space caching is enabled > [ 2.528540] dracut: Checking btrfs: > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > [ 2.530725] dracut: trying to mount > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > [ 2.558285] device label Linux data devid 1 transid 611917 /dev/sda4 > [ 2.559087] btrfs: disk space caching is enabled > [ 3.013103] dracut: btrfs: > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 is clean > [ 3.518807] dracut: Remounting > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 with -o > subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime,ro > [ 3.520031] device label Linux data devid 1 transid 611920 /dev/sda4 > [ 3.520680] btrfs: use lzo compression > [ 3.520684] btrfs: enabling auto defrag > [ 3.520688] btrfs: enabling inode map caching > [ 3.520689] btrfs: disk space caching is enabled > [ 4.066285] dracut: Mounted root filesystem /dev/sda4 > [ 4.120861] dracut: Switching root > [ 4.995934] btrfs: use lzo compression > [ 4.995938] btrfs: enabling auto defrag > [ 4.995942] btrfs: enabling inode map caching > [ 4.995943] btrfs: disk space caching is enabled > [ 5.010473] udevd[1330]: starting version 182 > > -- > Calvin Walton <calvin.walton@kepstin.ca>-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Apr 11, 2012 at 05:26:29PM +0200, Ahmet Inan wrote:> On Tue, Apr 10, 2012 at 5:16 PM, Josef Bacik <josef@redhat.com> wrote: > > On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: > >> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: > >> > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > >> > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > >> > > > Hi, > >> > > > > >> > > > I have a system that''s using a dracut-generated initramfs to mount a > >> > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > >> > > > noticed that the process of mounting the root filesystem takes much > >> > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > >> > >> > > And the bisect results are in: > >> > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > >> > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > >> > > Author: Josef Bacik <josef@redhat.com> > >> > > Date: Fri Jan 13 15:27:45 2012 -0500 > >> > > > >> > > Btrfs: remove the ideal caching code> > >> > > >> > Ok can you give this a whirl? You are going to have to boot/reboot a few times > >> > to let the cache get re-generated again to make sure it''s taken effect, but > >> > hopefully this will help out. Thanks, > >> > >> Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with > >> this patch applied I''m still seeing the same delay. > >> > > > > Ok drop that previous patch and give this one a whirl, it helped on my laptop. > > This is only half of the problem AFAICS, but it''s the easier half to fix, in > > the meantime I need to lock down why we''re not writing out cache for a bunch of > > block groups, but thats trickier since the messages I need are spit out while > > I''m shutting down, so I need to get creative. Let me know if/how much this > > helps. Thanks, > > i have tried your patch and my system still needs several minutes to boot > until it can be used. > Also tried to reboot several times - it doesn''t look like its getting better. > The last thing the system does when its shutting down is a read-only > remount of "/" so no umount. > Booting was much faster before i pulled for-linus a few weeks ago but > i couldn''t find the time to bisect it yet .. > > please also look at the attached dmesg.txt. > this is an core i3 system with 2x2TB BTRFS RAID1 and lots of > home directories and snapshots. > > I''m going to test this patch on twenty more computers but with > smaller HDDs and less files and see if it helps to speed up their > boot times. >Ok looks like you are running into a different problem. Could you maybe run bootchart and upload the resulting png somewhere so I can look and see what all is running while you boot? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Apr 11, 2012 at 11:38:33AM -0500, Chester wrote:> On Mon, Apr 9, 2012 at 10:53 AM, Calvin Walton <calvin.walton@kepstin.ca> wrote: > > > > Hi, > > > > I have a system that''s using a dracut-generated initramfs to mount a > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > noticed that the process of mounting the root filesystem takes much > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > > > Does anyone have any ideas? I''m going to try bisecting the issue to see > > if I can narrow down the cause. I''ve included excerpts from dmesg of the > > bad and good kernels here, and attached the complete dmesg from the bad > > kernel, in case it has anything interesting that I''ve trimmed out here. > > > > slow: > > [ 0.000000] Linux version 3.4.0-rc2 (cwalton@ayu) (gcc version 4.6.3 > > (Exherbo gcc-4.6.3) ) #57 SMP PREEMPT Mon Apr 9 11:19:43 EDT 2012 > > [ 0.000000] Command line: > > root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 > > rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime > > mtrr_chunk_size=512M quiet > > <snip> > > [ 1.058257] udevd[701]: starting version 182 > > [ 1.389606] device label Linux data devid 1 transid 611923 /dev/sda4 > > [ 1.498659] dracut: Checking, if btrfs device complete > > [ 1.644808] device label Linux data devid 1 transid 611923 /dev/sda4 > > [ 1.647993] btrfs: disk space caching is enabled > > [ 2.180836] device label Linux data devid 1 transid 611923 /dev/sda4 > > [ 2.181663] btrfs: use lzo compression > > [ 2.181667] btrfs: enabling auto defrag > > [ 2.181670] btrfs: enabling inode map caching > > [ 2.181672] btrfs: disk space caching is enabled > > [ 2.697845] dracut: Checking btrfs: > > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > > [ 2.699999] dracut: trying to mount > > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 > > [ 2.702637] device label Linux data devid 1 transid 611923 /dev/sda4 > > [ 2.704376] btrfs: disk space caching is enabled > > [ 3.081720] dracut: btrfs: > > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 is clean > > [ 29.934639] dracut: Remounting > > /dev/disk/by-uuid/43969cd0-4aca-4297-bfbe-952a692f7d55 with -o > > subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime,ro > > [ 29.936810] device label Linux data devid 1 transid 611926 /dev/sda4 > > [ 29.937720] btrfs: use lzo compression > > [ 29.937726] btrfs: enabling auto defrag > > [ 29.937733] btrfs: enabling inode map caching > > [ 29.937735] btrfs: disk space caching is enabled > > [ 30.388066] dracut: Mounted root filesystem /dev/sda4 > > [ 30.461884] dracut: Switching root > > [ 31.241729] udevd[1322]: starting version 182 > > [ 31.422905] btrfs: use lzo compression > > [ 31.422909] btrfs: enabling auto defrag > > [ 31.422913] btrfs: enabling inode map caching > > [ 31.422915] btrfs: disk space caching is enabled > > > > vs fast: > > [ 0.000000] Linux version 3.3.1 (cwalton@ayu) (gcc version 4.6.3 > > (Exherbo gcc-4.6.3) ) #53 SMP PREEMPT Wed Apr 4 01:20:47 EDT 2012 > > [ 0.000000] Command line: > > root=UUID=43969cd0-4aca-4297-bfbe-952a692f7d55 > > rootflags=subvolid=262,compress=lzo,autodefrag,space_cache,inode_cache,noatime > > Calvin, I noticed that you have the space_cache mount option set. I > remember josef saying that it is a one-time option where it''s a > permanent change after mounting for the first time, therefore each > subsequent mount will not require the option. In fact, if the option > is specified again, the space_cache is reset. > > Is this still up-to-date info? >No you can leave space_cache set, there''s another option for resetting the space cache, clear_cache iirc. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Apr 11, 2012 at 7:04 PM, Josef Bacik <josef@redhat.com> wrote:> On Wed, Apr 11, 2012 at 05:26:29PM +0200, Ahmet Inan wrote: >> On Tue, Apr 10, 2012 at 5:16 PM, Josef Bacik <josef@redhat.com> wrote: >> > On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: >> >> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: >> >> > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: >> >> > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: >> >> > > > Hi, >> >> > > > >> >> > > > I have a system that''s using a dracut-generated initramfs to mount a >> >> > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve >> >> > > > noticed that the process of mounting the root filesystem takes much >> >> > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! >> >> >> >> > > And the bisect results are in: >> >> > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit >> >> > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff >> >> > > Author: Josef Bacik <josef@redhat.com> >> >> > > Date: Fri Jan 13 15:27:45 2012 -0500 >> >> > > >> >> > > Btrfs: remove the ideal caching code> >> >> > >> >> > Ok can you give this a whirl? You are going to have to boot/reboot a few times >> >> > to let the cache get re-generated again to make sure it''s taken effect, but >> >> > hopefully this will help out. Thanks, >> >> >> >> Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with >> >> this patch applied I''m still seeing the same delay. >> >> >> > >> > Ok drop that previous patch and give this one a whirl, it helped on my laptop. >> > This is only half of the problem AFAICS, but it''s the easier half to fix, in >> > the meantime I need to lock down why we''re not writing out cache for a bunch of >> > block groups, but thats trickier since the messages I need are spit out while >> > I''m shutting down, so I need to get creative. Let me know if/how much this >> > helps. Thanks, >> >> i have tried your patch and my system still needs several minutes to boot >> until it can be used. >> Also tried to reboot several times - it doesn''t look like its getting better. >> The last thing the system does when its shutting down is a read-only >> remount of "/" so no umount. >> Booting was much faster before i pulled for-linus a few weeks ago but >> i couldn''t find the time to bisect it yet .. >> >> please also look at the attached dmesg.txt. >> this is an core i3 system with 2x2TB BTRFS RAID1 and lots of >> home directories and snapshots. >> >> I''m going to test this patch on twenty more computers but with >> smaller HDDs and less files and see if it helps to speed up their >> boot times. >> > > Ok looks like you are running into a different problem. Could you maybe run > bootchart and upload the resulting png somewhere so I can look and see what all > is running while you boot? Thanks,http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart.png i have tried your patch now on the twenty more computers i mentioned and still it takes a minute to remount rw "/" on those, even after several reboots. Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 12, 2012 at 11:22:51AM +0200, Ahmet Inan wrote:> On Wed, Apr 11, 2012 at 7:04 PM, Josef Bacik <josef@redhat.com> wrote: > > On Wed, Apr 11, 2012 at 05:26:29PM +0200, Ahmet Inan wrote: > >> On Tue, Apr 10, 2012 at 5:16 PM, Josef Bacik <josef@redhat.com> wrote: > >> > On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: > >> >> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: > >> >> > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > >> >> > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > >> >> > > > Hi, > >> >> > > > > >> >> > > > I have a system that''s using a dracut-generated initramfs to mount a > >> >> > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > >> >> > > > noticed that the process of mounting the root filesystem takes much > >> >> > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > >> >> > >> >> > > And the bisect results are in: > >> >> > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > >> >> > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > >> >> > > Author: Josef Bacik <josef@redhat.com> > >> >> > > Date: Fri Jan 13 15:27:45 2012 -0500 > >> >> > > > >> >> > > Btrfs: remove the ideal caching code> > >> >> > > >> >> > Ok can you give this a whirl? You are going to have to boot/reboot a few times > >> >> > to let the cache get re-generated again to make sure it''s taken effect, but > >> >> > hopefully this will help out. Thanks, > >> >> > >> >> Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with > >> >> this patch applied I''m still seeing the same delay. > >> >> > >> > > >> > Ok drop that previous patch and give this one a whirl, it helped on my laptop. > >> > This is only half of the problem AFAICS, but it''s the easier half to fix, in > >> > the meantime I need to lock down why we''re not writing out cache for a bunch of > >> > block groups, but thats trickier since the messages I need are spit out while > >> > I''m shutting down, so I need to get creative. Let me know if/how much this > >> > helps. Thanks, > >> > >> i have tried your patch and my system still needs several minutes to boot > >> until it can be used. > >> Also tried to reboot several times - it doesn''t look like its getting better. > >> The last thing the system does when its shutting down is a read-only > >> remount of "/" so no umount. > >> Booting was much faster before i pulled for-linus a few weeks ago but > >> i couldn''t find the time to bisect it yet .. > >> > >> please also look at the attached dmesg.txt. > >> this is an core i3 system with 2x2TB BTRFS RAID1 and lots of > >> home directories and snapshots. > >> > >> I''m going to test this patch on twenty more computers but with > >> smaller HDDs and less files and see if it helps to speed up their > >> boot times. > >> > > > > Ok looks like you are running into a different problem. Could you maybe run > > bootchart and upload the resulting png somewhere so I can look and see what all > > is running while you boot? Thanks, > > http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart.png > > i have tried your patch now on the twenty more computers i mentioned and > still it takes a minute to remount rw "/" on those, even after several reboots. >Oops responding to the whole list this time.. Um ouch your system appears to not be doing anything for like 300 seconds but sitting there. Can you hook up a console and capture sysrq+w while thats going on? Also you are mounting with -o space_cache right? Can I see your dmesg to make sure it''s doing what it''s supposed to? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 12, 2012 at 09:37:48AM -0400, Josef Bacik wrote:> On Thu, Apr 12, 2012 at 11:22:51AM +0200, Ahmet Inan wrote: > > On Wed, Apr 11, 2012 at 7:04 PM, Josef Bacik <josef@redhat.com> wrote: > > > On Wed, Apr 11, 2012 at 05:26:29PM +0200, Ahmet Inan wrote: > > >> On Tue, Apr 10, 2012 at 5:16 PM, Josef Bacik <josef@redhat.com> wrote: > > >> > On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: > > >> >> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: > > >> >> > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: > > >> >> > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: > > >> >> > > > Hi, > > >> >> > > > > > >> >> > > > I have a system that''s using a dracut-generated initramfs to mount a > > >> >> > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve > > >> >> > > > noticed that the process of mounting the root filesystem takes much > > >> >> > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! > > >> >> > > >> >> > > And the bisect results are in: > > >> >> > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit > > >> >> > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff > > >> >> > > Author: Josef Bacik <josef@redhat.com> > > >> >> > > Date: Fri Jan 13 15:27:45 2012 -0500 > > >> >> > > > > >> >> > > Btrfs: remove the ideal caching code> > > >> >> > > > >> >> > Ok can you give this a whirl? You are going to have to boot/reboot a few times > > >> >> > to let the cache get re-generated again to make sure it''s taken effect, but > > >> >> > hopefully this will help out. Thanks, > > >> >> > > >> >> Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with > > >> >> this patch applied I''m still seeing the same delay. > > >> >> > > >> > > > >> > Ok drop that previous patch and give this one a whirl, it helped on my laptop. > > >> > This is only half of the problem AFAICS, but it''s the easier half to fix, in > > >> > the meantime I need to lock down why we''re not writing out cache for a bunch of > > >> > block groups, but thats trickier since the messages I need are spit out while > > >> > I''m shutting down, so I need to get creative. Let me know if/how much this > > >> > helps. Thanks, > > >> > > >> i have tried your patch and my system still needs several minutes to boot > > >> until it can be used. > > >> Also tried to reboot several times - it doesn''t look like its getting better. > > >> The last thing the system does when its shutting down is a read-only > > >> remount of "/" so no umount. > > >> Booting was much faster before i pulled for-linus a few weeks ago but > > >> i couldn''t find the time to bisect it yet .. > > >> > > >> please also look at the attached dmesg.txt. > > >> this is an core i3 system with 2x2TB BTRFS RAID1 and lots of > > >> home directories and snapshots. > > >> > > >> I''m going to test this patch on twenty more computers but with > > >> smaller HDDs and less files and see if it helps to speed up their > > >> boot times. > > >> > > > > > > Ok looks like you are running into a different problem. Could you maybe run > > > bootchart and upload the resulting png somewhere so I can look and see what all > > > is running while you boot? Thanks, > > > > http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart.png > > > > i have tried your patch now on the twenty more computers i mentioned and > > still it takes a minute to remount rw "/" on those, even after several reboots. > > > > Oops responding to the whole list this time.. > > Um ouch your system appears to not be doing anything for like 300 seconds but > sitting there. Can you hook up a console and capture sysrq+w while thats going > on? Also you are mounting with -o space_cache right? Can I see your dmesg to > make sure it''s doing what it''s supposed to? Thanks, >Ok you don''t actually have space_cache enabled it looks like, make sure to add space_cache to your fstab so it gets enabled, and then reboot a few times to make sure everything gets cached right and then it should help. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 12, 2012 at 4:23 PM, Josef Bacik <josef@redhat.com> wrote:> On Thu, Apr 12, 2012 at 09:37:48AM -0400, Josef Bacik wrote: >> On Thu, Apr 12, 2012 at 11:22:51AM +0200, Ahmet Inan wrote: >> > On Wed, Apr 11, 2012 at 7:04 PM, Josef Bacik <josef@redhat.com> wrote: >> > > On Wed, Apr 11, 2012 at 05:26:29PM +0200, Ahmet Inan wrote: >> > >> On Tue, Apr 10, 2012 at 5:16 PM, Josef Bacik <josef@redhat.com> wrote: >> > >> > On Mon, Apr 09, 2012 at 05:20:46PM -0400, Calvin Walton wrote: >> > >> >> On Mon, 2012-04-09 at 16:54 -0400, Josef Bacik wrote: >> > >> >> > On Mon, Apr 09, 2012 at 01:10:04PM -0400, Calvin Walton wrote: >> > >> >> > > On Mon, 2012-04-09 at 11:53 -0400, Calvin Walton wrote: >> > >> >> > > > Hi, >> > >> >> > > > >> > >> >> > > > I have a system that''s using a dracut-generated initramfs to mount a >> > >> >> > > > btrfs root. After upgrading to kernel 3.4.0-rc2 to test it out, I''ve >> > >> >> > > > noticed that the process of mounting the root filesystem takes much >> > >> >> > > > longer with 3.4.0-rc2 than it did with 3.3.1 - nearly 30 seconds slower! >> > >> >> >> > >> >> > > And the bisect results are in: >> > >> >> > > 285ff5af6ce358e73f53b55c9efadd4335f4c2ff is the first bad commit >> > >> >> > > commit 285ff5af6ce358e73f53b55c9efadd4335f4c2ff >> > >> >> > > Author: Josef Bacik <josef@redhat.com> >> > >> >> > > Date: Fri Jan 13 15:27:45 2012 -0500 >> > >> >> > > >> > >> >> > > Btrfs: remove the ideal caching code> >> > >> >> > >> > >> >> > Ok can you give this a whirl? You are going to have to boot/reboot a few times >> > >> >> > to let the cache get re-generated again to make sure it''s taken effect, but >> > >> >> > hopefully this will help out. Thanks, >> > >> >> >> > >> >> Unfortunately, it doesn''t seem to help. Even after 3 or 4 reboots with >> > >> >> this patch applied I''m still seeing the same delay. >> > >> >> >> > >> > >> > >> > Ok drop that previous patch and give this one a whirl, it helped on my laptop. >> > >> > This is only half of the problem AFAICS, but it''s the easier half to fix, in >> > >> > the meantime I need to lock down why we''re not writing out cache for a bunch of >> > >> > block groups, but thats trickier since the messages I need are spit out while >> > >> > I''m shutting down, so I need to get creative. Let me know if/how much this >> > >> > helps. Thanks, >> > >> >> > >> i have tried your patch and my system still needs several minutes to boot >> > >> until it can be used. >> > >> Also tried to reboot several times - it doesn''t look like its getting better. >> > >> The last thing the system does when its shutting down is a read-only >> > >> remount of "/" so no umount. >> > >> Booting was much faster before i pulled for-linus a few weeks ago but >> > >> i couldn''t find the time to bisect it yet .. >> > >> >> > >> please also look at the attached dmesg.txt. >> > >> this is an core i3 system with 2x2TB BTRFS RAID1 and lots of >> > >> home directories and snapshots. >> > >> >> > >> I''m going to test this patch on twenty more computers but with >> > >> smaller HDDs and less files and see if it helps to speed up their >> > >> boot times. >> > >> >> > > >> > > Ok looks like you are running into a different problem. Could you maybe run >> > > bootchart and upload the resulting png somewhere so I can look and see what all >> > > is running while you boot? Thanks, >> > >> > http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart.png >> > >> > i have tried your patch now on the twenty more computers i mentioned and >> > still it takes a minute to remount rw "/" on those, even after several reboots. >> > >> >> Oops responding to the whole list this time.. >> >> Um ouch your system appears to not be doing anything for like 300 seconds but >> sitting there. Can you hook up a console and capture sysrq+w while thats going >> on? Also you are mounting with -o space_cache right? Can I see your dmesg to >> make sure it''s doing what it''s supposed to? Thanks, >> > > Ok you don''t actually have space_cache enabled it looks like, make sure to add > space_cache to your fstab so it gets enabled, and then reboot a few times to > make sure everything gets cached right and then it should help. Thanks,now i did enable space_cache in fstab and rebooted 4 times, still no improvement: http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_space_cache.png is it vital to put this space_cache option to the boot argument as well? mounting "/" readonly in initramfs and booting to it (until remount "/" rw) is quite fast. dmesg and fstab attached as requested. Ahmet
> dmesg and fstab attached as requested.Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote:>> dmesg and fstab attached as requested. > > Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period.here. i guess i should also increase dmesg history size next time. other than the slow boot, everything seems normal after 10-20minutes. fyi: the space_cache option is not really helping with those twenty computers. the one thing i observed is, that sometimes they reboot fast and only to reboot slow again after that. Ahmet
These are usb disks? Does that failure at 12.241517 (or related) happen every time? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Apr 13, 2012 at 2:57 PM, cwillu <cwillu@cwillu.com> wrote:> These are usb disks? Does that failure at 12.241517 (or related) > happen every time?no, 0CCD:0052 is the webcam. i dont have the modules for the webcam in initramfs, thats why. the real slowness is around 33.305370 The 2 SATA Disks are fine. I have many more Computers with varying Hardware but the same Kernel and see the same Problems there too. Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Apr 13, 2012 at 02:26:19PM +0200, Ahmet Inan wrote:> On Fri, Apr 13, 2012 at 1:22 PM, Ahmet Inan > <ainan@mathematik.uni-freiburg.de> wrote: > > On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote: > >>> dmesg and fstab attached as requested. > >> > >> Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. > > > > here. > > > > i guess i should also increase dmesg history size next time. > > other than the slow boot, everything seems normal after 10-20minutes. > > > > fyi: the space_cache option is not really helping with those > > twenty computers. the one thing i observed is, that sometimes > > they reboot fast and only to reboot slow again after that. > > sorry for spaming the list with my dmesg files, > please tell me, how i could do better. > > here a more complete dmesg.Hrm so you are getting blocked task warnings just trying to mount the filesystem, so either your disk is really really really slow or there''s something bigger going on. Let me think about this some and I''ll get back to you. Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Apr 13, 2012 at 3:47 PM, Josef Bacik <josef@redhat.com> wrote:> On Fri, Apr 13, 2012 at 02:26:19PM +0200, Ahmet Inan wrote: >> On Fri, Apr 13, 2012 at 1:22 PM, Ahmet Inan >> <ainan@mathematik.uni-freiburg.de> wrote: >> > On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote: >> >>> dmesg and fstab attached as requested. >> >> >> >> Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. >> > >> > here. >> > >> > i guess i should also increase dmesg history size next time. >> > other than the slow boot, everything seems normal after 10-20minutes. >> > >> > fyi: the space_cache option is not really helping with those >> > twenty computers. the one thing i observed is, that sometimes >> > they reboot fast and only to reboot slow again after that. >> >> sorry for spaming the list with my dmesg files, >> please tell me, how i could do better. >> >> here a more complete dmesg. > > Hrm so you are getting blocked task warnings just trying to mount the > filesystem, so either your disk is really really really slow or there''s > something bigger going on. Let me think about this some and I''ll get back to > you.Josef, i finally found out something: btrfs in kernel => fast boot btrfs as module => very slow boot and here see the results: http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_btrfs_in_kernel.png this is much, much better! I dont understand why btrfs as a module performs that bad on rotating disk. On SSD i had no issues and also i never used space_cache before. Im going to deploy the new kernel on our systems next week with btrfs in-kernel and come back with the results. Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Sergei Trofimovich
2012-Apr-22 07:59 UTC
Re: Boot speed/mount time regression with 3.4.0-rc2
On Sat, 21 Apr 2012 22:54:41 +0200 Ahmet Inan <ainan@mathematik.uni-freiburg.de> wrote:> On Fri, Apr 13, 2012 at 3:47 PM, Josef Bacik <josef@redhat.com> wrote: > > On Fri, Apr 13, 2012 at 02:26:19PM +0200, Ahmet Inan wrote: > >> On Fri, Apr 13, 2012 at 1:22 PM, Ahmet Inan > >> <ainan@mathematik.uni-freiburg.de> wrote: > >> > On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote: > >> >>> dmesg and fstab attached as requested. > >> >> > >> >> Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. > >> > > >> > here. > >> > > >> > i guess i should also increase dmesg history size next time. > >> > other than the slow boot, everything seems normal after 10-20minutes. > >> > > >> > fyi: the space_cache option is not really helping with those > >> > twenty computers. the one thing i observed is, that sometimes > >> > they reboot fast and only to reboot slow again after that. > >> > >> sorry for spaming the list with my dmesg files, > >> please tell me, how i could do better. > >> > >> here a more complete dmesg. > > > > Hrm so you are getting blocked task warnings just trying to mount the > > filesystem, so either your disk is really really really slow or there''s > > something bigger going on. Let me think about this some and I''ll get back to > > you. > > Josef, i finally found out something: > > btrfs in kernel => fast boot > btrfs as module => very slow boot > > and here see the results: > http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_btrfs_in_kernel.png > > this is much, much better! > > I dont understand why btrfs as a module performs that bad on rotating disk. > On SSD i had no issues and also i never used space_cache before.There is a suspiction that putting space_cache (or anything else) to fstab for rootfs does not get applied due to a bug: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg16039.html Do all of your affected computers run 3.4.0-rc kernels? -- Sergei
On Sun, Apr 22, 2012 at 9:59 AM, Sergei Trofimovich <slyich@gmail.com> wrote:> On Sat, 21 Apr 2012 22:54:41 +0200 > Ahmet Inan <ainan@mathematik.uni-freiburg.de> wrote: > >> On Fri, Apr 13, 2012 at 3:47 PM, Josef Bacik <josef@redhat.com> wrote: >> > On Fri, Apr 13, 2012 at 02:26:19PM +0200, Ahmet Inan wrote: >> >> On Fri, Apr 13, 2012 at 1:22 PM, Ahmet Inan >> >> <ainan@mathematik.uni-freiburg.de> wrote: >> >> > On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote: >> >> >>> dmesg and fstab attached as requested. >> >> >> >> >> >> Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. >> >> > >> >> > here. >> >> > >> >> > i guess i should also increase dmesg history size next time. >> >> > other than the slow boot, everything seems normal after 10-20minutes. >> >> > >> >> > fyi: the space_cache option is not really helping with those >> >> > twenty computers. the one thing i observed is, that sometimes >> >> > they reboot fast and only to reboot slow again after that. >> >> >> >> sorry for spaming the list with my dmesg files, >> >> please tell me, how i could do better. >> >> >> >> here a more complete dmesg. >> > >> > Hrm so you are getting blocked task warnings just trying to mount the >> > filesystem, so either your disk is really really really slow or there''s >> > something bigger going on. Let me think about this some and I''ll get back to >> > you. >> >> Josef, i finally found out something: >> >> btrfs in kernel => fast boot >> btrfs as module => very slow boot >> >> and here see the results: >> http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_btrfs_in_kernel.png >> >> this is much, much better! >> >> I dont understand why btrfs as a module performs that bad on rotating disk. >> On SSD i had no issues and also i never used space_cache before. > > There is a suspiction that putting space_cache (or anything else) to fstab for rootfs > does not get applied due to a bug: > > http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg16039.htmlThats right. space_cache only got enabled for real, after ive set the option in the boot argument once. Even if it looks like it gets enabled in dmesg via /etc/fstab, the next boot without space_cache doesnt show space_cache enabled. But this is another problem i will look into later. At the moment the biggest problem is having btrfs as a module.> Do all of your affected computers run 3.4.0-rc kernels?They run vanilla/linux-3.3.2 with btrfs/for-linus. The unaffected computers have <= vanilla/linux-3.2.9 with btrfs/for-linus from 2 months ago or so. Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Apr 22, 2012 at 11:16 AM, Ahmet Inan <ainan@mathematik.uni-freiburg.de> wrote:> On Sun, Apr 22, 2012 at 9:59 AM, Sergei Trofimovich <slyich@gmail.com> wrote: >> On Sat, 21 Apr 2012 22:54:41 +0200 >> Ahmet Inan <ainan@mathematik.uni-freiburg.de> wrote: >> >>> On Fri, Apr 13, 2012 at 3:47 PM, Josef Bacik <josef@redhat.com> wrote: >>> > On Fri, Apr 13, 2012 at 02:26:19PM +0200, Ahmet Inan wrote: >>> >> On Fri, Apr 13, 2012 at 1:22 PM, Ahmet Inan >>> >> <ainan@mathematik.uni-freiburg.de> wrote: >>> >> > On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote: >>> >> >>> dmesg and fstab attached as requested. >>> >> >> >>> >> >> Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. >>> >> > >>> >> > here. >>> >> > >>> >> > i guess i should also increase dmesg history size next time. >>> >> > other than the slow boot, everything seems normal after 10-20minutes. >>> >> > >>> >> > fyi: the space_cache option is not really helping with those >>> >> > twenty computers. the one thing i observed is, that sometimes >>> >> > they reboot fast and only to reboot slow again after that. >>> >> >>> >> sorry for spaming the list with my dmesg files, >>> >> please tell me, how i could do better. >>> >> >>> >> here a more complete dmesg. >>> > >>> > Hrm so you are getting blocked task warnings just trying to mount the >>> > filesystem, so either your disk is really really really slow or there''s >>> > something bigger going on. Let me think about this some and I''ll get back to >>> > you. >>> >>> Josef, i finally found out something: >>> >>> btrfs in kernel => fast boot >>> btrfs as module => very slow boot >>> >>> and here see the results: >>> http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_btrfs_in_kernel.png >>> >>> this is much, much better! >>> >>> I dont understand why btrfs as a module performs that bad on rotating disk. >>> On SSD i had no issues and also i never used space_cache before. >> >> There is a suspiction that putting space_cache (or anything else) to fstab for rootfs >> does not get applied due to a bug: >> >> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg16039.html > > Thats right. > > space_cache only got enabled for real, after ive set the option in the > boot argument once. > Even if it looks like it gets enabled in dmesg via /etc/fstab, > the next boot without space_cache doesnt show space_cache enabled. > > But this is another problem i will look into later.I just pulled for-linus and thank you for that patch Sergei. Now i can enable space_cache on all systems without playing with bootarguments or fstab: # mount -o remount,space_cache / it speeds up boot times after a couple reboots. perfect!> At the moment the biggest problem is having btrfs as a module.To be sure i just rebooted that troublesome computer again with btrfs as module and with space_cache enabled: http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_btrfs_as_module.png So it looks like btrfs as module only amplified that space_cache problem .. And now that space_cache is enabled and working even btrfs as module works ok. Conclusion: btrfs in kernel + space_cache solves all my slow boot problems. Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Apr 21, 2012 at 10:54 PM, Ahmet Inan <ainan@mathematik.uni-freiburg.de> wrote:> On Fri, Apr 13, 2012 at 3:47 PM, Josef Bacik <josef@redhat.com> wrote: >> On Fri, Apr 13, 2012 at 02:26:19PM +0200, Ahmet Inan wrote: >>> On Fri, Apr 13, 2012 at 1:22 PM, Ahmet Inan >>> <ainan@mathematik.uni-freiburg.de> wrote: >>> > On Fri, Apr 13, 2012 at 8:49 AM, cwillu <cwillu@cwillu.com> wrote: >>> >>> dmesg and fstab attached as requested. >>> >> >>> >> Need dmesg after you''ve hit alt-sysrq-w a couple times during the slow period. >>> > >>> > here. >>> > >>> > i guess i should also increase dmesg history size next time. >>> > other than the slow boot, everything seems normal after 10-20minutes. >>> > >>> > fyi: the space_cache option is not really helping with those >>> > twenty computers. the one thing i observed is, that sometimes >>> > they reboot fast and only to reboot slow again after that. >>> >>> sorry for spaming the list with my dmesg files, >>> please tell me, how i could do better. >>> >>> here a more complete dmesg. >> >> Hrm so you are getting blocked task warnings just trying to mount the >> filesystem, so either your disk is really really really slow or there''s >> something bigger going on. Let me think about this some and I''ll get back to >> you. > > Josef, i finally found out something: > > btrfs in kernel => fast boot > btrfs as module => very slow boot > > and here see the results: > http://aam.mathematik.uni-freiburg.de/IAM/homepages/ainan/bootchart_btrfs_in_kernel.png > > this is much, much better! > > I dont understand why btrfs as a module performs that bad on rotating disk. > On SSD i had no issues and also i never used space_cache before. > > Im going to deploy the new kernel on our systems next week with > btrfs in-kernel and come back with the results.FYI: a 100 computers here with all kinds of hardware constellations work perfectly since the kernel upgrade (3.3.2 + for-linus) one week ago. We are very happy with the better btrfs performance. Also boot times are very short again. Even upgrading from different and very old Kernel versions worked flawless: vanilla linux-2.6.38.8: "Old style space inode found, converting." great :-) To make a long story short: Thanks a lot! Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html