Martin Steigerwald
2011-Dec-17 17:33 UTC
3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Hi! Finally I tried scrubbing the / BTRFS filesystem mentioned in the thread "speeding up slow btrfs filesystem". However the machine looks up hard then. It repeats the last few seconds of audio all over again, no mouse and no ssh connection anymore: deepdance:~> btrfs scrub start / scrub started on /, fsid […] (pid=5737) deepdance:~> Write failed: Broken pipe After the second attempt of doing this the machine stops on booting after the space cache enabled message. Then I get backtraced of hung tasks: http://martin-steigerwald.de/tmp/btrfs/2011-17-12-deepdance-hang-at-boot/ I am able to mount the filesystem from grml 2011.12-rc1 with 3.1 kernel: root@grml ~ # Start lvm2 Setting up LVM Volume Groups Reading all physical volumes. This may take a while... Found volume group "deepdance" using metadata type lvm2 3 logical volume(s) in volume group "deepdance" now active . root@grml ~ # mount /mnt/debian root@grml ~ # mount /mnt/home root@grml ~ # dmesg | tail [ 55.479303] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 64.352088] eth0: no IPv6 routers present [ 75.744318] fuse init (API version 7.17) [ 75.801245] ipmi message handler version 39.2 [ 90.213880] end_request: I/O error, dev fd0, sector 0 [ 229.117153] Btrfs loaded [ 229.117962] device label debian devid 1 transid 201769 /dev/mapper/deepdance-debian [ 229.577244] btrfs: disk space caching is enabled [ 245.375875] device label home devid 1 transid 72021 /dev/mapper/deepdance-home [ 245.433137] btrfs: disk space caching is enabled root@grml ~ # umount /mnt/{debian,home} root@grml ~ # cat /proc/version Linux version 3.1.0-2-grml-486 (Debian 3.1.0-2+grml.3) (ch@grml.org) (gcc version 4.6.2 (Debian 4.6.2-5) ) #1 Fri Dec 9 00:25:07 UTC 2011 But after that 3.2-rc4 still doesn´t boot. I am now trying to install a 3.1.0 debian kernel via chroot. Ok, after installing the 3.1.0 kernel it seems to work with 3.2.0-rc4 as well again. Its again converting the old style inodes. Thus I think on a sudden crash there may be an issue with the inode cache in 3.2-rc4 that switching to 3.1.0, writing something, and then back to 3.2.0-rc4 works around. But then I had this after upgrading from 3.0 to 3.2-rc4 as well. I didn´t report it here cause I thought it was a one-time issue. Hopefully you can make something out of the backtraces I tried to snapshot with my digicam. Should I be concerned about the state of the filesystem? I will not try scrubbing it again for now ;) BTW the performance of the BTRFS fs while updating the inode cache is abysmal. The machine is trying to boot for about 5 minutes now and still no KDM to see. Hmm, at least there is an SSH now. No nothing about these hard lock ups in syslog as I have suspected. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2011-Dec-17 18:11 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Samstag, 17. Dezember 2011 schrieb Martin Steigerwald:> Hi! > > Finally I tried scrubbing the / BTRFS filesystem mentioned in the > thread "speeding up slow btrfs filesystem". However the machine looks > up hard then. It repeats the last few seconds of audio all over again, > no mouse and no ssh connection anymore: > > deepdance:~> btrfs scrub start / > scrub started on /, fsid […] (pid=5737) > deepdance:~> Write failed: Broken pipeFWIW the T520 can scrub its / BTRFS successfully. merkaba:~> btrfs scrub status / scrub status for […] scrub started at Sat Dec 17 18:34:49 2011 and finished after 82 seconds total bytes scrubbed: 13.62GB with 0 errors -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2011-Dec-20 20:46 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Hi again! Any hints about this one? Since scrubbing worked okay on other machines, I can also redo the BTRFS filesystem on this machine. Maybe it really has gained a corruption. (Still scrubbing should not lock up the kernel hard, but…) Thanks, Martin Am Samstag, 17. Dezember 2011 schrieb Martin Steigerwald:> Hi! > > Finally I tried scrubbing the / BTRFS filesystem mentioned in the > thread "speeding up slow btrfs filesystem". However the machine looks > up hard then. It repeats the last few seconds of audio all over again, > no mouse and no ssh connection anymore: > > deepdance:~> btrfs scrub start / > scrub started on /, fsid […] (pid=5737) > deepdance:~> Write failed: Broken pipe > > > After the second attempt of doing this the machine stops on booting > after the space cache enabled message. Then I get backtraced of hung > tasks: > > http://martin-steigerwald.de/tmp/btrfs/2011-17-12-deepdance-hang-at-boo > t/ > > > I am able to mount the filesystem from grml 2011.12-rc1 with 3.1 > kernel: > > root@grml ~ # Start lvm2 > Setting up LVM Volume Groups Reading all physical volumes. This may > take a while... > Found volume group "deepdance" using metadata type lvm2 > 3 logical volume(s) in volume group "deepdance" now active > . > root@grml ~ # mount /mnt/debian > root@grml ~ # mount /mnt/home > root@grml ~ # dmesg | tail > [ 55.479303] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). > [ 64.352088] eth0: no IPv6 routers present > [ 75.744318] fuse init (API version 7.17) > [ 75.801245] ipmi message handler version 39.2 > [ 90.213880] end_request: I/O error, dev fd0, sector 0 > [ 229.117153] Btrfs loaded > [ 229.117962] device label debian devid 1 transid 201769 > /dev/mapper/deepdance-debian > [ 229.577244] btrfs: disk space caching is enabled > [ 245.375875] device label home devid 1 transid 72021 > /dev/mapper/deepdance-home > [ 245.433137] btrfs: disk space caching is enabled > root@grml ~ # umount /mnt/{debian,home} > > root@grml ~ # cat /proc/version > Linux version 3.1.0-2-grml-486 (Debian 3.1.0-2+grml.3) (ch@grml.org) > (gcc version 4.6.2 (Debian 4.6.2-5) ) #1 Fri Dec 9 00:25:07 UTC 2011 > > > But after that 3.2-rc4 still doesn´t boot. > > I am now trying to install a 3.1.0 debian kernel via chroot. > > Ok, after installing the 3.1.0 kernel it seems to work with 3.2.0-rc4 > as well again. Its again converting the old style inodes. Thus I think > on a sudden crash there may be an issue with the inode cache in > 3.2-rc4 that switching to 3.1.0, writing something, and then back to > 3.2.0-rc4 works around. > > But then I had this after upgrading from 3.0 to 3.2-rc4 as well. I > didn´t report it here cause I thought it was a one-time issue. > > Hopefully you can make something out of the backtraces I tried to > snapshot with my digicam. > > Should I be concerned about the state of the filesystem? > > I will not try scrubbing it again for now ;) > > BTW the performance of the BTRFS fs while updating the inode cache is > abysmal. The machine is trying to boot for about 5 minutes now and > still no KDM to see. Hmm, at least there is an SSH now. No nothing > about these hard lock ups in syslog as I have suspected. > > Thanks,-- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Jan-21 10:49 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Dienstag, 20. Dezember 2011 schrieb Martin Steigerwald:> Hi again![…]> Am Samstag, 17. Dezember 2011 schrieb Martin Steigerwald: > > Hi! > > > > Finally I tried scrubbing the / BTRFS filesystem mentioned in the > > thread "speeding up slow btrfs filesystem". However the machine looks > > up hard then. It repeats the last few seconds of audio all over > > again, no mouse and no ssh connection anymore: > > > > deepdance:~> btrfs scrub start / > > scrub started on /, fsid […] (pid=5737) > > deepdance:~> Write failed: Broken pipe > > > > > > After the second attempt of doing this the machine stops on booting > > after the space cache enabled message. Then I get backtraced of hung > > tasks: > > > > http://martin-steigerwald.de/tmp/btrfs/2011-17-12-deepdance-hang-at-b > > oot/ > > > > > > I am able to mount the filesystem from grml 2011.12-rc1 with 3.1 > > kernel:I still have this with 3.2.0-1-pae - which is a debian kernel based on 3.2.1. When I do btrfs scrub start / the machine locks immediately up hard. Then usually on next boot it stops on space_cache enabled message, but not the one for /, but the one for /home which is mounted later. When I then boot with 3.1 it works. BTRFS redos the space_cache then while the machine takes ages to boot - I mean ages - 10 minutes till KDM prompt is no problem there. I thought I just mention it here. Since I got no hints on what to do, I probably redo both filesystems on the machine. Should that not work out, I switch the box to Ext4. btrfs filesystem scrub works on my ThinkPad T520 with 64-bit debian and Intel SSD 320 and one 2,5 inch external drive as well as a 3,5 inch external backup drive both via eSATA, so this seems to be no principal issue. It also works on a workstation at work which has 32-bit debian as well. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Jan-21 11:19 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald:> I still have this with 3.2.0-1-pae - which is a debian kernel based on > 3.2.1. > > When I do btrfs scrub start / the machine locks immediately up hard. > > Then usually on next boot it stops on space_cache enabled message, but > not the one for /, but the one for /home which is mounted later. > > When I then boot with 3.1 it works. BTRFS redos the space_cache then > while the machine takes ages to boot - I mean ages - 10 minutes till > KDM prompt is no problem there.I now tested scrubbing /home which is a different BTRFS filesystem on the same machine. Then the scrub is started, scrub status tells me so, but nothing happens, no block in/out activity in vmstat, no CPU related activity in top. btrfs scrub cancel then hangs, but not the complete machine, only the process. I had this once on my T520 with the internal Intel SSD 320 as well. The other time it worked. Well maybe that is due to BTRFS doing something else on my T23 now: deepdance:~> ps aux | grep ino-cache | grep -v grep root 1992 5.5 0.0 0 0 ? D 12:15 0:09 [btrfs- ino-cache] Hmmm, so I just let it sit for a while, maybe eventually it will scrub /home. At least it doesn´t lock up hard, so there might really be something strange with /. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Arne Jansen
2012-Jan-24 09:28 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On 21.01.2012 11:49, Martin Steigerwald wrote:> Am Dienstag, 20. Dezember 2011 schrieb Martin Steigerwald: > > When I do btrfs scrub start / the machine locks immediately up hard.Can you please give me the output of sysrq-w when the machine is locked up? Thanks, Arne> > Then usually on next boot it stops on space_cache enabled message, but not > the one for /, but the one for /home which is mounted later. > > When I then boot with 3.1 it works. BTRFS redos the space_cache then while > the machine takes ages to boot - I mean ages - 10 minutes till KDM prompt > is no problem there. > > I thought I just mention it here. > > Since I got no hints on what to do, I probably redo both filesystems on the > machine. Should that not work out, I switch the box to Ext4. > > btrfs filesystem scrub works on my ThinkPad T520 with 64-bit debian and > Intel SSD 320 and one 2,5 inch external drive as well as a 3,5 inch > external backup drive both via eSATA, so this seems to be no principal > issue. It also works on a workstation at work which has 32-bit debian as > well. > > Thanks,-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Jan-24 10:16 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Dienstag, 24. Januar 2012 schrieb Arne Jansen:> On 21.01.2012 11:49, Martin Steigerwald wrote: > > Am Dienstag, 20. Dezember 2011 schrieb Martin Steigerwald: > > > > When I do btrfs scrub start / the machine locks immediately up hard. > > Can you please give me the output of sysrq-w when the machine is locked > up?What would be the easiest way to do that? The machine is locked up, means, no mouse, no keyboard, no ping - as far as I remember I tested a ping, but I can verify -, no nothing. Serial console with USB serial adapter? Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Arne Jansen
2012-Jan-24 10:29 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On 24.01.2012 11:16, Martin Steigerwald wrote:> Am Dienstag, 24. Januar 2012 schrieb Arne Jansen: >> On 21.01.2012 11:49, Martin Steigerwald wrote: >>> Am Dienstag, 20. Dezember 2011 schrieb Martin Steigerwald: >>> >>> When I do btrfs scrub start / the machine locks immediately up hard. >> >> Can you please give me the output of sysrq-w when the machine is locked >> up? > > What would be the easiest way to do that? > > The machine is locked up, means, no mouse, no keyboard, no ping - as far > as I remember I tested a ping, but I can verify -, no nothing.Hm, I always use a (hardware) serial console, but I think on default loglevel the sysrq-output goes to dmesg only, so you have to adjust the loglevel also. Normally I have a ssh into the box running with a tail -f /var/log/messages. This locks up very seldom. Have you tried some sysrq combinations? It might work, even when the keyboard looks locked up in all other respects.> > Serial console with USB serial adapter? > > Thanks,-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Jan-24 10:39 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Dienstag, 24. Januar 2012 schrieb Arne Jansen:> On 24.01.2012 11:16, Martin Steigerwald wrote: > > Am Dienstag, 24. Januar 2012 schrieb Arne Jansen: > >> On 21.01.2012 11:49, Martin Steigerwald wrote: > >>> Am Dienstag, 20. Dezember 2011 schrieb Martin Steigerwald: > >>> > >>> When I do btrfs scrub start / the machine locks immediately up > >>> hard. > >> > >> Can you please give me the output of sysrq-w when the machine is > >> locked up? > > > > What would be the easiest way to do that? > > > > The machine is locked up, means, no mouse, no keyboard, no ping - as > > far as I remember I tested a ping, but I can verify -, no nothing. > > Hm, I always use a (hardware) serial console, but I think on default > loglevel the sysrq-output goes to dmesg only, so you have to adjust > the loglevel also. Normally I have a ssh into the box running with a > tail -f /var/log/messages. This locks up very seldom. > Have you tried some sysrq combinations? It might work, even when the > keyboard looks locked up in all other respects.Ok, so I may try the SSH way, maybe I get something before network gets nonfunctional. Or maybe I didn´t even test ssh/ping and network still works. But I think I tried accessing the machine via network, I usually do this in such cases. If that does not work and USB to serial adapter might work. But then as far as I remember even audio locked up hard replaying the same sample all over again, so I do not know whether the kernel will do anything at all. Aside from that I got the idea to try to scrubbing with kernel 3.1 to verify whether thats an regression. Well I try some things and report back. Since the / filesystem on that ThinkPad T23 is the only one it might have some wierd issues. Its one of my oldest BTRFS filesystems. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Feb-24 15:51 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald:> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > I still have this with 3.2.0-1-pae - which is a debian kernel based > > on 3.2.1. > > > > When I do btrfs scrub start / the machine locks immediately up hard. > > > > Then usually on next boot it stops on space_cache enabled message, > > but not the one for /, but the one for /home which is mounted > > later. > > > > When I then boot with 3.1 it works. BTRFS redos the space_cache then > > while the machine takes ages to boot - I mean ages - 10 minutes till > > KDM prompt is no problem there. > > I now tested scrubbing /home which is a different BTRFS filesystem on > the same machine. > > Then the scrub is started, scrub status tells me so, but nothing > happens, no block in/out activity in vmstat, no CPU related activity > in top. > > btrfs scrub cancel then hangs, but not the complete machine, only the > process. > > I had this once on my T520 with the internal Intel SSD 320 as well. The > other time it worked. > > Well maybe that is due to BTRFS doing something else on my T23 now: > > deepdance:~> ps aux | grep ino-cache | grep -v grep > root 1992 5.5 0.0 0 0 ? D 12:15 0:09 > [btrfs- ino-cache] > > Hmmm, so I just let it sit for a while, maybe eventually it will scrub > /home. > > At least it doesn´t lock up hard, so there might really be something > strange with /.FWIW a btrfs filesystem balance / does work. After this a btrfs scrub start / still locks the kernel. Anyway, I might be waiting for the new fsck and try it on the partition. Or redo the filesystem from scratch, cause I think trying to debug this will take way more time. I might also as well redo /home as well. Two fresh 3.2 or 3.3 kernel BTRFS and see whether they work better. -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Arne Jansen
2012-Feb-25 08:14 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Hi Martin, I just sent 2 patches to the list. Could you please test if these fix your problem with scrub? Thanks, Arne On 02/24/12 16:51, Martin Steigerwald wrote:> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: >>> I still have this with 3.2.0-1-pae - which is a debian kernel based >>> on 3.2.1. >>> >>> When I do btrfs scrub start / the machine locks immediately up hard. >>> >>> Then usually on next boot it stops on space_cache enabled message, >>> but not the one for /, but the one for /home which is mounted >>> later. >>> >>> When I then boot with 3.1 it works. BTRFS redos the space_cache then >>> while the machine takes ages to boot - I mean ages - 10 minutes till >>> KDM prompt is no problem there. >> >> I now tested scrubbing /home which is a different BTRFS filesystem on >> the same machine. >> >> Then the scrub is started, scrub status tells me so, but nothing >> happens, no block in/out activity in vmstat, no CPU related activity >> in top. >> >> btrfs scrub cancel then hangs, but not the complete machine, only the >> process. >> >> I had this once on my T520 with the internal Intel SSD 320 as well. The >> other time it worked. >> >> Well maybe that is due to BTRFS doing something else on my T23 now: >> >> deepdance:~> ps aux | grep ino-cache | grep -v grep >> root 1992 5.5 0.0 0 0 ? D 12:15 0:09 >> [btrfs- ino-cache] >> >> Hmmm, so I just let it sit for a while, maybe eventually it will scrub >> /home. >> >> At least it doesn´t lock up hard, so there might really be something >> strange with /. > > FWIW a btrfs filesystem balance / does work. After this a btrfs scrub start > / still locks the kernel. > > Anyway, I might be waiting for the new fsck and try it on the partition. > Or redo the filesystem from scratch, cause I think trying to debug this > will take way more time. > > I might also as well redo /home as well. Two fresh 3.2 or 3.3 kernel BTRFS > and see whether they work better. >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Feb-25 20:14 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Samstag, 25. Februar 2012 schrieb Arne Jansen:> Hi Martin, > > I just sent 2 patches to the list. Could you please test if these > fix your problem with scrub?I saved them and I try to. But no promises as to when. The machine is slow at compiling kernels. And there are two annoying Pulseaudio issues I´d like to take time to gather more infos about that Pulseaudio developers requested. So there is some sort of a backlog. Ciao, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-15 17:32 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Samstag, 25. Februar 2012 schrieb Arne Jansen:> On 02/24/12 16:51, Martin Steigerwald wrote: > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > >>> I still have this with 3.2.0-1-pae - which is a debian kernel based > >>> on 3.2.1. > >>> > >>> When I do btrfs scrub start / the machine locks immediately up > >>> hard. > >>> > >>> Then usually on next boot it stops on space_cache enabled message, > >>> but not the one for /, but the one for /home which is mounted > >>> later. > >>> > >>> When I then boot with 3.1 it works. BTRFS redos the space_cache > >>> then while the machine takes ages to boot - I mean ages - 10 > >>> minutes till KDM prompt is no problem there. > >> > >> I now tested scrubbing /home which is a different BTRFS filesystem > >> on the same machine. > >> > >> Then the scrub is started, scrub status tells me so, but nothing > >> happens, no block in/out activity in vmstat, no CPU related activity > >> in top. > >> > >> btrfs scrub cancel then hangs, but not the complete machine, only > >> the process. > >> > >> I had this once on my T520 with the internal Intel SSD 320 as well. > >> The other time it worked. > >> > >> Well maybe that is due to BTRFS doing something else on my T23 now: > >> > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > >> root 1992 5.5 0.0 0 0 ? D 12:15 0:09 > >> [btrfs- ino-cache] > >> > >> Hmmm, so I just let it sit for a while, maybe eventually it will > >> scrub /home. > >> > >> At least it doesn´t lock up hard, so there might really be something > >> strange with /. > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs scrub > > start / still locks the kernel. > > Hi Martin, > > I just sent 2 patches to the list. Could you please test if these > fix your problem with scrub?I didn´t yet test it but I tried the first balance then scrub stuff again: deepdance:~> btrfs filesystem balance / This time scrub didn´t lock up hard: deepdance:~> btrfs scrub start / scrub started on /, fsid 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 (pid=3347) deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Thu Mar 15 18:10:52 2012, running for 10 seconds total bytes scrubbed: 272.59MB with 0 errors deepdance:~> cat /proc/version Linux version 3.2.0-1-686-pae (Debian 3.2.6-1) (ben@decadent.org.uk) (gcc version 4.6.2 (Debian 4.6.2-14) ) #1 SMP Fri Feb 17 06:27:21 UTC 2012 deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Thu Mar 15 18:10:52 2012, running for 55 seconds total bytes scrubbed: 1.29GB with 0 errors deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Thu Mar 15 18:10:52 2012, running for 175 seconds total bytes scrubbed: 2.58GB with 0 errors But it seems to be stuck now: deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Thu Mar 15 18:10:52 2012, running for 295 seconds total bytes scrubbed: 2.58GB with 0 errors deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Thu Mar 15 18:10:52 2012, running for 395 seconds total bytes scrubbed: 2.58GB with 0 errors deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Thu Mar 15 18:10:52 2012, running for 636 seconds total bytes scrubbed: 2.58GB with 0 errors There is almost no activity in vmstat 5: 2 0 108812 63812 36 412716 0 22 0 437 515 1723 47 13 37 3 2 0 108812 63320 36 413168 0 0 81 0 453 1600 44 12 43 1 2 0 108812 68056 36 413512 0 0 64 0 480 1589 51 11 38 0 2 0 108812 68112 36 413516 0 0 0 0 460 1454 46 11 43 0 1 0 108812 68112 36 413536 0 0 0 1 454 1456 45 11 44 0 1 1 108812 67924 36 413692 0 0 0 330 491 1578 49 13 37 1 2 0 108812 67808 36 413792 0 0 0 6 473 1556 48 11 41 0 2 0 108812 67436 36 414124 0 0 0 424 523 1922 47 13 39 2 1 0 108812 67460 36 414136 0 0 0 0 453 1578 46 9 45 0 2 0 108812 67476 36 414136 0 0 0 0 465 1493 46 12 41 0 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3 0 108812 67492 36 414140 0 0 0 0 467 1609 48 10 42 0 2 0 108812 67500 36 414148 0 0 0 0 470 1686 46 12 41 0 1 0 108812 66872 36 414152 0 0 0 1 463 1585 46 11 43 0 1 0 108812 67492 36 414156 0 0 0 1 483 1612 48 13 39 0 2 0 108812 67428 36 414156 0 0 0 325 497 1654 48 9 41 1 0 0 108812 67452 36 414160 0 0 0 3 478 1588 46 15 39 0 2 0 108812 65840 36 415228 0 0 214 0 465 1408 54 10 36 0 0 0 108812 66124 36 415352 0 0 24 0 510 1476 62 14 22 2 Given the general slowness of the machine, I waited about 15 seconds on a tail -f /var/log/syslog while just a Konsole and Amarok was running and wait about that time or even longer when logging via SSH at times, I think next I try newest Debian kernel which should be up to 3.2.10. If that doesn´t work, I consider trying your patches. At least its a step forward: The machine doesn´t lock up hard anymore. Its still playing music. FWIW a cancel hangs: deepdance:~> btrfs scrub cancel / martin@deepdance:~> ps aux | grep btrfs | grep cancel root 3429 0.0 0.0 2128 304 pts/5 D+ 18:24 0:00 btrfs scrub cancel / Ah, now there we go, from dmesg: 16453.391004] btrfs: relocating block group 41393586176 flags 1 [16568.504524] btrfs: found 9891 extents [16660.810000] btrfs: found 9891 extents [16664.818271] btrfs: relocating block group 40319844352 flags 1 [16729.029078] btrfs: found 3593 extents [16755.771257] btrfs: found 3593 extents [16757.816472] btrfs: relocating block group 39246102528 flags 1 [16843.042862] btrfs: found 13147 extents [16898.078594] btrfs: found 13147 extents [16905.444650] btrfs: relocating block group 39237713920 flags 34 [16910.579963] btrfs: found 1 extents [16915.946553] btrfs: relocating block group 39103496192 flags 36 [16997.308374] btrfs: found 14552 extents [17002.239868] btrfs: relocating block group 38969278464 flags 36 [17165.253567] btrfs: found 22792 extents [17166.522513] btrfs: relocating block group 38835060736 flags 36 [17376.519168] btrfs: found 22857 extents [17377.526508] btrfs: relocating block group 38700843008 flags 36 [17559.413243] btrfs: found 24810 extents [17563.767590] btrfs: relocating block group 37627101184 flags 1 [17646.272226] btrfs: found 10900 extents [17694.323897] btrfs: found 10900 extents [17697.171187] btrfs: relocating block group 36553359360 flags 1 [17779.364231] btrfs: found 12149 extents [17828.395574] btrfs: found 12149 extents [17831.593055] btrfs: relocating block group 35479617536 flags 1 [17923.352624] btrfs: found 22174 extents [18014.147595] btrfs: found 22174 extents [18015.821756] btrfs: relocating block group 35345399808 flags 36 [18055.201043] btrfs: found 9166 extents [18061.692337] btrfs: relocating block group 34271657984 flags 1 [18173.986306] btrfs: found 29124 extents [18328.733328] btrfs: found 29124 extents [18333.141061] btrfs: relocating block group 33197916160 flags 1 [18427.523898] btrfs: found 26881 extents [18587.015489] btrfs: found 26880 extents [18590.183797] btrfs: relocating block group 33063698432 flags 36 [18590.183797] btrfs: relocating block group 33063698432 flags 36 [18624.883576] btrfs: found 6522 extents [18628.618415] btrfs: relocating block group 32929480704 flags 36 [18670.147756] btrfs: found 9876 extents [18671.426925] btrfs: relocating block group 32795262976 flags 36 [18863.313381] btrfs: found 27324 extents [18864.392712] btrfs: relocating block group 31721521152 flags 1 [18945.941315] btrfs: found 21748 extents [19076.164135] btrfs: found 21721 extents [20880.564211] INFO: task btrfs:3348 blocked for more than 120 seconds. [20880.564225] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20880.564234] btrfs D 000012d2 0 3348 1 0x00000000 [20880.564251] c0b72080 00000086 2cae1a17 000012d2 ec3ae5a0 00000000 000012d2 c14769c0 [20880.564275] c0b7222c c14769c0 00008050 e0b96ce0 ee000120 d1d5dae0 d1d5db20 00000028 [20880.564298] c105d451 00000092 c12b949b 00000092 f0b7037c 00000246 eee6073c c105d451 [20880.564321] Call Trace: [20880.564348] [<c105d451>] ? arch_local_irq_save+0xf/0x14 [20880.564368] [<c12b949b>] ? _raw_spin_lock_irqsave+0x8/0x21 [20880.564482] [<f0b7037c>] ? btrfs_queue_worker+0x1cd/0x1fc [btrfs] [20880.564497] [<c105d451>] ? arch_local_irq_save+0xf/0x14 [20880.564511] [<c12b949b>] ? _raw_spin_lock_irqsave+0x8/0x21 [20880.564526] [<c104d402>] ? prepare_to_wait+0x12/0x50 [20880.564601] [<f0b8a8c8>] ? btrfs_reada_wait+0x5b/0x84 [btrfs] [20880.564615] [<c104d4d0>] ? add_wait_queue+0x30/0x30 [20880.564687] [<f0b87e33>] ? scrub_stripe+0x34c/0xbda [btrfs] [20880.564729] [<f0a4da1f>] ? scsi_alloc_sgtable+0x1a/0x38 [scsi_mod] [20880.564786] [<f0b37a7f>] ? comp_keys+0x2f/0x34 [btrfs] [20880.564842] [<f0b37b4d>] ? generic_bin_search.constprop.31+0xc9/0x104 [btrfs] [20880.564856] [<c104d4d0>] ? add_wait_queue+0x30/0x30 [20880.564912] [<f0b37bbf>] ? bin_search+0x37/0x3d [btrfs] [20880.564987] [<f0b5e3d4>] ? __lookup_extent_mapping+0xe5/0x101 [btrfs] [20880.565061] [<f0b88781>] ? scrub_chunk.isra.13+0xc0/0xef [btrfs] [20880.565134] [<f0b889a3>] ? scrub_enumerate_chunks+0x1f3/0x23a [btrfs] [20880.565209] [<f0b89486>] ? btrfs_scrub_dev+0x215/0x34e [btrfs] [20880.565230] [<c10c061a>] ? __kmalloc_track_caller+0x9b/0xa7 [20880.565248] [<c102a280>] ? should_resched+0x5/0x1e [20880.565320] [<f0b74b14>] ? btrfs_ioctl+0xc55/0xda1 [btrfs] [20880.565335] [<c1029188>] ? kmap_atomic_prot+0x2f/0xe0 [20880.565353] [<c10ad3d5>] ? handle_mm_fault+0x1ee/0x1fd [20880.565426] [<f0b73ebf>] ? btrfs_ioctl_trans_end+0x49/0x49 [btrfs] [20880.565442] [<c10d7023>] ? do_vfs_ioctl+0x459/0x48f [20880.565457] [<c12bc3d7>] ? do_page_fault+0x2e0/0x2fc [20880.565469] [<c12bc3c4>] ? do_page_fault+0x2cd/0x2fc [20880.565484] [<c10d709d>] ? sys_ioctl+0x44/0x67 [20880.565499] [<c12bdd1f>] ? sysenter_do_call+0x12/0x28 [21000.564277] INFO: task btrfs:3348 blocked for more than 120 seconds. [21000.564292] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [21000.564302] btrfs D 000012d2 0 3348 1 0x00000000 [21000.564319] c0b72080 00000086 2cae1a17 000012d2 ec3ae5a0 00000000 000012d2 c14769c0 [21000.564344] c0b7222c c14769c0 00008050 e0b96ce0 ee000120 d1d5dae0 d1d5db20 00000028 [21000.564366] c105d451 00000092 c12b949b 00000092 f0b7037c 00000246 eee6073c c105d451 [21000.564390] Call Trace: [21000.564422] [<c105d451>] ? arch_local_irq_save+0xf/0x14 [21000.564442] [<c12b949b>] ? _raw_spin_lock_irqsave+0x8/0x21 [21000.564582] [<f0b7037c>] ? btrfs_queue_worker+0x1cd/0x1fc [btrfs] [21000.564600] [<c105d451>] ? arch_local_irq_save+0xf/0x14 [21000.564614] [<c12b949b>] ? _raw_spin_lock_irqsave+0x8/0x21 [21000.564630] [<c104d402>] ? prepare_to_wait+0x12/0x50 [21000.564707] [<f0b8a8c8>] ? btrfs_reada_wait+0x5b/0x84 [btrfs] [21000.564721] [<c104d4d0>] ? add_wait_queue+0x30/0x30 [21000.564793] [<f0b87e33>] ? scrub_stripe+0x34c/0xbda [btrfs] [21000.564840] [<f0a4da1f>] ? scsi_alloc_sgtable+0x1a/0x38 [scsi_mod] [21000.564899] [<f0b37a7f>] ? comp_keys+0x2f/0x34 [btrfs] [21000.564954] [<f0b37b4d>] ? generic_bin_search.constprop.31+0xc9/0x104 [btrfs] [21000.564968] [<c104d4d0>] ? add_wait_queue+0x30/0x30 [21000.565024] [<f0b37bbf>] ? bin_search+0x37/0x3d [btrfs] [21000.565099] [<f0b5e3d4>] ? __lookup_extent_mapping+0xe5/0x101 [btrfs] [21000.565173] [<f0b88781>] ? scrub_chunk.isra.13+0xc0/0xef [btrfs] [21000.565246] [<f0b889a3>] ? scrub_enumerate_chunks+0x1f3/0x23a [btrfs] [21000.565322] [<f0b89486>] ? btrfs_scrub_dev+0x215/0x34e [btrfs] [21000.565345] [<c10c061a>] ? __kmalloc_track_caller+0x9b/0xa7 [21000.565363] [<c102a280>] ? should_resched+0x5/0x1e [21000.565436] [<f0b74b14>] ? btrfs_ioctl+0xc55/0xda1 [btrfs] [21000.565451] [<c1029188>] ? kmap_atomic_prot+0x2f/0xe0 [21000.565471] [<c10ad3d5>] ? handle_mm_fault+0x1ee/0x1fd [21000.565544] [<f0b73ebf>] ? btrfs_ioctl_trans_end+0x49/0x49 [btrfs] [21000.565561] [<c10d7023>] ? do_vfs_ioctl+0x459/0x48f [21000.565577] [<c12bc3d7>] ? do_page_fault+0x2e0/0x2fc [21000.565589] [<c12bc3c4>] ? do_page_fault+0x2cd/0x2fc [21000.565603] [<c10d709d>] ? sys_ioctl+0x44/0x67 [21000.565619] [<c12bdd1f>] ? sysenter_do_call+0x12/0x28 [… it seems to go on this way …] Maybe something of that is helpful? I leave the machine running for a while, but I think it won´t hibernate with a process stuck in D state so eventually at some time I will shut it down and restart. OTOH I could leave it running till tomorrow or what. I will put log files aside for further diagnostics in any case. Ciao, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-15 17:39 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Donnerstag, 15. März 2012 schrieb Martin Steigerwald:> [20880.564225] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables this message. [20880.564234] btrfs D > 000012d2 0 3348 1 0x00000000 [20880.564251] c0b72080 > 00000086 2cae1a17 000012d2 ec3ae5a0 00000000 000012d2 c14769c0 > [20880.564275] c0b7222c c14769c0 00008050 e0b96ce0 ee000120 d1d5dae0 > d1d5db20 00000028 [20880.564298] c105d451 00000092 c12b949b 00000092 > f0b7037c 00000246 eee6073c c105d451Seems to be a thread of the btrfs scrub start process: martin@deepdance:~/scrub-hang-2012-03-15#1> ps aux | grep "btrfs " | grep -v grep root 3347 0.5 0.0 18520 112 pts/5 Sl 18:10 0:08 btrfs scrub start / root 3429 0.0 0.0 2128 304 pts/5 D+ 18:24 0:00 btrfs scrub cancel / deepdance:/proc/3348> cat cmdline btrfsscrubstart/# deepdance:/proc/3348> cat sched btrfs (3348, #threads: 3) --------------------------------------------------------- se.exec_start : 20693902.056600 se.vruntime : 4558167.340245 se.sum_exec_runtime : 8284.354850 nr_switches : 116554 nr_voluntary_switches : 115784 nr_involuntary_switches : 770 se.load.weight : 1024 policy : 0 prio : 120 clock-delta : 563 Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason
2012-Mar-15 17:42 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On Thu, Mar 15, 2012 at 06:32:19PM +0100, Martin Steigerwald wrote:> Am Samstag, 25. Februar 2012 schrieb Arne Jansen: > > On 02/24/12 16:51, Martin Steigerwald wrote: > > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > >>> I still have this with 3.2.0-1-pae - which is a debian kernel based > > >>> on 3.2.1. > > >>> > > >>> When I do btrfs scrub start / the machine locks immediately up > > >>> hard. > > >>> > > >>> Then usually on next boot it stops on space_cache enabled message, > > >>> but not the one for /, but the one for /home which is mounted > > >>> later. > > >>> > > >>> When I then boot with 3.1 it works. BTRFS redos the space_cache > > >>> then while the machine takes ages to boot - I mean ages - 10 > > >>> minutes till KDM prompt is no problem there. > > >> > > >> I now tested scrubbing /home which is a different BTRFS filesystem > > >> on the same machine. > > >> > > >> Then the scrub is started, scrub status tells me so, but nothing > > >> happens, no block in/out activity in vmstat, no CPU related activity > > >> in top. > > >> > > >> btrfs scrub cancel then hangs, but not the complete machine, only > > >> the process. > > >> > > >> I had this once on my T520 with the internal Intel SSD 320 as well. > > >> The other time it worked. > > >> > > >> Well maybe that is due to BTRFS doing something else on my T23 now: > > >> > > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > > >> root 1992 5.5 0.0 0 0 ? D 12:15 0:09 > > >> [btrfs- ino-cache] > > >> > > >> Hmmm, so I just let it sit for a while, maybe eventually it will > > >> scrub /home. > > >> > > >> At least it doesn´t lock up hard, so there might really be something > > >> strange with /. > > > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs scrub > > > start / still locks the kernel. > > > > Hi Martin, > > > > I just sent 2 patches to the list. Could you please test if these > > fix your problem with scrub? > > I didn´t yet test it but I tried the first balance then scrub stuff again:Looks like you''re on a 32 bit machine. The current for-linus branch has an important fix for scrub on 32 bit that should solve this. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-15 18:03 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Donnerstag, 15. März 2012 schrieb Chris Mason:> On Thu, Mar 15, 2012 at 06:32:19PM +0100, Martin Steigerwald wrote: > > Am Samstag, 25. Februar 2012 schrieb Arne Jansen: > > > On 02/24/12 16:51, Martin Steigerwald wrote: > > > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > >>> I still have this with 3.2.0-1-pae - which is a debian kernel > > > >>> based on 3.2.1. > > > >>> > > > >>> When I do btrfs scrub start / the machine locks immediately up > > > >>> hard. > > > >>> > > > >>> Then usually on next boot it stops on space_cache enabled > > > >>> message, but not the one for /, but the one for /home which > > > >>> is mounted later. > > > >>> > > > >>> When I then boot with 3.1 it works. BTRFS redos the space_cache > > > >>> then while the machine takes ages to boot - I mean ages - 10 > > > >>> minutes till KDM prompt is no problem there. > > > >> > > > >> I now tested scrubbing /home which is a different BTRFS > > > >> filesystem on the same machine. > > > >> > > > >> Then the scrub is started, scrub status tells me so, but nothing > > > >> happens, no block in/out activity in vmstat, no CPU related > > > >> activity in top. > > > >> > > > >> btrfs scrub cancel then hangs, but not the complete machine, > > > >> only the process. > > > >> > > > >> I had this once on my T520 with the internal Intel SSD 320 as > > > >> well. The other time it worked. > > > >> > > > >> Well maybe that is due to BTRFS doing something else on my T23 > > > >> now: > > > >> > > > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > > > >> root 1992 5.5 0.0 0 0 ? D 12:15 0:09 > > > >> [btrfs- ino-cache] > > > >> > > > >> Hmmm, so I just let it sit for a while, maybe eventually it will > > > >> scrub /home. > > > >> > > > >> At least it doesn´t lock up hard, so there might really be > > > >> something strange with /. > > > > > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs > > > > scrub start / still locks the kernel. > > > > > > Hi Martin, > > > > > > I just sent 2 patches to the list. Could you please test if these > > > fix your problem with scrub? > > > > I didn´t yet test it but I tried the first balance then scrub stuff > > again: > Looks like you''re on a 32 bit machine. The current for-linus branch > has an important fix for scrub on 32 bit that should solve this.Yes, thats 32-bit. Can this fix be applied to 3.2 as well? If yes, could you point me at it? Or otherwise is current for-linus somewhat stable? Cloning nonetheless - well after it finally installed git there which takes ages with audio playback lockups. Hopefully the /home BTRFS is faster than the / one ;). I have no cross-compiling set up. From atop: PAG | scan 11903 | stall 0 | | swin 25 | swout 875 | DSK | sda | busy 74% | read 297 | write 4240 | avio 1 ms | From vmstat 1: deepdance:~> vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 2 127012 109472 36 263096 1 6 661 740 516 1572 42 14 41 4 0 2 127012 110628 36 263088 0 0 0 2228 995 2114 41 14 0 44 0 0 127012 112800 36 263088 0 0 0 2916 1148 2306 38 19 2 41 1 0 127012 98416 36 277192 664 0 664 8 642 1203 82 13 3 2 2 0 127012 79320 36 294872 1072 0 1072 0 653 1226 77 16 0 7 9 0 127556 85952 36 279968 700 544 700 16456 1442 9767 37 63 0 0 3 0 127556 92584 36 281720 360 0 364 22408 1743 11684 44 56 0 0 0 2 127556 90964 36 282984 0 0 52 3104 915 2528 41 44 0 14 3 1 127556 91932 36 283036 0 0 0 2408 995 1969 46 14 0 40 0 1 127556 91760 36 283296 0 0 4 3004 980 2140 39 27 6 29 6 1 121000 190288 36 283884 0 0 612 8 596 1452 39 17 26 18 1 2 121000 181060 36 287732 0 0 3776 2104 791 1485 52 33 0 15 2 2 121000 181212 36 287724 0 0 4 2384 862 1936 40 23 1 37 0 1 121000 181444 36 287732 0 0 4 1888 870 2000 38 20 10 32 1 0 121000 181160 36 287740 0 0 4 2104 846 2170 45 25 9 20 3 1 121000 181156 36 287748 0 0 0 3528 916 2179 44 17 7 32 0 0 121000 181748 36 287756 0 0 0 1976 843 2199 41 19 17 23 3 0 121000 179252 36 290036 0 0 2240 2088 875 2197 42 30 1 28 These high values on wait luck suspicious to me. Anyway, cloning now. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason
2012-Mar-15 18:08 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On Thu, Mar 15, 2012 at 07:03:14PM +0100, Martin Steigerwald wrote:> Am Donnerstag, 15. März 2012 schrieb Chris Mason: > > > > > > I didn´t yet test it but I tried the first balance then scrub stuff > > > again: > > Looks like you''re on a 32 bit machine. The current for-linus branch > > has an important fix for scrub on 32 bit that should solve this. > > Yes, thats 32-bit. > > Can this fix be applied to 3.2 as well? If yes, could you point me at it? > Or otherwise is current for-linus somewhat stable? > > Cloning nonetheless - well after it finally installed git there which > takes ages with audio playback lockups. Hopefully the /home BTRFS > is faster than the / one ;). I have no cross-compiling set up. > > From atop: > > PAG | scan 11903 | stall 0 | | swin 25 | swout 875 | > DSK | sda | busy 74% | read 297 | write 4240 | avio 1 ms |http://git.kernel.org/?p=linux/kernel/git/mason/linux-btrfs.git;a=commit;h=a175423c831ea582c06784d1e172d2ce1d79923a The patch is so small you can edit it by hand. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-16 15:05 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Donnerstag, 15. März 2012 schrieb Chris Mason:> On Thu, Mar 15, 2012 at 06:32:19PM +0100, Martin Steigerwald wrote: > > Am Samstag, 25. Februar 2012 schrieb Arne Jansen: > > > On 02/24/12 16:51, Martin Steigerwald wrote: > > > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > >>> I still have this with 3.2.0-1-pae - which is a debian kernel > > > >>> based on 3.2.1. > > > >>> > > > >>> When I do btrfs scrub start / the machine locks immediately up > > > >>> hard. > > > >>> > > > >>> Then usually on next boot it stops on space_cache enabled > > > >>> message, but not the one for /, but the one for /home which > > > >>> is mounted later.[…]> > > >> I now tested scrubbing /home which is a different BTRFS > > > >> filesystem on the same machine. > > > >> > > > >> Then the scrub is started, scrub status tells me so, but nothing > > > >> happens, no block in/out activity in vmstat, no CPU related > > > >> activity in top. > > > >> > > > >> btrfs scrub cancel then hangs, but not the complete machine, > > > >> only the process. > > > >> > > > >> I had this once on my T520 with the internal Intel SSD 320 as > > > >> well. The other time it worked. > > > >> > > > >> Well maybe that is due to BTRFS doing something else on my T23 > > > >> now: > > > >> > > > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > > > >> root 1992 5.5 0.0 0 0 ? D 12:15 0:09 > > > >> [btrfs- ino-cache][…]> > > >> At least it doesn´t lock up hard, so there might really be > > > >> something strange with /. > > > > > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs > > > > scrub start / still locks the kernel. > > > > > > Hi Martin, > > > > > > I just sent 2 patches to the list. Could you please test if these > > > fix your problem with scrub? > > > > I didn´t yet test it but I tried the first balance then scrub stuff > > again: > Looks like you''re on a 32 bit machine. The current for-linus branch > has an important fix for scrub on 32 bit that should solve this.So finally - the machine did make-kpkg over night and complained about missing Documentation lguest, then I switched off lots from distro default config and just did the usual make stuff - I was able to scrub both partitions on that ThinkPad T23: deepdance:~> btrfs scrub status / scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 scrub started at Fri Mar 16 11:56:12 2012 and finished after 741 seconds total bytes scrubbed: 9.62GB with 0 errors deepdance:~> btrfs scrub status /home scrub status for a600de65-e1ab-4cbf-b150-bbaeaf9fa98d scrub started at Fri Mar 16 12:00:31 2012 and finished after 1708 seconds total bytes scrubbed: 36.63GB with 0 errors Thanks a lot for fixing this. Tested-by: Martin Steigerwald <martin@lichtvoll.de> Arne, if you want me to test your two patches to readahead as well, please tell me. Now since only a few files need to be recompiled it would be quite easy to test them. Next challenge is the slow performance at times. This morning as I booted into the new kernel afterwards on tty it took at least 15 seconds of heavy disk activity with hearable lots of seeks to just open a screen. But thats something for a different thread. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Arne Jansen
2012-Mar-16 15:37 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On 16.03.2012 16:05, Martin Steigerwald wrote:>> has an important fix for scrub on 32 bit that should solve this. > > So finally - the machine did make-kpkg over night and complained about > missing Documentation lguest, then I switched off lots from distro default > config and just did the usual make stuff - I was able to scrub both > partitions on that ThinkPad T23: > > deepdance:~> btrfs scrub status / > scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 > scrub started at Fri Mar 16 11:56:12 2012 and finished after 741 seconds > total bytes scrubbed: 9.62GB with 0 errors > > deepdance:~> btrfs scrub status /home > scrub status for a600de65-e1ab-4cbf-b150-bbaeaf9fa98d > scrub started at Fri Mar 16 12:00:31 2012 and finished after 1708 seconds > total bytes scrubbed: 36.63GB with 0 errors > > Thanks a lot for fixing this. > > Tested-by: Martin Steigerwald <martin@lichtvoll.de> > > Arne, if you want me to test your two patches to readahead as well, please > tell me. Now since only a few files need to be recompiled it would be > quite easy to test themAs Chris''s patch already solves your problem, I think we''re fine. The two patches are more important for future users of readahead. As I had them sitting in the queue there was a small hope they might help your problem ;) Thanks, Arne> > Next challenge is the slow performance at times. This morning as I booted > into the new kernel afterwards on tty it took at least 15 seconds of > heavy disk activity with hearable lots of seeks to just open a screen. > But thats something for a different thread. > > Thanks,-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-17 09:43 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Hi Greg, Chris, Arne, hi everyone, Am Freitag, 16. März 2012 schrieb Martin Steigerwald:> Am Donnerstag, 15. März 2012 schrieb Chris Mason: > > On Thu, Mar 15, 2012 at 06:32:19PM +0100, Martin Steigerwald wrote: > > > Am Samstag, 25. Februar 2012 schrieb Arne Jansen: > > > > On 02/24/12 16:51, Martin Steigerwald wrote: > > > > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > > >>> I still have this with 3.2.0-1-pae - which is a debian kernel > > > > >>> based on 3.2.1. > > > > >>> > > > > >>> When I do btrfs scrub start / the machine locks immediately > > > > >>> up hard. > > > > >>> > > > > >>> Then usually on next boot it stops on space_cache enabled > > > > >>> message, but not the one for /, but the one for /home which > > > > >>> is mounted later. > > […] > > > > > >> I now tested scrubbing /home which is a different BTRFS > > > > >> filesystem on the same machine. > > > > >> > > > > >> Then the scrub is started, scrub status tells me so, but > > > > >> nothing happens, no block in/out activity in vmstat, no CPU > > > > >> related activity in top. > > > > >> > > > > >> btrfs scrub cancel then hangs, but not the complete machine, > > > > >> only the process. > > > > >> > > > > >> I had this once on my T520 with the internal Intel SSD 320 as > > > > >> well. The other time it worked. > > > > >> > > > > >> Well maybe that is due to BTRFS doing something else on my T23 > > > > >> now: > > > > >> > > > > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > > > > >> root 1992 5.5 0.0 0 0 ? D 12:15 > > > > >> 0:09 [btrfs- ino-cache] > > […] > > > > > >> At least it doesn´t lock up hard, so there might really be > > > > >> something strange with /. > > > > > > > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs > > > > > scrub start / still locks the kernel. > > > > > > > > Hi Martin, > > > > > > > > I just sent 2 patches to the list. Could you please test if these > > > > fix your problem with scrub? > > > > > > I didn´t yet test it but I tried the first balance then scrub stuff > > > > > again: > > Looks like you''re on a 32 bit machine. The current for-linus branch > > has an important fix for scrub on 32 bit that should solve this. > > So finally - the machine did make-kpkg over night and complained about > missing Documentation lguest, then I switched off lots from distro > default config and just did the usual make stuff - I was able to scrub > both partitions on that ThinkPad T23: > > deepdance:~> btrfs scrub status / > scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 > scrub started at Fri Mar 16 11:56:12 2012 and finished after > 741 seconds total bytes scrubbed: 9.62GB with 0 errors > > deepdance:~> btrfs scrub status /home > scrub status for a600de65-e1ab-4cbf-b150-bbaeaf9fa98d > scrub started at Fri Mar 16 12:00:31 2012 and finished after > 1708 seconds total bytes scrubbed: 36.63GB with 0 errors > > Thanks a lot for fixing this. > > Tested-by: Martin Steigerwald <martin@lichtvoll.de>[…] Would that patch Btrfs: fix casting error in scrub reada code http://git.kernel.org/?p=linux/kernel/git/mason/linux- btrfs.git;a=commit;h=a175423c831ea582c06784d1e172d2ce1d79923a (sorry for line break. KMail insists on it even when I disable line breaks due to the minus sign in the URL.) that I tested above be something for stable? To my knowledge Debian Wheezy and Ubuntu 12.04 LTS will have 3.2. Thanks, -- Martin ''Helios'' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Greg Kroah-Hartman
2012-Mar-18 00:37 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On Sat, Mar 17, 2012 at 10:43:25AM +0100, Martin Steigerwald wrote:> Hi Greg, Chris, Arne, hi everyone, > > Am Freitag, 16. März 2012 schrieb Martin Steigerwald: > > Am Donnerstag, 15. März 2012 schrieb Chris Mason: > > > On Thu, Mar 15, 2012 at 06:32:19PM +0100, Martin Steigerwald wrote: > > > > Am Samstag, 25. Februar 2012 schrieb Arne Jansen: > > > > > On 02/24/12 16:51, Martin Steigerwald wrote: > > > > > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > > > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > > > >>> I still have this with 3.2.0-1-pae - which is a debian kernel > > > > > >>> based on 3.2.1. > > > > > >>> > > > > > >>> When I do btrfs scrub start / the machine locks immediately > > > > > >>> up hard. > > > > > >>> > > > > > >>> Then usually on next boot it stops on space_cache enabled > > > > > >>> message, but not the one for /, but the one for /home which > > > > > >>> is mounted later. > > > > […] > > > > > > > >> I now tested scrubbing /home which is a different BTRFS > > > > > >> filesystem on the same machine. > > > > > >> > > > > > >> Then the scrub is started, scrub status tells me so, but > > > > > >> nothing happens, no block in/out activity in vmstat, no CPU > > > > > >> related activity in top. > > > > > >> > > > > > >> btrfs scrub cancel then hangs, but not the complete machine, > > > > > >> only the process. > > > > > >> > > > > > >> I had this once on my T520 with the internal Intel SSD 320 as > > > > > >> well. The other time it worked. > > > > > >> > > > > > >> Well maybe that is due to BTRFS doing something else on my T23 > > > > > >> now: > > > > > >> > > > > > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > > > > > >> root 1992 5.5 0.0 0 0 ? D 12:15 > > > > > >> 0:09 [btrfs- ino-cache] > > > > […] > > > > > > > >> At least it doesn´t lock up hard, so there might really be > > > > > >> something strange with /. > > > > > > > > > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs > > > > > > scrub start / still locks the kernel. > > > > > > > > > > Hi Martin, > > > > > > > > > > I just sent 2 patches to the list. Could you please test if these > > > > > fix your problem with scrub? > > > > > > > > I didn´t yet test it but I tried the first balance then scrub stuff > > > > > > > again: > > > Looks like you''re on a 32 bit machine. The current for-linus branch > > > has an important fix for scrub on 32 bit that should solve this. > > > > So finally - the machine did make-kpkg over night and complained about > > missing Documentation lguest, then I switched off lots from distro > > default config and just did the usual make stuff - I was able to scrub > > both partitions on that ThinkPad T23: > > > > deepdance:~> btrfs scrub status / > > scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 > > scrub started at Fri Mar 16 11:56:12 2012 and finished after > > 741 seconds total bytes scrubbed: 9.62GB with 0 errors > > > > deepdance:~> btrfs scrub status /home > > scrub status for a600de65-e1ab-4cbf-b150-bbaeaf9fa98d > > scrub started at Fri Mar 16 12:00:31 2012 and finished after > > 1708 seconds total bytes scrubbed: 36.63GB with 0 errors > > > > Thanks a lot for fixing this. > > > > Tested-by: Martin Steigerwald <martin@lichtvoll.de> > […] > > Would that patch > > Btrfs: fix casting error in scrub reada code > > http://git.kernel.org/?p=linux/kernel/git/mason/linux- > btrfs.git;a=commit;h=a175423c831ea582c06784d1e172d2ce1d79923a > > (sorry for line break. KMail insists on it even when I disable line breaks > due to the minus sign in the URL.) > > that I tested above be something for stable?<formletter> This is not the correct way to submit patches for inclusion in the stable kernel tree. Please read Documentation/stable_kernel_rules.txt for how to do this properly. </formletter> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-19 08:31 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Sonntag, 18. März 2012 schrieb Greg Kroah-Hartman:> On Sat, Mar 17, 2012 at 10:43:25AM +0100, Martin Steigerwald wrote: > > Hi Greg, Chris, Arne, hi everyone, > > > > Am Freitag, 16. März 2012 schrieb Martin Steigerwald: > > > Am Donnerstag, 15. März 2012 schrieb Chris Mason: > > > > On Thu, Mar 15, 2012 at 06:32:19PM +0100, Martin Steigerwald wrote: > > > > > Am Samstag, 25. Februar 2012 schrieb Arne Jansen: > > > > > > On 02/24/12 16:51, Martin Steigerwald wrote: > > > > > > > Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > > > > >> Am Samstag, 21. Januar 2012 schrieb Martin Steigerwald: > > > > > > >>> I still have this with 3.2.0-1-pae - which is a debian kernel > > > > > > >>> based on 3.2.1. > > > > > > >>> > > > > > > >>> When I do btrfs scrub start / the machine locks immediately > > > > > > >>> up hard. > > > > > > >>> > > > > > > >>> Then usually on next boot it stops on space_cache enabled > > > > > > >>> message, but not the one for /, but the one for /home which > > > > > > >>> is mounted later. > > > > > > […] > > > > > > > > > >> I now tested scrubbing /home which is a different BTRFS > > > > > > >> filesystem on the same machine. > > > > > > >> > > > > > > >> Then the scrub is started, scrub status tells me so, but > > > > > > >> nothing happens, no block in/out activity in vmstat, no CPU > > > > > > >> related activity in top. > > > > > > >> > > > > > > >> btrfs scrub cancel then hangs, but not the complete machine, > > > > > > >> only the process. > > > > > > >> > > > > > > >> I had this once on my T520 with the internal Intel SSD 320 as > > > > > > >> well. The other time it worked. > > > > > > >> > > > > > > >> Well maybe that is due to BTRFS doing something else on my T23 > > > > > > >> now: > > > > > > >> > > > > > > >> deepdance:~> ps aux | grep ino-cache | grep -v grep > > > > > > >> root 1992 5.5 0.0 0 0 ? D 12:15 > > > > > > >> 0:09 [btrfs- ino-cache] > > > > > > […] > > > > > > > > > >> At least it doesn´t lock up hard, so there might really be > > > > > > >> something strange with /. > > > > > > > > > > > > > > FWIW a btrfs filesystem balance / does work. After this a btrfs > > > > > > > scrub start / still locks the kernel. > > > > > > > > > > > > Hi Martin, > > > > > > > > > > > > I just sent 2 patches to the list. Could you please test if these > > > > > > fix your problem with scrub? > > > > > > > > > > I didn´t yet test it but I tried the first balance then scrub stuff > > > > > > > > > again: > > > > Looks like you''re on a 32 bit machine. The current for-linus branch > > > > has an important fix for scrub on 32 bit that should solve this. > > > > > > So finally - the machine did make-kpkg over night and complained about > > > missing Documentation lguest, then I switched off lots from distro > > > default config and just did the usual make stuff - I was able to scrub > > > both partitions on that ThinkPad T23: > > > > > > deepdance:~> btrfs scrub status / > > > scrub status for 2bf5b1dc-1d89-4f0d-a561-1a5551a27275 > > > > > > scrub started at Fri Mar 16 11:56:12 2012 and finished after > > > > > > 741 seconds total bytes scrubbed: 9.62GB with 0 errors > > > > > > deepdance:~> btrfs scrub status /home > > > scrub status for a600de65-e1ab-4cbf-b150-bbaeaf9fa98d > > > > > > scrub started at Fri Mar 16 12:00:31 2012 and finished after > > > > > > 1708 seconds total bytes scrubbed: 36.63GB with 0 errors > > > > > > Thanks a lot for fixing this. > > > > > > Tested-by: Martin Steigerwald <martin@lichtvoll.de> > > > > […] > > > > Would that patch > > > > Btrfs: fix casting error in scrub reada code > > > > http://git.kernel.org/?p=linux/kernel/git/mason/linux- > > btrfs.git;a=commit;h=a175423c831ea582c06784d1e172d2ce1d79923a > > > > (sorry for line break. KMail insists on it even when I disable line > > breaks due to the minus sign in the URL.) > > > > that I tested above be something for stable? > > <formletter> > > This is not the correct way to submit patches for inclusion in the > stable kernel tree. Please read Documentation/stable_kernel_rules.txt > for how to do this properly. > > </formletter>What point of the rules do you refer to? Is it - It or an equivalent fix must already exist in Linus'' tree (upstream). ? Or do you want the patch quoted in the mail? Anyways its not yet in linus tree AFAIR and from my side it was a question whether it should be included at all. I thought it might be good ccing you on that question already. Chris, how about this patch for stable once it hits Linus'' tree? Thanks, -- Martin Steigerwald - teamix GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Greg Kroah-Hartman
2012-Mar-19 15:48 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
On Mon, Mar 19, 2012 at 09:31:24AM +0100, Martin Steigerwald wrote:> > <formletter> > > > > This is not the correct way to submit patches for inclusion in the > > stable kernel tree. Please read Documentation/stable_kernel_rules.txt > > for how to do this properly. > > > > </formletter> > > What point of the rules do you refer to? Is it > > - It or an equivalent fix must already exist in Linus'' tree (upstream).Yes. Without that, there''s nothing I can do with this.> Or do you want the patch quoted in the mail? > > Anyways its not yet in linus tree AFAIR and from my side it was a question > whether it should be included at all. I thought it might be good ccing you on > that question already.It''s nice, yes, but note that you sent this to me directly, didn''t cc: the proper stable mailing list so that others who maintain other stable trees also didn''t get the notice, and again, there''s really nothing that the stable maintainers can do about this at the moment. But, if this does go to Linus''s tree, then stable@vger.kernel.org would be very interested in knowing about it. thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Martin Steigerwald
2012-Mar-19 16:03 UTC
Re: 3.2-rc4: scrubbing locks up the kernel, then hung tasks on boot
Am Montag, 19. März 2012 schrieb Greg Kroah-Hartman:> On Mon, Mar 19, 2012 at 09:31:24AM +0100, Martin Steigerwald wrote: > > > <formletter> > > > > > > This is not the correct way to submit patches for inclusion in the > > > stable kernel tree. Please read Documentation/stable_kernel_rules.txt > > > for how to do this properly. > > > > > > </formletter> > > > > What point of the rules do you refer to? Is it > > > > - It or an equivalent fix must already exist in Linus'' tree (upstream). > > Yes. > > Without that, there''s nothing I can do with this. > > > Or do you want the patch quoted in the mail? > > > > Anyways its not yet in linus tree AFAIR and from my side it was a > > question whether it should be included at all. I thought it might be > > good ccing you on that question already. > > It''s nice, yes, but note that you sent this to me directly, didn''t cc: > the proper stable mailing list so that others who maintain other stable > trees also didn''t get the notice, and again, there''s really nothing that > the stable maintainers can do about this at the moment.I did, but got the address wrong, missed the vger in it. Anyway, I try to follow-up on this, when its in Linus tree (via git wait and notify for commit? - nah just kidding, but would be useful ;) Thanks, -- Martin Steigerwald - teamix GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html