Good morning btrfs list, I have been rsyncing files from an nfs mount to a btrfs filesystem. After an rsync run I ls random subvols or directorys to check the copy. About 60% to 70% of the time ls completely hangs. Ps aux shows it as running but even when I let it go for up to an hour it never finishes. Kill -9 will not stop the process. Dmesg shows nothing beyond a successful mount at boot. I can''t umount the system because "filesystem is busy". I find that a forced reboot is the only way to recapture the system. /var/log/messages has the only indication (that I can find) that anything abnormal is happening. A tail of the file is below. Thank you for any help and advice. Jim [root@btrfs ~]# tail -20 /var/log/messages Oct 18 05:06:36 btrfs kernel: [<ffffffff81142849>] sys_getdents+0x89/0xf0 Oct 18 05:06:36 btrfs kernel: [<ffffffff814b19ab>] system_call_fastpath+0x16/0x1b Oct 18 05:08:36 btrfs kernel: INFO: task ls:7100 blocked for more than 120 seconds. Oct 18 05:08:36 btrfs kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 18 05:08:36 btrfs kernel: ls D ffff88080bc843d8 0 7100 7047 0x00000084 Oct 18 05:08:36 btrfs kernel: ffff8804f526dd68 0000000000000086 ffff8804f526dd18 ffffffff00000000 Oct 18 05:08:36 btrfs kernel: ffff88080bc84000 0000000000011900 ffff8804f526dfd8 ffff8804f526c010 Oct 18 05:08:36 btrfs kernel: ffff8804f526dfd8 0000000000011900 ffff88080f64d7f0 ffff88080bc84000 Oct 18 05:08:36 btrfs kernel: Call Trace: Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a0f45>] wait_current_trans+0xa5/0x110 [btrfs] Oct 18 05:08:36 btrfs kernel: [<ffffffff8106c0e0>] ? wake_up_bit+0x40/0x40 Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a2540>] start_transaction+0x1e0/0x2b0 [btrfs] Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a2665>] btrfs_join_transaction+0x15/0x20 [btrfs] Oct 18 05:08:36 btrfs kernel: [<ffffffffa01ad360>] btrfs_dirty_inode+0x50/0x160 [btrfs] Oct 18 05:08:36 btrfs kernel: [<ffffffff8115631f>] __mark_inode_dirty+0x3f/0x200 Oct 18 05:08:36 btrfs kernel: [<ffffffff81148bf5>] touch_atime+0x115/0x150 Oct 18 05:08:36 btrfs kernel: [<ffffffff81142520>] ? sys_ioctl+0xb0/0xb0 Oct 18 05:08:36 btrfs kernel: [<ffffffff811426ce>] vfs_readdir+0xce/0xe0 Oct 18 05:08:36 btrfs kernel: [<ffffffff81142849>] sys_getdents+0x89/0xf0 Oct 18 05:08:36 btrfs kernel: [<ffffffff814b19ab>] system_call_fastpath+0x16/0x1b -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote:> Good morning btrfs list, > I have been rsyncing files from an nfs mount to a btrfs filesystem. > After an rsync run I ls random subvols or directorys to check the > copy. About 60% to 70% of the time ls completely hangs. Ps aux > shows it as running but even when I let it go for up to an hour it > never finishes. Kill -9 will not stop the process. Dmesg shows > nothing beyond a successful mount at boot. I can''t umount the > system because "filesystem is busy". I find that a forced reboot is > the only way to recapture the system. /var/log/messages has the > only indication (that I can find) that anything abnormal is > happening. A tail of the file is below. Thank you for any help and > advice.Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel you are on would be helpful. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I am on kernel 3.1.0-rc4. I am ssh''d into a remote machine so have no access to keyboard, any other way to get the info you need? Thanks again. Jim On 10/18/2011 09:31 AM, Josef Bacik wrote:> On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote: >> Good morning btrfs list, >> I have been rsyncing files from an nfs mount to a btrfs filesystem. >> After an rsync run I ls random subvols or directorys to check the >> copy. About 60% to 70% of the time ls completely hangs. Ps aux >> shows it as running but even when I let it go for up to an hour it >> never finishes. Kill -9 will not stop the process. Dmesg shows >> nothing beyond a successful mount at boot. I can''t umount the >> system because "filesystem is busy". I find that a forced reboot is >> the only way to recapture the system. /var/log/messages has the >> only indication (that I can find) that anything abnormal is >> happening. A tail of the file is below. Thank you for any help and >> advice. > Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel > you are on would be helpful. Thanks, > > Josef-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Jim wrote (ao):> I am on kernel 3.1.0-rc4. I am ssh''d into a remote machine so have > no access to keyboard, any other way to get the info you need?echo ''w'' > /proc/sysrq-trigger> On 10/18/2011 09:31 AM, Josef Bacik wrote: > >On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote: > >>Good morning btrfs list, > >>I have been rsyncing files from an nfs mount to a btrfs filesystem. > >>After an rsync run I ls random subvols or directorys to check the > >>copy. About 60% to 70% of the time ls completely hangs. Ps aux > >>shows it as running but even when I let it go for up to an hour it > >>never finishes. Kill -9 will not stop the process. Dmesg shows > >>nothing beyond a successful mount at boot. I can''t umount the > >>system because "filesystem is busy". I find that a forced reboot is > >>the only way to recapture the system. /var/log/messages has the > >>only indication (that I can find) that anything abnormal is > >>happening. A tail of the file is below. Thank you for any help and > >>advice. > >Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel > >you are on would be helpful. Thanks, > > > >Josef > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- Humilis IT Services and Solutions http://www.humilis.net -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Thank you, I should have thought of that. I will get back as soon as I have info. Jim On 10/18/2011 09:49 AM, Sander wrote:> Jim wrote (ao): >> I am on kernel 3.1.0-rc4. I am ssh''d into a remote machine so have >> no access to keyboard, any other way to get the info you need? > echo ''w''> /proc/sysrq-trigger > > >> On 10/18/2011 09:31 AM, Josef Bacik wrote: >>> On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote: >>>> Good morning btrfs list, >>>> I have been rsyncing files from an nfs mount to a btrfs filesystem. >>>> After an rsync run I ls random subvols or directorys to check the >>>> copy. About 60% to 70% of the time ls completely hangs. Ps aux >>>> shows it as running but even when I let it go for up to an hour it >>>> never finishes. Kill -9 will not stop the process. Dmesg shows >>>> nothing beyond a successful mount at boot. I can''t umount the >>>> system because "filesystem is busy". I find that a forced reboot is >>>> the only way to recapture the system. /var/log/messages has the >>>> only indication (that I can find) that anything abnormal is >>>> happening. A tail of the file is below. Thank you for any help and >>>> advice. >>> Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel >>> you are on would be helpful. Thanks, >>> >>> Josef >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I am getting no output from echo ''w'' > /proc/sysrq-trigger. Ls is hung again so I can try any thing else you suggest to get data. Jim On 10/18/2011 09:49 AM, Sander wrote:> Jim wrote (ao): >> I am on kernel 3.1.0-rc4. I am ssh''d into a remote machine so have >> no access to keyboard, any other way to get the info you need? > echo ''w''> /proc/sysrq-trigger > > >> On 10/18/2011 09:31 AM, Josef Bacik wrote: >>> On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote: >>>> Good morning btrfs list, >>>> I have been rsyncing files from an nfs mount to a btrfs filesystem. >>>> After an rsync run I ls random subvols or directorys to check the >>>> copy. About 60% to 70% of the time ls completely hangs. Ps aux >>>> shows it as running but even when I let it go for up to an hour it >>>> never finishes. Kill -9 will not stop the process. Dmesg shows >>>> nothing beyond a successful mount at boot. I can''t umount the >>>> system because "filesystem is busy". I find that a forced reboot is >>>> the only way to recapture the system. /var/log/messages has the >>>> only indication (that I can find) that anything abnormal is >>>> happening. A tail of the file is below. Thank you for any help and >>>> advice. >>> Hit sysrq+w when this happens and give us all the tracebacks. Also which kernel >>> you are on would be helpful. Thanks, >>> >>> Josef >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Oct 18, 2011 at 10:03:40AM -0400, Jim wrote:> I am getting no output from echo ''w'' > /proc/sysrq-trigger. Ls is > hung again so I can try any thing else you suggest to get data.Do echo 1> /proc/sys/kernel/sysrq and then try doing the echo w thing. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Yes, I had thought of that as well but still no output. Ls has been hung for about 30 min now and shows as D+ in ps. Do you think it would get us any information if I forced a reboot and redid the entire sequence running sysrq-trigger right after starting ls? Jim On 10/18/2011 10:04 AM, Josef Bacik wrote:> On Tue, Oct 18, 2011 at 10:03:40AM -0400, Jim wrote: >> I am getting no output from echo ''w''> /proc/sysrq-trigger. Ls is >> hung again so I can try any thing else you suggest to get data. > Do > > echo 1> /proc/sys/kernel/sysrq > > and then try doing the echo w thing. Thanks, > > Josef-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Josef Bacik wrote (ao):> On Tue, Oct 18, 2011 at 10:03:40AM -0400, Jim wrote: > > I am getting no output from echo ''w'' > /proc/sysrq-trigger. Ls is > > hung again so I can try any thing else you suggest to get data. > > Do > > echo 1> /proc/sys/kernel/sysrq > > and then try doing the echo w thing. Thanks,Jim, sorry if I''m stating the obvious here, but the output will be in dmesg (and syslog if possible). Sander -- Humilis IT Services and Solutions http://www.humilis.net -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Oct 18, 2011 at 10:08:30AM -0400, Jim wrote:> Yes, I had thought of that as well but still no output. Ls has been > hung for about 30 min now and shows as D+ in ps. Do you think it > would get us any information if I forced a reboot and redid the > entire sequence running sysrq-trigger right after starting ls?Maybe sysrq+t will get it? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Thank you both (Sander and Josef) I can forget anything so all reminders are appreciated. I will try again and also use the t argument. Jim On 10/18/2011 10:09 AM, Josef Bacik wrote:> On Tue, Oct 18, 2011 at 10:08:30AM -0400, Jim wrote: >> Yes, I had thought of that as well but still no output. Ls has been >> hung for about 30 min now and shows as D+ in ps. Do you think it >> would get us any information if I forced a reboot and redid the >> entire sequence running sysrq-trigger right after starting ls? > Maybe sysrq+t will get it? Thanks, > > Josef-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Thank you all for your prompt responses. I forced a reboot, remounted the fs and now cannot recreate the behavior. Ls appears to be working as expected. If it hangs later today I will get all the info I can and repost under this subject. Thank you again for your assistance. Jim On 10/18/2011 10:09 AM, Josef Bacik wrote:> On Tue, Oct 18, 2011 at 10:08:30AM -0400, Jim wrote: >> Yes, I had thought of that as well but still no output. Ls has been >> hung for about 30 min now and shows as D+ in ps. Do you think it >> would get us any information if I forced a reboot and redid the >> entire sequence running sysrq-trigger right after starting ls? > Maybe sysrq+t will get it? Thanks, > > Josef-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html