Christopher S. Aker
2011-Oct-25 19:48 UTC
[Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
Either right on boot, or shortly thereafter, the following occurs with either writeback or ordered filesystems. Multiple reports, all different machines, different users... Some people had success after I changed CONFIG_EXT3_DEFAULTS_TO_ORDERED off, however some still had problems... That was my only config change from 3.0.4. Them rebooting back into 3.0.4 fixed it. ------------[ cut here ]------------ WARNING: at fs/ext3/inode.c:1620 ext3_ordered_writepage+0x20a/0x230() Modules linked in: Pid: 2149, comm: flush-202:0 Not tainted 3.1.0-linode39 #1 Call Trace: [<c01346f8>] ? warn_slowpath_common+0x78/0xb0 [<c025666a>] ? ext3_ordered_writepage+0x20a/0x230 [<c025666a>] ? ext3_ordered_writepage+0x20a/0x230 [<c013474b>] ? warn_slowpath_null+0x1b/0x20 [<c025666a>] ? ext3_ordered_writepage+0x20a/0x230 [<c01067e4>] ? check_events+0x8/0xc [<c018c3c8>] ? __writepage+0x8/0x30 [<c018cbc6>] ? write_cache_pages+0x196/0x390 [<c018c3c0>] ? set_page_dirty+0x60/0x60 [<c018cdfe>] ? generic_writepages+0x3e/0x60 [<c01d4ae5>] ? writeback_single_inode+0x115/0x2b0 [<c0102b18>] ? xen_end_context_switch+0x8/0x10 [<c01d52c1>] ? writeback_sb_inodes+0x171/0x210 [<c01d58d5>] ? __writeback_inodes_wb+0x75/0xb0 [<c01d5b31>] ? wb_writeback+0x1a1/0x1b0 [<c0106007>] ? xen_force_evtchn_callback+0x17/0x30 [<c01067e4>] ? check_events+0x8/0xc [<c01d5cef>] ? wb_do_writeback+0x1af/0x1c0 [<c01d5d71>] ? bdi_writeback_thread+0x71/0x130 [<c01d5d00>] ? wb_do_writeback+0x1c0/0x1c0 [<c014e1a4>] ? kthread+0x74/0x80 [<c014e130>] ? kthread_worker_fn+0x170/0x170 [<c06c63b6>] ? kernel_thread_helper+0x6/0x10 ---[ end trace 702f842be742d945 ]--- And a different user: ------------[ cut here ]------------ WARNING: at fs/ext3/inode.c:1695 ext3_writeback_writepage+0x12a/0x150() Modules linked in: Pid: 2146, comm: flush-202:0 Not tainted 3.1.0-linode39 #1 Call Trace: [<c01346f8>] ? warn_slowpath_common+0x78/0xb0 [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150 [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150 [<c013474b>] ? warn_slowpath_null+0x1b/0x20 [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150 [<c01067e4>] ? check_events+0x8/0xc [<c018c3c8>] ? __writepage+0x8/0x30 [<c018cbc6>] ? write_cache_pages+0x196/0x390 [<c018c3c0>] ? set_page_dirty+0x60/0x60 [<c010dfd8>] ? sched_clock+0x8/0x10 [<c0154452>] ? sched_clock_local+0xb2/0x190 [<c018cdfe>] ? generic_writepages+0x3e/0x60 [<c01d4ae5>] ? writeback_single_inode+0x115/0x2b0 [<c0102b18>] ? xen_end_context_switch+0x8/0x10 [<c01d52c1>] ? writeback_sb_inodes+0x171/0x210 [<c01d58d5>] ? __writeback_inodes_wb+0x75/0xb0 [<c01d5b31>] ? wb_writeback+0x1a1/0x1b0 [<c0106007>] ? xen_force_evtchn_callback+0x17/0x30 [<c01067e4>] ? check_events+0x8/0xc [<c01d5cef>] ? wb_do_writeback+0x1af/0x1c0 [<c01d5d71>] ? bdi_writeback_thread+0x71/0x130 [<c01d5d00>] ? wb_do_writeback+0x1c0/0x1c0 [<c014e1a4>] ? kthread+0x74/0x80 [<c014e130>] ? kthread_worker_fn+0x170/0x170 [<c06c63b6>] ? kernel_thread_helper+0x6/0x10 ---[ end trace 0a37bc36a180e73c ]--- Am I barking up the wrong list? Thanks, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-25 20:12 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Tue, Oct 25, 2011 at 03:48:00PM -0400, Christopher S. Aker wrote:> Either right on boot, or shortly thereafter, the following occurs > with either writeback or ordered filesystems. Multiple reports, all > different machines, different users... Some people had success > after I changed CONFIG_EXT3_DEFAULTS_TO_ORDERED off, however some > still had problems... That was my only config change from 3.0.4. > Them rebooting back into 3.0.4 fixed it.Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18? What does the guest show for the disk? As in, is the barrier feature enabled? .. snip> ------------[ cut here ]------------ > WARNING: at fs/ext3/inode.c:1695 ext3_writeback_writepage+0x12a/0x150() > Modules linked in: > Pid: 2146, comm: flush-202:0 Not tainted 3.1.0-linode39 #1 > Call Trace: > [<c01346f8>] ? warn_slowpath_common+0x78/0xb0 > [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150 > [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150 > [<c013474b>] ? warn_slowpath_null+0x1b/0x20 > [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150 > [<c01067e4>] ? check_events+0x8/0xc > [<c018c3c8>] ? __writepage+0x8/0x30 > [<c018cbc6>] ? write_cache_pages+0x196/0x390 > [<c018c3c0>] ? set_page_dirty+0x60/0x60 > [<c010dfd8>] ? sched_clock+0x8/0x10 > [<c0154452>] ? sched_clock_local+0xb2/0x190 > [<c018cdfe>] ? generic_writepages+0x3e/0x60 > [<c01d4ae5>] ? writeback_single_inode+0x115/0x2b0 > [<c0102b18>] ? xen_end_context_switch+0x8/0x10 > [<c01d52c1>] ? writeback_sb_inodes+0x171/0x210 > [<c01d58d5>] ? __writeback_inodes_wb+0x75/0xb0 > [<c01d5b31>] ? wb_writeback+0x1a1/0x1b0 > [<c0106007>] ? xen_force_evtchn_callback+0x17/0x30 > [<c01067e4>] ? check_events+0x8/0xc > [<c01d5cef>] ? wb_do_writeback+0x1af/0x1c0 > [<c01d5d71>] ? bdi_writeback_thread+0x71/0x130 > [<c01d5d00>] ? wb_do_writeback+0x1c0/0x1c0 > [<c014e1a4>] ? kthread+0x74/0x80 > [<c014e130>] ? kthread_worker_fn+0x170/0x170 > [<c06c63b6>] ? kernel_thread_helper+0x6/0x10 > ---[ end trace 0a37bc36a180e73c ]--- > > Am I barking up the wrong list?Not sure.. It depends on what your dom0 version is and what the backend is? blktap or blkback? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Oct-25 20:40 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Oct 25, 2011, at 4:12 PM, Konrad Rzeszutek Wilk wrote:> Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18? > > What does the guest show for the disk? As in, is the barrier feature > enabled?2.6.18 dom0 and a 3.1.0 domU produces this on domU boot: EXT3-fs (xvda): using internal journal Adding 262140k swap on /dev/xvdb. Priority:-1 extents:1 across:262140k SS blkfront: barrier: empty write xvda op failed blkfront: xvda: barrier or flush: disabled end_request: I/O error, dev xvda, sector 1053048 end_request: I/O error, dev xvda, sector 1053048 Buffer I/O error on device xvda, logical block 131631 lost page write due to I/O error on xvda Aborting journal on device xvda. EXT3-fs (xvda): error: ext3_journal_start_sb: Detected aborted journal EXT3-fs (xvda): error: remounting filesystem read-only On my 3.0 based dom0 with a 3.1.0 domU blkfront doesn''t complain - and I *think* 3.1 domU may actually work as expected -- initial tests couldn''t get it to trace...> .. snip >> >> Am I barking up the wrong list? > > Not sure.. It depends on what your dom0 version is and what > the backend is? blktap or blkback?blkback. We suspected a dom0/domU incompatibility after doing some research. Hopefully this can get resolved within the guest since we have literally thousands of 2.6.18 dom0s deployed and rebooting everyone is not my idea of a good time :) Thanks, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-25 21:01 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Tue, Oct 25, 2011 at 04:40:09PM -0400, Christopher S. Aker wrote:> On Oct 25, 2011, at 4:12 PM, Konrad Rzeszutek Wilk wrote: > > Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18? > > > > What does the guest show for the disk? As in, is the barrier feature > > enabled? > > 2.6.18 dom0 and a 3.1.0 domU produces this on domU boot:You see the same exact thing with 3.0.6?> > EXT3-fs (xvda): using internal journal > Adding 262140k swap on /dev/xvdb. Priority:-1 extents:1 across:262140k SS > blkfront: barrier: empty write xvda op failed > blkfront: xvda: barrier or flush: disabled > end_request: I/O error, dev xvda, sector 1053048 > end_request: I/O error, dev xvda, sector 1053048 > Buffer I/O error on device xvda, logical block 131631 > lost page write due to I/O error on xvda > Aborting journal on device xvda. > EXT3-fs (xvda): error: ext3_journal_start_sb: Detected aborted journal > EXT3-fs (xvda): error: remounting filesystem read-only > > On my 3.0 based dom0 with a 3.1.0 domU blkfront doesn''t complain - and I *think* 3.1 domU may actually work as expected -- initial tests couldn''t get it to trace... > > > .. snip > >> > >> Am I barking up the wrong list? > > > > Not sure.. It depends on what your dom0 version is and what > > the backend is? blktap or blkback? > > blkback. > > We suspected a dom0/domU incompatibility after doing some research. Hopefully this can get resolved within the guest since we have literally thousands of 2.6.18 dom0s deployed and rebooting everyone is not my idea of a good time :) > > Thanks, > -Chris_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-25 21:15 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Tue, Oct 25, 2011 at 04:40:09PM -0400, Christopher S. Aker wrote:> On Oct 25, 2011, at 4:12 PM, Konrad Rzeszutek Wilk wrote: > > Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18? > > > > What does the guest show for the disk? As in, is the barrier feature > > enabled? > > 2.6.18 dom0 and a 3.1.0 domU produces this on domU boot: > > EXT3-fs (xvda): using internal journal > Adding 262140k swap on /dev/xvdb. Priority:-1 extents:1 across:262140k SS > blkfront: barrier: empty write xvda op failed > blkfront: xvda: barrier or flush: disabled > end_request: I/O error, dev xvda, sector 1053048 > end_request: I/O error, dev xvda, sector 1053048 > Buffer I/O error on device xvda, logical block 131631 > lost page write due to I/O error on xvda > Aborting journal on device xvda. > EXT3-fs (xvda): error: ext3_journal_start_sb: Detected aborted journal > EXT3-fs (xvda): error: remounting filesystem read-only > > On my 3.0 based dom0 with a 3.1.0 domU blkfront doesn''t complain - and I *think* 3.1 domU may actually work as expected -- initial tests couldn''t get it to trace...OK, that might be very well true as the 3.0 dom0 would export the ''feature-flush-cache''.. but you said that if you use 3.0 domU with 2.6.18 you do not get the type of warnings as above? Also, when the DomU guest starts it should report if it "thinks" it supports barriers. Something like: blkbfront: xvda: barrier Or is this the only message you get from blkfront??> > > .. snip > >> > >> Am I barking up the wrong list? > > > > Not sure.. It depends on what your dom0 version is and what > > the backend is? blktap or blkback? > > blkback. > > We suspected a dom0/domU incompatibility after doing some research. Hopefully this can get resolved within the guest since we have literally thousands of 2.6.18 dom0s deployed and rebooting everyone is not my idea of a good time :)> > Thanks, > -Chris > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Oct-26 20:26 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On 10/25/11 5:15 PM, Konrad Rzeszutek Wilk wrote:> You see the same exact thing with 3.0.6?It works with 3.0.4 - do you need me to try 3.0.6 specifically?> but you said that if you use 3.0 domU with 2.6.18 you do not get the > type of warnings as above?Correct.> Also, when the DomU guest starts it should report if it "thinks" it > supports barriers. Something like: blkbfront: xvda: barrier > > Or is this the only message you get from blkfront??2.6.18 dom0 + 3.0.4 domU produces this and nothing else: blkfront: xvda: barrier: enabled blkfront: xvdb: barrier: enabled 2.6.18 dom0 = works with 3.0.4, breaks with 3.1. Thanks, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-26 20:36 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Wed, Oct 26, 2011 at 04:26:40PM -0400, Christopher S. Aker wrote:> On 10/25/11 5:15 PM, Konrad Rzeszutek Wilk wrote: > >You see the same exact thing with 3.0.6? > > It works with 3.0.4 - do you need me to try 3.0.6 specifically?No need.> > >but you said that if you use 3.0 domU with 2.6.18 you do not get the > >type of warnings as above? > > Correct. > > >Also, when the DomU guest starts it should report if it "thinks" it > >supports barriers. Something like: blkbfront: xvda: barrier > > > >Or is this the only message you get from blkfront?? > > 2.6.18 dom0 + 3.0.4 domU produces this and nothing else: > > blkfront: xvda: barrier: enabled > blkfront: xvdb: barrier: enabled > > 2.6.18 dom0 = works with 3.0.4, breaks with 3.1.OK, so there is nothing that changed in 3.0 through 3.1 in xen-blkfront.c. Well, there was a name change, but nothing else. I am going to bet that if you compile 3.0.4 or 3.0.6 with that fancy new config option that turns EXT3_ORDERED_something you will see the same exact problem. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Oct-26 21:23 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote: > I am going to bet that if you compile 3.0.4 or 3.0.6 with that > fancy new config option that turns EXT3_ORDERED_something you will > see the same exact problem. Switching that back to match my 3.0 config is the first thing I tried, with no joy. I''ll reset and try a few more ideas... -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-27 15:41 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote:> On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote: > > I am going to bet that if you compile 3.0.4 or 3.0.6 with that > > fancy new config option that turns EXT3_ORDERED_something you will > > see the same exact problem. > > Switching that back to match my 3.0 config is the first thing I > tried, with no joy. > > I''ll reset and try a few more ideas...OK. I am kind of lost of what made all this happen - there was nothing between 3.0 and 3.1 in the Xen blkfront code. Perhaps there is something new and fancy in the Ext3 (or ext4) that is causing all this mayhem? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Oct-27 16:23 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Thu, Oct 27, 2011 at 11:41:59AM -0400, Konrad Rzeszutek Wilk wrote:> On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote: > > On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote: > > > I am going to bet that if you compile 3.0.4 or 3.0.6 with that > > > fancy new config option that turns EXT3_ORDERED_something you will > > > see the same exact problem. > > > > Switching that back to match my 3.0 config is the first thing I > > tried, with no joy. > > > > I''ll reset and try a few more ideas... > > OK. I am kind of lost of what made all this happen - there was nothing > between 3.0 and 3.1 in the Xen blkfront code. > > Perhaps there is something new and fancy in the Ext3 (or ext4) that is > causing all this mayhem? >Could it be?: http://kernelnewbies.org/Linux_3.1#head-12eb55d046763d8b789ff44c50b96c41b203d65e "Filesystem barriers enabled by default in Ext3" -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-27 16:41 UTC
Re: [Xen-devel] Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Thu, Oct 27, 2011 at 07:23:28PM +0300, Pasi Kärkkäinen wrote:> On Thu, Oct 27, 2011 at 11:41:59AM -0400, Konrad Rzeszutek Wilk wrote: > > On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote: > > > On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote: > > > > I am going to bet that if you compile 3.0.4 or 3.0.6 with that > > > > fancy new config option that turns EXT3_ORDERED_something you will > > > > see the same exact problem. > > > > > > Switching that back to match my 3.0 config is the first thing I > > > tried, with no joy. > > > > > > I''ll reset and try a few more ideas... > > > > OK. I am kind of lost of what made all this happen - there was nothing > > between 3.0 and 3.1 in the Xen blkfront code. > > > > Perhaps there is something new and fancy in the Ext3 (or ext4) that is > > causing all this mayhem? > > > > Could it be?: > http://kernelnewbies.org/Linux_3.1#head-12eb55d046763d8b789ff44c50b96c41b203d65e > > "Filesystem barriers enabled by default in Ext3"Oh, <smacks his head> I hadn''t realized this was a patch - thought it was some CONFIG option to turn it on. So Christopher if this: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=00eacd66cd8ab5fff9df49aa3f261ad43d495434 is reverted do you get the problems? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Dec-01 19:10 UTC
Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c
On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote:> On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote: > > I am going to bet that if you compile 3.0.4 or 3.0.6 with that > > fancy new config option that turns EXT3_ORDERED_something you will > > see the same exact problem. > > Switching that back to match my 3.0 config is the first thing I > tried, with no joy. > > I''ll reset and try a few more ideas...I think I know what the problem is. I believe you need the patch below in your hosts that are using the older dom0. AT least that solved the problem for me when I tried to use a 2.6.18 based dom0 with a 3.0 kernel with barriers. If you are using the linux-2.6.18.hg tree you might want to just pull it and rebuild it - it has this patch already. # HG changeset patch From: Jan Beulich <jbeulich@novell.com> # Date 1306409621 -3600 # Node ID 876a5aaac0264cf38cae6581e5714b93ec380aaa # Parent aedb712c05cf065e943e15d0f38597c2e80f7982 Subject: xen/blkback: don''t fail empty barrier requests The sector number on empty barrier requests may (will?) be uninitialized (neither bio_init() nor rq_init() set the respective fields), which allows for exceeding the actual (virtual) disk''s size. Inspired by Konrad''s "When writting barriers set the sector number to zero...", but instead of zapping the sector number (which is wrong for non-empty ones) just ignore the sector number when the sector count is zero. While at it also add overflow checking to the math in vbd_translate(). Signed-off-by: Jan Beulich <jbeulich@novell.com> diff -r aedb712c05cf -r 876a5aaac026 drivers/xen/blkback/vbd.c --- a/drivers/xen/blkback/vbd.c Thu May 26 08:09:04 2011 +0100 +++ b/drivers/xen/blkback/vbd.c Thu May 26 12:33:41 2011 +0100 @@ -108,8 +108,14 @@ int vbd_translate(struct phys_req *req, if ((operation != READ) && vbd->readonly) goto out; - if (unlikely((req->sector_number + req->nr_sects) > vbd_sz(vbd))) - goto out; + if (likely(req->nr_sects)) { + blkif_sector_t end = req->sector_number + req->nr_sects; + + if (unlikely(end < req->sector_number)) + goto out; + if (unlikely(end > vbd_sz(vbd))) + goto out; + } req->dev = vbd->pdevice; req->bdev = vbd->bdev;> > -Chris