Joe Jin
2011-Jul-08 07:14 UTC
[Xen-devel] xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
When we do block attach detach test with below steps, umount hang and the
guest unable to shutdown:
1. start guest with the latest kernel.
2. attach new disk by xm-attach in Dom0
3. mount new disk in guest
4. detach the disk by xm-detach in dom0
5. umount the partition/disk in guest, command hung. exactly at here, any
IO request to the partition/disk will hang.
Checking the code we found when xm-detach command set backend state to
Closing, will trigger blkback_changed() -> blkfront_closing() call.
At the moment, the disk still opened by guest, so frontend will refuse the
request, but in the blkfront_closing(), it send a notification to backend
said that the frontend state switched to Closing, when backend got the
event, it will disconnect from real device, at here any IO request will
be stuck, even tried to release the disk by umount.
Per our test, below patch fix this issue.
Signed-off-by: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Annie Li <annie.li@oracle.com>
---
xen-blkfront.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index b536a9c..f6d8ac2 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -1088,7 +1088,7 @@ blkfront_closing(struct blkfront_info *info)
if (bdev->bd_openers) {
xenbus_dev_error(xbdev, -EBUSY,
"Device in use; refusing to close");
- xenbus_switch_state(xbdev, XenbusStateClosing);
+ xbdev->state = XenbusStateClosing;
} else {
xlvbd_release_gendisk(info);
xenbus_frontend_closed(xbdev);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jul-08 16:04 UTC
[Xen-devel] Re: xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
On Fri, Jul 08, 2011 at 03:14:29PM +0800, Joe Jin wrote:> When we do block attach detach test with below steps, umount hang and the > guest unable to shutdown: > > 1. start guest with the latest kernel. > 2. attach new disk by xm-attach in Dom0 > 3. mount new disk in guest > 4. detach the disk by xm-detach in dom0I think you mean xm block-detach and xm-attach? I tried with and without your patch and in both cases I get this in my guest: sh-4.1# mount /dev/xvda /test [ 385.949749] EXT3-fs: barriers not enabled [ 385.960173] kjournald starting. Commit interval 5 seconds [ 385.960418] EXT3-fs (xvda): using internal journal [ 385.960427] EXT3-fs (xvda): mounted filesystem with writeback data mode sh-4.1# [ 411.176887] vbd vbd-51712: 16 Device in use; refusing to close The commands on the other side (Dom0) were: [root@tst009 ~]# xm block-list 6 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 12 770 /local/domain/0/backend/vbd/6/51712 [root@tst009 ~]# xm block-detach 6 51712 Error: Device 51712 (vbd) could not be disconnected. Usage: xm block-detach <Domain> <DevId> [-f|--force] Destroy a domain''s virtual block device. [root@tst009 ~]# xm block-detach 6 51712 -f> 5. umount the partition/disk in guest, command hung. exactly at here, any > IO request to the partition/disk will hang.I get that with the patch and without it: sh-4.1# sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles sh-4.1# df -h Filesystem Size Used Avail Use% Mounted on none 490M 120K 490M 1% /dev none 490M 131M 359M 27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty shm 10M 0 10M 0% /dev/shm var_tmp 10M 0 10M 0% /var/tmp /dev/xvda 20G 173M 19G 1% /test sh-4.1# umount /test Any ideas?> > Checking the code we found when xm-detach command set backend state to > Closing, will trigger blkback_changed() -> blkfront_closing() call. > At the moment, the disk still opened by guest, so frontend will refuse the > request, but in the blkfront_closing(), it send a notification to backend > said that the frontend state switched to Closing, when backend got the > event, it will disconnect from real device, at here any IO request will > be stuck, even tried to release the disk by umount. > > Per our test, below patch fix this issue. > > Signed-off-by: Joe Jin <joe.jin@oracle.com> > Signed-off-by: Annie Li <annie.li@oracle.com> > --- > xen-blkfront.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c > index b536a9c..f6d8ac2 100644 > --- a/drivers/block/xen-blkfront.c > +++ b/drivers/block/xen-blkfront.c > @@ -1088,7 +1088,7 @@ blkfront_closing(struct blkfront_info *info) > if (bdev->bd_openers) { > xenbus_dev_error(xbdev, -EBUSY, > "Device in use; refusing to close"); > - xenbus_switch_state(xbdev, XenbusStateClosing); > + xbdev->state = XenbusStateClosing; > } else { > xlvbd_release_gendisk(info); > xenbus_frontend_closed(xbdev); > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joe Jin
2011-Jul-09 00:26 UTC
[Xen-devel] Re: xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
Konrad, Thanks for the reply, see comments in lines. On 07/09/11 00:04, Konrad Rzeszutek Wilk wrote:> On Fri, Jul 08, 2011 at 03:14:29PM +0800, Joe Jin wrote: >> When we do block attach detach test with below steps, umount hang and the >> guest unable to shutdown: >> >> 1. start guest with the latest kernel. >> 2. attach new disk by xm-attach in Dom0 >> 3. mount new disk in guest >> 4. detach the disk by xm-detach in dom0 > > I think you mean xm block-detach and xm-attach?You are right and sorry for confusing.> > I tried with and without your patch and in both cases I get > this in my guest: > > sh-4.1# mount /dev/xvda /test > [ 385.949749] EXT3-fs: barriers not enabled > [ 385.960173] kjournald starting. Commit interval 5 seconds > [ 385.960418] EXT3-fs (xvda): using internal journal > [ 385.960427] EXT3-fs (xvda): mounted filesystem with writeback data mode > sh-4.1# [ 411.176887] vbd vbd-51712: 16 Device in use; refusing to close > > The commands on the other side (Dom0) were: > > [root@tst009 ~]# xm block-list 6 > Vdev BE handle state evt-ch ring-ref BE-path > 51712 0 0 4 12 770 /local/domain/0/backend/vbd/6/51712 > [root@tst009 ~]# xm block-detach 6 51712 > Error: Device 51712 (vbd) could not be disconnected. > Usage: xm block-detach <Domain> <DevId> [-f|--force] >The error caused by xm block-detach timeout to waiting the dev''s state switch to Closed.> Destroy a domain''s virtual block device. > [root@tst009 ~]# xm block-detach 6 51712 -f >With "--force", it always success but frontend did not disconnected if device opened by someone.> >> 5. umount the partition/disk in guest, command hung. exactly at here, any >> IO request to the partition/disk will hang. > > I get that with the patch and without it: > > sh-4.1# > sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles > > sh-4.1# df -h > Filesystem Size Used Avail Use% Mounted on > none 490M 120K 490M 1% /dev > none 490M 131M 359M 27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty > shm 10M 0 10M 0% /dev/shm > var_tmp 10M 0 10M 0% /var/tmp > /dev/xvda 20G 173M 19G 1% /test > sh-4.1# umount /test > > Any ideas?This caused by backend kthread stopped, any IO request to the real device will hang, that is the patch intend to resolving. Thanks, Joe>> >> Checking the code we found when xm-detach command set backend state to >> Closing, will trigger blkback_changed() -> blkfront_closing() call. >> At the moment, the disk still opened by guest, so frontend will refuse the >> request, but in the blkfront_closing(), it send a notification to backend >> said that the frontend state switched to Closing, when backend got the >> event, it will disconnect from real device, at here any IO request will >> be stuck, even tried to release the disk by umount. >> >> Per our test, below patch fix this issue. >> >> Signed-off-by: Joe Jin <joe.jin@oracle.com> >> Signed-off-by: Annie Li <annie.li@oracle.com> >> --- >> xen-blkfront.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c >> index b536a9c..f6d8ac2 100644 >> --- a/drivers/block/xen-blkfront.c >> +++ b/drivers/block/xen-blkfront.c >> @@ -1088,7 +1088,7 @@ blkfront_closing(struct blkfront_info *info) >> if (bdev->bd_openers) { >> xenbus_dev_error(xbdev, -EBUSY, >> "Device in use; refusing to close"); >> - xenbus_switch_state(xbdev, XenbusStateClosing); >> + xbdev->state = XenbusStateClosing; >> } else { >> xlvbd_release_gendisk(info); >> xenbus_frontend_closed(xbdev);_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jul-09 13:11 UTC
[Xen-devel] Re: xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
> >> 5. umount the partition/disk in guest, command hung. exactly at here, any > >> IO request to the partition/disk will hang. > > > > I get that with the patch and without it: > > > > sh-4.1# > > sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles > > > > sh-4.1# df -h > > Filesystem Size Used Avail Use% Mounted on > > none 490M 120K 490M 1% /dev > > none 490M 131M 359M 27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty > > shm 10M 0 10M 0% /dev/shm > > var_tmp 10M 0 10M 0% /var/tmp > > /dev/xvda 20G 173M 19G 1% /test > > sh-4.1# umount /test > > > > Any ideas? > > This caused by backend kthread stopped, any IO request to the real device will hang, that > is the patch intend to resolving.I get this hang with the patch (and without). Is there a corresponding patch to the backend? (The backend is the same exact version as DomU - 3.0-rc6 + #stable/for-jens). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joe Jin
2011-Jul-11 07:53 UTC
[Xen-devel] Re: xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
On 07/09/11 21:11, Konrad Rzeszutek Wilk wrote:>>>> 5. umount the partition/disk in guest, command hung. exactly at here, any >>>> IO request to the partition/disk will hang. >>> >>> I get that with the patch and without it: >>> >>> sh-4.1# >>> sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles >>> >>> sh-4.1# df -h >>> Filesystem Size Used Avail Use% Mounted on >>> none 490M 120K 490M 1% /dev >>> none 490M 131M 359M 27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty >>> shm 10M 0 10M 0% /dev/shm >>> var_tmp 10M 0 10M 0% /var/tmp >>> /dev/xvda 20G 173M 19G 1% /test >>> sh-4.1# umount /test >>> >>> Any ideas? >> >> This caused by backend kthread stopped, any IO request to the real device will hang, that >> is the patch intend to resolving. > > I get this hang with the patch (and without). Is there a corresponding patch to the backend? > (The backend is the same exact version as DomU - 3.0-rc6 + #stable/for-jens).My guest using linux-3.0 the latest kernel, backend based 2.6.32 kernel. Without the patch, xm block-detach timeout and umount hang, with the patch xm block-detach timeout but umount works fine. Thanks, Joe _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joe Jin
2011-Jul-12 03:31 UTC
[Xen-devel] Re: xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
On 07/09/11 21:11, Konrad Rzeszutek Wilk wrote:>>>> 5. umount the partition/disk in guest, command hung. exactly at here, any >>>> IO request to the partition/disk will hang. >>> >>> I get that with the patch and without it: >>> >>> sh-4.1# >>> sh-4.1# [ 519.814048] block xvda: device/vbd/51712 was hot-unplugged, 1 stale handles >>> >>> sh-4.1# df -h >>> Filesystem Size Used Avail Use% Mounted on >>> none 490M 120K 490M 1% /dev >>> none 490M 131M 359M 27% /lib/modules/3.0.0-rc6-00052-g3edce4b-dirty >>> shm 10M 0 10M 0% /dev/shm >>> var_tmp 10M 0 10M 0% /var/tmp >>> /dev/xvda 20G 173M 19G 1% /test >>> sh-4.1# umount /test >>> >>> Any ideas? >> >> This caused by backend kthread stopped, any IO request to the real device will hang, that >> is the patch intend to resolving. > > I get this hang with the patch (and without). Is there a corresponding patch to the backend? > (The backend is the same exact version as DomU - 3.0-rc6 + #stable/for-jens).I tried xen and linux the latest code, # xm dmesg | grep "Xen version" (XEN) Xen version 4.2-unstable (root@us.oracle.com) () Tue Jul 12 06:51:41 CST 2011 Dom0: # uname -r 3.0.0-rc6-XYZxen Guest # uname -r 3.0.0-rc6-XYZxen without the patch, after detach the device by xm block-detach timeout also umount hang in guest. with my patch, xm block-detach timeout in Dom0, umount in guest worked! Thanks, Joe _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jul-12 15:04 UTC
[Xen-devel] Re: xen-blkfront: Don''t send closing notification to backend in blkfront_closing()
> without the patch, after detach the device by xm block-detach timeout also umount hang in guest. > with my patch, xm block-detach timeout in Dom0, umount in guest worked!Ah, I failed to spot you did the unmount after the timeout. I tried to do the umount after forcing the block-detach. [and testing confirms your patch working properly] Looks good to me - with the only exception that the description needs to have ''xm block-attach'' and ''xm block-detach'' - please change that and resubmit and attach Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: stable@kernel.org and make sure to have Jens in the ''To:'' header. Thanks! Jens, Are you OK picking the amended patch (shortly to be posted) for 3.0-rc7? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel