Xen devel - Jul 2011 - request of blkfront may disorder after migration without waiting in suspend phase of migration

Hi konrad,

I found the mainline linux 3.0-rc6 xen-blkfront driver doesn''t
implement
suspend func. In my opinion, the blkfront should wait till responses of all
the pending requests come back, then domu can be suspended,
otherwise,  after the migration, vm of target host will request from target
host, meanwhile, the blkback of source host maybe do the pending requests
until they''s done.

if migrate the vm with high io pressure,  the requests of this vm may
disorder.

And i saw the gpl windows pv driver, blkfront will change its state and wait
for blkback closing, after blkback closing, the pending requests definitely
are done.

More, i did some test with migration,  while vm was running dd task,  xm
migrate -l .  before it suspend, gdb tapdisk2.  after migration, new vm
started. then quit gdb, the pending request were written back to disk after
new requests.

above mentioned is just my opinion, i need your advice and confirmation or
deny

any help is appreciated.


Regards

wanjia


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, 2011-07-10 at 00:36 -0400, alice wan wrote:
> Hi konrad,
> 
> I found the mainline linux 3.0-rc6 xen-blkfront driver doesn''t
implement suspend func. In my opinion, the blkfront should wait till responses
of all the pending requests come back, then domu can be suspended, otherwise, 
after the migration, vm of target host will request from target host, meanwhile,
the blkback of source host maybe do the pending requests until they''s
done.
It''s not done that way because the suspension alone is rather
transparent to the guest. There''s normally not a problem with that. As
long as I/O completed on the source host before resuming the new machine
image (which will reissue the requests not observed complete).

This is a relatively safe procedure as long as the toolstack makes sure
the backend shut down before handing over.
> if migrate the vm with high io pressure,  the requests of this vm may
disorder.
> 
> And i saw the gpl windows pv driver, blkfront will change its state and
wait for blkback closing, after blkback closing, the pending requests definitely
are done.
Huh? I don''t know much about the GPL PV drivers. I do believe however
shouldn''t be able observe blkback transitioning to Closing. The guest
should have been halted and destroyed before that even happens.
> More, i did some test with migration,  while vm was running dd task,  xm
migrate -l .  before it suspend, gdb tapdisk2.  after migration, new vm started.
then quit gdb, the pending request were written back to disk after new requests.
> 
> above mentioned is just my opinion, i need your advice and confirmation or
deny
> 
> any help is appreciated.
Woah. Could someone with xm know how confirm or deny that? The former
will break disk images. 

It''s quite fundamental that blkback was completely shut down (rm
backend/vbd doesn''t count, it state hit Closed, i.e. xenblkd gone and
bdev release()d) before resuming.

Daniel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, Jul 10, 2011 at 12:36:06PM +0800, alice wan
wrote:
> Hi konrad,
> 
> I found the mainline linux 3.0-rc6 xen-blkfront driver doesn''t
implement
> suspend func. In my opinion, the blkfront should wait till responses of all
> the pending requests come back, then domu can be suspended,
I think you mean blkback. It should be the job of blkback to account for
all I/Os and make sure they have completed before migrating?

I did have a patch for the 2.6.32 to deal with this but I never was able
to reproduce this. Are you able to reproduce this? Can you give me
step by step instructions on how to do it?
> otherwise,  after the migration, vm of target host will request from target
> host, meanwhile, the blkback of source host maybe do the pending requests
> until they''s done.
> 
> if migrate the vm with high io pressure,  the requests of this vm may
> disorder.
> 
> And i saw the gpl windows pv driver, blkfront will change its state and
wait
> for blkback closing, after blkback closing, the pending requests definitely
> are done.
> 
> More, i did some test with migration,  while vm was running dd task,  xm
> migrate -l .  before it suspend, gdb tapdisk2.  after migration, new vm
> started. then quit gdb, the pending request were written back to disk after
> new requests.
> 
> above mentioned is just my opinion, i need your advice and confirmation or
> deny
> 
> any help is appreciated.
> 
> 
> Regards
> 
> wanjia
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2011-07-12 at 19:23 +0100, Konrad Rzeszutek Wilk
wrote:
> On Sun, Jul 10, 2011 at 12:36:06PM +0800, alice wan wrote:
> > Hi konrad,
> > 
> > I found the mainline linux 3.0-rc6 xen-blkfront driver
doesn''t implement
> > suspend func. In my opinion, the blkfront should wait till responses
of all
> > the pending requests come back, then domu can be suspended,
> 
> I think you mean blkback. It should be the job of blkback to account for
> all I/Os and make sure they have completed before migrating?
blkfront has a resume hook which (AIUI) is intended to recover things
after a migration and this seemingly does not require any work on the
suspend path (or at least was presumably thought not to when it was
written...).

Ian.
> 
> I did have a patch for the 2.6.32 to deal with this but I never was able
> to reproduce this. Are you able to reproduce this? Can you give me
> step by step instructions on how to do it?
> 
> > otherwise,  after the migration, vm of target host will request from
target
> > host, meanwhile, the blkback of source host maybe do the pending
requests
> > until they''s done.
> > 
> > if migrate the vm with high io pressure,  the requests of this vm may
> > disorder.
> > 
> > And i saw the gpl windows pv driver, blkfront will change its state
and wait
> > for blkback closing, after blkback closing, the pending requests
definitely
> > are done.
> > 
> > More, i did some test with migration,  while vm was running dd task, 
xm
> > migrate -l .  before it suspend, gdb tapdisk2.  after migration, new
vm
> > started. then quit gdb, the pending request were written back to disk
after
> > new requests.
> > 
> > above mentioned is just my opinion, i need your advice and
confirmation or
> > deny
> > 
> > any help is appreciated.
> > 
> > 
> > Regards
> > 
> > wanjia
> 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

2011/7/13 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> On Sun, Jul 10, 2011 at 12:36:06PM +0800, alice wan wrote:
> > Hi konrad,
> >
> > I found the mainline linux 3.0-rc6 xen-blkfront driver
doesn''t implement
> > suspend func. In my opinion, the blkfront should wait till responses
of
> all
> > the pending requests come back, then domu can be suspended,
>
> I think you mean blkback. It should be the job of blkback to account for
> all I/Os and make sure they have completed before migrating?
>
> I did have a patch for the 2.6.32 to deal with this but I never was able
> to reproduce this. Are you able to reproduce this? Can you give me
> step by step instructions on how to do it?
>
yeah,  just because i didn''t see any code in blkback to waiting for IO
before migrating,
blkfront''s the rest one to do this job that it actively switches to
Closing
state, then blkback can observe it and call blkif_disconnect to account for
all IO.

my test procedure is like this
> More, i did some test with migration,  while vm was running dd task,  xm
> migrate -l .  before it suspend, gdb tapdisk2.  after migration, new vm
> started. then quit gdb, the pending request were written back to disk
after
> new requests.
gdb tapdisk2 , the IO of src host hanged, when xc_save did suspend, after
100s, socket was shutted down, Vm of target host resumed and continued to
dd, then quit gdb.

this test is quite extreme. normal migration depends on whether all IO can
be completed in 100s.

well, i need the IO have 100% safety, especially in the distributed fs.

of course, blkback is the best choice to ensure this.
>
> > otherwise,  after the migration, vm of target host will request from
> target
> > host, meanwhile, the blkback of source host maybe do the pending
requests
> > until they''s done.
> >
> > if migrate the vm with high io pressure,  the requests of this vm may
> > disorder.
> >
> > And i saw the gpl windows pv driver, blkfront will change its state
and
> wait
> > for blkback closing, after blkback closing, the pending requests
> definitely
> > are done.
> >
> > More, i did some test with migration,  while vm was running dd task, 
xm
> > migrate -l .  before it suspend, gdb tapdisk2.  after migration, new
vm
> > started. then quit gdb, the pending request were written back to disk
> after
> > new requests.
> >
> > above mentioned is just my opinion, i need your advice and
confirmation
> or
> > deny
> >
> > any help is appreciated.
> >
> >
> > Regards
> >
> > wanjia
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2011-07-13 at 05:09 -0400, Ian Campbell wrote: 
> On Tue, 2011-07-12 at 19:23 +0100, Konrad Rzeszutek Wilk wrote:
> > On Sun, Jul 10, 2011 at 12:36:06PM +0800, alice wan wrote:
> > > Hi konrad,
> > > 
> > > I found the mainline linux 3.0-rc6 xen-blkfront driver
doesn''t implement
> > > suspend func. In my opinion, the blkfront should wait till
responses of all
> > > the pending requests come back, then domu can be suspended,
> > 
> > I think you mean blkback. It should be the job of blkback to account
for
> > all I/Os and make sure they have completed before migrating?
> 
> blkfront has a resume hook which (AIUI) is intended to recover things
> after a migration and this seemingly does not require any work on the
> suspend path (or at least was presumably thought not to when it was
> written...).
>From a pre-copy migration perspective, adding a suspend hook to synch on
pending I/O is not ideal. It''s just going to add unsolicited memory
footprint, with fairly arbitrary amounts of additional memory getting
dirtied (''arbitrary'' depending on what else is allowed to keep
running
in the guest in that state).

I guess that''s why post-migration recovery was preferred, and I still
don''t see why that''s to blame.

Daniel
> Ian.
> 
> > 
> > I did have a patch for the 2.6.32 to deal with this but I never was
able
> > to reproduce this. Are you able to reproduce this? Can you give me
> > step by step instructions on how to do it?
> > 
> > > otherwise,  after the migration, vm of target host will request
from target
> > > host, meanwhile, the blkback of source host maybe do the pending
requests
> > > until they''s done.
> > > 
> > > if migrate the vm with high io pressure,  the requests of this vm
may
> > > disorder.
> > > 
> > > And i saw the gpl windows pv driver, blkfront will change its
state and wait
> > > for blkback closing, after blkback closing, the pending requests
definitely
> > > are done.
> > > 
> > > More, i did some test with migration,  while vm was running dd
task,  xm
> > > migrate -l .  before it suspend, gdb tapdisk2.  after migration,
new vm
> > > started. then quit gdb, the pending request were written back to
disk after
> > > new requests.
> > > 
> > > above mentioned is just my opinion, i need your advice and
confirmation or
> > > deny
> > > 
> > > any help is appreciated.
> > > 
> > > 
> > > Regards
> > > 
> > > wanjia
> > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel