Xen devel - Sep 2012 - Re: xen-vbd interface (segment size expansion) - FreeBSD host have it.

On Mon, Aug 20, 2012 at 06:56:02PM +0000, Justin Gibbs
wrote:
> Ronghui,
> 
> It has been a while since I''ve actively worked on the blkif stuff.
...
> 
> That said, I''m happy to help in whatever ways I can to help get
the blkif interface sorted out.  I see several steps that should be taken:
> 
> 1) Fix the Xen and QEMU builds so that QEMU keeps its own copy of the Xen
interface version.  This will allow interfaces to rev safely and in a
coordinated fashion (i.e. update interface in Xen, then add support for the new
interface in QEMU upstream).
> 
> 2) Complete support in drivers for the existing blkif interface so that you
get maximum performance against systems using the existing multi-page
extensions.
> 
> 3) Do something to allow for larger and more numerous requests.
> 
> On point 3, my approach was to try to perturb the existing protocol as
little as possible in the hopes that other implementations could quickly be
enhanced to support the feature.  However, there is a lot of ugliness in the
existing blkif interface.  I can certainly understand the desires of some to
just replace blkif with a blkif2.
> 
> What are your current plans in this area?  How can I be of assistance?
Hey Justin and Ronghui,

Note: I''ve expanded the email thread to include xen-devel.

I was wondering how the protocol you developed works when it comes
to migration to a host that does not support the new features?

Specifically how do deal with a guest which tries to replay in progress I/Os
that do not fit within the old-segment size (11)?

Hi, I am back from a long holiday. Sorry for the delay
replay.
> I was wondering how the protocol you developed works when it comes to
> migration to a host that does not support the new features?
>
For my part, when VM migrate to a host which do not support larger segments,
frontend will fall back to use original protocol with old-segments 11.
> Specifically how do deal with a guest which tries to replay in progress
I/Os
> that do not fit within the old-segment size (11)?
The in-progress bio which have larger segments > 11 will receive an io error.
I do not find a better way to handle it yet.
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 25, 2012 11:26 PM
> To: Justin Gibbs; xen-devel@lists.xensource.com
> Cc: Duan, Ronghui
> Subject: Re: xen-vbd interface (segment size expansion) - FreeBSD host have
> it.
> 
> On Mon, Aug 20, 2012 at 06:56:02PM +0000, Justin Gibbs wrote:
> > Ronghui,
> >
> > It has been a while since I''ve actively worked on the blkif
stuff.  ...
> >
> > That said, I''m happy to help in whatever ways I can to help
get the blkif
> interface sorted out.  I see several steps that should be taken:
> >
> > 1) Fix the Xen and QEMU builds so that QEMU keeps its own copy of the
> Xen interface version.  This will allow interfaces to rev safely and in a
> coordinated fashion (i.e. update interface in Xen, then add support for the
> new interface in QEMU upstream).
> >
> > 2) Complete support in drivers for the existing blkif interface so
that you
> get maximum performance against systems using the existing multi-page
> extensions.
> >
> > 3) Do something to allow for larger and more numerous requests.
> >
> > On point 3, my approach was to try to perturb the existing protocol as
little
> as possible in the hopes that other implementations could quickly be
> enhanced to support the feature.  However, there is a lot of ugliness in
the
> existing blkif interface.  I can certainly understand the desires of some
to just
> replace blkif with a blkif2.
> >
> > What are your current plans in this area?  How can I be of assistance?
> 
> Hey Justin and Ronghui,
> 
> Note: I''ve expanded the email thread to include xen-devel.
> 
> I was wondering how the protocol you developed works when it comes to
> migration to a host that does not support the new features?
> 
> Specifically how do deal with a guest which tries to replay in progress
I/Os
> that do not fit within the old-segment size (11)?

On Thu, Oct 18, 2012 at 01:59:36AM +0000, Duan, Ronghui
wrote:
> Hi, I am back from a long holiday. Sorry for the delay replay.
> > I was wondering how the protocol you developed works when it comes to
> > migration to a host that does not support the new features?
> >
> For my part, when VM migrate to a host which do not support larger
segments, frontend will fall back to use original protocol with old-segments 11.
> 
> > Specifically how do deal with a guest which tries to replay in
progress I/Os
> > that do not fit within the old-segment size (11)?
> The in-progress bio which have larger segments > 11 will receive an io
error. I do not find a better way to handle it yet.
Justin, how did you guys handle it in FreeBSD? Or is it dependent on
the backends always supporting these large segments?

thanks!
> 
> > -----Original Message-----
> > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> > Sent: Tuesday, September 25, 2012 11:26 PM
> > To: Justin Gibbs; xen-devel@lists.xensource.com
> > Cc: Duan, Ronghui
> > Subject: Re: xen-vbd interface (segment size expansion) - FreeBSD host
have
> > it.
> > 
> > On Mon, Aug 20, 2012 at 06:56:02PM +0000, Justin Gibbs wrote:
> > > Ronghui,
> > >
> > > It has been a while since I''ve actively worked on the
blkif stuff.  ...
> > >
> > > That said, I''m happy to help in whatever ways I can to
help get the blkif
> > interface sorted out.  I see several steps that should be taken:
> > >
> > > 1) Fix the Xen and QEMU builds so that QEMU keeps its own copy of
the
> > Xen interface version.  This will allow interfaces to rev safely and
in a
> > coordinated fashion (i.e. update interface in Xen, then add support
for the
> > new interface in QEMU upstream).
> > >
> > > 2) Complete support in drivers for the existing blkif interface
so that you
> > get maximum performance against systems using the existing multi-page
> > extensions.
> > >
> > > 3) Do something to allow for larger and more numerous requests.
> > >
> > > On point 3, my approach was to try to perturb the existing
protocol as little
> > as possible in the hopes that other implementations could quickly be
> > enhanced to support the feature.  However, there is a lot of ugliness
in the
> > existing blkif interface.  I can certainly understand the desires of
some to just
> > replace blkif with a blkif2.
> > >
> > > What are your current plans in this area?  How can I be of
assistance?
> > 
> > Hey Justin and Ronghui,
> > 
> > Note: I''ve expanded the email thread to include xen-devel.
> > 
> > I was wondering how the protocol you developed works when it comes to
> > migration to a host that does not support the new features?
> > 
> > Specifically how do deal with a guest which tries to replay in
progress I/Os
> > that do not fit within the old-segment size (11)?

On Oct 18, 2012, at 8:50 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com>
 wrote:
> On Thu, Oct 18, 2012 at 01:59:36AM +0000, Duan, Ronghui wrote:
>> Hi, I am back from a long holiday. Sorry for the delay replay.
>>> I was wondering how the protocol you developed works when it comes
to
>>> migration to a host that does not support the new features?
>>> 
>> For my part, when VM migrate to a host which do not support larger
segments, frontend will fall back to use original protocol with old-segments 11.
>> 
>>> Specifically how do deal with a guest which tries to replay in
progress I/Os
>>> that do not fit within the old-segment size (11)?
>> The in-progress bio which have larger segments > 11 will receive an
io error. I do not find a better way to handle it yet.
> 
> Justin, how did you guys handle it in FreeBSD? Or is it dependent on
> the backends always supporting these large segments?
The current API forces I/Os larger than 44k to get chopped up in
order to transit blkif.  The ability to negotiate a larger blkif
request size just means that you can sometimes reduce the amount
of "scatter-gather" work performed by the driver.  You must still
deal with the fact that a logical I/O submitted to blkfront may
need to be broken up, or may not completely fit within the size of
the negotiated ring.  Upon reconnect, the newly negotiated parameters
are in effect and reissued I/Os are subject to the rules that apply
to that connection.

I''d have to go review the FreeBSD blkfront driver to see if it
handles correctly all of the issues that arise with a ring that
shrinks (e.g. it may assume that an I/O will fit within an empty
ring), but there is certainly no technical reason that these issues
can''t be addressed.

--
Justin

On Thu, Oct 18, 2012 at 03:34:27PM +0000, Justin Gibbs
wrote:
> On Oct 18, 2012, at 8:50 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com>
>  wrote:
> 
> > On Thu, Oct 18, 2012 at 01:59:36AM +0000, Duan, Ronghui wrote:
> >> Hi, I am back from a long holiday. Sorry for the delay replay.
> >>> I was wondering how the protocol you developed works when it
comes to
> >>> migration to a host that does not support the new features?
> >>> 
> >> For my part, when VM migrate to a host which do not support larger
segments, frontend will fall back to use original protocol with old-segments 11.
> >> 
> >>> Specifically how do deal with a guest which tries to replay in
progress I/Os
> >>> that do not fit within the old-segment size (11)?
> >> The in-progress bio which have larger segments > 11 will
receive an io error. I do not find a better way to handle it yet.
> > 
> > Justin, how did you guys handle it in FreeBSD? Or is it dependent on
> > the backends always supporting these large segments?
> 
> The current API forces I/Os larger than 44k to get chopped up in
> order to transit blkif.  The ability to negotiate a larger blkif
> request size just means that you can sometimes reduce the amount
> of "scatter-gather" work performed by the driver.  You must still
> deal with the fact that a logical I/O submitted to blkfront may
> need to be broken up, or may not completely fit within the size of
> the negotiated ring.  Upon reconnect, the newly negotiated parameters
> are in effect and reissued I/Os are subject to the rules that apply
> to that connection.
> I''d have to go review the FreeBSD blkfront driver to see if it
> handles correctly all of the issues that arise with a ring that
> shrinks (e.g. it may assume that an I/O will fit within an empty
> ring), but there is certainly no technical reason that these issues
> can''t be addressed.
OK, so the one issue I keep on circling back to is there an benefit
of doing a seperate ring for segments - as opposed of re-using the existing
ring and just shrinking/expanding the number of segments?

I do like re-using the ring and just shrinking/expanding it b/c it
ends up being all neatly tucked inside the existing ring.

> 
> --
> Justin