Xen devel - Jan 2010 - Xen guest disk online resize, xenstore/blkback/blkfront questions

Hello,

I started implementing support for on-the-fly/online resizing of Xen PV
guest disks.

I added ''block-resize'' command to xm, and added
device_resize() function
to xend.

I''m able to do ''xm block-resize <guest>
<guest_disk>" and my new
device_resize() function in xend gets called. All fine so far.

Now I''m trying to understand the internals of how kernel
blkback/blkfront and xenstore interact with each other, so I can
implement the actual resizing.

This is how I''ve understood the flow of adding/creating a block device
in Xen:

- blkback driver in dom0 kernel runs xenbus_register_backend(), which
  makes xenstore to notify blkback about new block devices via xenbus.

- When xend DevController.createDevice() for new block device gets called, 
  it writes the initial block device configuration to xenstore. 
  
- Xenstore then notifies blkback via xenbus, and the blkback_probe() function 
  of blkback gets called.

- blkback_probe() creates the backend device in dom0 kernel, and
  sets up a xenbus_watch to get notified about
''physical-device'' changes
  in xenstore block device backend information.

- udev hotplug scripts get executed when the new backend device is
  created in the dom0 kernel, and when the udev hotplug scripts are done 
  they update the xenstore block device backend
''physical-device'' information
  to trigger blkback notification.
  (what do the udev hotplug scripts actually do here?)

- Xenstore block device backend ''physical-device'' updates
cause xenbus_watch to call
  blkback backend_changed(). Physical device major/minor, mode and type
  are then fetched from xenstore. Then vbd is created in the kernel, 
  and update_blkif_status() is called.

- update_blkif_status() calls connect() which tries to connect the
  backend with the blkfront frontend.

- connect() writes block device size in sectors and sector-size
  to xenstore, and changes the state to ''connected'' so
frontend can
  fetch the information from xenstore and bring itself online.


Is this more or less correct? Please correct if I''ve missed something
or
understood something wrong.

So, what I need to do next:

- LVM online-resize the guest disk LV in dom0.

- write something to xenstore block device backend structures, 
  to get the blkback driver notified about the ''block-resize''.

  Should I add a new xenbus_watch for some, say, ''resize''
field, so I could get
  callback to blkback device_resize() easily when xenstore is updated? 

- blkback driver then needs to update/fetch the new size of the vbd, 
  and update the xenstore /local/domain/0/backend/vbd/X/sectors field.

  Any problems getting the new size/sectors on-the-fly in the kernel? 

- blkback then needs to write something to xenstore block device frontend
  /local/domain/X/device/vbd/Y/ to notify the blkfront driver in the guest.

  Same thing here, should blkfront have a watch for some
''resize'' field
  or so?


How does that sound like? All comments and help appreciated!

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2010-01-06 at 20:26 +0000, Pasi Kärkkäinen wrote:
[...] 
> Is this more or less correct? Please correct if I''ve missed
something or
> understood something wrong.
All sounds vaguely familiar, although it''s been a while since I last
had
to follow those twisty paths around...
> So, what I need to do next:
> 
> - LVM online-resize the guest disk LV in dom0.
> 
> - write something to xenstore block device backend structures, 
>   to get the blkback driver notified about the
''block-resize''.
> 
>   Should I add a new xenbus_watch for some, say, ''resize''
field, so I could get
>   callback to blkback device_resize() easily when xenstore is updated? 
> 
> - blkback driver then needs to update/fetch the new size of the vbd, 
>   and update the xenstore /local/domain/0/backend/vbd/X/sectors field.
> 
>   Any problems getting the new size/sectors on-the-fly in the kernel? 
You might need to close and reopen the block device? 

Idle speculation: perhaps instead of an explicit "resize" field in
xenstore you could just have the backend continue watching the
''physical-device'' node even after everything is connected the
first time
and if it is rewritten (including to the same value) reopening the
physical device and setting things up again (picking up a size change as
a side effect).

There''s no particular reason why the physical device couldn''t
change
over this operation either, you could maybe imagine changing to a
different device mapper node (e.g. perhaps some sort of wierd snapshot
mechanism?) or perhaps implementing some sort of PV-CDROM media change
in the same way.
> - blkback then needs to write something to xenstore block device frontend
>   /local/domain/X/device/vbd/Y/ to notify the blkfront driver in the guest.
> 
>   Same thing here, should blkfront have a watch for some
''resize'' field
>   or so?
blkfront can probably just watch /local/domain/0/backend/vbd/X/sectors

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jan 06, 2010 at 08:44:25PM +0000, Ian Campbell
wrote:
> On Wed, 2010-01-06 at 20:26 +0000, Pasi Kärkkäinen wrote:
> [...] 
> > Is this more or less correct? Please correct if I''ve missed
something or
> > understood something wrong.
> 
> All sounds vaguely familiar, although it''s been a while since I
last had
> to follow those twisty paths around...
> 
> > So, what I need to do next:
> > 
> > - LVM online-resize the guest disk LV in dom0.
> > 
> > - write something to xenstore block device backend structures, 
> >   to get the blkback driver notified about the
''block-resize''.
> > 
> >   Should I add a new xenbus_watch for some, say,
''resize'' field, so I could get
> >   callback to blkback device_resize() easily when xenstore is updated?
> > 
> > - blkback driver then needs to update/fetch the new size of the vbd, 
> >   and update the xenstore /local/domain/0/backend/vbd/X/sectors field.
> > 
> >   Any problems getting the new size/sectors on-the-fly in the kernel? 
> 
> You might need to close and reopen the block device? 
> 
> Idle speculation: perhaps instead of an explicit "resize" field
in
> xenstore you could just have the backend continue watching the
> ''physical-device'' node even after everything is connected
the first time
> and if it is rewritten (including to the same value) reopening the
> physical device and setting things up again (picking up a size change as
> a side effect).
> 
Good idea. I''ll experiment with this.
> There''s no particular reason why the physical device
couldn''t change
> over this operation either, you could maybe imagine changing to a
> different device mapper node (e.g. perhaps some sort of wierd snapshot
> mechanism?) or perhaps implementing some sort of PV-CDROM media change
> in the same way.
> 
> > - blkback then needs to write something to xenstore block device
frontend
> >   /local/domain/X/device/vbd/Y/ to notify the blkfront driver in the
guest.
> > 
> >   Same thing here, should blkfront have a watch for some
''resize'' field
> >   or so?
> 
> blkfront can probably just watch /local/domain/0/backend/vbd/X/sectors
> 
Sounds good. 
Thanks for the reply!

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2010-01-06 at 16:57 -0500, Pasi Kärkkäinen wrote:
> > >   Any problems getting the new size/sectors on-the-fly in the
kernel?
> > 
> > You might need to close and reopen the block device? 
> > 
> > Idle speculation: perhaps instead of an explicit "resize"
field in
> > xenstore you could just have the backend continue watching the
> > ''physical-device'' node even after everything is
connected the first time
> > and if it is rewritten (including to the same value) reopening the
> > physical device and setting things up again (picking up a size change
as
> > a side effect).
> > 
> 
> Good idea. I''ll experiment with this.
The XCP kernel has support for transparent pause/unpause. Which takes of
the queue flush in order to get the bdev closed.

Below is that patch which enables switching the physical node,
safety-checking if the VBD is paused etc. Matters not so much for
snapshotting, rather for cross-vg stuff.

It happens to work otherwise as IanC suggested.

It may be more than you''re up to in order to get a resize done, e.g.
depends on whether you want to piggyback close/open into pause/unpause
or not (as a rule of thumb, it''s safe to say that pause applications
want a close anyway :]). 

Didn''t check for bitrot, sorry in advance.

HTH. Let me know if you''re missing something or how it''s
going.

Cheers,
Daniel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Jan 07, 2010 at 11:54:19PM -0800, Daniel Stodden
wrote:
> On Wed, 2010-01-06 at 16:57 -0500, Pasi Kärkkäinen wrote:
> 
> > > >   Any problems getting the new size/sectors on-the-fly in
the kernel?
> > > 
> > > You might need to close and reopen the block device? 
> > > 
> > > Idle speculation: perhaps instead of an explicit
"resize" field in
> > > xenstore you could just have the backend continue watching the
> > > ''physical-device'' node even after everything is
connected the first time
> > > and if it is rewritten (including to the same value) reopening
the
> > > physical device and setting things up again (picking up a size
change as
> > > a side effect).
> > > 
> > 
> > Good idea. I''ll experiment with this.
> 
> The XCP kernel has support for transparent pause/unpause. Which takes of
> the queue flush in order to get the bdev closed.
> 
> Below is that patch which enables switching the physical node,
> safety-checking if the VBD is paused etc. Matters not so much for
> snapshotting, rather for cross-vg stuff.
> 
> It happens to work otherwise as IanC suggested.
> 
> It may be more than you''re up to in order to get a resize done,
e.g.
> depends on whether you want to piggyback close/open into pause/unpause
> or not (as a rule of thumb, it''s safe to say that pause
applications
> want a close anyway :]). 
> 
> Didn''t check for bitrot, sorry in advance.
>
Interesting. I''ll take a look at this aswell. Thanks!
> HTH. Let me know if you''re missing something or how it''s
going.
> 
Yep. I''ll get back to you. 

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Has anyone tried is this method ok?

--
View this message in context:
http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447334.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

On Wed, Feb 01, 2012 at 04:29:58AM -0800, feisky wrote:
> Has anyone tried is this method ok?
> 
Yep, I''ve used it, and verified it works.

-- Pasi

Could you send your patch to me?

thanks.

On Wed, Feb 1, 2012 at 11:52 PM, Pasi Kärkkäinen [via Xen] <
ml-node+s1045712n5447888h47@n5.nabble.com> wrote:
> On Wed, Feb 01, 2012 at 04:29:58AM -0800, feisky wrote:
> > Has anyone tried is this method ok?
> >
>
> Yep, I''ve used it, and verified it works.
>
> -- Pasi
>
>
> _______________________________________________
> Xen-devel mailing list
> [hidden email]
<http://user/SendEmail.jtp?type=node&node=5447888&i=0>
> http://lists.xensource.com/xen-devel
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
>
http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447888.html
>  To unsubscribe from Xen guest disk online resize,
> xenstore/blkback/blkfront questions, click
here<http://xen.1045712.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2547847&code=ZmVpc2t5ZXJAZ21haWwuY29tfDI1NDc4NDd8MTg5Nzg0NTI4Nw==>
> .
>
NAML<http://xen.1045712.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>

--
View this message in context:
http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5449661.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Feb 01, 2012 at 09:31:54PM -0800, feisky wrote:
>    Could you send your patch to me?Â
>    thanks.
> 
It doesn't require patches. Support for online domU disk resize
is in upstream kernel.org kernels (and in rhel/sles vendor kernels).

It only requires support from xen-blkback and xen-blkfront,
and LVM volumes in dom0.

-- Pasi
>    On Wed, Feb 1, 2012 at 11:52 PM, Pasi KÀrkkÀinen [via Xen]
<[1][hidden
>    email]> wrote:
> 
>      On Wed, Feb 01, 2012 at 04:29:58AM -0800, feisky wrote:
>      > Has anyone tried is this method ok?
>      >
> 
>      Yep, I've used it, and verified it works.
> 
>      -- Pasi
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [2][hidden email]
>      [3]http://lists.xensource.com/xen-devel
> 
>   
--------------------------------------------------------------------------
> 
>      If you reply to this email, your message will be added to the
discussion
>      below:
>     
[4]http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447888.html
>      To unsubscribe from Xen guest disk online resize,
>      xenstore/blkback/blkfront questions, [5]click here.
>      [6]NAML
> 
>    -------------------------------------------
> 
>    View this message in context: [7]Re: Xen guest disk online resize,
>    xenstore/blkback/blkfront questions
>    Sent from the [8]Xen - Dev mailing list archive at Nabble.com.
> 
> References
> 
>    Visible links
>    1. file:///user/SendEmail.jtp?type=node&node=5449661&i=0
>    2. http://user/SendEmail.jtp?type=node&node=5447888&i=0
>    3. http://lists.xensource.com/xen-devel
>    4.
http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447888.html
>    5. file:///tmp/
>    6.
http://xen.1045712.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>    7.
http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5449661.html
>    8. http://xen.1045712.n5.nabble.com/Xen-Dev-f2473738.html
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel