Hi, list, One customer requests that we should show migration progress bar in ''xl migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so that they could see clearly what happened in migration period. To deal with that, we need to have a method to retrieve migration progress. And we hope such stuff could be finally merged to upstream. How do you think? Besides, currently, there is some debug messages about the transferred pages and remaining dirty pages in libxc xc_domain_save, but that could not be reported to upper layer. We may need a libxl API, which could save the migration status (that could be libxc passed to libxl through pipe or other way); and may need an asyncprogress callback to handle the async thing. Do you have any prefers about how it will be like? Regards, Chunyan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
2013/11/8 Chunyan Liu <cyliu@suse.com>> Hi, list, > > One customer requests that we should show migration progress bar in ''xl > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > that they could see clearly what happened in migration period. To deal with > that, we need to have a method to retrieve migration progress. And we hope > such stuff could be finally merged to upstream. How do you think? > > Besides, currently, there is some debug messages about the transferred > pages and remaining dirty pages in libxc xc_domain_save, but that could not > be reported to upper layer. We may need a libxl API, which could save the > migration status (that could be libxc passed to libxl through pipe or other > way); and may need an asyncprogress callback to handle the async thing. Do > you have any prefers about how it will be like? > > Regards, > Chunyan > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, 2013-11-08 at 16:10 +0800, Chunyan Liu wrote:> > > > > 2013/11/8 Chunyan Liu <cyliu@suse.com> > Hi, list, > > One customer requests that we should show migration progress > bar in ''xl migrate'' or ''virsh migrate'', like ''-h/--hash'' > option in ''rpm'' command, so that they could see clearly what > happened in migration period. To deal with that, we need to > have a method to retrieve migration progress. And we hope such > stuff could be finally merged to upstream. How do you think?The current code uses the xtl logging facilities, which IIRC includes support for progress reporting. xl by default uses the stdio output implementation provided by the library, which I think logs the progress as "\rFOO%" but this could be enhanced to use # instead, either by enhancing the xtl stdio module of by xl providing a customised version (I think I would prefer the first option). It may also be that the priority of these messages is not high enough that they are shown by default. Have you tried "xl -vvv migrate"? Perhaps --hash just needs to adjust the relevant thresholds. I''m not sure what meaning %age complete has for a live migration, doesn''t in guest activity make it hard to predict how far through the process you actually are?> Besides, currently, there is some debug messages about the > transferred pages and remaining dirty pages in libxc > xc_domain_save, but that could not be reported to upper layer.All this logging should be using the xtl infrastructure and should be shown if you use verbose logging (e.g. xl -vvv BLAH). If there are message which aren''t using the correct mechanism then please send patches. Likewise if you think the priority (DEBUG/INFO/WARN etc) given to particular messages is incorrect please propose patches.> We may need a libxl API, which could save the migration > status (that could be libxc passed to libxl through pipe or > other way); and may need an asyncprogress callback to handle > the async thing. Do you have any prefers about how it will be > like?Best to investigate improving the existing logging stuff before going down this route IMHO. BTW: there is no need for an IRC ping within an hour of sending the email. If you don''t hear anything for several days (e.g. up to a week) then maybe it would be appropriate to send a ping, perhaps via email. Otherwise please just assume that people will read their mail before too long. Ian.
Andrew Cooper
2013-Nov-08 11:13 UTC
Re: Discussion: Add API to retrieve migration progress
On 08/11/13 08:05, Chunyan Liu wrote:> Hi, list, > > One customer requests that we should show migration progress bar in > ''xl migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' > command, so that they could see clearly what happened in migration > period. To deal with that, we need to have a method to retrieve > migration progress. And we hope such stuff could be finally merged to > upstream. How do you think?There is no sensible way to determine timing here. A non-live migrate can be approximated based solely %age of ram transmitted. However, with a live migrate, the actions of the live guest affect how long the following iteration takes, and the longer an iteration takes, the more likely it is that further iterations will be needed later. Over the timescales required to live migrate a sensibly sized guest, changed in workload in dom0 can make a meaningful difference in time taken to send an iteration, meaning the live guest can dirty more ram and cause a larger next iteration. ~Andrew> > Besides, currently, there is some debug messages about the transferred > pages and remaining dirty pages in libxc xc_domain_save, but that > could not be reported to upper layer. We may need a libxl API, which > could save the migration status (that could be libxc passed to libxl > through pipe or other way); and may need an asyncprogress callback to > handle the async thing. Do you have any prefers about how it will be like? > > Regards, > Chunyan > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Bamvor Jian Zhang
2013-Nov-12 08:37 UTC
Re: Discussion: Add API to retrieve migration progress
Hi, Ian> On Fri, 2013-11-08 at 16:10 +0800, Chunyan Liu wrote: > > > > 2013/11/8 Chunyan Liu <cyliu@xxxxxxxx> > > Hi, list, > > > > One customer requests that we should show migration progress > > bar in ''xl migrate'' or ''virsh migrate'', like ''-h/--hash'' > > option in ''rpm'' command, so that they could see clearly what > > happened in migration period. To deal with that, we need to > > have a method to retrieve migration progress. And we hope such > > stuff could be finally merged to upstream. How do you think? > > The current code uses the xtl logging facilities, which IIRC includes > support for progress reporting. xl by default uses the stdio output > implementation provided by the library, which I think logs the progress > as "\rFOO%" but this could be enhanced to use # instead, either by > enhancing the xtl stdio module of by xl providing a customised version > (I think I would prefer the first option). It may also be that the > priority of these messages is not high enough that they are shown by > default. Have you tried "xl -vvv migrate"? Perhaps --hash just needs to > adjust the relevant thresholds.yes, "xl -vvv migrate" output lots of logs including migrate progress. it is ok for developers, meanwhile it is hard for users to know the step of current migration. is there any chance to output the migrate progress alone? it would be easier for libvirt or other toolstack to monitoring this progress. regards bamvor> > I''m not sure what meaning %age complete has for a live migration, > doesn''t in guest activity make it hard to predict how far through the > process you actually are? > > > Besides, currently, there is some debug messages about the > > transferred pages and remaining dirty pages in libxc > > xc_domain_save, but that could not be reported to upper layer. > > All this logging should be using the xtl infrastructure and should be > shown if you use verbose logging (e.g. xl -vvv BLAH). If there are > message which aren''t using the correct mechanism then please send > patches. Likewise if you think the priority (DEBUG/INFO/WARN etc) given > to particular messages is incorrect please propose patches. > > > We may need a libxl API, which could save the migration > > status (that could be libxc passed to libxl through pipe or > > other way); and may need an asyncprogress callback to handle > > the async thing. Do you have any prefers about how it will be > > like? > > Best to investigate improving the existing logging stuff before going > down this route IMHO. > > BTW: there is no need for an IRC ping within an hour of sending the > email. If you don''t hear anything for several days (e.g. up to a week) > then maybe it would be appropriate to send a ping, perhaps via email. > Otherwise please just assume that people will read their mail before too > long. > > Ian.
On Tue, 2013-11-12 at 01:37 -0700, Bamvor Jian Zhang wrote:> Hi, Ian > > On Fri, 2013-11-08 at 16:10 +0800, Chunyan Liu wrote: > > > > > > 2013/11/8 Chunyan Liu <cyliu@xxxxxxxx> > > > Hi, list, > > > > > > One customer requests that we should show migration progress > > > bar in ''xl migrate'' or ''virsh migrate'', like ''-h/--hash'' > > > option in ''rpm'' command, so that they could see clearly what > > > happened in migration period. To deal with that, we need to > > > have a method to retrieve migration progress. And we hope such > > > stuff could be finally merged to upstream. How do you think? > > > > The current code uses the xtl logging facilities, which IIRC includes > > support for progress reporting. xl by default uses the stdio output > > implementation provided by the library, which I think logs the progress > > as "\rFOO%" but this could be enhanced to use # instead, either by > > enhancing the xtl stdio module of by xl providing a customised version > > (I think I would prefer the first option). It may also be that the > > priority of these messages is not high enough that they are shown by > > default. Have you tried "xl -vvv migrate"? Perhaps --hash just needs to > > adjust the relevant thresholds. > yes, "xl -vvv migrate" output lots of logs including migrate progress. it > is ok for developers, meanwhile it is hard for users to know the step of > current migration. > is there any chance to output the migrate progress alone?Hopefully by adjusting the default thresholds as I suggested. If not then perhaps a custom xtl logger module, or an enhancement to the existing stdio module, might be needed. Ian.
George Dunlap
2013-Nov-12 11:05 UTC
Re: Discussion: Add API to retrieve migration progress
On Fri, Nov 8, 2013 at 11:13 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:> On 08/11/13 08:05, Chunyan Liu wrote: > > Hi, list, > > One customer requests that we should show migration progress bar in ''xl > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > that they could see clearly what happened in migration period. To deal with > that, we need to have a method to retrieve migration progress. And we hope > such stuff could be finally merged to upstream. How do you think? > > > There is no sensible way to determine timing here. > > A non-live migrate can be approximated based solely %age of ram transmitted. > However, with a live migrate, the actions of the live guest affect how long > the following iteration takes, and the longer an iteration takes, the more > likely it is that further iterations will be needed later. Over the > timescales required to live migrate a sensibly sized guest, changed in > workload in dom0 can make a meaningful difference in time taken to send an > iteration, meaning the live guest can dirty more ram and cause a larger next > iteration.I don''t think people necessarily want a 100% accurate prediction; they just want an idea how far along things are going (and a reassurance that things are actually moving). I think if the algorithm assumes 2 passes and then a clean-up phase, and just hard-codes in assumptions about what percentage each pass will take (e.g., 80% for first pass, 15% for 2nd pass, 5% for final clean-up), the results will be useful, and about as accurate as anyone can expect. -George
On Tue, 2013-11-12 at 11:05 +0000, George Dunlap wrote:> On Fri, Nov 8, 2013 at 11:13 AM, Andrew Cooper > <andrew.cooper3@citrix.com> wrote: > > On 08/11/13 08:05, Chunyan Liu wrote: > > > > Hi, list, > > > > One customer requests that we should show migration progress bar in ''xl > > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > > that they could see clearly what happened in migration period. To deal with > > that, we need to have a method to retrieve migration progress. And we hope > > such stuff could be finally merged to upstream. How do you think? > > > > > > There is no sensible way to determine timing here. > > > > A non-live migrate can be approximated based solely %age of ram transmitted. > > However, with a live migrate, the actions of the live guest affect how long > > the following iteration takes, and the longer an iteration takes, the more > > likely it is that further iterations will be needed later. Over the > > timescales required to live migrate a sensibly sized guest, changed in > > workload in dom0 can make a meaningful difference in time taken to send an > > iteration, meaning the live guest can dirty more ram and cause a larger next > > iteration. > > I don''t think people necessarily want a 100% accurate prediction; they > just want an idea how far along things are going (and a reassurance > that things are actually moving). I think if the algorithm assumes 2 > passes and then a clean-up phase, and just hard-codes in assumptions > about what percentage each pass will take (e.g., 80% for first pass, > 15% for 2nd pass, 5% for final clean-up), the results will be useful, > and about as accurate as anyone can expect.Note that the existing code has some algorithm, or at least it is calling xc_report_progress_step with some numbers. Whatever it is is probably "good enough". Ian.
>>> On 11/12/2013 at 07:16 PM, in message<1384255004.1883.54.camel@kazak.uk.xensource.com>, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Tue, 2013-11-12 at 11:05 +0000, George Dunlap wrote: > > On Fri, Nov 8, 2013 at 11:13 AM, Andrew Cooper > > <andrew.cooper3@citrix.com> wrote: > > > On 08/11/13 08:05, Chunyan Liu wrote: > > > > > > Hi, list, > > > > > > One customer requests that we should show migration progress bar in ''xl > > > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > > > that they could see clearly what happened in migration period. To deal > with > > > that, we need to have a method to retrieve migration progress. And we hope > > > such stuff could be finally merged to upstream. How do you think? > > > > > > > > > There is no sensible way to determine timing here. > > > > > > A non-live migrate can be approximated based solely %age of ram > transmitted. > > > However, with a live migrate, the actions of the live guest affect how > long > > > the following iteration takes, and the longer an iteration takes, the more > > > likely it is that further iterations will be needed later. Over the > > > timescales required to live migrate a sensibly sized guest, changed in > > > workload in dom0 can make a meaningful difference in time taken to send an > > > iteration, meaning the live guest can dirty more ram and cause a larger > next > > > iteration. > > > > I don''t think people necessarily want a 100% accurate prediction; they > > just want an idea how far along things are going (and a reassurance > > that things are actually moving). I think if the algorithm assumes 2 > > passes and then a clean-up phase, and just hard-codes in assumptions > > about what percentage each pass will take (e.g., 80% for first pass, > > 15% for 2nd pass, 5% for final clean-up), the results will be useful, > > and about as accurate as anyone can expect. > > Note that the existing code has some algorithm, or at least it is > calling xc_report_progress_step with some numbers. Whatever it is is > probably "good enough". >Well, I think the problem is not that the log information is enough or not, but that sometimes it''s not convenient if we only have logs. We may need a function like ''query-migrate'' as qemu does, so that we could actively check migrate progress or status at any time during migration, and based on this, we could supply user progress bar or whatever. Regards, Chunyan> Ian. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > >
On Wed, 2013-11-13 at 01:23 -0700, Chun Yan Liu wrote:> > >>> On 11/12/2013 at 07:16 PM, in message > <1384255004.1883.54.camel@kazak.uk.xensource.com>, Ian Campbell > <Ian.Campbell@citrix.com> wrote: > > On Tue, 2013-11-12 at 11:05 +0000, George Dunlap wrote: > > > On Fri, Nov 8, 2013 at 11:13 AM, Andrew Cooper > > > <andrew.cooper3@citrix.com> wrote: > > > > On 08/11/13 08:05, Chunyan Liu wrote: > > > > > > > > Hi, list, > > > > > > > > One customer requests that we should show migration progress bar in ''xl > > > > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > > > > that they could see clearly what happened in migration period. To deal > > with > > > > that, we need to have a method to retrieve migration progress. And we hope > > > > such stuff could be finally merged to upstream. How do you think? > > > > > > > > > > > > There is no sensible way to determine timing here. > > > > > > > > A non-live migrate can be approximated based solely %age of ram > > transmitted. > > > > However, with a live migrate, the actions of the live guest affect how > > long > > > > the following iteration takes, and the longer an iteration takes, the more > > > > likely it is that further iterations will be needed later. Over the > > > > timescales required to live migrate a sensibly sized guest, changed in > > > > workload in dom0 can make a meaningful difference in time taken to send an > > > > iteration, meaning the live guest can dirty more ram and cause a larger > > next > > > > iteration. > > > > > > I don''t think people necessarily want a 100% accurate prediction; they > > > just want an idea how far along things are going (and a reassurance > > > that things are actually moving). I think if the algorithm assumes 2 > > > passes and then a clean-up phase, and just hard-codes in assumptions > > > about what percentage each pass will take (e.g., 80% for first pass, > > > 15% for 2nd pass, 5% for final clean-up), the results will be useful, > > > and about as accurate as anyone can expect. > > > > Note that the existing code has some algorithm, or at least it is > > calling xc_report_progress_step with some numbers. Whatever it is is > > probably "good enough". > > > > Well, I think the problem is not that the log information is enough or not, but > that sometimes it''s not convenient if we only have logs. We may need a function > like ''query-migrate'' as qemu does, so that we could actively check migrate > progress or status at any time during migration, and based on this, we could > supply user progress bar or whatever.Why is the xentoollog_logger.progress callback insufficient for this use? If you wanted to then the application (be that xl or libvirt) could just stash the most recent progress and provide an API to the rest of the ap. Ian.
>>> On 11/13/2013 at 04:27 PM, in message<1384331255.25822.1.camel@dagon.hellion.org.uk>, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Wed, 2013-11-13 at 01:23 -0700, Chun Yan Liu wrote: > > > > >>> On 11/12/2013 at 07:16 PM, in message > > <1384255004.1883.54.camel@kazak.uk.xensource.com>, Ian Campbell > > <Ian.Campbell@citrix.com> wrote: > > > On Tue, 2013-11-12 at 11:05 +0000, George Dunlap wrote: > > > > On Fri, Nov 8, 2013 at 11:13 AM, Andrew Cooper > > > > <andrew.cooper3@citrix.com> wrote: > > > > > On 08/11/13 08:05, Chunyan Liu wrote: > > > > > > > > > > Hi, list, > > > > > > > > > > One customer requests that we should show migration progress bar in ''xl > > > > > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > > > > > that they could see clearly what happened in migration period. To deal > > > with > > > > > that, we need to have a method to retrieve migration progress. And we > hope > > > > > such stuff could be finally merged to upstream. How do you think? > > > > > > > > > > > > > > > There is no sensible way to determine timing here. > > > > > > > > > > A non-live migrate can be approximated based solely %age of ram > > > transmitted. > > > > > However, with a live migrate, the actions of the live guest affect how > > > long > > > > > the following iteration takes, and the longer an iteration takes, the > more > > > > > likely it is that further iterations will be needed later. Over the > > > > > timescales required to live migrate a sensibly sized guest, changed in > > > > > workload in dom0 can make a meaningful difference in time taken to send > an > > > > > iteration, meaning the live guest can dirty more ram and cause a larger > > > > next > > > > > iteration. > > > > > > > > I don''t think people necessarily want a 100% accurate prediction; they > > > > just want an idea how far along things are going (and a reassurance > > > > that things are actually moving). I think if the algorithm assumes 2 > > > > passes and then a clean-up phase, and just hard-codes in assumptions > > > > about what percentage each pass will take (e.g., 80% for first pass, > > > > 15% for 2nd pass, 5% for final clean-up), the results will be useful, > > > > and about as accurate as anyone can expect. > > > > > > Note that the existing code has some algorithm, or at least it is > > > calling xc_report_progress_step with some numbers. Whatever it is is > > > probably "good enough". > > > > > > > Well, I think the problem is not that the log information is enough or not, > but > > that sometimes it''s not convenient if we only have logs. We may need a > function > > like ''query-migrate'' as qemu does, so that we could actively check migrate > > progress or status at any time during migration, and based on this, we > could > > supply user progress bar or whatever. > > Why is the xentoollog_logger.progress callback insufficient for this > use? > > If you wanted to then the application (be that xl or libvirt) could just > stash the most recent progress and provide an API to the rest of the ap.Yes, that is just what we want. We could make use of the xentoollog_logger.progress callback, output the progress info to lg->file, meanwhile store the progress details (context, doing what, done, total, etc.) somewhere and provide an API to get those info for the rest. If I know correctly, only save/restore uses .progress callback, so we could do some changes in stdiostream_progress, and provide API in libxl. Libvirt could then use the logger and API. Chunyan> > Ian. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > >
On Wed, 2013-11-13 at 02:47 -0700, Chun Yan Liu wrote:> > >>> On 11/13/2013 at 04:27 PM, in message > <1384331255.25822.1.camel@dagon.hellion.org.uk>, Ian Campbell > <Ian.Campbell@citrix.com> wrote: > > On Wed, 2013-11-13 at 01:23 -0700, Chun Yan Liu wrote: > > > > > > >>> On 11/12/2013 at 07:16 PM, in message > > > <1384255004.1883.54.camel@kazak.uk.xensource.com>, Ian Campbell > > > <Ian.Campbell@citrix.com> wrote: > > > > On Tue, 2013-11-12 at 11:05 +0000, George Dunlap wrote: > > > > > On Fri, Nov 8, 2013 at 11:13 AM, Andrew Cooper > > > > > <andrew.cooper3@citrix.com> wrote: > > > > > > On 08/11/13 08:05, Chunyan Liu wrote: > > > > > > > > > > > > Hi, list, > > > > > > > > > > > > One customer requests that we should show migration progress bar in ''xl > > > > > > migrate'' or ''virsh migrate'', like ''-h/--hash'' option in ''rpm'' command, so > > > > > > that they could see clearly what happened in migration period. To deal > > > > with > > > > > > that, we need to have a method to retrieve migration progress. And we > > hope > > > > > > such stuff could be finally merged to upstream. How do you think? > > > > > > > > > > > > > > > > > > There is no sensible way to determine timing here. > > > > > > > > > > > > A non-live migrate can be approximated based solely %age of ram > > > > transmitted. > > > > > > However, with a live migrate, the actions of the live guest affect how > > > > long > > > > > > the following iteration takes, and the longer an iteration takes, the > > more > > > > > > likely it is that further iterations will be needed later. Over the > > > > > > timescales required to live migrate a sensibly sized guest, changed in > > > > > > workload in dom0 can make a meaningful difference in time taken to send > > an > > > > > > iteration, meaning the live guest can dirty more ram and cause a larger > > > > > > next > > > > > > iteration. > > > > > > > > > > I don''t think people necessarily want a 100% accurate prediction; they > > > > > just want an idea how far along things are going (and a reassurance > > > > > that things are actually moving). I think if the algorithm assumes 2 > > > > > passes and then a clean-up phase, and just hard-codes in assumptions > > > > > about what percentage each pass will take (e.g., 80% for first pass, > > > > > 15% for 2nd pass, 5% for final clean-up), the results will be useful, > > > > > and about as accurate as anyone can expect. > > > > > > > > Note that the existing code has some algorithm, or at least it is > > > > calling xc_report_progress_step with some numbers. Whatever it is is > > > > probably "good enough". > > > > > > > > > > Well, I think the problem is not that the log information is enough or not, > > but > > > that sometimes it''s not convenient if we only have logs. We may need a > > function > > > like ''query-migrate'' as qemu does, so that we could actively check migrate > > > progress or status at any time during migration, and based on this, we > > could > > > supply user progress bar or whatever. > > > > Why is the xentoollog_logger.progress callback insufficient for this > > use? > > > > If you wanted to then the application (be that xl or libvirt) could just > > stash the most recent progress and provide an API to the rest of the ap. > > Yes, that is just what we want. We could make use of the > xentoollog_logger.progress callback, output the progress info to lg->file, > meanwhile store the progress details (context, doing what, done, total, etc.) > somewhere and provide an API to get those info for the rest. If I know > correctly, only save/restore uses .progress callback, so we could do some > changes in stdiostream_progress, and provide API in libxl. Libvirt could then > use the logger and API.Wouldn''t you want libvirt to have its own logger which directs everything to the standard libvirt logging infrastructure? You are under no obligation to use the stdiostream logger if you don''t want to. Indeed I think it would make sense for libvirt to have its own. Ian.