Xen devel - Jun 2013 - Shutdown of HVM domains without PC drivers [was Re: Early ACPI events prevent subsequent ACPI functionality on xen 4.3 + HVM domU]

Sander,

(new thread started for you)

--On 27 June 2013 15:04:58 +0100 George Dunlap 
<george.dunlap@eu.citrix.com> wrote:
> On 21/05/13 18:31, Sander Eikelenboom wrote:
>
>> This reminds me that there still is the issue that a HVM domain without
>> PV drivers doesn''t shutdown properly with the standard init.d
scripts.
>
> BTW, if you want this to get attention, you should probably either reply
> to that thread, or start a new thread.  Starting a new topic on an
> unrelated thread is generally frowned upon. :-)
Something needs to tell the guest to shut down. Without pvdrivers,
you''ll
need to do it through ACPI. Do you have ACPI installed, and are you
sending an xl power event?

-- 
Alex Bligh

Thursday, June 27, 2013, 5:59:04 PM, you wrote:
> Sander,
> (new thread started for you)
> --On 27 June 2013 15:04:58 +0100 George Dunlap 
> <george.dunlap@eu.citrix.com> wrote:
>> On 21/05/13 18:31, Sander Eikelenboom wrote:
>>
>>> This reminds me that there still is the issue that a HVM domain
without
>>> PV drivers doesn''t shutdown properly with the standard
init.d scripts.
>>
>> BTW, if you want this to get attention, you should probably either
reply
>> to that thread, or start a new thread.  Starting a new topic on an
>> unrelated thread is generally frowned upon. :-)
> Something needs to tell the guest to shut down. Without pvdrivers,
you''ll
> need to do it through ACPI. Do you have ACPI installed, and are you
> sending an xl power event?

Hi Alex/George,

The problem is NOT with a manual sent command by a admin, who has both knowledge
and can (directly) respond to warning that is issued (although it''s a
bit harder in the --all case).

It''s with automated scripts using "xl shutdown [-a]", in
combination with HVM guests which do not have PV drivers installed.

For example:
- with the init.d/xendomains script on host shutdown when using complete
shutdown of guests instead of save/restore (not the default in xen hotplug
scripts)
  The guest shutdown will eventually timeout (default after 5 minutes) and the
guest will be destroyed, both are not very desirable.
- but also with other automated scripts using "xl shutdown [-a]", like
the tests scripts in automated xen test environment, which for quite some time
issue problems with guest stop.

There are many ways to solve this ..

a) Make the scripts issuing a "xl shutdown" smarter, but this gets
complicated for the "xl shutdown --all" case.
b) Make hotplug scripts and testscripts always use "-F" on host
shutdown
c) Inverting the -F logic, make "xl shutdown" always try acpi power
event when needed (no pv drivers) and make a option to suppress this behavior.
d) Make it possible to specify the desired behavior on guest shutdown in the
.cfg file

I personally have mitigated the issue by using option "b", and using
  XENDOMAINS_SHUTDOWN="--wait -F"
  XENDOMAINS_SHUTDOWN_ALL="--all --wait -F"
in my /etc/default/xendomains.

For the test environment that would mean IanJ has to change the xl shutdown
commands there to issue the -F as well (or install the pv-drivers)

For option "C", IanC was concerned for hvm guests which don''t
have the shutdown but a different action coupled to the acpi power event and no
pv-drivers installed.

So although it''s an itch, it''s not a major one.
(perhaps the biggest is the test environment spitting out failures when
it''s actually the currently desired behavior :-), as without this one
could get complete failure free runs .. which IMHO is a worthy goal to strive
for with automated testing)

--
Sander

Sander,

--On 27 June 2013 19:15:22 +0200 Sander Eikelenboom <linux@eikelenboom.it>
wrote:
>>> On 21/05/13 18:31, Sander Eikelenboom wrote:
>>>
>>>> This reminds me that there still is the issue that a HVM domain
without
>>>> PV drivers doesn''t shutdown properly with the standard
init.d scripts.
>>>
>>> BTW, if you want this to get attention, you should probably either
reply
>>> to that thread, or start a new thread.  Starting a new topic on an
>>> unrelated thread is generally frowned upon. :-)
>
>> Something needs to tell the guest to shut down. Without pvdrivers,
you''ll
>> need to do it through ACPI. Do you have ACPI installed, and are you
>> sending an xl power event?
>
>
> Hi Alex/George,
>
> The problem is NOT with a manual sent command by a admin, who has both
> knowledge and can (directly) respond to warning that is issued (although
> it''s a bit harder in the --all case).
>
> It''s with automated scripts using "xl shutdown [-a]", in
combination with
> HVM guests which do not have PV drivers installed.
Yes I know. We have this problem.

xl shutdown and friends only send the PV signals (I believe). As your
guest has no PV drivers, I believe this will do nothing (xl might wait
for a timeout and destroy)

You can use the xl power events to send ACPI signals to shutdown guests
without PV drivers. This also works for guests with PV drivers if they
have ACPI in.

So, the way we get round this is not to use the xl commands (well, libxl
calls in our case) that use the PV interface at all. We simply use
the ACPI stuff (the power events), which works whether or not the guest
has PV support. This requires the guest to have ACPI installed for a clean
shutdown, which is exactly the same requirement for KVM, and hence works
well.

We have found (for various reasons) it may be necessary to send the
power event multiple times until something happens, rather than simply
wait for a timeout.

If a shutdown is used, and you send it to early in the boot process,
then subsequent shutdowns may not work - I reported a bug on that
(it''s somewhere on the not-for-4.3 list). This in my book another
reason to avoid xl shutdown and use the power events instead. The
only disadvantage of this strategy is if you have guests which have
PV drivers but not ACPI; these tend to be machines built for PV rather
than HVM, being run on HVM (e.g. AMIs someone has stuck a kernel
in and hoped for the best).

-- 
Alex Bligh

Thursday, June 27, 2013, 9:31:53 PM, you wrote:
> Sander,
> --On 27 June 2013 19:15:22 +0200 Sander Eikelenboom
<linux@eikelenboom.it>
> wrote:
>>>> On 21/05/13 18:31, Sander Eikelenboom wrote:
>>>>
>>>>> This reminds me that there still is the issue that a HVM
domain without
>>>>> PV drivers doesn''t shutdown properly with the
standard init.d scripts.
>>>>
>>>> BTW, if you want this to get attention, you should probably
either reply
>>>> to that thread, or start a new thread.  Starting a new topic on
an
>>>> unrelated thread is generally frowned upon. :-)
>>
>>> Something needs to tell the guest to shut down. Without pvdrivers,
you''ll
>>> need to do it through ACPI. Do you have ACPI installed, and are you
>>> sending an xl power event?
>>
>>
>> Hi Alex/George,
>>
>> The problem is NOT with a manual sent command by a admin, who has both
>> knowledge and can (directly) respond to warning that is issued
(although
>> it''s a bit harder in the --all case).
>>
>> It''s with automated scripts using "xl shutdown
[-a]", in combination with
>> HVM guests which do not have PV drivers installed.
> Yes I know. We have this problem.
> xl shutdown and friends only send the PV signals (I believe). As your
> guest has no PV drivers, I believe this will do nothing (xl might wait
> for a timeout and destroy)
> You can use the xl power events to send ACPI signals to shutdown guests
> without PV drivers. This also works for guests with PV drivers if they
> have ACPI in.
> So, the way we get round this is not to use the xl commands (well, libxl
> calls in our case) that use the PV interface at all. We simply use
> the ACPI stuff (the power events), which works whether or not the guest
> has PV support. This requires the guest to have ACPI installed for a clean
> shutdown, which is exactly the same requirement for KVM, and hence works
> well.
> We have found (for various reasons) it may be necessary to send the
> power event multiple times until something happens, rather than simply
> wait for a timeout.
> If a shutdown is used, and you send it to early in the boot process,
> then subsequent shutdowns may not work - I reported a bug on that
> (it''s somewhere on the not-for-4.3 list). This in my book another
> reason to avoid xl shutdown and use the power events instead. The
> only disadvantage of this strategy is if you have guests which have
> PV drivers but not ACPI; these tend to be machines built for PV rather
> than HVM, being run on HVM (e.g. AMIs someone has stuck a kernel
> in and hoped for the best).
Well with my little adjustments its working for my situation just fine.
You are right there is still a time window that neither the pv method or the
acpi method will work (during guest boot).
The only thing that can be done seems to be:

- Retry on intervals, under the assumption that the guest is still booting and
hope it will get to a point it can be shutdown by either pv or acpi method, and
timeout and destroy eventually.

The point is, is the current (default) behavior of "xl shutdown" (only
try pv method) sensible ..
That the Xen automated test system run into this problem seems to suggest it is
at least debatable.

So it''s not a major issue for me, just a tiny nit that people can run
into when using only xl and the default scripts for domain management.


--
Sander