Xen devel - Sep 2010 - PV driver domains and S3 sleep

Hello,

The topic is self-explanatory: how to ensure that a PV driver domain correctly 
prepares its PCI devices for S3 sleep?
If I do "pm-suspend" in dom0, and the driver domain has active network
interfaces,
suspend hangs the system. Yes, in case of this particular machine, suspend works
fine when there is no driver domain. 
It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/ in
driver
domain. In the test case, "ifconfig down wlan0" in the driver domain
allows
the suspend to go smoothly. But generally, is it enough ? The kernel device
driver should
prepare the PCI device properly for S3, shouldn''t it ?  
Would it be more proper to [somehow] notify a driver domain _kernel_ that we are
going to S3 (just like dom0 kernel is notified), and let it execute all
necessary actions
(including, but not only, launching of usermode pm-utils scripts), just like
dom0 kernel
does ? Would it work at all, considering that driver domain kernel has no access
to
ACPI tables ? 
Currently, how are these issues taken care of in the mainstream Xen? 

Thanks in advance,
Rafal Wojtczuk


 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/09/2010 12:44, "Rafal Wojtczuk"
<rafal@invisiblethingslab.com> wrote:
> The topic is self-explanatory: how to ensure that a PV driver domain
correctly
> prepares its PCI devices for S3 sleep?
> If I do "pm-suspend" in dom0, and the driver domain has active
network
> interfaces, 
> suspend hangs the system. Yes, in case of this particular machine, suspend
> works
> fine when there is no driver domain.
> It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/
in
> driver 
> domain. In the test case, "ifconfig down wlan0" in the driver
domain allows
> the suspend to go smoothly. But generally, is it enough ? The kernel device
> driver should 
> prepare the PCI device properly for S3, shouldn''t it ?
> Would it be more proper to [somehow] notify a driver domain _kernel_ that
we
> are 
> going to S3 (just like dom0 kernel is notified), and let it execute all
> necessary actions
> (including, but not only, launching of usermode pm-utils scripts), just
like
> dom0 kernel 
> does ? Would it work at all, considering that driver domain kernel has no
> access to 
> ACPI tables ? 
> Currently, how are these issues taken care of in the mainstream Xen?
I don''t think it currently is handled. HVM driver domains (using VT-d
or
equivalent) can be put into virtual S3. We would need an equivalent concept
for PV driver domains. Or for devices to be hot-unplugged from the driver
domain, and re-plugged on resume?

 -- Keir





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/16/10 13:52, Keir Fraser wrote:
> On 16/09/2010 12:44, "Rafal Wojtczuk"
<rafal@invisiblethingslab.com> wrote:
> 
>> The topic is self-explanatory: how to ensure that a PV driver domain
correctly
>> prepares its PCI devices for S3 sleep?
>> If I do "pm-suspend" in dom0, and the driver domain has
active network
>> interfaces, 
>> suspend hangs the system. Yes, in case of this particular machine,
suspend
>> works
>> fine when there is no driver domain.
>> It is possible to manually invoke scripts from
/usr/lib64/pm-utils/sleep.d/ in
>> driver 
>> domain. In the test case, "ifconfig down wlan0" in the driver
domain allows
>> the suspend to go smoothly. But generally, is it enough ? The kernel
device
>> driver should 
>> prepare the PCI device properly for S3, shouldn''t it ?
>> Would it be more proper to [somehow] notify a driver domain _kernel_
that we
>> are 
>> going to S3 (just like dom0 kernel is notified), and let it execute all
>> necessary actions
>> (including, but not only, launching of usermode pm-utils scripts), just
like
>> dom0 kernel 
>> does ? Would it work at all, considering that driver domain kernel has
no
>> access to 
>> ACPI tables ? 
>> Currently, how are these issues taken care of in the mainstream Xen?
> 
> I don''t think it currently is handled. HVM driver domains (using
VT-d or
> equivalent) can be put into virtual S3. We would need an equivalent concept
> for PV driver domains. Or for devices to be hot-unplugged from the driver
> domain, and re-plugged on resume?
> 
But, can you explain how Xen notifies Dom0 when the system enters S3,
and if the same mechanism could be (easily) used to do the same for a
driver PV domain?

Thanks,
joanna.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/16/2010 12:04 PM, Joanna Rutkowska wrote:
> On 09/16/10 13:52, Keir Fraser wrote:
>> On 16/09/2010 12:44, "Rafal Wojtczuk"
<rafal@invisiblethingslab.com> wrote:
>>
>>> The topic is self-explanatory: how to ensure that a PV driver
domain correctly
>>> prepares its PCI devices for S3 sleep?
>>> If I do "pm-suspend" in dom0, and the driver domain has
active network
>>> interfaces, 
>>> suspend hangs the system. Yes, in case of this particular machine,
suspend
>>> works
>>> fine when there is no driver domain.
>>> It is possible to manually invoke scripts from
/usr/lib64/pm-utils/sleep.d/ in
>>> driver 
>>> domain. In the test case, "ifconfig down wlan0" in the
driver domain allows
>>> the suspend to go smoothly. But generally, is it enough ? The
kernel device
>>> driver should 
>>> prepare the PCI device properly for S3, shouldn''t it ?
>>> Would it be more proper to [somehow] notify a driver domain
_kernel_ that we
>>> are 
>>> going to S3 (just like dom0 kernel is notified), and let it execute
all
>>> necessary actions
>>> (including, but not only, launching of usermode pm-utils scripts),
just like
>>> dom0 kernel 
>>> does ? Would it work at all, considering that driver domain kernel
has no
>>> access to 
>>> ACPI tables ? 
>>> Currently, how are these issues taken care of in the mainstream
Xen?
>> I don''t think it currently is handled. HVM driver domains
(using VT-d or
>> equivalent) can be put into virtual S3. We would need an equivalent
concept
>> for PV driver domains. Or for devices to be hot-unplugged from the
driver
>> domain, and re-plugged on resume?
>>
> But, can you explain how Xen notifies Dom0 when the system enters S3,
> and if the same mechanism could be (easily) used to do the same for a
> driver PV domain?
The dom0 kernel initiates S3 itself (possibly in response to a
lid-switch or the like), so it knows it is going into S3.  As part of
that it can do something to notify all the other domains that they in
turn need to do something.

I think the simplest thing to do is just do a regular PV save/restore on
the domains, but without needing to save their pages to disk.  That way
their regular device model suspend/resume will do the right thing to the
hardware devices.  The only change that may be needed is to make sure
that the normal PCI suspend/resume bus calls are done on the Xen
save/restore path.

Or perhaps the domains will need to know whether its a normal
save/restore, vs S3, vs S5.  In that case we could add a xenstore key
which they could watch to know they need to do something.  But it would
need a bit of thought to handshake the whole process.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Sep 16, 2010 at 01:44:24PM +0200, Rafal Wojtczuk
wrote:
> Hello,
> 
> The topic is self-explanatory: how to ensure that a PV driver domain
correctly
> prepares its PCI devices for S3 sleep?
> If I do "pm-suspend" in dom0, and the driver domain has active
network interfaces,
> suspend hangs the system. Yes, in case of this particular machine, suspend
works
> fine when there is no driver domain. 
> It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/
in driver
> domain. In the test case, "ifconfig down wlan0" in the driver
domain allows
> the suspend to go smoothly. But generally, is it enough ? The kernel device
driver should
The pci_disable calls that are made do put the devices in the D3 (or is it D0
state).
However those calls are not made when you do ''ifconfig X down''
(I think). You need
to do ''rmmod ipw2100'' to trigger those calls, or trigger the
drivers'' suspend call
invocation.

The drivers'' suspend call invocation is a twisty maze of dependency
(ie, must first
suspend the driver, and only after that you can suspend the PCI bus).

The S3 suspend on Linux also freezez the user space, cgroups, and whole bunch
of other stuff. But you don''t care about that.

What I think you care about is to put the device in the appropiate D state.
> prepare the PCI device properly for S3, shouldn''t it ?  
> Would it be more proper to [somehow] notify a driver domain _kernel_ that
we are
> going to S3 (just like dom0 kernel is notified), and let it execute all
necessary actions
> (including, but not only, launching of usermode pm-utils scripts), just
like dom0 kernel
> does ? Would it work at all, considering that driver domain kernel has no
access to
> ACPI tables ? 
I think that depends on the PCI device. In laptop world, the wireless card can
do some weird stuff when you press Ctrl-F5 for example - it would invoke some 
ACPI code (well, the Linux kernel AML driver would invoke it), which then
disables/unloads the driver as appropiate. With DomU having no ACPI support, it
means
that the Dom0 would yank the PCI device away from the DomU - which actually
considering
that we are using pciback as the owner, would mean you could pass a request to
the
DomU saying: "Hey, reconfigure now. Device going away." And I think
that might
work today actually.

But back to putting the device in the appropiate D state. You could
pass in DomU a call akin to doing "suspend" > /sys/power/state
which should do the appropiate PCI move.
> Currently, how are these issues taken care of in the mainstream Xen? 
Never explored I fear.
> 
> Thanks in advance,
> Rafal Wojtczuk
> 
> 
>  
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Sep 16, 2010 at 12:52:02PM +0100, Keir Fraser
wrote:
> > The topic is self-explanatory: how to ensure that a PV driver domain
correctly
> > prepares its PCI devices for S3 sleep?
[cut]
> > Currently, how are these issues taken care of in the mainstream Xen?
> I don''t think it currently is handled. HVM driver domains (using
VT-d or
> equivalent) can be put into virtual S3. We would need an equivalent concept
> for PV driver domains. Or for devices to be hot-unplugged from the driver
> domain, and re-plugged on resume?
The idea of using PCI hotplug is nice, however, PCI hotplug does not seem to
work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works, however the 
following hotplug makes the driver domain kernel spit out the following:

Sep 24 09:46:01 localhost kernel: [  113.045927] pcifront pci-0: Rescanning
PCI Frontend Bus 0000:00
Sep 24 09:46:15 localhost kernel: [  126.843990] pcifront pci-0: Rescanning
PCI Frontend Bus 0000:00
Sep 24 09:46:15 localhost kernel: [  126.846217] pcifront pci-0: New device
on 0000:00:01.00 found.
Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0: device
not available (can''t reserve [mem 0xf8000000-0xf8001fff 64bit])

^C
[root@localhost ~]# cat /proc/iomem 
f6000000-f600ffff : 0000:00:00.0
  f6000000-f600ffff : tg3
[root@localhost ~]# lspci
00:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit
Ethernet PCI Express (rev 02)
00:01.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN
[Kedron] Network Connection (rev 61)

Nothing suspicious in xend, Xen and dom0 logs.

The domU and dom0 kernels are the same, 2.6.34.1-10.xenlinux (SUSE patches
for 2.6.34.1).

With old pvops (2.6.31.9-1.pvops0) in domU, the message on the hot-plug is
similar:
Sep 24 09:50:40 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
Bus 0000:00
Sep 24 09:50:51 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
Bus 0000:00
Sep 24 09:50:51 localhost kernel: pcifront pci-0: New device on
0000:00:01.00 found.
Sep 24 09:50:51 localhost kernel: iwlagn 0000:00:01.0: device not available
because of BAR 0 [0xf8000000-0xf8001fff] collisions

Others seem to experience similar problems (e.g.
http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
anyone know the solution ?

Regards,
Rafal Wojtczuk

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Sep 16, 2010 at 05:22:50PM -0700, Jeremy Fitzhardinge
wrote:
> >>> The topic is self-explanatory: how to ensure that a PV driver
domain correctly
> >>> prepares its PCI devices for S3 sleep?
[cut]
> I think the simplest thing to do is just do a regular PV save/restore on
> the domains, but without needing to save their pages to disk.  That way
I suspect suspend/resume of the driver domain will kill established net
backend/frontend
connections ? So we also would have to network-detach all VMs interfaces, and
reattach. It does not look pretty. 

Regards,
Rafal Wojtczuk

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/24/2010 07:30 AM, Rafal Wojtczuk wrote:
> On Thu, Sep 16, 2010 at 05:22:50PM -0700, Jeremy Fitzhardinge wrote:
>>>>> The topic is self-explanatory: how to ensure that a PV
driver domain correctly
>>>>> prepares its PCI devices for S3 sleep?
> [cut]
>> I think the simplest thing to do is just do a regular PV save/restore
on
>> the domains, but without needing to save their pages to disk.  That way
> I suspect suspend/resume of the driver domain will kill established net
backend/frontend
> connections ? So we also would have to network-detach all VMs interfaces,
and
> reattach. It does not look pretty. 
>
Not generally.  The blkfront and netfront drivers don''t really do
anything on a save; they certainly don''t change the xenbus connection
state.  The normal mode of operation for save/restore or migration is
that after resuming the frontends suddenly find their backends are no
longer connected and will quietly attempt to reconnect before doing on,
resulting in just a little IO hiccup.

In this case, the backends will still be there and will remain in a
connected state, so the frontends won''t even notice after resuming.

However, the pcifront driver would implement the suspend method and make
sure the pci bus does its normal suspend operation.

My main concern is that I''m not sure how the handshake with dom0 would
work, so that it knows the suspend is finished - oh I guess the normal
way, waiting for the domain to suspend itself (or timeout mysteriously).

But it might just be better to add a pciback xenstore key to tell
pcifront to do whatever''s required for an ACPI suspend (S3, S5 or
whatever), and have a corresponding pcifront xenstore key to handshake
that the suspend has completed (or failed).  At least that way you could
get some useful diagnostics on failure.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Sep 24, 2010 at 04:24:58PM +0200, Rafal Wojtczuk
wrote:
> On Thu, Sep 16, 2010 at 12:52:02PM +0100, Keir Fraser wrote:
> > > The topic is self-explanatory: how to ensure that a PV driver
domain correctly
> > > prepares its PCI devices for S3 sleep?
> [cut]
> > > Currently, how are these issues taken care of in the mainstream
Xen?
> 
> > I don''t think it currently is handled. HVM driver domains
(using VT-d or
> > equivalent) can be put into virtual S3. We would need an equivalent
concept
> > for PV driver domains. Or for devices to be hot-unplugged from the
driver
> > domain, and re-plugged on resume?
> 
> The idea of using PCI hotplug is nice, however, PCI hotplug does not seem
to
> work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works, however
the
> following hotplug makes the driver domain kernel spit out the following:
> 
> Sep 24 09:46:01 localhost kernel: [  113.045927] pcifront pci-0: Rescanning
> PCI Frontend Bus 0000:00
> Sep 24 09:46:15 localhost kernel: [  126.843990] pcifront pci-0: Rescanning
> PCI Frontend Bus 0000:00
> Sep 24 09:46:15 localhost kernel: [  126.846217] pcifront pci-0: New device
> on 0000:00:01.00 found.
> Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0:
device
> not available (can''t reserve [mem 0xf8000000-0xf8001fff 64bit])
> 
> ^C
> [root@localhost ~]# cat /proc/iomem 
> f6000000-f600ffff : 0000:00:00.0
>   f6000000-f600ffff : tg3
> [root@localhost ~]# lspci
> 00:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit
> Ethernet PCI Express (rev 02)
> 00:01.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN
> [Kedron] Network Connection (rev 61)
> 
> Nothing suspicious in xend, Xen and dom0 logs.
> 
> The domU and dom0 kernels are the same, 2.6.34.1-10.xenlinux (SUSE patches
> for 2.6.34.1).
> 
> With old pvops (2.6.31.9-1.pvops0) in domU, the message on the hot-plug is
similar:
> Sep 24 09:50:40 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
> Bus 0000:00
> Sep 24 09:50:51 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
> Bus 0000:00
> Sep 24 09:50:51 localhost kernel: pcifront pci-0: New device on
> 0000:00:01.00 found.
> Sep 24 09:50:51 localhost kernel: iwlagn 0000:00:01.0: device not available
> because of BAR 0 [0xf8000000-0xf8001fff] collisions
> 
> Others seem to experience similar problems (e.g.
> http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
> anyone know the solution ?
I had an off-mailing list conversation with that fellow and I spun out
a bunch of patches to fix his issue.

You need these patches:
Konrad Rzeszutek Wilk (3):
      xen-pcifront: Enforce scanning of device functions on initial execution.
      xen-pcifront: Claim PCI resources before going live.
      xen-pcifront: Don''t race with udev when discovering new devices.

I think they are in Jeremy''s upstream tree.. ah, right you guys
aren''t using
Jeremy''s tree.

Get them from: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

pv/pcifront-2.6.34

you also might want to update your pciback driver too
(pv/pciback-2.6.32)
> 
> Regards,
> Rafal Wojtczuk
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 27, 2010 at 01:07:05PM -0400, Konrad Rzeszutek Wilk wrote:
> > The idea of using PCI hotplug is nice, however, PCI hotplug does not
seem to
> > work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works,
however the
> > following hotplug makes the driver domain kernel spit out the
following:
[cut]
> > Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0:
device
> > not available (can''t reserve [mem 0xf8000000-0xf8001fff
64bit])
[cut]
> > Others seem to experience similar problems (e.g.
> > http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
> > anyone know the solution ?
> 
> I had an off-mailing list conversation with that fellow and I spun out
> a bunch of patches to fix his issue.
> 
> You need these patches:
> Konrad Rzeszutek Wilk (3):
>       xen-pcifront: Enforce scanning of device functions on initial
execution.
>       xen-pcifront: Claim PCI resources before going live.
>       xen-pcifront: Don''t race with udev when discovering new
devices.
> 
> I think they are in Jeremy''s upstream tree.. ah, right you guys
aren''t using
> Jeremy''s tree.
> 
> Get them from: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> 
> pv/pcifront-2.6.34
Indeed these patches help, thank you.
There is one more problem with the linux-2.6.18-xen.hg pcifront (that affect
derived code, e.g. OpenSUSE kernel, too). unbind_from_irqhandler() is
mistakenly passed evtchn, instead of irq. Compare line 68 of
http://xenbits.xensource.com/linux-2.6.18-xen.hg?file/a66a7c64b1d0/drivers/xen/pcifront/xenbus.c
with pvops equivalent
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/pci/xen-pcifront.c;h=10868aeae818d69980b8519f8a77b38d6ab58a4c;hb=HEAD#l758

The following patch helps.
Regards,
Rafal Wojtczuk


unbind_from_irqhandler takes irq, not evtchn, as its first argument.

Signed-off-by: Rafal Wojtczuk <rafal@invisiblethingslab.com>
--- linux-2.6.34.1/drivers/xen/pcifront/xenbus.c.orig   2010-09-29
16:47:39.961674359 +0200
+++ linux-2.6.34.1/drivers/xen/pcifront/xenbus.c        2010-09-29
16:47:49.458675391 +0200
@@ -61,7 +61,7 @@ static void free_pdev(struct pcifront_de

        /*For PCIE_AER error handling job*/
        flush_scheduled_work();
-       unbind_from_irqhandler(pdev->evtchn, pdev);
+       unbind_from_irqhandler(irq_from_evtchn(pdev->evtchn), pdev);

        if (pdev->evtchn != INVALID_EVTCHN)
                xenbus_free_evtchn(pdev->xdev, pdev->evtchn);


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 01.10.10 at 16:24, Rafal Wojtczuk
<rafal@invisiblethingslab.com> wrote:
> There is one more problem with the linux-2.6.18-xen.hg pcifront (that
affect
> derived code, e.g. OpenSUSE kernel, too). unbind_from_irqhandler() is
> mistakenly passed evtchn, instead of irq. Compare line 68 of
>
http://xenbits.xensource.com/linux-2.6.18-xen.hg?file/a66a7c64b1d0/drivers/xen
> /pcifront/xenbus.c
> with pvops equivalent
>
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/pc
> i/xen-pcifront.c;h=10868aeae818d69980b8519f8a77b38d6ab58a4c;hb=HEAD#l758
> 
> The following patch helps.
Except there is no irq_from_evtchn() in the original tree. I''ll post a
better, more complete patch later.

Jan
> Regards,
> Rafal Wojtczuk
> 
> 
> unbind_from_irqhandler takes irq, not evtchn, as its first argument.
> 
> Signed-off-by: Rafal Wojtczuk <rafal@invisiblethingslab.com>
> --- linux-2.6.34.1/drivers/xen/pcifront/xenbus.c.orig   2010-09-29 
> 16:47:39.961674359 +0200
> +++ linux-2.6.34.1/drivers/xen/pcifront/xenbus.c        2010-09-29 
> 16:47:49.458675391 +0200
> @@ -61,7 +61,7 @@ static void free_pdev(struct pcifront_de
> 
>         /*For PCIE_AER error handling job*/
>         flush_scheduled_work();
> -       unbind_from_irqhandler(pdev->evtchn, pdev);
> +       unbind_from_irqhandler(irq_from_evtchn(pdev->evtchn), pdev);
> 
>         if (pdev->evtchn != INVALID_EVTCHN)
>                 xenbus_free_evtchn(pdev->xdev, pdev->evtchn);


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel