After deploying stubdomains in a production environment with xen 3.4.1 and the official 2.6.18-xen kernel I''ve had a lot of crashes. Each server hosts several PV and HVM domUs. Whenever the storage backend gets slow (RAID rebuild, domU with heavy disk IO, etc) the stubdomains begin dying off. I have two questions: 1. Has anyone else seen this issue? 2. fs-backend is still running so is there any way to attach a new stubdom to the existing (orphaned) HVM domU? Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2009-Sep-29 18:38 UTC
Re: [Xen-devel] slow storage backend kills stubdomains
On Mon, Sep 28, 2009 at 04:43:17PM -0400, Keith Coleman wrote:> After deploying stubdomains in a production environment with xen 3.4.1 > and the official 2.6.18-xen kernel I''ve had a lot of crashes. Each > server hosts several PV and HVM domUs. Whenever the storage backend > gets slow (RAID rebuild, domU with heavy disk IO, etc) the stubdomains > begin dying off. I have two questions: > > 1. Has anyone else seen this issue? > 2. fs-backend is still running so is there any way to attach a new > stubdom to the existing (orphaned) HVM domU? >Did you apply this patch? http://lists.xensource.com/archives/html/xen-devel/2009-08/msg00335.html -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2009-Sep-29 22:45 UTC
Re: [Xen-devel] slow storage backend kills stubdomains
On Tue, Sep 29, 2009 at 2:38 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Mon, Sep 28, 2009 at 04:43:17PM -0400, Keith Coleman wrote: >> After deploying stubdomains in a production environment with xen 3.4.1 >> and the official 2.6.18-xen kernel I''ve had a lot of crashes. Each >> server hosts several PV and HVM domUs. Whenever the storage backend >> gets slow (RAID rebuild, domU with heavy disk IO, etc) the stubdomains >> begin dying off. I have two questions: >> >> 1. Has anyone else seen this issue? >> 2. fs-backend is still running so is there any way to attach a new >> stubdom to the existing (orphaned) HVM domU? >> > > Did you apply this patch? > > http://lists.xensource.com/archives/html/xen-devel/2009-08/msg00335.html > > -- Pasi > >This is a different. The patch that you suggested was already in place since I contributed to the discovery and resolution of that problem. This particular issue seems to be related to high IO latency. Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2009-Oct-01 19:58 UTC
Re: [Xen-devel] slow storage backend kills stubdomains
On Tue, Sep 29, 2009 at 6:45 PM, Keith Coleman <list.keith@scaltro.com> wrote:> On Tue, Sep 29, 2009 at 2:38 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote: >> On Mon, Sep 28, 2009 at 04:43:17PM -0400, Keith Coleman wrote: >>> After deploying stubdomains in a production environment with xen 3.4.1 >>> and the official 2.6.18-xen kernel I''ve had a lot of crashes. Each >>> server hosts several PV and HVM domUs. Whenever the storage backend >>> gets slow (RAID rebuild, domU with heavy disk IO, etc) the stubdomains >>> begin dying off. I have two questions: >>> >>> 1. Has anyone else seen this issue? >>> 2. fs-backend is still running so is there any way to attach a new >>> stubdom to the existing (orphaned) HVM domU? >>> >> >> Did you apply this patch? >> >> http://lists.xensource.com/archives/html/xen-devel/2009-08/msg00335.html >> >> -- Pasi >> >> > > This is a different. The patch that you suggested was already in place > since I contributed to the discovery and resolution of that problem. > This particular issue seems to be related to high IO latency. > > > Keith Coleman >This is likely another instance of the same issue: http://lists.xensource.com/archives/html/xen-users/2009-10/msg00009.html Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-02 11:42 UTC
[Xen-devel] Re: slow storage backend kills stubdomains
On Mon, 28 Sep 2009, Keith Coleman wrote:> After deploying stubdomains in a production environment with xen 3.4.1 > and the official 2.6.18-xen kernel I''ve had a lot of crashes. Each > server hosts several PV and HVM domUs. Whenever the storage backend > gets slow (RAID rebuild, domU with heavy disk IO, etc) the stubdomains > begin dying off. I have two questions: > > 1. Has anyone else seen this issue? > 2. fs-backend is still running so is there any way to attach a new > stubdom to the existing (orphaned) HVM domU? >Could you please let me know how can I reproduce this problem in the simplest possible way (for example with just 1 linux HVM guest)? Also some logs of the crash would be useful. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-05 13:53 UTC
[Xen-devel] Re: slow storage backend kills stubdomains
On Mon, 28 Sep 2009, Keith Coleman wrote:> After deploying stubdomains in a production environment with xen 3.4.1 > and the official 2.6.18-xen kernel I''ve had a lot of crashes. Each > server hosts several PV and HVM domUs. Whenever the storage backend > gets slow (RAID rebuild, domU with heavy disk IO, etc) the stubdomains > begin dying off. I have two questions: > > 1. Has anyone else seen this issue? > 2. fs-backend is still running so is there any way to attach a new > stubdom to the existing (orphaned) HVM domU? >There is a good chance that the last patch series I sent to the list fixes this issues. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel