I was investigating a bug report[1] about newer kernels (>3.1) not booting as
HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
it lead me at least close and with some crash dump data I think I figured the
problem.
commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
Author: Olaf Hering <olaf@aepfle.de>
Date: Thu Sep 22 16:14:49 2011 +0200
xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
kernel
This change introduced a xs_reset_watches() call. The problem seems to be that
there is at least some version of Xen (I was able to reproduce with a 3.4.3
version which I admit to deliberately not having updated) for which xenstore
will not return any reply.
At least the backtraces in crash showed that xs_init had been calling
xs_reset_watches() and that was happily idling in read_reply(). Effectively
nothing was going on and the boot just hung.
By just not doing that xs_reset_watches() call, I was able to boot under the
same host. And for what it is worth there has not been an issue with Xen 4.1.1
and a 3.0 dom0 kernel. Just this "older" release is trouble.
Now the big question is, should this never happen and the host needs urgent
updating. Or, should xs_talkv() set up a time limit and assume failure when not
receiving a message after that? I could imagine the latter might lead at least
to a more helpful "there is something wrong here, dude" than just
hanging around
without any response. ;)
-Stefan
On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:> I was investigating a bug report[1] about newer kernels (>3.1) not booting as > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but > it lead me at least close and with some crash dump data I think I figured the > problem.Stefan, thanks for finding this. Olaf, what are your thoughts? Should I prep a patch to revert the patch below and then we can work on 3.3 and rethink this in 3.3? The clock is ticking for 3.2 and there is not much runway to fix stuff.> > commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05 > Author: Olaf Hering <olaf@aepfle.de> > Date: Thu Sep 22 16:14:49 2011 +0200 > > xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old > kernel > > This change introduced a xs_reset_watches() call. The problem seems to be that > there is at least some version of Xen (I was able to reproduce with a 3.4.3 > version which I admit to deliberately not having updated) for which xenstore > will not return any reply.And oxenstore too, but Ian prepped a patch for this. Perhaps that is what Amazon is running.> > At least the backtraces in crash showed that xs_init had been calling > xs_reset_watches() and that was happily idling in read_reply(). Effectively > nothing was going on and the boot just hung.So at least we should have a timeout read_reply. But I don''t see anything in the code that we could immediately use.> By just not doing that xs_reset_watches() call, I was able to boot under the > same host. And for what it is worth there has not been an issue with Xen 4.1.1 > and a 3.0 dom0 kernel. Just this "older" release is trouble. > > Now the big question is, should this never happen and the host needs urgent > updating. Or, should xs_talkv() set up a time limit and assume failure when not > receiving a message after that? I could imagine the latter might lead at least > to a more helpful "there is something wrong here, dude" than just hanging around > without any response. ;) > > -Stefan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
On 15.12.2011 20:39, Konrad Rzeszutek Wilk wrote:> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote: >> I was investigating a bug report[1] about newer kernels (>3.1) not booting as >> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but >> it lead me at least close and with some crash dump data I think I figured the >> problem. > > Stefan, thanks for finding this. >I realize I wanted to add the reference to our bug report but completely forgot to do so. So just for completeness: http://bugs.launchpad.net/bugs/901305> Olaf, what are your thoughts? Should I prep a patch to revert the patch > below and then we can work on 3.3 and rethink this in 3.3? The clock is > ticking for 3.2 and there is not much runway to fix stuff. > >> >> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05 >> Author: Olaf Hering <olaf@aepfle.de> >> Date: Thu Sep 22 16:14:49 2011 +0200 >> >> xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old >> kernel >> >> This change introduced a xs_reset_watches() call. The problem seems to be that >> there is at least some version of Xen (I was able to reproduce with a 3.4.3 >> version which I admit to deliberately not having updated) for which xenstore >> will not return any reply. > > And oxenstore too, but Ian prepped a patch for this. Perhaps that is > what Amazon is running. >> >> At least the backtraces in crash showed that xs_init had been calling >> xs_reset_watches() and that was happily idling in read_reply(). Effectively >> nothing was going on and the boot just hung. > > So at least we should have a timeout read_reply. But I don''t see > anything in the code that we could immediately use. > >> By just not doing that xs_reset_watches() call, I was able to boot under the >> same host. And for what it is worth there has not been an issue with Xen 4.1.1 >> and a 3.0 dom0 kernel. Just this "older" release is trouble. >> >> Now the big question is, should this never happen and the host needs urgent >> updating. Or, should xs_talkv() set up a time limit and assume failure when not >> receiving a message after that? I could imagine the latter might lead at least >> to a more helpful "there is something wrong here, dude" than just hanging around >> without any response. ;) >> >> -Stefan >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel
On Thu, 2011-12-15 at 19:20 +0000, Stefan Bader wrote:> I was investigating a bug report[1] about newer kernels (>3.1) not booting as > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but > it lead me at least close and with some crash dump data I think I figured the > problem. > > commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05 > Author: Olaf Hering <olaf@aepfle.de> > Date: Thu Sep 22 16:14:49 2011 +0200 > > xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old > kernel > > This change introduced a xs_reset_watches() call. The problem seems to be that > there is at least some version of Xen (I was able to reproduce with a 3.4.3 > version which I admit to deliberately not having updated) for which xenstore > will not return any reply. > > At least the backtraces in crash showed that xs_init had been calling > xs_reset_watches() and that was happily idling in read_reply(). Effectively > nothing was going on and the boot just hung. > By just not doing that xs_reset_watches() call, I was able to boot under the > same host. And for what it is worth there has not been an issue with Xen 4.1.1 > and a 3.0 dom0 kernel. Just this "older" release is trouble.I sent a patch to fix exactly this issue in oxenstored (the ocaml xenstore) just this week. Is there any chance that you are running C xenstored with Xen 4.1.1 and oxenstored with Xen 3.4.3?> Now the big question is, should this never happen and the host needs urgent > updating. Or, should xs_talkv() set up a time limit and assume failure when not > receiving a message after that? I could imagine the latter might lead at least > to a more helpful "there is something wrong here, dude" than just hanging around > without any response. ;) > > -Stefan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
On 15.12.2011 21:53, Ian Campbell wrote:> On Thu, 2011-12-15 at 19:20 +0000, Stefan Bader wrote: >> I was investigating a bug report[1] about newer kernels (>3.1) not booting as >> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but >> it lead me at least close and with some crash dump data I think I figured the >> problem. >> >> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05 >> Author: Olaf Hering <olaf@aepfle.de> >> Date: Thu Sep 22 16:14:49 2011 +0200 >> >> xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old >> kernel >> >> This change introduced a xs_reset_watches() call. The problem seems to be that >> there is at least some version of Xen (I was able to reproduce with a 3.4.3 >> version which I admit to deliberately not having updated) for which xenstore >> will not return any reply. >> >> At least the backtraces in crash showed that xs_init had been calling >> xs_reset_watches() and that was happily idling in read_reply(). Effectively >> nothing was going on and the boot just hung. >> By just not doing that xs_reset_watches() call, I was able to boot under the >> same host. And for what it is worth there has not been an issue with Xen 4.1.1 >> and a 3.0 dom0 kernel. Just this "older" release is trouble. > > I sent a patch to fix exactly this issue in oxenstored (the ocaml > xenstore) just this week. Is there any chance that you are running C > xenstored with Xen 4.1.1 and oxenstored with Xen 3.4.3?Thanks for the pointer, I missed that thread. Now dumb question, would oxenstored be named that way? Or iow, how do I quickly find out what is running? The binary running in 3.4.3 is xenstored which is a linked executable (same in 4.1.1). But I guess, whatever version is running, any oxenstored would not have the bugfix because things take longer to reach any packaged versions. I rather would suspect that in 4.1.1, the reset watches message probably is just known and thus avoiding the problem. Unfortunately it is near impossible to tell for sure what exactly EC2 is running. The major point here probably is that when the upstream kernels are calling that message and there are versions of xenstored in production that will just ignore it while the kernel blocks waiting, this is a painful path. Production systems tend to update slowly and the symptoms are not that obvious. Having a timeout maybe could be useful not only for this case, but clearly it is nothing that should be rushed. So reverting the patch introducing that call (at least in the distro kernel) may be the best thing to do (knowing that this will be bought by loosing the fix for kexec boots fo crash kernels). -Stefan>> Now the big question is, should this never happen and the host needs urgent >> updating. Or, should xs_talkv() set up a time limit and assume failure when not >> receiving a message after that? I could imagine the latter might lead at least >> to a more helpful "there is something wrong here, dude" than just hanging around >> without any response. ;) >> >> -Stefan >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
On Fri, 2011-12-16 at 09:18 +0000, Stefan Bader wrote:> On 15.12.2011 21:53, Ian Campbell wrote: > > On Thu, 2011-12-15 at 19:20 +0000, Stefan Bader wrote: > >> I was investigating a bug report[1] about newer kernels (>3.1) not booting as > >> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but > >> it lead me at least close and with some crash dump data I think I figured the > >> problem. > >> > >> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05 > >> Author: Olaf Hering <olaf@aepfle.de> > >> Date: Thu Sep 22 16:14:49 2011 +0200 > >> > >> xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old > >> kernel > >> > >> This change introduced a xs_reset_watches() call. The problem seems to be that > >> there is at least some version of Xen (I was able to reproduce with a 3.4.3 > >> version which I admit to deliberately not having updated) for which xenstore > >> will not return any reply. > >> > >> At least the backtraces in crash showed that xs_init had been calling > >> xs_reset_watches() and that was happily idling in read_reply(). Effectively > >> nothing was going on and the boot just hung. > >> By just not doing that xs_reset_watches() call, I was able to boot under the > >> same host. And for what it is worth there has not been an issue with Xen 4.1.1 > >> and a 3.0 dom0 kernel. Just this "older" release is trouble. > > > > I sent a patch to fix exactly this issue in oxenstored (the ocaml > > xenstore) just this week. Is there any chance that you are running C > > xenstored with Xen 4.1.1 and oxenstored with Xen 3.4.3? > > Thanks for the pointer, I missed that thread. Now dumb question, would > oxenstored be named that way? Or iow, how do I quickly find out what is running?The process name will be oxenstored instead of xenstored.> The binary running in 3.4.3 is xenstored which is a linked executable (same in > 4.1.1).The sounds like you are running the C xenstored in both cases so this is a red-herring.> But I guess, whatever version is running, any oxenstored would not have the > bugfix because things take longer to reach any packaged versions.Correct.> I rather would suspect that in 4.1.1, the reset watches message probably is just > known and thus avoiding the problem. Unfortunately it is near impossible to tell > for sure what exactly EC2 is running. > > The major point here probably is that when the upstream kernels are calling that > message and there are versions of xenstored in production that will just ignore > it while the kernel blocks waiting, this is a painful path. Production systems > tend to update slowly and the symptoms are not that obvious.Yes. It is unfortunate that xenstored in the field has this bug but it does mean that the approach taken here with this new message cannot work. FWIW there weren''t all that many changes to C xenstored between 3.4.x and current unstable so it wouldn''t be hard to identify where the fix is, but that doesn''t really help people stuck with an older xenstored.> Having a timeout > maybe could be useful not only for this case, but clearly it is nothing that > should be rushed.A timeout may not help, it depends what the daemon does after the invalid message. It looks as if oxenstored just throws it away and will process the next message fine so that is OK but I didn''t check the C version. I think (or hope!) that Olaf tested with the most recent version of C xenstored which did not support this new message and so I presume that it correct returns an error for an unknown message but that doesn''t help us with the older C xenstored which you have. In any case someone really needs to check both the ocaml and the version''s behaviour.> So reverting the patch introducing that call (at least in the distro kernel) may > be the best thing to do (knowing that this will be bought by loosing the fix for > kexec boots fo crash kernels).Agreed. We should revert the kernel change for now and revisit it. One potential solution, depending on the actual behaviour of the daemons, would be to follow the potentially unsupported command with an innocuous well established one and use the ID field to identify which we get a response to. Does the target kernel know that it has been kexec''d? Perhaps we should only reset xenstore watches if we are booting after a kexec. Worst case the kexec tool can add a command line argument to trigger this. Doing it this way means there is no possibility of regressions for normal boot and kexec wasn''t supported on older xenstored anyway. Ian.> > -Stefan > > >> Now the big question is, should this never happen and the host needs urgent > >> updating. Or, should xs_talkv() set up a time limit and assume failure when not > >> receiving a message after that? I could imagine the latter might lead at least > >> to a more helpful "there is something wrong here, dude" than just hanging around > >> without any response. ;) > >> > >> -Stefan > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel >
On Thu, Dec 15, Konrad Rzeszutek Wilk wrote:> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote: > > I was investigating a bug report[1] about newer kernels (>3.1) not booting as > > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but > > it lead me at least close and with some crash dump data I think I figured the > > problem. > > Stefan, thanks for finding this. > > Olaf, what are your thoughts? Should I prep a patch to revert the patch > below and then we can work on 3.3 and rethink this in 3.3? The clock is > ticking for 3.2 and there is not much runway to fix stuff.Sometimes guest changes expose bugs in the host. Its my understanding that hosts should be kept uptodate so that it can serve both old and new guests well. In my testing with Xen4 based hosts their xenstored did properly ignore the new command. I proposed several ways to get rid of existing watches, but finally we came to the conclusion that a new xenstored command would be the cleanest way. Wether adding a timeout is a good idea has to be decided. I can imagine that a busy host may take some time to respond to guest commands. Perhaps we should figure out what exactly EC2 is using as host and why it only breaks with upstream kernels. So far I havent received reports for SLES11 guests. SP1 got an update recently, so their HVM guests would have seen the hang as well. The not yet released SP2 sends XS_RESET_WATCHES as well since quite some time. Olaf
On Fri, Dec 16, 2011 at 12:33:00PM +0100, Olaf Hering wrote:> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote: > > > On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote: > > > I was investigating a bug report[1] about newer kernels (>3.1) not booting as > > > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but > > > it lead me at least close and with some crash dump data I think I figured the > > > problem. > > > > Stefan, thanks for finding this. > > > > Olaf, what are your thoughts? Should I prep a patch to revert the patch > > below and then we can work on 3.3 and rethink this in 3.3? The clock is > > ticking for 3.2 and there is not much runway to fix stuff. > > Sometimes guest changes expose bugs in the host. Its my understanding > that hosts should be kept uptodate so that it can serve both old and new > guests well. > > In my testing with Xen4 based hosts their xenstored did properly ignore > the new command. > > I proposed several ways to get rid of existing watches, but finally we > came to the conclusion that a new xenstored command would be the > cleanest way. > > Wether adding a timeout is a good idea has to be decided. I can imagine > that a busy host may take some time to respond to guest commands. > > > Perhaps we should figure out what exactly EC2 is using as host and why > it only breaks with upstream kernels. So far I havent received reportsGood point. Stefan were you able to provide to Scott a kernel without the git commit mentioned to see if that fixed the issue? CC-ing here Vincent in hopes of getting some hints..> for SLES11 guests. SP1 got an update recently, so their HVM guests would > have seen the hang as well. The not yet released SP2 sends > XS_RESET_WATCHES as well since quite some time. > > > Olaf
On Fri, Dec 16, Ian Campbell wrote:> I think (or hope!) that Olaf tested with the most recent version of C > xenstored which did not support this new message and so I presume that > it correct returns an error for an unknown message but that doesn''t help > us with the older C xenstored which you have.I just checked a SLES10SP3 host (xen 3.2.3) with a SLES11SP2 hvm guest and it booted fine. So I suspect EC2 is broken in this case. Olaf
by backing out ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05, 3.2.0-rc5 boots just fine. -Alessandro- Here i am, A young man, A crashing computer program, Here is a pen, write out my name... (from: The Servant - Orchestra) On Fri, Dec 16, 2011 at 09:01, Olaf Hering <olaf@aepfle.de> wrote:> On Fri, Dec 16, Ian Campbell wrote: > >> I think (or hope!) that Olaf tested with the most recent version of C >> xenstored which did not support this new message and so I presume that >> it correct returns an error for an unknown message but that doesn''t help >> us with the older C xenstored which you have. > > I just checked a SLES10SP3 host (xen 3.2.3) with a SLES11SP2 hvm guest > and it booted fine. > > So I suspect EC2 is broken in this case. > > Olaf > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
On Fri, 2011-12-16 at 11:33 +0000, Olaf Hering wrote:> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote: > > > On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote: > > > I was investigating a bug report[1] about newer kernels (>3.1) not booting as > > > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but > > > it lead me at least close and with some crash dump data I think I figured the > > > problem. > > > > Stefan, thanks for finding this. > > > > Olaf, what are your thoughts? Should I prep a patch to revert the patch > > below and then we can work on 3.3 and rethink this in 3.3? The clock is > > ticking for 3.2 and there is not much runway to fix stuff. > > Sometimes guest changes expose bugs in the host. Its my understanding > that hosts should be kept uptodate so that it can serve both old and new > guests well.In an ideal world yes but we need to balance this against breaking stuff which is still widely used. It seems like in this case we may have gotten the balance wrong because people are reporting bugs. What''s wrong with only doing this reset if we know we are kexec''d? If that can''t be automatically detected then e.g. using an explicit reset_watches command line option. You could even make a tenuous argument for hanging this off reset_devices? [...]> Perhaps we should figure out what exactly EC2 is using as host and why > it only breaks with upstream kernels.and in the meantime we leave upstream (and any distros which picks up a new enough kernel) on EC2? I think at this stage in the rc cycle we''d be better off reverting and trying again for 3.3.> So far I havent received reports > for SLES11 guests. SP1 got an update recently, so their HVM guests would > have seen the hang as well. The not yet released SP2 sends > XS_RESET_WATCHES as well since quite some time.This thread contains 3 reports of actual regressions. Stefan explicitly reported an EC2 bug and a reproduction on 3.4.3 and Alessandro by implication has reported breakage in his environment. Ian.> > > Olaf > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
On Tue, Dec 20, Ian Campbell wrote:> What''s wrong with only doing this reset if we know we are kexec''d? If > that can''t be automatically detected then e.g. using an explicit > reset_watches command line option. You could even make a tenuous > argument for hanging this off reset_devices?The kexec kernel does not know that it was loaded via kexec. We could make the reset_devices option mandatory for kexec in PVonHVM guests, so the change to drivers/xen/xenbus/xenbus_xs.c would be very small, like "if (hvm && reset_devices) xs_reset_watches();"> > Perhaps we should figure out what exactly EC2 is using as host and why > > it only breaks with upstream kernels. > > and in the meantime we leave upstream (and any distros which picks up a > new enough kernel) on EC2? I think at this stage in the rc cycle we''d be > better off reverting and trying again for 3.3.If EC2 is unable to fix it in time (or provide info what exactly they use), I''m ok with reverting/disabling the call to xs_reset_watches(). I can continue to work on this next year. Olaf
On Tue, Dec 20, 2011 at 02:15:33PM +0100, Olaf Hering wrote:> On Tue, Dec 20, Ian Campbell wrote: > > > What''s wrong with only doing this reset if we know we are kexec''d? If > > that can''t be automatically detected then e.g. using an explicit > > reset_watches command line option. You could even make a tenuous > > argument for hanging this off reset_devices? > > The kexec kernel does not know that it was loaded via kexec. > We could make the reset_devices option mandatory for kexec in PVonHVM > guests, so the change to drivers/xen/xenbus/xenbus_xs.c would be very > small, like "if (hvm && reset_devices) xs_reset_watches();"<nods> OK that would be one way. Granted if one tried to kexec under Amazon EC2 an PVonHVM domain we would hit this bug again. But then I don''t think kexecing without this patch works, so that scenario is probably moot.> > > > Perhaps we should figure out what exactly EC2 is using as host and why > > > it only breaks with upstream kernels. > > > > and in the meantime we leave upstream (and any distros which picks up a > > new enough kernel) on EC2? I think at this stage in the rc cycle we''d be > > better off reverting and trying again for 3.3. > > If EC2 is unable to fix it in time (or provide info what exactly they > use), I''m ok with reverting/disabling the call to xs_reset_watches().By my reckoning the 3.2 is going to come out Dec 29th (60 days after 3.1 was released) or it might slip. With folks buying presents online (and potentially using Amazon) they [Amazon] is not going to fix anything - they are in "must work now to sell stuff mode" - which means fix only those $1M bugs. With the craze of purchases stopping around January I think they could start addressing this sometime in Janurary - which would be past the 3.2 release date. Sorry Olaf, have to revert that commit.> I can continue to work on this next year.Ok. I need to serioulsy get a free Amazon EC2 instance to run nightly tests. This is the third breakage this year.
Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of
doom"):> Sorry Olaf, have to revert that commit.
I agree. When we introduced it we weren''t aware that most existing
implementations of xenstored simply ignore unknown commands rather
than replying with an error. If we had known this we would not have
approved Olaf''s patch.
That they ignore unknown commands is of course a bug but expecting
everyone to update is no good. Really the best approach would be some
kind of discovery mechanism.
Maybe we should have a special path @xenstore/fail_unknown_commands
which you could read, or something. But this time we should try it
against old implementations.
Ian.
On Tue, 2011-12-20 at 17:29 +0000, Ian Jackson wrote:> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"): > > Sorry Olaf, have to revert that commit. > > I agree. When we introduced it we weren''t aware that most existing > implementations of xenstored simply ignore unknown commands rather > than replying with an error. If we had known this we would not have > approved Olaf''s patch. > > That they ignore unknown commands is of course a bug but expecting > everyone to update is no good. Really the best approach would be some > kind of discovery mechanism. > > Maybe we should have a special path @xenstore/fail_unknown_commands > which you could read, or something. But this time we should try it > against old implementations.I was sure I''d seen some precedent (and therefore an existing path) for this sort of thing at some point but I can''t for the life of me find it. The closest I could find is the /local/domain/<N>/control/platform-feature-multiprocessor-suspend node which we write statically for every domain. I suppose putting it under /local/domain/<N> is consistent with restricting the domain to mostly it''s own home area. Ian.
On 16.12.2011 12:33, Olaf Hering wrote:> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote: > >> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote: >>> I was investigating a bug report[1] about newer kernels (>3.1) not booting as >>> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but >>> it lead me at least close and with some crash dump data I think I figured the >>> problem. >> >> Stefan, thanks for finding this. >> >> Olaf, what are your thoughts? Should I prep a patch to revert the patch >> below and then we can work on 3.3 and rethink this in 3.3? The clock is >> ticking for 3.2 and there is not much runway to fix stuff. > > Sometimes guest changes expose bugs in the host. Its my understanding > that hosts should be kept uptodate so that it can serve both old and new > guests well.That would be the ideal world. Unfortunately, in reality, hosts stick to a particular version and maybe get updated with what is provided as stable or security updates.> In my testing with Xen4 based hosts their xenstored did properly ignore > the new command.I can only take what evidence I see publicly on running EC2 instances. The closest we can get there is something Redhat or CentOS based with a variation of 3.x based hypervisors (my test box is keeping CentOS 5.6 and Xen 3.4.3 to verify bugs). Checking against our Xen host reintroduced with 11.10, there is no problem either.> I proposed several ways to get rid of existing watches, but finally we > came to the conclusion that a new xenstored command would be the > cleanest way. > > Wether adding a timeout is a good idea has to be decided. I can imagine > that a busy host may take some time to respond to guest commands. > > > Perhaps we should figure out what exactly EC2 is using as host and why > it only breaks with upstream kernels. So far I havent received reports > for SLES11 guests. SP1 got an update recently, so their HVM guests would > have seen the hang as well. The not yet released SP2 sends > XS_RESET_WATCHES as well since quite some time.As said before, it is hard to say exactly. We can even see variations of Xen 3.x (I can at least remember having seen three versions ranging from 3.0 to 3.4.3). So to a certain degree there are updates going on. But it is impossible to say what and when. But definitely the problem was still there with 3.4.3 (I stopped updating the test box because I was rather glad to have a state that exhibits most of the weirdness that happens in the cloud). -Stefan> > > Olaf
On 16.12.2011 16:25, Konrad Rzeszutek Wilk wrote:> On Fri, Dec 16, 2011 at 12:33:00PM +0100, Olaf Hering wrote: >> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote: >> >>> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote: >>>> I was investigating a bug report[1] about newer kernels (>3.1) not booting as >>>> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but >>>> it lead me at least close and with some crash dump data I think I figured the >>>> problem. >>> >>> Stefan, thanks for finding this. >>> >>> Olaf, what are your thoughts? Should I prep a patch to revert the patch >>> below and then we can work on 3.3 and rethink this in 3.3? The clock is >>> ticking for 3.2 and there is not much runway to fix stuff. >> >> Sometimes guest changes expose bugs in the host. Its my understanding >> that hosts should be kept uptodate so that it can serve both old and new >> guests well. >> >> In my testing with Xen4 based hosts their xenstored did properly ignore >> the new command. >> >> I proposed several ways to get rid of existing watches, but finally we >> came to the conclusion that a new xenstored command would be the >> cleanest way. >> >> Wether adding a timeout is a good idea has to be decided. I can imagine >> that a busy host may take some time to respond to guest commands. >> >> >> Perhaps we should figure out what exactly EC2 is using as host and why >> it only breaks with upstream kernels. So far I havent received reports > > Good point. Stefan were you able to provide to Scott a kernel without the > git commit mentioned to see if that fixed the issue?Sorry have been off over the end of year and I try to be seriously off when I am off. ;) Am working my way through email now and maybe this is already obsolete as I see some submissions which I have not read, yet. But doing that was on my list. -Stefan> CC-ing here Vincent in hopes of getting some hints.. > >> for SLES11 guests. SP1 got an update recently, so their HVM guests would >> have seen the hang as well. The not yet released SP2 sends >> XS_RESET_WATCHES as well since quite some time. >> >> >> Olaf
On Tue, Dec 20, Ian Campbell wrote:> On Tue, 2011-12-20 at 17:29 +0000, Ian Jackson wrote: > > Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"): > > > Sorry Olaf, have to revert that commit. > > > > I agree. When we introduced it we weren''t aware that most existing > > implementations of xenstored simply ignore unknown commands rather > > than replying with an error. If we had known this we would not have > > approved Olaf''s patch. > > > > That they ignore unknown commands is of course a bug but expecting > > everyone to update is no good. Really the best approach would be some > > kind of discovery mechanism. > > > > Maybe we should have a special path @xenstore/fail_unknown_commands > > which you could read, or something. But this time we should try it > > against old implementations. > > I was sure I''d seen some precedent (and therefore an existing path) for > this sort of thing at some point but I can''t for the life of me find it.If you could come up with a patch to present xenstored (or other dom0/toolstack) features, that would be great. Olaf
On Mon, 2012-01-02 at 17:16 +0000, Olaf Hering wrote:> On Tue, Dec 20, Ian Campbell wrote: > > > On Tue, 2011-12-20 at 17:29 +0000, Ian Jackson wrote: > > > Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"): > > > > Sorry Olaf, have to revert that commit. > > > > > > I agree. When we introduced it we weren''t aware that most existing > > > implementations of xenstored simply ignore unknown commands rather > > > than replying with an error. If we had known this we would not have > > > approved Olaf''s patch. > > > > > > That they ignore unknown commands is of course a bug but expecting > > > everyone to update is no good. Really the best approach would be some > > > kind of discovery mechanism. > > > > > > Maybe we should have a special path @xenstore/fail_unknown_commands > > > which you could read, or something. But this time we should try it > > > against old implementations. > > > > I was sure I''d seen some precedent (and therefore an existing path) for > > this sort of thing at some point but I can''t for the life of me find it. > > > If you could come up with a patch to present xenstored (or other > dom0/toolstack) features, that would be great.As I said in the bit you trimmed the precedent seems to be to write /local/domain/<N>/control/platform-feature-<X> for each guest that you want to expose the feature to. I think a per-guest key makes sense since even if something seems obviously global you never know when you might need to hide it from a guest (e.g. to workaround an in-guest bug). I''m afraid I don''t have time to work on this myself but it seems pretty simple? Ian.
On Tue, Jan 03, Ian Campbell wrote:> As I said in the bit you trimmed the precedent seems to be to > write /local/domain/<N>/control/platform-feature-<X> for each guest that > you want to expose the feature to. I think a per-guest key makes sense > since even if something seems obviously global you never know when you > might need to hide it from a guest (e.g. to workaround an in-guest bug). > > I''m afraid I don''t have time to work on this myself but it seems pretty > simple?After looking at this, the (compile tested) patch below should be enough: diff -r 3a22ed3ec534 tools/xenstore/xs.c --- a/tools/xenstore/xs.c +++ b/tools/xenstore/xs.c @@ -884,6 +884,10 @@ bool xs_introduce_domain(struct xs_handl char mfn_str[MAX_STRLEN(mfn)]; char eventchn_str[MAX_STRLEN(eventchn)]; struct iovec iov[3]; + static const char feat[] = "control/platform-feature-xs_reset_watches"; + int len; + char *dom_path, *feat_path; + bool ret; snprintf(domid_str, sizeof(domid_str), "%u", domid); snprintf(mfn_str, sizeof(mfn_str), "%lu", mfn); @@ -896,8 +900,21 @@ bool xs_introduce_domain(struct xs_handl iov[2].iov_base = eventchn_str; iov[2].iov_len = strlen(eventchn_str) + 1; - return xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov, + ret = xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov, ARRAY_SIZE(iov), NULL)); + + dom_path = xs_get_domain_path(h, domid); + if (dom_path) { + len = strlen(dom_path) + 1 + strlen(feat); + feat_path = malloc(len + 1); + if (feat_path) { + snprintf(feat_path, len + 1, "%s/%s", dom_path, feat); + xs_write(h, XBT_NULL, feat_path, "1", 1); + free(feat_path); + } + free(dom_path); + } + return ret; } bool xs_set_target(struct xs_handle *h,
On Wed, 2012-01-04 at 15:57 +0000, Olaf Hering wrote:> On Tue, Jan 03, Ian Campbell wrote: > > > As I said in the bit you trimmed the precedent seems to be to > > write /local/domain/<N>/control/platform-feature-<X> for each guest that > > you want to expose the feature to. I think a per-guest key makes sense > > since even if something seems obviously global you never know when you > > might need to hide it from a guest (e.g. to workaround an in-guest bug). > > > > I''m afraid I don''t have time to work on this myself but it seems pretty > > simple? > > After looking at this, the (compile tested) patch below should be enough: > > > diff -r 3a22ed3ec534 tools/xenstore/xs.c > --- a/tools/xenstore/xs.c > +++ b/tools/xenstore/xs.cI think the toolstack needs to be involved with this. IOW this should be done in libxl__domain_make() around the same place as control/platform-feature-multiprocessor-suspend is written. Ian.> @@ -884,6 +884,10 @@ bool xs_introduce_domain(struct xs_handl > char mfn_str[MAX_STRLEN(mfn)]; > char eventchn_str[MAX_STRLEN(eventchn)]; > struct iovec iov[3]; > + static const char feat[] = "control/platform-feature-xs_reset_watches"; > + int len; > + char *dom_path, *feat_path; > + bool ret; > > snprintf(domid_str, sizeof(domid_str), "%u", domid); > snprintf(mfn_str, sizeof(mfn_str), "%lu", mfn); > @@ -896,8 +900,21 @@ bool xs_introduce_domain(struct xs_handl > iov[2].iov_base = eventchn_str; > iov[2].iov_len = strlen(eventchn_str) + 1; > > - return xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov, > + ret = xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov, > ARRAY_SIZE(iov), NULL)); > + > + dom_path = xs_get_domain_path(h, domid); > + if (dom_path) { > + len = strlen(dom_path) + 1 + strlen(feat); > + feat_path = malloc(len + 1); > + if (feat_path) { > + snprintf(feat_path, len + 1, "%s/%s", dom_path, feat); > + xs_write(h, XBT_NULL, feat_path, "1", 1); > + free(feat_path); > + } > + free(dom_path); > + } > + return ret; > } > > bool xs_set_target(struct xs_handle *h,
On Wed, Jan 04, Ian Campbell wrote:> I think the toolstack needs to be involved with this. IOW this should be > done in libxl__domain_make() around the same place as > control/platform-feature-multiprocessor-suspend is written.Isnt XS_RESET_WATCHES a feature of xenstored, so it should add its own stuff itself? Olaf
On Wed, 2012-01-04 at 16:27 +0000, Olaf Hering wrote:> On Wed, Jan 04, Ian Campbell wrote: > > > I think the toolstack needs to be involved with this. IOW this should be > > done in libxl__domain_make() around the same place as > > control/platform-feature-multiprocessor-suspend is written. > > Isnt XS_RESET_WATCHES a feature of xenstored, so it should add its own stuff itself?Ultimately I think it needs to be the toolstack so that it can control if this gets written or not based on configuration should that become necessary. This is similar to how control/platform-feature-multiprocessor-suspend is a feature of libxc but libxl still writes the node. In general we tend to consider the toolstack as a matched set so in a given release the toolstack is allowed to rely on features (and in particular bugfixes) of xenstored in that release. Ian.
On Thu, Jan 05, Ian Campbell wrote:> On Wed, 2012-01-04 at 16:27 +0000, Olaf Hering wrote: > > On Wed, Jan 04, Ian Campbell wrote: > > > > > I think the toolstack needs to be involved with this. IOW this should be > > > done in libxl__domain_make() around the same place as > > > control/platform-feature-multiprocessor-suspend is written. > > > > Isnt XS_RESET_WATCHES a feature of xenstored, so it should add its own stuff itself? > > Ultimately I think it needs to be the toolstack so that it can control > if this gets written or not based on configuration should that become > necessary.I have sent a patch to add control/platform-feature-xs_reset_watches from libxl/xend. With this patch a guest can now decide wether it can issue the XS_RESET_WATCHES command. Olaf