xen.org
2011-Sep-01 15:54 UTC
[Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
branch xen-unstable
xen branch xen-unstable
job test-amd64-i386-rhel6hvm-intel
test xen-install
Tree: linux git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
Tree: qemu git://hg.uk.xensource.com/HG/qemu-xen-unstable.git
Tree: xen http://hg.uk.xensource.com/xen-unstable.hg
*** Found and reproduced problem changeset ***
Bug is in tree: xen http://hg.uk.xensource.com/xen-unstable.hg
Bug introduced: bb9b81008733
Bug not present: d54cfae72cd1
changeset: 23802:bb9b81008733
user: Laszlo Ersek <lersek@redhat.com>
date: Wed Aug 31 15:16:14 2011 +0100
x86: Increase the default NR_CPUS to 256
Changeset 21012:ef845a385014 bumped the default to 128 about one and a
half years ago. Increase it now to 256, as systems with eg. 160
logical CPUs are becoming (have become) common.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
For bisection revision-tuple graph see:
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-amd64-i386-rhel6hvm-intel.xen-install.html
Revision IDs in each graph node refer, respectively, to the Trees above.
----------------------------------------
Searching for failure / basis pass:
8791 fail [host=earwig] / 8769 [host=itch-mite] 8760 [host=bush-cricket] 8739
[host=itch-mite] 8735 [host=bush-cricket] 8731 [host=itch-mite] 8729
[host=itch-mite] 8727 [host=gall-mite] 8726 [host=field-cricket] 8725
[host=gall-mite] 8724 [host=bush-cricket] 8723 [host=itch-mite] 8722
[host=bush-cricket] 8721 [host=field-cricket] 8718 [host=gall-mite] 8717
[host=bush-cricket] 8715 [host=itch-mite] 8713 [host=gall-mite] 8712
[host=gall-mite] 8711 [host=gall-mite] 8710 [host=field-cricket] 8707
[host=gall-mite] 8696 [host=gall-mite] 8687 [host=gall-mite] 8674 ok.
Failure / basis pass flights: 8791 / 8674
Tree: linux git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
Tree: qemu git://hg.uk.xensource.com/HG/qemu-xen-unstable.git
Tree: xen http://hg.uk.xensource.com/xen-unstable.hg
Latest 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 4a4882df5649
Basis pass ada3f6a1ba43e163aab95c7808f11b88fc7c79e6
cd776ee9408ff127f934a707c1a339ee600bc127 fc2be6cb89ad
Generating revisions with ./adhoc-revtuple-generator
git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git#ada3f6a1ba43e163aab95c7808f11b88fc7c79e6-1c3f03ccc5258887f5f2cafc0836a865834f9205
git://hg.uk.xensource.com/HG/qemu-xen-unstable.git#cd776ee9408ff127f934a707c1a339ee600bc127-cd776ee9408ff127f934a707c1a339ee600bc127
http://hg.uk.xensource.com/xen-unstable.hg#fc2be6cb89ad-4a4882df5649
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare
git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
/export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
pulling from http://hg.uk.xensource.com/xen-unstable.hg
searching for changes
no changes found
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare
git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
/export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
pulling from http://hg.uk.xensource.com/xen-unstable.hg
searching for changes
no changes found
Loaded 2260 nodes in revision graph
Searching for test results:
8674 pass ada3f6a1ba43e163aab95c7808f11b88fc7c79e6
cd776ee9408ff127f934a707c1a339ee600bc127 fc2be6cb89ad
8664 pass ada3f6a1ba43e163aab95c7808f11b88fc7c79e6
cd776ee9408ff127f934a707c1a339ee600bc127 fc2be6cb89ad
8712 [host=gall-mite]
8722 [host=bush-cricket]
8713 [host=gall-mite]
8687 [host=gall-mite]
8735 [host=bush-cricket]
8723 [host=itch-mite]
8696 [host=gall-mite]
8715 [host=itch-mite]
8707 [host=gall-mite]
8724 [host=bush-cricket]
8710 [host=field-cricket]
8717 [host=bush-cricket]
8711 [host=gall-mite]
8718 [host=gall-mite]
8725 [host=gall-mite]
8721 [host=field-cricket]
8729 [host=itch-mite]
8726 [host=field-cricket]
8727 [host=gall-mite]
8731 [host=itch-mite]
8739 [host=itch-mite]
8786 pass 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 2c687e70a343
8806 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 bb9b81008733
8781 [host=field-cricket]
8760 [host=bush-cricket]
8790 pass 20a27c1e25b8550066902c9d6ca91631e656dfa3
cd776ee9408ff127f934a707c1a339ee600bc127 41f00cf6b822
8792 [host=field-cricket]
8793 [host=field-cricket]
8769 [host=itch-mite]
8794 [host=field-cricket]
8791 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 4a4882df5649
8795 [host=field-cricket]
8796 pass 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 ac9aa65050e9
8797 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 51983821efa4
8776 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 4a4882df5649
8798 pass 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 d54cfae72cd1
8782 pass ada3f6a1ba43e163aab95c7808f11b88fc7c79e6
cd776ee9408ff127f934a707c1a339ee600bc127 fc2be6cb89ad
8799 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 bb9b81008733
8784 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 4a4882df5649
8800 pass 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 d54cfae72cd1
8804 fail 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 bb9b81008733
8805 pass 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 d54cfae72cd1
Searching for interesting versions
Result found: flight 8664 (pass), for basis pass
Result found: flight 8776 (fail), for basis failure
Repro found: flight 8782 (pass), for basis pass
Repro found: flight 8784 (fail), for basis failure
0 revisions at 1c3f03ccc5258887f5f2cafc0836a865834f9205
cd776ee9408ff127f934a707c1a339ee600bc127 d54cfae72cd1
No revisions left to test, checking graph state.
Result found: flight 8798 (pass), for last pass
Result found: flight 8799 (fail), for first failure
Repro found: flight 8800 (pass), for last pass
Repro found: flight 8804 (fail), for first failure
Repro found: flight 8805 (pass), for last pass
Repro found: flight 8806 (fail), for first failure
*** Found and reproduced problem changeset ***
Bug is in tree: xen http://hg.uk.xensource.com/xen-unstable.hg
Bug introduced: bb9b81008733
Bug not present: d54cfae72cd1
pulling from http://hg.uk.xensource.com/xen-unstable.hg
searching for changes
no changes found
changeset: 23802:bb9b81008733
user: Laszlo Ersek <lersek@redhat.com>
date: Wed Aug 31 15:16:14 2011 +0100
x86: Increase the default NR_CPUS to 256
Changeset 21012:ef845a385014 bumped the default to 128 about one and a
half years ago. Increase it now to 256, as systems with eg. 160
logical CPUs are becoming (have become) common.
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Revision graph left in
/home/xc_osstest/results/bisect.xen-unstable.test-amd64-i386-rhel6hvm-intel.xen-install.{dot,ps,png,html}.
----------------------------------------
8806: ALL FAIL
flight 8806 xen-unstable real-bisect [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/8806/
jobs:
build-i386 fail
test-amd64-i386-rhel6hvm-intel fail
------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://www.chiark.greenend.org.uk/~xensrcts/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ian Jackson
2011-Sep-01 16:26 UTC
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
xen.org writes ("[Xen-devel] [xen-unstable bisection] complete
test-amd64-i386-rhel6hvm-intel"):> branch xen-unstable
> xen branch xen-unstable
> job test-amd64-i386-rhel6hvm-intel
> test xen-install
>
> Tree: linux git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> Tree: qemu git://hg.uk.xensource.com/HG/qemu-xen-unstable.git
> Tree: xen http://hg.uk.xensource.com/xen-unstable.hg
>
> *** Found and reproduced problem changeset ***
>
> Bug is in tree: xen http://hg.uk.xensource.com/xen-unstable.hg
> Bug introduced: bb9b81008733
> Bug not present: d54cfae72cd1
>
>
> changeset: 23802:bb9b81008733
> user: Laszlo Ersek <lersek@redhat.com>
> date: Wed Aug 31 15:16:14 2011 +0100
>
> x86: Increase the default NR_CPUS to 256
>
> Changeset 21012:ef845a385014 bumped the default to 128 about one and
a
> half years ago. Increase it now to 256, as systems with eg. 160
> logical CPUs are becoming (have become) common.
>
> Signed-off-by: Laszlo Ersek <lersek@redhat.com>
My bisector is pretty reliable nowadays. Looking at the revision
graph it tested before/after/before/after/before/after, ie three times
each on the same host.
This change looks innocuous enough TBH. Is there any way this change
could have broken a PV-on-HVM guest ? Note that RHEL6, which is what
this is testing, seems to generally be full of bugs.
If the problem is indeed a bug in the current RHEL6 then I will add
this test to the "do not care" list.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Laszlo Ersek
2011-Sep-01 17:22 UTC
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
On 09/01/11 18:26, Ian Jackson wrote:>> job test-amd64-i386-rhel6hvm-intel>> changeset: 23802:bb9b81008733 >> user: Laszlo Ersek<lersek@redhat.com> >> date: Wed Aug 31 15:16:14 2011 +0100 >> >> x86: Increase the default NR_CPUS to 256 >> >> Changeset 21012:ef845a385014 bumped the default to 128 about one and a >> half years ago. Increase it now to 256, as systems with eg. 160 >> logical CPUs are becoming (have become) common. >> >> Signed-off-by: Laszlo Ersek<lersek@redhat.com> > > My bisector is pretty reliable nowadays. Looking at the revision > graph it tested before/after/before/after/before/after, ie three times > each on the same host. > > This change looks innocuous enough TBH. Is there any way this change > could have broken a PV-on-HVM guest ? Note that RHEL6, which is what > this is testing, seems to generally be full of bugs. > > If the problem is indeed a bug in the current RHEL6 then I will add > this test to the "do not care" list.In what way was the guest broken? How many physical cores/threads was the hypervisor running on? Thanks, lacos _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Laszlo Ersek
2011-Sep-01 17:48 UTC
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
On 09/01/11 18:26, Ian Jackson wrote:>> changeset: 23802:bb9b81008733 >> user: Laszlo Ersek<lersek@redhat.com> >> date: Wed Aug 31 15:16:14 2011 +0100 >> >> x86: Increase the default NR_CPUS to 256 >> >> Changeset 21012:ef845a385014 bumped the default to 128 about one and a >> half years ago. Increase it now to 256, as systems with eg. 160 >> logical CPUs are becoming (have become) common. >> >> Signed-off-by: Laszlo Ersek<lersek@redhat.com>FWIW, the hypervisor shipped in RHEL-5 has been built for 256 CPUs since April 2009, using the max_phys_cpus make macro. I posted the patch because now we changed the in-source macro definition too. lacos _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Jones
2011-Sep-01 19:28 UTC
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
----- Original Message -----> xen.org writes ("[Xen-devel] [xen-unstable bisection] complete > test-amd64-i386-rhel6hvm-intel"): > > branch xen-unstable > > xen branch xen-unstable > > job test-amd64-i386-rhel6hvm-intel > > test xen-install > > > > Tree: linux > > git://git.eu.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git > > Tree: qemu git://hg.uk.xensource.com/HG/qemu-xen-unstable.git > > Tree: xen http://hg.uk.xensource.com/xen-unstable.hg > > > > *** Found and reproduced problem changeset *** > > > > Bug is in tree: xen http://hg.uk.xensource.com/xen-unstable.hg > > Bug introduced: bb9b81008733 > > Bug not present: d54cfae72cd1 > > > > > > changeset: 23802:bb9b81008733 > > user: Laszlo Ersek <lersek@redhat.com> > > date: Wed Aug 31 15:16:14 2011 +0100 > > > > x86: Increase the default NR_CPUS to 256 > > > > Changeset 21012:ef845a385014 bumped the default to 128 about > > one and a > > half years ago. Increase it now to 256, as systems with eg. > > 160 > > logical CPUs are becoming (have become) common. > > > > Signed-off-by: Laszlo Ersek <lersek@redhat.com> > > My bisector is pretty reliable nowadays. Looking at the revision > graph it tested before/after/before/after/before/after, ie three times > each on the same host. > > This change looks innocuous enough TBH. Is there any way this change > could have broken a PV-on-HVM guest ? Note that RHEL6, which is what > this is testing, seems to generally be full of bugs.It''s seems unlikely this change could break a guest, but without any output from you tests it''s impossible to tell. The fact it failed on the same host each of the three times is probably a clue worth looking further at. I take it that it succeeded on other hosts? Which RHEL6 kernel release do you test with? When you say "full of bugs", where have the bugs been filed? Are those bugs only present with the pv-on-hvm drivers? IMO, the HV should support the guest (especially an HVM guest), even if it was based on something as "old" as 2.6.32. So the bugs you''re finding should likely be looked at from both the host and the guest sides, certainly not ignored.> > If the problem is indeed a bug in the current RHEL6 then I will add > this test to the "do not care" list. >This attitude won''t get anybody anywhere.> Ian. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Sep-02 07:11 UTC
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
On Thu, 2011-09-01 at 18:22 +0100, Laszlo Ersek wrote:> On 09/01/11 18:26, Ian Jackson wrote: > > >> job test-amd64-i386-rhel6hvm-intel > > >> changeset: 23802:bb9b81008733 > >> user: Laszlo Ersek<lersek@redhat.com> > >> date: Wed Aug 31 15:16:14 2011 +0100 > >> > >> x86: Increase the default NR_CPUS to 256 > >> > >> Changeset 21012:ef845a385014 bumped the default to 128 about one and a > >> half years ago. Increase it now to 256, as systems with eg. 160 > >> logical CPUs are becoming (have become) common. > >> > >> Signed-off-by: Laszlo Ersek<lersek@redhat.com> > > > > My bisector is pretty reliable nowadays. Looking at the revision > > graph it tested before/after/before/after/before/after, ie three times > > each on the same host. > > > > This change looks innocuous enough TBH. Is there any way this change > > could have broken a PV-on-HVM guest ? Note that RHEL6, which is what > > this is testing, seems to generally be full of bugs. > > > > If the problem is indeed a bug in the current RHEL6 then I will add > > this test to the "do not care" list. > > In what way was the guest broken? How many physical cores/threads was > the hypervisor running on?This is just confusion over the way the failure is reported. The bisector was running the test-amd64-i386-rhel6hvm-intel job but it was actually failing at the build/install Xen stage and not getting anywhere near actually testing rhel6hvm. This confused me (and apparently IanJ) too. For future reference the thing to look at is the report''s header which in this case said: job test-amd64-i386-rhel6hvm-intel test xen-install i.e. the xen-install stage failed while running the test-amd64-i386-rhel6hvm-intel sequence. The selection of the test-amd64-i386-rhel6hvm-intel sequence for bisecting is apparently just an arbitrary choice out of all the sequences which suffered this failure. The actual fix for this issue was identified and posted in the "8803: regressions - FAIL" thread. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2011-Sep-02 11:08 UTC
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-rhel6hvm-intel
Andrew Jones writes ("Re: [Xen-devel] [xen-unstable bisection] complete
test-amd64-i386-rhel6hvm-intel"):> It''s seems unlikely this change could break a guest, but without
any
> output from you tests it''s impossible to tell. The fact it failed
on
> the same host each of the three times is probably a clue worth looking
> further at. I take it that it succeeded on other hosts?
Sorry, this particular problem was a Xen build failure and nothing to
do with RHEL6.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel