thr3ads.net - Xen devel - [xen-unstable bisection] complete test-i386-i386-xl [Jan 2012]

If this information is useful, please help other people find it:
Share via:

xen.org

2012-Jan-14 12:05 UTC

[xen-unstable bisection] complete test-i386-i386-xl

branch xen-unstable
xen branch xen-unstable
job test-i386-i386-xl
test guest-localmigrate

Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg

*** Found and reproduced problem changeset ***

  Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
  Bug introduced:  3c4c5df2cd0e
  Bug not present: 92630d4b093e


  changeset:   24471:3c4c5df2cd0e
  user:        Paul Durrant <paul.durrant@citrix.com>
  date:        Fri Dec 16 14:54:14 2011 +0000
      
      tools: VM generation ID save/restore and migrate.
      
      Add code to track the address of the VM generation ID buffer across a
      save/restore or migrate, and increment it as necessary.
      The address of the buffer is written into xenstore by hvmloader at
      boot time. It must be read from xenstore by the caller of
      xc_domain_save() and then written back again by the caller of
      xc_domain_restore().
      
      Note that the changes to xc_save.c and xc_restore.c are merely
      sufficient for them to build.
      
      Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
      Committed-by: Ian Jackson <ian.jackson.citrix.com>
      
      


For bisection revision-tuple graph see:
  
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-i386-i386-xl.guest-localmigrate.html
Revision IDs in each graph node refer, respectively, to the Trees above.

----------------------------------------
Searching for failure / basis pass:
 10700 fail [host=woodlouse] / 10649 ok.
Failure / basis pass flights: 10700 / 10649
Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
Latest 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
Basis pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
Generating revisions with ./adhoc-revtuple-generator 
git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git#59b1ffe039ce366305d48176b7fccce29ce729d3-59b1ffe039ce366305d48176b7fccce29ce729d3
git://xenbits.xen.org/staging/qemu-xen-unstable.git#bb36d632e4cabf47882adff07a45c6702c4a5b30-bb36d632e4cabf47882adff07a45c6702c4a5b30
http://xenbits.xen.org/staging/xen-unstable.hg#5b2676ac1321-6d8888519e3a
pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
searching for changes
no changes found
pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
searching for changes
no changes found
Loaded 49 nodes in revision graph
Searching for test results:
 10697 [host=potato-beetle]
 10708 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 db22b1aa11d3
 10689 [host=itch-mite]
 10698 [host=potato-beetle]
 10715 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
 10709 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
 10681 [host=potato-beetle]
 10645 [host=potato-beetle]
 10699 [host=itch-mite]
 10646 [host=moss-bug]
 10649 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
 10701 [host=itch-mite]
 10680 [host=lace-bug]
 10710 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 1e3fa01b0c9e
 10702 [host=itch-mite]
 10703 [host=itch-mite]
 10711 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
 10700 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
 10704 [host=itch-mite]
 10712 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
 10705 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
 10706 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
 10713 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
 10707 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 32dd444700bd
 10714 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
Searching for interesting versions
 Result found: flight 10649 (pass), for basis pass
 Result found: flight 10700 (fail), for basis failure
 Repro found: flight 10705 (pass), for basis pass
 Repro found: flight 10706 (fail), for basis failure
 0 revisions at 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
No revisions left to test, checking graph state.
 Result found: flight 10709 (pass), for last pass
 Result found: flight 10711 (fail), for first failure
 Repro found: flight 10712 (pass), for last pass
 Repro found: flight 10713 (fail), for first failure
 Repro found: flight 10714 (pass), for last pass
 Repro found: flight 10715 (fail), for first failure

*** Found and reproduced problem changeset ***

  Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
  Bug introduced:  3c4c5df2cd0e
  Bug not present: 92630d4b093e

pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
searching for changes
no changes found

  changeset:   24471:3c4c5df2cd0e
  user:        Paul Durrant <paul.durrant@citrix.com>
  date:        Fri Dec 16 14:54:14 2011 +0000
      
      tools: VM generation ID save/restore and migrate.
      
      Add code to track the address of the VM generation ID buffer across a
      save/restore or migrate, and increment it as necessary.
      The address of the buffer is written into xenstore by hvmloader at
      boot time. It must be read from xenstore by the caller of
      xc_domain_save() and then written back again by the caller of
      xc_domain_restore().
      
      Note that the changes to xc_save.c and xc_restore.c are merely
      sufficient for them to build.
      
      Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
      Committed-by: Ian Jackson <ian.jackson.citrix.com>
      
      

Revision graph left in
/home/xc_osstest/results/bisect.xen-unstable.test-i386-i386-xl.guest-localmigrate.{dot,ps,png,html}.
----------------------------------------
10715: ALL FAIL

flight 10715 xen-unstable real-bisect [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/10715/


jobs:
 test-i386-i386-xl                                            fail    


------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary

Ian Campbell

2012-Jan-16 10:21 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

On Sat, 2012-01-14 at 12:05 +0000, xen.org wrote:> branch xen-unstable
> xen branch xen-unstable
> job test-i386-i386-xl
> test guest-localmigrate
> 
> Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
>   Bug introduced:  3c4c5df2cd0e
>   Bug not present: 92630d4b093e
> 
> 
>   changeset:   24471:3c4c5df2cd0e
>   user:        Paul Durrant <paul.durrant@citrix.com>
>   date:        Fri Dec 16 14:54:14 2011 +0000
>       
>       tools: VM generation ID save/restore and migrate.
>       
>       Add code to track the address of the VM generation ID buffer across a
>       save/restore or migrate, and increment it as necessary.
>       The address of the buffer is written into xenstore by hvmloader at
>       boot time. It must be read from xenstore by the caller of
>       xc_domain_save() and then written back again by the caller of
>       xc_domain_restore().
>       
>       Note that the changes to xc_save.c and xc_restore.c are merely
>       sufficient for them to build.
>       
>       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
>       Committed-by: Ian Jackson <ian.jackson.citrix.com>
The logs show a guest kernel crash but I think this is a second order
effect (which I''ll investigate) related to xl not recovering from a
failed migration in a way that guest kernels actually like.

The original failure seems to be a PV migration failure. Paul can you
look into that aspect please. Probably an imbalance for PV save/restore
due to something not correctly gated on HVM on one side or the other.

Thanks,
Ian.
>       
>       
> 
> 
> For bisection revision-tuple graph see:
>   
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-i386-i386-xl.guest-localmigrate.html
> Revision IDs in each graph node refer, respectively, to the Trees above.
> 
> ----------------------------------------
> Searching for failure / basis pass:
>  10700 fail [host=woodlouse] / 10649 ok.
> Failure / basis pass flights: 10700 / 10649
> Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> Latest 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
> Basis pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
> Generating revisions with ./adhoc-revtuple-generator 
git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git#59b1ffe039ce366305d48176b7fccce29ce729d3-59b1ffe039ce366305d48176b7fccce29ce729d3
git://xenbits.xen.org/staging/qemu-xen-unstable.git#bb36d632e4cabf47882adff07a45c6702c4a5b30-bb36d632e4cabf47882adff07a45c6702c4a5b30
http://xenbits.xen.org/staging/xen-unstable.hg#5b2676ac1321-6d8888519e3a
> pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> searching for changes
> no changes found
> pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> searching for changes
> no changes found
> Loaded 49 nodes in revision graph
> Searching for test results:
>  10697 [host=potato-beetle]
>  10708 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 db22b1aa11d3
>  10689 [host=itch-mite]
>  10698 [host=potato-beetle]
>  10715 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
>  10709 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
>  10681 [host=potato-beetle]
>  10645 [host=potato-beetle]
>  10699 [host=itch-mite]
>  10646 [host=moss-bug]
>  10649 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
>  10701 [host=itch-mite]
>  10680 [host=lace-bug]
>  10710 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 1e3fa01b0c9e
>  10702 [host=itch-mite]
>  10703 [host=itch-mite]
>  10711 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
>  10700 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
>  10704 [host=itch-mite]
>  10712 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
>  10705 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
>  10706 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
>  10713 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
>  10707 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 32dd444700bd
>  10714 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
> Searching for interesting versions
>  Result found: flight 10649 (pass), for basis pass
>  Result found: flight 10700 (fail), for basis failure
>  Repro found: flight 10705 (pass), for basis pass
>  Repro found: flight 10706 (fail), for basis failure
>  0 revisions at 59b1ffe039ce366305d48176b7fccce29ce729d3
bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
> No revisions left to test, checking graph state.
>  Result found: flight 10709 (pass), for last pass
>  Result found: flight 10711 (fail), for first failure
>  Repro found: flight 10712 (pass), for last pass
>  Repro found: flight 10713 (fail), for first failure
>  Repro found: flight 10714 (pass), for last pass
>  Repro found: flight 10715 (fail), for first failure
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
>   Bug introduced:  3c4c5df2cd0e
>   Bug not present: 92630d4b093e
> 
> pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> searching for changes
> no changes found
> 
>   changeset:   24471:3c4c5df2cd0e
>   user:        Paul Durrant <paul.durrant@citrix.com>
>   date:        Fri Dec 16 14:54:14 2011 +0000
>       
>       tools: VM generation ID save/restore and migrate.
>       
>       Add code to track the address of the VM generation ID buffer across a
>       save/restore or migrate, and increment it as necessary.
>       The address of the buffer is written into xenstore by hvmloader at
>       boot time. It must be read from xenstore by the caller of
>       xc_domain_save() and then written back again by the caller of
>       xc_domain_restore().
>       
>       Note that the changes to xc_save.c and xc_restore.c are merely
>       sufficient for them to build.
>       
>       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
>       Committed-by: Ian Jackson <ian.jackson.citrix.com>
>       
>       
> 
> Revision graph left in
/home/xc_osstest/results/bisect.xen-unstable.test-i386-i386-xl.guest-localmigrate.{dot,ps,png,html}.
> ----------------------------------------
> 10715: ALL FAIL
> 
> flight 10715 xen-unstable real-bisect [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/10715/
> 
> 
> jobs:
>  test-i386-i386-xl                                            fail    
> 
> 
> ------------------------------------------------------------
> sg-report-flight on woking.cam.xci-test.com
> logs: /home/xc_osstest/logs
> images: /home/xc_osstest/images
> 
> Logs, config files, etc. are available at
>     http://www.chiark.greenend.org.uk/~xensrcts/logs
> 
> Test harness code can be found at
>     http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Paul Durrant

2012-Jan-16 10:32 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

> -----Original Message-----
> From: Ian Campbell
> Sent: 16 January 2012 10:22
> To: Ian Jackson; Paul Durrant
> Cc: xen-devel@lists.xensource.com; Keir (Xen.org); Stefano Stabellini
> Subject: Re: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl
> 
> On Sat, 2012-01-14 at 12:05 +0000, xen.org wrote:
> > branch xen-unstable
> > xen branch xen-unstable
> > job test-i386-i386-xl
> > test guest-localmigrate
> >
> > Tree: linux
> > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> > Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> > Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> >
> > *** Found and reproduced problem changeset ***
> >
> >   Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
> >   Bug introduced:  3c4c5df2cd0e
> >   Bug not present: 92630d4b093e
> >
> >
> >   changeset:   24471:3c4c5df2cd0e
> >   user:        Paul Durrant <paul.durrant@citrix.com>
> >   date:        Fri Dec 16 14:54:14 2011 +0000
> >
> >       tools: VM generation ID save/restore and migrate.
> >
> >       Add code to track the address of the VM generation ID buffer
across a
> >       save/restore or migrate, and increment it as necessary.
> >       The address of the buffer is written into xenstore by hvmloader
at
> >       boot time. It must be read from xenstore by the caller of
> >       xc_domain_save() and then written back again by the caller of
> >       xc_domain_restore().
> >
> >       Note that the changes to xc_save.c and xc_restore.c are merely
> >       sufficient for them to build.
> >
> >       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
> >       Committed-by: Ian Jackson <ian.jackson.citrix.com>
> 
> The logs show a guest kernel crash but I think this is a second order
effect
> (which I''ll investigate) related to xl not recovering from a
failed migration in
> a way that guest kernels actually like.
> 
> The original failure seems to be a PV migration failure. Paul can you look
> into that aspect please. Probably an imbalance for PV save/restore due to
> something not correctly gated on HVM on one side or the other.
> 
Yep, I''ll check now.

  Paul
> Thanks,
> Ian.
> 
> >
> >
> >
> >
> > For bisection revision-tuple graph see:
> >
> > http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstabl
> > e.test-i386-i386-xl.guest-localmigrate.html
> > Revision IDs in each graph node refer, respectively, to the Trees
above.
> >
> > ----------------------------------------
> > Searching for failure / basis pass:
> >  10700 fail [host=woodlouse] / 10649 ok.
> > Failure / basis pass flights: 10700 / 10649
> > Tree: linux
> > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> > Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> > Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> > Latest 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a Basis pass
> > 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321 Generating
> > revisions with ./adhoc-revtuple-generator
> > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git#59b1ffe03
> > 9ce366305d48176b7fccce29ce729d3-
> 59b1ffe039ce366305d48176b7fccce29ce729
> > d3
> > git://xenbits.xen.org/staging/qemu-xen-
> unstable.git#bb36d632e4cabf4788
> > 2adff07a45c6702c4a5b30-bb36d632e4cabf47882adff07a45c6702c4a5b30
> > http://xenbits.xen.org/staging/xen-unstable.hg#5b2676ac1321-
> 6d8888519e
> > 3a pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> > searching for changes
> > no changes found
> > pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> > searching for changes
> > no changes found
> > Loaded 49 nodes in revision graph
> > Searching for test results:
> >  10697 [host=potato-beetle]
> >  10708 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 db22b1aa11d3
> >  10689 [host=itch-mite]
> >  10698 [host=potato-beetle]
> >  10715 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
> >  10709 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
> >  10681 [host=potato-beetle]
> >  10645 [host=potato-beetle]
> >  10699 [host=itch-mite]
> >  10646 [host=moss-bug]
> >  10649 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
> >  10701 [host=itch-mite]
> >  10680 [host=lace-bug]
> >  10710 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 1e3fa01b0c9e
> >  10702 [host=itch-mite]
> >  10703 [host=itch-mite]
> >  10711 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
> >  10700 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
> >  10704 [host=itch-mite]
> >  10712 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
> >  10705 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
> >  10706 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
> >  10713 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
> >  10707 fail 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 32dd444700bd
> >  10714 pass 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e Searching for
> > interesting versions  Result found: flight 10649 (pass), for basis
> > pass  Result found: flight 10700 (fail), for basis failure  Repro
> > found: flight 10705 (pass), for basis pass  Repro found: flight 10706
> > (fail), for basis failure
> >  0 revisions at 59b1ffe039ce366305d48176b7fccce29ce729d3
> > bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e No revisions
> left to test, checking graph state.
> >  Result found: flight 10709 (pass), for last pass  Result found:
> > flight 10711 (fail), for first failure  Repro found: flight 10712
> > (pass), for last pass  Repro found: flight 10713 (fail), for first
> > failure  Repro found: flight 10714 (pass), for last pass  Repro found:
> > flight 10715 (fail), for first failure
> >
> > *** Found and reproduced problem changeset ***
> >
> >   Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
> >   Bug introduced:  3c4c5df2cd0e
> >   Bug not present: 92630d4b093e
> >
> > pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> > searching for changes
> > no changes found
> >
> >   changeset:   24471:3c4c5df2cd0e
> >   user:        Paul Durrant <paul.durrant@citrix.com>
> >   date:        Fri Dec 16 14:54:14 2011 +0000
> >
> >       tools: VM generation ID save/restore and migrate.
> >
> >       Add code to track the address of the VM generation ID buffer
across a
> >       save/restore or migrate, and increment it as necessary.
> >       The address of the buffer is written into xenstore by hvmloader
at
> >       boot time. It must be read from xenstore by the caller of
> >       xc_domain_save() and then written back again by the caller of
> >       xc_domain_restore().
> >
> >       Note that the changes to xc_save.c and xc_restore.c are merely
> >       sufficient for them to build.
> >
> >       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
> >       Committed-by: Ian Jackson <ian.jackson.citrix.com>
> >
> >
> >
> > Revision graph left in
/home/xc_osstest/results/bisect.xen-unstable.test-
> i386-i386-xl.guest-localmigrate.{dot,ps,png,html}.
> > ----------------------------------------
> > 10715: ALL FAIL
> >
> > flight 10715 xen-unstable real-bisect [real]
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/10715/
> >
> >
> > jobs:
> >  test-i386-i386-xl                                            fail
> >
> >
> > ------------------------------------------------------------
> > sg-report-flight on woking.cam.xci-test.com
> > logs: /home/xc_osstest/logs
> > images: /home/xc_osstest/images
> >
> > Logs, config files, etc. are available at
> >     http://www.chiark.greenend.org.uk/~xensrcts/logs
> >
> > Test harness code can be found at
> >     http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>

Paul Durrant

2012-Jan-16 10:59 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Paul Durrant
> Sent: 16 January 2012 10:33
> To: Ian Campbell; Ian Jackson
> Cc: xen-devel@lists.xensource.com; Keir (Xen.org); Stefano Stabellini
> Subject: Re: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl
> 
> > -----Original Message-----
> > From: Ian Campbell
> > Sent: 16 January 2012 10:22
> > To: Ian Jackson; Paul Durrant
> > Cc: xen-devel@lists.xensource.com; Keir (Xen.org); Stefano Stabellini
> > Subject: Re: [Xen-devel] [xen-unstable bisection] complete
> > test-i386-i386-xl
> >
> > On Sat, 2012-01-14 at 12:05 +0000, xen.org wrote:
> > > branch xen-unstable
> > > xen branch xen-unstable
> > > job test-i386-i386-xl
> > > test guest-localmigrate
> > >
> > > Tree: linux
> > > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> > > Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> > > Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> > >
> > > *** Found and reproduced problem changeset ***
> > >
> > >   Bug is in tree:  xen
http://xenbits.xen.org/staging/xen-unstable.hg
> > >   Bug introduced:  3c4c5df2cd0e
> > >   Bug not present: 92630d4b093e
> > >
> > >
> > >   changeset:   24471:3c4c5df2cd0e
> > >   user:        Paul Durrant <paul.durrant@citrix.com>
> > >   date:        Fri Dec 16 14:54:14 2011 +0000
> > >
> > >       tools: VM generation ID save/restore and migrate.
> > >
> > >       Add code to track the address of the VM generation ID
buffer across a
> > >       save/restore or migrate, and increment it as necessary.
> > >       The address of the buffer is written into xenstore by
hvmloader at
> > >       boot time. It must be read from xenstore by the caller of
> > >       xc_domain_save() and then written back again by the caller
of
> > >       xc_domain_restore().
> > >
> > >       Note that the changes to xc_save.c and xc_restore.c are
merely
> > >       sufficient for them to build.
> > >
> > >       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
> > >       Committed-by: Ian Jackson <ian.jackson.citrix.com>
> >
> > The logs show a guest kernel crash but I think this is a second order
> > effect (which I''ll investigate) related to xl not recovering
from a
> > failed migration in a way that guest kernels actually like.
> >
> > The original failure seems to be a PV migration failure. Paul can you
> > look into that aspect please. Probably an imbalance for PV
> > save/restore due to something not correctly gated on HVM on one side
or
> the other.
> >
> 
> Yep, I''ll check now.
> 
It all seems to be gated correctly on the whether the VM is HVM and/or the
existence of the appropriate save record.

   Paul

Ian Jackson

2012-Jan-16 12:17 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

Paul Durrant writes ("RE: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl"):> It all seems to be gated correctly on the whether the VM is HVM and/or the
existence of the appropriate save record.
Well, the bisector, when it fingers a changeset, is pretty reliable.
The revision graph shows that it tried each of 3c4c5df2cd0e and
92630d4b093e three times and got consistent results.

So as you were the author of the original patch, can you please try to
reproduce the problem and fix it ?

Thanks,
Ian.

Paul Durrant

2012-Jan-16 13:17 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

> -----Original Message-----
> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> Sent: 16 January 2012 12:18
> To: Paul Durrant
> Cc: Ian Campbell; xen-devel@lists.xensource.com; Keir (Xen.org); Stefano
> Stabellini
> Subject: RE: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl
> 
> Paul Durrant writes ("RE: [Xen-devel] [xen-unstable bisection]
complete
> test-i386-i386-xl"):
> > It all seems to be gated correctly on the whether the VM is HVM and/or
> the existence of the appropriate save record.
> 
> Well, the bisector, when it fingers a changeset, is pretty reliable.
> The revision graph shows that it tried each of 3c4c5df2cd0e and
> 92630d4b093e three times and got consistent results.
> 
> So as you were the author of the original patch, can you please try to
> reproduce the problem and fix it ?
> 
Already on it.

  Paul

Ian Jackson

2012-Jan-16 14:30 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

Paul Durrant writes ("RE: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl"):> > -----Original Message-----
> > From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
...> > So as you were the author of the original patch, can you please try to
> > reproduce the problem and fix it ?
> > 
> 
> Already on it.
Great, thanks.  If you need any help, or to borrow the machine from my
test pool, or something, just let me know.

Ian.

Paul Durrant

2012-Jan-17 10:04 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

So far I''ve been unable to repro. with a build of staging tip from
yesterday (running a 512MB PV debian guest with a 3.2.0-rc2+ kernel).
I''ll revert to the appropriate c/s and see if I can repro. with that.
The errors unfortunately don''t give much away other than the receiving
process clearly didn''t like something about the migration data stream.

libxl: error: libxl_utils.c:409:libxl_read_exactly: file/stream truncated
reading ready message from migration receiver stream
libxl: info: libxl_exec.c:124:libxl_report_child_exitstatus: migration target
process [14446] exited with error status 255

  Paul
> -----Original Message-----
> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> Sent: 16 January 2012 14:31
> To: Paul Durrant
> Cc: Ian Campbell; xen-devel@lists.xensource.com; Keir (Xen.org); Stefano
> Stabellini
> Subject: RE: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl
> 
> Paul Durrant writes ("RE: [Xen-devel] [xen-unstable bisection]
complete
> test-i386-i386-xl"):
> > > -----Original Message-----
> > > From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> ...
> > > So as you were the author of the original patch, can you please
try
> > > to reproduce the problem and fix it ?
> > >
> >
> > Already on it.
> 
> Great, thanks.  If you need any help, or to borrow the machine from my test
> pool, or something, just let me know.
> 
> Ian.

Paul Durrant

2012-Jan-17 11:05 UTC

head link

Re: [xen-unstable bisection] complete test-i386-i386-xl

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Paul Durrant
> Sent: 17 January 2012 10:05
> To: Ian Jackson
> Cc: xen-devel@lists.xensource.com; Keir (Xen.org); Ian Campbell; Stefano
> Stabellini
> Subject: Re: [Xen-devel] [xen-unstable bisection] complete
test-i386-i386-xl
> 
> So far I''ve been unable to repro. with a build of staging tip from
yesterday
> (running a 512MB PV debian guest with a 3.2.0-rc2+ kernel). I''ll
revert to the
> appropriate c/s and see if I can repro. with that. The errors unfortunately
> don''t give much away other than the receiving process clearly
didn''t like
> something about the migration data stream.
> 
> libxl: error: libxl_utils.c:409:libxl_read_exactly: file/stream truncated
> reading ready message from migration receiver stream
> libxl: info: libxl_exec.c:124:libxl_report_child_exitstatus: migration
target
> process [14446] exited with error status 255
> 
Apologies for the previous top-post; it seems impossible to configure outlook to
not do that by default :-(

I reproduced the problem. The bug has been identified and a fix verified. I
posted the patch a couple of minutes ago.

  Paul

Xen devel - Jan 2012 - [xen-unstable bisection] complete test-i386-i386-xl

[xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl

Re: [xen-unstable bisection] complete test-i386-i386-xl