llvm dev - Sep 2017 - Please help: SVN is broken

Hi all, for the past several days I've been dealing with intermittent bot
failures due to a recent series of patches which changes some things about
lit.

Most of the dust has settled, but there's several bots that just won't
stay
green.

An example is this one:

http://lab.llvm.org:8011/builders/llvm-hexagon-elf

What happens in all of these cases is that SVN encounters a tree conflict
when updating, and then it gets stuck on some messed up of the code where
the test suite doesn't match up with the code being tested.

But at this point I honestly think this is some kind of bug in SVN or
buildbot and I'm not entirely sure what to do about it.  For these affected
bots, I can force a clobber by setting build revision to 0 and forcing a
build.  This will rm -rf the source tree and check out a new one.  Surely
this should resolve any conflicts right?  But it doesn't.  For example:

In this build I clobber the source tree.  You can see by clicking the
svn-llvm step.  You can see that all tests pass.
http://lab.llvm.org:8011/builders/llvm-hexagon-elf/builds/9746

The next build, 9747 also passes and if you click the svn-llvm step you'll
see that it just updates some files.

But all of a sudden, 9749 fails.  If you look at the svn-llvm step again,
you see this:

Updating '.':
Skipped 'utils/lit/lit/llvm' -- Node remains in conflict

and now everything starts failing.

But it shouldn't even be trying to update this!   No file in that directory
has changed.  If you look at 9747, 9748, and 9749, none of those CLs touch
any files in that folder.

So, I'm really at a loss here.  I don't think reverting is going to fix
anything, because I believe the damage to be related to some intermediate
commit, and the revert is just going to get stacked onto the end of the
commit list.  Something seems fundamentally messed up here.

Can anyone provide some insight?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170920/cd587447/attachment.html>

On 9/19/2017 6:33 PM, Zachary Turner via llvm-dev wrote:
> Hi all, for the past several days I've been dealing with intermittent 
> bot failures due to a recent series of patches which changes some 
> things about lit.
>
> Most of the dust has settled, but there's several bots that just
won't
> stay green.
>
> An example is this one:
>
> http://lab.llvm.org:8011/builders/llvm-hexagon-elf
>
> What happens in all of these cases is that SVN encounters a tree 
> conflict when updating, and then it gets stuck on some messed up of 
> the code where the test suite doesn't match up with the code being
tested.
>
> But at this point I honestly think this is some kind of bug in SVN or 
> buildbot and I'm not entirely sure what to do about it.  For these 
> affected bots, I can force a clobber by setting build revision to 0 
> and forcing a build.  This will rm -rf the source tree and check out a 
> new one.  Surely this should resolve any conflicts right?  But it 
> doesn't.  For example:
>
> In this build I clobber the source tree.  You can see by clicking the 
> svn-llvm step.  You can see that all tests pass.
> http://lab.llvm.org:8011/builders/llvm-hexagon-elf/builds/9746
>
> The next build, 9747 also passes and if you click the svn-llvm step 
> you'll see that it just updates some files.
>
> But all of a sudden, 9749 fails.  If you look at the svn-llvm step 
> again, you see this:
There are two buildslaves hooked up to that builder, and those two 
builds did not run on the same slave.

-Eli

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux
Foundation Collaborative Project

On llvm-hexagon-elf, the two builds you mention actually came from two 
different machines.  One of them fails consistently, the other one 
works.  Since the builds are dispatched to whichever one is free, it may 
appear that they fail randomly.  Let me see if I can fix it, if not, 
I'll complain to someone who can.

-Krzysztof

On 9/19/2017 8:33 PM, Zachary Turner via llvm-dev wrote:
> Hi all, for the past several days I've been dealing with intermittent 
> bot failures due to a recent series of patches which changes some things 
> about lit.
> 
> Most of the dust has settled, but there's several bots that just
won't
> stay green.
> 
> An example is this one:
> 
> http://lab.llvm.org:8011/builders/llvm-hexagon-elf
> 
> What happens in all of these cases is that SVN encounters a tree 
> conflict when updating, and then it gets stuck on some messed up of the 
> code where the test suite doesn't match up with the code being tested.
> 
> But at this point I honestly think this is some kind of bug in SVN or 
> buildbot and I'm not entirely sure what to do about it.  For these 
> affected bots, I can force a clobber by setting build revision to 0 and 
> forcing a build.  This will rm -rf the source tree and check out a new 
> one.  Surely this should resolve any conflicts right?  But it doesn't.
> For example:
> 
> In this build I clobber the source tree.  You can see by clicking the 
> svn-llvm step.  You can see that all tests pass.
> http://lab.llvm.org:8011/builders/llvm-hexagon-elf/builds/9746
> 
> The next build, 9747 also passes and if you click the svn-llvm step 
> you'll see that it just updates some files.
> 
> But all of a sudden, 9749 fails.  If you look at the svn-llvm step 
> again, you see this:
> 
> Updating '.':
> Skipped 'utils/lit/lit/llvm' -- Node remains in conflict
> 
> and now everything starts failing.
> 
> But it shouldn't even be trying to update this!   No file in that 
> directory has changed.  If you look at 9747, 9748, and 9749, none of 
> those CLs touch any files in that folder.
> 
> So, I'm really at a loss here.  I don't think reverting is going to
fix
> anything, because I believe the damage to be related to some 
> intermediate commit, and the revert is just going to get stacked onto 
> the end of the commit list.  Something seems fundamentally messed up here.
> 
> Can anyone provide some insight?
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

It seems like everything is getting back to normal?  (Disregarding the fact
that every bot is currently red, but at least it's an actual bug in
someone's patch this time)

Thanks

On Wed, Sep 20, 2017 at 5:27 AM Krzysztof Parzyszek via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On llvm-hexagon-elf, the two builds you mention actually came from two
> different machines.  One of them fails consistently, the other one
> works.  Since the builds are dispatched to whichever one is free, it may
> appear that they fail randomly.  Let me see if I can fix it, if not,
> I'll complain to someone who can.
>
> -Krzysztof
>
> On 9/19/2017 8:33 PM, Zachary Turner via llvm-dev wrote:
> > Hi all, for the past several days I've been dealing with
intermittent
> > bot failures due to a recent series of patches which changes some
things
> > about lit.
> >
> > Most of the dust has settled, but there's several bots that just
won't
> > stay green.
> >
> > An example is this one:
> >
> > http://lab.llvm.org:8011/builders/llvm-hexagon-elf
> >
> > What happens in all of these cases is that SVN encounters a tree
> > conflict when updating, and then it gets stuck on some messed up of
the
> > code where the test suite doesn't match up with the code being
tested.
> >
> > But at this point I honestly think this is some kind of bug in SVN or
> > buildbot and I'm not entirely sure what to do about it.  For these
> > affected bots, I can force a clobber by setting build revision to 0
and
> > forcing a build.  This will rm -rf the source tree and check out a new
> > one.  Surely this should resolve any conflicts right?  But it
doesn't.
> > For example:
> >
> > In this build I clobber the source tree.  You can see by clicking the
> > svn-llvm step.  You can see that all tests pass.
> > http://lab.llvm.org:8011/builders/llvm-hexagon-elf/builds/9746
> >
> > The next build, 9747 also passes and if you click the svn-llvm step
> > you'll see that it just updates some files.
> >
> > But all of a sudden, 9749 fails.  If you look at the svn-llvm step
> > again, you see this:
> >
> > Updating '.':
> > Skipped 'utils/lit/lit/llvm' -- Node remains in conflict
> >
> > and now everything starts failing.
> >
> > But it shouldn't even be trying to update this!   No file in that
> > directory has changed.  If you look at 9747, 9748, and 9749, none of
> > those CLs touch any files in that folder.
> >
> > So, I'm really at a loss here.  I don't think reverting is
going to fix
> > anything, because I believe the damage to be related to some
> > intermediate commit, and the revert is just going to get stacked onto
> > the end of the commit list.  Something seems fundamentally messed up
> here.
> >
> > Can anyone provide some insight?
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> --
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by The Linux Foundation
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170920/ecaf779a/attachment.html>