llvm dev - Dec 2021 - Status of Bugzilla Migration

Dear All,

Some of you who are checking the migration notes
(https://bit.ly/3HVjr7a) might already have noticed that we're stuck
again. Let me provide more information about what is going on now and
what the plans are.

As a reminder, previously we imported all issues in the archive repo
and essentially the very last step remained: migration to the live
llvm-project repo. This step is crucial and one-way, once started we
cannot undo the steps we'd made. We also have to rely on GitHub here
as we cannot do it via rate-limited API calls

During the final checks two issues were revealed:
  - Notifications are still sent in some cases
  - Migration sets the last modification date of the closed issues (it
looks like it was implemented like "re-open issue, transfer and close
again"). As a result, all closed issues essentially got sorted
chronologically before the real open ones.

These issues were fixed at GitHub side and we proceeded with
re-checking everything. It turned out that another issue appeared: the
labels were silently lost and the migrated issues were completely
labelless, despite being annotated by 140+ labels we had originally.
For now this is a show-stopper issue. The issue was reported and
acknowledged by GitHub, however, not ETA was provided.

Our current options are:
  1. Abandon the migration
  2. Wait until the issue is resolved on GitHub side
  3. Try to find alternative solutions to workaround GitHub issue

2. is essentially not an option. I am proposing to abandon the
migration and unlock the bugzilla if the solution will not be found by
the end of this week.

The only alternative I'm seeing is to apply the labels post-migration.
There are important downsides:
  - This has to be done via GitHub API and we're rate limited to ~5000
requests per hour, so this means that the labelling will take ~20
hours. I was told that there is no way for us to have the API rate
limit increased.
  - This might trigger notifications. My quick check via web ui does
not, but I cannot be 100% with anything here
  - (the most important) This will screw the "last modified" timestamp
as label setting is an event that is recorded in the issue. There is
no way to set some "old" timestamp, it is assigned by GitHub
automatically.

 For now I'm testing the script for 3. and waiting for any news from GitHub.

I will keep you updated.

-- 
With best regards, Anton Korobeynikov
On behalf of LLVM Foundation

What bad stuff happens if you just open up
https://github.com/llvm/llvm-bugzilla-archive/issues (even if you then make
another historical archive later) to use as the bug tracker until you and
github have ironed out all the migration from one project to another
project issues? rather than going all the way back to bugzillia which is
then going to impose some other multi day migration at a later point.

In my mind I've already divorced from bugzilla, I'm ready to move on
with
my life with github!

MyDeveloperDay

On Thu, Dec 2, 2021 at 7:36 AM Anton Korobeynikov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Dear All,
>
> Some of you who are checking the migration notes
> (https://bit.ly/3HVjr7a) might already have noticed that we're stuck
> again. Let me provide more information about what is going on now and
> what the plans are.
>
> As a reminder, previously we imported all issues in the archive repo
> and essentially the very last step remained: migration to the live
> llvm-project repo. This step is crucial and one-way, once started we
> cannot undo the steps we'd made. We also have to rely on GitHub here
> as we cannot do it via rate-limited API calls
>
> During the final checks two issues were revealed:
>   - Notifications are still sent in some cases
>   - Migration sets the last modification date of the closed issues (it
> looks like it was implemented like "re-open issue, transfer and close
> again"). As a result, all closed issues essentially got sorted
> chronologically before the real open ones.
>
> These issues were fixed at GitHub side and we proceeded with
> re-checking everything. It turned out that another issue appeared: the
> labels were silently lost and the migrated issues were completely
> labelless, despite being annotated by 140+ labels we had originally.
> For now this is a show-stopper issue. The issue was reported and
> acknowledged by GitHub, however, not ETA was provided.
>
> Our current options are:
>   1. Abandon the migration
>   2. Wait until the issue is resolved on GitHub side
>   3. Try to find alternative solutions to workaround GitHub issue
>
> 2. is essentially not an option. I am proposing to abandon the
> migration and unlock the bugzilla if the solution will not be found by
> the end of this week.
>
> The only alternative I'm seeing is to apply the labels post-migration.
> There are important downsides:
>   - This has to be done via GitHub API and we're rate limited to ~5000
> requests per hour, so this means that the labelling will take ~20
> hours. I was told that there is no way for us to have the API rate
> limit increased.
>   - This might trigger notifications. My quick check via web ui does
> not, but I cannot be 100% with anything here
>   - (the most important) This will screw the "last modified"
timestamp
> as label setting is an event that is recorded in the issue. There is
> no way to set some "old" timestamp, it is assigned by GitHub
> automatically.
>
>  For now I'm testing the script for 3. and waiting for any news from
> GitHub.
>
> I will keep you updated.
>
> --
> With best regards, Anton Korobeynikov
> On behalf of LLVM Foundation
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211202/977706b8/attachment.html>

On Thu, Dec 2, 2021 at 2:36 AM Anton Korobeynikov via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
>
> Dear All,
>
> Some of you who are checking the migration notes
> (https://bit.ly/3HVjr7a) might already have noticed that we're stuck
> again. Let me provide more information about what is going on now and
> what the plans are.
>
> As a reminder, previously we imported all issues in the archive repo
> and essentially the very last step remained: migration to the live
> llvm-project repo. This step is crucial and one-way, once started we
> cannot undo the steps we'd made. We also have to rely on GitHub here
> as we cannot do it via rate-limited API calls
>
> During the final checks two issues were revealed:
>   - Notifications are still sent in some cases
>   - Migration sets the last modification date of the closed issues (it
> looks like it was implemented like "re-open issue, transfer and close
> again"). As a result, all closed issues essentially got sorted
> chronologically before the real open ones.
>
> These issues were fixed at GitHub side and we proceeded with
> re-checking everything. It turned out that another issue appeared: the
> labels were silently lost and the migrated issues were completely
> labelless, despite being annotated by 140+ labels we had originally.
> For now this is a show-stopper issue. The issue was reported and
> acknowledged by GitHub, however, not ETA was provided.
>
> Our current options are:
>   1. Abandon the migration
>   2. Wait until the issue is resolved on GitHub side
>   3. Try to find alternative solutions to workaround GitHub issue
>
> 2. is essentially not an option. I am proposing to abandon the
> migration and unlock the bugzilla if the solution will not be found by
> the end of this week.
Thank you for all of the hard work you've put into this so far, and
thank you for the detailed update on the unfortunate place we're at.

When you say "abandon the migration", do you mean temporarily or
permanently? I'd be strongly in favor of temporarily abandoning the
migration so that we can continue to do useful work against bugs while
we sort this out. If you're thinking of abandoning permanently, I
could be in support of that as well, but I'd want to know what our
aspirational goals are for the bug database long-term before giving my
support.

~Aaron
>
> The only alternative I'm seeing is to apply the labels post-migration.
> There are important downsides:
>   - This has to be done via GitHub API and we're rate limited to ~5000
> requests per hour, so this means that the labelling will take ~20
> hours. I was told that there is no way for us to have the API rate
> limit increased.
>   - This might trigger notifications. My quick check via web ui does
> not, but I cannot be 100% with anything here
>   - (the most important) This will screw the "last modified"
timestamp
> as label setting is an event that is recorded in the issue. There is
> no way to set some "old" timestamp, it is assigned by GitHub
> automatically.
>
>  For now I'm testing the script for 3. and waiting for any news from
GitHub.
>
> I will keep you updated.
>
> --
> With best regards, Anton Korobeynikov
> On behalf of LLVM Foundation
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

Thanks for all the work and info, Anton. Based on your writeup, I think
option 3 is best.

Losing the last update timestamps on all the issues is unfortunate, but I
think it's OK. We already know the migration doesn't have perfect
fidelity,
and that's OK.

I also think we can wait a day to get labels on the migrated issues. I
think my bigger concern with the rate-limited APIs is that it's hard to
test scripts that take 20 hours to run, so there is some risk that the
label migration script fails or mislabels issues. Still, I would just hope
for the best here. It's not critical to get labels on old issues on day 1.
Maybe one way to deal with this is to apply labels to recently modified
issues first.

Notifications are concerning, but your test via the web UI gives me enough
confidence to want to push forward.

Finally, you are sort of the one in the hot seat here doing the work, so I
favor any solution that takes the pressure off you. :) That means either
going back to bugzilla temporarily, or moving forward with the migration
and fixing the labels as best we can over time.

On Wed, Dec 1, 2021 at 11:37 PM Anton Korobeynikov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Dear All,
>
> Some of you who are checking the migration notes
> (https://bit.ly/3HVjr7a) might already have noticed that we're stuck
> again. Let me provide more information about what is going on now and
> what the plans are.
>
> As a reminder, previously we imported all issues in the archive repo
> and essentially the very last step remained: migration to the live
> llvm-project repo. This step is crucial and one-way, once started we
> cannot undo the steps we'd made. We also have to rely on GitHub here
> as we cannot do it via rate-limited API calls
>
> During the final checks two issues were revealed:
>   - Notifications are still sent in some cases
>   - Migration sets the last modification date of the closed issues (it
> looks like it was implemented like "re-open issue, transfer and close
> again"). As a result, all closed issues essentially got sorted
> chronologically before the real open ones.
>
> These issues were fixed at GitHub side and we proceeded with
> re-checking everything. It turned out that another issue appeared: the
> labels were silently lost and the migrated issues were completely
> labelless, despite being annotated by 140+ labels we had originally.
> For now this is a show-stopper issue. The issue was reported and
> acknowledged by GitHub, however, not ETA was provided.
>
> Our current options are:
>   1. Abandon the migration
>   2. Wait until the issue is resolved on GitHub side
>   3. Try to find alternative solutions to workaround GitHub issue
>
> 2. is essentially not an option. I am proposing to abandon the
> migration and unlock the bugzilla if the solution will not be found by
> the end of this week.
>
> The only alternative I'm seeing is to apply the labels post-migration.
> There are important downsides:
>   - This has to be done via GitHub API and we're rate limited to ~5000
> requests per hour, so this means that the labelling will take ~20
> hours. I was told that there is no way for us to have the API rate
> limit increased.
>   - This might trigger notifications. My quick check via web ui does
> not, but I cannot be 100% with anything here
>   - (the most important) This will screw the "last modified"
timestamp
> as label setting is an event that is recorded in the issue. There is
> no way to set some "old" timestamp, it is assigned by GitHub
> automatically.
>
>  For now I'm testing the script for 3. and waiting for any news from
> GitHub.
>
> I will keep you updated.
>
> --
> With best regards, Anton Korobeynikov
> On behalf of LLVM Foundation
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211202/35b97cfc/attachment-0001.html>

>- This has to be done via GitHub API and we're rate limited to ~5000
requests per hour, so this means that the labelling will take ~20
hours. I was told that there is no way for us to have the API rate
limit increased.

This 5000 request per hour limit, is that per repo or per access token? Could we
potentially make a pool access token from multiple github accounts to sidestep
the issue? Say 20 tokens to do the migration in 1 hour?

--Jeff Miller
-------- Original Message --------
On Dec 1, 2021, 11:36 PM, Anton Korobeynikov via cfe-dev wrote:
> Dear All,
>
> Some of you who are checking the migration notes
> (https://bit.ly/3HVjr7a) might already have noticed that we're stuck
> again. Let me provide more information about what is going on now and
> what the plans are.
>
> As a reminder, previously we imported all issues in the archive repo
> and essentially the very last step remained: migration to the live
> llvm-project repo. This step is crucial and one-way, once started we
> cannot undo the steps we'd made. We also have to rely on GitHub here
> as we cannot do it via rate-limited API calls
>
> During the final checks two issues were revealed:
> - Notifications are still sent in some cases
> - Migration sets the last modification date of the closed issues (it
> looks like it was implemented like "re-open issue, transfer and close
> again"). As a result, all closed issues essentially got sorted
> chronologically before the real open ones.
>
> These issues were fixed at GitHub side and we proceeded with
> re-checking everything. It turned out that another issue appeared: the
> labels were silently lost and the migrated issues were completely
> labelless, despite being annotated by 140+ labels we had originally.
> For now this is a show-stopper issue. The issue was reported and
> acknowledged by GitHub, however, not ETA was provided.
>
> Our current options are:
> 1. Abandon the migration
> 2. Wait until the issue is resolved on GitHub side
> 3. Try to find alternative solutions to workaround GitHub issue
>
> 2. is essentially not an option. I am proposing to abandon the
> migration and unlock the bugzilla if the solution will not be found by
> the end of this week.
>
> The only alternative I'm seeing is to apply the labels post-migration.
> There are important downsides:
> - This has to be done via GitHub API and we're rate limited to ~5000
> requests per hour, so this means that the labelling will take ~20
> hours. I was told that there is no way for us to have the API rate
> limit increased.
> - This might trigger notifications. My quick check via web ui does
> not, but I cannot be 100% with anything here
> - (the most important) This will screw the "last modified"
timestamp
> as label setting is an event that is recorded in the issue. There is
> no way to set some "old" timestamp, it is assigned by GitHub
> automatically.
>
> For now I'm testing the script for 3. and waiting for any news from
GitHub.
>
> I will keep you updated.
>
> --
> With best regards, Anton Korobeynikov
> On behalf of LLVM Foundation
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211202/bb0d3ca6/attachment.html>

On Wed, Dec 1, 2021 at 11:36 PM Anton Korobeynikov via flang-dev <
flang-dev at lists.llvm.org> wrote:
> Dear All,
>
> Some of you who are checking the migration notes
> (https://bit.ly/3HVjr7a) might already have noticed that we're stuck
> again. Let me provide more information about what is going on now and
> what the plans are.
>
> As a reminder, previously we imported all issues in the archive repo
> and essentially the very last step remained: migration to the live
> llvm-project repo. This step is crucial and one-way, once started we
> cannot undo the steps we'd made. We also have to rely on GitHub here
> as we cannot do it via rate-limited API calls
>
> During the final checks two issues were revealed:
>   - Notifications are still sent in some cases
>   - Migration sets the last modification date of the closed issues (it
> looks like it was implemented like "re-open issue, transfer and close
> again"). As a result, all closed issues essentially got sorted
> chronologically before the real open ones.
>
> These issues were fixed at GitHub side and we proceeded with
> re-checking everything. It turned out that another issue appeared: the
> labels were silently lost and the migrated issues were completely
> labelless, despite being annotated by 140+ labels we had originally.
> For now this is a show-stopper issue. The issue was reported and
> acknowledged by GitHub, however, not ETA was provided.
>
> Our current options are:
>   1. Abandon the migration
>   2. Wait until the issue is resolved on GitHub side
>   3. Try to find alternative solutions to workaround GitHub issue
>
> 2. is essentially not an option. I am proposing to abandon the
> migration and unlock the bugzilla if the solution will not be found by
> the end of this week.
>
Seems reasonable to me!

>
> The only alternative I'm seeing is to apply the labels post-migration.
> There are important downsides:
>   - This has to be done via GitHub API and we're rate limited to ~5000
> requests per hour, so this means that the labelling will take ~20
> hours. I was told that there is no way for us to have the API rate
> limit increased.
>   - This might trigger notifications. My quick check via web ui does
> not, but I cannot be 100% with anything here
>   - (the most important) This will screw the "last modified"
timestamp
> as label setting is an event that is recorded in the issue. There is
> no way to set some "old" timestamp, it is assigned by GitHub
> automatically.
>
>  For now I'm testing the script for 3. and waiting for any news from
> GitHub.
>
Thanks for the work :)
I hope you can get your script working!
Maybe if you can share this on a public repo, others here can help to do
small test runs in private forks and cross-validate or help fix issues with
it?

-- 
Mehdi


>
> I will keep you updated.
>
> --
> With best regards, Anton Korobeynikov
> On behalf of LLVM Foundation
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211202/b35cb015/attachment.html>