llvm dev - Dec 2021 - [cfe-dev] [Openmp-dev] Bugzilla migration is stopped again

On Sat, Dec 4, 2021 at 5:46 AM Anton Korobeynikov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> [...]
> * Surely*, if the community will re-decide that these are unimportant
> things
> *we can push the existing code into a blank archive fairly quickly*.
>
Please, test the above claim this week, *on a blank repo*.  Let's actually
find out whether it works, instead of relying on "Surely...".

At this point I'm offering my own technical assistance, just to get the
thing done and stop getting these emails every day. Send me your Bugzilla
export script; I'll test it out this week on a blank repo, with the goal of
mirroring a 100-bug subset of the LLVM Bugzilla publicly visible in
https://github.com/Quuxplusone/LLVMBugzillaTest/ by EOW.

(Credentials: I was SRE at Mixpanel for ~3 years and performed several
100GB cluster migrations with zero downtime. I *have* seen the "We'll
do it
live!" attitude be successful, but I have also seen it fail spectacularly.
The alternative "plan it carefully, write down your deploy plan, test what
can be tested ahead of time, do a practice run, *then* do it live" approach
usually works better. At this point it looks like Anton's initial pass at
"We'll do it live!" clearly was *not* successful, in the sense
that if it
*were* successful the repo would have been migrated circa Thanksgiving
weekend. So this is the giant-honking-red-flashing-light alert that it's
time to shift from "We'll do it live!" to "Let's make a
deploy plan.")

Respectfully, yet frustrated with the never-ending email thread,
–Arthur
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211204/0680adc0/attachment.html>

Dear Arthur,
> Respectfully, yet frustrated with the never-ending email thread,
I understand your frustration and please rest assured that my own
frustration is certainly not less than yours. I'm also very exhausted
at the moment as the things are beyond my control. The constant
pushing from this and similar emails does not help in resolving the
situation. I certainly have to note that your accusations in "we'll do
it live section" are not quite accurate in many aspects - if you have
not seen the outcome of test imports, then it does not mean that there
were none. I would say even more, this means that they were successful
as nothing triggered excessive notifications (we made such a mistake
once – you could even find the reports of this in the MLs) . For your
information: the last "dry run import" which gated the migration was
14th full (52k issues) try. In all previous runs issues were found and
either a workaround was prepared or they were reported to GitHub.

Now let's proceed from the emotions to the real things. I do
appreciate your willingness to provide the help. Please see my notes
below.
> I'll test it out this week on a blank repo, with the goal of mirroring
a 100-bug subset of the LLVM Bugzilla publicly visible in
https://github.com/Quuxplusone/LLVMBugzillaTest/ by EOW.
First of all, proof-of-concept should certainly be 10k issues on
non-blank repo. It should already have closed issues / pull requests
in order to represent the real llvm-project repo.

Now down to details:
0. I would suggest you not to use the GitHub API. YMMV, but from our
experience: API is rate limited, and many things are outside your
control including:
  - ids
  - timestamps
  - notifications
1. The real migration starts from a local gitlab instance, where you
import all bugzilla issues. You can certainly skip this step in your
own experiments and proceed directly to step 2, but this will allow
you to check the outcome of the import. The script we used could be
found at https://github.com/llvm/bugzilla2gitlab/tree/llvm

2. Then you need to prepare the dump which could be consumed by GitHub
Enterprise Migration API:
https://docs.github.com/en/rest/reference/migrations We are using
gitlab-to-github scripts provided by GitHub. I'm not sure I can share
them as they are not public – I will ask GH support engineers on
Monday and will return to you.

3. After the dump is prepared you need to upload it via GitHub
Enterprise Migration API. Note that import is only possible into empty
repo (it is essentially created). If the import failed you'd need to
ask GitHub engineers whether the error is real or whether it could be
ignored. If the error is real, then you'd likely need to restart from
scratch – it is possible to resume, but practice shows that this might
create duplicate comments.

4. After the import finished check the results: number of objects
(issues, comments, attachments) that were imported. If there are any
objects that failed import, then you need to figure out which ones and
what to do. Your options are: ignore or restart the import. Here is
the checklist I'm using for the content:
https://docs.google.com/document/d/1G6DZ6AxzSaOlrtTxoxtqYKnD4Myv40QfKK4wj54y8ms/edit

5. At this point one should have something similar to
https://github.com/llvm/llvm-bugzilla-archive

6. In order to transfer issues from the archive to the live repo there
are two options:
  - Use GitHub rate-limited API
  - Ask GitHub folks
The first variant triggers notifications to everyone mentioned,
assigned or commented on the issue. There is no way to silence these
notifications.
In our case here we are relying on GitHub support engineers that do
this migration step for us. There is no API, no script, nothing that
is within our control. We did several test migrations from dry-run
repo to another repo (and this is how we found all bugs wrt issue
transfer in the past). As I already said, the circular reference
rewriting was not included into my original checklist - I expected
that this feature "just works" and was only spotted later.

Hope this helps. Should you have more questions, I will certainly be
happy to help you. I'm interested in finishing this 2+ year project
more than anyone else.

-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University

On Sat, Dec 4, 2021 at 9:16 AM Arthur O'Dwyer <arthur.j.odwyer at
gmail.com>
wrote:
> On Sat, Dec 4, 2021 at 5:46 AM Anton Korobeynikov via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> [...]
>> * Surely*, if the community will re-decide that these are unimportant
>> things
>> *we can push the existing code into a blank archive fairly quickly*.
>>
>
> Please, test the above claim this week, *on a blank repo*.  Let's
> actually find out whether it works, instead of relying on
"Surely...".
>
> At this point I'm offering my own technical assistance, just to get the
> thing done and stop getting these emails every day. Send me your Bugzilla
> export script; I'll test it out this week on a blank repo, with the
goal of
> mirroring a 100-bug subset of the LLVM Bugzilla publicly visible in
> https://github.com/Quuxplusone/LLVMBugzillaTest/ by EOW.
>
The promised EOW update: I have written Python scripts for the Export,
Transform, and (dumbed-down, see below) Load stages of a bugzilla-to-github
migration. You can find them at
https://github.com/Quuxplusone/BugzillaToGithub#bugzilla-to-github
and the resulting GitHub issues list (which is just partial, so far) lives
at
https://github.com/Quuxplusone/LLVMBugzillaTest/issues
This is merely the result of five evenings of work, so e.g. the formatting
of message bodies still isn't perfect, and as of this morning I'm aware
of
at least one bug (that GitHub's import API doesn't like a comment to
have *empty
string* as its `body`). And of course the biggest issue is that I was
noodling around without special access to GitHub staff, who are the only
people able to forge issue/comment authorship; so my script just puts
everything under the username of the person-or-bot that runs it.  I
guarantee GitHub SRE can help with that.

Arthur
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211210/1c03c6b1/attachment.html>