llvm dev - Dec 2021 - [cfe-dev] Bugzilla migration is stopped again

Hello
> 1) we already migrated in 1300 issues
Way more – 13k.
> 3) its not possible to remove the existing issues and start again
Right
> 4) so if any of the links are wrong in the 1300 then we can't do
anything with them other than correct by hand? (is that correct?)
Likely. GitHub engineers are still investigating what are the options here.
> 5) GitHub say they don't recommend post-migration writing? Do they mean
they don't recommend using an api to do that? Or doing it by hand?
Both actually. There are multiple concerns including notifications
that will be sent and the last changed time updated. Also, before
rewriting by ourselves we will need to build a map from bz id to
github id, so we will know what is the target issue id.
> 6) We can edit the comments by hand (can you only edit your own comments or
can we edit someone else's comments, I'm thinking its only our own based
on testing I've done with other repos)
Yes, only admins can edit everything.
> Assuming there is no obvious/immediate fix, Do we have any choice but to
move ahead with the existing import and fix the comments by hand retrospectively
(assuming 6)
This is what I asked GitHub engineers. They essentially asked for yet
another day to figure out the possible options. My rough estimate that
at least 5k issues will have broken links.

-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University

On Tue, Dec 7, 2021 at 7:33 AM Anton Korobeynikov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
>
> > 6) We can edit the comments by hand (can you only edit your own
comments
> or can we edit someone else's comments, I'm thinking its only our
own based
> on testing I've done with other repos)
> > - isn't this a requirement in order to fix up the
"code-blocks"?
>

Yes, only admins can edit everything.
>
I noticed this yesterday with the existing test migration: compare
https://bugs.llvm.org/show_bug.cgi?id=52598
versus
https://github.com/llvm/llvm-bugzilla-archive/issues/52598

The current script seems to be forgetting that GitHub issues use Markdown,
and so every existing Bugzilla comment needs to be wrapped in
triple-backticks to preserve its semantics.
(You could do *cleverer* things, like "don't wrap comments that are
only
one line long," but doing anything *less-clever* will be a non-starter.)
> Assuming there is no obvious/immediate fix, Do we have any choice but to
> move ahead with the existing import and fix the comments by hand
> retrospectively (assuming 6)
> This is what I asked GitHub engineers. They essentially asked for yet
> another day to figure out the possible options. My rough estimate that
> at least 5k issues will have broken links.

Anton: I see about 35,000 issues in
https://github.com/llvm/llvm-bugzilla-archive/issues
but only 228 (i.e. essentially none, presumably just historical noise from
newbie GitHub users) in
https://github.com/llvm/llvm-project/issues
Where are the 13,000 issues you are saying have already been migrated?

IIUC, it's *very fortunate* that there aren't yet 13,000 issues in
https://github.com/llvm/llvm-project/issues .  That means that it is still
an option to do a "practice" migration into a test repo — e.g.,
https://github.com/llvm/llvm-bugzilla-archive2 (and then if it works as
intended, you can either "blow away
https://github.com/llvm/llvm-bugzilla-archive and rename
https://github.com/llvm/llvm-bugzilla-archive2 to
https://github.com/llvm/llvm-bugzilla-archive", or "blow away
https://github.com/llvm/llvm-bugzilla-archive and repeat the migration just
to prove it works *reproducibly*".
Only once the whole migration has been tested end-to-end on a test repo,
would I recommend starting the migration into the production repo
https://github.com/llvm/llvm-project.

Thanks for the links to https://github.com/llvm/bugzilla2gitlab/tree/llvm
 and
https://docs.google.com/document/d/1G6DZ6AxzSaOlrtTxoxtqYKnD4Myv40QfKK4wj54y8ms/edit
 .
Those make it clear that someone's done a little bit of work to script this
stuff; but the Google Doc also makes it clear that there is a long way to
go to accomplish a "deploy plan": someone needs to take that English
description and turn it into code (Python or even Bash or whatever) that
can be
(A) reviewed for correctness, without running it
(B) run multiple times with guaranteed same behavior, with no risk that
some human will accidentally forget a step in the middle

Step 1, getting the XML files from Bugzilla, turns out to be super easy
because there's a public API for that:
https://github.com/Quuxplusone/BugzillaToGithub
Step 3, transforming XML to GitHub's JSON schema, requires knowing what
GitHub's schema looks like. I've found
https://gist.github.com/jonmagic/5282384165e0f86ef105#start-an-issue-import
although it's not real clear what the schema is or if that even still works
(I haven't tried yet). Also, there seems to be no way for one GitHub user
to create a comment or issue putatively authored by some other GitHub user.
(Which certainly makes sense.) So this would result in issues and comments
filed by "LLVM Import Bot" or whatever... but I think that's fine,
and
might even avoid some issues that you'd have otherwise, with scenarios like
"Joe User created his GitHub account in 2015, but was making comments on
LLVM issues back in 2012."

Vice versa, btw, you've currently got some issues being incorrectly
imported with the reporter listed* in the issue summary itself* as "LLVM
Bugzilla Contributor"; e.g. this one from Chris Burel.
https://github.com/llvm/llvm-bugzilla-archive/issues/52567
It certainly makes sense that you won't have a GitHub *username* for some
people, but you still shouldn't throw away the information about their
human name just because we're migrating from one platform to another.

–Arthur
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211207/7c8fa681/attachment.html>

> > 4) so if any of the links are wrong in the 1300 then we can't do
anything with them other than correct by hand? (is that correct?)
> Likely. GitHub engineers are still investigating what are the options here.
I've been told that the bug was fixed at the GitHub side. After
checking we might proceed with the migration.

-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University