Pkg xen devel - Aug 2018 - git workflow, redux

Summary:

I have tried the packaging-only repo and I really don't like it at
all.  I don't know how anyone copes with this - such hard work!
IMO we should switch to git-debrebase.  (As an alternative, if
you don't trust git-debrebase because it's my own tool, gbp pq
would be better, too, even though it's not as good as git-debrebase.)

Particularly, now that we have more people making substantial
contributions, it is important to have something that works for as
many as possible, and that is low friction for common tasks.


Knorrie wrote on irc:

  ... I only want to discuss it if you have packaging scenarios and
  want to do them in both ways and then discuss the result, so we can
  discuss based on actual work/commands/etc instead of feelings

Well, I don't propose to actually try doing the same task two
different ways, at least unless I completely fail to manage it the
first way I try.

But I will consider some scenarios:

                    packaging-only      git-debrebase

  Build binaries    100 lines of        dgit sbuild -A
  for release       instructions!         or
                    + pbuilder/sbuild   dgit pbuilder [options]

  Build binaries    ??? no idea         dpkg-buildpackage -uc -b
  ad hoc for         README.source
  testing            dirties the tree

  New upstream      ??? no idea how     # choose VERSION and COMMIT
  rebase              to do this        git tag upstream/VERSION COMMIT
                    fix up conflicts    git debrebase new-upstream VERSION
                     in quilt omg!      # fix up conflicts in git
                                        # edit changelog at some point

  Add a patch       obtain tree         # edit upstream files
                    with upstream src   git commit
                    do quilt stuff 

  Amend a patch     obtain tree         git debrebase -i
                    with upstream src   # is just like git-rebase
                    do quilt stuff

  Drop a patch      edit series or      git debrebase -i
                     soemthing ?        # is just like git-rebase

  Cherry pick       git format-patch    git cherry-pick
  from upstream     edit series?
                    how to check it
                    applies, or build?

  Update d/control  edit control.in     # edit control.in
                    run rules to        debian/rules debian/control
                     see output,        git commit debian/control
                    debdifff?           git diff

  git blame on      not possible        git blame
   upstream source

  machinery needed  orig targets        README.source saying
   in tree          genorig             `see dgit-maint-debrebase(7)'
                    README.source       and mentioning d/control.in

  push to salsa     check that          git debrebase prepush
                     patches apply?     # ^ won't fail
                    git push            git push

  uploading         100 lines of        git deborig
                    pratting about      dgit push-source
                    debsign, dput       (or dgit push if NEW)

You'll observe that the left hand column contains references to
documentation.  It also contains some ??? because I haven't bothered
to RTFM quilt recently.  (I used it once, a long time ago.  We have
much better tools now.)

You'll see that the right hand column is wider and longer.  That
is because it contains the complete actual command lines.


In fact, when I was doing the upload to experimental just now, I so
much missed git that I did the following:
  * Get the relevant branch and tag from salsa
  * Follow the instructions in README.source.md to generate a
    source package
  * dgit import-dsc
  * lots of git diff (sadly this procedure produces a not very
     helpful history)
  * dgit sbuild -A
  * dgit push



I'm going to quote some snippets from irc:

17:17 <babilen> My take on this is, is that I'm more than confident to
  deal with any security issue that arises in the future with the
  packaging style used for 4.8 in the stretch-security uploads

Likewise.  (I'd like to take this chance to publicly thank Wolodja for
his work on the stretch security updates.)


17:27 <Knorrie> also, a relevant question is whether we want to have a
  patch-burden-heavy workflow in debian stable, or if we could just
  follow the upstream stable branch, like the kernel team does with
  stable kernel releases

I don't think it is realistic to expect the situation to be different
in buster than it was in stretch.  When I uploaded 4.8 to stretch, it
was an upstream RC without patches.  The way upstream provide security
patches means we sometimes have to take upstream stable branches plus
some patches from advisories; sometimes upstream unstable branches (!)
and sometimes some unholy mixture.

It's awkward but git-debrebase makes this reasonably straightforward.
Doing this stuff with quilt would be nearly unworkable.


23:41 <Knorrie> babilen: doing stable security updates for a single
  xen version is a whole different flow than organizing a rolling
  release (with stable-backports) which follows latest upstream
  release in unstable/testing (and upstream rc in experimental), while
  also allowing to easily charry-pick things or merge branches between
  them

I think git-debrebase is going to be easier for all these things than
the current approach.


Ian.

-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my
own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Hi team,

On 08/23/2018 08:07 PM, Ian Jackson wrote:
> Summary:
> 
> I have tried the packaging-only repo and I really don't like it at
> all.  I don't know how anyone copes with this - such hard work!
> IMO we should switch to git-debrebase.  (As an alternative, if
> you don't trust git-debrebase because it's my own tool, gbp pq
> would be better, too, even though it's not as good as git-debrebase.)
> 
> Particularly, now that we have more people making substantial
> contributions, it is important to have something that works for as
> many as possible, and that is low friction for common tasks.
Ack, especially the low friction part.

Xen is a broad product that can be used in many different ways, and bugs
that appear might not be easily reproducible (or fixes tested) by anyone
else than the reporter, having specific hardware and configurations.

The users of Xen in Debian are usually a bit more technical, since it's
not very easy to get everything running correctly. I suspect that when
someone in our user group runs into problems, they are able to not only
sit in a corner and wait until someone else fixes their specific
problem, but are willing to participate in getting it done.

The "onboarding experience" of working with the packaging is a crucial
thing here. It's preferable that a user can try out things himself,
without having to ask someone else to apply some patch and make a test
build all the time. When doing this, our goal should be that the amount
of time spent on things that are not executing the actual idea to
fix/improve/whatever for the Xen related thing is reduced to an absolute
minimum.

So, yes, that's also the 'as many as possible' part indeed.
> Knorrie wrote on irc:
> 
>   ... I only want to discuss it if you have packaging scenarios and
>   want to do them in both ways and then discuss the result, so we can
>   discuss based on actual work/commands/etc instead of feelings
> 
> Well, I don't propose to actually try doing the same task two
> different ways, at least unless I completely fail to manage it the
> first way I try.
Yes. What I mean is that in discussions that need to have a decision to
change things, or not, as result, I'm allergic to arguments that are
based on generalizations and feelings ("much more work!",
"hundreds of
things!", "so difficult!", "omg!", "but I'm
used to this"), because they
display a feeling instead of facts, and/or often display lack of
interest in any other opinion than ones current one.
> But I will consider some scenarios:
> 
>                     packaging-only      git-debrebase
> 
>   Build binaries    100 lines of        dgit sbuild -A
>   for release       instructions!         or
>                     + pbuilder/sbuild   dgit pbuilder [options]
The left column could be:

git clean -dfx; debian/rules orig; debian/rules debian/control; pdebuild
--use-pdebuild-internal --auto-debsign --debsign-k XXX --configfile
.pbuilderrc-sid

This is a oneliner, which can go into an alias, or just in the shell
history (ctrl-r!). This is not "100 lines of instructions" and a
README
is something that you only have to read once for background information
to understand why you are doing what you are doing.
>   Build binaries    ??? no idea         dpkg-buildpackage -uc -b
>   ad hoc for         README.source
>   testing            dirties the tree
What you express here, "it's dirty" is a "ugh! dirty!
ew!" feeling,
because you apparently do not like the fact that actual instructions
about what to do are needed at all.

Do realize that someone else who has to use tooling you wrote has to go
through the same process of "wtf! no idea how to do this! what is this
shit! ew!" first, while it's the most normal thing in the world to you
now.
>   New upstream      ??? no idea how     # choose VERSION and COMMIT
>   rebase              to do this        git tag upstream/VERSION COMMIT
>                     fix up conflicts    git debrebase new-upstream VERSION
>                      in quilt omg!      # fix up conflicts in git
>                                         # edit changelog at some point
A one line edit in debian/changelog is all, you're exaggerating again,
which is not helpful.

And yes, if there are conflicts (which I've only seen happening when
converting from 4.8 -> 4.9 -> 4.10 -> 4.11), doing this with quilt and
having a git diff that is a diff of diffs is absolutely suboptimal
compared to doing the same in "normal" git conflicts.
> 
>   Add a patch       obtain tree         # edit upstream files
>                     with upstream src   git commit
>                     do quilt stuff 
> 
>   Amend a patch     obtain tree         git debrebase -i
>                     with upstream src   # is just like git-rebase
>                     do quilt stuff
> 
>   Drop a patch      edit series or      git debrebase -i
>                      soemthing ?        # is just like git-rebase
git revert abcd (where the patch was added), and add some information in
the commit message why it can be dropped.
> 
>   Cherry pick       git format-patch    git cherry-pick
>   from upstream     edit series?
>                     how to check it
>                     applies, or build?
Yes, it looks like the most interesting part of this discussion is going
to be about patch management. The packaging-only approach with quilt and
detached patches is absolutely painful compared to using tooling that
has been written with the goal to exactly do that better.
>   Update d/control  edit control.in     # edit control.in
>                     run rules to        debian/rules debian/control
>                      see output,        git commit debian/control
>                     debdifff?           git diff
> 
>   git blame on      not possible        git blame
>    upstream source
Not possible? Well, not in the same repo no, but I can't imagine you
don't know how to do a git blame on the upstream source. Sorry, but this
does not remotely make sense at all.
> 
>   machinery needed  orig targets        README.source saying
>    in tree          genorig             `see dgit-maint-debrebase(7)'
>                     README.source       and mentioning d/control.in
Yes, the genorig and other really old python stuff is a dead horse that
is being dragged around. Instead of trying to rewrite it, it would be
preferable to use tooling that is well-maintained and have all of this
removed, since it's distracting from the actual work that should be
done, improving the Xen stuff.
>   push to salsa     check that          git debrebase prepush
>                      patches apply?     # ^ won't fail
>                     git push            git push
> 
>   uploading         100 lines of        git deborig
>                     pratting about      dgit push-source
>                     debsign, dput       (or dgit push if NEW)
> 
> You'll observe that the left hand column contains references to
> documentation.  It also contains some ??? because I haven't bothered
> to RTFM quilt recently.  (I used it once, a long time ago.  We have
> much better tools now.)
> 
> You'll see that the right hand column is wider and longer.  That
> is because it contains the complete actual command lines.
> 
> 
> In fact, when I was doing the upload to experimental just now, I so
> much missed git that I did the following:
>   * Get the relevant branch and tag from salsa
>   * Follow the instructions in README.source.md to generate a
>     source package
>   * dgit import-dsc
>   * lots of git diff (sadly this procedure produces a not very
>      helpful history)
>   * dgit sbuild -A
>   * dgit push
I'm not going to respond to these ramblings.
> I'm going to quote some snippets from irc:
> 
> 17:17 <babilen> My take on this is, is that I'm more than
confident to
>   deal with any security issue that arises in the future with the
>   packaging style used for 4.8 in the stretch-security uploads
> 
> Likewise.  (I'd like to take this chance to publicly thank Wolodja for
> his work on the stretch security updates.)
Woohoo! +1 Thanks!
> 17:27 <Knorrie> also, a relevant question is whether we want to have
a
>   patch-burden-heavy workflow in debian stable, or if we could just
>   follow the upstream stable branch, like the kernel team does with
>   stable kernel releases
> 
> I don't think it is realistic to expect the situation to be different
> in buster than it was in stretch.  When I uploaded 4.8 to stretch, it
> was an upstream RC without patches.  The way upstream provide security
> patches means we sometimes have to take upstream stable branches plus
> some patches from advisories; sometimes upstream unstable branches (!)
> and sometimes some unholy mixture.
> 
> It's awkward but git-debrebase makes this reasonably straightforward.
> Doing this stuff with quilt would be nearly unworkable.
Because of the complexity of the work that's flying by in the XSAs and
generally in the code, the thing I'm mostly interested in here is which
approach is the best to end up with a working result for everyone, which
is the thing that we call 'stable' in Debian.

Is it giving the stable-X branch to our users, because we know it passed
all the upstream tests? Or is it cobbling together our own collection of
changes?

Obviously, a patch rebase tool can make something apply and help it
compile again. But it does not detect that if you have patch X and Y and
not Z which seemed to be unrelated but actually breaks a corner case of
some user while the combination of them is stable-X upstream...

But, this is more about how upstream organizes things, how often stable
releases are tagged and there's more discussion about it upstream, which
should be participated in (about maturity level of the patch publishing
process etc), instead of discussing it here.

Also, I am fully aware of the fact that having someone actually working
at $upstream and also actively participating in our packaging is a
luxury. (Yay!) But, for doing the work, it's not optimal to sometimes
have to rely on this person spot invalid combinations of patches added.

Right here, I should be able to mention a specific case where this went
(almost) wrong. I can dig up things from IRC logs if anyone wants.
> 23:41 <Knorrie> babilen: doing stable security updates for a single
>   xen version is a whole different flow than organizing a rolling
>   release (with stable-backports) which follows latest upstream
>   release in unstable/testing (and upstream rc in experimental), while
>   also allowing to easily charry-pick things or merge branches between
>   them
> 
> I think git-debrebase is going to be easier for all these things than
> the current approach.
Ok, let's try it! Thanks a lot for doing the above writeup. After
filtering out the irrelevant stuff, I certainly see that it is better
for us and our users to change and spend time to invest in improving the
external tooling itself instead of improving the tooling inside this
packaging-only repo.

The comment about me not trusting the tools you wrote is irrelevant,
since I don't see any reason to do so at all.

How to proceed:

Can you provide me with some rtfm pointers and instructions about how
you would like to see things being transformed? Or would you like to do
it together? Do we want to do this before or after Sep 10?

I actually already have some questions about the git-debrebase behavior
to start with:

1: meta information

When writing software and git commits, my main incentive is to have a
future reader understand what I'm doing and why. I tend to put quite
some effort in this when I do things, not only because...

I'm also on the other side a lot and not being able to find information
in the history and read *why* something was done instead of *what* was
done, and I see how it hurts future development. This is a major waste
of time and blocking progress. Don't hesitate to ask me for examples.

The quilt workflow allows to add a patch and add (meta-) comments to the
git commit adding that patch, why it needed to be added etc. When
directly adding patches on git level, this is not possible. How is this
meta information stored for future readers? Does the reader have to
search in the debian/changelog and hope something is mentioned about it?
Is there something else? I'd like to see some solution/process for this,
instead of having the information in the head of the person who did it.

2: usage of git

One thing that is quite disturbing for me personally, is the result that
is left behind in a git repository when using git-debrebase. Instead of
having a meaningful history (which is my favourite thing to end up with,
something you can "browse" and read meaningful things, see above) git
seems to be used as some sort of ftp server, where a rm *; cp -a * is
done every time.

This results in the git history looking like a football-field wide rake,
listing the same patches over and over and over and over again, like the
history in the git-dpm repo that we have archived.

The relationship between patches and changes in them is not stored in
git history because every time things end up in new unrelated commits.

The history of the current master branch, doing the packaging from 4.8
via 4.9 and 4.10 to 4.11 and all the changes still does fit on your
screen. I like this. I can just read back what I did and why.

So my question is: how does debrebase care about future readers? How do
I quickly find out what *actual* changes have happened to the packaging
from the large commit blurb that I'm looking at? Does this need a README
with git commands to filter commits in debian/ or other things, or does
the debrebase tooling need to get features to analyze the git history
again to produce a useful overview of this information?

Knorrie

Responding separately here to the part about how security updates are
done (and in particular, what choices we make about what combination
of things from usptream to make into each update):

Hans van Kranenburg writes ("Re: git workflow,
redux"):
> On 08/23/2018 08:07 PM, Ian Jackson wrote:
> > I don't think it is realistic to expect the [security patch
> > application process] situation to be different in buster than it
> > was in stretch.  [...]
> > 
> > It's awkward but git-debrebase makes this reasonably
straightforward.
> > Doing this stuff with quilt would be nearly unworkable.
> 
> Because of the complexity of the work that's flying by in the XSAs and
> generally in the code, the thing I'm mostly interested in here is which
> approach is the best to end up with a working result for everyone, which
> is the thing that we call 'stable' in Debian.
That is precisely the same consideration Wolodja and I have been using
when deciding how to prepare each security update for stretch.
> Is it giving the stable-X branch to our users, because we know it passed
> all the upstream tests? Or is it cobbling together our own collection of
> changes?
That depends primarily on whether a stable-X branch is available with
the necessary fixes.  Often the right answer is to use the upstream
stable-X branch, but supplemented with patches from advisories.  That
is close to what the upstream security team have been reviewing,
testing, etc.
> But, this is more about how upstream organizes things, how often stable
> releases are tagged and there's more discussion about it upstream,
which
> should be participated in (about maturity level of the patch publishing
> process etc), instead of discussing it here.
This is a very active topic upstream.  Debian are not the only
downstream who are complaining that it is very awkward (and, as you
point out, that it can be risky).  We can hope that things will
improve.  But I wouldn't count on it.

Thanks,
Ian.

Hans van Kranenburg writes ("Re: git workflow,
redux"):
> On 08/23/2018 08:07 PM, Ian Jackson wrote:
> > I think git-debrebase is going to be easier for all these things than
> > the current approach.
> 
> Ok, let's try it! Thanks a lot for doing the above writeup.
Great, thanks.  (I hope it's OK that I have snipped most of your
responses to the discussion, that didn't seem to want a reply.)

> How to proceed:
> 
> Can you provide me with some rtfm pointers and instructions about how
> you would like to see things being transformed? Or would you like to do
> it together? Do we want to do this before or after Sep 10?
I would like to do this as soon as possible.

There is no pre-cooked recipe or tool for converting a packaging-only
style branch to a git-debrebase branch.  And this kind of surgery
involves more a thorough understanding of the gdr branch format than
merely using the tool on an existing branch does.

If you would like to try it then you're welcome to, but maybe it would
be better for me to do it.  How about I do this today, and push it to
a wip branch on salsa for you and Wolodja to take a look at.

(FTR: I think that git-merge to bring in upstream, plus git-debrebase
convert-from-gbp, might do the right thing, so I will try that first.)

> I actually already have some questions about the git-debrebase behavior
> to start with:
Of course.  This is quite new tool and in particular the user
documentation hasn't had much contact with users who aren't intimately
familiar with how it works.  I think this is particularly true for
people who are more familiar with older Debian packaging workflows.

So please do ask me questions here, or on irc, or whatever.

> 1: meta information
> 
> When writing software and git commits, my main incentive is to have a
> future reader understand what I'm doing and why. I tend to put quite
> some effort in this when I do things, not only because...
Absolutely.
> The quilt workflow allows to add a patch and add (meta-) comments to the
> git commit adding that patch, why it needed to be added etc. When
> directly adding patches on git level, this is not possible.
I'm not sure what you mean.  The metainformation which would
previously go into the patch description, now goes into the commit
message.
> How is this meta information stored for future readers? Does the
> reader have to search in the debian/changelog and hope something is
> mentioned about it?
Certainly not - quite the opposite.  The commit will show up in gitk
and git log and so on.  You can look at its commit message.

> 2: usage of git
> 
> One thing that is quite disturbing for me personally, is the result that
> is left behind in a git repository when using git-debrebase. Instead of
> having a meaningful history (which is my favourite thing to end up with,
> something you can "browse" and read meaningful things, see above)
git
> seems to be used as some sort of ftp server, where a rm *; cp -a * is
> done every time.
Again, I'm not sure what you mean.  Perhaps you're referring to what
you see in gitk without --date-order and without --first-parent ?
It's true that that the full history view is rather complex.

Or perhaps you are running git diff, git log, etc. on debian/patches.
When using git-debrebase, don't do that.  In a git-debrebase branch,
the files in debian/patches are an output which is best ignored.

Instead, look at the history of the actual upstream files.

> The relationship between patches and changes in them is not stored in
> git history because every time things end up in new unrelated commits.
It is true that you can't interdiff patches so easily.  IME one
probably wants to do that rarely anyway.

> The history of the current master branch, doing the packaging from 4.8
> via 4.9 and 4.10 to 4.11 and all the changes still does fit on your
> screen. I like this. I can just read back what I did and why.
> 
> So my question is: how does debrebase care about future readers? How do
> I quickly find out what *actual* changes have happened to the packaging
> from the large commit blurb that I'm looking at? Does this need a
README
> with git commands to filter commits in debian/ or other things, or does
> the debrebase tooling need to get features to analyze the git history
> again to produce a useful overview of this information?
The normal git commands for viewing subsets of the history can be
used.  Let me make some suggestions:

  History of package in Debian (disregards upstream history)
      gitk --first-parent
  In a laundered branch, the delta queue is at the top.

  History of the packaging (excluding the delta queue)
      gitk :/debian :!/debian/patches

  Just the delta queue (ie, Debian's changes to upstream):
      gitk --first-parent -- :/ :!/debian

  Full history including old versions of the delta queue:
      gitk --date-order
  The "Declare fast forward" commits you see have an older
  history (usually, an older delta queue) as one parent,
  and a newer history as the other.  --date-order makes
  gitk show the delta queues in order.

  Show complete diff since the last upload:
      git diff dgit/dgit/sid..HEAD -- :/ :!/debian/patches
  (Includes changes to upstream files.)

  Show interdiff of patches, if you really want that:
      git debrebase make-patches
      git diff dgit/dgit/sid..HEAD -- debian/patches

  Also of course there is
      git debrebase status

I think some of this information should be in the manpages.
I'll file a bug about that.

> >   Update d/control  edit control.in     # edit control.in
> >                     run rules to        debian/rules debian/control
> >                      see output,        git commit debian/control
> >                     debdifff?           git diff
> > 
> >   git blame on      not possible        git blame
> >    upstream source
> 
> Not possible? Well, not in the same repo no, but I can't imagine you
> don't know how to do a git blame on the upstream source. Sorry, but
this
> does not remotely make sense at all.
This part I wanted to comment on, since it's a useful feature of
git-debrebase branches.  What I mean is that, if I look at some file
in the dgit/stretch branch, which was done by git-debrebase, and do
`git-blame' or `git-log' on that file, I see the upstream history
*and* the patches, all combined.  That is, in the blame output: for
lines which were edited in a patch, it shows the patch, and for lines
which are from upstream, it shows the relevant upstream commit.


Thanks,
Ian.