Pkg xen devel - Apr 2018 - debian-xen git workflow

Hi Ian, team,

    In the week of Feb 23 (time flies!:), we had a very productive day
\o/ working on the stretch security/stable package and some irc
discussions in oftc #debian-xen which ended with an open end.

Ian, you were a bit surprised that 1) my debian-xen repo started with an
import of a 4.8 package 2) it was not a git-pbuilder style repo with
upstream source included, and asked me to at least make a conscious
decision about why not doing those two things instead.

To start with the conclusion: I indeed want to continue in the same way
as the current repository on salsa is set up, since I believe it fits
the type of package and work (and workflow for contributors) quite well.

Summarizing that's:
 * packaging-only repo
 * debian changelog version is an unambiguous reference to an upstream
version/commit
 * generated files (4.x version dependent) are in the git ignore
 * branching done like explained at
https://salsa.debian.org/xen-team/debian-xen/wikis/home

I'd appreciate if you just would let me do this for a while and see what
track record we achieve while rolling forward.

The rest of this wall of text is a collection of thoughts and things
that happened in the last months. Feel free to read or ignore (but not
ask questions which are answered here already) or let me know how I
could have provided the same information using less words.

-------- >8 --------

For Xen packaging, there are two different types of work running in
parallel:
 * Following the latest greatest releases and putting them in Debian
unstable (yes, there's actually an audience for that, and for
stable-backports)
 * Provide fixes for security and stability issues for Debian stable.

Also, for Xen packaging, there are two peculiarities which make it look
a bit different from many other packages in Debian:
 * Instead of simple a linear development model, upstream supports
multiple releases in parallel (e.g. 4.8, 4.9, 4.10). There is no path
forward from a stable branch to the next release (e.g 4.10-stable ->
4.11) in upstream version control.
 * There are quite some binary packages, and a bunch of them have a
version number in them, which changes when moving to a new 4.x release.

    In Nov/Dec 2017, the first thing I tried was searching for an
existing packaging git repo.

I could find an old repository on alioth, which had some packaging from
2014 and 2015, but nothing recent. Since the way of working in there was
very different again (git-dpm I guess), and about 75% of the full commit
history is the same set of patches over and over and over and over and
over again, and since I could find only exacty one commit in there that
has a bit more explanation than a single meaningless line of ("Update
XYZ") which describes why the change is done and what it solves
(092785a0d8), I respectfully ignored this and later archived it at
https://salsa.debian.org/xen-team/xen-2014-2015

    Since I couldn't find anything else, I assumed the current packages
were just "based on changing the previous version" and then uploading
that. Ubuntu already had a Xen 4.9 package, and based on that work, the
first thing I tried was to create a git-pbuilder style repo with
upstream + debian packaging.

But, I quickly ran into a set of problems and inconveniences:
 * git-pbuilder can't handle the git-attributes replacement magic when
creating an orig tar, so it always ends up with a failing build because
of unexpected upstream changes. I couldn't figure out a way to work
around this.
 * Having the upstream source attached to the packaging is not a good
match for the parallel-development style repo that Xen upstream has.
There is no path forward from e.g 4.10-stable to 4.11. Merge commits of
upstream source make wip-rebasing, branching and merging of packaging
changes between versions very awkward, noisy and highly meaningless.
 * The ubuntu package obviously had been constructed by modifying
generated content (all the generated files and files with the 4.x
version in their name) in place, and having these files around meant I
had to rename them from 4.9 to 4.10, resulting in noisy error-prone
non-optimal git commits with no default history for the new files.

    At that moment I discovered these version-dependent files could be
generated, and when taking a closer look, I also discovered that the
packaging looked much alike the packaging repo of the debian linux
kernel team. Since I've been following and using their packaging for
quite a bunch of years, I was pleasantly surprised about this. BOOM!

What I did directly after this is weeding out all generated content, and
that's where the repo started.

I documented a few things in messages to the team mailing list:

https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2017-December/007118.html
https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2018-January/007153.html

Those messages got zero feedback from anyone(!!), so I just continued
working in that way.

It was only in March 2018 when I learned about the dgit system and
discovered the part of the packaging for the current Stretch release.
However, after playing around for a few hours with dgit, I wasn't able
to construct a local repository which made much sense. What I ended up
with was multiple initial commits and unrelated branches based without
finding out how to combine them into a meaningful history.

There's undoubtedly a lot more to say, but this message is already too
long :)

Let useful discussions continue, but aside from that, I would really
like to end up with an upload of Xen 4.10 to Debian Unstable soon.

Hans

Hi Hans,

Thanks for the detailed message, while I haven’t actually done any work on Xen
in Debian (yet, well, aside from that small UEFI stuff), it does make a lot of
sense to me to use the style currently used in Salsa, the old(er) system doesn’t
track well with upstream indeed, and since Debian uses the same structure for
the kernel packages I don’t see why continuing with the older style helps
anyone.
From my perspective: good job!

John
> On 30 Apr 2018, at 21:36, Hans van Kranenburg <hans at knorrie.org>
wrote:
> 
> Hi Ian, team,
> 
>    In the week of Feb 23 (time flies!:), we had a very productive day
> \o/ working on the stretch security/stable package and some irc
> discussions in oftc #debian-xen which ended with an open end.
> 
> Ian, you were a bit surprised that 1) my debian-xen repo started with an
> import of a 4.8 package 2) it was not a git-pbuilder style repo with
> upstream source included, and asked me to at least make a conscious
> decision about why not doing those two things instead.
> 
> To start with the conclusion: I indeed want to continue in the same way
> as the current repository on salsa is set up, since I believe it fits
> the type of package and work (and workflow for contributors) quite well.
> 
> Summarizing that's:
> * packaging-only repo
> * debian changelog version is an unambiguous reference to an upstream
> version/commit
> * generated files (4.x version dependent) are in the git ignore
> * branching done like explained at
> https://salsa.debian.org/xen-team/debian-xen/wikis/home
> 
> I'd appreciate if you just would let me do this for a while and see
what
> track record we achieve while rolling forward.
> 
> The rest of this wall of text is a collection of thoughts and things
> that happened in the last months. Feel free to read or ignore (but not
> ask questions which are answered here already) or let me know how I
> could have provided the same information using less words.
> 
> -------- >8 --------
> 
> For Xen packaging, there are two different types of work running in
> parallel:
> * Following the latest greatest releases and putting them in Debian
> unstable (yes, there's actually an audience for that, and for
> stable-backports)
> * Provide fixes for security and stability issues for Debian stable.
> 
> Also, for Xen packaging, there are two peculiarities which make it look
> a bit different from many other packages in Debian:
> * Instead of simple a linear development model, upstream supports
> multiple releases in parallel (e.g. 4.8, 4.9, 4.10). There is no path
> forward from a stable branch to the next release (e.g 4.10-stable ->
> 4.11) in upstream version control.
> * There are quite some binary packages, and a bunch of them have a
> version number in them, which changes when moving to a new 4.x release.
> 
>    In Nov/Dec 2017, the first thing I tried was searching for an
> existing packaging git repo.
> 
> I could find an old repository on alioth, which had some packaging from
> 2014 and 2015, but nothing recent. Since the way of working in there was
> very different again (git-dpm I guess), and about 75% of the full commit
> history is the same set of patches over and over and over and over and
> over again, and since I could find only exacty one commit in there that
> has a bit more explanation than a single meaningless line of ("Update
> XYZ") which describes why the change is done and what it solves
> (092785a0d8), I respectfully ignored this and later archived it at
> https://salsa.debian.org/xen-team/xen-2014-2015
> 
>    Since I couldn't find anything else, I assumed the current packages
> were just "based on changing the previous version" and then
uploading
> that. Ubuntu already had a Xen 4.9 package, and based on that work, the
> first thing I tried was to create a git-pbuilder style repo with
> upstream + debian packaging.
> 
> But, I quickly ran into a set of problems and inconveniences:
> * git-pbuilder can't handle the git-attributes replacement magic when
> creating an orig tar, so it always ends up with a failing build because
> of unexpected upstream changes. I couldn't figure out a way to work
> around this.
> * Having the upstream source attached to the packaging is not a good
> match for the parallel-development style repo that Xen upstream has.
> There is no path forward from e.g 4.10-stable to 4.11. Merge commits of
> upstream source make wip-rebasing, branching and merging of packaging
> changes between versions very awkward, noisy and highly meaningless.
> * The ubuntu package obviously had been constructed by modifying
> generated content (all the generated files and files with the 4.x
> version in their name) in place, and having these files around meant I
> had to rename them from 4.9 to 4.10, resulting in noisy error-prone
> non-optimal git commits with no default history for the new files.
> 
>    At that moment I discovered these version-dependent files could be
> generated, and when taking a closer look, I also discovered that the
> packaging looked much alike the packaging repo of the debian linux
> kernel team. Since I've been following and using their packaging for
> quite a bunch of years, I was pleasantly surprised about this. BOOM!
> 
> What I did directly after this is weeding out all generated content, and
> that's where the repo started.
> 
> I documented a few things in messages to the team mailing list:
> 
>
https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2017-December/007118.html
>
https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2018-January/007153.html
> 
> Those messages got zero feedback from anyone(!!), so I just continued
> working in that way.
> 
> It was only in March 2018 when I learned about the dgit system and
> discovered the part of the packaging for the current Stretch release.
> However, after playing around for a few hours with dgit, I wasn't able
> to construct a local repository which made much sense. What I ended up
> with was multiple initial commits and unrelated branches based without
> finding out how to combine them into a meaningful history.
> 
> There's undoubtedly a lot more to say, but this message is already too
> long :)
> 
> Let useful discussions continue, but aside from that, I would really
> like to end up with an upload of Xen 4.10 to Debian Unstable soon.
> 
> Hans
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-xen-devel

Hans van Kranenburg writes ("debian-xen git
workflow"):
> I'd appreciate if you just would let me do this for a while and see
what
> track record we achieve while rolling forward.
OK.
> The rest of this wall of text is a collection of thoughts and things
> that happened in the last months. Feel free to read or ignore (but not
> ask questions which are answered here already) or let me know how I
> could have provided the same information using less words.
> 
> -------- >8 --------
I have read everything you wrote.  Thanks for the writeup.  I will
only reply to the bits I had any kind of comment on:

...
> But, I quickly ran into a set of problems and inconveniences:
>  * git-pbuilder can't handle the git-attributes replacement magic when
> creating an orig tar, so it always ends up with a failing build because
> of unexpected upstream changes. I couldn't figure out a way to work
> around this.
The gitattributes are very annoying.  My approach with the security
releases has been to (i) completely disable them (ii) generate my own
orig tarballs using appropriate `git archive' runes.
>  * Having the upstream source attached to the packaging is not a good
> match for the parallel-development style repo that Xen upstream has.
> There is no path forward from e.g 4.10-stable to 4.11. Merge commits of
> upstream source make wip-rebasing, branching and merging of packaging
> changes between versions very awkward, noisy and highly meaningless.
I have a new tool (`git-debrebase') in development which may make some
of this easier, but I don't get the impression that you want to be an
early adopter of novel git tools, so think it would be best for you to
avoid it until it's more mature.
> It was only in March 2018 when I learned about the dgit system and
> discovered the part of the packaging for the current Stretch release.
> However, after playing around for a few hours with dgit, I wasn't able
> to construct a local repository which made much sense. What I ended up
> with was multiple initial commits and unrelated branches based without
> finding out how to combine them into a meaningful history.
The stretch branches are not in a very convenient state for anyone but
the most hardcore git expert.  My apologies for that.
> Let useful discussions continue, but aside from that, I would really
> like to end up with an upload of Xen 4.10 to Debian Unstable soon.
Yes.

Regards,
Ian.

-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my
own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

On 05/14/2018 04:52 PM, Ian Jackson wrote:
> Hans van Kranenburg writes ("debian-xen git workflow"):
>> I'd appreciate if you just would let me do this for a while and see
what
>> track record we achieve while rolling forward.
> 
> OK.
Thanks for the feedback.
> I have read everything you wrote.  Thanks for the writeup.  I will
> only reply to the bits I had any kind of comment on:
Doing the same now.
>>  * Having the upstream source attached to the packaging is not a good
>> match for the parallel-development style repo that Xen upstream has.
>> There is no path forward from e.g 4.10-stable to 4.11. Merge commits of
>> upstream source make wip-rebasing, branching and merging of packaging
>> changes between versions very awkward, noisy and highly meaningless.
> 
> I have a new tool (`git-debrebase') in development which may make some
> of this easier, but I don't get the impression that you want to be an
> early adopter of novel git tools, so think it would be best for you to
> avoid it until it's more mature.
For sure I'm interested in learning more about this, and if I can,
contribute some testing time and ideas to help.

One of the things I really like about the dgit approach is that you work
towards getting rid of manual actions needed to manage debian/patches. I
think we agree that just using git is less of a headache than vim and
quilt if you want to make custom changes to upstream.

However, the downside of this approach is that it instantly gets less
obvious for someone browsing the history to understand which git commits
are part of the real upstream code or were cherry-picked by a
maintainer, because they look exactly the same. This works when you're a
sole maintainer and have this information in your head, or you can
reverse engineer this information by looking at debian/patches again,
but meh... :|

And, updating to a new upstream version means that you can end up with
conflicts caused by changes to upstream added by yourself...

In the case of Xen, I've been looking at and thinking about the current
patches a bit.

* Most of the patches are related to the abiname/prefix thing, and we
already know that we'd like to find a solution for this which can be
upstreamed. I'm not an expert in this area, but I will try to help in
whatever way I can when it's going to happen. Also: [0]
* There might be some cherry-picked patches, like including the xen-diag
tool for 4.8 now (#880554), but those kind of patches are not the
painful ones to add manually to debian/patches. Just use the right git
format patch command and boom!, it's done.
* And, I'd really like to support a "fewer patches is better"
culture,
trying to build a good relationship with upstream, so that instead of
piling up patches, there can be some discussion back and forth about
what issues users run into and how they can be solved upstream. Adding
patches to the debian packaging should be a last resort.
>> It was only in March 2018 when I learned about the dgit system and
>> discovered the part of the packaging for the current Stretch release.
>> However, after playing around for a few hours with dgit, I wasn't
able
>> to construct a local repository which made much sense. What I ended up
>> with was multiple initial commits and unrelated branches based without
>> finding out how to combine them into a meaningful history.
> 
> The stretch branches are not in a very convenient state for anyone but
> the most hardcore git expert.  My apologies for that.
Well, as you told me earlier, you kind of adopted the packaging because
it got abandoned, and at least made sure there were new security updates
for users. Yay for that!

If I was in the same position and also developing new tools for package
management at the same time, I think I would also have gone the way of
trying them on whatever I was working on. :)
>> Let useful discussions continue, but aside from that, I would really
>> like to end up with an upload of Xen 4.10 to Debian Unstable soon.
> 
> Yes.
\o/

Can you share your thoughts about what work we need to do to reach this
goal?

I'm currently already running the Xen 4.10 packages at work in a
pre-production environment. Except for some serious major breakage [1]
it just works (tm) as long as we don't try to use live migrate.

One thing I noticed is that there are an awful lot of warnings and
interesting messages spewed out during the build, also related to
debian-specific things. And, the packages I end up also report a quite
interesting list of lintian errors and warnings.

I think the lintian report is at least something that we should have a
look at. I can run this on my most recent packages and compose a message
to the list in which I try to triage them. Or, I could just split them
out and create gitlab issues for every one of them, in which anyone can
add a thought about how important it is and how it could be solved?

What do you think?

Hans

[0] I have been wondering... and I haven't tried to test this myself
yet... This hypervisor/utils ABI which the debian packaging relies on...
Is there some upstream test in the test suite to make sure this does not
change during a 4.x release? I can imagine that making huge changes like
the spectre/meltdown stuff might accidentally change this. How is this
managed?

[1] At Mendix, we've ran into a very nasty live migration problem,
leading to corruption in the dom0 network stack (or worse things looking
like random memory corruption causing spectacular crashes and even disk
curruption in the domU) and haven't been able to trace down the exact
cause yet. However, we can reproduce the same issue with Xen 4.8 from
Debian Stable and are really puzzled about what's so different about our
setup that causes this.

Since we can reproduce using the current stretch xen 4.8 and kernel in
dom0, I don't think this should be a total blocker for 4.10 to enter
unstable, because all other users of it are probably not hitting this
problem? (insert puzzled face icon here)

Hi,

On 05/01/2018 12:59 AM, John Keates wrote:
> 
> Thanks for the detailed message, while I haven’t actually done any
> work on Xen in Debian (yet, well, aside from that small UEFI stuff),
> it does make a lot of sense to me to use the style currently used in
> Salsa, the old(er) system doesn’t track well with upstream indeed,
> and since Debian uses the same structure for the kernel packages I
> don’t see why continuing with the older style helps anyone. From my
> perspective: good job!
Thanks!

Did you try to build and run 4.10 Xen packages with your change in it
already?

"Doing work on Xen in Debian" might start with just asking around if
there's something you could help with, or, mentioning things that are
missing and getting them added, like you did already.

Don't underestimate the fact that you're a user who is actually using
that "small UEFI stuff". There's a huge amount of different use
cases
and there's a huge amount of configuration options and all of them
result in a specific user doing specific things with it, which could
result in some weird specific bug that they're running into.

So, don't assume someone else will also be running the exact same Xen
setup as you do, and let your voice be heard if you encounter anything. :)

Have fun,
Hans