Pkg xen devel - Jan 2018 - Xen packaging in Debian - Progress update

To: Debian xen and kernel team list, Ian Jackson
Cc: Stefan Bader, maintainer of xen packages in Ubuntu

Hi all,

Short version: Hi! I'd like to help with the Xen packaging in Debian.

Long version:

Q: Who are you? How are you related to Debian an the Xen project?
A: Hi, I'm Hans van Kranenburg, nickname Knorrie, I live in the
Netherlands. I'm a Debian user since 2002, and have been using Debian
and Xen at work (Mendix) since 2006, which has grown from a handful
dom0s with 4GiB memory each to a bunch of clusters with let's say
somewhere between 10TiB and 20TiB of physical memory in the dom0s
together, running a production environment for customer application hosting.

My general interests are filesystems and networking and I like
programming in Python. At work, we've been maintaining our own debian
repository for many years, with our own packages, changed Debian
packages and custom backports, which caused me to pick up some Debian
packaging skills along the way.

Q: And why the sudden interest?
A: Some problems we ran into in the last year+ caused me to have a
better look at the Xen releases and the packages in Debian.

The situation I ran into is best described as: "Wait... I'm running a
Xen version from Debian Stable (at the time of realizing that's Jessie)
that was released before the type of hardware I have here was invented
and manufactured, it's out of support and out of security support
upstream and I'm surprised weird things are going on?" Maybe we should
reverse this a bit and see if we can keep up to date with Xen stable
releases that know about certain quirks of the hardware.

The sad part is that I quickly discovered the xen packages aren't really
actively maintained in Debian. Luckily we got a newer version in Stretch
just before the freeze and currently Ian Jackson is keeping everything
on life support (thanks!!). Current packaging is not tracked in version
control (well, not on a level of granularity that I would deem
acceptable) and the contents are being changed based on unpacking the
previous upload and changing old generated files in place, disregarding
the way how the package was set up in the past (which is quite similar
to how the linux kernel packaging for Debian works).

Q: So, let's sit in a corner and cry?
A: No, we can do better. A few days ago I started reaching out to see if
I could find members of the Xen team and ended up talking to Ian Jackson
in #debian-kernel. He encouraged me to take a further look and was
immediately available to help and answer any questions I would have.

What I did in the last few days is basically clean up the packaging to
get it back to a state where it's usable again. So, move the packaging
back to git, import the latest release and then fix enough things to be
able to at least produce new security updates for Stretch and get a
newer package into unstable.

I uploaded the work it to github [0] for now. It can be moved anywhere
else later. I tend to write things in commit messages, so please have a
look.

Besides preparing a new version for stretch-security as an example, I
moved the packaging forward to Xen 4.9, by taking the relevant changes
done by Stefan Bader in Ubuntu (thanks!!) and merge them back into the
packaging.

I (smoke)tested the resulting packages in my test environment at work.
To be able to properly test I put them in a repository at [1].

Note that I'm not a Debian Maintainer (yet). I do have packages in
Debian with my work on btrfs, the uploads are sponsored by Adam
Borowski. [2]

I guess that it'd be good to finally take the step to apply for Debian
Maintainer status when starting to work on low level security sensitive
packages like this. Luckily, we have a Debian keysigning party in the
next days in The Netherlands, so I have a quick opportinity to get
things in order. :]

I have already identified quite a few topics I'd like to discuss next,
but, let's take it one step at a time, I typed enough already here. :)

Regards,
Hans van Kranenburg

[0] https://github.com/knorrie/debian-xen/
[1] https://packages.knorrie.org/xen/debian/pool/main/x/xen/
[2] https://qa.debian.org/developer.php?login=hans%40knorrie.org

Hi,

On 12/22/2017 02:01 AM, Hans van Kranenburg wrote:
> To: Debian xen and kernel team list, Ian Jackson
I'm replying to my own email, since there has not been a reply on the
lists to it yet.

For Ian Jackson: There's a question for you below (section "Moving
packaging repository"), can you please answer that.

== Whaaaaat's happening? (skip this section if you're busy) =
Three weeks ago, when I started working on this, I didn't remotely know
what would happen on Jan 3, but I like the rollercoaster ride.

In the past days:
* ...I've been working to get the packaging moved on from 4.9 to Xen 4.10.
* ...I could help the kernel team a bit with handling xen-related
regression bugs that were reported on the updated linux packages.
* ...replied to a few other open Xen bugs.
* Moritz from the security team mailed me like "Hey! Did you get any
response on that email? What's gonna happen to xen?" and I told him
what
was going on and that I'd like to help wherever I can.
* ...the #debian-xen IRC channel on OFTC has become active again, others
have been joining and offering more help, and we're discussing all kind
of things, about packaging, but also about upgrade tactics for any size
Xen cluster we're managing ourselves.

== Security update for Stretch =
On IRC I got some questions about the already earlier released XSA
patches, which still aren't in Stretch.

Question for security team: If you want to have it, I've prepared an
update for in the mean time. [0] [1] [2]. I'm not a Maintainer yet, but
at least I have some GPG signatures of Debian Developers now. ;]

== Xen 4.10 =
We're still all looking at upstream to find out what they're going to do
in the next days.

My gut feeling is that a not-neglectible part of users is looking into
upgrading to Xen 4.10 and Linux 4.14 in the domU. It would be really
great if Debian could have Xen 4.10 in testing and stretch-backports.

So, in the meantime I'm trying to get a proper Xen 4.10 package in order
for unstable. The build doesn't work completely yet, but if there's a
dpkg-shlibdeps expert around, maybe you can help. More info on IRC.

In general, If someone with more intimate knowledge of Xen can help
review the changes, always welcome, as well as beta-testers for new
packages.

== Moving packaging repository =
It makes sense to get the cleaned up packaging code moved to a new
Debian Xen team owned repo at the new Debian Gitlab hosting.

Ian: Can you please ACK that it's ok if we go on with this and take
ownership to get it set up? Thanks.

== Thanks for your time =
Regards,
Hans van Kranenburg

[0]
https://github.com/knorrie/debian-xen/commit/d3922c423010894d5badfc5381a7312b90715cbf

[1]
https://github.com/knorrie/debian-xen/releases/tag/debian%2F4.8.2%2Bxsa245-0%2Bdeb9u2

[2] https://syrinx.knorrie.org/~knorrie/xen/stretch-security/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20180109/3f3e05c3/attachment.sig>

Hi all,

Thanks for sending this update Hans.  I too am interested in helping out with
xen packaging.  I am trying to dig into the dpkg-shlibdeps issues - I built it
successfully on one box but haven?t been able to reproduce.  Either way, I will
plan to hang around in IRC to help where I can.

Stephen
> On Jan 9, 2018, at 4:05 PM, Hans van Kranenburg <hans at knorrie.org>
wrote:
> 
> Hi,
> 
> On 12/22/2017 02:01 AM, Hans van Kranenburg wrote:
>> To: Debian xen and kernel team list, Ian Jackson
> 
> I'm replying to my own email, since there has not been a reply on the
> lists to it yet.
> 
> For Ian Jackson: There's a question for you below (section "Moving
> packaging repository"), can you please answer that.
> 
> == Whaaaaat's happening? (skip this section if you're busy) => 
> Three weeks ago, when I started working on this, I didn't remotely know
> what would happen on Jan 3, but I like the rollercoaster ride.
> 
> In the past days:
> * ...I've been working to get the packaging moved on from 4.9 to Xen
4.10.
> * ...I could help the kernel team a bit with handling xen-related
> regression bugs that were reported on the updated linux packages.
> * ...replied to a few other open Xen bugs.
> * Moritz from the security team mailed me like "Hey! Did you get any
> response on that email? What's gonna happen to xen?" and I told
him what
> was going on and that I'd like to help wherever I can.
> * ...the #debian-xen IRC channel on OFTC has become active again, others
> have been joining and offering more help, and we're discussing all kind
> of things, about packaging, but also about upgrade tactics for any size
> Xen cluster we're managing ourselves.
> 
> == Security update for Stretch => 
> On IRC I got some questions about the already earlier released XSA
> patches, which still aren't in Stretch.
> 
> Question for security team: If you want to have it, I've prepared an
> update for in the mean time. [0] [1] [2]. I'm not a Maintainer yet, but
> at least I have some GPG signatures of Debian Developers now. ;]
> 
> == Xen 4.10 => 
> We're still all looking at upstream to find out what they're going
to do
> in the next days.
> 
> My gut feeling is that a not-neglectible part of users is looking into
> upgrading to Xen 4.10 and Linux 4.14 in the domU. It would be really
> great if Debian could have Xen 4.10 in testing and stretch-backports.
> 
> So, in the meantime I'm trying to get a proper Xen 4.10 package in
order
> for unstable. The build doesn't work completely yet, but if there's
a
> dpkg-shlibdeps expert around, maybe you can help. More info on IRC.
> 
> In general, If someone with more intimate knowledge of Xen can help
> review the changes, always welcome, as well as beta-testers for new
> packages.
> 
> == Moving packaging repository => 
> It makes sense to get the cleaned up packaging code moved to a new
> Debian Xen team owned repo at the new Debian Gitlab hosting.
> 
> Ian: Can you please ACK that it's ok if we go on with this and take
> ownership to get it set up? Thanks.
> 
> == Thanks for your time => 
> Regards,
> Hans van Kranenburg
> 
> [0]
>
https://github.com/knorrie/debian-xen/commit/d3922c423010894d5badfc5381a7312b90715cbf
> 
> [1]
>
https://github.com/knorrie/debian-xen/releases/tag/debian%2F4.8.2%2Bxsa245-0%2Bdeb9u2
> 
> [2] https://syrinx.knorrie.org/~knorrie/xen/stretch-security/
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-xen-devel

Hans van Kranenburg <hans at knorrie.org> writes:
> == Security update for Stretch =>
> On IRC I got some questions about the already earlier released XSA
> patches, which still aren't in Stretch.
It would be a lovely if a security upload that includes patches for the
following XSAs could be prepared, given that many people will reboot
their hypervisors these days:

- https://xenbits.xen.org/xsa/advisory-248.html
- https://xenbits.xen.org/xsa/advisory-249.html
- https://xenbits.xen.org/xsa/advisory-250.html
- https://xenbits.xen.org/xsa/advisory-251.html

These are all single patches that apply cleanly to 4.8 (with some fuzz)
and will have been deployed in locally built packages by many.

Thanks for all your efforts!
-- 
Wolodja <debian at babilen5.org>

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC