pkg kde talk - Mar 2011 - New build system (dhmk based) for Qt/KDE packages (qt-kde-team/2/*)

Hello,

In my opinion, it is time to move away from pkg-kde-tools/qt-kde-team/1/* 
which is based on CDBS. As most of you know, I wholeheartedly dislike CDBS for 
a couple reasons. In short:

1) The way it sequences debhelper commands (dh_command -ppackage for each 
command+package, i.e. command_count * package_count forks instead of 
command_count forks) prolongs build time significantly for huge source 
packages. As you know we do maintain such packages, actually quite a few of 
them. What''s more, build logs are unnecessary cluttered due to this.

2) It''s insanely complex. And I''m not referring to its
internals (which are
mind blowing by themselves) but to its "user" interface:

2a) first of all, it''s controlled by a bunch of variables which names
are not
always consistent (e.g. DEB_INSTALL_DOCS_ALL vs. DH_MAKESHLIBS_ARGS_ALL 
despite both passing arguments to the respective debhelper programs);

2b) secondly, each time you want to hook something up to it you have to read 
cdbs internals and copy&paste targets. Hook names are not obvious but even 
then you can''t hook to all places if you really need to.

3) So finally, I would rather spend my time doing actual work (packaging) than 
reading cdbs internals or waiting until it wastes time while I''m doing
test
builds.

Now there is dh(1) sequencer. It''s pretty nice because:

1) User interface is pretty straightforward. In particular override_dh_command 
targets are nice and `dh $@ --extra-option1 --extra-option2` command line 
interface for options is fine too.

2) Sequencer is written in perl. So are dh addons which push you to write a 
new debhelper program (also perl based) when you want to extend functionally 
for general use (i.e. outside debian/rules). As a result, "templating"
is
pretty limited to perl which is not optimal due to many reasons.

Therefore I wrote dhmk which qt-kde-team/2/* is based upon (since pkg-kde-
tools 0.10). dhmk.* is a basic sequencer that is driven by make and can be 
*templated* with make snippets. debian-qt-kde.mk and friends hook a couple of 
custom stuff to general sequences.

dhmk design goals:

1) dhmk is make-based. Unlike dh(1), it does not track progress per command, 
but per action (configure, build, install etc.). It has sense of stamped 
(configure, build) and dynamic (install, binary) targets like traditional 
debian/rules would.

2) dhmk is compatible with dh as much as possible, e.g. it runs the same 
debhelper commands dh(1) would (by default) and it''s possible to reuse
dh(1)
addons to extend the sequencer. Like with dh(1), source code is built by 
dh_auto_* commands. Sune expressed some criticism towards them, but I 
personally don''t see why we should not use official, well supported and
widely
tested debhelper helpers given that they already do some important work behind 
the scenes (parallel, verbose output etc.).

3) There is the dhmk.pl part which is responsible for generating a dynamic 
debian/dhmk_rules.mk file. It''s used for:

	a) parsing commands file (static) and converting it to a bunch of variable 
definitions which dhmk.mk would understand;
	b) executing dh addons (specified using --with option to dh variable) and 
integrating their sequence changes to debian/dhmk_rules.mk file;
    c) parsing the rest of options in the dh variable (i.e. like dh(1) would)
and passing them to all dh commands being run (and also exporting them in 
DHMK_OPTIONS environment variables for use in overrides);
	d) detecting which debhelper commands have overrides in debian/rules and 
specifying that info in debian/dhmk_rules.mk for later use by dhmk.mk.

Now yeah, dhmk.pl is still perl. But if there is ever a need to drop perl, 
debian/dhmk_rules.mk could be turned static and d) replaced with sed script 
(8de511c5 commit [1] has an example which need to be tweaked). However, 
support for b) and c) would be lost in that case. Auto-generated dhmk_rules.mk 
is a bunch of variable definitions which are not very handy to edit by hand.

4) dhmk.mk is a core sequencer and it''s 100% extensible with make
snippets.
Snippets may hook to run the code before/after *each* debhelper command or 
after/before each standard target. There is no need to know perl to extend 
general functionality. See library-packages.mk, policy.mk and debian-qt-kde.mk 
for examples of such make-based dhmk extensions.

5) While make snippets use hooks, debian/rules should use override_dh_command 
targets. So here dhmk is not reinventing the wheel and borrowing the best what 
dh(1) has to offer. Contrary to dh(1), dhmk does not repass extra options 
(defined in the dh variable) to the overriden dh_command''s. Those are
made
available in the DHMK_OPTIONS envvar for later use if needed. Only arch/indep 
options (-i/-a) are handled implicitly.

6) dhmk.mk code itself, while pretty short, is not very straightforward (a 
bunch of advanced make stuff like $(foreach), $(if), $(filter) etc. and 
extensive use of internal variables all over the place. In order to understand 
what''s going on, have a look at what variables get defined in
dhmk_rules.mk.
Unfortunately, extensive templating options make dhmk UI a bit more cluttered 
than the one of dh(1) but it''s still far far less complex than cdbs (in
my
opinion anyway). Some more information is available in qt-kde-team/2/README.

Therefore, I would like to see Qt/KDE packages getting converted to dhmk [2]. 
Unless somebody has to offer something better, they are welcome to. But 
personally, I no longer see "sticking to CDBS" as an option. While
qt-kde-
team/2/* is not perfect, I tried to decouple it from tight bound with perl 
while adhering to the best successful practises established by dh(1).

[1] http://git.debian.org/?p=pkg-kde/pkg-kde-tools.git;a=commit;h=8de511c5
[2] I have recently converted phonon and its backends will follow soon.

-- 
Modestas Vainius <modestas at vainius.eu>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110309/e01935a4/attachment.pgp>

Before commenting any further, I must be honest that I (still) never built KDE 
itself, but always "small" packages. Nontheless, I have been trying to
understand CDBS before... and I was lucky enough to be on time for dh.

On Wednesday 09 March 2011 15:23:00 Modestas Vainius
wrote:
> Hello,
> 
> In my opinion, it is time to move away from pkg-kde-tools/qt-kde-team/1/*
> which is based on CDBS. As most of you know, I wholeheartedly dislike CDBS
> for a couple reasons. In short:
[snip] 
> 3) So finally, I would rather spend my time doing actual work (packaging)
> than reading cdbs internals or waiting until it wastes time while
I''m
> doing test builds.
Fully and wholeheartdly agree on this.
> Now there is dh(1) sequencer. It''s pretty nice because:
> 
> 1) User interface is pretty straightforward. In particular
> override_dh_command targets are nice and `dh $@ --extra-option1
> --extra-option2` command line interface for options is fine too.
Agree.
 
[snip]
> dhmk design goals:
[big snip]

I really like them, although I must admit that I have little knowledge of 
makefiles (wich is easily fixable, as my job/NMUs/etc stuff has been showing 
me), although I don''t have any perl experience... and I don''t
plan to have it
in the near future.
> Therefore, I would like to see Qt/KDE packages getting converted to dhmk
> [2]. Unless somebody has to offer something better, they are welcome to.
> But personally, I no longer see "sticking to CDBS" as an option.
While
> qt-kde- team/2/* is not perfect, I tried to decouple it from tight bound
> with perl while adhering to the best successful practises established by
> dh(1).
As for me, I really like the idea. But once again, I may not be the most 
suitable person to say this.

Kinds regards, Lisandro.

-- 
Contrary to popular belief, Unix is user friendly. It just happens to be
very selective about who it decides to make friends with.
  Unknown - http://www.linfo.org/q_unix.html

Lisandro Dami?n Nicanor P?rez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110310/ac8d2910/attachment.pgp>

Hello,
> [2] I have recently converted phonon and its backends will follow soon.
I have already switched a couple of kde-req and kde-std packages to qt-kde-
team/2/* [1]. So far so good. Checklist:

1) Build-Depend on pkg-kde-tools (>= 0.11).

2) s,/usr/share/pkg-kde-tools/qt-kde-team/1,/usr/share/pkg-kde-tools/qt-kde-
team/2,.

3) Replace cdbs variables and hooks with override_{dh_command} targets. Avoid 
using dhmk hooks in debian/rules directly (for the sake of clarity and 
consistency).

4) In most cases call overriden commands via $(overriden_command) in the 
override targets (unless its effect is not desirable). $(overriden_command) 
expands to the full original command line of the overriden command including 
additional arguments added by dh addons and "dh" variable (if any).

5) Do not forget to specify --dbg-package manually (either via 
"override_dh_strip:\n\t$(overriden_command) --dbg-package=pkg"
(recommended as
it''s less verbose in the build log output) or "dh
--dbg-package=pkg". Thats
needed because cdbs detects dbg package automatically while qt-kde-team/2/* 
does not (and should not).

6) library-packages.mk was rewritten and variable names were changed. Change 
them accordingly in debian/rules.

7) Test if the package is actually acted upon before doing anything non-
debhelper related with it in the overrides (esp. if it''s arch: all).
That''s
needed due to the different set of packages operated on during binary and 
binary-arch builds. You can use if [ -d debian/pkgname ] (faster) or if [ -n 
"$(filter pkgname,$(shell dh_listpackages))" ] or similar techniques
for this.

[1] phonon, phonon-backend-*, soprano, akonadi, polkit-qt-1, qimageblitz ...

-- 
Modestas Vainius <modax at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110314/7417b3cd/attachment.pgp>