I fixed this in rawhide kernel.spec over the weekend and left a somewhat detailed comment in the changed bit. But I thought I''d bring this up directly and explain the story. The short version of What Broke is "what, you sha1 on me? no, i sha1 on you!" Module signing works by feeding all the allocated sections of the .ko into gpg and adding that signature to the .ko as the .module_sig section. The build ID note now included in each .ko when first built is an allocated section, so its contents contribute to that signature. As explained somewhere down the middle of [1], debugedit (the workhorse of find-debuginfo.sh) recomputes fresh build ID bits after editting the debuginfo, and edits the build ID bits in the object. This final build ID is computed with sha1 on most of the contents of the editted binary (including debuginfo). It''s done this way to ensure that repeating the same rpmbuild twice with all the same constituents in the buildroot yields identical binaries both times (with the same build IDs), unaffected by the differences that debugedit removes, like the $RPM_BUILD_DIR name. In the old procedure, the modules were signed in place after being built. Then find-debuginfo.sh comes along at the end of the build, and edits the build ID bits in each .ko. Pow, the signature does not match the contents. The first fix I tried was changing module signing and verification so it skips allocated SHT_NOTE sections, which the build ID is. That worked. But that is not so hot in two ways. One, the signature is not serving as guarantor of the trustability of the module->build ID association. Worse, since the final build ID bits depend on everything in the installed .ko--including the .module_sig section--now repeated builds get different build IDs for the identical code compiled from identical sources. So I punted that. What kernel.spec now does is to sign the stripped .ko files only at the very end, after find-debuginfo.sh has done its work. This requires some spec macro hackery to get in at the right spot. Now a repeated build will still generate a new module-signing key and different signatures every time. But the build ID bits inside the signed modules will be the same every time. Now you know if a module got loaded with no unsigned taint, then the build ID bits it claims to have really are those the kernel packager wanted you to see. (Not that this is worth a lot.) Incidentally, if someone wants to freshen up the modsign patches, I think it would be better to change the format slightly. Thanks, Roland [1] http://fedoraproject.org/wiki/RolandMcGrath/BuildID
Roland McGrath wrote:> I fixed this in rawhide kernel.spec over the weekend and left a somewhat > detailed comment in the changed bit. But I thought I''d bring this up > directly and explain the story. > > The short version of What Broke is "what, you sha1 on me? no, i sha1 on you!"Something I''m still seeing: $ rpmbuild --bb --with baseonly --without debuginfo kernel.spec [...] Processing files: kernel-headers-2.6.23-0.104.rc3.vsc.fc8 Provides: glibc-kernheaders = 3.0-46 Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <3.0.3-1 Obsoletes: glibc-kernheaders Processing files: kernel-devel-2.6.23-0.104.rc3.vsc.fc8 warning: File listed twice: /usr/src/kernels/2.6.23-0.104.rc3.vsc.fc8-x86_64 warning: File listed twice: /usr/src/kernels/2.6.23-0.104.rc3.vsc.fc8-x86_64/.config [...repeat for every file in kernel-devel...] warning: File listed twice: /usr/src/kernels/2.6.23-0.104.rc3.vsc.fc8-x86_64/usr/Makefile Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/kernel-2.6.23-0.104.rc3.vsc.fc8-root-x86_64 Wrote: /usr/src/redhat/RPMS/x86_64/kernel-2.6.23-0.104.rc3.vsc.fc8.x86_64.rpm Wrote: /usr/src/redhat/RPMS/x86_64/kernel-headers-2.6.23-0.104.rc3.vsc.fc8.x86_64.rpm Wrote: /usr/src/redhat/RPMS/x86_64/kernel-devel-2.6.23-0.104.rc3.vsc.fc8.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.34783 + umask 022 + cd /usr/src/redhat/BUILD + cd kernel-2.6.22 + rm -rf /var/tmp/kernel-2.6.23-0.104.rc3.vsc.fc8-root-x86_64 + exit 0 Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.42722 + umask 022 + cd /usr/src/redhat/BUILD + rm -rf kernel-2.6.22 + exit 0 -- Jarod Wilson jwilson@redhat.com
This has nothing to do with the modsign issue.
Roland McGrath wrote:> This has nothing to do with the modsign issue.Hrm... It didn''t start happening (that I noticed) until after some of the recent changes you''d made, so I thought you might have an inkling as to what was going on. However, the behavior of the resulting kernel build *does* appear to be related. I can''t load any modules: # modprobe e1000 FATAL: Error inserting e1000 (/lib/modules/.../e1000.ko): Key was rejected by service. Best guess is that something goes haywire since I did this build ''--without debuginfo''. -- Jarod Wilson jwilson@redhat.com
> Roland McGrath wrote: > > This has nothing to do with the modsign issue. > > Hrm... It didn''t start happening (that I noticed) until after some of > the recent changes you''d made, so I thought you might have an inkling as > to what was going on.I do, but it''s harmless and not related to the modsign issue.> However, the behavior of the resulting kernel build *does* appear to be > related. I can''t load any modules: > > # modprobe e1000 > FATAL: Error inserting e1000 (/lib/modules/.../e1000.ko): Key was > rejected by service.The is the failure mode that revision 1.84 of kernel.spec fixed. Are you using the current spec?> Best guess is that something goes haywire since I did this build > ''--without debuginfo''.I can''t see how that would break the fix for module signing. Thanks, Roland
>> Roland McGrath wrote: >>> This has nothing to do with the modsign issue. >> Hrm... It didn''t start happening (that I noticed) until after some of >> the recent changes you''d made, so I thought you might have an inkling as >> to what was going on. >=20 > I do, but it''s harmless and not related to the modsign issue.My main point was to make sure it was going to get fixed, and if you already know what''s up, I presume you''ll fix it and I don''t need to spend time looking into it. :)>> However, the behavior of the resulting kernel build *does* appear to be >> related. I can''t load any modules: >> >> # modprobe e1000 >> FATAL: Error inserting e1000 (/lib/modules/.../e1000.ko): Key was >> rejected by service. >=20 > The is the failure mode that revision 1.84 of kernel.spec fixed. > Are you using the current spec?That was from revision 1.87.>> Best guess is that something goes haywire since I did this build >> ''--without debuginfo''. >=20 > I can''t see how that would break the fix for module signing.Was just a guess and perhaps not even an educated one. I''ll poke at it a bit more. --=20 Jarod Wilson jwilson@redhat.com
> That was from revision 1.87.Hmm. Actually the failure mode from before was not "modprobe" problems, but never getting that far because the insmods in the initrd failed. Do you have some magical setup that needs no modules? Or did it really succeed at boot-time insmods and then modprobe fails? Thanks, Roland
Roland McGrath wrote:>> That was from revision 1.87. > > Hmm. Actually the failure mode from before was not "modprobe" problems, > but never getting that far because the insmods in the initrd failed. > Do you have some magical setup that needs no modules? Or did it really > succeed at boot-time insmods and then modprobe fails?Looks like it really succeeded at boot-time insmods, I''ve got dm_*, ata_piix, ata_generic, libata, sd_mod, scsi_mod, ext3, jbd, mbcache and {e,o,u}hci_hcd modules all loaded. -- Jarod Wilson jwilson@redhat.com
> Roland McGrath wrote: > >> That was from revision 1.87. > > > > Hmm. Actually the failure mode from before was not "modprobe" problems, > > but never getting that far because the insmods in the initrd failed. > > Do you have some magical setup that needs no modules? Or did it really > > succeed at boot-time insmods and then modprobe fails? > > Looks like it really succeeded at boot-time insmods, I''ve got dm_*, > ata_piix, ata_generic, libata, sd_mod, scsi_mod, ext3, jbd, mbcache and > {e,o,u}hci_hcd modules all loaded.Ok, color me ungentlemanly, but I have to ask if you''ve got the right things installed on disk where modprobe is looking. I mean, where did you get those modules with good signatures for the initrd if the ones on disk are bad? This is encouraging me to add the hack I had in mind the other day to make the kernel tell you the public key it''s using in a /sys file. Thanks, Roland
Roland McGrath wrote:>> Roland McGrath wrote: >>>> That was from revision 1.87. >>> Hmm. Actually the failure mode from before was not "modprobe" problems, >>> but never getting that far because the insmods in the initrd failed. >>> Do you have some magical setup that needs no modules? Or did it really >>> succeed at boot-time insmods and then modprobe fails? >> Looks like it really succeeded at boot-time insmods, I''ve got dm_*, >> ata_piix, ata_generic, libata, sd_mod, scsi_mod, ext3, jbd, mbcache and >> {e,o,u}hci_hcd modules all loaded. > > Ok, color me ungentlemanly, but I have to ask if you''ve got the right > things installed on disk where modprobe is looking.So far as I know, yeah. I built an rpm and installed it, then booted from the entry the kernel %post added to grub.conf, same as I''ve always done... :)> I mean, where did you > get those modules with good signatures for the initrd if the ones on disk > are bad?Ya got me, but upon unpacking the initrd, modinfo tells me the bits in the initrd have the right vermagic. However, the file sizes don''t match. In fact, they aren''t even close. # (cd /tmp/initrd-104/lib; ll ext3.ko) -rw------- 1 root root 189096 2007-08-14 15:31 ext3.ko # (cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kerne/fs/ext3; ll ext3.ko) -rw-r--r-- 1 root root 2719832 2007-08-14 12:46 ext3.ko Okay, so I rebuilt the initrd and bounced the box... And there''s the expected kernel panic. So now I''m thoroughly confused as to where the hell the modules that at least booted the system came from... -- Jarod Wilson jwilson@redhat.com
On 08/14/2007 03:48 PM, Jarod Wilson wrote:> > Ya got me, but upon unpacking the initrd, modinfo tells me the bits in > the initrd have the right vermagic. However, the file sizes don''t match. > In fact, they aren''t even close. > > # (cd /tmp/initrd-104/lib; ll ext3.ko) > -rw------- 1 root root 189096 2007-08-14 15:31 ext3.ko > > # (cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kerne/fs/ext3; ll ext3.ko) > -rw-r--r-- 1 root root 2719832 2007-08-14 12:46 ext3.ko >Looks like one has the debug info in it and the other doesn''t.
> Ya got me, but upon unpacking the initrd, modinfo tells me the bits in > the initrd have the right vermagic.That doesn''t tell you anything useful. Compare the signature sections, e.g. readelf -x .module_sig on each.> However, the file sizes don''t match. > In fact, they aren''t even close. > > # (cd /tmp/initrd-104/lib; ll ext3.ko) > -rw------- 1 root root 189096 2007-08-14 15:31 ext3.ko > > # (cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kerne/fs/ext3; ll ext3.ko) > -rw-r--r-- 1 root root 2719832 2007-08-14 12:46 ext3.komkinitrd runs strip -g on the modules copied to the initrd. I hadn''t noticed that before, but it should not be a problem. (Its affect on the signature issue should not have changed.)> Okay, so I rebuilt the initrd and bounced the box... And there''s the > expected kernel panic. So now I''m thoroughly confused as to where the > hell the modules that at least booted the system came from...Ok, we''ll call the first experience a mysterious hiccup then. Did you save your rpmbuild log? Can you double-check that it has no debugedit or find-debuginfo.sh runs that follow the modsign.sh runs? Also, you could try setting MODSIGN_DEBUG in kernel/module-verify-sig.c (linux-2.6-modsign-core.patch) and booting with "debug" to see those msgs. Thanks, Roland
Roland McGrath wrote:>> Ya got me, but upon unpacking the initrd, modinfo tells me the bits in >> the initrd have the right vermagic. > > That doesn''t tell you anything useful. Compare the signature sections, > e.g. readelf -x .module_sig on each.The signature sections are identical. Triple-checked that I was comparing with the ext3.ko from the initrd that booted the system.>> However, the file sizes don''t match. >> In fact, they aren''t even close. >> >> # (cd /tmp/initrd-104/lib; ll ext3.ko) >> -rw------- 1 root root 189096 2007-08-14 15:31 ext3.ko >> >> # (cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kerne/fs/ext3; ll ext3.ko) >> -rw-r--r-- 1 root root 2719832 2007-08-14 12:46 ext3.ko > > mkinitrd runs strip -g on the modules copied to the initrd. > I hadn''t noticed that before, but it should not be a problem. > (Its affect on the signature issue should not have changed.)And indeed, the size matches if I manually run strip -g on the unstripped ko. To make it even more interesting: # cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kernel/drivers/net/e1000 # insmod e1000.ko Modules signature verification failed insmod: error inserting ''e1000.ko'': -1 Key was rejected by service # strip -g e1000.ko # insmod e1000.ko # lsmod |grep e1000 e1000 125977 0>> Okay, so I rebuilt the initrd and bounced the box... And there''s the >> expected kernel panic.Only, if mkinitrd is stripping the modules, based on the above info, this *should* have worked...>> So now I''m thoroughly confused as to where the >> hell the modules that at least booted the system came from... > > Ok, we''ll call the first experience a mysterious hiccup then. > > Did you save your rpmbuild log? Can you double-check that it has no > debugedit or find-debuginfo.sh runs that follow the modsign.sh runs?I didn''t save it, but I can do a rebuild with the same options.> Also, you could try setting MODSIGN_DEBUG in kernel/module-verify-sig.c > (linux-2.6-modsign-core.patch) and booting with "debug" to see those msgs.Sure, I''ll add that too. -- Jarod Wilson jwilson@redhat.com
> The signature sections are identical. Triple-checked that I was > comparing with the ext3.ko from the initrd that booted the system.[...]> To make it even more interesting: > > # cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kernel/drivers/net/e1000 > # insmod e1000.ko > Modules signature verification failed > insmod: error inserting ''e1000.ko'': -1 Key was rejected by service > # strip -g e1000.ko > # insmod e1000.ko > # lsmod |grep e1000 > e1000 125977 0Ok. This makes me think that the signature generation and/or verification are looking at something they shouldn''t be. i.e., something strip changed.> > Also, you could try setting MODSIGN_DEBUG in kernel/module-verify-sig.c > > (linux-2.6-modsign-core.patch) and booting with "debug" to see those msgs. > > Sure, I''ll add that too.Also hack modsign.sh to pass -v to mod-extract. The logs from mod-extract for a given module and the printks from verification looking at that module should give us something to go on. Thanks, Roland
Roland McGrath wrote:>> The signature sections are identical. Triple-checked that I was >> comparing with the ext3.ko from the initrd that booted the system. > [...] >> To make it even more interesting: >> >> # cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kernel/drivers/net/e1000 >> # insmod e1000.ko >> Modules signature verification failed >> insmod: error inserting ''e1000.ko'': -1 Key was rejected by service >> # strip -g e1000.ko >> # insmod e1000.ko >> # lsmod |grep e1000 >> e1000 125977 0 > > Ok. This makes me think that the signature generation and/or verification > are looking at something they shouldn''t be. i.e., something strip changed. > >>> Also, you could try setting MODSIGN_DEBUG in kernel/module-verify-sig.c >>> (linux-2.6-modsign-core.patch) and booting with "debug" to see those msgs. >> Sure, I''ll add that too. > > Also hack modsign.sh to pass -v to mod-extract. The logs from mod-extract > for a given module and the printks from verification looking at that module > should give us something to go on.The build is just about done, but I gotta run. I''ll poke at it some more in a few hours though... -- Jarod Wilson jwilson@redhat.com
Jarod Wilson wrote:> Roland McGrath wrote: >>> The signature sections are identical. Triple-checked that I was >>> comparing with the ext3.ko from the initrd that booted the system. >> [...] >>> To make it even more interesting: >>> >>> # cd /lib/modules/2.6.23-0.104.rc3.vsc.fc8/kernel/drivers/net/e1000 >>> # insmod e1000.ko >>> Modules signature verification failed >>> insmod: error inserting ''e1000.ko'': -1 Key was rejected by service >>> # strip -g e1000.ko >>> # insmod e1000.ko >>> # lsmod |grep e1000 >>> e1000 125977 0 >> Ok. This makes me think that the signature generation and/or verification >> are looking at something they shouldn''t be. i.e., something strip changed. >> >>>> Also, you could try setting MODSIGN_DEBUG in kernel/module-verify-sig.c >>>> (linux-2.6-modsign-core.patch) and booting with "debug" to see those msgs. >>> Sure, I''ll add that too. >> Also hack modsign.sh to pass -v to mod-extract. The logs from mod-extract >> for a given module and the printks from verification looking at that module >> should give us something to go on. > > The build is just about done, but I gotta run. I''ll poke at it some more > in a few hours though...Hrm, something appears to have gone a touch screwy with my output redirection, but most of the good stuff (from mod-extract -v) is there: http://people.redhat.com/jwilson/misc/buildlog dmesg output (as much as possible) post-boot here: http://people.redhat.com/jwilson/misc/dmesg-modsign -- Jarod Wilson jwilson@redhat.com