Syslinux - Oct 2010 - Stumped by extlinux After OS Upgrade

Hello fellow syslinux/extlinux users!  Please forgive the long post, but I 
believe my problem warrants the background info to understand the issue.

I've developed a specialized spin of Fedora that runs off CompactFlash. 
The spin is actually produced by the livecd-tools package and is then 
transferred to the CF card -- which has been formatted with ext3 -- using 
livecd-tools' livecd-iso-to-disk.  I have hundreds of embedded systems 
running this spin.  Every now and then we need to upgrade these systems 
and needless to say swapping CF cards is labor expensive.  Thus I 
developed an in-place upgrade which essentially downloads a new squashfs 
image and syslinux directory, both of which are obtained from a newer 
spin.  These downloads do not overwrite those of the running system; they 
get a name suffix of ".new".  Then a special initrd is booted into
which
looks for the new stuff and puts them in place after renaming the 
originals with an ".old" suffix.  Another reboot occurs and the new
spin
is then running.  Currently I'm trying to effect an upgrade that will take 
these systems from a Fedora 10 base to Fedora 13.

Now the fun part and my problem.  Before shipping CF cards out with F10 on 
them, I went through rigorous functional testing and one step was to 
verify that future upgrades would be successful as best that I could. 
Since F10 was the latest Fedora at the time it was only a pseudo-test that 
tried to upgrade the F10 spin to an alternate, distinguishable F10 spin. I 
made that work, but found it necessary to run "extlinux --update 
/sysroot/syslinux" after swapping in the replacement squashfs image and 
syslinux directory.  I don't recall exactly why it was necessary, but 
suffice it to say it solved the problem and I could reboot the system and 
be running the alternate F10 image.

Now I'm doing it for real but instead of that alternate F10 spin, the 
upgrade would bring F13 into place.  Everything works smoothly -- the 
extlinux update runs without error and has the expected message indicating 
"/sysroot/syslinux is device /dev/sda1" -- right up until that final 
reboot which should take us into F13.  What happens?  Well, I get a 
"boot:" prompt for a second or two and then it disappears and then
repeats
endlessly.

I've tried a variety of things and have learned much, but not enough to 
understand the problem and have become rather perplexed.  Inspecting the 
CF card on my desktop system, I find what I'd expect.  I can make notes of 
kernel parameters as found in extlinux.conf, return the card to the 
device, power up and provide those same parameters at the "boot:"
prompt
and the device will boot into F13 just fine.  Once there I can manually 
run "extlinux --update /mnt/live/syslinux" and the system will reboot
fine
thereafter w/o intervention.  (The path changes, but is effectively the 
same directory, it's just mounted differently between the initrd and the 
regular running system.)  I have also manually run the "extlinux
--update"
from my desktop system (also F13) and found that also will "repair"
the CF
card to good working order.

My hunch is that either extlinux cannot find it's conf file or maybe the 
path difference between initrd and regular somehow does matter.  Neither 
seems likely though since my F10-F10 test upgrade worked.  That makes me 
wonder about using "extlinux --update" from F10 on a syslinux
directory
that was already populated by a F13 system and some compatibility issue. 
I've noticed that my manually run "extlinux --update" makes
extlinux.sys
grow a bit too as if the run from the initrd may have left something out, 
but I really know nothing about that file (yet).  Lastly, there's been a 
change in the "root=" kernel parameter between F10 and F13 that maybe
is
at play here.  In F10, this read as "root=UUID=blablabla" but now
reads as
"root=live:UUID=blablabla".  When I provide the kernel parameters
manually
at the "boot:" prompt I must use the F13 form for a fully successful
boot,
but even if I use the F10 format it get's much further than on its own -- 
it fails around the pivot_root point.  Maybe the "live" bit is
confusing
the syslinux package in the initrd of F10?

Any ideas would be greatly appreciated!

--
John Florian

On Thu, Oct 14, 2010 at 10:31,  <John.Florian at dart.biz>
wrote:
> Hello fellow syslinux/extlinux users! ?Please forgive the long post, but I
> believe my problem warrants the background info to understand the issue.
My comments are inline but I'd suggest you follow my advice at the end
of my email as a first thing to try.
> I've developed a specialized spin of Fedora that runs off CompactFlash.
> The spin is actually produced by the livecd-tools package and is then
> transferred to the CF card -- which has been formatted with ext3 -- using
> livecd-tools' livecd-iso-to-disk. ?I have hundreds of embedded systems
> running this spin. ?Every now and then we need to upgrade these systems
> and needless to say swapping CF cards is labor expensive. ?Thus I
> developed an in-place upgrade which essentially downloads a new squashfs
> image and syslinux directory, both of which are obtained from a newer
> spin. ?These downloads do not overwrite those of the running system; they
> get a name suffix of ".new". ?Then a special initrd is booted
into which
> looks for the new stuff and puts them in place after renaming the
> originals with an ".old" suffix. ?Another reboot occurs and the
new spin
> is then running. ?Currently I'm trying to effect an upgrade that will
take
> these systems from a Fedora 10 base to Fedora 13.
>
> Now the fun part and my problem. ?Before shipping CF cards out with F10 on
> them, I went through rigorous functional testing and one step was to
> verify that future upgrades would be successful as best that I could.
> Since F10 was the latest Fedora at the time it was only a pseudo-test that
> tried to upgrade the F10 spin to an alternate, distinguishable F10 spin. I
> made that work, but found it necessary to run "extlinux --update
> /sysroot/syslinux" after swapping in the replacement squashfs image
and
> syslinux directory. ?I don't recall exactly why it was necessary, but
> suffice it to say it solved the problem and I could reboot the system and
> be running the alternate F10 image.
Which version(s) of EXTLINUX?  If you're attempting to use an old
version of EXTLINUX to patch the extlinux.sys of a new version, it
probably won't work the way you want.  I think "extlinux --update"
also refreshes the VBR (Volume boot record), which contains a pointer
to the extlinux.sys file.

I normally take the approach that if the core
binary(extlinux.sys/ldlinux.sys) is moved (including the directory it
sits in), renamed, or modified, it should be redone with --update for
maximum reliability.  I can't say for certain if it's needed.  After
your initrd shuffles the directories around, it might be wise to
re-run your extlinux update (from the F13 installer, of course).
> Now I'm doing it for real but instead of that alternate F10 spin, the
> upgrade would bring F13 into place. ?Everything works smoothly -- the
> extlinux update runs without error and has the expected message indicating
> "/sysroot/syslinux is device /dev/sda1" -- right up until that
final
> reboot which should take us into F13. ?What happens? ?Well, I get a
> "boot:" prompt for a second or two and then it disappears and
then repeats
> endlessly.
Strange to say the least.  Is this "second or two" possibly your
TIMEOUT in your configuration?  Also, what does your configuration
look like?  Do your kernel, initrd, and squashfs sit in another
directory?

If you're running the F10 extlinux from your squashfs, it's probably
not accomplishing much.
> I've tried a variety of things and have learned much, but not enough to
> understand the problem and have become rather perplexed. ?Inspecting the
> CF card on my desktop system, I find what I'd expect. ?I can make notes
of
> kernel parameters as found in extlinux.conf, return the card to the
> device, power up and provide those same parameters at the "boot:"
prompt
> and the device will boot into F13 just fine. ?Once there I can manually
> run "extlinux --update /mnt/live/syslinux" and the system will
reboot fine
> thereafter w/o intervention. ?(The path changes, but is effectively the
> same directory, it's just mounted differently between the initrd and
the
> regular running system.) ?I have also manually run the "extlinux
--update"
> from my desktop system (also F13) and found that also will
"repair" the CF
> card to good working order.
Which version loaded?  Which config did it load?  It might be the old
version loading the old config still but I don't understand why it
just reboots.
> My hunch is that either extlinux cannot find it's conf file or maybe
the
> path difference between initrd and regular somehow does matter. ?Neither
> seems likely though since my F10-F10 test upgrade worked. ?That makes me
> wonder about using "extlinux --update" from F10 on a syslinux
directory
> that was already populated by a F13 system and some compatibility issue.
> I've noticed that my manually run "extlinux --update" makes
extlinux.sys
> grow a bit too as if the run from the initrd may have left something out,
> but I really know nothing about that file (yet). ?Lastly, there's been
a
> change in the "root=" kernel parameter between F10 and F13 that
maybe is
> at play here. ?In F10, this read as "root=UUID=blablabla" but now
reads as
> "root=live:UUID=blablabla". ?When I provide the kernel parameters
manually
> at the "boot:" prompt I must use the F13 form for a fully
successful boot,
> but even if I use the F10 format it get's much further than on its own
--
> it fails around the pivot_root point. ?Maybe the "live" bit is
confusing
> the syslinux package in the initrd of F10?
If it didn't find a configuration file, it would have complained
stating so.  The messages when there is no configuration file found
have improved through the versions.

Also, --update always installs a fresh copy of the core binary.  This
"growth" is either replacing the F10 version with F13 like you wanted
initially or effectively just patching it with the needed data.
> Any ideas would be greatly appreciated!
>
> --
> John Florian
Shuffle the directories to their final destination then run extlinux
from F13.  This probably can be done on the live system (assuming no
library issues) or from your initrd as your live system shouldn't have
any of those files open.

-- 
-Gene

> From: Gene Cumm <gene.cumm at gmail.com>
 
> My comments are inline but I'd suggest you follow my advice at the end
> of my email as a first thing to try.
First off, thanks for the reply!

I agree, so that's where I'm heading first and have already ran into 
difficulty -- detailed at the end.  I will also respond inline to you 
various questions just to be thorough.

> > I've developed a specialized spin of Fedora that runs off 
CompactFlash.
> > The spin is actually produced by the livecd-tools package and is then
> > transferred to the CF card -- which has been formatted with ext3 -- 
using
> > livecd-tools' livecd-iso-to-disk.  I have hundreds of embedded
systems
> > running this spin.  Every now and then we need to upgrade these 
systems
> > and needless to say swapping CF cards is labor expensive.  Thus I
> > developed an in-place upgrade which essentially downloads a new 
squashfs
> > image and syslinux directory, both of which are obtained from a newer
> > spin.  These downloads do not overwrite those of the running system; 
they
> > get a name suffix of ".new".  Then a special initrd is
booted into
which
> > looks for the new stuff and puts them in place after renaming the
> > originals with an ".old" suffix.  Another reboot occurs and
the new
spin
> > is then running.  Currently I'm trying to effect an upgrade that
will
take
> > these systems from a Fedora 10 base to Fedora 13.
> >
> > Now the fun part and my problem.  Before shipping CF cards out with 
F10 on
> > them, I went through rigorous functional testing and one step was to
> > verify that future upgrades would be successful as best that I could.
> > Since F10 was the latest Fedora at the time it was only a pseudo-test 
that
> > tried to upgrade the F10 spin to an alternate, distinguishable F10 
spin. I
> > made that work, but found it necessary to run "extlinux --update
> > /sysroot/syslinux" after swapping in the replacement squashfs
image
and
> > syslinux directory.  I don't recall exactly why it was necessary,
but
> > suffice it to say it solved the problem and I could reboot the system 
and
> > be running the alternate F10 image.
> 
> Which version(s) of EXTLINUX?

Fedora 13 is using syslinux-3.84 whereas Fedora 10 has syslinux-3.61.

>  If you're attempting to use an old
> version of EXTLINUX to patch the extlinux.sys of a new version, it
> probably won't work the way you want.

I was afraid I'd hear that.

> I think "extlinux --update"
> also refreshes the VBR (Volume boot record), which contains a pointer
> to the extlinux.sys file.
> 
> I normally take the approach that if the core
> binary(extlinux.sys/ldlinux.sys) is moved (including the directory it
> sits in), renamed, or modified, it should be redone with --update for
> maximum reliability.  I can't say for certain if it's needed.

I wish it were not necessary, but given the behavior I suspect it must. 
extlinux doesn't understand file systems like, say grub, does it?  Please 
correct me if I'm wrong, but it's my guess that extlinux behaves more
like
lilo than grub.

>  After
> your initrd shuffles the directories around, it might be wise to
> re-run your extlinux update (from the F13 installer, of course).
> 
> > Now I'm doing it for real but instead of that alternate F10 spin,
the
> > upgrade would bring F13 into place.  Everything works smoothly -- the
> > extlinux update runs without error and has the expected message 
indicating
> > "/sysroot/syslinux is device /dev/sda1" -- right up until
that final
> > reboot which should take us into F13.  What happens?  Well, I get a
> > "boot:" prompt for a second or two and then it disappears
and then
repeats
> > endlessly.
> 
> Strange to say the least.  Is this "second or two" possibly your
> TIMEOUT in your configuration?

Well, the timeout in the old (F10) extlinux.conf is 10 and the new (F13) 
is 30.  So if those are in 1/10s, I'd guess no.  I really wish all boot 
loaders would include their name and version in their prompt.  When trying 
to unravel chain-loading "boot:" doesn't tell one much.  I
don't think
there is any chain-loading going on here, but the Fedora Live images are 
originally intended for a bootable CD that then gets transferred to the CF 
card.  I know extlinux is here for sure; isolinux too?  I don't think so, 
but a detailed prompt would help alleviate such concerns.

>  Also, what does your configuration
> look like?  Do your kernel, initrd, and squashfs sit in another
> directory?

/mnt/live/syslinux
/mnt/live/syslinux/vmlinuz0
/mnt/live/syslinux/memtest
/mnt/live/syslinux/isolinux.bin
/mnt/live/syslinux/vesamenu.c32
/mnt/live/syslinux/extlinux.conf
/mnt/live/syslinux/splash.jpg
/mnt/live/syslinux/extlinux.sys
/mnt/live/syslinux/boot.cat
/mnt/live/syslinux/initrd0.img
/mnt/live/LiveOS
/mnt/live/LiveOS/osmin.img
/mnt/live/LiveOS/squashfs.img

Here /mnt/live is the root file system of the CF card.  The initrd0.img 
has a script to loop mount squashfs.img and find the regular kernel 
within.

 
> If you're running the F10 extlinux from your squashfs, it's
probably
> not accomplishing much.
> 
> > I've tried a variety of things and have learned much, but not
enough
to
> > understand the problem and have become rather perplexed.  Inspecting 
the
> > CF card on my desktop system, I find what I'd expect.  I can make 
notes of
> > kernel parameters as found in extlinux.conf, return the card to the
> > device, power up and provide those same parameters at the
"boot:"
prompt
> > and the device will boot into F13 just fine.  Once there I can 
manually
> > run "extlinux --update /mnt/live/syslinux" and the system
will reboot
fine
> > thereafter w/o intervention.  (The path changes, but is effectively 
the
> > same directory, it's just mounted differently between the initrd
and
the
> > regular running system.)  I have also manually run the "extlinux 
--update"
> > from my desktop system (also F13) and found that also will
"repair"
the CF
> > card to good working order.
> 
> Which version loaded?  Which config did it load?  It might be the old
> version loading the old config still but I don't understand why it
> just reboots.

I'm not sure I understand the Qs.  Are you asking what extlinux version 
loads after I've manually made the repair?  I don't know how to ask the 
"boot:" prompt to tell me anything useful here.  For example, if I
respond
with "help" all I get back is "Could not find kernel image:
help" and then
the perpetual two second timeout before it reboots unless I try typing 
something else before the timeout expires.

 
> > My hunch is that either extlinux cannot find it's conf file or
maybe
the
> > path difference between initrd and regular somehow does matter. 
 Neither
> > seems likely though since my F10-F10 test upgrade worked.  That makes 
me
> > wonder about using "extlinux --update" from F10 on a
syslinux
directory
> > that was already populated by a F13 system and some compatibility 
issue.
> > I've noticed that my manually run "extlinux --update"
makes
extlinux.sys
> > grow a bit too as if the run from the initrd may have left something 
out,
> > but I really know nothing about that file (yet).  Lastly, there's
been
a
> > change in the "root=" kernel parameter between F10 and F13
that maybe
is
> > at play here.  In F10, this read as "root=UUID=blablabla"
but now
reads as
> > "root=live:UUID=blablabla".  When I provide the kernel
parameters
manually
> > at the "boot:" prompt I must use the F13 form for a fully
successful
boot,
> > but even if I use the F10 format it get's much further than on its
own
--
> > it fails around the pivot_root point.  Maybe the "live" bit
is
confusing
> > the syslinux package in the initrd of F10?
> 
> If it didn't find a configuration file, it would have complained
> stating so.  The messages when there is no configuration file found
> have improved through the versions.

That alone is good to know.  Do you know if 3.61 would be that 
informative?


 
> Also, --update always installs a fresh copy of the core binary.  This
> "growth" is either replacing the F10 version with F13 like you
wanted
> initially or effectively just patching it with the needed data.

That was my guess, but good to have confirmed.

> Shuffle the directories to their final destination then run extlinux
> from F13.  This probably can be done on the live system (assuming no
> library issues) or from your initrd as your live system shouldn't have
> any of those files open.

Well that seems like the sanest thing to try next, but it isn't going to 
be easy.  I've gotten myself a shell from within the initrd and had 
intended to loop mount the new squashfs image so that I could run extlinux 
from there, but the mount fails from the F10-based initrd with "SQUASHFS 
error: Major/Minor mismatch, trying to mount newer 4.0 filesystem".  I 
seem to recall that Fedora did update SquashFS in this time frame.  I 
guess this means I'm going to need to identify and extract the various 
syslinux bits from an F13 system and provide them to the initrd outside of 
the squashfs.img.

--
John Florian