CentOS - Jun 2008 - Root-filesystem remounts as read-only during 5.2 upgrade (system completely shoot)

Judging from the frequency of my messages here one could think that
I'm too stupid to upgrade a workstation to 5.2 (but the servers I've
tried work without problem)

OK. The problem: I've tried to upgrade a 5.1-x86_64-workstation to 5.2.
During the upgrade immidiatly after (according to the
/var/log/messages) upgrading the two (64 & 32-bit) libgcc-packages an
EXT3-error occurs and the root-partition gets remounted in
read-only-mode. Consequently ALL following package upgrades throw
errors saying this or that can't be done because he can't write to
/etc, /usr but in the end yum says everything is OK.

(Side info: /var is on a different partition, so it is still
writeable)

At reboot the machine wants a manual fsck which throws a lot of
errors. 
Then the machine reboots with with a lot of error messages (basically
because it can't find /lib64/libgcc_s.so.1 (nothing a symlink can't
fix). After that I try to fix things by manually upgrading the
libgcc-packages (otherwise yum won'T run). "rpm -q" for selected
packages shows that the majority of the packages is still in 5.1-state.
Trying to "yum upgrade" again fail because the machine can't find
any
servers (but network functionality seems OK)

So I try to reboot. And now the really strange thing happens: A simple
"rpm -q rpm" says that no rpm is installed. Rebuilding the
rpm-database doesn't help.

So I'm a bit stuck here with a machine that is in limbo.

My questions: what could have been the cause for this (the machine was
working OK before that). The only thing I feel a little guilty about
was deinstalling the nvidia-kernel-driver and not reboot, but this
can't f### up the file-system, can it?

Any suggestions what I can do to get the machine into a normal state
again (apart from reinstall from scratch)

OK. I managed to beat the machine into submission. But a slight
incertainty remains.
>>>>> On Wed, 25 Jun 2008 18:34:26 +0200
>>>>> "BG" == Bernhard Gschaider <bgschaid_lists at
ice-sf.at> wrote:
    BG> Judging from the frequency of my messages here one could think
    BG> that I'm too stupid to upgrade a workstation to 5.2 (but the
    BG> servers I've tried work without problem)

    BG> OK. The problem: I've tried to upgrade a
    BG> 5.1-x86_64-workstation to 5.2.  During the upgrade immidiatly
    BG> after (according to the /var/log/messages) upgrading the two
    BG> (64 & 32-bit) libgcc-packages an EXT3-error occurs and the
    BG> root-partition gets remounted in read-only-mode. Consequently
    BG> ALL following package upgrades throw errors saying this or
    BG> that can't be done because he can't write to /etc, /usr but
in
    BG> the end yum says everything is OK.

    BG> (Side info: /var is on a different partition, so it is still
    BG> writeable)

That is my problem: Did that step of the "upgrade" leave the
rpm-database in a state that is not in tune with what is actually on
the disk

    BG> At reboot the machine wants a manual fsck which throws a lot
    BG> of errors. Then the machine reboots with with a lot of error
    BG> messages (basically because it can't find /lib64/libgcc_s.so.1
    BG> (nothing a symlink can't fix). After that I try to fix things
    BG> by manually upgrading the libgcc-packages (otherwise yum won'T
    BG> run). "rpm -q" for selected packages shows that the
majority
    BG> of the packages is still in 5.1-state.  Trying to "yum
    BG> upgrade" again fail because the machine can't find any
servers
    BG> (but network functionality seems OK)

The problem was that the centos-release was removed at the start of
the upgrade (and that is needed by yum to determine which $releasever
to use)

    BG> So I try to reboot. And now the really strange thing happens:
    BG> A simple "rpm -q rpm" says that no rpm is
    BG> installed. Rebuilding the rpm-database doesn't help.

    BG> So I'm a bit stuck here with a machine that is in limbo.

By manually reinstalling the rpm.rpm and some other packages I managed
to kickstart the upgrade again (and it seems to have succeeded)

Only problem: in the second upgrade run less packages were listed as
due to be updated (roughly a half). So I'm not sure: Are they marked
as upgraded in the rpm-database but in reality there are the old
versions on the disk. 

Is there a way to say: "Hey RPM, have a look whether really the files
in your database are on the disk)" ? 

    BG> My questions: what could have been the cause for this (the
    BG> machine was working OK before that). The only thing I feel a
    BG> little guilty about was deinstalling the nvidia-kernel-driver
    BG> and not reboot, but this can't f### up the file-system, can
    BG> it?

That question remains. But it is academic

    BG> Any suggestions what I can do to get the machine into a normal
    BG> state again (apart from reinstall from scratch)

As I said: solved

Thanks for listening