CentOS - Nov 2015 - synthesizing yum transactions

Ok, I'm trying way too hard to shovel myself out of a hole.

We have a bunch of remote CentOS 6 servers, that were configured
with kickstart.

They've subsequently had additional RPMs installed/updated via the
'rpm' utility.

We have reason to occasionally rewind the state of the server back
to it's original set of RPMs.

Of late, we've found that 'yum' handles this scenario quite nicely,
IFF (if and only if) that all RPM changes were handled via the 'yum'
utility itself, such that yum can manage a list of transactions.
In this case 'yum history rollback 1' does exactly what we need.

My problem is that we did not exclusively use 'yum' to do this work,
and even though there are newer RPMs in place, yum only knows about
transaction ID 1, from the initial kickstart installation.  Hence,
it thinks there's nothing to rollback from.

So, given the knowledge of what RPMs are currently installed, is
there a way to invent a new transaction incorporating them, without
reinstalling the RPMs in question?

I feel I've come close, with this methodology, basically coercing
yum to rebuild a database in a private directory:

  # installroot for yum
  workdir=/home/yum-hack

  rm -rf ${workdir}
  mkdir -p ${workdir}
  cd ${workdir}

  # list of newer RPMs (after transaction ID 1)
  new=new_rpms

  # dup repos and rpm/yum databases
  for tree in /var/lib/rpm/ /var/cache/yum/ /var/lib/yum/ /etc/yum.repos.d/; do
    echo ${tree}
    mkdir -p ./${tree}
    cp -pr ${tree}/* ./${tree}
  done

  cat new_rpm | xargs /usr/bin/yum -y --nogpgcheck --installroot=${workdir} \
    --setopt=tsflags=noscripts reinstall

On the surface, this seems to work, in that I now can see a transaction
list:

  yum --installroot=/home/yum-hack history list all

But, any effort to rollback to tid 1 yields:

  Transaction history is incomplete, before 5.
   You can use 'history rollback force', to try anyway.
  Error: Failed history rollback, incomplete

So, I've clearly tricked myself.

Does anyone have any suggestions about how I _should_ retroactively
invent an additional yum transaction?

This there some state I failed to introduce into the the above
private directory?

I've only begun to dive into the yum source to try to track down
what triggers this error, but if anyone has some immediate knowledge,
or suggestion for a better forum where I can ask this question?

Some details of my environment, if that's helpful:

  [root at 172-20-245-145 ~]# cat /etc/redhat-release
  CentOS release 6.5 (Final)
  [root at 172-20-245-145 ~]# rpm -q yum rpm
  yum-3.2.29-69.el6.centos.noarch
  rpm-4.8.0-37.el6.x86_64

Thanks for any advice...

-- 
Brian Reichert				<reichert at numachi.com>
BSD admin/developer at large

On Thu, Nov 05, 2015 at 11:39:41AM -0500, Brian Reichert
wrote:
> So, given the knowledge of what RPMs are currently installed, is
> there a way to invent a new transaction incorporating them, without
> reinstalling the RPMs in question?
> 
> I feel I've come close, with this methodology, basically coercing
> yum to rebuild a database in a private directory:
So, I'm closer, but I'm getting a new symptom.  My test setup:

First: to replicate the effect of not using yum consistently:

Install server from CentOS-6.7-x86_64-minimal.iso

Install repoquery, etc.:

	yum -y install yum-utils rsync

Install GCC and dependencies, but do so with the rpm utility:

	yum --setopt=keepcache=1 --downloadonly -y install gcc
	find /var/cache/yum/ -name \*.rpm | xargs rpm -U

This has the side effect of updating glibc, but yum doesn't have a
transaction related to that:

	[root at localhost ~]# rpm -q glibc
	glibc-2.12-1.166.el6_7.3.x86_64
	[root at localhost ~]# yum history package glibc
	Loaded plugins: fastestmirror
	ID     | Action(s)      | Package
	---------------------------------------------------------
		 1 | Install        | glibc-2.12-1.166.el6.x86_64
	Warning: RPMDB altered outside of yum.
	history package

The question is: how can I invent an additonal yum transaction such
that I can roll back to tid 1?

This looks like it works, up to a point, in terms of creating a new
transaction:

	workdir=/home/yum-hack
	rm -rf ${workdir}
	mkdir -p ${workdir}
	cd ${workdir}
	
	# Dup repos and rpm/yum databases
	for tree in /var/lib/rpm/ /var/cache/yum/ /var/lib/yum/ /etc/yum.repos.d/;
	do
  	  mkdir -p ./${tree}
  	  cp -pr ${tree}/* ./${tree}
	done

	# Reinstall new RPMs, via yum, in private work directory
	# Avoid scriplets; we don't want to depend on other RPMs

	find /var/cache/yum/ -name \*.rpm | xargs rpm -q -p | \
	  xargs /usr/bin/yum -y -C --nogpgcheck --installroot=${workdir} \
  	  --setopt=tsflags=noscripts reinstall

	# sync the private yum/rpm databases back out
	rsync -av --delete ${workdir}/var/lib/yum/ /var/lib/yum/
	rsync -av --delete ${workdir}/var/lib/rpm/ /var/lib/rpm/

Now, I see another transaction ID:

	[root at localhost ~]# yum history list
	Loaded plugins: fastestmirror
	ID     | Login user               | Date and time    | Action(s)      |
	Altered
	-------------------------------------------------------------------
     	3 | root <root>              | 2015-11-05 11:37 | Reinstall      |  
11
	<
     	2 | root <root>              | 2015-11-05 10:36 | Install        |  
2
	>
     	1 | System <unset>           | 2015-11-05 10:21 | Install        | 
205
	history list

Looks like I'm full of win, right?

However, when I then try a rollback to tid 1, it does undo the
yum-utils+ rsync transaction (tid 2), but then decides it's going
to reinstall everything from tid 3, for some reason.

	[root at localhost ~]# yum  history rollback force 1
	...
	Transaction Summary
	=========================================================	Remove        2
Package(s)
	Reinstall    11 Package(s)
	...

Obviously, I'm way off into uncharted/unsupported territory, but
if anyone has a more clue about what yum is doing here; I'd be
thrilled to hear from you...

-- 
Brian Reichert				<reichert at numachi.com>
BSD admin/developer at large