OpenPKG Advent Calendar 2006

Sunday 2006-12-10: Backstage
How the OpenPKG "Bootstrap" Works

One of the most essential but most tricky and hence far away from easy to understood parts of OpenPKG is the openpkg package, also known as our bootstrap. It allows one to bootstrap an OpenPKG instance from scratch by creating the filesystem area, and installing RPM, its database and a bunch of related tools into it.

OpenPKG RPM packaged by OpenPKG RPM itself

The main complexity of the OpenPKG RPM bootstrap package results from the fact that we force us to treat it equal to every other regular OpenPKG RPM package. First, this implies that the packaging tool RPM is packaged with itself as an OpenPKG RPM package (means: its build procedure is a real RPM *.spec file and it can be installed and upgraded through a binary or source RPM package). Second, RPM is installed into the same filesystem hierarchy as all other packages. Third, RPM manages its own files. The reason for this approach should be obvious: 100% consistency for the whole OpenPKG software packaging facility!

The drawback is that this package requires a very tricky bootstrapping procedure which had cost a lot of time to figure out and establish. If you ever wanted to know the gory details, here they are...

Emulated Package Building

The first step was that we wrote the regular openpkg.spec file for building the bootstrap package with OpenPKG RPM under the assumption that OpenPKG RPM is already available. This way we can provide OpenPKG RPM as an RPM package. Just remains the problem how we actually bootstrap in case where OpenPKG RPM is still not available, i.e., when we reach a new platform and have to build the package from scratch. Here the openpkg.boot script comes into play. It executes the openpkg.spec build procedure very similar to the way the real OpenPKG RPM would do ("openpkg rpm -bb"). That is, openpkg.boot partly emulates OpenPKG RPM — just enough that openpkg.spec works. As a result, openpkg.spec cannot use any fancy OpenPKG RPM features or other things before openpkg.boot is able to emulate it, of course.

Reduced Package Rebuilding

After openpkg.boot executed the %prep, %build and %install scripts of openpkg.spec, there is a fresh version of the target filesystem hierarchy staying under a temporary "build root". The openpkg.boot script then creates a very special temporary "openpkg rpm" command which allows the installed "openpkg rpm" command inside the "build root" to work (although it is built for the final target filesystem path). Then the $RPM_BOOT variable is set and the package is again build via openpkg.spec — but this time with the real OpenPKG RPM. To avoid unneccessary re-compilation, the openpkg.spec skips %prep, %build and %install sections if $RPM_BOOT is defined. So, on this second build phase, only the %files section is executed, i.e., a binary OpenPKG RPM package openpkg-V-R.P-T.rpm is rolled from the files in the "build root". Additionally, a source OpenPKG RPM package openpkg-V-R.src.rpm is rolled for consistency reasons.

Installing Package into its own Database

Finally, we override the installation in the "build root" again by installing the now rolled binary OpenPKG RPM package openpkg-V-R.P-T.rpm by using the real OpenPKG RPM. This way we achieve that OpenPKG RPM is remembered as a real OpenPKG RPM package in the RPM database. We just have to make sure the package is still relocated to the "build root" while installing. For this we could use "--prefix=$RPM_BUILD_ROOT%{l_prefix}", but this would create an incorrect file list for the package openpkg in the RPM database. Instead we use the tricky "--justdb" option for "openpkg rpm" which means "openpkg rpm" behaves as it would install into the real location, but does not actually install anything. But as a side-effect, the database inside the "build root" is now correct.

Rolling Bootstrap Shell Packages

After this procedure, the "build root" contains the target filesystem hierarchy with OpenPKG RPM installed with itself. What is now just remaining is to roll a bootstrap package openpkg with this stuff for initial installation without OpenPKG RPM. For this the "build root" is packed into a "tarball", compressed, again wrapped into another tarball together with the uncompression tools ("bzip2" and "tar"), and finally wrapped into a self-extracting shell script by appending the aux.wrapbin.sh script (padded to 64KB for easier unpacking of the attached tarball) to its front.

The result is the binary bootstrap script openpkg-V-R.P-T.sh which can be used to install the target hierarchy from scratch without any pre-installed OpenPKG RPM. Nevetheless, the installed target hierarchy looks exactly as it would have been installed with OpenPKG RPM. If one later wants to upgrade this hierarchy one can just use the generated (or a newer) openpkg-V-R.P-T.rpm.

To allow one to easily repeat this from-source bootstrapping procedure on other machines, one can run "./openpkg.boot -s" which rolls a openpkg-V-R.src.sh script which is a self-extracting script containing an attached tarball of the sources of this directory. This script contains the same contents like openpkg-V-R.src.rpm, but is intended for running the described bootstrapping procedure from scratch without any OpenPKG RPM.