OpenPKG Advent Calendar 2006

Monday 2006-12-11: Backstage
What are "back-ported" security fixes?

The OpenPKG Enterprise product of the OpenPKG GmbH is mainly about a longer life-time (minimum 1 year instead of the maximum 4 months of OpenPKG Community) and the delivering of security updates based on back-ported security fixes. What does this "back-porting" mean and why is this only available with the OpenPKG Enterprise E1.X-SOLID branch and its predecessor (the OpenPKG Community 2.X-SOLID branch)?

The Anarchy of Open Source Vendors

Upstream Open Source software vendors are working completely independent from each other, both from a release schedule and release scope point of view. For instance some release early and often and with just small change sets per released software version, others release just on demand and then rather large change sets per released software version. In case a security incident occurs, the upstream vendor usually fixes its software and as fast as possible releases a new version of its software.

The crux is that between the time of the security incident and the last release date, the upstream vendor usually has incorporated lots of additional bugfixes and features into its software. As many upstream vendors do not follow a strict software engineering approach based on source code versioning, they either release just a single new version which contains the security fix plus the accumulated bugfixes and feature additions — all merged together. Or at best they additionally provide the security fix as an extracted stand-alone patch — but usually just against their previous single software version.

The Challenge of OpenPKG

The challenge for software distribution vendors like OpenPKG is that they have to update multiple branches and in each branch a different version of the upstream vendor's software. Here they have to master one or even two major challenges.

Security Fix Extraction

In case no stand-alone patch for the security fix is available, they have to reverse engineer it by extracting it from the total difference between the latest and the previous versions of the upstream vendor software. This task can be a matter of minutes or hours and days, mainly depending on how large the total difference is and how much the security fix is syntactically and semantically overlayed with other bugfixes or feature additions.

Security Fix Back-Porting

Second, the extracted patch (which is just against the previous version) has to be "back-ported" to all the particular versions contained in the packages of the distribution vendor's branch. This is again a matter of minutes or hours and even days. Sometimes the patch can be applied with some "fuzz factor", i.e., it still applies but with few lines of code offset. Sometimes some patch hunks can be applied manually only as their context completely changed. And sometimes the patch (or parts of it) has to be completely reverse engineered from scratch as the upstream vendor sources have already changed too much.

What makes all this "back-porting" really nasty is that (1) the distribution vendor is forced to work on totally foreign source code, (2) still has to guarranty its own customers backward compatibility and (3) do all this under high time pressure as the customer's services are affected and exploits are perhaps already available.

Back-Porting is Important but Expensive

For the production environments of customers it is very important to provide security updates based on back-ported security fixes as only this way the production environment can be still kept fully stable.

Everyone who has reasonable programming experience knows very well that nothing is more nasty than having to work on a piece of foreign code as different coding styles, unknown program workflows and other constraints easily sum up to a real problem. And on top of this, the time pressure aspect is what makes it a real practical nightmare for the engineers.

As usually nobody is willing to choose this just for fun, it is obvious that this important but expensive service cannot be provided in a free of charge offering like OpenPKG Community (at least not without lots of sponsoring). That's why this is available with the commercial OpenPKG Enterprise product only as only there people can be paid by the OpenPKG GmbH for doing this job. For the OpenPKG Community series the OpenPKG Foundation e.V. does a compromise: they also fix the security issue, but instead of using back-ported patches, they are just upgrading to the latest upstream vendor version at the accepted drawback of on-demand introducing incompatibilities.