The New [Debian] Security Build Infrastructure

  From:	 Anthony Towns   To:	 debian-devel-announce@lists.debian.org  Subject: The New Security Build Infrastructure  Date:	 Sat, 8 Jun 2002 18:26:46 +1000  

So, there's probably a whole bunch of people out there who're interested in the form new security infrastructure will take. If you're not, feel free to skip this message.

Historically, which is to say, currently, the security.debian.org archive has been managed pretty manually: a member of the security team will patch a program with a security bug and test it; then ssh to machines of each different architecture in the stable release and build the new source there; then, once this is done everywhere, prepare an advisory including the md5sums of all the newly built packages; then move the packages into the appropriate place on security.debian.org; then run dpkg-scanpackages and dpkg-scansources; and, finally, sign and send out the advisory. (The above is the general idea aiui, anyway. I'm sure if anyone cares the security team can correct the details)

This has a whole bunch of problems: since almost everything's done by hand, it's easy to forget to do something or to make mistakes, and this has occasionally resulted in broken security updates or confusing advisories. Additionally, it takes a lot of time and care, which can delay updates, and certainly makes the job a lot harder than it might otherwise be. But the real problem is it doesn't scale well: the ever growing number of packages alone gives the security team an ever increasing amount of work to do, with woody having twice the number of ports as potato we've made every single security update require twice as much effort from the security team, and that's just too much.

Well, that's the problem statement anyway: managing security updates, and especially getting them built, needs to be much easier.

So, what's the solution? For a fair while (at least a few months that I've been aware of), the security team were attempting to get "rbuilder" [0] installed on machines of each architecture so that they could use that to build security updates, but for various reasons didn't have much success. As such, in early to mid April, the buck was passed, although it wasn't for a couple of weeks after that that we figured out who it could reasonably be passed to.

The new buildd infrastructure consists of two components that've been on the agenda for a while now:

1. Converting security.debian.org to be run by katie
2. Modifiying the central wanna-build infrastructure to do "Accepted-Autobuilding"

Meanwhile, the latter is a nifty idea that the archive changes for crypto-in-main have allowed: since packages are checked (for signatures and such) within about fifteen minutes of them being uploaded, there's no longer any reason to have the autobuilders wait until the daily archive run before trying to build them. So it's basically a matter of setting up the buildds' sources.list to point to a new url [0], and add newly accepted packages to the wanna-build database, and -- presto chango! -- packages get automatically built thirty minutes or an hour or so after they're uploaded! As it happens, there're a bunch of little complications beyond this, to the point where the current setup is the third reimplementation of the above basic description in the past month or so, but you get the idea.

While the above makes for a good answer to the scalability problems the security team are facing -- the archive code and the autobuilders already cope with much larger numbers of packages than the security team produce -- there are a number of additional difficulties in trying to manage security uploads that have to be taken into account.

The major difference is that security updates frequently shouldn't be made public as soon as they're ready; in some cases they only need to wait until they can be tested further or an advisory written, in others they need to wait for a week or more to avoid publicising the flaw until all vendors have had a reasonable chance to fix it. This is pretty much in direct opposition to the way normal uploads work: we want those to be publically available as soon as possible. And, of course, there's an additional problem here in that in order to do accepted autobuilding, packages need to be made available to the autobuilders as early as possible, which is well before thay should be made available to the wider public.

So! How does it all work? Here's the general idea:

• a) Someone finds a security problem.

• b) Someone fixes the problem, and makes an upload to security.debian.org's incoming. That is to say, a package is uploaded to:

   	security.debian.org:/org/security.debian.org/queue/unchecked
     (aka ) 

  Its changelog needs to point it at "stable-security" or "testing-security" in place of the usual "unstable". It should be signed in the usual manner. It should, obviously, be tested fairly thoroughly. Anyone can do this, but presumably it'll usually be a member of the security team. If you're _not_ a member of the security team, you should definitely contact them before doing this. Further, while anyone can upload files here, only the security team can see what people have uploaded.

• c) The upload gets checked and processed by "jennifer" and moved into queue/accepted, and the buildds are notified. Files in here can be accessed by the security team and (somewhat indirectly) by the buildds.

• d) Security-enabled buildds pick up the source package (prioritized over normal builds), build it, and send the logs to the security team.

• e) The security team reply to the logs, and the newly built packages are uploaded to queue/unchecked, where they're processed by jennifer, and moved into queue/accepted.

• f) When the security team are happy with the source package (ie, that it's been correctly built for all applicable architectures and that it fixes the security hole and doesn't introduce new problems of its own) they run a script (viz, "amber") which:
  • installs the package into the security archive
  • updates the Packages, Sources and Release files in the usual way
  • sets up a template advisory that the security team can finish off
  • (optionally) forwards the packages to the appropriate proposed-updates so that it can be included in the real archive ASAP

And that's pretty much it.

All of the above has been implemented and tested in one way or another, although some of this hasn't been in entirely realistic conditions yet. It does, though, seem pretty safe to say that any further changes that'll be needed will be small bugfixes rather than redesigns or rewrites. katie for security.debian.org has been setup on satie.debian.org [1] and basically works, and the buildd changes are being live tested on the alpha, i386, powerpc and sparc autobuilders.

Credit for the katie implementation work go to James Troup, for the buildd changes, Ryan Murray, and for all the -devel and other flamewars, yours truly. ;)

What you can expect next is for the security infrastructure to move over to satie permanently and into the hands of the security team, and an update on all the other things that affect woody which have come up over the past month and a bit.

It's possible that some of these changes will let us handle security updates in better ways in future (I'm hopeful it'll let us does security updates for testing in a more timely manner, eg), but that probably won't become clear until the security team's had more of a chance to get comfortable with it all.

Cheers, aj (woody release manager)

[0] , as it happens. These are IP/password restricted and shouldn't be used except by autobuilders. If you want to poke around just for curiousity's sake, auric:/org/incoming.debian.org/ is one place to start.

[1] Also accessible via a reference to this project's codename, "security by obscurity", as obscurity.debian.org.

-- Anthony Towns