Tales from the underfunded cousin of DevOps, while trying to get research done.

These are my notes for creating .deb packages, mostly for my future self for the next time I need to do this. I'm putting them up here in case anybody else finds this useful. Note that I'm trying to strike a balance between doing "proper" debian packages that might eventually picked up by the Debian Med community or Debian in general, and getting stuff done quickly so I can do back to sciencing.

Some helpful ressources

There's some nice docs on Debian package building, but it's spread all over the place, and some of it is outdated. Also, I've found that none of the articles got me all the way to creating properly signed packages. Still, if you're completely new to Debian packaging, you might want to have a look at Building a Debian Package which explains how to modify an existing package, and also at the Intro to Debian Packaging, which covers the basics of creating a package from scratch. I will try to not repeat things in this post that are covered in these two documents, but instead deal with the things that I had to figure out the hard way over the last couple of days.

Supporting multiple distributions

One of my main difficulties was that I wanted to easily support packaging for both the current Debian stable release "jessie" (8.0) and the current Ubuntu long term support (LTS) release "trusty" (14.04). When I have some time, I might also want to build for the non-LTS Ubuntu release of the week, whatever that happens to be at a given moment. What I didn't want was to set up a virtual machine for every distro, because building on VMs is slow, and I'd have to figure out how to get my packaging signing key onto the VMs. I've decided to go with pbuilder, a build tool that utilises a chroot-based distribution installs. Doesn't help with multiple architectures, but honestly, if you want to run antiSMASH on your smartphone, you can build our ARM packages yourself. I'm fine with just supporting x86_64 at this point.

Publishing a Debian repository for apt

The main reason for creating Debian packages in the first place is to be able to automaticall install them (and get updates) via apt. So in addition to figuring out how to build debs, I also wanted to figure out creating a repository. The Debian docs list a gazillion options, I ended up going with aptly, which has great docs, is easy to install and seems to do all I need.

Initial setup

I'm doing all my builds in a directory called /data/packaging/deb. If you want to go for something different, replace the paths accordingly. Now, I set up some additional directories:

$ mkdir -p /data/packaging/deb/{jessie,trusty,pool}

Here, my jessie and trusty build files will go into the respective directories, and the generated files will be put into pool.

Then I also configure pbuiler using pbuilder-dist.

$ pbuilder-dist jessie
# grab a coffee while the basic chroot is created
$ pbuilder-dist trusty
# grab another coffee

Updating existing packages

I'll pick ncbi-blast+ for this example, because that was upgraded to 2.2.30 a while ago, but both Debian and Ubuntu currently ship older versions. As it turns out, there already is a version 2.2.30 package in Debian experimental, so let's go to the debian package page and grab the upstream tarball and Debian's changeset.

$ cd /data/packaging/deb
$ wget http://http.debian.net/debian/pool/main/n/ncbi-blast+/ncbi-blast+_2.2.30.orig.tar.gz
$ wget http://http.debian.net/debian/pool/main/n/ncbi-blast+/ncbi-blast+_2.2.30-2.debian.tar.xz
$ cd jessie
$ tar xf ../ncbi-blast+_2.2.30.orig.tar.gz
# create a symlink, the reason will be explained below
$ ln -s ../ncbi-blast+_2.2.30.orig.tar.gz ncbi-blast+_2.2.30~jessie.orig.tar.gz
$ cd ncbi-blast-2.2.30+-src
$ tar xf ../../ncbi-blast+_2.2.30-2.debian.tar.xz

Now we need to tweak a bit, because we want to build this for both jessie and trusty, and we want to keep the debs in the same pool later on. So we'll go in and cheat a bit and create .dsc files in two versions. One for jessie and one for trusty. Note: This probably doesn't matter for NCBI blast+, but I'm doing that with all packages for consistency's sake.

So, let's open the debian/changelog file and change the first line to

ncbi-blast+ (2.2.30~jessie-2) experimental; urgency=medium

Now, we can build the source package

$ debuild -S -us -uc

This is why we also created the symlink, so the debuild can find the tarball for our modified version 2.2.30~jessie. In the parent directory, this will end you up with a file called ncbi-blast+_2.2.30~jessie-2.dsc.

We can now do the proper build using pbuilder:

$ cd /data/packaging/deb/jessie
$ pbuilder-dist jessie build ncbi-blast+_2.2.30~jessie-2.dsc --buildresult ../pool

Once this is done, rinse and repeat for trusty.

Creating python library packages from scratch, the easy way

The easy way to create deb from Python libraries is to simply use Jordan Sissel's brilliant fpm tool. This isn't really cross-distribution-capable, but for pure python packages for python 2.7, this currently works fine for jessie and trusty. fpm makes creating a debian package as easy as

$ fpm -s python -t deb --iteration 1 straight.plugin==1.4.0-post-1

Note that I've picked a complicated example here, building straight.plugin in exactly version 1.4.0-post-1, as all other versions currently are broken for me. In any case, this leaves you with a Debian package for your library in a few seconds. Can't really beat that. As a bonus, creating RPMs with fpm is just as easy, just replace the -t deb with -t rpm. You really can't go wrong with fpm for stuff that you can easily package. Just don't mention this to Debian packaging people, they get a bit agressive about people not doing things "properly".

Creating "proper" python library packages from scratch

Sometimes, unfortunately, you run into a more complicated library that fpm won't handle, and then you have to do this the proper way. I'm basing this section on the Debian Python Library Style Guide, with my notes added, as the library style guide is just some guide, not a step-by-step instruction thing.

Let's use pysvg as an example. Grab it from PYPI. PYPI likes to give you zipfiles, for packaging we need a tarball, but that also gives us the opportunity to rename pysvg to python-pysvg, which conforms to the Debian way of naming things.

$ unzip pysvg-0.2.2.zip
$ mv pysvg-0.2.2/ python-pysvg-0.2.2
# Use XZ compression, modern Debian prefers that.
$ tar cJf python-pysvg_0.2.2.orig.tar.xz python-pysvg-0.2.2
$ mv python-pysvg-0.2.2 jessie

Now we need to create the debian/ directory and all the related packaging infrastructure. Fortunately, there's a package called python-stdeb that takes care of a lot of that already. Once installed, it integrates with setuptools, so we can use it from setup.py.

$ cd jessie/python-pysvg-0.2.2
$ python setup.py --command-packages=stdeb.command debianize

This has created a debian directory, but we need to adjust things a bit. First, again fix debian/changelog. Use dch to edit it, and just delete the autogenerated content the stdeb tool added and add your own changelog info.

We also need to create a debian/copyright file with licensing information. For pysvg, the file looks as follows:

Files: *
Copyright: 2008-2012 Kerim Mansour
License: BSD

Files: debian/*
Copyright: 2015 Kai Blin
License: BSD

License: BSD
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those
of the authors and should not be interpreted as representing official policies,
either expressed or implied, of the FreeBSD Project.

Now that this is out of the way, we can quicly create the ~jessie tarball and do the source package generation, followed by the proper pbuilder build.

$ ln -s ../../python-pysvg_0.2.2.orig.tar.xz ../pysvg_0.2.2~jessie.orig.tar.xz
$ debuild -S -us -uc
$ cd ..
$ pbuilder-dist jessie build pysvg_0.2.2~jessie-1.dsc --buildresult ../pool

Rinse and repeat for trusty.

To be continued

And as this blog post has grown huge already, I'll follow up with another one about building debs from scratch for compiled packages soon.


Comments

comments powered by Disqus