Contents¶

Required tools¶

This packaging infrastructure relies on few basic tools.

Note

Debian has all the necessary tooling build rpm and create the rpm repo. However CentOS doesn’t have the debian tooling.

Warning

What is detailed here is only the common tools used to build packages and repo. For each package, you need to install its own build dependencies (stuff like gcc, python-setuptools, etc) if building outside a chroot.

Debian/Ubuntu tools¶

To install the Debian requirements:

# Debian/Ubuntu (deb)
$ apt-get install make debhelper reprepro cowbuilder

RHEL/CentOS/Fedora tools¶

To install the RHEL requirements:

# CentOS/RHEL (rpm)
$ yum install rpm-sign expect rpm-build createrepo make mock

# Fedora (rpm)
$ dnf install rpm-sign expect rpm-build createrepo make mock

Create a package¶

Creating a package foo involves the following steps:

Initialize the packaging directories.
Fill foo/Makefile (used for common metadata and upstream source recovery and preparation).
Do the distribution specific stuff (dependencies in debian/control and foo.spec, pre/post install scripts, init scripts, etc)
Build the package

Here is the general packaging workflow:

The steps in orange are common for all packages and must not be modified.
The steps in green are package specific, it’s those steps which must be customized for each package.

Initialize package skeleton¶

To create a package “foo” skeleton, run the follwing command:

$ ./common/init_pkg.sh -n foo

This script will create the following tree:

foo
├── buildenv -> ../common/buildenv
├── debian
│   ├── changelog
│   ├── compat
│   ├── conffiles
│   ├── control
│   ├── copyright
│   ├── foo.cron.d.ex
│   ├── foo.default.ex
│   ├── init.d.ex
│   ├── postinst.ex
│   ├── postrm.ex
│   ├── preinst.ex
│   ├── prerm.ex
│   ├── rules
│   └── source
│       └── format
├── rpm
│   └── component.spec
├── Makefile
└── MANIFEST

This tree contains two main directories, two main files, and a symlink:

debian: deb packaging stuff.
rpm: rpm packaging stuff (component.spec and optionally additional content like .service files).
Makefile: used to download and prepare upstream sources.
MANIFEST: listing the downloaded files and their hash.
buildenv: symlink to the shared build resources (Makefile.common, and various helper scripts).

Note

Don’t rename component.spec, build script for rpm expect this file to exist.

At this point, with default content, “empty” .rpm and .deb packages can be built:

$ cd foo/   # go in dir
# make help # display Makefile help
$ make deb  # build deb
$ make rpm  # build rpm
$ dpkg -c out/foo_0.0.1-1\~up+deb00_all.deb # look .deb content
$ rpm -qpl out/foo-0.0.1-1.00.noarch.rpm    # look .rpm content

Package metadata¶

It’s necessary to setup the package metadata (version, description) to their proper values. Package metadata are declared at the top of the package Makefile:

# Version
# if possible, keep the upstream version
VERSION=0.0.1

# Revision number
# increment it when fixing packaging for a given release
# reset it to 1 if VERSION is increased
RELEASE=1

# URL of the upstream project
URL=http://example.org/stuff

# short summary of what the package provides
SUMMARY=My package summary

# long version of the summary, (but I could be lazy)
DESCRIPTION=$(SUMMARY)

Note

During the package build, these variables are automatically substitute in packaging files. This is done by simple running sed -s ‘s|@VAR@|$(VAR)|’ on these files.

Don’t remove the @VAR@ (ex: @SUMMARY@, @URL@, @VERSION@) in the packaging files.

Download upstream sources¶

This packaging infrastructure comes with a small tool, ./common/buildenv/wget_sum.sh to handle downloads.

This tool role is:

Download upstream sources.
Check the integrity of the upstream source against the MANIFEST file (sha512 sum).
(Re)Build the MANIFEST file if requested.
Handle a local download cache to avoid downloading sources at each build.

Download tool usage¶

Inside the Makefile, use it as followed:

$(WGS) -u <url> -o $(BUILD_DIR)/<output file>

Example:

# Name of the package
NAME = libemf2svg

# Version
VERSION = 1.0.1

# URL of the project
URL=https://github.com/kakwa/libemf2svg

# Source recovery url
URL_SRC=$(URL)/archive/$(VERSION).tar.gz

# Including common rules and targets
include buildenv/Makefile.common

$(SOURCE_ARCHIVE): $(SOURCE_DIR) $(CACHE) Makefile MANIFEST
        $(WGS) -u $(URL_SRC) -o $(SOURCE_ARCHIVE)

Building the MANIFEST file¶

To create the MANIFEST file, just run the following command:

make manifest

Source preparation¶

The source preparation is made in the $(SOURCE_ARCHIVE) target.

The goal of this rule is to create the tar.gz archive $(SOURCE_ARCHIVE).

The root directory of the source archive should be $(NAME)-$(VERSION). For example:

tar -tvf cache/mk-sh-skel_1.0.0.orig.tar.gz
drwxrwxr-x root/root         0 2015-11-27 00:26 mk-sh-skel-1.0.0/
-rw-rw-r-- root/root      1135 2015-11-27 00:26 mk-sh-skel-1.0.0/LICENSE
-rw-rw-r-- root/root       145 2015-11-27 00:26 mk-sh-skel-1.0.0/Makefile
-rw-rw-r-- root/root       972 2015-11-27 00:26 mk-sh-skel-1.0.0/README.md
-rw-rw-r-- root/root      1037 2015-11-27 00:26 mk-sh-skel-1.0.0/mksh-skel

In ideal cases, it’s only a matter of downloading the upstream sources as these conventions are quite standards. For example:

# Version
VERSION = 1.0.1

# URL of the project
URL=https://github.com/kakwa/mk-sh-skel

# example of source recovery url
URL_SRC=$(URL)/archive/$(VERSION).tar.gz

# Basic source archive recovery,
# this works fine if upstream is clean
$(SOURCE_ARCHIVE): $(SOURCE_DIR) $(CACHE) Makefile MANIFEST
        $(WGS) -u $(URL_SRC) -o $(SOURCE_ARCHIVE)

But in some cases, it might be necessary to modify the upstream sources content.

For that two helper variables are provided:

$(SOURCE_DIR): source directory (with proper naming convention) where to put sources before building the source archive.
$(SOURCE_TAR_CMD): once $(SOURCE_DIR) is filled with content, just call this variable, it will generate the $(SOURCE_ARCHIVE) tar.gz and do some cleanup. If present, $(SOURCE_TAR_CMD) should be the last step in $(SOURCE_ARCHIVE) target.

For example:

# Version
VERSION = 1.0.7

# URL of the project
URL=http://repos.entrouvert.org/python-rfc3161.git

# example of source recovery url
URL_SRC=$(URL)/snapshot/python-rfc3161-$(VERSION).tar.gz

# preparation of the sources with removal of upstream, unwanted debian/ packaging
# it does the following:
# * recover upstream archive
# * uncompress it
# * upstream modification (remove the unwanted debian/ dir from upstream source)
# * move remaining stuff to $(SOURCE_DIR)
# * do some cleanup
# * build the archive

$(SOURCE_ARCHIVE): $(SOURCE_DIR) $(CACHE) Makefile MANIFEST
        $(WGS) -u $(URL_SRC) -o $(BUILD_DIR)/python-rfc3161-$(VERSION).tar.gz
        mkdir -p $(BUILD_DIR)/tmp
        tar -vxf $(BUILD_DIR)/$(NAME)-$(VERSION).tar.gz -C $(BUILD_DIR)/tmp
        rm -rf $(BUILD_DIR)/tmp/python-rfc3161-$(VERSION)/debian
        mv $(BUILD_DIR)/tmp/python-rfc3161-$(VERSION)/* $(SOURCE_DIR)
        rm -rf $(BUILD_DIR)/tmp
        rm -f $(BUILD_DIR)/python-rfc3161-$(VERSION).tar.gz
        $(SOURCE_TAR_CMD)

Distribution specific packaging¶

For the most part, just package according to deb/rpm documentation, filling the rpm/component.spec, debian/rules, debian/control, or any other packaging files if necessary.

Note

I would advise you to try to respect the distributions guidelines and standards such as the FHS (https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard).

deb¶

For Debian packages, just leverage the usual packaging patterns such as the PKG.init, PKG.default, PKG.service, ... files and the override_dh_* targets in debian/rules, and then, add your dependencies and architecture(s) in the debian/control file.

Note

In many cases, with clean upstreams, there is nearly nothing to do except dependencies and architecture, the various dh_helpers will do their magic and build a clean package.

If you are unlucky, uncomment the export DH_VERBOSE=1 in debian/rules and customize the build as necessary using the override_dh_* targets.

rpm¶

For rpm, fill the various sections of the rpm/component.spec file such as BuildRequires:, Requires: or BuildArch: parameters and the various sections like %install.

If additional files a required for packaging, an init script for example, put these files in the rpm/ directory.

All additional files in the rpm/ directory are copied in the rpmbuild SOURCES directory. This means that it’s possible to treat them as additional source files in component.spec with the Source[0-9]: directives.

Example for ldapcherry.service systemd service file and it’s associated files:

# rpm/ directory content
tree rpm/
rpm/
├── component.spec
├── ldapcherry
├── ldapcherry.conf
└── ldapcherry.service

# component.spec relevant sections
Source: %{pkgname}-%{version}.tar.gz
Source1: ldapcherry
Source2: ldapcherry.conf
Source3: ldapcherry.service

# install section
%install

# install the .service, the sysconfig file and tmpfiles.d (for pid file creation as non-root user)
mkdir -p %{buildroot}%{_unitdir}
mkdir -p %{buildroot}/usr/lib/tmpfiles.d/
mkdir -p %{buildroot}/etc/sysconfig/
install -pm644 %{SOURCE1} %{buildroot}/etc/sysconfig/
install -pm644 %{SOURCE2} %{buildroot}/usr/lib/tmpfiles.d/
install -pm644 %{SOURCE3} %{buildroot}%{_unitdir}

Distribution version specific packaging files¶

Depending on the OS version targeted, there might be some differences in packaging. A common difference is the dependency names.

For handling those cases, the present packaging framework provides a simple mechanism.

To override any file <FILE> in either the rpm/ or debian/ directories if targeting distribution version <DIST>, just create a new version of the <FILE> with the following name: <FILE>.dist.<DIST>.

For example, with the debian/control file and distribution jessie:

debian/control             # will be used as default
debian/control.dist.jessie # will be used build is called with DIST=jessie

It also permits to handle additional files for specific distribution versions.

Build a package¶

To build a package foo, just go in foo/pkg directory and run make deb and/or make rpm.

The resulting package(s) are located in the out directory. Source package(s) are in the src-out directory.

Cleaning cache, out and src-out directories¶

Example with with package python-asciigraph:

# go inside the component directory
$ cd python-asciigraph

# optionnally, clean cache/ and out/ directory
$ make clean

Build rpm package¶

Example with with package python-asciigraph:

# go inside the component directory
$ cd python-asciigraph

# build rpm package
$ make rpm

Here are the results:

# output packages:
$ ls out/
python-asciigraph-1.1.3-1.unk.noarch.rpm

# output source package
$ ls src-out/
python-asciigraph-1.1.3-1.unk.src.rpm

Build rpm inside a clean chroot¶

Warning

Not implemented yet

Build deb package¶

Example with with package python-asciigraph:

# go inside the component directory
$ cd python-asciigraph

# build deb package
$ make deb

Here are the results:

# output packages:
$ ls out/
python-asciigraph_1.1.3-1_all.deb

# output source package
$ ls src-out/
python-asciigraph_1.1.3-1.debian.tar.xz  python-asciigraph_1.1.3.orig.tar.gz
python-asciigraph_1.1.3-1.dsc

Build deb package inside a clean chroot¶

This build system can leverage cowbuilder from Debian to build in a clean chroot.

This is the recommended way to build packages targeted to be used in production.

Building in chroot is heavier but has multiple gains:

It permits to build in a clean environment every time
It rapidly exits in error if the build dependencies are not properly declared
It permits to target different version of Debian (stretch, jessie, wheezy)
It manages build dependencies, installing them automatically (if properly declared)
It permits to avoid having to install all build dependencies on your main system

# go inside the component directory
$ cd python-asciigraph

# build deb package for dist jessie
$ make deb_chroot DIST=jessie

Note

Building the chroot can be a long and heavy step but there are several way to accelerate it.

The first is to used a local mirror, this can be done using the DEB_MIRROR option when calling deb_chroot:

make deb_chroot DEB_MIRROR=http://your.local.mirror/debian

The second is to use a tmpfs for building, it requires a few GB of RAM however (at least 1.5GB per distro version targeted, but this may vary depending on the number packages and the size of their dependencies):

# as root
mount -t tmpfs -o size=16G tmpfs /var/cache/pbuilder/

# in fstab
tmpfs /var/cache/pbuilder/ tmpfs defaults,size=16G 0 0

Warning

Building in chroot requires root permission (it’s necessary for creating the chroot environment).

If make deb_chroot is run as a standard user, sudo will be used for cowbuilder calls.

The only command that needs to be white listed in sudoers configuration is cowbuilder:

# replace build-user with the user used to generate the packages
build-user ALL=(ALL) NOPASSWD: /usr/sbin/cowbuilder

Build the complete repositories¶

Repository metadata¶

The repository has a few metadata which must be filled:

# Name of the gpg key to use
GPG_KEY="kakwa"
# Output directory for the repos
OUTPUT="out/"
# Package provider
ORIGIN="kakwa"

Clean before build¶

Optionally, it’s possible to clean everything before build:

# optionnally, cleaning everything
$ make clean

Create the repositories¶

To build the repositories, just run:

Note

use -j <number of jobs> to run multiple packaging jobs in parallele

Note

use ERROR=skip to ignore package build failures when calling make <pkg>_repo and keep continuing building the repo.

Build deb repository¶

# create the deb repository
$ make deb_repo -j 4

# same ignoring individual package build errors
$ make deb_repo -j 4 ERROR=skip

Build the rpm repository¶

# create the rpm repository
$ make rpm_repo -j 4

Create both repo in one command¶

# create everything
$ make all -j 4

# same ignoring individual package build errors
$ make rpm_repo -j 4 ERROR=skip

Result repositories¶

The resulting repositories will look like that:

out
├── deb
│   └── sid
│       └── amd64
│           ├── conf
│           │   └── distributions
│           ├── db
│           │   ├── checksums.db
│           │   ├── contents.cache.db
│           │   ├── packages.db
│           │   ├── references.db
│           │   ├── release.caches.db
│           │   └── version
│           ├── dists
│           │   └── sid
│           │       └── contrib
│           │           └── binary-amd64
│           └── pool
│               └── contrib
│                   ├── d
│                   │   └── dwm-desktop
│                   │       └── dwm-desktop_5.9.0-1_amd64.deb
│                   ├── g
│                   │   └── gogs
│                   │       └── gogs_0.7.22-1_amd64.deb
│                   ├── m
│                   │   └── mksh-skel
│                   │       └── mksh-skel_1.0.0-1_all.deb
│                   └── p
│                       ├── python-asciigraph
│                       │   └── python-asciigraph_1.1.3-1_all.deb
│                       ├── python-dnscherry
│                       │   └── python-dnscherry_0.1.3-1_all.deb
│                       ├── python-ldapcherry
│                       │   └── python-ldapcherry_0.2.2-1_all.deb
│                       ├── python-ldapcherry-ppolicy-cracklib
│                       │   └── python-ldapcherry-ppolicy-cracklib_0.1.0-1_all.deb
│                       └── python-pygraph-redis
│                           └── python-pygraph-redis_0.2.1-1_all.deb
├── pub.gpg
└── rpm
    └── debU
        └── x86_64
            ├── repodata
            │   ├── 454e22ec768a30aa8e0c169454729501bbcd60f4365ce920d8125f2f4692d987-primary.xml.gz
            │   ├── 8f0383e61bd158979fd85db8a8e26a269b65f2327b183f99ba5139b559dd0336-other.xml.gz
            │   ├── a91c0afbd9bfef2cfb0a00fb3fe5a7490520dbf6d55ea098826cc6f253354552-other.sqlite.bz2
            │   ├── b49576332c4b8277aa173f57ee86b94db25edf2790e5712a39f22044c4c31669-filelists.xml.gz
            │   ├── b7cc2998becaa1b7c4592c3fa81fe5eca4bb522726d8634362cf2054ef01fae2-filelists.sqlite.bz2
            │   ├── e6e5b087813b07eef01de6cbfa9df8ec496affb79141cef026c28a812096dd4b-primary.sqlite.bz2
            │   └── repomd.xml
            └── RPMS
                ├── dwm-desktop-5.9.0-1.debU.x86_64.rpm
                ├── gogs-0.7.22-1.debU.x86_64.rpm
                ├── mksh-skel-1.0.0-1.debU.noarch.rpm
                ├── python-asciigraph-1.1.3-1.debU.noarch.rpm
                ├── python-dnscherry-0.1.3-1.debU.noarch.rpm
                ├── python-ldapcherry-0.2.2-1.debU.noarch.rpm
                ├── python-ldapcherry-ppolicy-cracklib-0.1.0-1.debU.noarch.rpm
                └── python-pygraph-redis-0.2.1-1.debU.noarch.rpm

Misc documentation¶

GPG cheat sheet¶

Package are signed by a gpg key.

Here are some useful commands to manage this key:

Generate the GPG key:

$ gpg --gen-key

List the keys:

$ gpg -K

Export the private key (multiple hosts):

$ gpg --export-secret-key -a "kakwa" > priv.gpg

Import the private key:

$ gpg --import priv.gpg

import the key in debian:

$ cat pub.gpg | apt-key add -

Name¶

amkecpak is just an anagram of makepack (for Make Package).

Motivation¶

I’ve implemented this set of Makefiles and script to more easily build .deb and .rpm packages.

I found that packaging .deb or .rpm is a little annoying for various reasons:

I’ve limited memory and I could not remember all the options of rpmbuild and dpkg-buildpackage. I deeply prefer having simple, easy to remember commands like make deb or make rpm.
I also wanted to build several packages in parallel easily.
I wanted to avoid upstream sources recovery by hand as it’s somewhat annoying.
I wanted to be capable to easily update my packages if a new upstream version is available.
I didn’t want huge build dependencies for this build system, only usual tools like make, tar, posix shell...
I didn’t want to mask the actual complexity of packaging software properly, I wanted to keep most of the knobs of regular .deb or .rpm packaging.

To achieve that, I used a set of Makefiles (for the parallel build capacity, the error handling, and the delta (re)build capacity). Plus some basic helper scripts to help with upstream source recovery, package initialization, or determining OS version.

I’ve also stolen some patterns I really liked from Gentoo.

Most upstream normalize how they ship sources. It’s generally a tar.gz file with a fixed pattern like <NAME>-<VERSION>.tar.gz, this pattern is even automatically implemented by hosting solution like GitHub.

Consequently it’s relatively easy to “templatize” a download URL with a $(VERSION) variable, this variable could also be used for setting the package version. This is generally how .ebuild files in Gentoo point to their respective upstreams.

Another thing I liked was that they keep a manifest file with the hashes of upstream sources. It permits to have a safe guard against modification of an existing upstream release, gaining some basic guaranties about build reproductibility and avoiding surprises...

Also the recovery of the upstream sources in Gentoo .ebuild files is a clearly distinct step from the build and install steps, no call to pip or wget in the middle of a compilation.

I’ve copied these ideas in this build “infrastructure”.

Resources¶

Amkecpak, a makefile based packaging framework.

Doc:	Documentation on ReadTheDoc
Dev:	GitHub
Author:	Pierre-Francois Carpentier - copyright © 2015

Packaging documentation in a nutshell¶

# Install the packaing tools
$ apt-get install make debhelper reprepro
# or
$ yum install rpm-sign expect rpm-build createrepo make

# Init a package foo
$ ./common/init_pkg.sh -n foo

$ cd foo/

# Implementing the package
$ vim Makefile
$ vim debian/rules ; vim debian/control
$ vim rpm/component.spec

# Help for the various targets
$ make help

# Building the packages
$ make deb
$ make rpm

$ cd ../

# gpg key generation (one time thing)
$ gpg --gen-key

# Building the repositories
# Use ERROR=skip to ignore package build failures and continue building the repo
$ make deb_repo -j 4 # ERROR=skip
$ make rpm_repo -j 4 # ERROR=skip

If you need more information, read the detailed documentation.