Debian and Ubuntu system administration tools

Welcome to the documentation of debuntu-tools version 0.9.1! The following sections are available:

User documentation

The readme is the best place to start reading, it’s targeted at all users:

debuntu-tools: Debian and Ubuntu system administration tools

The debuntu-tools package is my playground for experiments in automating system administration tasks on Debian and Ubuntu Linux systems. Right now there’s just four programs and no test suite, but I intend to keep working on this package in order make it a lot more useful. For usage instructions please refer to following sections and the documentation.

Contents

Status

Right now debuntu-tools is just an experiment and as such I’m not making any claims about or commitments towards usability, reliability or backwards compatibility. I guess we’ll see how long it’s going to take me to consider this more than an experiment 😊. The programs in this package have been manually tested on and are being used to manage headless Linux servers running Ubuntu 16.04 and 18.04.

Installation

The debuntu-tools package is available on PyPI which means installation should be as simple as:

$ pip install debuntu-tools

There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).

Requirements

• Several Python packages are required by debuntu-tools but installation of the Python package should automatically pull in those dependencies for you.
• The debuntu-kernel-manager program expects to be running on a Debian or Ubuntu derived Linux distribution, more specifically you need a functional dpkg installation. This enables version sorting according to the semantics used by dpkg, which is quite significant if your goal is to remove older kernels but preserve newer ones :-). To actually install and remove kernel packages you need apt-get and sudo privileges on the system whose kernels are being managed.
• The unlock-remote-system program expects a remote Linux system that has been configured in such a way that the pre-boot environment (the initial ramdisk) enables a static IP address and starts an SSH server like dropbear. More information about how to set this up is available in the documentation.
• The upgrade-remote-system builds on top of debuntu-kernel-manager as well as unlock-remote-system (in the form of reboot-remote-system) and so all of the requirements above apply.

Usage

There are two ways to use the debuntu-tools package:

1. The command line interfaces which are described below.
2. The Python API which is documented on Read the Docs.

The following programs are documented here:

debuntu-kernel-manager

Usage: debuntu-kernel-manager [OPTIONS] – [APT_OPTIONS]

Detect and remove old Linux kernel header, image and modules packages that can be safely removed to conserve disk space and speed up apt-get runs that install or remove kernels.

By default old packages are detected and reported on the command line but no changes are made. To actually remove old packages you need to use the -c, --clean or --remove option. Using the following command you can perform a dry run that shows you what will happen without actually doing it:

$ debuntu-kernel-manager --remove -- --dry-run

The debuntu-kernel-manager program is currently in alpha status, which means a first attempt at a usable program has been published but there are no guarantees about what it actually does. You have been warned :-).

Supported options:

Option	Description
-c, --clean, --remove	Remove Linux kernel header and/or image packages that are deemed to be safe to remove. The use of this option requires sudo access on the system in order to run the ‘apt-get remove’ command.
-f, --force	When more than one Linux kernel meta package is installed the -c, --clean and --remove options will refuse to run apt-get and exit with an error instead. Use the -f or --force option to override this sanity check.
-p, --preserve-count=NUMBER	Preserve the NUMBER newest versions of the kernel packages (defaults to 2).
-r, --remote-host=ALIAS	Detect and remove old Linux kernel header and image packages on a remote host over SSH. The ALIAS argument gives the SSH alias that should be used to connect to the remote host.
-v, --verbose	Increase verbosity (can be repeated).
-q, --quiet	Decrease verbosity (can be repeated).
-h, --help	Show this message and exit.

debuntu-nodejs-installer

Usage: debuntu-nodejs-installer [OPTIONS]

Install an up to date Node.js binary distribution on a Debian or Ubuntu system by configuring and using the NodeSource binary package repositories.

Due to the time it takes for new software releases to find their way into the Debian and Ubuntu ecosystems versus the speed with which the Node.js community is currently moving, the system packages that provide Node.js are hopelessly out of date. Fortunately the folks at NodeSource maintain Debian and Ubuntu package repositories that provide up to date Node.js binary distributions.

NodeSource makes installation scripts available and the suggested way to run these is to download and pipe them straight to a shell. That kind of rubs me the wrong way :-) but I’ve nevertheless had to set up NodeSource installations a dozen times now. One thing led to another and now there is this program.

Supported options:

Option	Description
-i, --install	Configure the system to use one of the NodeSource binary package repositories and install the ‘nodejs’ package from the repository.
-V, --version=NODEJS_VERSION	Set the version of Node.js to be installed. You can find a list of available versions on the following web page: https://github.com/nodesource/distributions/ Default: node_10.x (active LTS)
-s, --sources-file=FILENAME	Set the pathname of the ‘package resource list’ that will be added to the system during configuration of the NodeSource binary package repository. Default: /etc/apt/sources.list.d/nodesource.list
-r, --remote-host=ALIAS	Perform the requested action(s) on a remote host over SSH. The ALIAS argument gives the SSH alias that should be used to connect to the remote host.
-v, --verbose	Increase verbosity (can be repeated).
-q, --quiet	Decrease verbosity (can be repeated).
-h, --help	Show this message and exit.

reboot-remote-system

Usage: reboot-remote-system [OPTIONS] [SSH_ALIAS]

Reboot a remote system and wait for the system to come back online. If the SSH alias matches a section in the ‘unlock-remote-system’ configuration, the root disk encryption of the remote system will be unlocked after it is rebooted.

Supported options:

Option	Description
-s, --shell	Start an interactive shell on the remote system after it has finished booting.
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

unlock-remote-system

Usage: unlock-remote-system [OPTIONS] PRE_BOOT [POST_BOOT]

Boot a remote Linux system that’s waiting for the root disk encryption password to be entered into an interactive prompt by connecting to the remote system over the network using SSH and entering the password non-interactively. The remote Linux system needs to be configured in such a way that the pre-boot environment enables a static IP address and starts an SSH server like Dropbear.

The PRE_BOOT argument defines how to connect to the pre-boot environment:

• Its value is assumed to be a host name, IP address or SSH alias.
• It can optionally start with a username followed by an ‘@’ sign.
• It can optionally end with a ‘:’ followed by a port number.

The default username is ‘root’ and the default port number 22. The optional POST_BOOT argument defines how to connect to the post-boot environment, this is useful when the pre and post-boot environments run SSH servers on different port numbers.

If the PRE_BOOT argument matches the name of a user defined configuration section the options in that section define how unlock-remote-system operates.

Supported options:

Option	Description
-i, --identity-file=KEY_FILE	Use the private key stored in KEY_FILE for SSH connections to the pre-boot environment. The post-boot environment is expected to use your default private key or have a suitable configuration in ~/.ssh/config.
-k, --known-hosts=HOSTS_FILE	Use HOSTS_FILE as the “known hosts file” for SSH connections to the pre-boot environment. When this option is not given host key verification will be disabled to avoid conflicts between the host keys of the different SSH servers running in the pre-boot and post-boot environments.
-p, --password=NAME	Get the password for the root disk encryption of the remote system from the local password store in ~/.password-store using the ‘pass’ program. The NAME argument gives the full name of the password.
-r, --remote-host=SSH_ALIAS	Connect to the remote system through an SSH proxy.
-s, --shell	Start an interactive shell on the remote system after it has finished booting.
-w, --watch	Start monitoring the remote system and automatically unlock the root disk encryption when the remote system is rebooted. The monitoring continues indefinitely.
-a, --all	Enable monitoring of all configured systems when combined with --watch.
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

upgrade-remote-system

Usage: upgrade-remote-system [OPTIONS] [SSH_ALIAS]

Upgrade the system packages on a remote Debian or Ubuntu system, reboot the system when this is required due to security updates or because the system isn’t yet running the newest kernel, remove old Linux kernel and header packages and optionally remove ‘auto-removable’ system packages.

If the given SSH alias matches a section in the ‘unlock-remote-system’ configuration, the root disk encryption of the remote system will be automatically unlocked when the system is rebooted.

Supported options:

Option	Description
-s, --shell	Start an interactive shell on the remote system afterwards.
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

Configuration files

unlock-remote-system

Configuration files are text files in the subset of ini syntax supported by Python’s configparser module. They can be located in the following places:

Directory	Main configuration file	Modular configuration files
/etc	/etc/unlock-remote-system.ini	/etc/unlock-remote-system.d/*.ini
~	~/.unlock-remote-system.ini	~/.unlock-remote-system.d/*.ini
~/.config	~/.config/unlock-remote-system.ini	~/.config/unlock-remote-system.d/*.ini

The available configuration files are loaded in the order given above, so that user specific configuration files override system wide configuration files.

Each section of the configuration applies to a single host. The following options are supported in these sections:

Configuration option	Default value
boot-timeout	5 minutes
connect-timeout	60 seconds
cryptroot-config	/conf/conf.d/cryptroot
cryptroot-program	/scripts/local-top/cryptroot
key-script	/tmp/keyscript.sh
known-hosts-file	(no value)
named-pipe	/lib/cryptsetup/passfifo
password	(no value)
password-name	(no value)
password-store	(no value)
post-boot	(no value)
pre-boot	(no value)
retry-interval	1 second
scan-timeout	5 seconds
ssh-proxy	(no value)

The links in the table above lead to the Python API documentation which explains the purpose of each of these options.

Contact

The latest version of debuntu-tools is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.

License

This software is licensed under the MIT license.

© 2018 Peter Odding.

Miscellaneous documents:

Remote root disk encryption

This document contains notes about setting up remote root disk encryption on a Linux server. My experiences are based on Ubuntu 12.04, 14.04, 16.04 and 18.04 but I’m guessing most of these things will work very similar on Debian and its derivatives. In fact some of these notes apply to all Linux distributions.

We will configure the system to enable network connectivity in the initramfs so that we can connect using SSH and inject the root disk encryption pass phrase into the boot process. This makes it possible to set this up for headless servers.

Contents

Set up root disk encryption

It’s much easier to start an Ubuntu installation with root disk encryption than to set this up later [1] because the installer takes care of a lot of the details involved. Of course doing this on a headless server assumes you have access to something like a VNC connection.

[1]	It is possible to add root disk encryption to an existing system, but it can be a bit tricky to get all the details right and not end up with a system that won’t boot :-).

Enable SSH in the initramfs

Dropbear is a lightweight SSH server and BusyBox is a lightweight collection of command line utilities. We’ll need both of these in our initramfs. They can be installed on Debian/Ubuntu using the following command:

$ sudo apt-get install busybox dropbear

This should [2] automatically configure Dropbear to start in the initramfs using the shell script /usr/share/initramfs-tools/hooks/dropbear. Depending on the version of the dropbear package this may or may not generate an SSH public/private key pair during installation:

• If a key pair is generated then the private key file will be located at /etc/initramfs-tools/root/.ssh/id_rsa. You can download this file and remove it from the server, because it won’t be needed there.
• If a key pair isn’t generated, or if you prefer to use your own key pair, you can install your own public key into the initramfs (see the next section).

[2]

The script /usr/share/initramfs-tools/hooks/dropbear should automatically include Dropbear in the initramfs image when it detects root disk encryption. If this doesn’t work you can force Dropbear to be included by adding (or uncommenting) the line DROPBEAR=y in /usr/share/initramfs-tools/conf-hooks.d/dropbear.

Install your own SSH key

You can install your own public SSH key using the following steps:

On Ubuntu >= 18.04

1. Open the file /etc/dropbear-initramfs/authorized_keys in a text editor of your choosing to add your public key:

   $ sudo vim /etc/dropbear-initramfs/authorized_keys
   

2. Update your initramfs images to include the key you added:

   $ sudo update-initramfs -uk all
   

On Ubuntu < 18.04

1. First make sure the /etc/initramfs-tools/root/.ssh directory exists:

   $ sudo mkdir -p /etc/initramfs-tools/root/.ssh
   

2. Open the file /etc/initramfs-tools/root/.ssh/authorized_keys in a text editor of your choosing to add your public key:

   $ sudo vim /etc/initramfs-tools/root/.ssh/authorized_keys
   

3. Update your initramfs images to include the key you added:

   $ sudo update-initramfs -uk all
   

Enable networking in the initramfs

The network interfaces of Debian/Ubuntu servers are usually configured in /etc/network/interfaces which is not available in initramfs images. What we can do instead is to provide a suitable (static) network configuration directly to the kernel via a kernel parameter. On Debian/Ubuntu this is done by changing the GRUB_CMDLINE_LINUX variable defined in the configuration file /etc/default/grub. Here’s an example based on my VPS:

GRUB_CMDLINE_LINUX="ip=149.210.193.173::149.210.193.1:255.255.255.0::eth0:none"

As you can see this compresses an entire static network interface configuration into a single ip=… kernel parameter, with fields separated by colons. The kernel documentation contains details on the ‘ip’ kernel parameter, but for your convenience here is an overview of the supported fields:

ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns0-ip>:<dns1-ip>

To elaborate even further, here is how the values in the example above map to the named fields:

• client-ip is 149.210.193.173
• server-ip is empty (we’re not using an NFS root)
• gw-ip is the gateway IP, in this case 149.210.193.1 (if you hadn’t already guessed, my VPS is hosted at TransIP 😉)
• netmask is 255.255.255.0
• hostname is empty (we’re not using DHCP so there’s no point in configuring a DHCP client id)
• device is eth0
• autoconf is none (to disable autoconfiguration)

Make sure to run the following command after editing /etc/default/grub:

$ sudo update-grub

Consistent device naming

The example above uses the device name eth0 however with the introduction of consistent network device naming the eth* names were retired. If you need or want them back you can add the kernel parameters biosdevname=0 and net.ifnames=0 to the $GRUB_CMDLINE_LINUX variable.

On a Raspberry Pi

In June 2018 I got Ubuntu 16.04 with root disk encryption running on a Raspberry Pi. Because the boot process of a Raspberry Pi is very different from a regular computer, the way you configure it is also different. In this case I needed to add the ip=… kernel parameter to the file /boot/firmware/cmdline.txt.

Slow boot issues

When the ip= kernel parameter and /etc/network/interfaces both define a static network interface configuration, you may encounter slow boot issues. If you were to look at the messages emitted by the boot process you would most likely see a message along the lines of:

Waiting for network configuration..

This can slow down the boot process by two or three minutes, making you doubt whether a server is going to come back online! Fortunately there’s an easy way to avoid this problem. Open /etc/network/interfaces in your favorite text editor and add the line pre-up ip addr flush dev eth0, similar to this:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 149.210.193.173
  netmask 255.255.255.0
  gateway 149.210.193.1
  pre-up ip addr flush dev eth0

I originally found this trick on the Ubuntu Forums in October 2014 when I created my first headless server with root disk encryption based on Ubuntu 12.04 and I still need the workaround at the time of writing, on that same server, which has since been upgraded to 14.04 and then to 16.04.

External references

• The cryptsetup package on Debian/Ubuntu contains notes on how to setup remote unlocking in /usr/share/doc/cryptsetup/README.remote.gz, this is how I initially got started back in 2014.
• The StackExchange question SSH to decrypt encrypted LVM during headless server boot? received some interesting answers including a honorable mention of Mandos [3].

[3]	You would not believe how much time I’ve invested in getting Mandos to unlock my servers unattended, I even went so far as to (cross) compile the “latest & greatest” versions for multiple CPU architectures in a desperate attempt to get it to work. I never did.

Sample output of debuntu-kernel-manager

On this page you can find sample outputs of debuntu-kernel-manager.

Contents

Removal forbidden

The following sample output shows that debuntu-kernel-manager detects the presence of multiple Linux kernel image meta packages and refuses to remove any packages.

2016-06-15 18:22:56 VERBOSE Sanity checking meta packages on remote system (package-sandbox) ..
2016-06-15 18:22:56 INFO Found 2 installed Linux kernel image meta packages:
2016-06-15 18:22:56 INFO  - linux-image-generic-lts-wily (4.2.0.38.31)
2016-06-15 18:22:56 INFO  - linux-image-generic-lts-xenial (4.4.0.24.14)
2016-06-15 18:22:56 WARNING You have more than one Linux kernel image meta package installed (linux-image-generic-lts-wily and linux-image-generic-lts-xenial) which means automatic package removal can be unreliable!
2016-06-15 18:22:56 VERBOSE It's probably best to stick to one Linux kernel image meta package, preferably the one that matches the newest kernel :-)
2016-06-15 18:22:56 INFO Found 1 installed Linux kernel header meta package:
2016-06-15 18:22:56 INFO  - linux-headers-generic-lts-xenial (4.4.0.24.14)
2016-06-15 18:22:56 VERBOSE Checking for removable packages on remote system (package-sandbox) ..
2016-06-15 18:22:57 INFO Found 9 installed Linux kernel image packages:
2016-06-15 18:22:57 INFO  - linux-image-3.13.0-63-generic and linux-image-extra-3.13.0-63-generic (removable)
2016-06-15 18:22:57 INFO  - linux-image-3.13.0-73-generic (removable)
2016-06-15 18:22:57 INFO  - linux-image-4.2.0-38-generic and linux-image-extra-4.2.0-38-generic (removable)
2016-06-15 18:22:57 INFO  - linux-image-4.4.0-21-generic and linux-image-4.4.0-21-lowlatency (one of 2 newest kernels)
2016-06-15 18:22:57 INFO  - linux-image-4.4.0-24-generic and linux-image-extra-4.4.0-24-generic (the active kernel)
2016-06-15 18:22:57 INFO Found 5 installed Linux kernel header packages:
2016-06-15 18:22:57 INFO  - linux-headers-3.13.0-63 and linux-headers-3.13.0-63-generic (removable)
2016-06-15 18:22:57 INFO  - linux-headers-3.13.0-88 (removable)
2016-06-15 18:22:57 INFO  - linux-headers-4.4.0-24 and linux-headers-4.4.0-24-generic (the active kernel)
2016-06-15 18:22:57 INFO Found 8 Linux kernel packages that can be removed.
2016-06-15 18:22:57 VERBOSE Command to remove packages: apt-get remove --purge linux-headers-3.13.0-88 linux-headers-3.13.0-63-generic linux-image-extra-3.13.0-63-generic linux-headers-3.13.0-63 linux-image-3.13.0-63-generic linux-image-3.13.0-73-generic linux-image-extra-4.2.0-38-generic linux-image-4.2.0-38-generic
2016-06-15 18:22:57 ERROR Refusing to cleanup kernel related packages on remote system (package-sandbox) because results can be unreliable when multiple Linux kernel meta packages are installed! You can use the -f, --force option to override this sanity check.

Forced removal

The following sample output shows that debuntu-kernel-manager detects the presence of multiple Linux kernel image meta packages but blindly marches on at the explicit request of the operator (it’s just a dry run after all :-). The result is what I intended to do though!

2016-06-15 18:22:57 VERBOSE Sanity checking meta packages on remote system (package-sandbox) ..
2016-06-15 18:22:58 INFO Found 2 installed Linux kernel image meta packages:
2016-06-15 18:22:58 INFO  - linux-image-generic-lts-wily (4.2.0.38.31)
2016-06-15 18:22:58 INFO  - linux-image-generic-lts-xenial (4.4.0.24.14)
2016-06-15 18:22:58 WARNING You have more than one Linux kernel image meta package installed (linux-image-generic-lts-wily and linux-image-generic-lts-xenial) which means automatic package removal can be unreliable!
2016-06-15 18:22:58 VERBOSE It's probably best to stick to one Linux kernel image meta package, preferably the one that matches the newest kernel :-)
2016-06-15 18:22:58 INFO Found 1 installed Linux kernel header meta package:
2016-06-15 18:22:58 INFO  - linux-headers-generic-lts-xenial (4.4.0.24.14)
2016-06-15 18:22:58 VERBOSE Checking for removable packages on remote system (package-sandbox) ..
2016-06-15 18:22:58 INFO Found 9 installed Linux kernel image packages:
2016-06-15 18:22:59 INFO  - linux-image-3.13.0-63-generic and linux-image-extra-3.13.0-63-generic (removable)
2016-06-15 18:22:59 INFO  - linux-image-3.13.0-73-generic (removable)
2016-06-15 18:22:59 INFO  - linux-image-4.2.0-38-generic and linux-image-extra-4.2.0-38-generic (removable)
2016-06-15 18:22:59 INFO  - linux-image-4.4.0-21-generic and linux-image-4.4.0-21-lowlatency (one of 2 newest kernels)
2016-06-15 18:22:59 INFO  - linux-image-4.4.0-24-generic and linux-image-extra-4.4.0-24-generic (the active kernel)
2016-06-15 18:22:59 INFO Found 5 installed Linux kernel header packages:
2016-06-15 18:22:59 INFO  - linux-headers-3.13.0-63 and linux-headers-3.13.0-63-generic (removable)
2016-06-15 18:22:59 INFO  - linux-headers-3.13.0-88 (removable)
2016-06-15 18:22:59 INFO  - linux-headers-4.4.0-24 and linux-headers-4.4.0-24-generic (the active kernel)
2016-06-15 18:22:59 INFO Found 8 Linux kernel packages that can be removed.
2016-06-15 18:22:59 VERBOSE Command to remove packages: apt-get remove --purge --dry-run --quiet --quiet linux-headers-3.13.0-88 linux-headers-3.13.0-63-generic linux-image-extra-3.13.0-63-generic linux-headers-3.13.0-63 linux-image-3.13.0-63-generic linux-image-3.13.0-73-generic linux-image-extra-4.2.0-38-generic linux-image-4.2.0-38-generic
2016-06-15 18:22:59 INFO Removing 8 packages on remote system (package-sandbox) ..
Pseudo-terminal will not be allocated because stdin is not a terminal.
Purg linux-headers-3.13.0-63-generic [3.13.0-63.103]
Purg linux-headers-3.13.0-63 [3.13.0-63.103]
Purg linux-headers-3.13.0-88 [3.13.0-88.135]
Purg linux-image-extra-3.13.0-63-generic [3.13.0-63.103]
Purg linux-image-3.13.0-63-generic [3.13.0-63.103]
Purg linux-image-3.13.0-73-generic [3.13.0-73.116]
Purg linux-image-generic-lts-wily [4.2.0.38.31]
Purg linux-image-extra-4.2.0-38-generic [4.2.0-38.45~14.04.1]
Purg linux-image-4.2.0-38-generic [4.2.0-38.45~14.04.1]
2016-06-15 18:23:00 INFO Done! (took 2.56 seconds)

API documentation

The following API documentation is automatically generated from the source code:

API documentation

The following documentation is based on the source code of version 0.9.1 of the debuntu-tools package.

Available modules

debuntu_tools

The top level debuntu_tools module.

debuntu_tools.__version__ = '0.9.1'

The global version number of the debuntu-tools package (a string).

debuntu_tools.start_interactive_shell(context)

Start an interactive shell in the given execution context.

Parameters:context – An execution context created by executor.contexts.

Swallows return code 130 which can be caused by the operator typing Control-C followed by Control-D.

debuntu_tools.kernel_manager

Usage: debuntu-kernel-manager [OPTIONS] – [APT_OPTIONS]

Detect and remove old Linux kernel header, image and modules packages that can be safely removed to conserve disk space and speed up apt-get runs that install or remove kernels.

By default old packages are detected and reported on the command line but no changes are made. To actually remove old packages you need to use the -c, --clean or --remove option. Using the following command you can perform a dry run that shows you what will happen without actually doing it:

$ debuntu-kernel-manager --remove -- --dry-run

The debuntu-kernel-manager program is currently in alpha status, which means a first attempt at a usable program has been published but there are no guarantees about what it actually does. You have been warned :-).

Supported options:

Option	Description
-c, --clean, --remove	Remove Linux kernel header and/or image packages that are deemed to be safe to remove. The use of this option requires sudo access on the system in order to run the ‘apt-get remove’ command.
-f, --force	When more than one Linux kernel meta package is installed the -c, --clean and --remove options will refuse to run apt-get and exit with an error instead. Use the -f or --force option to override this sanity check.
-p, --preserve-count=NUMBER	Preserve the NUMBER newest versions of the kernel packages (defaults to 2).
-r, --remote-host=ALIAS	Detect and remove old Linux kernel header and image packages on a remote host over SSH. The ALIAS argument gives the SSH alias that should be used to connect to the remote host.
-v, --verbose	Increase verbosity (can be repeated).
-q, --quiet	Decrease verbosity (can be repeated).
-h, --help	Show this message and exit.

debuntu_tools.kernel_manager.REBOOT_REQUIRED_FILE = '/var/run/reboot-required'

The absolute pathname of the file that exists when a system reboot is required (a string).

debuntu_tools.kernel_manager.REBOOT_REQUIRED_PACKAGES_FILE = '/var/run/reboot-required.pkgs'

The absolute pathname of a file with details about why a system reboot is required (a string).

debuntu_tools.kernel_manager.main()

Command line interface for debuntu-kernel-manager.

class debuntu_tools.kernel_manager.KernelPackageManager(**kw)

Python API for automated Linux kernel image package cleanup on Debian based systems.

Here’s an overview of the KernelPackageManager class:

Superclass:	PropertyManager
Public methods:	cleanup_packages() and render_summary()
Properties:	active_kernel_package, active_kernel_release, apt_options, cleanup_command, context, dry_run, force, installed_header_meta_packages, installed_header_packages, installed_image_meta_packages, installed_kernel_packages, installed_modules_packages, installed_package_groups, installed_packages, preserve_count, reboot_required, removable_header_packages, removable_kernel_packages, removable_modules_packages, removable_package_groups, removable_packages and running_newest_kernel

When you initialize a KernelPackageManager object you are required to provide values for the apt_options, context, force and preserve_count properties. You can set the values of the apt_options, context, force and preserve_count properties by passing keyword arguments to the class initializer.

apt_options

A list of strings with command line options to pass to apt-get.

Note

The apt_options property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named apt_options (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

context

An execution context created by executor.contexts.

Note

The context property is a custom_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named context (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

force

Whether to continue with removal despite warnings (a boolean, defaults to False).

Note

The force property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named force (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

preserve_count

The number of kernel packages to preserve (an integer, defaults to 2).

Note

The preserve_count property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named preserve_count (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

dry_run

True if cleanup_command performs a dry run, False otherwise.

installed_packages

A dictionary that maps package names (strings) to MaybeKernelPackage objects.

The value of this property is generated by parsing the output of the dpkg --list command.

Note

The installed_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

installed_header_packages

A list of MaybeKernelPackage objects for the installed Linux kernel header packages.

Note

The installed_header_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

installed_kernel_packages

A list of MaybeKernelPackage objects for the installed kernel images.

Note

The installed_kernel_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

installed_modules_packages

A list of MaybeKernelPackage objects for the installed kernel modules packages.

Note

The installed_modules_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

installed_header_meta_packages

A list of MaybeKernelPackage objects with installed meta packages for kernel headers.

Note

The installed_header_meta_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

installed_image_meta_packages

A list of MaybeKernelPackage objects with installed meta packages for kernel images.

Note

The installed_image_meta_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

installed_package_groups

A list of sets with MaybeKernelPackage objects for installed header and kernel packages.

Note

The installed_package_groups property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

active_kernel_release

The output of uname --kernel-release (a string).

Note

The active_kernel_release property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

active_kernel_package

The package name for the running kernel (a string).

Note

The active_kernel_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

reboot_required

True if REBOOT_REQUIRED_FILE exists, False otherwise.

Note

The reboot_required property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

removable_package_groups

A list of sets with MaybeKernelPackage objects considered to be removable.

Candidates for removal are selected from installed_package_groups, ignoring active_kernel_package and the newest preserve_count kernel images (minus one when active_kernel_package was ignored).

Note

The removable_package_groups property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

removable_header_packages

A list of MaybeKernelPackage objects for header packages considered to be removable.

Note

The removable_header_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

removable_kernel_packages

A list of MaybeKernelPackage objects for kernel packages considered to be removable.

Note

The removable_kernel_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

removable_modules_packages

A list of MaybeKernelPackage objects for modules packages considered to be removable.

Note

The removable_modules_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

removable_packages

A list of MaybeKernelPackage objects for kernel related packages considered to be removable.

Note

The removable_packages property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

running_newest_kernel

True if the newest kernel is currently active, False otherwise.

Note

The running_newest_kernel property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

cleanup_command

A list of strings with the apt-get command to remove old packages.

Note

The cleanup_command property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

render_summary()

Render a summary of installed and removable kernel packages on the terminal.

cleanup_packages(**options)

Run apt-get to cleanup removable kernel related packages.

Parameters:options – Any keyword arguments are passed on to the execute() method of the context object. Returns:True if a system reboot is required (to switch to the newest installed kernel image or because security updates have been installed), False otherwise. Raises:CleanupError when multiple Linux kernel meta packages are installed and force is False.

class debuntu_tools.kernel_manager.MaybeKernelPackage(**kw)

Dumb container for entries parsed from dpkg --list output.

Here’s an overview of the MaybeKernelPackage class:

Superclass:	PropertyManager
Properties:	is_header_meta_package, is_header_package, is_image_meta_package, is_installed, is_kernel_package, is_modules_package, is_supported_package, kernel_type, name, status, tokenized_name, version and version_in_name

When you initialize a MaybeKernelPackage object you are required to provide values for the name, status and version properties. You can set the values of the name, status and version properties by passing keyword arguments to the class initializer.

name

The name of the package (a string).

Note

The name property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

version

The version of the package (a Version object).

Note

The version property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named version (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

status

The status of the package (a string of two characters, refer to the dpkg man pages for details).

Note

The status property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named status (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

is_installed

True if the package is installed (or configuration files remain), False otherwise.

tokenized_name

The tokenized name of the package (a list of strings).

The value of this property is computed by calling tokenize_package_name() on name.

Note

The tokenized_name property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

is_header_meta_package

True if the package is a Linux kernel header meta package, False otherwise.

Note

The is_header_meta_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

is_image_meta_package

True if the package is a Linux kernel image meta package, False otherwise.

Note

The is_image_meta_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

is_header_package

True if the package is a specific version of the Linux kernel headers, False otherwise.

Note

The is_header_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

is_kernel_package

True if the package is a specific version of the Linux kernel image, False otherwise.

Note

The is_kernel_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

is_modules_package

True if the package contains Linux kernel modules, False otherwise.

Note

The is_modules_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

is_supported_package

True if the package concerns a Linux kernel image or modules or headers, False otherwise.

Note

The is_supported_package property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

version_in_name

The version encoded in the name of the package (a string or None).

Note

The version_in_name property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

kernel_type

The kernel type encoded in the name of the package (a string or None).

Note

The kernel_type property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

debuntu_tools.kernel_manager.tokenize_package_name(package_name)

Split a (kernel) package name into tokens.

Parameters:package_name – The name of a package (a string). Returns:A list of strings.

The tokenize_package_name() function uses a regular expression to tokenize package names by splitting on dashes while ignoring dashes with a digit on both sides (i.e. dashes that are part of a version number). For example:

>>> from debuntu_tools.kernel_manager import tokenize_package_name
>>> tokenize_package_name('linux-image-4.4.0-72-generic')
['linux', 'image', '4.4.0-72', 'generic']

debuntu_tools.kernel_manager.is_kernel_version(token)

Check whether a token in a package name looks like a kernel version.

Parameters:token – The token to check (a string). Returns:True if the token looks like a kernel version, False otherwise.

The is_kernel_version() function uses a regular expression to check whether the given string looks like a kernel version of the form 4.4.0-72. The string is assumed to be a package name token returned by tokenize_package_name().

exception debuntu_tools.kernel_manager.CleanupError

Custom exception to detect known problems.

Raised by KernelPackageManager when multiple Linux kernel meta packages are installed but force is False.

debuntu_tools.nodejs_installer

Usage: debuntu-nodejs-installer [OPTIONS]

Install an up to date Node.js binary distribution on a Debian or Ubuntu system by configuring and using the NodeSource binary package repositories.

Due to the time it takes for new software releases to find their way into the Debian and Ubuntu ecosystems versus the speed with which the Node.js community is currently moving, the system packages that provide Node.js are hopelessly out of date. Fortunately the folks at NodeSource maintain Debian and Ubuntu package repositories that provide up to date Node.js binary distributions.

NodeSource makes installation scripts available and the suggested way to run these is to download and pipe them straight to a shell. That kind of rubs me the wrong way :-) but I’ve nevertheless had to set up NodeSource installations a dozen times now. One thing led to another and now there is this program.

Supported options:

Option	Description
-i, --install	Configure the system to use one of the NodeSource binary package repositories and install the ‘nodejs’ package from the repository.
-V, --version=NODEJS_VERSION	Set the version of Node.js to be installed. You can find a list of available versions on the following web page: https://github.com/nodesource/distributions/ Default: node_10.x (active LTS)
-s, --sources-file=FILENAME	Set the pathname of the ‘package resource list’ that will be added to the system during configuration of the NodeSource binary package repository. Default: /etc/apt/sources.list.d/nodesource.list
-r, --remote-host=ALIAS	Perform the requested action(s) on a remote host over SSH. The ALIAS argument gives the SSH alias that should be used to connect to the remote host.
-v, --verbose	Increase verbosity (can be repeated).
-q, --quiet	Decrease verbosity (can be repeated).
-h, --help	Show this message and exit.

debuntu_tools.nodejs_installer.main()

Command line interface for debuntu-nodejs-installer.

class debuntu_tools.nodejs_installer.NodeInstaller(**kw)

Python API for the Node.js installer.

The logic in this installer is based on the manual installation instructions and setup scripts provided by NodeSource and my experience with system administration of Debian and Ubuntu systems.

The main method of this class is install() and the behavior of this method can be configured by setting the context, nodejs_version and sources_file properties by passing keyword arguments to the class initializer.

Here’s an overview of the NodeInstaller class:

Superclass:	PropertyManager
Public methods:	install(), install_https_transport(), install_package(), install_signing_key(), install_sources_file(), update_package_lists() and validate_system()
Properties:	context, nodejs_version and sources_file

When you initialize a NodeInstaller object you are required to provide a value for the context property. You can set the values of the context, nodejs_version and sources_file properties by passing keyword arguments to the class initializer.

context

An execution context created using executor.contexts.

Note

The context property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named context (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

nodejs_version

The Node.js version to install (a string, defaults to node_10.x).

Note

The nodejs_version property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

sources_file

The absolute pathname of the ‘package resource list’ used to enable the NodeSource repositories (a string).

Defaults to /etc/apt/sources.list.d/nodesource.list.

Note

The sources_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

install()

Enable the NodeSource repository and install the nodejs package.

Raises:UnsupportedSystemError when validate_system() fails.

validate_system()

Make sure the system is running a supported version of Debian or Ubuntu.

Raises:UnsupportedSystemError when validation fails.

install_signing_key(key_url='https://deb.nodesource.com/gpgkey/nodesource.gpg.key')

Install the signing key for the NodeSource repositories.

install_https_transport()

Install the apt-transport-https system package.

install_sources_file()

Install a ‘package resource list’ that points apt to the NodeSource repository.

update_package_lists()

Run apt-get update (with compensation if things break).

install_package()

Install the Node.js package on a Debian or Ubuntu system.

exception debuntu_tools.nodejs_installer.UnsupportedSystemError

Raised when an unsupported operating system is detected.

debuntu_tools.remote_reboot

Usage: reboot-remote-system [OPTIONS] [SSH_ALIAS]

Reboot a remote system and wait for the system to come back online. If the SSH alias matches a section in the ‘unlock-remote-system’ configuration, the root disk encryption of the remote system will be unlocked after it is rebooted.

Supported options:

Option	Description
-s, --shell	Start an interactive shell on the remote system after it has finished booting.
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

debuntu_tools.remote_reboot.main()

Command line interface for reboot-remote-system.

debuntu_tools.remote_reboot.reboot_remote_system(context=None, name=None)

Reboot a remote Linux system (unattended).

Parameters:

• context – A RemoteContext object (or None).
• name – The name of the unlock-remote-system configuration section for the remote host (a string) or None.

Raises:

ValueError when the remote system appears to be using root disk encryption but there’s no unlock-remote-system configuration section available. The reasoning behind this is to err on the side of caution when we suspect we won’t be able to get the remote system back online.

This function reboots a remote Linux system, waits for the system to go down and then waits for it to come back up.

If the ssh_alias of the context matches a section in the unlock-remote-system configuration, the root disk encryption of the remote system will be unlocked after it is rebooted.

debuntu_tools.remote_reboot.get_uptime(context, multiplexed=False)

Get the uptime of a remote Linux system by reading /proc/uptime.

Parameters:context – An execution context created by executor.contexts. Returns:The uptime of the remote system (as a float).

This function is used by reboot_remote_system() to wait until a remote Linux system has successfully rebooted.

The uptime check uses ssh -S none $REMOTE_HOST cat /proc/uptime where -S none opts out of OpenSSH connection multiplexing, because in my experience the master connection for multiplexed OpenSSH connections handles remote reboots rather ungracefully (it can take a minute or two before the client finally considers the master connection dead and successfully establishes a fresh connection to the rebooted remote host).

Any output on the standard error stream is silenced because 99% of the time this output will consist of SSH client connection errors due to the “retry until success” approach taken by reboot_remote_system().

debuntu_tools.remote_reboot.get_post_context(name)

Get an execution context for the post-boot environment of a remote host.

Parameters:name – The configuration section name or SSH alias of the remote host (a string). Returns:A RemoteContext object.

debuntu_tools.remote_reboot.is_encrypted(context)

Detect whether a remote system is using root disk encryption.

Parameters:context – A RemoteContext object. Returns:True if root disk encryption is being used, False otherwise.

debuntu_tools.remote_unlock

Usage: unlock-remote-system [OPTIONS] PRE_BOOT [POST_BOOT]

Boot a remote Linux system that’s waiting for the root disk encryption password to be entered into an interactive prompt by connecting to the remote system over the network using SSH and entering the password non-interactively. The remote Linux system needs to be configured in such a way that the pre-boot environment enables a static IP address and starts an SSH server like Dropbear.

The PRE_BOOT argument defines how to connect to the pre-boot environment:

• Its value is assumed to be a host name, IP address or SSH alias.
• It can optionally start with a username followed by an ‘@’ sign.
• It can optionally end with a ‘:’ followed by a port number.

The default username is ‘root’ and the default port number 22. The optional POST_BOOT argument defines how to connect to the post-boot environment, this is useful when the pre and post-boot environments run SSH servers on different port numbers.

If the PRE_BOOT argument matches the name of a user defined configuration section the options in that section define how unlock-remote-system operates.

Supported options:

Option	Description
-i, --identity-file=KEY_FILE	Use the private key stored in KEY_FILE for SSH connections to the pre-boot environment. The post-boot environment is expected to use your default private key or have a suitable configuration in ~/.ssh/config.
-k, --known-hosts=HOSTS_FILE	Use HOSTS_FILE as the “known hosts file” for SSH connections to the pre-boot environment. When this option is not given host key verification will be disabled to avoid conflicts between the host keys of the different SSH servers running in the pre-boot and post-boot environments.
-p, --password=NAME	Get the password for the root disk encryption of the remote system from the local password store in ~/.password-store using the ‘pass’ program. The NAME argument gives the full name of the password.
-r, --remote-host=SSH_ALIAS	Connect to the remote system through an SSH proxy.
-s, --shell	Start an interactive shell on the remote system after it has finished booting.
-w, --watch	Start monitoring the remote system and automatically unlock the root disk encryption when the remote system is rebooted. The monitoring continues indefinitely.
-a, --all	Enable monitoring of all configured systems when combined with --watch.
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

debuntu_tools.remote_unlock.EXPRESSION_PATTERN = <_sre.SRE_Pattern object>

A compiled regular expression pattern to parse connection profile expressions.

debuntu_tools.remote_unlock.HOST_KEYS_FILE = '~/.config/unlock-remote-system.d/known-hosts.ini'

The configuration file that’s used to store SSH host keys (a string).

debuntu_tools.remote_unlock.main()

Command line interface for unlock-remote-system.

debuntu_tools.remote_unlock.find_local_username()

Find the username of the current user on the local system.

debuntu_tools.remote_unlock.get_password_from_store(name, store=None)

Get the disk encryption password from the ‘pass’ program.

debuntu_tools.remote_unlock.prompt_for_password(hostname)

Prompt the operator to interactively enter the disk encryption password.

class debuntu_tools.remote_unlock.EncryptedSystem(**kw)

Python API for the unlock-remote-system program.

This class implements a Python API for remote unlocking of Linux systems with root disk encryption over SSH. The internals of this class differentiate between the pre-boot and post-boot environments:

• The pre_boot and post_boot properties are ConnectionProfile objects that define how to connect to the SSH server in the pre-boot environment (usually this is Dropbear running in the initial ram disk) and the SSH server in the post-boot environment (usually this is OpenSSH).
• The pre_context and post_context properties are RemoteContext objects that enable command execution in the pre-boot and post-boot environments. The values of these properties are created based on the corresponding connection profiles.
• In addition to the command execution contexts for the pre-boot and post-boot environments there is the context property which provides a command execution context for commands outside of the remote system. This defaults to the local system on which unlock-remote-system is running but will be a remote system when ssh_proxy is set.

Here’s an overview of the EncryptedSystem class:

Superclass:	PropertyManager
Special methods:	__enter__() and __exit__()
Public methods:	check_ssh_connection(), create_key_script(), find_process_id(), get_known_host_keys(), kill_emergency_shell(), kill_interactive_prompt(), offer_password(), parse_host_keys(), parse_server_header(), run_cryptroot_program(), scan_ssh_server(), store_host_keys(), test_ssh_connection(), unlock_system(), wait_for_post_boot(), wait_for_pre_boot(), watch_system(), write_file() and write_to_named_pipe()
Properties:	boot_timeout, config, config_loader, config_section, connect_timeout, context, control_directory, cryptroot_config, cryptroot_program, have_cryptroot_config, have_cryptroot_program, have_named_pipe, interactive, key_script, known_hosts_file, named_pipe, password, post_boot, post_context, pre_boot, pre_context, retry_interval, scan_timeout, ssh_proxy and watch_interval

When you initialize a EncryptedSystem object you are required to provide values for the post_boot and pre_boot properties. You can set the values of the boot_timeout, config_loader, config_section, connect_timeout, control_directory, cryptroot_config, cryptroot_program, interactive, key_script, known_hosts_file, named_pipe, password, post_boot, pre_boot, retry_interval, scan_timeout, ssh_proxy and watch_interval properties by passing keyword arguments to the class initializer.

boot_timeout

The number of seconds to wait for the system to boot (a number, defaults to 5 minutes).

Note

The boot_timeout property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

config

A dictionary with configuration options.

See also config and config_section.

Note

The config property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

config_loader

A ConfigLoader object.

See also config and config_section.

Note

The config_loader property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

config_section

The configuration section to use (a string or None).

When this option is set config can be loaded and store_host_keys() can persist SSH host keys.

See also config and config_loader.

Note

The config_section property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

connect_timeout

How long to wait for the system to become reachable (an integer, defaults to 2 minutes).

Note

The connect_timeout property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

context

The command execution context from which the remote system is being controlled.

The computed value of this property is a command execution context created by executor.contexts. When ssh_proxy is set this will be a RemoteContext object, otherwise it will be a LocalContext object.

Note

The context property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

control_directory

A temporary directory for SSH control sockets (a string).

Note

The control_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

cryptroot_config

The pathname of the ‘cryptroot’ configuration file (a string).

The value of this property sets the pathname of the ‘cryptroot’ configuration file in the pre-boot environment (the initial ram disk). It defaults to ‘/conf/conf.d/cryptroot’.

Note

The cryptroot_config property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

cryptroot_program

The pathname of the ‘cryptroot’ program (a string).

The value of this property sets the pathname of the ‘cryptroot’ program in the pre-boot environment (the initial ram disk). It defaults to ‘/scripts/local-top/cryptroot’.

Note

The cryptroot_program property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

have_cryptroot_config

True if cryptroot_config exists, False otherwise.

The existence of the /conf/conf.d/cryptroot configuration file is taken as confirmation that the remote system is currently in its pre-boot environment (initial ram disk).

have_cryptroot_program

True if cryptroot_program exists, False otherwise.

have_named_pipe

True if named_pipe exists, False otherwise.

The named pipe configured by the named_pipe property provides a simple and robust way to inject the root disk encryption pass phrase into the boot sequence. When the named pipe is available it will be used as the preferred method.

In my experience this works on Ubuntu 16.04 but it doesn’t work on Ubuntu 14.04 (because the named pipe doesn’t exist).

interactive

True to allow user interaction, False otherwise.

The value of interactive defaults to the return value of connected_to_terminal() when given sys.stdin.

Note

The interactive property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

key_script

The pathname of the generated key script (defaults to '/tmp/keyscript.sh').

Note

The key_script property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

known_hosts_file

The filename of the “known hosts file” to use (a string or None).

Note

The known_hosts_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

named_pipe

The pathname of the named pipe used by cryptsetup (a string).

The value of this property sets the pathname of the named pipe used by cryptsetup in the pre-boot environment (the initial ram disk). It defaults to ‘/lib/cryptsetup/passfifo’.

Note

The named_pipe property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

password

The password that unlocks the root filesystem of the remote host (a string or None).

If the configuration section contains the option password-name then get_password_from_store() will be used to get the password by executing the pass program.

The optional configuration option password-store can be used to set the $PASSWORD_STORE_DIR environment variable.

Note

The password property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

post_boot

A connection profile for the post-boot environment (a ConnectionProfile object).

Note

The post_boot property is a custom_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named post_boot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

post_context

A command execution context for the post-boot environment.

The computed value of this property is a command execution context created by executor.contexts, more specifically it’s a RemoteContext object.

Note

The post_context property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

pre_boot

A connection profile for the pre-boot environment (a ConnectionProfile object).

Note

The pre_boot property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named pre_boot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

pre_context

The command execution context inside the pre-boot environment.

The computed value of this property is a command execution context created by executor.contexts, more specifically it’s a RemoteContext object.

Note

The pre_context property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

retry_interval

The time between connection attempts (an integer, defaults to 1 second).

Note

The retry_interval property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

scan_timeout

The timeout for ssh-keyscan (an integer, defaults to 5 seconds).

Note

The scan_timeout property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

ssh_proxy

The SSH alias of a proxy to the remote system (a string or None).

Note

The ssh_proxy property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

watch_interval

The time between pre-boot environment checks (an integer, defaults to 60 seconds).

Note

The watch_interval property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

__enter__()

Prepare control_directory.

__exit__(exc_type=None, exc_value=None, traceback=None)

Cleanup control_directory.

check_ssh_connection()

Verify SSH connectivity to the pre-boot environment.

create_key_script()

Create a key script in the pre-boot environment (initial ram disk).

This method creates a minimal key script (a shell script containing a single echo command) in the pre-boot environment and modifies the cryptroot_config file so that the key script is used to unlock the root disk.

find_process_id(program, **options)

Determine the process id of a program or script in the pre-boot environment.

Parameters:program – The name of a program or script (a string). Returns:A process id (an integer) or None.

get_known_host_keys(option_name)

Get the configured SSH host keys.

Parameters:option_name – The name of the configuration option that holds known host keys (a string). Returns:A set of strings.

kill_emergency_shell()

Kill the emergency shell process to resume the boot process.

kill_interactive_prompt()

Kill the process responsible for the interactive prompt in the pre-boot environment.

offer_password()

Make the root disk encryption pass phrase available to the remote host.

Raises:UnsupportedSystemError when have_named_pipe, have_cryptroot_config and have_cryptroot_program are all False.

If have_named_pipe is True and password is set, the internal method write_to_named_pipe() is used to write password to the named pipe. This is the preferred way to make the password available to the remote host because it’s simple and robust.

When the named pipe isn’t available but have_cryptroot_config is True the following internal methods are used instead:

• kill_interactive_prompt()
• create_key_script() (this is only called when password is set)
• run_cryptroot_program()
• kill_emergency_shell()

This is more convoluted than the named pipe but it works :-).

parse_host_keys(cmd)

Find the SSH host keys in the output of ssh-keyscan.

Parameters:cmd – A RemoteCommand object. Returns:A set of strings.

parse_server_header(cmd)

Find the SSH server header in the output of ssh-keyscan.

Parameters:cmd – A RemoteCommand object. Returns:The SSH server header (a string).

scan_ssh_server(profile)

Get the SSH server header and host keys.

Parameters:profile – A ConnectionProfile object. Returns:A ServerDetails object.

The scan_ssh_server() method runs ssh-keyscan in context and parses its output to get the SSH server header and host keys.

run_cryptroot_program(interactive=True)

Unlock the root disk encryption by running cryptroot_program.

store_host_keys(pre_server, post_server)

Store the SSH host keys in the configuration.

Parameters:

• pre_server – The ServerDetails object from wait_for_pre_boot().
• post_server – The ServerDetails object from wait_for_post_boot().

test_ssh_connection(profile, context)

Verify SSH connectivity to the pre-boot environment.

unlock_system(server=None)

Validate the pre-boot environment and unlock the root filesystem encryption.

Parameters:server – A ServerDetails object or None.

When the server argument isn’t given wait_for_pre_boot() is used to wait for the pre-boot environment to become available.

wait_for_post_boot(pre_server)

Wait for the post-boot environment to come online.

Parameters:pre_server – A ServerDetails object created by wait_for_pre_boot().

wait_for_pre_boot()

Wait for the pre-boot environment to become available.

Returns:

A ServerDetails object.

Raises:

The following exceptions can be raised:

• SystemUnreachableError when connect_timeout seconds have passed and we still haven’t managed to query the SSH server in the pre-boot environment.
• UnlockAbortedError when the post-boot environment is detected and the operator aborts the unlock sequence.

watch_system()

Start monitoring the remote system for reboots.

When the remote system is rebooted, the root disk encryption will be unlocked automatically. The monitoring continues indefinitely.

write_file(filename, contents)

Write a file in the initial ram disk of the pre-boot environment.

Parameters:

• filename – The pathname of the file to write (a string).
• contents – The contents to write to the file (a string).

This method writes a file in the initial ram disk by running a remote ‘sh’ shell that redirects its standard input to the given filename.

write_to_named_pipe()

Write password to the named pipe configured by named_pipe.

class debuntu_tools.remote_unlock.ConnectionProfile(**kw)

SSH connection profiles.

Here’s an overview of the ConnectionProfile class:

Superclass:	PropertyManager
Properties:	expression, hostname, identity_file, port_number and username

When you initialize a ConnectionProfile object you are required to provide a value for the hostname property. You can set the values of the expression, hostname, identity_file, port_number and username properties by passing keyword arguments to the class initializer.

expression

The SSH connection profile encoded as a string.

Note

The expression property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

hostname

The host name, IP address or SSH alias of the remote system (a string).

Note

The hostname property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named hostname (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

port_number

The port number for SSH connections (an integer, defaults to 22).

Note

The port_number property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

username

The username used to log in to the remote system (a string, defaults to ‘root’).

Note

The username property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

identity_file

The pathname of the identity file used to connect to the remote system (a string or None).

Note

The identity_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class debuntu_tools.remote_unlock.ServerDetails(**kw)

Properties that can be used to uniquely identify SSH servers.

Here’s an overview of the ServerDetails class:

Superclass:	PropertyManager
Public methods:	match_header()
Properties:	header and host_keys

When you initialize a ServerDetails object you are required to provide values for the header and host_keys properties. You can set the values of the header and host_keys properties by passing keyword arguments to the class initializer.

header

The SSH server header (a string).

This is the first line of output emitted by the SSH server when a client opens a TCP connection.

Note

The header property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named header (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

host_keys

The SSH server host keys (a frozenset of strings).

Note

The host_keys property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named host_keys (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

match_header(substring)

Check if header contains the given substring.

Parameters:substring – The substring to check (a string). Returns:True if the header matches the substring, False otherwise.

exception debuntu_tools.remote_unlock.EncryptedSystemError

Base class for custom exceptions raised by debuntu_tools.remote_unlock.

exception debuntu_tools.remote_unlock.UnlockAbortedError

Raised when the operator changes their minds.

exception debuntu_tools.remote_unlock.SystemUnreachableError

Raised when connecting to the SSH server in the pre-boot environment fails.

exception debuntu_tools.remote_unlock.BootTimeoutError

Raised when the remote system doesn’t boot properly after unlocking.

exception debuntu_tools.remote_unlock.UnsupportedSystemError

Raised when the configuration file ‘/conf/conf.d/cryptroot’ is not found.

debuntu_tools.upgrade_system

Usage: upgrade-remote-system [OPTIONS] [SSH_ALIAS]

Upgrade the system packages on a remote Debian or Ubuntu system, reboot the system when this is required due to security updates or because the system isn’t yet running the newest kernel, remove old Linux kernel and header packages and optionally remove ‘auto-removable’ system packages.

If the given SSH alias matches a section in the ‘unlock-remote-system’ configuration, the root disk encryption of the remote system will be automatically unlocked when the system is rebooted.

Supported options:

Option	Description
-s, --shell	Start an interactive shell on the remote system afterwards.
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

debuntu_tools.upgrade_system.main()

Command line interface for upgrade-remote-system.

debuntu_tools.upgrade_system.upgrade_remote_system(context, force_reboot=False)

Perform standard system maintenance tasks on a remote Debian or Ubuntu system.

Parameters:context – An execution context created by executor.contexts.

This function performs the following system maintenance tasks:

1. The apt-get update command is run (using the Python API of the apt-mirror-updater program).
2. The apt-get dist-upgrade command is run [1].
3. The apt-get clean command is run.
4. If the file /var/run/reboot-required exists (indicating that a reboot is required due to security updates) the remote system is rebooted using the Python API of the reboot-remote-system program, to enable automatic unlocking of remote root disk encryption.
5. Old kernel packages are removed (using the Python API of the debuntu-kernel-manager program). If more than one meta package is installed a warning message is logged but no exception is raised.
6. The apt-get autoremove --purge command is run to optionally [1] remove any ‘auto-removable’ system packages.

[1]	(1, 2) Because the apt-get option --yes is not used, the operator will be asked to confirm using an interactive confirmation prompt.

Change log

The change log lists notable changes to the project:

Changelog

The purpose of this document is to list all of the notable changes to this project. The format was inspired by Keep a Changelog. This project adheres to semantic versioning.

Release 0.9.1 (2020-08-17)

Speed up reboot-remote-system by opting out of OpenSSH connection multiplexing during reboot check.

Release 0.9 (2020-05-20)

Noteworthy changes:

• Switch from requests to six (six.moves.urllib.request).

  The debuntu-nodejs-installer program needs to make two HTTPS requests and until now used requests to do so. However requests pulls in quite a few dependencies (certifi, chardet, idna and urllib3).

  On older Python 2.7 releases requests was needed to provide proper TLS support including SNI, however with most of the world moving on to modern Python releases the simple responsibility of making two HTTPS requests no longer warrants five dependencies…

  Besides, six was already part of the transitive requirements and the code changes required were minimal. I also got to remove most of the complexity from setup.py.

• Make it possible to instruct the Python API of upgrade-remote-system to perform a reboot regardless of whether this is required by package updates.

• Updated the usage messages embedded in the readme.

Miscellaneous changes:

• Update Ubuntu releases mentioned in readme.
• Update PyPI and RTD links in readme.
• Use console highlighting in readme.
• Refactored makefile (use Python 3 for local development, treat Sphinx warnings as errors, etc).
• Fixed existing Sphinx reference warnings.
• Bumped requirements, fixed deprecated imports.

Release 0.8 (2019-06-23)

• debuntu-nodejs-installer: Bump Node.js version to 10.x. This was triggered by the build failure at https://travis-ci.org/xolox/python-npm-accel/jobs/549226950 which caused me to wonder why I had never bothered to update this default. So here it is :-).
• Bug fix for reboot-remote-system: Don’t run lsblk on regular files.
• Bug fix for reboot-remote-system: Ignore usernames in config check.

Release 0.7 (2019-04-10)

• Improved upgrade-remote-system (reboot when running old kernel).
• Bug fix for reboot-remote-system (always confirm SSH connectivity).
• Updated the remote root disk encryption how to (Ubuntu 18.04 compatibility).

Release 0.6.4 (2018-11-17)

The debuntu-kernel-manager program now supports cleaning up Linux kernel modules packages (linux-modules-*).

Release 0.6.3 (2018-10-24)

Bump connection timeout of unlock-remote-system from 60 seconds to 2 minutes.

In the past months the reboot-remote-system command has failed to reboot my Raspberry Pi in an unattended fashion in about half of my attempts, because after the reboot command is given it takes more than 60 seconds for the pre-boot environment to become available… 😒

Now on the one hand this is just a single use case based on crappy hardware, and I could have just configured a longer connect-timeout in the configuration file of course. On the other hand I do intend for tools like reboot-remote-system to be as much “do what I mean” as possible and picking reasonable defaults is part of that.

Also I have plenty of experience with server hardware and I know that some of those servers take more than a minute to finish initializing their hardware and actually booting the OS, so even with fancy hardware boot times can be long 😇.

Because I didn’t see the harm in bumping the connect-timeout for all users I decided to do that instead of configuring this on my end, potentially “obscuring a bad default”. Anyone who disagrees is free to define a more restrictive connect-timeout using a configuration file.

Release 0.6.2 (2018-10-24)

• Improve header package detection of debuntu-kernel-manager: While doing routine maintenance on the Raspberry Pi that handles DHCP and DNS in my home network I noticed that while the package linux-headers-4.4.0-1096-raspi2 was recognized the package linux-raspi2-headers-4.4.0-1096 was not suggested for removal by debuntu-kernel-manager. This is now fixed.
• I’ve also reduced code duplication in debuntu-kernel-manager. While this isn’t intended to change the behavior of the program I haven’t gone to great lengths to actually verify this, however it seems to me that only in obscure theoretical corner cases would there be an actual observable difference in behavior.

Release 0.6.1 (2018-07-03)

Bumped linux-utils requirement to pull in an upstream bug fix:

• An exception was being raised by the upgrade-remote-system program (at the point where it calls into reboot-remote-system) because the file /etc/crypttab didn’t exist.
• However experience tells me that /etc/crypttab doesn’t exist in default Debian and Ubuntu installations (unless that system was specifically set up with root disk encryption using the installation wizard).
• Furthermore this was in the code path responsible for figuring out whether a given system has any encrypted filesystems. Because “none” is definitely a valid answer, I’ve changed linux-utils to log a notice that the file couldn’t be found but not raise any exceptions.

Release 0.6 (2018-06-28)

• Added upgrade-remote-system program.
• Improved reboot-remote-system API (it’s now possible to give a name to reboot_remote_system() and leave it up to that function to get the execution context from the configuration file).
• Documentation about remote root disk encryption on Raspberry Pi.

Release 0.5 (2018-05-26)

• Make it possible to interactively enter the root disk encryption password into an interactive prompt on the remote system, while connected over SSH.
• Added documentation about remote root disk encryption.
• Fixed a confusing typo in logging output of reboot-remote-system.
• Improved reboot_remote_system() API documentation.
• Added this changelog, restructured the online documentation.
• Integrated property_manager.sphinx in online documentation.
• Added license='MIT' key to setup.py script.
• Include documentation in source distributions.
• Fixed broken reStructuredText reference in nodejs_installer.py.
• Fixed unaligned reStructuredText headings.

Release 0.4.1 (2018-04-03)

• Bug fix for unlocking in reboot-remote-system.
• Set interactive=False for unlock-remote-system --watch --all.
• Cleanup debuntu_tools.remote_unlock.main().

Release 0.4 (2018-04-01)

• Added the unlock-remote-system program for unattended unlocking of remote root disk encryption over SSH.
• Added the reboot-remote-system program for rebooting of remote systems (optionally with root disk encryption).

Release 0.3.8 (2017-07-11)

• Try to improve security requirements handling.
• Changed the Sphinx theme of the online documentation.

Release 0.3.7 (2017-04-17)

Improved package name parsing in debuntu-kernel-manager.

Recently I installed the Linux kernel image meta package linux-image-generic-hwe-16.04 on my Ubuntu 16.04 laptop and since then I noticed that debuntu-kernel-manager got confused by the -16.04 suffix. This is now fixed.

Release 0.3.6 (2017-01-18)

Reduced tty usage and code duplication in debuntu-kernel-manager.

Release 0.3.5 (2016-10-31)

Expose the “kernel preserve count” in the debuntu-kernel-manager command line interface.

Release 0.3.4 (2016-10-31)

Bug fix: Always run apt-auto-removal script with root privileges.

Release 0.3.3 (2016-10-25)

Bug fix: Automatically update the list of auto-removable kernels after cleanup.

Release 0.3.2 (2016-10-25)

• Bug fix: Never remove signal files when performing a dry-run.
• Simplified the dpkg -l package status handling.

Release 0.3.1 (2016-10-25)

Bug fix: Don’t complain when multiple header meta packages are installed.

Release 0.3 (2016-09-07)

Added the debuntu-nodejs-installer program to install Node.js from the NodeSource binary repositories.

Release 0.2 (2016-06-23)

• Remove the /var/run/reboot-required file when it seems safe to do so.
• Rename s/collector/manager/g throughout the package.

Release 0.1 (2016-06-15)

The initial release of debuntu-tools contained only the program debuntu-kernel-manager. Half the value for me in creating this program was getting to know how Debian and Ubuntu kernel image/header meta packages worked. My initial goal was to create a safer alternative to sudo apt-get autoremove --purge with the ultimate goal of completely automating the cleanup of old kernel packages.