Manifests¶
Official EC2 manifests¶
The official Debian images for EC2 are built with bootstrap-vz.
In the folder manifests/official/ec2
you will find the various manifests
that are used to create the different flavors of Debian AMIs for EC2.
You can read more about those official images in the Debian wiki.
The official images can be found on the AWS marketplace.
Official GCE manifests¶
The manifest file is the primary way to interact with bootstrap-vz. Every configuration and customization of a Debian installation is specified in this file.
The manifest format is YAML or JSON. It is near impossible to run the bootstrapper with an invalid configuration, since every part of the framework supplies a json-schema that specifies exactly which configuration settings are valid in different situations.
Manifest variables¶
Many of the settings in the example manifests use strings like
debian-{system.release}-{system.architecture}-{{"{%y"}}}{{"{%m"}}}{{"{%d"}}}
.
These strings make use of manifest variables, which can cross reference
other settings in the manifest or specific values supplied by the
bootstrapper (e.g. all python date formatting variables are available).
Any reference uses dots to specify a path to the desired manifest
setting. Not all settings support this though, to see whether embedding
a manifest variable in a setting is possible, look for the
manifest vars
label.
Sections¶
The manifest is split into 7 sections.
Provider¶
The provider section contains all provider specific settings and the name of the provider itself.
name
: target virtualization platform of the installationrequired
Consult the providers section of the documentation for a list of valid values.
Bootstrapper¶
This section concerns the bootstrapper itself and its behavior. There are 4 possible settings:
workspace
: Path to where the bootstrapper should place images and intermediate files. Any volumes will be mounted under that path.required
tarball
: debootstrap has the option to download all the software and pack it up in a tarball. When starting the actual bootstrapping process, debootstrap can then be pointed at that tarball and use it instead of downloading anything from the internet. If you plan on running the bootstrapper multiple times, this option can save you a lot of bandwidth and time. This option just specifies whether it should create a new tarball or not. It will search for and use an available tarball if it already exists, regardless of this setting.optional
Valid values:true, false
Default:false
mirror
: The mirror debootstrap should download software from. It is advisable to specify a mirror close to your location (or the location of the host you are bootstrapping on), to decrease latency and improve bandwidth. If not specified, the configured aptitude mirror URL is used.optional
include_packages
: Extra packages to be installed during bootstrap. Accepts a list of package names.optional
exclude_packages
: Packages to exclude during bootstrap phase. Accepts a list of package names.optional
guest_additions
: This setting is only relevant for the virtualbox provider. It specifies the path to the VirtualBox Guest Additions ISO, which, when specified, will be mounted and used to install the VirtualBox Guest Additions.optional
Image¶
The image section configures anything pertaining directly to the image that will be created.
name
: The name of the resulting image. When bootstrapping cloud images, this would be the name visible in the interface when booting up new instances. When bootstrapping for VirtualBox or kvm, it’s the filename of the image.required
manifest vars
description
: Description of the image. Where this setting is used depends highly on which provider is set. At the moment it is only used for AWS images.required for ec2 provider
manifest vars
bucket
: When bootstrapping an S3 backed image for AWS, this will be the bucket where the image is uploaded to.required for S3 backing
region
: Region in which the AMI should be registered.required for S3 backing
System¶
This section defines anything that pertains directly to the bootstrapped system and does not fit under any other section.
architecture
: The architecture of the system. Valid values:i386, amd64
required
bootloader
: The bootloader for the system. Depending on the bootmethod of the virtualization platform, the options may be restricted. Valid values:grub, extlinux, pv-grub
required
charmap
: The default charmap of the system. Valid values: Any valid charmap likeUTF-8
,ISO-8859-
orGBK
.required
hostname
: hostname to preconfigure the system with.optional
locale
: The default locale of the system. Valid values: Any locale mentioned in/etc/locale.gen
required
release
: Defines which debian release should be bootstrapped. Valid values:squeeze
,wheezy
,jessie
,sid
,oldstable
,stable
,testing
,unstable
required
timezone
: Timezone of the system. Valid values: Any filename from/usr/share/zoneinfo
required
Packages¶
The packages section allows you to install custom packages from a variety of sources.
install
: A list of strings that specify which packages should be installed. Valid values: Package names optionally followed by a/target
or paths to local.deb
files. Note that packages are installed in the order they are listed. The installer invocations are bundled by package type (remote or local), meaning if you install two local packages, then two remote packages and then another local package, there will be two calls todpkg -i ...
and a single call toapt-get install ...
.install_standard
: Defines if the packages of the"Standard System Utilities"
option of the Debian installer, provided by tasksel, should be installed or not. The problem is that with justdebootstrap
, the system ends up with very basic commands. This is not a problem for a machine that will not be used interactively, but otherwise it is nice to have at hand tools likebash-completion
,less
,locate
, etc.optional
Valid values:true
,false
Default:false
mirror
: The default aptitude mirror.optional
Default:http://httpredir.debian.org/debian/
sources
: A map of additional sources that should be added to the aptitude sources list. The key becomes the filename in/etc/apt/sources.list.d/
(with.list
appended to it), while the value is an array with each entry being a line.optional
components
: A list of components that should be added to the default apt sources. For examplecontrib
ornon-free
optional
Default:['main']
trusted-keys
: List of paths to.gpg
keyrings that should be added to the aptitude keyring of trusted signatures for repositories.optional
preferences
: Allows you to pin packages through apt preferences. The setting is an object where the key is the preference filename in/etc/apt/preferences.d/
. The keymain
is special and refers to the file/etc/apt/preferences
, which will be overwritten if specified.optional
The values are objects with three keys:package
: The package to pin (wildcards allowed)pin
: The release to pin the package to.pin-priority
: The priority of this pin.
Volume¶
bootstrap-vz allows a wide range of options for configuring the disk layout of the system. It can create unpartitioned as well as partitioned volumes using either the gpt or msdos scheme. At most, there are only three partitions with predefined roles configurable though. They are boot, root and swap.
backing
: Specifies the volume backing. This setting is very provider specific. Valid values:ebs
,s3
,vmdk
,vdi
,raw
required
partitions
: A map of the partitions that should be created on the volume.type
: The partitioning scheme to use. When usingnone
, only root can be specified as a partition. Valid values:none
,gpt
,msdos
required
root
: Configuration of the root partition.required
size
: The size of the partition. Valid values: Any datasize specification up to TB (e.g. 5KiB, 1MB, 6TB).required
filesystem
: The filesystem of the partition. When choosingxfs
, thexfsprogs
package will need to be installed. Valid values:ext2
,ext3
,ext4
,xfs
required
format_command
: Command to format the partition with. This optional setting overrides the command bootstrap-vz would normally use to format the partition. The command is specified as a string array where each option/argument is an item in that array (much like the commands plugin).optional
The following variables are available:{fs}
: The filesystem of the partition.{device_path}
: The device path of the partition.{size}
: The size of the partition.
The default command used by boostrap-vz is
['mkfs.{fs}', '{device_path}']
.boot
: Configuration of the boot partition. The three settings equal those of the root partition.optional
swap
: Configuration of the swap partition. Since the swap partition has its own filesystem you can only specify the size for this partition.optional
Plugins¶
The plugins section is a map of plugin names to whatever configuration a plugin requires. Go to the plugin section of the documentation, to see the configuration for a specific plugin.
Providers¶
Azure¶
This provider generates raw images for Microsoft Azure computing platform.
Setup¶
qemu-img >= 1.7.0 required to convert raw image to vhd fixed size disk. This release is available in wheezy-backports.
wget must be installed on local computer.
Manifest must use the raw format, provider will automatically transform the disk to a vhd disk format.
Do not create swap space on the OS disk:
The Windows Azure Linux Agent can automatically configure swap space using the local resource disk that is attached to the VM after provisioning on Azure. Modify the following parameters in /etc/waagent.conf appropriately:
ResourceDisk.Format=y
ResourceDisk.Filesystem=ext4
ResourceDisk.MountPoint=/mnt/resource
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=2048 ## NOTE: set this to whatever you need it to be.
You can specify a waagent.conf file to replace the default one in the manifest in the azure/waagent section of the provider:
"system" : {
"waagent" : {
"conf": "path_to_my_conf_file", # optional
"version" : "2.0.4" # mandatory
}
}, ...
Waagent versions are available at: https://github.com/Azure/WALinuxAgent/releases
EC2¶
The EC2 provider automatically creates a volume for bootstrapping (be it EBS or S3), makes a snapshot of it once it is done and registers it as an AMI. EBS volume backing only works on an EC2 host while S3 backed volumes should work locally (at this time however they do not, a fix is in the works).
Unless the cloud-init plugin
is used, special startup scripts will be installed that automatically fetch the
configured authorized_key from the instance metadata and save or run
any userdata supplied (if the userdata begins with #!
it will be
run). Set the variable install_init_scripts
to False
in order
to disable this behaviour.
Manifest settings¶
Credentials¶
The AWS credentials can be configured in two ways: Via the manifest or through environment variables. To bootstrap S3 backed instances you will need a user certificate and a private key in addition to the access key and secret key, which are needed for bootstraping EBS backed instances.
The settings describes below should be placed in the credentials
key
under the provider
section.
access-key
: AWS access-key. May also be supplied via the environment variable$AWS_ACCESS_KEY
required for EBS & S3 backing
secret-key
: AWS secret-key. May also be supplied via the environment variable$AWS_SECRET_KEY
required for EBS & S3 backing
certificate
: Path to the AWS user certificate. Used for uploading the image to an S3 bucket. May also be supplied via the environment variable$AWS_CERTIFICATE
required for S3 backing
private-key
: Path to the AWS private key. Used for uploading the image to an S3 bucket. May also be supplied via the environment variable$AWS_PRIVATE_KEY
required for S3 backing
user-id
: AWS user ID. Used for uploading the image to an S3 bucket. May also be supplied via the environment variable$AWS_USER_ID
required for S3 backing
Example:
---
provider:
name: ec2
credentials:
access-key: AFAKEACCESSKEYFORAWS
secret-key: thes3cr3tkeyf0ryourawsaccount/FS4d8Qdva
Virtualization¶
EC2 supports both paravirtual and hardware virtual machines. The virtualization type determines various factors about the virtual machine performance (read more about this in the EC2 docs).
virtualization
: The virtualization type Valid values:pvm
,hvm
required
Example:
---
provider:
name: ec2
virtualization: hvm
Enhanced networking¶
Install enhanced networking drivers to take advantage of SR-IOV capabilities on hardware virtual machines. Read more about this in the EC2 docs.
Example:
---
provider:
name: ec2
virtualization: hvm
enhanced_networking: simple
Google Compute Engine¶
The GCE provider
can creates image as expected by GCE - i.e. raw disk image in *.tar.gz
file. It can upload created images to Google Storage Engine (to URI
provided in manifest by gcs_destination
) and can register image to
be used by Google Compute Engine to project provided in manifest by
gce_project
. Both of those functionalities are not fully tested yet.
KVM¶
The KVM provider creates virtual images for Linux Kernel-based Virtual Machines. It supports the installation of virtio kernel modules (paravirtualized drivers for IO operations).
VirtualBox¶
The VirtualBox provider can bootstrap to both .vdi and .vmdk images (raw images are also supported but do not run in VirtualBox). It’s advisable to always use vmdk images for interoperability (e.g. OVF files should support vdi files, but since they have no identifier URL not even VirtualBox itself can import them). VirtualBox Guest Additions can be installed automatically if the ISO is provided in the manifest.
Providers in bootstrap-vz represent various cloud providers and virtual machines.
bootstrap-vz is an extensible platform with loose coupling and a significant amount of tooling, which allows for painless implementation of new providers.
The virtualbox provider for example is implemented in only 89 lines of python, since most of the building blocks are a part of the common task library. Only the kernel and guest additions installation are specific to that provider.
Plugins¶
Admin user¶
This plugin creates a user with passwordless sudo privileges. It also disables the SSH root login. If the EC2 init scripts are installed, the script for fetching the SSH authorized keys will be adjust to match the username specified.
Settings¶
username
: The username of the account to create.required
APT Proxy¶
This plugin creates a proxy configuration file for APT, so you could
enjoy the benefits of using cached packages instead of downloading them
from the mirror every time. You could just install apt-cacher-ng
on
the host machine and then add "address": "127.0.0.1"
and
"port": 3142
to the manifest file.
Settings¶
address
: The IP or host of the proxy server.required
port
: The port (integer) of the proxy server.required
username
: The username for authentication against the proxy server. This is ignored ifpassword
is not also set.optional
password
: The password for authentication against the proxy server. This is ignored ifusername
is not also set.optional
persistent
: Whether the proxy configuration file should remain on the machine or not. Valid values:true
,false
Default:false
.optional
cloud-init¶
This plugin installs and configures cloud-init on the system. Depending on the release it installs it from either backports or the main repository.
cloud-init is only compatible with Debian wheezy and upwards.
Settings¶
username
: The username of the account to create.required
disable_modules
: A list of strings specifying which cloud-init modules should be disabled.optional
metadata_sources
: A string that sets the datasources that cloud-init should try fetching metadata from. The source is automatically set when using the ec2 provider.optional
Commands¶
This plugin allows you to run arbitrary commands during the bootstrap process. The commands are run at an indeterminate point after packages have been installed, but before the volume has been unmounted.
Settings¶
commands
: A list of lists containing strings. Each top-level item is a single command, while the strings inside each list comprise parts of a command. This allows for proper shell argument escaping. To circumvent escaping, simply put the entire command in a single string, the command will additionally be evaluated in a shell (e.g. globbing will work). In addition to the manifest variables{root}
is also available. It points at the root of the image volume.required
manifest vars
Example¶
Create an empty index.html in /var/www and delete all locales except english. .. code:: yaml
- commands:
- commands:
- [touch, ‘{root}/var/www/index.html’]
- [‘rm -rf /usr/share/locale/[^en]*’]
Docker daemon¶
Install docker daemon in the image. Uses init scripts for the official repository.
This plugin can only be used if the distribution being bootstrapped is
at least wheezy
, as Docker needs a kernel version 3.8
or higher,
which is available at the wheezy-backports
repository. There’s also
an architecture requirement, as it runs only on amd64
.
Settings¶
version
: Selects the docker version to install. To select the latest version simply omit this setting. Default:latest
optional
minimize size¶
This plugin can be used to reduce the size of the resulting image. Often virtual volumes are much smaller than their reported size until any data is written to them. During the bootstrapping process temporary data like the aptitude cache is written to the volume only to be removed again.
The minimize size plugin employs three different strategies to keep a low volume footprint:
- Mount folders from the host into key locations of the image volume to avoid any unneccesary disk writes.
- Use zerofree to deallocate unused sectors on the volume. On an unpartitioned volume this will be done for the entire volume, while it will only happen on the root partition for partitioned volumes.
- Use vmware-vdiskmanager to shrink the real volume size (only applicable when using vmdk backing). The tool is part of the VMWare Workstation package.
Settings¶
zerofree
: Specifies if it should mark unallocated blocks as zeroes, so the volume could be better shrunk after this. Valid values: true, false Default: falseoptional
shrink
: Whether the volume should be shrunk. This setting works best in conjunction with the zerofree tool. Valid values: true, false Default: falseoptional
NTP¶
This plugins installs the Network Time Protocol daemon and optionally defines which time servers it should use.
Settings¶
servers
: A list of strings specifying which servers should be used to synchronize the machine clock.optional
Open Nebula¶
This plugin adds OpenNebula contextualization to the image, which sets up the network configuration and SSH keys.
The virtual machine context should be configured as follows:
ETH0_DNS $NETWORK[DNS, NETWORK_ID=2]
ETH0_GATEWAY $NETWORK[GATEWAY, NETWORK_ID=2]
ETH0_IP $NIC[IP, NETWORK_ID=2]
ETH0_MASK $NETWORK[MASK, NETWORK_ID=2]
ETH0_NETWORK $NETWORK[NETWORK, NETWORK_ID=2]
FILES path_to_my_ssh_public_key.pub
The plugin will install all .pub files in the root authorized_keys file. When using the ec2 provider, the USER_EC2_DATA will be executed if present.
Settings¶
This plugin has no settings. To enable it add "opennebula":{}
to the
plugin section of the manifest.
Pip install¶
Install packages from the Python Package Index via pip.
Installs build-essential
and python-dev
debian packages, so
Python extension modules can be built.
Settings¶
packages
: Python packages to install, a list of strings. The list can contain anything thatpip install
would accept as an argument, for exampleawscli==1.3.13
.
prebootstrapped¶
When developing for bootstrap-vz, testing can be quite tedious since the bootstrapping process can take a while. The prebootstrapped plugin solves that problem by creating a snapshot of your volume right after all the software has been installed. The next time bootstrap-vz is run, the plugin replaces all volume preparation and bootstrapping tasks and recreates the volume from the snapshot instead.
The plugin assumes that the users knows what he is doing (e.g. it doesn’t check whether bootstrap-vz is being run with a partitioned volume configuration, while the snapshot is unpartitioned).
When no snapshot or image is specified the plugin creates one and outputs its ID/path. Specifying an ID/path enables the second mode of operation which recreates the volume from the specified snapshot instead of creating it from scratch.
Settings¶
snapshot
: ID of the EBS snapshot to use. This setting only works with EBS backed EC2 configurations.image
: Path to the loopbackvolume snapshot. This setting works with all configurable volume backings except EBS.
Puppet¶
Installs puppet and optionally applies a manifest inside the chroot. You can also have it copy your puppet configuration into the image so it is readily available once the image is booted.
Keep in mind that when applying a manifest, the system is in a chrooted environment. This can prevent daemons from running properly (e.g. listening to ports), they will also need to be shut down gracefully (which bootstrap-vz cannot do) before unmounting the volume. It is advisable to avoid starting any daemons inside the chroot at all.
Settings¶
manifest
: Path to the puppet manifest that should be applied.optional
assets
: Path to puppet assets. The contents will be copied into/etc/puppet
on the image. Any existing files will be overwritten.optional
enable_agent
: Whether the puppet agent daemon should be enabled.optional
root password¶
Sets the root password. This plugin removes the task that disables the SSH password authentication.
Settings¶
password
: The password for the root user.required
Salt¶
Install salt minion in the image. Uses salt-bootstrap script to install.
Settings¶
install_source
: Source to install salt codebase from.stable
for current stable,daily
for installing the daily build, andgit
to install from git repository.required
version
: Only needed if you are installing fromgit
.develop
to install current development head, or provide any tag name or commit hash from salt repooptional
master
: Salt master FQDN or IPoptional
grains
: Set salt grains for this minion. Accepts a map with grain name as key and the grain data as value.optional
Unattended upgrades¶
Enables the unattended update/upgrade feature in aptitude. Enable it to have your system automatically download and install security updates automatically with a set interval.
Settings¶
update_interval
: Days between runningapt-get update
.required
download_interval
: Days between runningapt-get upgrade --download-only
required
upgrade_interval
: Days between installing any security upgrades.required
Vagrant¶
Vagrant is a tool to quickly create virtualized environments. It uses “boxes” to make downloading and sharing those environments easier. A box is a tarball containing a virtual volumes accompanied by an OVF specification of the virtual machine.
This plugin creates a vagrant box that is ready to be shared or deployed. At the moment it is only compatible with the VirtualBox provider and doesn’t requires any additional settings.
Plugins are a key feature of bootstrap-vz. Despite their small size (most plugins do not exceed 100 source lines of code) they can modify the behavior of bootstrapped systems to a great extent.
Below you will find documentation for all plugins available for bootstrap-vz. If you cannot find what you are looking for, consider developing it yourself and contribute to this list!
Supported builds¶
The following is a list of supported manifest combinations.
Note that grub cannot boot from unpartitioned volumes.
Additionally grub installation is not supported on squeeze. This is not a technical limitation, but simply stems from a lack of motivation to implement support for it.
Azure¶
TODO
EC2¶
EBS (wheezy & jessie)¶
Bootloader / Partitioning | none | msdos | gpt |
---|---|---|---|
pvgrub (paravirtualized) | supported | supported | supported |
extlinux (hvm) | supported | supported | supported |
grub (hvm) | not supported | supported | supported |
EBS (squeeze)¶
Bootloader / Partitioning | none | msdos | gpt |
---|---|---|---|
pvgrub (paravirtualized) | supported | supported | supported |
extlinux (hvm) | supported | supported | supported |
grub (hvm) | not supported | not implemented | not implemented |
S3 (all releases)¶
Bootloader / Partitioning | none | msdos | gpt |
---|---|---|---|
pvgrub (paravirtualized) | supported | not implemented | not implemented |
extlinux (hvm) | not implemented | not implemented | not implemented |
grub (hvm) | not supported | not implemented | not implemented |
GCE¶
TODO
KVM¶
TODO
VirtualBox¶
Bootloader / Partitioning | none | msdos | gpt |
---|---|---|---|
extlinux | supported | supported | supported |
grub | not supported | supported | supported |
Logfile¶
Every run creates a new logfile in the logs/
directory. The filename
for each run consists of a timestamp (%Y%m%d%H%M%S
) and the basename
of the manifest used. The log also contains debugging statements
regardless of whether the --debug
switch was used.
Remote bootstrapping¶
bootstrap-vz is able to bootstrap images not only on the machine on which it is invoked, but also on remote machines that have bootstrap-vz installed.
This is helpful when you create manifests on your own workstation, but have a beefed up remote build server which can create images quickly. There may also be situations where you want to build multiple manifests that have different providers and require the host machines to be running on that provider (e.g. EBS backed AMIs can only be created on EC2 instances), when doing this multiple times SSHing into the machines and copying the manifests can be a hassle.
Lastly, the main motivation for supporting remote bootstrapping is the automation of integration testing. As you will see further down, bootstrap-vz is able to select which build server is required for a specific test and run the bootstrapping procedure on said server.
bootstrap-vz-remote¶
Normally you’d use bootstrap-vz
to start a bootstrapping process.
When bootstrapping remotely simply use bootstrap-vz-remote
instead,
it takes the same arguments plus a few additional ones:
--servers <path>
: Path to a list of build-servers (see build-servers.yml for more info)--name <name>
: Selects a specific build-server from the list of build-servers--release <release>
: Restricts the autoselection of build-servers to the ones with the specified release
Much like when bootstrapping directly, you can press Ctrl+C
at any time
to abort the bootstrapping process.
The remote process will receive the keyboard interrupt signal
and begin cleaning up - pressing Ctrl+C
a second time will abort that as
well and kill the connection immediately.
Note that there is also a bootstrap-vz-server
, this file is not meant to be
invoked directly by the user, but is instead launched by bootstrap-vz on the
remote server when connecting to it.
Dependencies¶
For the remote bootstrapping procedure to work, you will need to install
bootstrap-vz as well as the sudo
command on the remote machine.
Also make sure that all the needed dependencies for bootstrapping your image
are installed.
Locally the pip package Pyro4 is needed.
build-servers.yml¶
The file build-servers.yml
informs bootstrap-vz about the different
build servers you have at your disposal.
In its simplest form you can just add your own machine like this:
local:
type: local
can_bootstrap: [virtualbox]
release: jessie
build_settings: {}
type
specifies how bootstrap-vz should connect to the build-server.
local
simply means that it will call the bootstrapping procedure directly,
no new process is spawned.
can_bootstrap
tells bootstrap-vz for which providers this machine is capable
of building images. With the exception of the EC2 provider,
the accepted values match the accepted provider names in the manifest.
For EC2 you can specify ec2-s3
and/or ec2-ebs
.
ec2-ebs
specifies that the machine in question can bootstrap EBS backed
images and should only be used when the it is located on EC2.
ec2-s3
signifies that the machine is capable of bootstrapping S3 backed
images.
Beyond being a string, the value of release
is not enforced in any way.
It’s only current use is for bootstrap-vz-remote
where you can restrict
which build-server should be autoselected.
Remote settings¶
The other (and more interesting) setting for type
is ssh
,
which requires a few more configuration settings:
local_vm:
type: ssh
can_bootstrap:
- virtualbox
- ec2-s3
release: wheezy
# remote settings below here
address: 127.0.0.1
port: 2222
username: admin
keyfile: path_to_private_key_file
server_bin: /root/bootstrap/bootstrap-vz-server
The last 5 settings specify how bootstrap-vz can connect
to the remote build-server.
While the initial handshake is achieved through SSH, bootstrap-vz mainly
communicates with its counterpart through RPC (the communication port is
automatically forwarded through an SSH tunnel).
address
, port
, username
and keyfile
are hopefully
self explanatory (remote machine address, SSH port, login name and path to
private SSH key file).
server_bin
refers to the aboved mentioned
bootstrap-vz-server executable. This is the command bootstrap-vz executes
on the remote machine to start the RPC server.
Be aware that there are a few limitations as to what bootstrap-vz is able to deal with, regarding the remote machine setup (in time they may be fixed by a benevolent contributor):
- The login user must be able to execute sudo without a password
- The private key file must be added to the ssh-agent before invocation (alternatively it may not be password protected)
- The server must already be part of the known_hosts list
(bootstrap-vz uses
ssh
directly and cannot handle interactive prompts)
Build settings¶
The build settings allow you to override specific manifest properties.
This is useful when for example the VirtualBox guest additions ISO is located
at /root/guest_additions.iso
on server 1, while server 2 has it at
/root/images/vbox.iso
.
local:
type: local
can_bootstrap:
- virtualbox
- ec2-s3
release: jessie
build_settings:
guest_additions: /root/images/VBoxGuestAdditions.iso
apt_proxy:
address: 127.0.0.1
port: 3142
ec2-credentials:
access-key: AFAKEACCESSKEYFORAWS
secret-key: thes3cr3tkeyf0ryourawsaccount/FS4d8Qdva
certificate: /root/manifests/cert.pem
private-key: /root/manifests/pk.pem
user-id: 1234-1234-1234
s3-region: eu-west-1
guest_additions
specifies the path to the VirtualBox guest additions ISO on the remote machine.apt_proxy
sets the configuration for the apt_proxy plugin <../plugins/apt_proxy>.ec2-credentials
contains all the settings you know from EC2 manifests, note that when running integration tests, these credentials are also used when running instances.s3-region
overrides the s3 bucket region when bootstrapping S3 backed images.
Changelog¶
2015-06-29¶
- Alex Adriaanse:
- Fix DKMS kernel version error
- Add support for Btrfs
- Add EC2 Jessie HVM manifest
2015-05-08¶
- Alexandre Derumier:
- Fix #219: ^PermitRootLogin regex
2015-05-02¶
- Anders Ingemann:
- Fix #32: Add image_commands example
- Fix #99: rename image_commands to commands
- Fix #139: Vagrant / Virtualbox provider should set ostype when 32 bits selected
- Fix #204: Create a new phase where user modification tasks can run
2015-04-29¶
- Anders Ingemann:
- Fix #104: Don’t verify default target when adding packages
- Fix #217: Implement get_version() function in common.tools
2015-04-28¶
- Jonh Wendell:
- root_password: Enable SSH root login
2015-04-27¶
- John Kristensen:
- Add authentication support to the apt proxy plugin
2015-04-25¶
- Anders Ingemann (work started 2014-08-31, merged on 2015-04-25):
- Introduce remote bootstrapping
- Introduce integration testing (for VirtualBox and EC2)
- Merge the end-user documentation into the sphinx docs (plugin & provider docs are now located in their respective folders as READMEs)
- Include READMEs in sphinx docs and transform their links
- Docs for integration testing
- Document the remote bootstrapping procedure
- Add documentation about the documentation
- Add list of supported builds to the docs
- Add html output to integration tests
- Implement PR #201 by @jszwedko (bump required euca2ools version)
- grub now works on jessie
- extlinux is now running on jessie
- Issue warning when specifying pre/successors across phases (but still error out if it’s a conflict)
- Add salt dependencies in the right phase
- extlinux now works with GPT on HVM instances
- Take @ssgelm’s advice in #155 and copy the mount table – df warnings no more
- Generally deny installing grub on squeeze (too much of a hassle to get working, PRs welcome)
- Add 1 sector gap between partitions on GPT
- Add new task: DeterminKernelVersion, this can potentially fix a lot of small problems
- Disable getty processes on jessie through logind config
- Partition volumes by sectors instead of bytes This allows for finer grained control over the partition sizes and gaps Add new Sectors unit, enhance Bytes unit, add unit tests for both
- Don’t require qemu for raw volumes, use truncate instead
- Fix #179: Disabling getty processes task fails half the time
- Split grub and extlinux installs into separate modules
- Fix extlinux config for squeeze
- Fix #136: Make extlinux output boot messages to the serial console
- Extend sed_i to raise Exceptions when the expected amount of replacements is not met
- Jonas Bergler:
- Fixes #145: Fix installation of vbox guest additions.
- Tiago Ilieve:
- Fixes #142: msdos partition type incorrect for swap partition (Linux)
2015-04-23¶
- Tiago Ilieve:
- Fixes #212: Sparse file is created on the current directory
2014-11-23¶
- Noah Fontes:
- Add support for enhanced networking on EC2 images
2014-07-12¶
- Tiago Ilieve:
- Fixes #96: AddBackports is now a common task
2014-07-09¶
- Anders Ingemann:
- Allow passing data into the manifest
- Refactor logging setup to be more modular
- Convert every JSON file to YAML
- Convert “provider” into provider specific section
2014-07-02¶
- Vladimir Vitkov:
- Improve grub options to work better with virtual machines
2014-06-30¶
- Tomasz Rybak:
- Return information about created image
2014-06-22¶
- Victor Marmol:
- Enable the memory cgroup for the Docker plugin
2014-06-19¶
- Tiago Ilieve:
- Fixes #94: allow stable/oldstable as release name on manifest
- Vladimir Vitkov:
- Improve ami listing performance
2014-06-07¶
- Tiago Ilieve:
- Download gsutil tarball to workspace instead of working directory
- Fixes #97: remove raw disk image created by GCE after build
2014-06-06¶
- Ilya Margolin:
- pip_install plugin
2014-05-23¶
- Tiago Ilieve:
- Fixes #95: check if the specified APT proxy server can be reached
2014-05-04¶
- Dhananjay Balan:
- Salt minion installation & configuration plugin
- Expose debootstrap –include-packages and –exclude-packages options to manifest
2014-05-03¶
- Anders Ingemann:
- Require hostname setting for vagrant plugin
- Fixes #14: S3 images can now be bootstrapped outside EC2.
- Added enable_agent option to puppet plugin
2014-05-02¶
- Tomasz Rybak:
- Added Google Compute Engine Provider
Developers¶
Contributing¶
Sending pull requests¶
Do you want to contribute to the bootstrap-vz project? Nice! Here is the basic workflow:
- Read the development guidelines
- Fork this repository.
- Make any changes you want/need.
- Check the coding style of your changes using tox by running tox -e flake8 and fix any warnings that may appear. This check will be repeated by Travis CI once you send a pull request, so it’s better if you check this beforehand.
- If the change is significant (e.g. a new plugin, manifest setting or security fix) add your name and contribution to the changelog.
- Commit your changes.
- Squash the commits if needed. For instance, it is fine if you have multiple commits describing atomic units of work, but there’s no reason to have many little commits just because of corrected typos.
- Push to your fork, preferably on a topic branch.
- Send a pull request to the master branch.
Please try to be very descriptive about your changes when you write a pull request, stating what it does, why it is needed, which use cases this change covers, etc. You may be asked to rebase your work on the current branch state, so it can be merged cleanly. If you push a new commit to your pull request you will have to add a new comment to the PR, provided that you want us notified. Github will otherwise not send a notification.
Be aware that your modifications need to be properly documented. Please take a look at the documentation section to see how to do that.
Happy hacking! :-)
Development guidelines¶
The following guidelines should serve as general advice when developing providers or plugins for bootstrap-vz. Keep in mind that these guidelines are not rules , they are advice on how to better add value to the bootstrap-vz codebase.
The manifest should always fully describe the resulting image¶
The outcome of a bootstrapping process should never depend on settings specified elsewhere.
This allows others to easily reproduce any setup other people are running and makes it possible to share manifests. The official debian EC2 images for example can be reproduced using the manifests available in the manifest directory of bootstrap-vz.
The bootstrapper should always be able to run fully unattended¶
For end users, this guideline minimizes the risk of errors. Any required input would also be in direct conflict with the previous guideline that the manifest should always fully describe the resulting image.
Additionally developers may have to run the bootstrap process multiple times though, any prompts in the middle of that process may significantly slow down the development speed.
The bootstrapper should only need as much setup as the manifest requires¶
Having to shuffle specific paths on the host into place
(e.g. /target
has to be created manually) to get the bootstrapper
running is going to increase the rate of errors made by users.
Aim for minimal setup.
Exceptions are of course things such as the path to
the VirtualBox Guest Additions ISO or tools like parted
that
need to be installed on the host.
Roll complexity into which tasks are added to the tasklist¶
If a run()
function checks whether it should do any work or simply be
skipped, consider doing that check in resolve_tasks()
instead and
avoid adding that task alltogether. This allows people looking at the
tasklist in the logfile to determine what work has been performed.
If a task says it will modify a file but then bails , a developer may get confused when looking at that file after bootstrapping. He could conclude that the file has either been overwritten or that the search & replace does not work correctly.
Control flow should be directed from the task graph¶
Avoid creating complicated run()
functions. If necessary, split up
a function into two semantically separate tasks.
This allows other tasks to interleave with the control-flow and add extended functionality (e.g. because volume creation and mounting are two separate tasks, the prebootstrapped plugin can replace the volume creation task with a task of its own that creates a volume from a snapshot instead, but still reuse the mount task).
Task classes should be treated as decorated run() functions¶
Tasks should not have any state, thats what the BootstrapInformation object is for.
Only add stuff to the BootstrapInformation object when really necessary¶
This is mainly to avoid clutter.
Use a json-schema to check for allowed settings¶
The json-schema may be verbose but it keeps the bulk of check work outside the python code, which is a big plus when it comes to readability. This only applies bas long as the checks are simple. You can of course fall back to doing the check in python when that solution is considerably less complex.
When invoking external programs, use long options whenever possible¶
This makes the commands a lot easier to understand, since the option names usually hint at what they do.
When invoking external programs, don’t use full paths, rely on $PATH
¶
This increases robustness when executable locations change.
Example: Use log_call(['wget', ...])
instead of log_call(['/usr/bin/wget', ...])
.
Coding style¶
bootstrap-vz is coded to comply closely with the PEP8 style guidelines. There however a few exceptions:
- Max line length is 110 chars, not 80.
- Multiple assignments may be aligned with spaces so that the = match vertically.
- Ignore
E101
: Indent with tabs and align with spaces - Ignore
E221 & E241
: Alignment of assignments - Ignore
E501
: The max line length is not 80 characters - Ignore
W191
: Indent with tabs not spaces
The codebase can be checked for any violations quite easily, since those rules are already specified in the tox configuration file.
tox -e flake8
Documentation¶
When developing a provider or plugin, make sure to update/create the README.rst located in provider/plugin folder. Any links to other rst files should be relative and work, when viewed on github. For information on how to build the documentation and how the various parts fit together, refer to the documentation about the documentation :-)
Developing plugins¶
Developing a plugin for bootstrap-vz is a fairly straightforward process, since there is very little code overhead.
The process is the same whether you create an internal or an external plugin (though you need to add some code for package management when creating an external plugin)
Start by creating an __init__.py
in your plugin folder.
The only obligatory function you need to implement is resolve_tasks()
.
This function adds tasks to be run to the tasklist:
def resolve_tasks(taskset, manifest):
taskset.add(tasks.DoSomething)
The manifest variable holds the manifest the user specified, with it you can determine settings for your plugin and e.g. check of which release of Debian bootstrap-vz will create an image.
A task is a class with a static run()
function and some meta-information:
class DoSomething(Task):
description = 'Doing something'
phase = phases.volume_preparation
predecessors = [PartitionVolume]
successors = [filesystem.Format]
@classmethod
def run(cls, info):
pass
To read more about tasks and their ordering, check out the section on how bootstrap-vz works.
Besides the resolve_tasks()
function, there is also the resolve_rollback_tasks()
function, which comes into play when something has gone awry while bootstrapping.
It should be used to clean up anything that was created during the bootstrapping
process. If you created temporary files for example, you can add a task to the
rollback taskset that deletes those files, you might even already have it because
you run it after an image has been successfully bootstrapped:
def resolve_rollback_tasks(taskset, manifest, completed, counter_task):
counter_task(taskset, tasks.DoSomething, tasks.UndoSomething)
In resolve_rollback_tasks()
you have access to the taskset
(this time it contains tasks that will be run during rollback), the manifest, and
the tasks that have already been run before the bootstrapping aborted (completed
).
The last parameter is the counter_task()
function, with it you can specify that
a specific task (2nd param) has to be in the taskset (1st param) for the rollback
task (3rd param) to be added. This saves code and makes it more readable than
running through the completed tasklist and checking each completed task.
You can also specify a validate_manifest()
function.
Typically it looks like this:
def validate_manifest(data, validator, error):
import os.path
schema_path = os.path.normpath(os.path.join(os.path.dirname(__file__), 'manifest-schema.yml'))
validator(data, schema_path)
This code validates the manifest against a schema in your plugin folder. The schema is a JSON schema, since bootstrap-vz supports yaml, you can avoid a lot of curly braces quotes:
$schema: http://json-schema.org/draft-04/schema#
title: Example plugin manifest
type: object
properties:
plugins:
type: object
properties:
example:
type: object
properties:
message: {type: string}
required: [message]
additionalProperties: false
In the schema above we check that the example
plugin has a single property
named message
with a string value (setting additionalProperties
to false
makes sure that users don’t misspell optional attributes).
Internal plugins¶
Internal plugins are part of the bootstrap-vz package and distributed with it. If you have developed a plugin that you think should be part of the package because a lot of people might use it you can send a pull request to get it included (just remember to read the guidelines first).
External plugins¶
External plugins are packages distributed separately from bootstrap-vz.
Separate distribution makes sense when your plugin solves a narrow problem scope
specific to your use-case or when the plugin contains proprietary code that you
would not like to share.
They integrate with bootstrap-vz by exposing an entry-point through setup.py
:
setup(name='example-plugin',
version=0.9.5,
packages=find_packages(),
include_package_data=True,
entry_points={'bootstrapvz.plugins': ['plugin_name = package_name.module_name']},
install_requires=['bootstrap-vz >= 0.9.5'],
)
Beyond setup.py
the package might need a MANIFEST.in
so that assets
like manifest-schema.yml
are included when the package is built:
include example/manifest-schema.yml
include example/README.rst
To test your package from source you can run python setup.py develop
to register the package so that bootstrap-vz can find the entry-point of your
plugin.
An example plugin is available at https://github.com/andsens/bootstrap-vz-example-plugin, you can use it as a starting point for your own plugin.
Installing external plugins¶
Some plugins may not find their way to the python package index (especially if it’s in a private repo). They can of course still be installed using pip:
pip install git+ssh://git@github.com/username/repo#egg=plugin_name
Documentation¶
Both the end-user and developer documentation is combined into a single sphinx build (the two were previously split between github pages and sphinx).
Building¶
To build the documentation, simply run tox -e docs
in the project root.
Serving the docs through http can be achieved by subsequently running
(cd docs/_build/html; python -m SimpleHTTPServer 8080)
and accessing them
on http://localhost:8080/
.
READMEs¶
Many of the folders in the project have a README.rst which describes the purpose of the contents in that folder. These files are automatically included when building the documentation, through use of the include directive.
Include files for the providers and plugins are autogenerated through the sphinx conf.py script.
Links¶
All links in rst files outside of docs/
(but also docs/README.rst
) that
link to other rst files are relative and reference folder names when the link
would point at a README.rst otherwise. This is done to take advantage of the
github feature where README files are displayed when viewing its parent folder.
When accessing the manifests/
folder for example, the documentation for how
manifests work is displayed at the bottom.
When sphinx generates the documentation, these relative links are
automatically converted into relative links that work inside the generated
html pages instead.
If you are interested in how this works, take a look at the
link transformation module in docs/transform_github_links
.
Commandline switches¶
As a developer, there are commandline switches available which can make your life a lot easier.
--debug
: Enables debug output in the console. This includes output from all commands that are invoked during bootstrapping.--pause-on-error
: Pauses the execution when an exception occurs before rolling back. This allows you to debug by inspecting the volume at the time the error occured.--dry-run
: Prevents therun()
function from being called on all tasks. This is useful if you want to see whether the task order is correct.
Taskoverview¶
How bootstrap-vz works¶
Tasks¶
At its core bootstrap-vz is based on tasks that perform units of work. By keeping those tasks small and with a solid structure built around them a high degree of flexibility can be achieved. To ensure that tasks are executed in the right order, each task is placed in a dependency graph where directed edges dictate precedence. Each task is a simple class that defines its predecessor tasks and successor tasks via attributes. Here is an example:
class MapPartitions(Task):
description = 'Mapping volume partitions'
phase = phases.volume_preparation
predecessors = [PartitionVolume]
successors = [filesystem.Format]
@classmethod
def run(cls, info):
info.volume.partition_map.map(info.volume)
In this case the attributes define that the task at hand should run
after the PartitionVolume
task — i.e. after volume has been
partitioned (predecessors
) — but before formatting each
partition (successors
).
It is also placed in the volume_preparation
phase.
Phases are ordered and group tasks together. All tasks in a phase are
run before proceeding with the tasks in the next phase. They are a way
of avoiding the need to list 50 different tasks as predecessors and
successors.
The final task list that will be executed is computed by enumerating all tasks in the package, placing them in the graph and sorting them topologically. Subsequently the list returned is filtered to contain only the tasks the provider and the plugins added to the taskset.
System abstractions¶
There are several abstractions in bootstrap-vz that make it possible
to generalize things like volume creation, partitioning, mounting and
package installation. As a rule these abstractions are located in the
base/
folder, where the manifest parsing and task ordering algorithm
are placed as well.
API¶
Base functionality¶
The base module represents concepts of the bootstrapping process that tasks can interact with and handles the gather, sorting and running of tasks.
Filesystem handling¶
Volume¶
-
class
bootstrapvz.base.fs.volume.
Volume
(partition_map)[source]¶ Represents an abstract volume. This class is a finite state machine and represents the state of the real volume.
-
_before_link_dm_node
(e)[source]¶ Links the volume using the device mapper This allows us to create a ‘window’ into the volume that acts like a volume in itself. Mainly it is used to fool grub into thinking that it is working with a real volume, rather than a loopback device or a network block device.
Parameters: e (_e_obj) – Event object containing arguments to create() Keyword arguments to link_dm_node() are:
Parameters: - logical_start_sector (int) – The sector the volume should start at in the new volume
- start_sector (int) – The offset at which the volume should begin to be mapped in the new volume
- sectors (int) – The number of sectors that should be mapped
Read more at: http://manpages.debian.org/cgi-bin/man.cgi?query=dmsetup&apropos=0&sektion=0&manpath=Debian+7.0+wheezy&format=html&locale=en
Raises VolumeError: When a free block device cannot be found.
-
Partitionmaps¶
Abstract Partitionmap¶
-
class
bootstrapvz.base.fs.partitionmaps.abstract.
AbstractPartitionMap
(bootloader)[source]¶ Abstract representation of a partiton map This class is a finite state machine and represents the state of the real partition map
-
create
(volume)[source]¶ Creates the partition map
Parameters: volume (Volume) – The volume to create the partition map on
-
get_total_size
()[source]¶ Returns the total size the partitions occupy
Returns: The size of all partitions Return type: Sectors
-
is_blocking
()[source]¶ Returns whether the partition map is blocking volume detach operations
Return type: bool
-
GPT Partitionmap¶
MS-DOS Partitionmap¶
No Partitionmap¶
-
class
bootstrapvz.base.fs.partitionmaps.none.
NoPartitions
(data, sector_size, bootloader)[source]¶ Represents a virtual ‘NoPartitions’ partitionmap. This virtual partition map exists because it is easier for tasks to simply always deal with partition maps and then let the base abstract that away.
Partitions¶
Abstract partition¶
-
class
bootstrapvz.base.fs.partitions.abstract.
AbstractPartition
(size, filesystem, format_command)[source]¶ Abstract representation of a partiton This class is a finite state machine and represents the state of the real partition
-
add_mount
(source, destination, opts=[])[source]¶ Associate a mount with this partition Automatically mounts it
Parameters: - source (str,AbstractPartition) – The source of the mount
- destination (str) – The path to the mountpoint
- opts (list) – Any options that should be passed to the mount command
-
get_end
()[source]¶ Gets the end of the partition
Returns: The end of the partition Return type: Sectors
-
Base partition¶
-
class
bootstrapvz.base.fs.partitions.base.
BasePartition
(size, filesystem, format_command, previous)[source]¶ Represents a partition that is actually a partition (and not a virtual one like ‘Single’)
-
create
(volume)[source]¶ Creates the partition
Parameters: volume (Volume) – The volume to create the partition on
-
get_index
()[source]¶ Gets the index of this partition in the partition map
Returns: The index of the partition in the partition map Return type: int
-
GPT partition¶
MS-DOS partition¶
Single¶
Package handling¶
Package list¶
-
class
bootstrapvz.base.pkg.packagelist.
PackageList
(manifest_vars, source_lists)[source]¶ Represents a list of packages
-
PackageList.
add
(name, target=None)[source]¶ Adds a package to the install list
Parameters: - name (str) – The name of the package to install, may contain manifest vars references
- target (str) – The name of the target release for the package, may contain manifest vars references
Raises: - PackageError – When a package of the same name but with a different target has already been added.
- PackageError – When the specified target release could not be found.
-
Sources list¶
-
class
bootstrapvz.base.pkg.sourceslist.
SourceLists
(manifest_vars)[source]¶ Represents a list of sources lists for apt
Bootstrap information¶
-
class
bootstrapvz.base.bootstrapinfo.
BootstrapInformation
(manifest=None, debug=False)[source]¶ The BootstrapInformation class holds all information about the bootstrapping process. The nature of the attributes of this class are rather diverse. Tasks may set their own attributes on this class for later retrieval by another task. Information that becomes invalid (e.g. a path to a file that has been deleted) must be removed.
-
_BootstrapInformation__create_manifest_vars
(manifest, additional_vars={})[source]¶ Creates the manifest variables dictionary, based on the manifest contents and additional data.
Parameters: - manifest (Manifest) – The Manifest
- additional_vars (dict) – Additional values (they will take precedence and overwrite anything else)
Returns: The manifest_vars dictionary
Return type: dict
-
Manifest¶
The Manifest module contains the manifest that providers and plugins use to determine which tasks should be added to the tasklist, what arguments various invocations should have etc..
-
class
bootstrapvz.base.manifest.
Manifest
(path=None, data=None)[source]¶ This class holds all the information that providers and plugins need to perform the bootstrapping process. All actions that are taken originate from here. The manifest shall not be modified after it has been loaded. Currently, immutability is not enforced and it would require a fair amount of code to enforce it, instead we just rely on tasks behaving properly.
-
load_data
(data=None)[source]¶ Loads the manifest and performs a basic validation. This function reads the manifest and performs some basic validation of the manifest itself to ensure that the properties required for initalization are accessible (otherwise the user would be presented with some cryptic error messages).
-
parse
()[source]¶ Parses the manifest. Well... “parsing” is a big word. The function really just sets up some convenient attributes so that tasks don’t have to access information with info.manifest.data[‘section’] but can do it with info.manifest.section.
-
schema_validator
(data, schema_path)[source]¶ This convenience function is passed around to all the validation functions so that they may run a json-schema validation by giving it the data and a path to the schema.
Parameters: - data (dict) – Data to validate (normally the manifest data)
- schema_path (str) – Path to the json-schema to use for validation
-
validate
()[source]¶ Validates the manifest using the provider and plugin validation functions. Plugins are not required to have a validate_manifest function
-
validation_error
(message, data_path=None)[source]¶ This function is passed to all validation functions so that they may raise a validation error because a custom validation of the manifest failed.
Parameters: - message (str) – Message to user about the error
- data_path (list) – A path to the location in the manifest where the error occurred
Raises ManifestError: With absolute certainty
-
Tasklist¶
The tasklist module contains the TaskList class.
-
class
bootstrapvz.base.tasklist.
TaskList
(tasks)[source]¶ The tasklist class aggregates all tasks that should be run and orders them according to their dependencies.
-
bootstrapvz.base.tasklist.
check_ordering
(task)[source]¶ Checks the ordering of a task in relation to other tasks and their phases.
This function checks for a subset of what the strongly connected components algorithm does, but can deliver a more precise error message, namely that there is a conflict between what a task has specified as its predecessors or successors and in which phase it is placed.
Parameters: task (Task) – The task to check the ordering for Raises TaskListError: If there is a conflict between task precedence and phase precedence
-
bootstrapvz.base.tasklist.
create_list
(taskset, all_tasks)[source]¶ Creates a list of all the tasks that should be run.
-
bootstrapvz.base.tasklist.
get_all_classes
(path=None, prefix='', excludes=[])[source]¶ Given a path to a package, this function retrieves all the classes in it
Parameters: - path (str) – Path to the package
- prefix (str) – Name of the package followed by a dot
- excludes (list) – List of str matching module names that should be ignored
Returns: A generator that yields classes
Return type: generator
Raises Exception: If a module cannot be inspected.
-
bootstrapvz.base.tasklist.
get_all_tasks
(modules)[source]¶ Gets a list of all task classes in the package
Returns: A list of all tasks in the package Return type: list
-
bootstrapvz.base.tasklist.
load_tasks
(function, manifest, *args)[source]¶ Calls
function
on the provider and all plugins that have been loaded by the manifest. Any additional arguments are passed directly tofunction
. The function that is called shall accept the taskset as its first argument and the manifest as its second argument.Parameters: - function (str) – Name of the function to call
- manifest (Manifest) – The manifest
- args (list) – Additional arguments that should be passed to the function that is called
-
bootstrapvz.base.tasklist.
strongly_connected_components
(graph)[source]¶ Find the strongly connected components in a graph using Tarjan’s algorithm.
Source: http://www.logarithmic.net/pfh-files/blog/01208083168/sort.py
Parameters: graph (dict) – mapping of tasks to lists of successor tasks Returns: List of tuples that are strongly connected comoponents Return type: list
-
bootstrapvz.base.tasklist.
topological_sort
(graph)[source]¶ Runs a topological sort on a graph.
Source: http://www.logarithmic.net/pfh-files/blog/01208083168/sort.py
Parameters: graph (dict) – mapping of tasks to lists of successor tasks Returns: A list of all tasks in the graph sorted according to ther dependencies Return type: list
Logging¶
This module holds functions and classes responsible for formatting the log output both to a file and to the console.
-
class
bootstrapvz.base.log.
ColorFormatter
(fmt=None, datefmt=None)[source]¶ Colorizes log messages depending on the loglevel
-
class
bootstrapvz.base.log.
FileFormatter
(fmt=None, datefmt=None)[source]¶ Formats log statements for output to file Currently this is just a stub
-
class
bootstrapvz.base.log.
SourceFormatter
(fmt=None, datefmt=None)[source]¶ Adds a [source] tag to the log message if it exists The python docs suggest using a LoggingAdapter, but that would mean we’d have to use it everywhere we log something (and only when called remotely), which is not feasible.
-
bootstrapvz.base.log.
get_console_handler
(debug, colorize)[source]¶ Returns a log handler for the console The handler color codes the different log levels
Params bool debug: Whether to set the log level to DEBUG (otherwise INFO) Params bool colorize: Whether to colorize console output Returns: The console logging handler
-
bootstrapvz.base.log.
get_file_handler
(path, debug)[source]¶ Returns a log handler for the given path If the parent directory of the logpath does not exist it will be created The handler outputs relative timestamps (to when it was created)
Params str path: The full path to the logfile Params bool debug: Whether to set the log level to DEBUG (otherwise INFO) Returns: The file logging handler
-
bootstrapvz.base.log.
get_log_filename
(manifest_path)[source]¶ Returns the path to a logfile given a manifest The logfile name is constructed from the current timestamp and the basename of the manifest
Parameters: manifest_path (str) – The path to the manifest Returns: The path to the logfile Return type: str
Task¶
-
class
bootstrapvz.base.task.
Task
[source]¶ The task class represents a task that can be run. It is merely a wrapper for the run function and should never be instantiated.
-
classmethod
run
(info)[source]¶ The run function, all work is done inside this function
Parameters: info (BootstrapInformation) – The bootstrap info object.
-
classmethod
Testing¶
Unit tests¶
Integration tests¶
Integration tests test bootstrap-vz in its entirety. This testing includes building images from manifests and creating/booting said images.
Since hardcoding manifests for each test, bootstrapping them and booting the resulting images is too much code for a single test, a testing harness has been developed that reduces each test to it’s bare essentials:
- Combine available manifest partials into a single manifest
- Boot an instance from a manifest
- Run tests on the booted instance
In order for the integration testing harness to be able to bootstrap it must know about your build-servers. Depending on the manifest that is bootstrapped, the harness chooses a fitting build-server, connects to it and starts the bootstrapping process.
When running integration tests, the framework will look for build-servers.yml
at the root of the repo and raise an error if it is not found.
Manifest combinations¶
The tests mainly focus on varying key parts of an image (e.g. partitioning, Debian release, bootloader, ec2 backing, ec2 virtualization method) that have been problem areas. Essentially the tests are the cartesian product of these key parts.
Aborting a test¶
You can press Ctrl+C
at any time during the testing to abort -
the harness will automatically clean up any temporary resources and shut down
running instances. Pressing Ctrl+C
a second time stops the cleanup and quits
immediately.
Manifest partials¶
Instead of creating manifests from scratch for each single test, reusable parts are factored out into partials in the manifest folder. This allows code like this:
partials = {'vdi': '{provider: {name: virtualbox}, volume: {backing: vdi}}',
'vmdk': '{provider: {name: virtualbox}, volume: {backing: vmdk}}',
}
def test_unpartitioned_extlinux_oldstable():
std_partials = ['base', 'stable64', 'extlinux', 'unpartitioned', 'root_password']
custom_partials = [partials['vmdk']]
manifest_data = merge_manifest_data(std_partials, custom_partials)
The code above produces a manifest for Debian stable 64-bit unpartitioned
virtualbox VMDK image.
root_password
is a special partial in that the actual password is
randomly generated on load.
Missing parts¶
The integration testing harness is in no way complete.
- It still has no support for providers other than virtualbox and EC2.
- Creating an SSH connection to a booted instance is cumbersome and does not happen in any of the tests - this would be particularly useful when manifests are to be tested beyond whether they boot up.
Integration test providers¶
The testing framework consists of two parts: The unit tests and the integration tests.
The unit tests are responsible for testing individual parts of bootstrap-vz, while the integration tests test entire manifests by bootstrapping and booting them.
Selecting tests¶
To run one specific test suite simply append the module path to tox:
$ tox -e unit tests.unit.releases_tests
Specific tests can be selected by appending the function name with a colon to the modulepath – to run more than one tests, simply attach more arguments.
$ tox -e unit tests.unit.releases_tests:test_lt tests.unit.releases_tests:test_eq
bootstrap-vz¶
bootstrap-vz is a bootstrapping framework for Debian that creates ready-to-boot images able to run on a number of cloud providers and virtual machines. bootstrap-vz runs without any user intervention and generates ready-to-boot images for a number of virtualization platforms. Its aim is to provide a reproducable bootstrapping process using manifests as well as supporting a high degree of customizability through plugins.
bootstrap-vz was coded from scratch in python once the bash script architecture that was used in the build-debian-cloud bootstrapper reached its limits.
Documentation¶
The documentation for bootstrap-vz is available at bootstrap-vz.readthedocs.org. There, you can discover what the dependencies for a specific cloud provider are, see a list of available plugins and learn how you create a manifest.
Note to developers: The documentaion is generated in a rather peculiar and nifty way.
Installation¶
bootstrap-vz has a master branch for stable releases and a development for, well, development.
After checking out the branch of your choice you can install the
python dependencies by running python setup.py install
. However,
depending on what kind of image you’d like to bootstrap, there are
other debian package dependencies as well, at the very least you will
need debootstrap
.
The documentation
explains this in more detail.
Note that bootstrap-vz will tell you which tools it requires when they aren’t present (the different packages are mentioned in the error message), so you can simply run bootstrap-vz once to get a list of the packages, install them, and then re-run.
Quick start¶
Here are a few quickstart tutorials for the most common images.
If you plan on partitioning your volume, you will need the parted
package and kpartx
:
root@host:~# apt-get install parted kpartx
Note that you can always abort a bootstrapping process by pressing
Ctrl+C
, bootstrap-vz will then initiate a cleanup/rollback process,
where volumes are detached/deleted and temporary files removed, pressing
Ctrl+C
a second time shortcuts that procedure, halts the cleanup and
quits the process.
VirtualBox Vagrant¶
user@host:~$ sudo -i # become root
root@host:~# git clone https://github.com/andsens/bootstrap-vz.git # Clone the repo
root@host:~# apt-get install qemu-utils debootstrap python-pip # Install dependencies from aptitude
root@host:~# pip install termcolor jsonschema fysom docopt pyyaml # Install python dependencies
root@host:~# bootstrap-vz/bootstrap-vz bootstrap-vz/manifests/virtualbox-vagrant.manifest.yml
If you want to use the minimize_size plugin,
you will have to install the zerofree
package and VMWare Workstation as well.
Amazon EC2 EBS backed AMI¶
user@host:~$ sudo -i # become root
root@host:~# git clone https://github.com/andsens/bootstrap-vz.git # Clone the repo
root@host:~# apt-get install debootstrap python-pip # Install dependencies from aptitude
root@host:~# pip install termcolor jsonschema fysom docopt pyyaml boto # Install python dependencies
root@host:~# bootstrap-vz/bootstrap-vz bootstrap-vz/manifests/ec2-ebs-debian-official-amd64-pvm.manifest.yml
To bootstrap S3 backed AMIs, bootstrap-vz will also need the
euca2ools
package. However, version 3.2.0 is required meaning you
must however install it directly from the eucalyptus repository like
this:
apt-get install --no-install-recommends python-dev libxml2-dev libxslt-dev gcc
pip install git+git://github.com/eucalyptus/euca2ools.git@v3.2.0
Cleanup¶
bootstrap-vz tries very hard to clean up after itself both if a run was successful but also if it failed. This ensures that you are not left with volumes still attached to the host which are useless. If an error occurred you can simply correct the problem that caused it and rerun everything, there will be no leftovers from the previous run (as always there are of course rare/unlikely exceptions to that rule). The error messages should always give you a strong hint at what is wrong, if that is not the case please consider opening an issue and attach both the error message and your manifest (preferably as a gist or similar).
Dependencies¶
bootstrap-vz has a number of dependencies depending on the target platform and the selected plugins. At a bare minimum the following python libraries are needed:
To bootstrap Debian itself debootstrap is needed as well.
Any other requirements are dependent upon the manifest configuration and are detailed in the corresponding sections of the documentation. bootstrap-vz will however warn you if a requirement has not been met, before the bootstrapping process begins.
Developers¶
The API documentation, development guidelines and an explanation of bootstrap-vz internals can be found at bootstrap-vz.readthedocs.org.
Contributing¶
Contribution guidelines are described in the documentation under Contributing. There’s also a topic regarding the coding style.