NOTICE!!

tl;dr: This project was renamed to Stapled, which can be found here.

As of now this project has been renamed to Stapled because there are various other projects with the name ocspd which is confusing. Plus we want to implement functional tests in the near future, which are probably going to be based on a package also called ocsdp. This project will be kept here until June 30th 2018, in case you have this repo set as a dependency of some project.

From now on you will get a warning when you install the package, that tells you to use Stapled instead. From the 1st of January 2018 you will get an error instead. From June 30th 2018 onward the installation will stop working entirely.

You can find Stapled here.

Introduction

Why do I need ocspd?

ocspd is meant to be a helper daemon for HAProxy which doesn’t do OCSP stapling out of the box. However HAProxy can serve staple files if they are place in the certificate directory, which is what we use to our benefit.

NOTICE!!

tl;dr: This project was renamed to Stapled, which can be found here.

As of now this project has been renamed to Stapled because there are various other projects with the name ocspd which is confusing. Plus we want to implement functional tests in the near future, which are probably going to be based on a package also called ocsdp. This project will be kept here until June 30th 2018, in case you have this repo set as a dependency of some project.

From now on you will get a warning when you install the package, that tells you to use Stapled instead. From the 1st of January 2018 you will get an error instead. From June 30th 2018 onward the installation will stop working entirely.

You can find Stapled here.

Quick start

Documentation

Read the full documentation on Read the docs.

System requirements

This application requires Python 3.3+ or Python 2.7.9 and an installed version of PIP for the Python version you are using. It is also convenient to have virtualenv installed so you can make a separate environment for ocspd’s dependencies.

Installation

Before installation make sure you have met the System requirements. You can install the ocsp daemon from the source code repository on our gitlab instance.

From github (for developers)

# Download the source from the repo
git clone https://github.com/greenhost/ocspd.git
# Enter the source directory
cd ocspd/
# Setup a virtualenv
virtualenv -p python3 env/
# Load the virtualenv
source env/bin/activate
# Install a dependency that is not yet it PyPi
pip install git+https://github.com/wbond/certvalidator.git@4383a4bfd5e769679bc4eedd1e4d334eb0c7d85a
# Install the current directory with pip. This allows you to edit the code
pip install .

Every time you want to run ocspd you will need to run source env/bin/activate to load the virtualenv first. Alternatively you can start the daemon by running ocspd

Upgrading

If you had previously installed a version of ocspd from github, to upgrade run the following:

# Deactivate the virtualenv if active
deactivate
# Delete the virtualenv (we will start clean)
rm -rf ./env
# Make a new virtualenv
virtualenv -p python3 env/
# Update to the latest version
git pull
# Install a dependency that is not yet it PyPi
pip install git+https://github.com/wbond/certvalidator.git@4383a4bfd5e769679bc4eedd1e4d334eb0c7d85a --upgrade
# Install the current directory with pip. This allows you to edit the code
pip install . --upgrade

Debian package

We package ocspd for Debian, but it will still have depenfencies that are not available as debian packages. This means you need to either still use PIP to install those dependencies, or you need to package them yourself.

There is a build script in the root of this project: build_deb_pkg.sh. It will automatically download the dependencies master branches from Github and package them, the finished packages including a package for ocspd will be in the build directory.

Warning

Do not use this, none of the source code you are about to check out will be audited, you will need to vet it yourself. Also it will cause side effects inluding but not limited to loss of hair, stress and diziness. This is not for production use. We do not take any responsibility for what you do with this script, we provide it as is, it will probably fail anyway but we may also stop supporting it at any time, in fact this is highly likely.

You have been warned, now please don’t continue at your own risk or go for the PIP install.

# Install available dependencies
apt install python-future python-all python-configargparse
# Download remaining dependencies and convert them to debian packages
./build_deb_pkg.sh
# Install all packages
dpkg -i build/*.deb

Using ocspd

Update OCSP staples from CA’s and store the result so HAProxy can serve them to clients.

usage: ocspd [-h] [-c CONFIG] [--minimum-validity MINIMUM_VALIDITY]
             [-t RENEWAL_THREADS] [--verbosity VERBOSITY] [-v] [-D]
             [--file-extensions FILE_EXTENSIONS] [-r REFRESH_INTERVAL]
             [-l [LOGDIR]] [--syslog] [-q]
             [-s HAPROXY_SOCKETS [HAPROXY_SOCKETS ...]] -d DIRECTORIES
             [DIRECTORIES ...] [--no-recycle] [-i IGNORE [IGNORE ...]]

Named Arguments

-c, --config	Override the default config file locations (default=~/.ocspd.conf, /etc/ocspd/ocspd.conf)
--minimum-validity
	If the staple is valid for less than this time in seconds an attempt will be made to get a new, valid staple (default: 7200).
-t, --renewal-threads
	Amount of threads to run for renewing staples. (default=2)
--verbosity	Verbose output argument should be an integer between 0 and 4, canbe overridden by the -v argument.
-v	Verbose output, repeat to increase verbosity, overrides the verbosity argument if provided
-D, --daemon	Daemonise the process, release from shell and process group, run under new process group.
--file-extensions
	Files with which extensions should be scanned? Comma separated list (default: crt,pem,cer)
-r, --refresh-interval
	Minimum time to wait between parsing cert dirs and certificates (default=60).
-l, --logdir	Enable logging to ‘/var/log/ocspd/’. It is possible to supply another directory. Traces of unexpected exceptions are placed here as well.
--syslog	Output to syslog.
-q, --quiet	Don’t print messages to stdout
-s, --haproxy-sockets
	Sockets to connect to HAProxy. Each directory you pass with the directory argument, should have its own haproxy socket. The order of the socket arguments should match the order of the directory arguments.Example:I have a directory /etc/haproxy1 with certificates, and a HAProxy that serves these certificates and has stats socket /etc/haproxy1/haproxy.sock. I have another directory /etc/haproxy2 with certificates and another haproxy instance that serves these and has stats socket /etc/haproxy2/haproxy.sock. I would then start ocspd as follows:./ocspd /etc/haproxy1 /etc/haproxy2 -s /etc/haproxy1.sock /etc/haproxy2.sock
-d, --directories
	Directories containing the certificates used by HAProxy. Multiple directories may be specified separated by a space.
--no-recycle	Don’t re-use existing staples, force renewal.
-i, --ignore	Ignore files matching this pattern. Multiple paths may be specified separated by a space. You can escape the pattern to let the daemon evaluate it instead of letting your shell evaluate it. You can use globbing patterns with * or ?. Relative paths are also allowed.If the path starts with / it will be considered absolute if it does not, the pattern will be compared to the last part of found files.

The daemon will not serve OCSP responses, it can however inform HAPRoxy about the staples it creates using the --haproxy-sockets. argument. Alternatively you can configureHAPRoxy or another proxy (e.g. nginx has support for serving OCSP staples) to serve the OCSP staples manually.

Testing ocspd

Testing an application like this is hard, but that is no excuse not to do testing. We want to have unit tests but to do that correctly we need to run an OCSP server locally, quite a setup. So until now we didn’t do so yet. Note that if you have experience with this kind of setup and you want to help this project move forward, you are welcome to help.

Obviously we do test ocspd, admittedly a little bit primitively. You can find a script in scripts/ called refresh_testdata.sh. It will delete any directory named testdata in the root of the project and create a fresh one. Then it will download 3 certificate chains from live servers. These will be placed in subdirectories with the same name as the domain name.

Next you can run python ocspd -vvvv -d testdata/* to get output printed to your terminal. The testdata/[domain].[tld] directories will be populated with [domain].[tld].ocsp files.

Module description

ocspd consists of several modules that interact with each other in order to keep OCSP staples up-to-date. In short, these are the modules:

Scheduler:It is possible to schedule a task with the scheduler. It will wait for the scheduled moment and add the task to a queue to be handled by one of the other modules. Finder:Finds certificates in the specified directories. When new file are found, or existing files are changed it schedules a parsing for these certificates. Parser:Parses certificates and parses them. If certificates are correct, it schedules a renewal for these certificates. Renewer:The renewer takes input from the scheduler. It contacts the CA to renew an OCSP staple. After renewing the staple it schedules a new renewal and tells the scheduler to call the adder right away. Adder:This is a module that can talk to the HAProxy socket to add OCSP staples without restarting HAProxy.

This graph explains their interaction. Every arrow passes a OCSPTaskContext instance to the other module.

Daemon documentation

Source code

ocspd.main

Initialise the ocspd module.

This file only contains some variables we need in the ocspd name space.

ocspd.FILE_EXTENSIONS_DEFAULT = 'crt,pem,cer'

The extensions the daemon will try to parse as certificate files

ocspd.DEFAULT_REFRESH_INTERVAL = 60

The default refresh interval for the ocspd.core.certfinder.CertFinderThread.

ocspd.MAX_RESTART_THREADS = 3

How many times should we restart threads that crashed.

ocspd.LOG_DIR = '/var/log/ocspd/'

Directory where logs and traces will be saved.

ocspd.DEFAULT_CONFIG_FILE_LOCATIONS = ['~/.ocspd.conf', '/etc/ocspd/ocspd.conf']

Default locations to look for config files in order of importance.

ocspd.core.daemon

This module bootstraps the ocspd process by starting threads for:

• 1x ocspd.scheduling.SchedulerThread

  Can be used to create action queues that where tasks can be added that are either added to the action queue immediately or at a set time in the future.

• 1x ocspd.core.certfinder.CertFinderThread

  • Finds certificate files in the specified directories at regular intervals.
  • Removes deleted certificates from the context cache in ocspd.core.daemon.run.models.
  • Add the found certificate to the the parse action queue of the scheduler for parsing the certificate file.

• 1x ocspd.core.certparser.CertParserThread

  • Parses certificates and caches parsed certificates in ocspd.core.daemon.run.models.
  • Add the parsed certificate to the the renew action queue of the scheduler for requesting or renewing the OCSP staple.

• 2x (or more depending on the -t CLI argument) ocspd.core.ocsprenewer.OCSPRenewerThread

  • Gets tasks from the scheduler in self.scheduler which is a ocspd.scheduling.Scheduler object passed by this module.

  • For each task:

    • Validates the certificate chains.
    • Renews the OCSP staples.
    • Validates the certificate chains again but this time including the OCSP staple.
    • Writes the OCSP staple to disk.
    • Schedules a renewal at a configurable time before the expiration of the OCSP staple.

  The main reason for spawning multiple threads for this is that the OCSP request is a blocking action that also takes relatively long to complete. If any of these request stall for long, the entire daemon doesn’t stop working until it is no longer stalled.

• 1x ocspd.core.ocspadder.OCSPAdder (optional)

  Takes tasks haproxy-add from the scheduler and communicates OCSP staples updates to HAProxy through a HAProxy socket.

ocspd.core.taskcontext

This module defines an extended version of the general purpose scheduling.ScheduledTaskContext for use in the OCSP daemon.

class ocspd.core.taskcontext.OCSPTaskContext(task_name, model, sched_time=None, **attributes)

Adds the following functionality to the scheduling.ScheduledTaskContext:

• Keep track of the exception that occurred last, and how many times it occurred.
• Renames ScheduledTaskContext’s subject argument to model.

__init__(task_name, model, sched_time=None, **attributes)

Initialise a OCSPTaskContext with a task name, cert model, and optional scheduled time.

Parameters:

• task_name (str) – A task name corresponding to an existing queue in the scheduler.
• model (ocspd.core.certmodel.CertModel) – A certificate model.
• sched_time (datetime.datetime|int) – Absolute time (datetime.datetime object) or relative time in seconds (int) to execute the task or None for processing ASAP.
• attributes (kwargs) – Any data you want to assign to the context, avoid using names already defined in the context: scheduler, task_name, subject, model, sched_time, reschedule.

set_last_exception(exc)

Set the exception that occurred just now, this function will return the amount of times the same exception has occurred in a row.

Parameters:exc (Exception) – The last exception. Return int:Count of same exceptions in a row.

Todo

Make sure two similar exceptions are treated as identical, e.g. ignore attributes that will be different every time. https://code.greenhost.net/open/ocspd/issues/15

ocspd.core.certfinder

This module locates certificate files in the supplied directories and parses them. It then keeps track of the following:

• If cert is found for the first time (thus also when the daemon is started), the cert is added to the ocspd.core.certfinder.CertFinder.scheduler so the CertParserThread can parse the certificate. The file modification time is recorded so file changes can be detected.
• If a cert is found a second time, the modification time is compared to the recorded modification time. If it differs, if it differs, the file is added to the scheduler for parsing again, any scheduled actions for the old file are cancelled.
• When certificates are deleted from the directories, the entries are removed from the cache in ocspd.core.daemon.run.models. Any scheduled actions for deleted files are cancelled.

The cache of parsed files is volatile so every time the process is killed files need to be indexed again (thus files are considered “new”).

class ocspd.core.certfinder.CertFinderThread(*args, **kwargs)

This searches directories for certificate files. When found, models are created for the certificate files, which are wrapped in a ocspd.core.taskcontext.OCSPTaskContext which are then scheduled to be processed by the ocspd.core.certparser.CertParserThread ASAP.

Pass refresh_interval=None if you want to run it only once (e.g. for testing)

__init__(*args, **kwargs)

Initialise the thread with its parent threading.Thread and its arguments.

Parameters:

• models (dict) – A dict to maintain a model cache (required).
• directories (iter) – The directories to index (required).
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we add new parse tasks to. (required).
• refresh_interval (int) – The minimum amount of time (s) between search runs, defaults to 10 seconds. Set to None to run only once (optional).
• file_extensions (array) – An array containing the file extensions of file types to check for certificate content (optional).

run()

Start the certificate finder thread.

refresh()

Wraps up the internal CertFinder._update_cached_certs() and CertFinder._find_new_certs() functions.

Note

This method is automatically called by CertFinder.run()

_find_new_certs()

Locate new files, schedule them for parsing.

Raises:ocspd.core.exceptions.CertFileAccessError – When the certificate file can’t be accessed.

_del_model(filename)

Delete model from ocspd.core.daemon.run.models in a thread-safe manner, if another thread deleted it, we should ignore the KeyError making this function omnipotent.

Parameters:filename (str) – The filename of the model to forget about.

_update_cached_certs()

Loop through the list of files that were already found and check whether they were deleted or changed.

If a file was modified since it was last seen, the file is added to the scheduler to get the new certificate data parsed.

Deleted files are removed from the model cache in ocspd.core.daemon.run.models. Any scheduled tasks for the model’s task context are cancelled.

Raises:ocspd.core.exceptions.CertFileAccessError – When the certificate file can’t be accessed.

check_ignore(path)

Check if a file path matches any pattern in the ignore list.

Parameters:path (str) – Path to a file to match.

static compile_pattern(pattern)

Compile a glob pattern and return a compiled regex object.

Parameters:pattern (str) – Glob pattern.

ocspd.core.certparser

This module parses certificate in a queue so the data contained in the certificate can be used to request OCSP responses. After parsing a new ocspd.core.taskcontext.OCSPTaskContext is created for the ocspd.core.oscprenewe.OCSPRenewer which is then scheduled to be processed ASAP.

class ocspd.core.certparser.CertParserThread(*args, **kwargs)

This object makes sure certificate files are parsed, after which a task context is created for the ocspd.core.oscprenewer.OCSPRenewer which is scheduled to be executed ASAP.

__init__(*args, **kwargs)

Initialise the thread with its parent threading.Thread and its arguments.

Parameters:

• models (dict) – A dict to maintain a model cache (required).
• minimum_validity (int) – The amount of seconds the OCSP staple should be valid for before a renewal is scheduled (required).
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we can get parser tasks from and add renew tasks to. (required).
• no_recycle (bool) – Don’t recycle existing staples (default=False)

run()

Start the certificate parser thread.

parse_certificate(model)

Parse certificate files and check whether an existing OCSP staple that is still valid exists. If so, use it, if not request a new OCSP staple. If the staple is valid but not valid for longer than the minimum_validity, the staple is loaded but a new request is still scheduled.

ocspd.core.ocsprenewer

This module takes renew task contexts from the scheduler which contain certificate models that consist of parsed certificates. It then generates an OCSP request and sends it to the OCSP server(s) that is/are found in the certificate and saves both the request and the response in the model. It also generates a file containing the respone (the OCSP staple) and creates a new ocspd.core.taskcontext.OCSPTaskContext to schedule a renewal before the staple expires. Optionally creates a ocspd.core.taskcontext.OCSPTaskContext task context for the ocspd.core.oscpadder.OCSPAdder and schedules it to be run ASAP.

class ocspd.core.ocsprenewer.OCSPRenewerThread(*args, **kwargs)

This object requests OCSP responses for certificates, after which a new task context is created for the ocspd.core.oscprenewer.OCSPRenewer which is scheduled to be executed before the new staple expires. Optionally a task is created for the ocspd.core.oscpadder.OCSPAdder to tell HAProxy about the new staple.

__init__(*args, **kwargs)

Initialise the thread’s arguments and its parent threading.Thread.

Parameters:

• minimum_validity (int) – The amount of seconds the OCSP staple is still valid for, before starting to attempt to request a new OCSP staple (required).
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we can get tasks from and add new tasks to. (required).

run()

Start the renewer thread.

schedule_renew(model, sched_time=None)

Schedule to renew this certificate’s OCSP staple in sched_time seconds.

Parameters:

• context (ocspd.core.certmodel.CertModel) – CertModel instance None to calculate it automatically.
• shed_time (int) – Amount of seconds to wait for renewal or None to calculate it automatically.

Raises:

ValueError – If context.ocsp_staple.valid_until is None

ocspd.core.ocspadder

Module for adding OCSP Staples to a running HAProxy instance.

class ocspd.core.ocspadder.OCSPAdder(*args, **kwargs)

This class is used to add a OCSP staples to a running HAProxy instance by sending it over a socket. It runs a thread that keeps connections to sockets open for each of the supplied haproxy sockets. Code from collectd haproxy connection under the MIT license, was used for inspiration.

Tasks are taken from the ocspd.scheduling.SchedulerThread, as soon

as a task context is received, an OCSP response is read from the model within it, it is added to a HAProxy socket found in self.socks[<certificate directory>].

TASK_NAME = 'proxy-add'

The name of this task in the scheduler

OCSP_ADD = 'set ssl ocsp-response {}'

The haproxy socket command to add OCSP staples. Use string.format to add the base64 encoded OCSP staple

__init__(*args, **kwargs)

Initialise the thread with its parent threading.Thread and its arguments.

Parameters:

• socket_paths (dict) – A mapping from a directory (typically the directory containing TLS certificates) to a HAProxy socket that serves certificates from that directory. These sockets are used to communicate new OCSP staples to HAProxy, so it does not have to be restarted.
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we can get “haproxy-adder” tasks from (required).

_open_socket(key, socket_path)

Opens a socket located at socket_path and saves it in self.socks[key]. Subsequently it asks for a prompt to keep the socket connection open, so several commands can be sent without having to close and re-open the socket.

Parameters:

• key – the identifier of the socket in self.socks
• socket_path (str) – A valid HAProxy socket path.

:raises :exc:ocspd.core.exceptions.SocketError: when the socket can not

be opened.

__del__()

Close the sockets on exit.

run()

The main loop: send any commands that enter the command queue

Raises:ValueError – if the command queue is empty.

add_staple(model)

Create and send the command that adds a base64 encoded OCSP staple to the HAProxy

Parameters:model – An object that has a binary string ocsp_staple in it and a filename filename.

send(socket_key, command)

Send the command through self.socks[socket_key] (using self.socket_paths)

Parameters:

• socket_key (str) – Identifying dictionary key of the socket. This is typically the directory HAProxy serves certificates from.
• command (str) – String with the HAProxy command. For a list of possible commands, see the haproxy documentation

:raises IOError if an error occurs and it’s not errno.EAGAIN or

errno.EINTR

ocspd.core.certmodel

This module defines the ocspd.core.certmodel.CertModel class which is used to keep track of certificates that are found by the ocspd.core.certfinder.CertFinderThread, then parsed by the ocspd.core.certparser.CertParserThread, an OCSP request is generated by the ocspd.core.ocsprenewer.OCSPRenewer, a response from an OCSP server is returned. All data generated and returned like the request and the response are stored in the context.

The following logic is contained within the context class:

• Parsing the certificate.
• Validating parsed certificates and their chains.
• Generating OCSP requests.
• Sending OCSP requests.
• Processing OCSP responses.
• Validating OCSP responses with the respective certificate and its chain.

class ocspd.core.certmodel.CertModel(filename)

Model for certificate files.

__init__(filename)

Initialise the CertModel model object, and read the certificate data from the passed filename.

Raises:ocspd.core.exceptions.CertFileAccessError – When the certificate file can’t be accessed.

parse_crt_file()

Parse certificate, wraps the _read_full_chain() and the _validate_cert() methods. Wicth extract the certificate (end_entity) and the chain intermediates*), and validates the certificate chain.

recycle_staple(minimum_validity)

Try to find an existing staple that is still valid for more than the minimum_validity period. If it is not valid for longer than the minimum_validity period, but still valid, add it to the context but still ask for a new one by returning False.

If anything goes wrong during this process, False is returned without any error handling, we can always try to get a new staple.

Return bool:False if a new staple should be requested, True if the current one is still valid for more than minimum_validity

renew_ocsp_staple()

Renew the OCSP staple, validate it and save it to the file path of the certificate file (certificate.pem.ocsp).

Note

This method handles a lot of exceptions, some of then are non-fatal and might lead to retries. When they are fatal, one of the exceptions documented below is raised. Exceptions are handled by the ocspd.core.excepthandler.ocsp_except_handle() context.

Note

There can be several OCSP URLs. When the first URL fails, the error handler will increase the url_index and schedule a new renewal until all URLS have been tried, then continues with retries from the first again.

Raises:

• RenewalRequirementMissing – A requirment for the renewal is missing.
• OCSPBadResponse – Response is empty, invalid or the status is not “good”.
• urllib.error.URLError – An OCSP url can’t be opened (Python3).
• urllib2.URLError – An OCSP url can’t be opened (Python2).

Raises:

urllib.error.URLError/urllib2.URLError - when a URL/HTTP error occurs

Raises:

socket.error - when a socket error occurs

Todo

Send merge request to ocspbuider, for setting the hostname in the headers while fetching OCSP records. If accepted the request library won’t be needed anymore.

_check_ocsp_response(ocsp_staple, url)

Check that the OCSP response says that the status is good. Also sets ocspd.core.certmodel.CertModel.ocsp_staple.valid_until.

Raises:OCSPBadResponse – If an empty response is received.

_read_full_chain()

Parses binary data in self.crt_data and parses the content. The server certificate a.k.a. end_entity is put in self.end_entity, anything else that has a CA extension is added to self.intermediates.

Note

At this point it is not clear yet which of the intermediates is the root and which are actual intermediates.

Raises:CertParsingError – If the certificate file can’t be read, it contains errors or parts of the chain are missing.

_validate_cert(ocsp_staple=None)

Validates the certificate and its chain, including the OCSP staple if there is one in self.ocsp_staple.

Parameters:ocsp_staple (asn1crypto.core.Sequence) – Binary ocsp staple data. Return array:Validated certificate chain. Raises:CertValidationError – If there is any problem with the certificate chain and/or the staple, e.g. certificate is revoked, chain is incomplete or invalid (i.e. wrong intermediate with server certificate), certificate is simply invalid, etc.

Note

At this point it becomes known what the role of the certiticates in the chain is. With the exception of the root, which is usually not kept with the intermediates and the certificate because ever client has its own copy of it.

__repr__()

We return the file name here because this way we can use it as a short-cut when we assign this object to something.

__str__()

Return a formatted string representation of the object containing: "<CertModel {}>".format("".join(self.filename)) so it’s clear it’s an object and which file it concerns.

__weakref__

list of weak references to the object (if defined)

Scheduler documentation

Scheduler source code

scheduling

This is a general purpose scheduler. It does best effort scheduling and execution of expired items in the order they are added. This also means that there is no guarantee the tasks will be executed on time every time, in fact they will always be late, even if just by milliseconds. If you need it to be done on time, you schedule it early, but remember that it will still be best effort.

The way this scheduler is supposed to be used is to add a scheduling queue, then you can add tasks to the queue to either be put in a task queue ASAP, or at or an absolute time in the future. The queue should be consumed by a worker thread.

This module defines the following objects:

• ocspd.scheduling.ScheduledTaskContext

  A context that wraps around any data you want to pass to the scheduler and which will be added to the task queue when the schedule time expires.

• ocspd.scheduling.SchedulerThread

  An object that is capable of scheduling and unscheduling tasks that you can define with ocspd.scheduling.ScheduledTaskContext.

class ocspd.scheduling.ScheduledTaskContext(task_name, subject, sched_time=None, **attributes)

A context for scheduled tasks, this context can be updated with an exception count for the last exception, so it can be re-scheduled if it is the appropriate action.

__init__(task_name, subject, sched_time=None, **attributes)

Initialise a ScheduledTaskContext with a task name, subject and optional scheduled time. Any remaining keyword arguments are set as attributes of the task context.

Parameters:

• task (str) – A task corresponding to an existing queue in the target scheduler.
• sched_time (datetime.datetime|int) – Absolute time (datetime.datetime object) or relative time in seconds (int) to schedule the task.
• subject (obj) – A subject for the context instance this can be whatever object you want to pass along to the worker.
• attributes (kwargs) – Any additional data you want to assign to the context, avoid using names already defined in the context: scheduler, task, subject, sched_time, reschedule.

scheduler = None

This attribute will be set automatically when the context is passed to a scheduler.

reschedule(sched_time=None)

Reschedule this context itself.

Parameters:sched_time (datetime.datetime) – When should this context be added back to the task queue

__weakref__

list of weak references to the object (if defined)

class ocspd.scheduling.SchedulerThread(*args, **kwargs)

This object can be used to schedule tasks for contexts.

The context should be a ScheduledTaskContext or an extension of it.. When the scheduled time has passed, the context will be added back to the internal task queue(s), where it can be consumed by a worker thread. When a task is scheduled you can choose to have it added to the task queue ASAP or at a specified absolute or relative point in time. If you add it with an absolute time in the past, or a negative relative number, it will be added to the task queue the first time the scheduler checks expired tasks schedule times. If you want to run a task ASAP, you probably don’t that, you should pass sched_time=None instead, it will bypass the scheduling mechanism and place your task directly into the worker queue.

__init__(*args, **kwargs)

Initialise the thread’s arguments and its parent threading.Thread.

Parameters:

• queues (iterable) – A list, tuple or any iterable that returns strings that should be the names of queues.
• sleep (int|float) – The sleep time in seconds between checking the expired items in the queue (default=1)

Raises:

KeyError – If the queue name is already taken (only when queues kwarg is used).

schedule = None

The schedule contains items indexed by time.

scheduled_by_context = None

Keeping the tasks in reverse order helps for faster unscheduling.

scheduled_by_queue = None

Keeping the tasks per queue name helps faster queue deletion.

scheduled_by_subject = None

To allow removing by subject we keep the scheduled tasks by subject.

add_queue(name, max_size=0)

Add a scheduled queue to the scheduler.

Parameters:

• name (str) – A unique name for the queue.
• max_size (int) – Maximum queue depth, [default=0 (unlimited)].

Raises:

KeyError – If the queue name is already taken.

remove_queue(name)

Remove a scheduled queue from the scheduler.

Parameters:name (str) – The name of the existing queue. Raises:KeyError – If the queue doesn’t exist.

add_task(ctx)

Add a ScheduledTaskContext to be added to the task queue either ASAP, or at a specific time.

If the context is not unique, the scheduled task will be cancelled before scheduling the new task.

Parameters:

ctx (ScheduledTaskContext) – A context containing data for a worker thread.

Raises:

• queue.Queue.Full – If the underlying task queue is full.
• TypeError – If the passed context is not a ScheduledTaskContext
• KeyError – If the task queue doesn’t exist.

cancel_task(ctx)

Remove a task from the scheduler.

Note

Tasks that were already queued for a worker to process can’t be canceled anymore.

Parameters:ctx (ScheduledTaskContext) – A context containing data for a worker thread. Return bool:True for successfully cancelled task or False.

get_task(task_name, blocking=True, timeout=None)

Get a task context from the task queue task.

Parameters:

• task_name (str) – Task name that refers to an existsing scheduler queue.
• blocking (bool) – Wait until there is something to return from the queue.

Raises:

• Queue.Empty – If the underlying task queue is empty and blocking is False or the timout expires.
• KeyError – If the task queue does not exist.

task_done(task_name)

Mark a task done on a queue, this up the queue’s counter of completed tasks.

Parameters:task_name (str) – The task queue name. Raises:KeyError – If the task queue does not exist.

run()

Start the scheduler thread.

run_all()

Run all tasks currently queued regardless schedule time.

_run(all_tasks=False)

Runs all scheduled tasks that have a scheduled time < now.

cancel_by_subject(subject)

Cancel scheduled tasks by the task’s context’s subject.

This comes down to: delete anything from the scheduler that relates to my object X.

Parameters:subject (obj) – The object you want all scheduled tasks cancelled for.

Exception handling

During the OCSP renewal proces lots of things could go wrong, some errors are recoverable, others can be ignored, still others could be cause by temporary issues e.g.: a service interruption of the OCSP server in question. So extensive error handling is done to keep the daemons threads running.

The following is an overview of what can be expected when exceptions occur.

Exception	Source	Raised when?	Action
IOError/OSError	certfinder	Directory can’t be read.	Ignore, certfinder will try at every refresh.
CertFileAccessError	certfinder	Certificate file can’t be read.	Schedule retry 3x n*60s, then 3x, every hour, then ignore. [1]
CertParsingError	certparser	Can’t access the certificate file, doesn’t parse or part of the chain is missing.	Ignore, certfinder will try at every refresh.
OCSPBadResponse	ocsprenewer	The response is empty, invalid or the status is not “good”.	Schedule retry 3x n*60s, then 3x, every hour, then twice a day. indefinately. If it’s not a server issue, wait for the file to change [1]
urllib.error.URLError	ocsprenewer	An OCSP url can’t be opened.	We can try again later, maybe there is a server side issue. Some certificates contain multiple URL’s so we will try each one with 10 seconds intervals and then start from the first again. Schedule retry 3x n*60s, then 3x, every hour, then then twice a day.
requests.exceptions.Timeout		Data didn’t reach us within the expected time frame.
requests.exceptions.ReadTimeout
requests.exceptions.ConnectTimeout		A connection can’t be established because the server doesn’t reply within the expected time frame.
requests.exceptions.TooManyRedirects		When the OCSP server redirects us too many times. Limit is quite high so probably something is wrong with the OCSP server.
requests.exceptions.HTTPError		A HTTP error code was returned, this can be a 4xx or 5xx status code.
requests.exceptions.ConnectionError		A connection to the OCSP server can’t be established.
SocketError	ocspadder	A HAProxy socket can not be opened	Log a critical error. Every “send” action will try to re-open the socket.
BrokenPipeError		A HAProxy socket consistently has a broken pipe
OCSPAdderBadResponse		HAProxy does not respond with ‘OCSP Response updated!’	Schedule a retry 3x n*60s, then 3x, every hour, then ignore.

[1]	(1, 2) When the certificate file is changed, certfinder will add the file back to the parsing queue.

ocspd.core.exceptions

This module holds the application specific exceptions.

exception ocspd.core.exceptions.OCSPBadResponse

Gets raised when a OCSP staple is not valid.

exception ocspd.core.exceptions.RenewalRequirementMissing

Gets raised when a OCSP renewal is run while not all requirements are met.

exception ocspd.core.exceptions.SocketError

Gets raised by the OCSPAdder when it is impossible to connect to or use its socket.

exception ocspd.core.exceptions.OCSPAdderBadResponse

Gets raised when the HAProxy does not respond with “OCSP Response updated”

exception ocspd.core.exceptions.CertFileAccessError

Gets raised when a file can’t be accessed at all.

exception ocspd.core.exceptions.CertParsingError

Gets raised when something went wrong while parsing the certificate file.

exception ocspd.core.exceptions.CertValidationError

Gets raised when something went wrong while validating the certificate chain.

ocspd.core.excepthandler

This module defines a context in which we can run actions that are likely to fail because they have intricate dependencies e.g. network connections, file access, parsing certificates and validating their chains, etc., without stopping execution of the application. Additionally it will log these errors and depending on the nature of the error reschedule the task at a time that seems reasonable, i.e.: we can reasonably expect the issue to be resolved by that time.

It is generally considered bad practice to catch all remaining exceptions, however this is a daemon. We can’t afford it to get stuck or crashed. So in the interest of staying alive, if an exception is not caught specifically, the handler will catch it, generate a stack trace and save if in a file in the current working directory. A log entry will be created explaining that there was an exception, inform about the location of the stack trace dump and that the context will be dropped. It will also kindly request the administrator to contact the developers so the exception can be caught in a future release which will probably increase stability and might result in a retry rather than just dropping the context.

Dropping the context effectively means that a retry won’t occur and since the context will have no more references, it will be garbage collected. There is however still a reference to the certificate model in core.daemon.run.models. With no scheduled actions it will just sit idle, until the finder detects that it is either removed – which will cause the entry in core.daemon.run.models to be deleted, or it is changed. If the certificate file is changed the finder will schedule schedule a parsing action for it and it will be picked up again. Hopefully the issue that caused the uncaught exception will be resolved, if not, if will be caught again and the cycle continues.

ocspd.core.excepthandler.LOG_DIR = '/var/log/ocspd/'

This is a global variable that is overridden by ocspd.__main__ with the command line argument: --logdir

ocspd.core.excepthandler.ocsp_except_handle(ctx=None)

Handle lots of potential errors and reschedule failed action contexts.

ocspd.core.excepthandler.delete_ocsp_for_context(ctx)

When something bad happens, sometimes it is good to delete a related bad OCSP file so it can’t be served any more.

Todo

Check that HAProxy doesn’t cache this, it probably does, we need to be able to tell it not to remember it.

ocspd.core.excepthandler.dump_stack_trace(ctx, exc)

Examine the last exception and dump a stack trace to a file, if it fails due to an IOError or OSError, log that it failed so the a sysadmin may make the directory writeable.

Indices and tables

• Index
• Module Index
• Search Page