Reactors SDK

Nearly any language can be used to build functions for TACC’s Abaco serverless computing platform. This is the documentation for a specific, opinionated approach that embeds a client-side Python2.7/3.6 support library called Reactors in the container that hosts an Abaco function.

Reactors extends the Abaco Actors concept with:

• YAML-based configuration mechanism with environment overrides
• Support for first-class mocking and local-side testing
• Semantic aliases for Actors and other TACC.cloud API assets
• Helper methods for working with integrations like Slack and IFTTT
• Introspection of the actor’s platform-level attributes
• Advanced logging with support for redacting sensitive text
• Optimized TACC.cloud API operations

Pre-requisites

1. You must have installed and configured the Agave CLI and the Abaco CLI.
2. You should have gone through the Abaco Quickstart and be familiar deploying, running, and querying an Actor

Start a Project

Concept: The Abaco CLI will automatically generate a skeleton for your Reactor that is preconfigured for easy building, testing, and deployment. This same structure lends itself well to adopting continuous integration and unit testing if you should need those practices. The list of templates is limited at present, and constrained to Python language support. This is expected to change in the future. In addition, it is expected that deeper integration with Github and Gitlab will be added to the Reactors workflow.

Let’s go!

Run abaco init, specifying language (python2 or python3 for now) and a URL-safe name. A good rule-of-thumb is to name a Reactors project like one would a Git repository.

$ abaco init -l python3 -n hello_world
$ ls hello_world/
Dockerfile       config.yml      reactor.py      requirements.txt
TEMPLATE.md      message.jsonschema  reactor.rc      secrets.json.sample

Details on what each project file does are provided in the User Guide.

Configure the Project

Concept: The Dockerfile is a recipe to build the environment where our function will run. The function itself is implemented in reactor.py. A Python module built into the base Docker image (reactors) works with config.yml and message.jsonschema to provide declarative configuration and validation. The requirements.txt file is used with pip in the container image to specify additional Python modules to install. Finally, the Reactors workflow uses reactor.rc to specify name, version, and other metadata, and secrets.json as a way to pass sensitive information into a function without committing it to the container image.

Step 1: Edit config.rc

Naivigate to the project directory and edit DOCKER_HUB_ORG in config.rc to reflect either your Docker Hub username or an organization where you have push and pull access. For example, if a person with the Docker Hub username taco is a member of Docker Hub group cabana, they can choose either taco or cabana as the value for DOCKER_HUB_ORG

Step 2: Edit config.yml

Change the config file to read as follows.

---
logs:
  level: INFO
  token: ~
dont_reveal: ~

Step 3: Create secrets.json

Write a JSON file with the following contents.

{"_REACTORS_DONT_REVEAL": "This is a secret"}

Write some code

Concept: An Abaco function is a script or binary that is set as the default command in a container, accepts a message and parameters from environment variables, and can (optionally) make use of a pre-authenticated Agave API client. Functions can be written in any language, but the Reactors Python SDK streamlines these processes and adds support for some experimental platform features.

Action: Replace the contents of reactor.py

 from reactors.runtime import Reactor


 def main():
     """Main function"""
     r = Reactor()
     r.logger.info("I received: {}".format(r.context['raw_message']))
     r.logger.debug("This is a DEBUG message from actor {}".format(r.uid))
     r.logger.info("This is an INFO message from actor {}".format(r.uid))
     r.logger.warning("This is a warning from actor {}".format(r.uid))

     r.logger.info("Here's that secret value: {}".format(
         r.settings.dont_reveal))

 if __name__ == '__main__':
     main()

This example illustrates use of the Reactor object, specifically, its settings, context, and logging functions. More features and use cases are described in the User Guide and Scenarios sections.

Deploy the Reactor

Concept: Functions can be deployed with the abaco create CLI command using a Docker image that has been built and pushed to a public registry. This is a very flexible approach, but it requires the authorone to execute the same series of steps each time. The abaco deploy command implements a streamlined workflow that, with configuration guidance from reactor.rc, automatically builds the image, pushes it, gathers environment variables, and deploys or updates the Reactor.

Action: Ensure the image builds correctly with a dry run

$ abaco deploy -R

 [INFO]   Build Options: --rm=true --pull
 Sending build context to Docker daemon  10.75kB
 Step 1/1 : FROM sd2e/reactors:python3
 python3: Pulling from sd2e/reactors
 Digest: sha256:789c9057306d618168193c75a6c47ca5c500bc6fcdb60dc30f27f9bf8b1af404
 Status: Image is up to date for sd2e/reactors:python3
 # Executing 5 build triggers
  ---> Using cache
  ---> Using cache
  ---> c06a54dcc66c
 Successfully built c06a54dcc66c
 Successfully tagged taco/hello_world:0.1
 [INFO] Stopping deployment as this was only a dry run!

Action: Deploy the Reactor

$ abaco deploy

 [INFO]   Build Options: --rm=true --pull
 Sending build context to Docker daemon  10.75kB
 Step 1/1 : FROM sd2e/reactors:python3
 python3: Pulling from sd2e/reactors
 Digest: sha256:789c9057306d618168193c75a6c47ca5c500bc6fcdb60dc30f27f9bf8b1af404
 Status: Image is up to date for sd2e/reactors:python3
 # Executing 5 build triggers
  ---> Using cache
  ---> Using cache
  ---> Using cache
  ---> Using cache
  ---> Using cache
  ---> c06a54dcc66c
 Successfully built c06a54dcc66c
 Successfully tagged taco/hello_world:0.1
 The push refers to repository [docker.io/taco/hello_world]
 f9dde2603ec7: Pushed
 87f9719c8a1d: Mounted from sd2e/reactors
 913edbb0371b: Mounted from sd2e/reactors
 0.1: digest: sha256:a944131700e2ae540dc76f2c1c2d72e3909fdfd287b42a505c339ff79615bac7 size: 7184
 [INFO] Pausing to let Docker Hub register that the repo has been pushed
 [INFO] Reading environment variables from secrets.json
 Successfully deployed actor with ID: e6rkEBlzJ8vG4

Validate deployment

Concept: A Reactor that has been deployed successfully will be accessible via the actors API and will report a status of SUBMITTED while the function is being deployed, then READY when it is prepared to accept messages.

Action: List the new actor by its identifier

$ abaco ls e6rkEBlzJ8vG4

The expected response should resemble this JSON document:

{
    "message": "Actor retrieved successfully.",
    "result": {
    "_links": {
      "executions": "https://api.sd2e.org/actors/v2/e6rkEBlzJ8vG4/executions",
      "owner": "https://api.sd2e.org/profiles/v2/taco",
      "self": "https://api.sd2e.org/actors/v2/e6rkEBlzJ8vG4"
    },
    "createTime": "2018-06-21 14:39:16.435800",
    "defaultEnvironment": {
      "_REACTORS_DONT_REVEAL": "This is a secret"
    },
    "description": "",
    "gid": 845005,
    "id": "e6rkEBlzJ8vG4",
    "image": "taco/hello_world:0.1",
    "lastUpdateTime": "2018-06-21 14:39:16.435800",
    "mounts": [
      {
        "container_path": "/work",
        "host_path": "/work",
        "mode": "rw"
      },
      {
        "container_path": "/corral",
        "host_path": "/corral/projects/TACC-Cloud",
        "mode": "rw"
      }
    ],
    "name": "hello_world",
    "owner": "taco",
    "privileged": false,
    "state": {},
    "stateless": false,
    "status": "READY",
    "statusMessage": " ",
    "tasdir": "05201/taco",
    "type": "none",
    "uid": 845005,
    "useContainerUid": false
    },
    "status": "success",
    "version": "0.8.0"
}

Note that result.status is READY - this means the actor is ready to do work. If it reads SUBMITTED, deployment is stil in progress. If it reads ERROR, a problem has been encountered.

Test by sending a message

User Guide

The client-side SDK is based around the Reactor class, which wraps key aspects of the current actor’s execution in a single object to provide helper functions enabled by submodules abacoid, agaveutils, aliases, jsonmessages, and process. This guide will detail usage of Reactor, then describe functions in the submodules that are usable on their own.

Reactor

Concept: A Reactor object has attributes that directly reflect the current Abaco context, attributes that provide linkages to a couple of other handy encapsulations, and some helper functions that provide or require an Agave API client. Critically, an instance of Reactor can be invoked outside the Abaco execution environment and will be populated with mock (but valid-y) attribute values and helper functions for working with TACC APIs initialized with the user’s own credentials.

Attributes

Every Reactor has the following attributes, built from either the Reactor’s execution environment or by means of some clever mocking code in a local test environment:

name	type	contents	source
aliases	aliases.store.AliasStore	A helper for managing App and Actor aliases	Configured on instantiation with current client
client	agavepy.agave.Agave	An active Agave API client	Values provided by Abaco platform using agavepy.actors or local Agave API credentials.
context	dict	Variables passed by Abaco platform to the current execution	Values passed by Abaco and materialized via agavepy.actors or generated by mocking support functions.
execid	string	The current Abaco execution ID	`context.execution_id
local	boolean	True or False to indicate whether the function is running under Abaco or in a testing environment.	The LOCALONLY environment variable, which can be set in local testing configurations.
logger	`logging.StreamHandler`	Pointer to the screen logger of loggers
loggers	dict	Python loggers screen, which logs to both STDERR and (optionally) a structured log aggregator, and slack which can log directly to a Slack channel.	Configured on instantiation, with credentials for Slack and other services provided by environment variables.
nickname	string	A semi-random, human memorable string. (e.g. sleek-lemur)	A call to the petname library
pemagent	agaveutils.recursive.PemAgent	A helper for applying recursive Agave API files permissions. Calls to it will eventually be an asynchronous.	Configured on instantiation with current client
session	string	A correlation key for connecting related events.	Query parameters SESSION or x-session, then nickname
settings	attrdict.AttrDict	Contents of config.yml, where keys are accessible in dot.notation.	Populated at instantiation from config.yml via tacconfig.load_settings()
state	dict	The current Abaco actor state variable	context.state
uid	string	The current Abaco actor ID	`context.actor_id
username	string	TACC.cloud username on whose behalf the actor’s current execution is being undertaken.	Provided by `context.username under Abaco or by inspecting client when running locally.

aliases

This is an instance of AliasStore with which one can create, get, remove, and share alias entries for any actor. See AliasStore documentation for details.

Example Usage

Get an alias

 >>> r = Reactor()
 >>> id = r.aliases.get('slackbot')
 >>> print(id)
 GGPg6KqVrqPl6

Use the ‘me’ built-in

 >>> r = Reactor()
 >>> print(r.uid)
 e50Yz5jaBA4Eo
 >>> id = r.aliases.get('me')
 >>> print(id)
 e50Yz5jaBA4Eo

client

This is an active AgavePy API client used to make authenticated API calls on behalf of the user that invoked execution of the Reactor. All AgavePy functions are supported.

Example Usage

List available public apps

 >>> r = Reactor()
 >>> apps = r.client.apps.list(publicOnly=True)
 >>> type(apps)
 <type 'list'>
 >>> for a in apps:
 >>>    print(a.name)
 psap-0.1.0u1
 xplan-0.1.0u1
 lcms-pyquant-0.1.1u1
 ...

context

This is a dictionary populated from environment variables passed to the container by Abaco.

Example Usage

An example context

 >>> r = Reactor()
 >>> print(r.context)
 AttrDict({'HOSTNAME': '4647e1acfe46', 'LOCALONLY': '1', '_REACTOR_TEMP': '/mnt/ephemeral-01', 'raw_message_parse_log': 'Error parsing message: malformed node or string: None', 'message_dict': {}, 'SCRATCH': '/mnt/ephemeral-01', 'REACTORS_VERSION': '0.7.5', '_': '/usr/bin/python3', 'SHLVL': '1', 'PWD': '/', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'TERM': 'xterm', 'HOME': '/root', '_PROJ_STOCKYARD': '/work/projects/', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'OLDPWD': '/mnt/ephemeral-01', 'MSG': 'Hello There', _USER_WORK': '', '_PROJ_CORRAL': '/corral', 'actor_dbid': '5GJbZRQYk0VmY', 'username': 'tacocat', 'actor_id': '5GJbZRQYk0VmY', 'state': {}, 'execution_id': 'DeAzrr6ABO1pr', 'raw_message': 'Hello There', 'content_type': 'application/json'})
 >>> print(r.context.raw_message)
 Hello There

Context combines environment variables inherited from the container image, set in the container’s worker by Abaco, and passed to the specific execution by Abaco into a single dictionary. In addition to the string value keys, there are two important dict objects: state and message_dict

• state is a dict that can be read from and modified to pass information between executions of an actor without relying on an external database.
• message_dict is populated by parsing a JSON message passed to the actor into a Python dictionary. This is done automatically and safely via Pythons ast and json.loads functions. If a message can’t be parsed, message_dict is set to an empty AttrDict. If you believe you’re sending valid JSON but it’s not resolving as a dictionary, context.raw_message_parse_log can be inspected for clues to what is causing the failure.

execid

This is the unique identifier for the current execution.

Example Usage

Get the execution identifier

 >>> r = Reactor()
 >>> print(r.execid)
 ePmJlZOg0W03

local

This is a boolean value set based on the state of the LOCALONLY environment variable. It is intended to be used to selectively disable or enable code branches when running under local emulation.

An environment-specific print() statement

 r = Reactor()
 if r.local is not True:
     print('This code is not running under a test environment')
 else:
     print('This code is running locally')

The state of local is set automatically by some CI support scripts, and can be set in a pytest environment using the monkeypatch fixture.

Monkeypatching local to return True

 def test_demo_local(monkeypatch):
     monkeypatch.setenv('LOCALONLY', 1)
     r = Reactor()
     assert r.local is True

logger

This is an ready-to-use Python logger.

Example Usage

Get the execution identifier

 >>> r = Reactor()
 >>> r.logger.info('This is a log message')
 5AB11Q8XxwPK5 INFO This is a log message

A nicely formatted message is printed to STDERR that includes the current actor ID. All logging levels (debug, info, warning, critical) are available. Logging level is set in the logs stanza of config.yml.

At the same time a plaintext message is sent to the standard log, it can (optionally) be sent over the network in a structured format. This is described in the advanced topics section.

Log redaction

Sensitive data passed into a Reactor using the secrets mechanism are redacted automatically when logged. For instance, assume API credentials for AWS have been set in a Reactor. Under a standard logging scheme, it would be very easy to print those sensitive data in build logs, screenshots, commits, and such, where it could be easily discoverable.

Redaction in action

 >>> r = Reactor()
 >>> api_secret = r.settings.api.secret
 >>> r.logger.info('Here are credentials: {}'.format(api_secret))
 Pk4B11Q8Xxw INFO Here are credentials: ****
 >>> api_url = r.settings.api.url
 >>> r.logger.info('Here is API server: {}'.format(api_urk))
 Pk4B11Q8Xxw INFO Here is API server: https://tacos.tacc.cloud/api/v1

loggers

This dict holds references to all loggers configured by a Reactor. At present, two loggers are established. The default logger, screen, prints to STDERR and (optionally) a log aggregator. The other logger, slack, allows logging directly to Slack assuming a webhook is provided when the actor is configured.

Slack logger configuration

---
slack:
  channel: "notifications"
  icon_emoji: ":smile:"
  username: "tacobot"
  webhook: ~

Using loggers

 >>> r = Reactor()
 >>> r.loggers['screen'].info('This actor is a logger and a slacker')
 5AB11Q8XxwPK5 INFO This actor is a logger and a slacker
 >>> r.loggers['slack'].info('This actor is a logger and a slacker')

nickname

Inspired by the naming of Docker containers and other cloud resources, each Reactor invocation is assigned a “human meaningful, decentralized, secure” nickname generated by the petname library. By default, two-word nicknames are used, but this can be overridden with an entry in config.yml

---
nickname:
  length: 3

Example Usage

View the current nickname

 >>> r = Reactor()
 >>> print(r.nickname)
 sleepy-wallaby

pemagent

This is an instance of PemAgent, a helper for recursive Agave files permissions management. Most permissions operations should be handled directly using AgavePy files.*Permissions commands. The pemagent helper exists provide an optimized, and potentially asynchronous, method for doing batch operations.

Example Usage

Let user tacopal read a specific directory

 r = Reactor()
 r.pemsagent.grant(systemId='data.tacc.cloud',
                   absPath='/demo',
                   username='tacopal',
                   pem='READ')

session

This string is a correlation identifier among platform events with a common ancestry. Its value is set as follows:

1. If environment variable x-session is not empty, session takes on its value
2. Else, if environment variable SESSION is not empty, session takes on its value
3. Else, session is set to the value of nickname

Critically, actors can message other actors. When this is done using Reactor.send_message, the session value is forwarded along to the receipients as x-session and will thus become their session identifier.

Example Usage

An Agave API job may send a string, such as job name or ID as SESSION, when messaging an Abaco actor. In this example, demojob is sent as a value for SESSION.

{ "notifications": [
    {
      "url": "https://api.tacc.cloud/actors/v2/eZE7XDPLzZOwo/messages?x-nonce=SD2E_KbyGjq4XOM4&channel=agavejobs&SESSION=demojob",
      "event": "FINISHED",
      "persistent": false
    }
}

The value of session in the downstream Reactor instance will be demojob. If that Reactor messages another reactor, the downstream entity’s session will also be demojob.

settings

state

uid

username

Functions

Assistants

abacoids

agaveutils

aliases

jsonmessages

process

Third-party Webhooks

Agave API Notifications

Finite State Machine

Schedule Actions

Automate Deployment

Unit Testing

RabbitMQ

AWS SNS

Getting Help

TACC.cloud Slack

You are welcome to join the developers and users of TACC.cloud services in TACC.cloud Slack. Helpful channels to join include #support and #announcements

Tenant-specific Assistance

If you are a user from any of the following organizations, you can get help from additional listed resources.

• CyVerse
• DesignSafe
• Synergistic Discovery and Design (SD2)

Other Tutorials and Guides

Use third-party Docker images

Import from other Serverless systems

Extend to other languages

Contribute to Reactors

Indices and tables

• Index
• Module Index
• Search Page