Holmes Processing: Distributed Large-Scale Analysis

Holmes Processing was born out of the need to rapidly process and analyse large volumes data in the computer security community. At its core, Holmes Processing is a catalyst for extracting useful information and generate meaningful intelligence. Furthermore, the robust distributed architecture allows the system to scale while also providing the flexibility needed to evolve.

Holmes Processing Architecture is based on 3 core pillars.

• Resilient: Failures should be gracefully handled and not affect other parts of the system.
• Scalable: The system should be easily able to scale vertically and horizontally.
• Flexible: Components should be interchangeable and new features should be easy to add.

Introduction

What is Holmes?

TODO

How does Holmes Work?

TODO

Architecture Design

HolmesProcessing architecture is based on Skald (A Scalable Architecture for Feature Extraction, Multi-User Analysis, and Real-Time Information Sharing)

Data-Information-Knowledge-Wisdom (DIKW)

Skald

Planner

Transport

Services

Information Extraction

Knowledge Generation

Functionality

Quick Start

This section explains how to quick start HolmesProcessing.

Deployment Strategies

Components

Holmes Totem

What is Totem?

Architecture Overview

Installation

Dependencies

In order to compile Holmes-Totem you have to install sbt (Scala Build Tool). See the official `website <sbt_>`_ for more info.

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt.list > /dev/null
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install -y sbt

Configuration

There’s two files of interest here:

• config/docker-compose.yml
• config/totem.conf

The docker-compose config file is responsible for service launch configuration. Each entry in the services: section is of the form:

<name>:
  build:
    context: ../src/main/scala/org/holmesprocessing/totem/services/<name>
    args:
      conf: ${CONFSTORAGE_<name_uppercase>}service.conf
  ports:
    - "<port>:8080"
  restart: unless-stopped
  volumes:
    - /tmp:/tmp:ro

<name>	The name of the service, all lowercase usually and the same as the folder name.
<name_uppercase>	The same as the <name>, but uppercase.
<port>	The service port. Convention is 7700,7710,7720, ... for the services processing files and 9700,9710,9720, ... for the services that do not. The simple reason for this is, that scaling the service deployment using a solution like docker-compose requires some free ports.

The totem config file mainly tells Totem what services are available and where to find them. But you can also configure its request timeouts here.

download_settings {
  connection_pooling = true
  connection_timeout = 1000
  download_directory = "/tmp/"
  thread_multiplier = 4
  request_timeout = 1000
}

tasking_settings {
  default_service_timeout = 180
  prefetch = 3
  retry_attempts = 3
}

rabbit_settings {
  requeueKey = "requeue.static.totem"
  host {
    server = "127.0.0.1"
    port = 5672
    username = "guest"
    password = "guest"
    vhost = "/"
  }

  ...
}

services {
  asnmeta {
    uri = ["http://127.0.0.1:9700/analyze/?obj="]
    resultRoutingKey = "asnmeta.result.static.totem"
  }

  ...
}

The above is an excerpt of the default configuration file. It is important to note that the connection_timeout and request_timeout are (counter-intuitively) not just associated with downloading samples. They apply to “downloading” results from services as well. If you experience a lot of service failures due to timeouts consider increasing these values. Additionally the tasking_settings.default_service_timeout may need changing, too. (The former two are given as milliseconds, the latter as seconds)

Most settings regarding RabbitMQ are of no interest to a regular user. The only things that need to be adjusted are the credentials and the address.

More interesting are the service entries. They offer the ability to configure multiple URIs for each service for automatic load balancing. The schema for the URIs and the routing key is always the same though.

"http://<address>/analyze/?obj="

"<servicename>.result.stastic.totem"

The suffix result.static.totem always corresponds to the suffix defined in the RabbitMQ settings.

Running

After installing sbt clone Holmes-Totem from the GitHub repository and build it from source.

mkdir -p /data/holmes-totem
cd /data/holmes-totem
git clone https://github.com/HolmesProcessing/Holmes-Totem.git .
sbt assembly

This will produce a jar-file. To start Totem, issue:

java -jar target/scala-2.11/totem-assembly-1.0.jar config/totem.conf

However, you’ll need to configure it first. (See the configuration documentation)

Executing Tasks

Services

Overview

Holmes Totem Static Services perform analysis on a given object and provide a result.

The Services in Holmes Processing are designed as microservices. The Planners in Holmes are the central orchestration engine for ensuring that the Services performing the given task at an optimal level. Under this umbrella, Holmes Processing uses Holmes Totem Services which is built on the ideas of micro services. These are pluggable services that we can use or remove easily according to the needs.

Holmes Totem Services are kind of web servers that communicate with Totem via HTTP protocol. They do analysis using a particular malware analysis library (we call it analyzer library) and returns result. Totem sends a request for analysis and Services responds with final result of analysis along with HTTP error code. This final result will eventually gets stored in Holmes Storage in JSON format.

Each Service runs in an isolated environment with the illusion that they are the only process running in the system. We use Docker containers to isolate a Totem Service. Each Totem-Services are loosely coupled and does analysis individually. They don’t depend on other Totem-Service to create the final output. Also, there will not be any interprocess communication between Totem Services. The reason for this design choice is that this improves fault-tolerance and provides great flexibility.

Analyser Library

Analyser library is a module to parse and work with a particular file type.

These modules parse the file and get all the information/properties of the file type. Later Interrogation planner will infer and score the sample.

The Totem Service takes a sample through GET request, and gives it to the analyser library. The library will parse the file and provides an output. The Service then uses this output and properly serialize the data and returns the result to TOTEM.

The analyzer library can also be any malware analysis tool. We can either directly import the library or directly parse the command line output.

If the analyzer library is in Python, we would suggest creating a Totem Service in Python. If the analyser library is in C or if the tool is command line, then we would suggest using Go

Structure

A Holmes Totem Service requires four files. These files are

• Scala File : Logic for communication with totem.
• Configuration File : Configuration settings which can be changed as needed.
• Service Logic : Web Server and Logic of the entire service.
• Dockerfile : To containerize and isolate the entire Service

Configuration file

Let’s discuss how Holmes being a distributed system, uses configuration files. Configuration files configure initial settings for the Services. The settings of each of these Services will be stored in a configuration file.

Service Configuration for Holmes Totem

The essential settings needed for starting Totem Service are located in configuration file. The file format of Holmes’ configuration files is .conf. The format of the text is JSON. A user can change these values to change the behavior of the Service.

For each Service, The Service author should create a file called service.conf where the essential configuration settings for the service like HTTPBinding, Maximum number of objects etc. A type Totem Service configuration file will look like this:

{
        "settings": {
                "httpbinding": 8080
        },
        "<service_name": {
                "<Key": "Value",
                "<key>": "Value"
        }
}

Warning

All the services by default should run on port 8080 so that docker compose can port forward it to a specified port

Centralised Service configuration of Holmes Totem

Holmes system allows an admin to store the Totem Service configurations in a central location and make the system automatically load it from there upon upstart. This is useful when you start up multiple Totem at different locations all working with the same service configuration because copying of all the Services everywhere is quite tedious. Instead, you only have to modify the config file on one machine and upload it and then rebuilt the containers on all the machines. It makes distributed service configuration changes easier.

Holmes Storage is extended to allow for storing configuration-files (uploaded over HTTP) into its database and query them (also over HTTP). The upload_config.sh. script can be used for uploading configuration to Storage. This script creates environment variables CONFSTORAGE which storage the URI where the configuration files are stored in Storage and configuration files are uploaded to storage. The compose_download_conf.sh. script creates the environment variable for each Service like for example CONFSTORAGE_ASNMETA which contains the URI of the configuration file for the Service

The docker-compose.yml.example therefore takes a look at environment variables pointing to the running instance of Holmes Storage and sets these arguments correctly for each Service. By doing this, whenever the containers are built, the configuration-files are pulled from the server. As you can see in the docker-compose.yml.example, the Services always have a line like this:

conf: ${CONFSTORAGE_ASNMETA}service.conf

Usually the environment variable CONFSTORAGE_ASNMETA (and all the others as well) are empty, so the Dockerfiles just get the local config version

Also The following are the modifications done for Dockerfile to accept an argument specifying the location of the file service.conf

# add the configuration file (possibly from a storage uri)
ARG conf=service.conf
ADD $conf /service/service.conf

If docker-compose did not set the conf-argument, it defaults to service.conf, otherwise it is left as it was. Docker’s ADD can also download the file via HTTP.

Reading Configuration files for Services.

Each Service runs independently in an isolated docker container. The configuration settings for the Service has to be provided in service.conf. When The service runs, it first looks for the configuration file, in order to read its settings, and applies them to the Service. The configuration file has to be in JSON format.

Service Logic

Communication

Currently Holmes Totem Services uses REST approach which leverages the HTTP protocol for communication. Totem requests a task to the Service by GET request. The format of url is

GET http://address:port/analyze/obj?=sample-id

When a request to /analyze route is made, totem looks for the sample in Holmes Storage, if the sample if found, that will be submitted for analysis to the Service and the Service responds with result. If TOTEM could not find the sample in Storage, It simply returns 404 HTTP error. The various endpoints through which a Service can be interacted with is written in API Endpoints

API Endpoints

GET /

Returns general documentation information about the Service. * Resource URL: http://address:port/ * Parameters: None

GET /analyse

Returns the analysis result for a given sample

• Resource URL: http://address:port/analyze/?obj=sample-id
• Parameters: The name of the object to be analysed.
• Example Request : http://address:port/analyze/?obj=sample-id

Example Response : For example response, please refer to README.md of any Service

Scala File

Holmes-Totem schedules the execution of its Services. Holmes Totem Services are web servers that receive tasks via HTTP request. This file tells Service How to interact with Totem. Totem imports this file in driver.scala and schedules the task

Containerization

Since we are trying to analyse malware sample, there could be a risk that analysis could could damage a environment in which Service is running on. To minimize this risk, we should we use sandbox environment.

For our purpose, we generally need a Virtual Machine. But only need virtualization for the sake of isolation. And we want them to be lightweight. So Docker is ideal for our requirements. To pack and isolate the above discussed parts (except Scala file), we need to do containerization. A typical Dockerfile of Holmes Totem Service will look like:

FROM <base-image>

# Create folder
RUN mkdir -p /service
WORKDIR /service

# Get Language dependencies
RUN apk add --no-cache \
        git \
        && rm -rf /var/cache/apk/*

# Get Analyzer Library dependencies
RUN apk add --no-cache \
           && rm -rf /var/cache/apk/*

# Clean Up
RUN apk del --purge \
        git

# Set environment variables.

# Installing Analyzer library

# add Service files to the container.
COPY LICENSE /service
COPY README.md /service
COPY service.{go, py} /service

# build the service file

# add the configuration file (possibly from a storage uri)
ARG conf=service.conf
ADD $conf /service/service.conf

# Run the Service
CMD ["./service", "--config=service.conf"]

BASE IMAGE:

The FROM directive in dockerfile is used to mention the base image. To make the container light weight, we use Alpine Linux.

You need to choose the docker image suitable for the task you are trying to achieve. That is choosing the right container for the language in which in you are writing Service.

For Go: FROM golang:alpine

For Python: FROM python:alpine

Services-Output

HTTP Error Codes

Holmes Services attempts to return appropriate HTTP error code when a requested for analysis. This codes will be used by the watchdog to manage results.

The Error codes table follow this pattern:

• 200 (OK) - Service returns analysis results in JSON format.
• 4XX (client errors)
• 5XX (server errors) from Service logic section
• 6XX - Custom Errors

These are general error codes:

Code	Message	Description
400	Bad request	Trying use new endpoint other than /`and `/analyse
401	Unauthorised	Invalid authentication
404	Not Found	The resource cannot be found.
500	Internal Server Error	When there is problem in the service logic
503	Cannot Create JSON	JSON encoding error

Apart from these, you can define your own custom error codes. We would suggest doing so starting with 600. (refer to PEMETA service which uses error codes to report error to TOTEM)

Code	Message	Description
601	ALLOCATION FAILURE	Cannot allocate memory to LIBPE function

These error codes will notify watchdog about the behavior of the service and based on these error codes, watchdog can manage results more intelligently

Based on the error codes, the behavior of TOTEM changes which will be discussed in Holmes-TOTEM sections.

JSON

The mime type of the data returned by Totem Service is JSON. The output should be meaningful so that it it becomes easier for the interrogation planner to produce knowledge from the results.

The output is a typical json (* part of output for returned by pemeta*)

{
        "Exports": {},
        "Resources": "",
        "Imports": [
                {
                "DllName": "SHELL32.dll",
                        "Functions": [
                                "ShellExecuteA",
                                "FindExecutableA"
                        ]
                }
        ]
}

Note

All the first level keys of the output JSON should be property of the file that the Service is supposed to analysed. If the a particular property is absent in the file, then the value for the key should empty.

In the above snippet, Imports is a property of PE file. Imports is a list of DLLs. Each item of the list contains Dll Name and functions present in that DLL

Also, Exports value is an empty struct. This means that the service has analysed Exports, but could not find any results.

Extending Holmes Processing

Services for Totem

This chapter contains detailed information about how to write a new Service and a Service example.Totem’s Services are pretty simple to understand. Any programming language could be used to implement your service. They build upon JSON as a messaging format, use a RESTful API and are completely independent of Totem itself.

The Scala you need to add one is, in most cases, as simple as it can be: copy and paste.

This tutorial uses Python because it’s simple and the services for the popular analysis platform CRITs are written in it and Golang because in Go you can call C programs or functions and most of the binary analysis tool are written in C language.

Also, any kind of virtualization method can be used to restrict the environment of your Service. This tutorial shows how to use Docker.

Writing a Service

Overview

For a service to be accepted into the main repository, the proper additions to the following files need to be made:

• Config Files

  • totem.conf
  • docker-compose.yml.example
  • compose_download_conf.sh

• Scala Files

  • driver.scala

Additionally the following files need to be added:

• Service Files

  • Dockerfile
  • LICENSE
  • README.md
  • acl.conf
  • service.conf.example
  • watchdog.scala
  • <SERVICE_NAME_CAMELCASE>REST.scala

• Any additional files your service needs

  • Python e.g.: service.py
  • Go e.g.: service.go

Warning

Try to stick to the naming convention (uppercase/lowercase/camelcase were appropriate) to avoid confusion!

If you did not write a service yet, please consult the subcategories.

Details

Files To Add

The following table declares variables used in the sections below.

These need to be replaced by their respective values.

Compare with other service implementations for suitable values.

• All additional files required by your service also belong into the folder src/main/scala/org/holmesprocessing/totem/services/SERVICE_NAME.

SERVICE_NAME	The name of the service, e.g. My_Totem_Service
SERVICE_NAME_CAMELCASE	CamelCase version of your service’s name, e.g. MyTotemService
SERVICE_NAME_UPPERCASE	Upper case version of the service’s name, e.g. MY_TOTEM_SERVICE
SERVICE_NAME_LOWERCASE	Lower case version of your service’s name, e.g. my_totem_service
CONFSTORAGE_SERVICE_NAME	String CONFSTORAGE_ followed by an upper case version of your service’s name, e.g. CONFSTORAGE_MY_TOTEM_SERVICE
SERVICE_IP	The IP at which the service is reachable. Usually this should be 127.0.0.1.
SERVICE_PORT	The convention for ports is simple, file processing services are in the 77xx range, no-file services are in the 79xx range. Additionally by default there is a gap of 10 free ports per service, so if the first service starts at 7700 the second starts at 7710.
SERVICE_CLASS_SUCCESS	The name of the service’s success class, e.g. MyTotemServiceSuccess
SERVICE_CLASS_WORK	The name of the service’s work class, e.g. MyTotemServiceWork

Service Files

All files in this section belong into the folder src/main/scala/org/holmesprocessing/totem/services/SERVICE_NAME.

Dockerfile

In order to make optimal use of Docker’s caching ability, you must use the given Dockerfiles and extend them according to your services needs.

• Dockerfile for Python2:

  FROM python:2-alpine
  
  # add tornado
  RUN pip install tornado
  
  # create folder
  RUN mkdir -p /service
  WORKDIR /service
  
  # add holmeslibrary
  RUN apk add --no-cache \
          wget \
      && wget https://github.com/HolmesProcessing/Holmes-Totem-Service-Library/archive/v0.1.tar.gz \
      && tar xf v0.1.tar.gz \
      && mv Holmes-Totem-Service* holmeslibrary \
      && rm -rf /var/cache/apk/* v0.1.tar.gz
  
  ###
  # service specific options
  ###
  
  # INSTALL SERVICE DEPENDENCIES
  #
  #   RUN pip install <stuff>
  #   RUN apk add --no-cache <stuff>
  #   RUN wget <url> && tar xf <stuff> ...
  #   ...
  #
  
  # add the files to the container
  COPY LICENSE /service
  COPY README.md /service
  COPY service.py /service
  
  # ADD FURTHER SERVICE FILES
  #
  #   COPY specialLibrary/ /service/specialLibrary
  #   COPY extraFile.py /service
  #   ...
  #
  
  # add the configuration file (possibly from a storage uri)
  ARG conf=service.conf
  ADD $conf /service/service.conf
  
  CMD ["python2", "service.py"]
  

• Dockerfile for Python3

  FROM python:alpine
  
  # add tornado
  RUN pip3 install tornado
  
  # create folder
  RUN mkdir -p /service
  WORKDIR /service
  
  # add holmeslibrary
  RUN apk add --no-cache \
          wget \
      && wget https://github.com/HolmesProcessing/Holmes-Totem-Service-Library/archive/v0.1.tar.gz \
      && tar xf v0.1.tar.gz \
      && mv Holmes-Totem-Service* holmeslibrary \
      && rm -rf /var/cache/apk/* v0.1.tar.gz
  
  ###
  # service specific options
  ###
  
  # INSTALL SERVICE DEPENDENCIES
  #
  #   RUN pip install <stuff>
  #   RUN apk add --no-cache <stuff>
  #   RUN wget <url> && tar xf <stuff> ...
  #   ...
  #
  
  # add the files to the container
  COPY LICENSE /service
  COPY README.md /service
  COPY service.py /service
  
  # ADD FURTHER SERVICE FILES
  #
  #   COPY specialLibrary/ /service/specialLibrary
  #   COPY extraFile.py /service
  #   ...
  #
  
  # add the configuration file (possibly from a storage uri)
  ARG conf=service.conf
  ADD $conf /service/service.conf
  
  CMD ["python3", "service.py"]
  

• Dockerfile for Go 1.7:

  FROM golang:alpine
  
  # create folder
  RUN mkdir -p /service
  WORKDIR /service
  
  # get go dependencies
  RUN apk add --no-cache \
          git \
      && go get github.com/julienschmidt/httprouter \
      && rm -rf /var/cache/apk/*
  
  ###
  # passivetotal specific options
  ###
  
  # INSTALL SERVICE DEPENDENCIES
  #
  #   RUN go get <stuff>
  #   RUN apk add --no-cache <stuff>
  #   RUN wget <url> && tar xf <stuff> ...
  #   ...
  #
  
  # create directory to hold sources for compilation
  RUN mkdir -p src/SERVICE_NAME_LOWERCASE
  
  # add files to the container
  # sources files to GOPATH instead of /service for compilation
  COPY LICENSE /service
  COPY README.md /service
  COPY service.go /service
  
  # ADD FURTHER SERVICE FILES
  #
  #   COPY specialLibrary/ /service/specialLibrary
  #   COPY extraFile.go /service
  #   ...
  #
  
  # add the configuration file (possibly from a storage uri)
  ARG conf=service.conf
  ADD $conf /service/service.conf
  
  # build service
  RUN go build -o service.run *.go
  
  # clean up git
  # clean up behind the service build
  # clean up golang it is not necessary anymore
  RUN apk del --purge \
          git \
      && rm -rf /var/cache/apk/* \
      && rm -rf $GOPATH \
      && rm -rf /usr/local/go
  
  CMD ["./service.run"]
  

  Warning

  If you require a more complex namespacing in your service’s code, check out the Passivetotal service’s Dockerfile

LICENSE

• The license under which the service is distributed.

README.md

• An appropriate readme file for your service (also displayed if the service’s info url is looked up)

  Example:

  # Passivetotal service for Holmes-Totem
  
  ## Description
  
  A simple service to check PassiveTotal for additional enrichment data.
  If you do not have an API key, visit http://www.passivetotal.org to get one.
  
  ## Usage
  
  Build and start the docker container using the included Dockerfile.
  
  Upon building the Dockerfile downloads a list of TLDs from iana.org.
  To update this list of TLDs, the image needs to be built again.
  
  The service accepts domain names, IP addresses and emails as request objects.
  These have to be supplied as a parameter after the request URL.
  (If the analysisURL parameter is set to /passivetotal, then a request for the
  domain www.passivetotal.org would look like this: /passivetotal/www.passivetotal.org)
  
  The service performs some checks to determine the type of the input object.
  If a passed domain name contains an invalid TLD, it is invalid and rejected.
  If a passed email address contains an invalid domain, it is rejected.
  If a passed IP is in a reserved range, it is rejected. (IETF RFC6890, RFC4291)
  
  Only if a request object is determined valid, it is sent to selected passivetotal
  api endpoints. The maximum of simultaneous requests is 9.
  If an error is encountered in any of the api queries, the request fails and returns
  an appropriate error code. Check the local logs for detailed information.
  If the query succeeds, a JSON struct containing all 9 api endpoints is returned.
  Those endpoints that were not queried are set to null.
  

acl.conf

• Currently empty

service.conf.example

• JSON file containing service settings like internal port (default must be 8080), but also service specific settings like maybe output limits or parsing limits.

  Example:

  {
      "port": 8080,
      "max-output-lines": 10000
  }
  

watchdog.scala

• Currently empty

YourServiceREST.scala

Note

For most services YourServiceREST.scala can be copy & pasted and the names adjusted.

Now that Totem knows about your service and where to find it, we need to tell it how to communicate with the service. This is done in a separate file, that defines 3 (or more) classes and one object:

case class YourServiceWork
case class YourServiceSuccess
case class YourServiceFailure
object YourServiceREST

Full working example:

package org.holmesprocessing.totem.services.yourservice

import dispatch.Defaults._
import dispatch.{url, _}
import org.json4s.JsonAST.{JString, JValue}
import org.holmesprocessing.totem.types.{TaskedWork, WorkFailure, WorkResult, WorkSuccess}
import collection.mutable


case class yourserviceWork(key: Long, filename: String, TimeoutMillis: Int, WorkType: String, Worker: String, Arguments: List[String]) extends TaskedWork {
  def doWork()(implicit myHttp: dispatch.Http): Future[WorkResult] = {

    val uri = yourserviceREST.constructURL(Worker, filename, Arguments)
    val requestResult = myHttp(url(uri) OK as.String)
      .either
      .map({
      case Right(content) =>
        yourserviceSuccess(true, JString(content), Arguments)

      case Left(StatusCode(404)) =>
        yourserviceFailure(false, JString("Not found (File already deleted?)"), Arguments)

      case Left(StatusCode(500)) =>
        yourserviceFailure(false, JString("Objdump service failed, check local logs"), Arguments) //would be ideal to print response body here

      case Left(StatusCode(code)) =>
        yourserviceFailure(false, JString("Some other code: " + code.toString), Arguments)

      case Left(something) =>
        yourserviceFailure(false, JString("wildcard failure: " + something.toString), Arguments)
    })
    requestResult
  }
}


case class yourserviceSuccess(status: Boolean, data: JValue, Arguments: List[String], routingKey: String = "yourservice.result.static.totem", WorkType: String = "YOURSERVICE") extends WorkSuccess
case class yourserviceFailure(status: Boolean, data: JValue, Arguments: List[String], routingKey: String = "", WorkType: String = "YOURSERVICE") extends WorkFailure


object yourserviceREST {
  def constructURL(root: String, filename: String, arguments: List[String]): String = {
    arguments.foldLeft(new mutable.StringBuilder(root+filename))({
      (acc, e) => acc.append(e)}).toString()
  }
}

Explanation

The YourServiceWork class initiates the request with your service, creating the final uri via the YourServiceREST object.

The request result is gathered and depending on what the returned HTTP status code was, a specific class (YourServiceSuccess or YourServiceFailure) is instantiated with the result as a parameter and returned.

Warning

The 2 generic cases at the end of the map should be there in any case to avoid exceptions.

The YourServiceSuccess and YourServiceFailure classes should be self-explanatory. They extend the default interfaces for success and failure and are very convenient for mapping cases as done in driver.scala, for example.

YourServiceREST object should be self-explanatory as well, it defines how the request address for your service gets constructed from the supplied parameters.

Service-logic file

This is the file that makes the Service act like a web server. The service can be accessible from 2 endpoints.

Endpoint	Operation
/	Provide information about the service
/analyze?obj=?	Perform tasking and return results

Endpoint - ‘/’

You can use this page to view the information about the Service. Basically this page should state every aspect of Service the you are creating.

The INFO page should contain

1. Author name.
2. Service name and version. ( or any metadata about the service. )
3. Brief Description about the Service.
4. Licence
5. General information about how to use the Service and expected JSON output.

Info-output Generation in Go

func info_output(f_response http.ResponseWriter, r *http.Request, ps httprouter.Params) {
    fmt.Fprintf(f_response, `<p>%s - %s</p>
        <hr>
        <p>%s</p>
        <hr>
        <p>%s</p>
        `,
        metadata.Name,
        metadata.Version,
        metadata.Description,
        metadata.License)
}

Info-Output Generation in Python

class InfoHandler(tornado.web.RequestHandler):
        # Emits a string which describes the purpose of the analytics
        def get(self):
            info = """
  <p>{name:s} - {version:s}</p>
  <hr>
  <p>{description:s}</p>
  <hr>
  <p>{license:s}</p>
  <hr>
  <p>{copyright:s}</p>
            """.strip().format(
                name        = name,
                version     = version,
                description = description,
                license     = license,
                copyright   = copyright
            )
            self.write(info)
    return InfoHandler

Endpoint ‘/analyze?obj=’

In this Endpoint you write the logic for interacting with analyer library and producing the JSON output and appropriate error codes.

Reading configuration file

Before you start about writing the service logic, you first need to parse settings from service.conf ( renamed from service.conf.example ). Since the format Service’s configuration file JSON, you can use any JSON parsing library.

Reading configuration in Golang

With the json package it’s a snap to read JSON data into your Go programs. The json package provides Decoder and Encoder types to support the common operation of reading and writing streams of JSON data. We read the configuration file and then we fit the output to Config struct

package main

import (
    "encoding/json"
    "flag"
    "os"
)

// ....

var (
    config *Config
    configPath string
)

// ....

type Config struct {
HTTPBinding string MaxNumberOfObjects int
}

// ....

flag.StringVar(&configPath, "config", "", "Path to the configuration file") flag.Parse()

config := &Config{}

cfile, _ := os.Open(configPath) dec := json.NewDecoder(cfile) // reading from json data

if err := dec.Decode(&config); err != nil {
// handle error

}

Reading configuration for Python

Try opening the path, reading it all in and parsing it as a JSON. If an error occurs, throw a tornado.web.HTTPError (well define behaviour by tornado for these) If parsing succeeds, update provided config dictionary.

# reading configuration file
  configPath = "./service.conf"
  config = json.loads(open(configPath).read())

HTTP Error Codes

Holmes totem service is RESTful service which communicates with HTTP protocol. The first line of the HTTP response is called the status line and includes a numeric status code (such as “404”) and a textual reason phrase (such as “Not Found”). Also when something went wrong in the service, we should return a HTTP error code so that user agent can debug accordingly.

HTTP Error Code	Summary	Operation
200	OK	Services returns result
400	Bad Operation	Missing argument Obj.
401	Authorization required	currently empty
404	Not Found	Invalid URL format. A user should follow URL API scheme to submit objects.
500	Internal Server Error	Generating JSON failed

• Raising error codes in Go

func returnCode500(w http.ResponseWriter, r *http.Request) {
 http.Error(f_response, "Generating JSON failed'", 500)
}

func returnCode400(w http.ResponseWriter, r *http.Request) {
 http.Error(f_response, " Missing argument Obj", 400)
}

func returnCode404(w http.ResponseWriter, r *http.Request) {
 http.Error(f_response, " Invalid URL format. ", 404)
}

• Raising error codes in Python

# For python tornado.
raise tornado.web.HTTPError(status_code=code, log_message=custom_msg)

Files to Edit

Config Files

The following files can be found in the config/ folder within the Holmes-Totem repository.

totem.conf.example

• The entry in the totem.conf tells Totem your service exists, where to reach it, and where to store its results.

  The uri field supports multiple address entries for automatic load balancing.

  totem {
      services {
          SERVICE_NAME_LOWERCASE {
              uri = ["http://SERVICE_IP:SERVICE_PORT/analyze/?obj="]
              resultRoutingKey = "SERVICE_NAME_LOWERCASE.result.static.totem"
          }
      }
  }
  

docker-compose.yml.example

• Holmes-Totem relies on Docker to provide the services. As such all services need to provide an entry in the docker-compose file.

  services:
    SERVICE_NAME_LOWERCASE:
      build:
        context: ../src/main/scala/org/holmesprocessing/totem/services/SERVICE_NAME_LOWERCASE
        args:
          conf: ${CONFSTORAGE_SERVICE_NAME}service.conf
      ports:
        - "SERVICE_PORT:8080"
      restart: unless-stopped
  

  If the service processes files (i.e. it needs access to /tmp/ on the host), the following option needs to be added additionally to build, ports, and restart:

  volumes:
    - /tmp:/tmp:ro
  

compose_download_conf.sh

• This file runs docker-compose with certain environmental variables set, that allow fetching service configuration files from a server. Add an export statement like this:

  export CONFSTORAGE_SERVICE_NAME=${CONFSTORAGE}zipmeta/
  

  Warning

  In the above example, ${CONFSTORAGE} is the actual term and nothing needs to be replaced there.

Scala Files

The following files can be found in the src/main/scala/org/holmesprocessing/totem/ folder within the Holmes-Totem repository

driver.scala

• Import your services scala classes (see the respective section for information on these classes).

  import org.holmesprocessing.totem.services.SERVICE_NAME_LOWERCASE.{
      SERVICE_CLASS_SUCCESS,
      SERVICE_CLASS_WORK
  }
  

• Add a case to the method GeneratePartial

  def GeneratePartial(work: String): String = {
    work match {
      case "SERVICE_NAME_UPPERCASE" => Random.shuffle(services.getOrElse("SERVICE_NAME_LOWERCASE", List())).head
    }
  }
  

• Add a case to the method enumerateWork.

  Warning

  If your service does not process files but rather the input string, use uuid_filename instead of orig_filename below.

  def enumerateWork(key: Long, orig_filename: String, uuid_filename: String, workToDo: Map[String, List[String]]): List[TaskedWork] = {
    val w = workToDo.map({
      case ("SERVICE_NAME_UPPERCASE", li: List[String]) => SERVICE_CLASS_WORK(key, orig_filename, taskingConfig.default_service_timeout, "SERVICE_NAME_UPPERCASE", GeneratePartial("SERVICE_NAME_UPPERCASE"), li)
    }).collect({
      case x: TaskedWork => x
    })
    w.toList
  }
  

• Add a case to the method workRoutingKey

  def workRoutingKey(work: WorkResult): String = {
    work match {
      case x: SERVICE_CLASS_SUCCESS => conf.getString("totem.services.SERVICE_NAME_LOWERCASE.resultRoutingKey")
    }
  }
  

Example Service: Hello World

Let’s implement a service as simple as possible: Hello World.

• totem.conf

totem {
  version = "0.5.0"

  download_settings {
    connection_pooling = true
    connection_timeout = 1000
    download_directory = "/tmp/"
    thread_multiplier = 4
    request_timeout = 1000
    validate_ssl_cert = true
  }

  tasking_settings {
    default_service_timeout = 180
    prefetch = 3
    retry_attempts = 3
  }

  rabbit_settings {
    requeueKey = "requeue.static.totem"
    host {
      server = "127.0.0.1"
      port = 5672
      username = "guest"
      password = "guest"
      vhost = "/"
    }
    exchange {
      name = "totem"
      type = "topic"
      durable = true
    }
    workqueue {
      name = "totem_input"
      routing_key = "work.static.totem"
      durable = true
      exclusive = false
      autodelete = false
    }
    resultsqueue {
      name = "totem_output"
      routing_key = "*.result.static.totem"
      durable = true
      exclusive = false
      autodelete = false
    }
    misbehavequeue {
      name = "totem_misbehave"
      routing_key = "misbehave.static.totem"
      durable = true
      exclusive = false
      autodelete = false
    }
  }

  services {
    helloworld {
      uri = ["http://127.0.0.1:8888/analyze/?obj="]
      resultRoutingKey = "helloworld.result.static.totem"
    }
  }
}

• driver.scala

package org.holmesprocessing.totem.driver

import java.util.concurrent.{Executors, ExecutorService}

import akka.actor.{ActorRef, ActorSystem, Props}
import org.holmesprocessing.totem.actors._
import org.holmesprocessing.totem.services.helloworld.{helloworldSuccess, helloworldWork}

import org.holmesprocessing.totem.types._
import org.holmesprocessing.totem.util.DownloadSettings

import org.holmesprocessing.totem.util.Instrumented

import java.io.File

import com.typesafe.config.{Config, ConfigFactory}

import scala.util.Random

object driver extends App with Instrumented {
  // Define constants
  val DefaultPathConfigFile = "./config/totem.conf"

  lazy val execServ: ExecutorService = Executors.newFixedThreadPool(4000)
  val conf: Config = if (args.length > 0) {
    println("Using manual config file: " + args(0))
    ConfigFactory.parseFile(new File(args(0)))
  } else {
    println("Using default config file: " + DefaultPathConfigFile)
    ConfigFactory.parseFile(new File(DefaultPathConfigFile))
  }
  val system = ActorSystem("totem")

  println("Configuring details for Totem Tasking")
  val taskingConfig = TaskingSettings(
    conf.getInt("totem.tasking_settings.default_service_timeout"),
    conf.getInt("totem.tasking_settings.prefetch"),
    conf.getInt("totem.tasking_settings.retry_attempts")
  )

  println("Configuring details for downloading objects")
  val downloadConfig = DownloadSettings(
    conf.getBoolean("totem.download_settings.connection_pooling"),
    conf.getInt("totem.download_settings.connection_timeout"),
    conf.getString("totem.download_settings.download_directory"),
    conf.getInt("totem.download_settings.thread_multiplier"),
    conf.getInt("totem.download_settings.request_timeout"),
    conf.getBoolean("totem.download_settings.validate_ssl_cert")
  )

  println("Configuring details for Rabbit queues")
  val hostConfig = HostSettings(
    conf.getString("totem.rabbit_settings.host.server"),
    conf.getInt("totem.rabbit_settings.host.port"),
    conf.getString("totem.rabbit_settings.host.username"),
    conf.getString("totem.rabbit_settings.host.password"),
    conf.getString("totem.rabbit_settings.host.vhost")
  )

  val exchangeConfig = ExchangeSettings(
    conf.getString("totem.rabbit_settings.exchange.name"),
    conf.getString("totem.rabbit_settings.exchange.type"),
    conf.getBoolean("totem.rabbit_settings.exchange.durable")
  )

  val workqueueKeys = List[String](
    conf.getString("totem.rabbit_settings.workqueue.routing_key"),
    conf.getString("totem.rabbit_settings.requeueKey")
  )
  val workqueueConfig = QueueSettings(
    conf.getString("totem.rabbit_settings.workqueue.name"),
    workqueueKeys,
    conf.getBoolean("totem.rabbit_settings.workqueue.durable"),
    conf.getBoolean("totem.rabbit_settings.workqueue.exclusive"),
    conf.getBoolean("totem.rabbit_settings.workqueue.autodelete")
  )

  val resultQueueConfig = QueueSettings(
    conf.getString("totem.rabbit_settings.resultsqueue.name"),
    List[String](conf.getString("totem.rabbit_settings.resultsqueue.routing_key")),
    conf.getBoolean("totem.rabbit_settings.resultsqueue.durable"),
    conf.getBoolean("totem.rabbit_settings.resultsqueue.exclusive"),
    conf.getBoolean("totem.rabbit_settings.resultsqueue.autodelete")
  )

  val misbehaveQueueConfig = QueueSettings(
    conf.getString("totem.rabbit_settings.misbehavequeue.name"),
    List[String](conf.getString("totem.rabbit_settings.misbehavequeue.routing_key")),
    conf.getBoolean("totem.rabbit_settings.misbehavequeue.durable"),
    conf.getBoolean("totem.rabbit_settings.misbehavequeue.exclusive"),
    conf.getBoolean("totem.rabbit_settings.misbehavequeue.autodelete")
  )

  println("Configuring setting for Services")
  class TotemicEncoding(conf: Config) extends ConfigTotemEncoding(conf) { //this is a class, but we can probably make it an object. No big deal, but it helps on mem. pressure.
    def GeneratePartial(work: String): String = {
      work match {
        case "HELLOWORLD" => Random.shuffle(services.getOrElse("helloworld", List())).head
        case _ => ""
      }
    }

    def enumerateWork(key: Long, orig_filename: String, uuid_filename: String, workToDo: Map[String, List[String]]): List[TaskedWork] = {
      val w = workToDo.map({

        case ("HELLOWORLD", li: List[String]) =>
          pdfparseWork(key, uuid_filename, taskingConfig.default_service_timeout, "HELLOWORLD", GeneratePartial("HELLOWORLD"), li)
        case (s: String, li: List[String]) =>
          UnsupportedWork(key, orig_filename, 1, s, GeneratePartial(s), li)
        case _ => Unit //need to set this to a non Unit type.
      }).collect({
        case x: TaskedWork => x
      })
      w.toList
    }

    def workRoutingKey(work: WorkResult): String = {
      work match {
        case x: helloworldSuccess => conf.getString("totem.services.helloworld.resultRoutingKey")
        case _ => ""
      }
    }
  }

  println("Completing configuration")
  val encoding = new TotemicEncoding(conf)

  println("Creating Totem Actors")
  val myGetter: ActorRef = system.actorOf(RabbitConsumerActor.props[ZooWork](hostConfig, exchangeConfig, workqueueConfig, encoding, Parsers.parseJ, downloadConfig, taskingConfig).withDispatcher("akka.actor.my-pinned-dispatcher"), "consumer")
  val mySender: ActorRef = system.actorOf(Props(classOf[RabbitProducerActor], hostConfig, exchangeConfig, resultQueueConfig, misbehaveQueueConfig, encoding, conf.getString("totem.rabbit_settings.requeueKey"), taskingConfig), "producer")

  println("Totem version " + conf.getString("totem.version") + " is running and ready to receive tasks")

}

• helloworldREST.scala

In Golang

This tutorial will show how to This tutorial will show how to utilize the Go Programming Language to write the actual service.

Install Dependencies

First of all, Go is required:

# for ubuntu
apt-get install golang
# for macOS
brew install golang

Whilst Go natively has a very good webserver, it lacks a good router. More specific, the router lacks the ability to parse request URIs directly into variables. In our example we will use httprouter:

go get github.com/julienschmidt/httprouter

Further, we need a way to parse a config file:

go get gopkg.in/ini.v1

If you have any further dependencies, they need to be installed in this step as well. (For example any additional frameworks)

Dockerfile

# choose the operating system image to base of, refer to docker.com for available images
FROM golang:aplpine

# create a folder to contain your service's files
RUN mkdir -p /service
WORKDIR /service

# add Go dependencies
RUN apk add --no-cache \
                    git \
                && go get github.com/julienschmidt/httprouter \
                && rm -rf /var/cache/apk/*

# add dependencies for helloworld


# add all files relevant for running your service to your container
COPY helloworld.py /service
COPY README.md /service


# build the service
RUN go build helloworld.go

# add the configuration file (possibly from a storage URI)
ARG conf=service.conf
ADD $conf /service/service.conf

CMD ["./helloworld", "--config=service.conf"]

It is important to think about the ordering the commands have in the Dockerfile, as that can speed up or slow down the container build process heavily. Stuff that does not need to be done on every build should go to the front of the Dockerfile, stuff that changes should go towards the end of the file.

(Docker caches previous build steps and if nothing changes, those build steps will be reused on the next build, speeding it up by a lot, especially when installing python like in this Dockerfile)

helloworld.go

package main

// These were all the imports required for this tutorial. If there are any
// further dependencies those go inside this import section, too.

import (
  "encoding/json"
  "flag"
  "github.com/julienschmidt/httprouter"
  "os"
  "net/http"
  "fmt"
)

var (
  config   *Config
  helloworld string
  metadata Metadata = Metadata{
    Name:        "helloworld",
    Version:     "0.1",
    Description: "./README.md",
    Copyright:   "Copyright 2017 Holmes Group LLC",
    License:     "./LICENSE",
  }
)

type Config struct {
  HTTPBinding        string
  MaxNumberOfObjects int
}

type Metadata struct {
  Name        string
  Version     string
  Description string
  Copyright   string
  License     string
}

type Result struct {
  key string
}
func main() {

  var configPath string

  flag.StringVar(&configPath, "config", "", "Path to the configuration file")
  flag.Parse()

  // reading configuration file.
  config := &Config{}
  cfile, _ := os.Open(configPath)
  json.NewDecoder(cfile).Decode(&config)


  router := httprouter.New()
  router.GET("/analyze/", handler_analyze)
  router.GET("/", handler_info)
  http.ListenAndServe(":8080", router)
}

func handler_info(f_response http.ResponseWriter, r *http.Request, ps httprouter.Params) {
  // info-output
  fmt.Fprintf(f_response, `<p>%s - %s</p>
    <hr>
    <p>%s</p>
    <hr>
    <p>%s</p>
    `,
    metadata.Name,
    metadata.Version,
    metadata.Description,
    metadata.License)

}

func handler_analyze(f_response http.ResponseWriter, request *http.Request, params httprouter.Params) {
  obj := request.URL.Query().Get("obj")
  if obj == "" {
    http.Error(f_response, "Missing argument 'obj'", 400)
    return
  }
  sample_path := "/tmp/" + obj
  if _, err := os.Stat(sample_path); os.IsNotExist(err) {
    http.NotFound(f_response, request)
    return
  }

  //-----------------------------------------------------------------//
  //                                                                 //
  //                    Write your service logic.                    //
  //                                                                 //
  //-----------------------------------------------------------------//
  result := &Result{
    key : "value",
  }

  f_response.Header().Set("Content-Type", "text/json; charset=utf-8")
  json2http := json.NewEncoder(f_response)

  if err := json2http.Encode(result); err != nil {
    http.Error(f_response, "Generating JSON failed", 500)
    return
  }
}

In Python

Install Dependencies

First of all, Python is required, as well as pip:

# for ubuntu
apt-get install python python-pip

As already explained in the section Service logic the Service needs to act as a webserver, accepting requests from Totem. For details please read up in the corresponding section.

One easy way of providing such a webserver is to use Tornado:

pip install tornado

That’s all dependencies we’ll need for a simple service that does basically nothing. If you have any further dependencies, they need to be installed in this step as well. (Like additional Python frameworks)

Dockerfile

# choose the operating system image to base of, refer to docker.com for available images
FROM golang:apline

 # create a folder to contain your service's files
RUN mkdir -p /service
WORKDIR /service

# add Tornado or Flask or any WSGI compliant wrapper
RUN pip install tornado

# add dependencies for helloworld
RUN pip3 install <....>

# add all files relevant for running your service to your container
COPY helloworld.py /service
COPY LICENSE /service

# add the configuration file (possibly from a storage URI)
ARG conf=service.conf
ADD $conf /service/service.conf

CMD ["python3", "helloworld.py"]

helloworld.py

import tornado
import tornado.web
import tornado.httpserver
import tornado.ioloop
import json

import os
from os import path


# reading configuration file
configPath = "./service.conf"
config = json.loads(open(configPath).read())

# service logic
class Service (tornado.web.RequestHandler):
    def get(self, filename):
      # Getting object submitteed through URL
      object = self.get_argument('obj', strip=False)
        data = {
            "message": "Hello World!"
        }
        self.write(data)

      # return appropriate error codes
      raise tornado.web.HTTPError(status_code=code, log_message=custom_msg)

# Generating info-output
class Info(tornado.web.RequestHandler):
    def get(self):
        description = """
            <p>Copyright 2017 Holmes Processing
            <p>Description: This is the HelloWorld Service for Totem.
        """
        self.write(description)


class Application(tornado.web.Application):
    def __init__(self):
        handlers = [
            (r'/', Info),
            (r'/analyze/', Service),
        ]
        settings = dict(
            template_path=path.join(path.dirname(__file__), 'templates'),
            static_path=path.join(path.dirname(__file__), 'static'),
        )
        tornado.web.Application.__init__(self, handlers, **settings)
        self.engine = None


def main():
    server = tornado.httpserver.HTTPServer(Application())
    server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()


if __name__ == '__main__':
    main()

The port in the main function needs to be adjusted as necessary and all the services work should go either into the Service class or should be called from there.

Warning

Please note, that while the Info class writes a string, the Service class must write a dictionary. (Totem communicates via JSON!)

Holmes Totem Dynamic

What is Holmes Totem Dynamic?

Architecture Overview

Holmes Totem Dynamic

Dependencies

Configuration

Executing Tasks

Services

Services

Structure

Configuration

Configuration file

TODO

Port Selection

TODO

API Endpoints

Endpoint - /

Endpoint - /analyze?obj=

Endpoint - /feed?obj=

Endpoint - /check?taskid=

Endpoint - /result?taskid=

Endpoint - /status

Output

HTTP Error Codes

Results

Holmes Storage

Overview

Holmes-Storage is responsible for managing the interaction of Holmes Processing with the database backends. At its core, Holmes-Storage organizes the information contained in Holmes Processing and provides a RESTful and AMQP interface for accessing the data. Additionally, Holmes-Storage provides an abstraction layer between the specific database types. This allows a Holmes Processing system to change database types and combine different databases together for optimization.

When running, Holmes-Storage will:

• Automatically fetch the analysis results from Holmes-Totem and Holmes-Totem-Dynamic over AMQP for storage
• Support the submission of objects via a RESTful API
• Support the retrieval of results, raw objects, object meta-data, and object submission information via a RESTful API

We have designed Holmes-Storage to operate as a reference implementation. In doing so, we have optimized the system to seamlessly plug into other parts of Holmes Processing and optimized the storage of information for generic machine learning algorithms and queries through a web frontend. Furthermore, we have separated the storage of binary blobs and textual data in order to better handle how data is stored and compressed. As such, Holmes-Storage will store file based objects (i.e. ELF, PE32, PDF, etc) in an S3 compatible backend and the meta information of the objects and results from the analysis in Cassandra. With respect to non-file type objects, these are purely stored in Cassandra. In our internal production systems, this scheme has supported 10s of million of objects along with the results from associated Totem and Totem-Dynamic Services with minimal effort. However, as with any enterprise system, customization will be required to improve the performance for custom use cases. Anyway, we hope this serves you well or at least helps guide you in developing custom Holmes-Storage Planners.

Installation

Dependencies

Supported Databases

Holmes-Storage supports multiple databases and splits them into two categories: Object Stores and Document Stores. Object Stores are responsible for storing the file-based malicious objects collected by the analyst: binary files such as PE32 and ELF, PDFs, HTML code, Zips files etc. Document Stores contain the output of Holmes-Totem and Holmes-Totem-Dynamic Services. This was done to enable users to more easily select their preferred solutions while also allowing the mixing of databases for optimization purposes. In production environments, we strongly recommend using an S3 compatible Object Store, such as RIAK-CS, and a clustered deployment of Cassandra for the Document Stores.

Object Stores

We support two primary object storage databases.

• S3 compatible
• (Soon) MongoDB Gridfs

There are several tools you can use for implementing Object Stores. Depending on the intended scale of your work with Holmes-Storage, we would recommend:

Framework	Workstation	Mid-scale	Large-scale
AWS	[]	[]	[x]
RIAK-CS	[]	[]	[x]
LeoFS	[]	[]	[x]
Pithos	[]	[x]	[]
Minio	[]	[x]	[]
Fake-S3	[x]	[]	[]

If you want to run Holmes-Storage on your local machine for testing or development purposes, we recommend you use lightweight servers compatible with the Amazon S3 API. This will make the installation and usage of Holmes-Storage faster and more efficient. There are several great options to fulfill this role: Fake-S3, Minio, Pithos etc. The above-mentioned frameworks are only suggestions, any S3 compatible storage will do. Check out their documentation to find out which option is more suitable for the work you intend to do.

Document Stores

We support two primary object storage databases.

• Cassandra
• MongoDB

We recommend a Cassandra cluster for large deployments.

Configuration

RiakCS

It is recommended to use RIAK-CS only for large scale or industry deployments. Follow this tutorial for installation of RiakCS.

After successful installation, the user’s access key and secret key are returned in the key_id and key_secret fields respectively. Use these keys to update key and secret your config file ( storage.conf.example )

Holmes-Storage uses Amazon S3 signature version 4 for authentication. To enable authV4 on riak-cs, add {auth_v4_enabled, true} to advanced.config file ( should be in /riak-cs/etc/)

Fake-S3

We recommend Fake-S3 as a simple starting point for exploring the system functionality. Most of the current developers of Holmes-Storage are using Fake-S3 for quick testing purposes. Minio is also encouraged if you want to engage more with development and do more testing.

Check out the documentation of Fake-S3 on how to install and run it. Afterwards, go to /config/storage.conf of Holmes-Storage and set the IP and Port your ObjectStorage server is running on. You can decide whether you want your Holmes client to send HTTP or HTTPS requests to the server through the Secure parameter.

Cassandra

Holmes-Storage supports single node or cluster installation of Cassandra version 3.10 and higher. The version requirement is because of the significant improvement in system performance when leveraging the newly introduced SASIIndex for secondary indexing and Materialized Views. We highly recommend deploying Cassandra as a cluster with a minimum of three Cassandra nodes in production environments.

New Cassandra clusters will need to be configured before Cassandra is started for the first time. We have highlighted a few of the configuration options that are critical or will improve performance. For additional options, please see the Cassandra installation guide.

To edit these values, please open the Cassandra configuration file in your favorite editor. The Cassandra configuration file is typically located in /etc/cassandra/cassandra.yaml.

The Cassandra “cluster_name” must be set and the same on all nodes. The name you select does not much matter but again it should be identical on all nodes. cluster_name: 'Holmes Processing'

Cassandra 3.x has an improved token allocation algorithm. As such, 256 is not necessary and should be decreased to 64 or 128 tokens. num_tokens: 128

You should populate the “seeds” value with the IP addresses for at least two additional Cassandra nodes. seeds: <ip node1>,<ip node2>

The “listen_address” should be set to the external IP address for the current Cassandra node. listen_address: <external ip address>

Installation

Copy the default configuration file located in config/storage.conf.example and change it according to your needs.

$ cp storage.conf.example storage.conf

Update the storage.conf file in config folder and adjust the ports and IPs to point to your cluster nodes. To build the Holmes-Storage, just run

$ go build

Setup the database by calling

$ ./Holmes-Storage --config <path_to_config> --setup

This will create the configured keyspace if it does not exist yet. For Cassandra, the default keyspace will use the following replication options:

{'class': 'NetworkTopologyStrategy', 'dc': '2'}

If you want to change this, you can do so after the setup by connecting with cqlsh and changing it manually. For more information about that we refer to the official documentation of Cassandra (Cassandra Replication Altering Keyspace) You can also create the keyspace with different replication options before executing the setup and the setup won’t overwrite that. The setup will also create the necessary tables and indices.

Setup the object storer by calling:

$ ./Holmes-Storage --config <path_to_config> --objSetup

Execute storage by calling:

$ ./Holmes-Storage --config <path_to_config>

Best Practices

On a new cluster, Holmes-Storage will setup the database in an optimal way for the average user. However, we recommend Cassandra users to please read the Cassandra’s Operations website for more information Cassandra best practices. We want to expand on three particular practices that in our experience have been proven to be very meaningful in keeping the database healthy.

Nodetool

Based on the CAP theorem, Cassandra can be classified as an AP database. The cost for strong consistency is higher latency, so the database has its own mechanisms to ensure eventual consistency in the system. However, human intervention is often necessary. It is critical that the Cassandra cluster is regularly repaired using the nodetool repair and nodetool compact command. The following documentations 1, 2 give an overview of the nodetool functionality. We suggest exploring the Cassandra-Reaper tool as a potential way to automate the repair process.

Nodetool Repair

The purpose of this command is to enforce consistency in the tables across the cluster. We recommend that this is executed on every node, one at a time, at least once a week.

Nodetool Compact

This is another important maintenance command. Cassandra has its own methodology for storing and deleting data, which requires compaction to take place in regular intervals in order to save space and maintain efficiency. For more details follow the links above to learn more about Cassandra.

Indexing

Holmes-Storage uses SASIIndex for indexing the Cassandra database. This indexing allows for querying of large datasets with minimal overhead. When leveraging Cassandra, most of the Holmes Processing tools will automatically use SASI indexes for speed improvements. Power users wishing to learn more about how to utilize these indexes should please visit the excellent blog post by Doan DyuHai.

However, while SASI is powerful, it is not meant to be a replacement for advanced search and aggregation engines like Solr, Elasticsearch, or leveraging Spark. Additionally, Holmes Storage by default does not implement SASI on the table for storing the results of TOTEM Services (results.results). This is because indexing this field can increase storage costs by approximately 40% on standard deployments. If you still wish to leverage SASI on results.results, the following Cassandra command will provide a sane level of indexing.

SASI indexing of TOTEM Service results. WARNING: this will greatly increase storage requirement:

CREATE CUSTOM INDEX results_results_idx ON holmes_testing.results (results)
USING 'org.apache.cassandra.index.sasi.SASIIndex'
WITH OPTIONS = {
        'analyzed' : 'true',
        'analyzer_class' : 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
        'tokenization_enable_stemming' : 'false',
        'tokenization_locale' : 'en',
        'tokenization_normalize_lowercase' : 'true',
        'tokenization_skip_stop_words' : 'true',
        'max_compaction_flush_memory_in_mb': '512'
};

Holmes Gateway

Overview

Holmes-Gateway orchestrates the submission of objects and tasks to HolmesProcessing. Foremost, this greatly simplifies the tasking and enables the ability to automatically route tasks to Holmes-Totem and Holmes-Totem-Dynamic at a Service level. In addition, Holmes-Gateway provides validation and authentication. Finally, Holmes-Gateway provides the technical foundation for collaboration between organizations.

Holmes-Gateway is meant to prevent a user from directly connecting to Holmes-Storage or RabbitMQ. Instead, tasking-requests and object upload pass through Holmes-Gateway, which performs validity checking, enforces ACL, and forwards the requests.

If the user wants to upload samples, he sends the request to /samples/ along with his credentials, and the request will be forwarded to storage.

If the user wants to task the system, he sends the request to /task/ along with his credentials. Holmes-Gateway will parse the submitted tasks and find partnering organizations (or the user’s own organization) which have access to the sources that are specified by the tasks. It will then forward the tasking-requests to the corresponding Gateways and these will check the task and forward it to their AMQP queues. The Gateway can be configured to push different services into different queues.

This way Gateway can push long-lasting tasks (usually those that perform dynamic analysis) into different queues than quick tasks and thus distribute those tasks among different instances of Holmes-Totem and Holmes-Totem-Dynamic.

Highlights

• Collaborative Tasking: Holmes-Gateway allows organizations to enable other organizations to execute analysis-tasks on their samples without actually giving them access to these samples.
• ACL enforcement: Users who want to submit tasks or new objects need to authenticate before they can do so.
• The Central point for tasking and sample upload: Without Holmes-Gateway, a user who wants to task the system needs access to RabbitMQ, while a user who wants to upload samples needs access to Holmes-Storage.

Installation

Dependencies

• Golang

Configuration

Before going into detail about configuration options for Holmes-Gateway, you need a good understanding how it works. Take a look at the following picture:

As you can see, users interact with master gateways. When a user submits a task to a master gateway, it creates a ticket and sends it to the appropriate slave gateway. Which in turn pushes the task on the appropriate AMQP queue. The reason for this architecture is that it enables sharing of sources and services with other organizations, whilst enabling access control on two layers.

Each organization:

• has full control which organizations can submit tickets, based on their slave gateway settings
• has full control which users can submit tasks, based on their master gateway settings

Note

The following configuration options for Holmes-Gateway might have misleading names, if you are not aware of their functionality. Be sure to read the descriptions carefully and look at the examples.

Holmes-Gateway is configured by the files found in config/. The gateway-master.conf configures master instances of Gateway, whilst the gateway.conf configures the slave instances.

Configuration options for a Slave Gateway:

Config-Key	Value-Type	Description
HTTP	String	HTTP address to listen to.
SourcesKeysPath	String	The path to search for source public keys. Keys in here must be in the PEM format and have .pub as their file extension.
TicketKeysPath	String	The Slave Gateway holds the public key associated with the Master Gateways private key. It is used to verify that a ticket was really sent by the Master Gateway.
SampleStorageURI	String	Address of a Holmes-Storage instance, e.g.: http://127.0.0.1:8016/samples/.
AllowedTasks	Object{String->List[String]}	JSON object mapping organization identifiers to lists of services that the respective organization may execute. Wildcard * means all services are allowed.
RabbitURI	String	Address of the RabbitMQ (or other AMQP broker).
RabbitUser	String	Username to use for AMQP
RabbitPassword	String	Password to use for AMQP
RabbitDefault	Object	JSON object with the following entries: Queue String Name of the AMQP queue to use Exchange String Name of the AMQP exchange to use RoutingKey String Name of the AMQP routing key to use
Rabbit	Object{String->Object}	Routing settings per service. Key is the service name (uppercase), value a JSON object with the same key-value pairs as the RabbitDefault setting

Example:

{
  "HTTP":               ":8080",
  "SourcesKeysPath":    "config/keys/sources/",
  "TicketKeysPath":     "config/keys/tickets/",
  "SampleStorageURI":   "http://localhost:8016/samples/",
  "AllowedTasks":       {"org1": ["*"], "org2": ["PEINFO"]},
  "RabbitURI":          "localhost:5672/",
  "RabbitUser":         "guest",
  "RabbitPassword":     "guest",
  "RabbitDefault":      {"Queue": "totem_input", "Exchange": "totem", "RoutingKey": "work.static.totem"},
  "Rabbit":             {"CUCKOO":     {"Queue": "totem_dynamic_input", "Exchange": "totem_dynamic", "RoutingKey": "work.static.totem"},
                           "DRAKVUF":    {"Queue": "totem_dynamic_input", "Exchange": "totem_dynamic", "RoutingKey": "work.static.totem"},
                           "VIRUSTOTAL": {"Queue": "totem_dynamic_input", "Exchange": "totem_dynamic", "RoutingKey": "work.static.totem"}}
}

Configuration options for a Master Gateway:

Config-Key	Value-Type	Description
HTTP	String	HTTP address to listen to.
SourcesKeysPath	String	The path to search for source public keys. Keys in here must be in the PEM format and have .pub as their file extension.
TicketSignKeyPath	String	Path to the private key of the gateway used for signing tickets.
StorageURI	String	Address of a Holmes-Storage instance, e.g.: http://127.0.0.1:8016/samples/.
RabbitURI	String	Address of the RabbitMQ server (or other AMQP broker).
RabbitUser	String	Username to use for AMQP
RabbitPassword	String	Password to use for AMQP
RabbitDefault	Object	JSON object with the following entries: Queue String Name of the AMQP queue to use Exchange String Name of the AMQP exchange to use RoutingKey String Name of the AMQP routing key to use
Rabbit	Object{String->Object}	Routing settings per service. Key is the service name (uppercase), value a JSON object with the same key-value pairs as the RabbitDefault setting
AllowedUsers	List[Object]	JSON list of JSON objects describing users and their logins. Each object is of the form: Name String Username Pw String Password-Hash, Hashalgorithm: Blowfish. ID Integer User-ID, must be unique.
OwnOrganization	String	The identifier of your own organization.
Organizations	List[Object]	JSON list of JSON objects describing an organization. Each object is of the form: Name String Organizations name Uri String TODO Sources List[String] Names of sources that this organization may access
AutoTasks	Object{String->Object{String->List[String]}}	Maps automatic execution instructions to filetypes. It is basically Object[Filetype->Object[Servicename->Servicearguments]].

Example:

{
  "HTTP":              ":8090",
  "SourcesKeysPath":   "config/keys/sources/",
  "TicketSignKeyPath": "config/keys/tickets/org1.priv",
  "Organizations":     [{"Name": "Org1", "Uri": "http://localhost:8080/task/", "Sources": ["src1","src2"]},
                        {"Name": "Org2", "Uri": "http://localhost:8081/task/", "Sources": ["src3"]}],
  "OwnOrganization":   "Org1",
  "AllowedUsers":      [{"name": "test", "pw":"$2a$06$fLcXyZd6xs60iPj8sBXf8exGfcIMnxZWHH5Eyf1.fwkSnuNq0h6Aa", "id":0},
                        {"name": "test2", "pw":"$2a$06$fLcXyZd6xs60iPj8sBXf8exGfcIMnxZWHH5Eyf1.fwkSnuNq0h6Aa", "id":1}],
  "StorageURI":        "http://localhost:8016/samples/",
  "AutoTasks":         {"PE32":{"PEINFO":[],"PEID":[]}, "":{"YARA":[]}},
  "CertificateKeyPath":"cert-key.pem",
  "CertificatePath":   "cert.pem"
  "MaxUploadSize":     200
}

In addition to the regular config files, Holmes-Gateway requires RSA keys to provide its services. The structure explained above requires these for security reasons.

Two rules apply:

• Each sample source is assigned a key pair
• Each master gateway is assigned a key pair

The procedure when a master gateway receives a tasking request is as follows:

1. It checks if the user is allowed
2. It checks if it possesses the public key associated with the requested source
3. It creates a ticket
4. It encrypts the task in the ticket using the public key of the source
5. It signs the ticket using its private key
6. It sends the ticket to the target slave gateway

The receiving slave gateway follows these steps:

1. It checks if it possesses the public key of the master gateway
2. It verifies the signature of the request using that public key
3. It checks if it possesses the private key of the source
4. It decrypts the task
5. It relays the task to the transport (AMQP)

As a result, these rules apply:

• A master gateway will reject creation of tickets if the user is not allowed or if it misses the appropriate public key (key of the source)
• A slave gateway will reject tickets that it cannot verify or decrypt
• Keys must be named accurately - source keys must be named exactly as the corresponding source - organization keys must be named exactly like the corresponding organization - all keys must have the appropriate .priv or .pub suffix

In summary, a master gateway requires:

• A keypair for signing requests
• The public keys of all sources it should have access to

A slave gateway requires:

• The public keys of all master gateways that are allowed to send tickets
• The private keys of all sources it should have access to

All the required keys can be created using e.g. the OpenSSL libraries.

Note

Only unencrypted RSA keys in the PEM format are supported.

For ease of use, we distribute a small convenience program that creates 2048-bit keys for you. After git cloning https://github.com/HolmesProcessing/Holmes-Gateway.git open the repository folder in a terminal and do:

cd config/keys
go build

Now you can use ./keys <path> to create said keys. For example, to create the master gateway key use:

./keys tickets/org_holmesprocessing

This will create a keypair (org_holmesprocessing.priv and org_holmesprocessing.pub) in config/keys/tickets.

Similarly, if you execute:

./keys sources/source_holmesprocessing

You will get the files source_holmesprocessing.priv and source_holmesprocessing.pub saved to config/keys/sources.

Note

Key changes are recognized at runtime, allowing for removal or addition of keys without system downtime.

Installation

Holmes-Gateway is the endpoint that users interact with when creating tasks for Holmes-Totem or Holmes-Totem-Dynamic.

mkdir -p /data/holmes-gateway
cd /data/holmes-gateway
git clone https://github.com/HolmesProcessing/Holmes-Gateway.git .
go build .

The framework requires one Holmes-Gateway running in Master mode and the Master Gateway needs an SSL certificate to function. If you don’t have an SSL certificate at hand, you can simply create a self-signed one by using the provided shell script:

./mkcert.sh

Routing Services through Different queues

By modifying gateway’s config-file, it is possible to push different services into different AMQP-queues / exchanges. This way, it is possible to route some services to Holmes-Totem-Dynamic. The keys AMQPDefault and AMQPSplitting are used for this purpose. AMQPSplitting consists of a dictionary mapping Service names to Queues, Exchanges, and RoutingKeys. If the service is not found in this dictionary, the values from AMQPDefault are taken. e.g.

"AMQPDefault":   {"Queue": "totem_input", "Exchange": "totem", "RoutingKey": "work.static.totem"},
"AMQPSplitting": {"CUCKOO":     {"Queue": "totem_dynamic_input", "Exchange": "totem_dynamic", "RoutingKey": "work.static.totem"},
                  "DRAKVUF":    {"Queue": "totem_dynamic_input", "Exchange": "totem_dynamic", "RoutingKey": "work.static.totem"},
                  "VIRUSTOTAL": {"Queue": "totem_dynamic_input", "Exchange": "totem_dynamic", "RoutingKey": "work.static.totem"}}

This configuration will route services CUCKOO and DRAKVUF to the queue “totem_dynamic_input”, while every other service is routed to “totem_input”.

Uploading Samples

In order to upload samples to storage, the user sends an HTTPS-encrypted POST request to /samples/ of the Holmes-Gateway. Gateway will forward every request for this URI directly to storage. If storage signals a success, Gateway will immediately issue a tasking-request for the new samples, if the configuration-option AutoTasks is not empty.

You can use Holmes-Toolbox for this purpose. Just replace the storage-URI with the URI of Gateway. Also, make sure, your SSL-Certificate is accepted. You can do so either by adding it to your system’s certificate store or by using the command-line option –insecure. The following command uploads all files from the directory $dir to the Gateway instance residing at 127.0.0.1:8090 using 5 threads.

./Holmes-Toolbox --gateway https://127.0.0.1:8090 --user test --pw test --dir $dir --src foo --comment something --workers 5 --insecure

Requesting Tasks

In order to request a task, a user sends an HTTPS-request (GET/POST) to Gateway containing the following form fields:

username: The user’s login name password: The user’s password task: The task which should be executed in JSON-form, as described below. A task consists of the following attributes:

primaryURI: The user enters only the sha256-sum here, as Gateway will prepend this with the URI to its version of Holmes-Storage secondaryURI (optional): A secondary URI to the sample, if the primaryURI isn’t found filename: The name of the sample tasks: All the tasks that should be executed, as a dictionary tags: A list of tags associated with this task attempts: The number of attempts. Should be zero source: The source this sample belongs to. The executing organization is chosen mainly based on this value download: A boolean specifying, whether totem has to download the file given as PrimaryURI For this purpose, any web browser or command line utility can be used. The following demonstrates an exemplary evocation using CURL. The –insecure parameter is used, to disable certificate checking.

curl --data 'username=test&password=test&task=[{"primaryURI":"3a12f43eeb0c45d241a8f447d4661d9746d6ea35990953334f5ec675f60e36c5","secondaryURI":"","filename":"myfile","tasks" :{"PEID":[],"YARA":[]},"tags":["test1"],"attempts":0,"source":"src1","download":true}]' --insecure https://localhost:8090/task/

Alternatively, it is possible to use Holmes-Toolbox for this task, as well. First a file must be prepared containing a line with the sha256-sum, the filename, and the source (separated by single spaces) for each sample.

./Holmes-Toolbox --gateway https://127.0.0.1:8090 --tasking --file sampleFile --user test --pw test --tasks '{"PEID":[], "YARA":[]}' --tags '["mytag"]' --comment 'mycomment' --insecure

If no error occurred, nothing or an empty list will be returned. Otherwise, a list containing the faulty tasks, as well as a description of the errors will be returned.

You can also use the Web-Interface by opening the file submit_task.html in your browser. However, you will need to create an exception for the certificate by visiting the website of the Gateway manually, before you can use the web interface.

Holmes Interrogation

Holmes Presentation

Transport

Toolbox and Helper scripts

FAQ