margaritashotgun¶
Python Remote Memory Aquisition
Quick Start¶
First, Install margaritashotgun.
Capture A Single Machine¶
A single machine can be captured using only the command line arguments for margaritashotgun.
First specify the server and user with the --server
and --username
flags.
Next provide a path to an ssh key with --key
(or use a password with the --password
flag).
Finally provide a lime kernel module with --module
and specify an output file with --filename
margaritashotgun --server 172.16.20.10 --username root --key root_access.pem --module lime-3.13.0-74-generic.ko --filename 172.16.20.10-mem.lime
Save Memory In S3¶
To save a file to s3 replace the filename
flag with --bucket
. Ensure that you have aws credentials configured prior to executing the following command.
margaritashotgun --server 172.16.20.10 --username root --key root_access.pem --module lime-3.13.0-74-generic.ko --bucket memory_capture_bucket
Capture Multiple Machines¶
Run margaritashotgun with a configuration file like parallel_config.yml.example
aws:
bucket: memory_dump_example
hosts:
- addr: 52.36.191.XXX
port: 22
username: ec2-user
key: access.pem
module: lime-4.1.19-24.31.amzn1.x86_64.ko
- addr: 52.36.170.XXX
port: 22
username: ec2-user
key: access.pem
module: lime-4.1.19-24.31.amzn1.x86_64.ko
- addr: 52.36.210.XXX
port: 22
username: ubuntu
key: dev.pem
module: lime-3.13.0-74-generic.ko
- addr: 52.36.90.XXX
port: 22
username: ubuntu
key: dev.pem
module: lime-3.13.0-74-generic.ko
workers: 2
Note
In this example parallelism is limited to 2 workers.
Run the capture with:
margaritashotgun -c your_custom_config.yml.
Installation¶
System Requirements¶
Margarita Shotgun is supported on common linux distributions, for other operating systems see the Install with Docker section.
While margaritashotgun is written purely in python, some of the libraries used require additional system packages.
Fedora / RHEL Distributions¶
- python-devel (2.X or 3.X)
- python-pip
- libffi-devel
- openssl-devel
Debian Distributions¶
- python-dev (2.X or 3.X)
- python-pip
- libffi-dev
- libssl-dev
Install From PyPi¶
$ pip install margaritashotgun
Install with Docker¶
Pull and run the python docker image.
$ docker pull python:3
$ docker run -ti python:3 bash
$ root@3009a5bc9817:/# pip install margaritashotgun
Note
If you plan on streaming memory to S3 ensure you setup IAM access keys in the docker container.
Set -e ACCESS_KEY_ID=ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=ACCESS_KEY
in the docker run
command above.
Alternately follow Amazon’s guide for configuring credentials once the docker container is running.
Install From Github¶
$ pip install git+ssh://git@github.com/ThreatResponse/margaritashotgun.git@master
$ margaritashotgun -h
Local Build and Install¶
$ git clone https://github.com/ThreatResponse/margaritashotgun.git
$ cd margaritashotgun
$ python setup.py sdist
$ pip install dist/margarita_shotgun-*.tar.gz
$ margaritashotgun -h
Local Execution¶
In the previous two example dependencies are automatically resolved, if you simply want to run margaritashotgun using the script bin/margaritashotgun
you will have to manually install dependencies
$ git clone https://github.com/ThreatResponse/margaritashotgun.git
$ cd margaritashotgun
$ pip install -r requirements.txt
$ ./bin/margaritashotgun -h
User Guide¶
Command Line¶
Note
See the quickstart for common examples.
Usage¶
Run margaritashotgun -h
at the command line, detailed information on flags is below.
Quick Reference¶
Flag | Use | Notes |
--config |
path to config file | See the Configuration File section |
--server |
ip of remote server | DNS records may also be used |
--version |
print version info | |
--bucket |
output S3 bucket | Incommpatible with -o |
--output-dir |
local output folder | Incompatible with -b |
--port |
ssh port | 22 is used unless specified |
--username |
ssh username | Username for ssh authentication |
--module |
lime kernel module | Required if no repository is enabled |
--password |
ssh password | Unlockes RSA key when used with -k |
--key |
RSA Key | Unlocked via -p if supplied |
--jump-server |
ip of jump host | DNS records may also be used |
--jump-port |
jump host ssh port | 22 is used unless specified |
--jump-username |
jump host ssh username | Username for jump host ssh authentication |
--jump-password |
jump host ssh password | |
--jump-key |
jump host RSA key | |
--filename |
output file | |
--repository |
enable kernel repo | Default state is disabled |
--repository-url |
custom repo url | Defaults to threat response modules |
--repository-manifest |
custom repo url | Defaults to “primary” |
--gpg-no-verify |
disable signature checks | |
--workers |
worker count | Constrains parallel captures |
--verbose |
log debug messages | |
--log-dir |
log directory | |
--log-prefix |
log file prefix |
Config¶
The -c
and --config
flags accept a relative or absolute path to a yml config file.
The structure of this file is outlided in the Configuration
section below.
Server¶
The --server
flag specifies the server being targeted for memory capture.
A DNS record or IP address are valid inputs.
Version¶
The --version
flag prints the module version.
Bucket¶
The --bucket
flag specifies the destination bucket when dumping memory to s3.
This flag cannot be used in conjunction wth -o
or --output-dir
.
Output-Dir¶
The --output-dir
flags specify the destination folder when dumping memory to the local filesystem.
This flag cannot be used in conjunction with --bucket
.
Port¶
The --port
flag specifies the port that ssh is running on the remote server specified by --server
.
This flag is optional and port 22
will be assumed if no value is provided.
Username¶
The --username
flag specifies the user account to authenticate with when connecting to the remote server specified by --server
.
Module¶
The --module
flag accepts a relative or absolute path to a LiME kernel module.
This flag is required if no kernel module repository is enabled with the --repository
flag.
Password¶
The --password
flag specifies the password used for authentication with connection to the remote server specified by --server
.
When used in conjuction with the --key
flag this password will be used to unlock a password protected private key file.
Key¶
The --key
flag accepts a relative or absolute path to a a private key file used for authentication when connecting to the server specified by -server
.
If the private key file specified is password protected use the -p
or --password
flags to specify the password that unlocks the private key.
Filename¶
The --filename
flags specify the name of the file memory will be saved to when dumping to the local filesystem.
The file will be saved to the local directory unless the --output-dir
option is configured.
Repository¶
The --repository
flag enables automatic kernel module resolution via the repository configured with --repository-url
.
Margarita Shotgun will not query any repositories unless explicitly enabled with the --repository
flag.
Repository-Url¶
The --repository-url
flag specifies where to search for kernel modules. The default public repository provided by Threat Response is availible at https://threatresponse-lime-modules.s3.amazonaws.com
Repository-manifest¶
The --repository-manifest
flag specifies alternate kernel module manifests in the remote repository configured by --repository-url
. For more information on repository structure and manifests see the architecture page or lime-compiler repository.
Gpg-no-verify¶
The --gpg-no-verify
flag disables gpg verification of kernel modules downloaded from a remote repository.
Workers¶
The --workers
flag specifies how many worker processes will be spawned to process memory captures in parallel.
The default value for this flag is auto
which will spawn a process per remote host up to the number of cpu cores on the local system.
Integer values can be provided instead of the auto
keyword.
Eg. --workers 3
will process 3 memory captures simultaneously.
Verbose¶
The --verbose
flag enables debug logging, including each command executed on remote hosts as a part of the memory capture process.
Log-Dir¶
The --log-dir
flag specifies the directory in which log files will be saved during memory capture.
Log-Prefix¶
The --log-prefix
flag specifies a custom case number that is prepended onto log files.
Configuration File¶
Example configuration files are availible in the repository. More documentation about the configuration file format is in the works.
Managing AWS Credentials¶
Margarita Shotgun does not support explicitly declaring aws credentials. Currently the only way to interact with S3 is by configuring an aws profile.
A feature is planned to allow selecting a profile other than the default
profile. Until that feature is completed the default
profile must be used.
Recommended IAM Policy¶
Margarita Shotgun only requires PutObject on a specified bucket.
Example¶
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::member-berries/*"
}
]
}
Wrapping Margarita Shotgun¶
Margarita Shotgun can be driven by another program when included as a python module. The configuration object passed to the margaritashotgun client must have the exact structure of the configuration file outlined above.
Example¶
>>> import margaritashotgun
>>> config = dict(aws dict(bucket = 'case-bucket'),
... hosts = [ dict(addr = '10.10.12.10',
... port = 22,
... username = 'ec2-user',
... key = '/path/to/private-key') ]
... workers = 'auto',
... logging = dict(log_dir = 'logs/',
... prefix = 'casenumber-10.10.12.10'),
... repository = dict(enabled = true,
... url = 'your-custom-kernel-module-repo.io'))
...
>>> capture_client = margaritashotgun.client(name='mem-capture', config=config,
... library=True, verbose=False)
...
>>> response = capture_client.run()
>>> print(response)
{'total':1,'failed':[],'completed':['10.10.12.10']}
Note that calling capture_client.run()
is a blocking operation.
Reference Guide¶
Authentication¶
Client¶
-
class
margaritashotgun.client.
Client
(config=None, library=True, name=None, verbose=False)¶ Client for parallel memory capture with LiME
-
__init__
(config=None, library=True, name=None, verbose=False)¶ Parameters:
-
__module__
= 'margaritashotgun.client'¶
-
map_config
()¶
-
run
()¶ Captures remote hosts memory
-
statistics
(results)¶
-
Cli¶
-
class
margaritashotgun.cli.
Cli
¶ -
__module__
= 'margaritashotgun.cli'¶
-
check_directory_path
(path)¶ Ensure directory exists at the provided path
Parameters: path (string) – path to directory to check
-
check_directory_paths
(*args)¶ Ensure all arguments correspond to directories
-
check_file_path
(path)¶ Ensure file exists at the provided path
Parameters: path (string) – path to directory to check
-
check_file_paths
(*args)¶ Ensure all arguments provided correspond to a file
-
configure
(arguments=None, config=None)¶ Merge command line arguments, config files, and default configs
Params arguments: Arguments produced by Cli.parse_args Params config: configuration dict to merge and validate
-
configure_args
(arguments)¶ Create configuration has from command line arguments
Params arguments: arguments produced by Cli.parse_args()
-
get_env_default
(variable, default)¶ Fetch environment variables, returning a default if not found
-
load_config
(path)¶ Load configuration from yaml file
Parameters: path (string) – path to configuration file
-
merge_config
(base, config)¶
-
Exceptions¶
-
exception
margaritashotgun.exceptions.
AuthenticationMethodMissingError
¶ Raised when no ssh authentication methods are specified
-
__init__
()¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
AuthenticationMissingUsernameError
¶ Raised when authentication method is configured without a username
-
__init__
()¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
ConfigurationMergeError
(reason)¶ Raised when merging user configuration with the base config fails
-
__init__
(reason)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
InvalidConfigurationError
(key, value, reason='unsupported configuration')¶ Raised when an unsupported configuration option is supplied
-
__init__
(key, value, reason='unsupported configuration')¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
KernelModuleNotFoundError
(kernel_version, repo_url)¶ Raised when no kernel module is provided and a suitable module cannot be found
-
__init__
(kernel_version, repo_url)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
KernelModuleNotProvidedError
(kernel_version)¶ Raised when no kernel module is provided and repository is disabled
-
__init__
(kernel_version)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
LimeRetriesExceededError
(retries)¶ Raised when max number of retries are exceeded waiting for LiME to load.
-
__init__
(retries)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
MargaritaShotgunError
¶ Base Error Class
-
__module__
= 'margaritashotgun.exceptions'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
exception
margaritashotgun.exceptions.
MemoryCaptureAttributeMissingError
(attribute)¶ Raised when memory capture is missing a required attribute
-
__init__
(attribute)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
MemoryCaptureOutputMissingError
(remote_host)¶ Raised when no output is configured when capturing memory
-
__init__
(remote_host)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
NoConfigurationError
¶ Raised when no configuration is supplied while operating as a library
-
__init__
()¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
RepositoryError
(metadata_url, reason)¶ Raised when malformed repository metadata is found
-
__init__
(metadata_url, reason)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
RepositoryMissingKeyMetadataError
(url)¶ Raised when signing public key is missing from repository
-
__init__
(url)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
RepositoryMissingSignatureError
(signature_url)¶ Raised when a detached signature is missing in remote repository”
-
__init__
(signature_url)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
RepositoryMissingSigningKeyError
(url)¶ Raised when signing public key is missing from repository
-
__init__
(url)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
RepositorySignatureError
(url, signature_url)¶ Raised when signature verification fails
-
__init__
(url, signature_url)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
-
exception
margaritashotgun.exceptions.
RepositoryUntrustedSigningKeyError
(url, fingerprint)¶ Raised when repository signing key is not trusted
-
__init__
(url, fingerprint)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.exceptions'¶
-
Logging¶
-
class
margaritashotgun.logger.
Logger
(*args, **kwargs)¶ -
__init__
(*args, **kwargs)¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
__module__
= 'margaritashotgun.logger'¶
-
-
margaritashotgun.logger.
cleanup
(log_file)¶
-
margaritashotgun.logger.
get_times
()¶
-
margaritashotgun.logger.
listener
(queue, name, log_file, desc)¶
Memory¶
-
class
margaritashotgun.memory.
Memory
(remote_addr, mem_size, progressbar=False, recv_size=1048576, sock_timeout=1)¶ -
__init__
(remote_addr, mem_size, progressbar=False, recv_size=1048576, sock_timeout=1)¶ Parameters:
-
__module__
= 'margaritashotgun.memory'¶
-
capture
(tunnel_addr, tunnel_port, filename=None, bucket=None, destination=None)¶ Captures memory based on the provided OutputDestination
Parameters: - tunnel_port (int) – ssh tunnel hostname or ip
- tunnel_port – ssh tunnel port
- filename (str) – memory dump output filename
- bucket (str) – output s3 bucket
- destination (
margaritashotgun.memory.OutputDestinations
) – OutputDestinations member
-
cleanup
()¶ Release resources used during memory capture
-
max_size
(mem_size, padding_percentage)¶ Calculates the excpected size in bytes of the memory capture
Parameters:
-
to_file
(filename, tunnel_addr, tunnel_port)¶ Writes memory dump to a local file
Parameters:
-
to_s3
(bucket, filename, tunnel_addr, tunnel_port)¶ Writes memory dump to s3 bucket
Parameters:
-
update_progress
(complete=False)¶ Logs capture progress
Params complete: toggle to finish ncurses progress bar
-
Remote Host¶
-
class
margaritashotgun.remote_host.
Host
¶ -
__init__
()¶
-
__module__
= 'margaritashotgun.remote_host'¶
-
capture_memory
(destination, filename, bucket, progressbar)¶
-
check_for_lime
(pattern)¶ Check to see if LiME has loaded on the remote system
Parameters:
-
cleanup
()¶ Release resources used by supporting classes
-
connect
(username, password, key, address, port, jump_host)¶ Connect ssh tunnel and shell executor to remote host
Parameters:
-
kernel_version
()¶ Returns the kernel kernel version of the remote host
-
load_lime
(remote_path, listen_port, dump_format='lime')¶ Load LiME kernel module from remote filesystem
Parameters:
-
log_async_result
(future)¶
-
mem_size
()¶ Returns the memory size in bytes of the remote host
-
start_tunnel
(local_port, remote_address, remote_port)¶ Start ssh forward tunnel
Parameters:
-
unload_lime
()¶ Remove LiME kernel module from remote host
-
upload_module
(local_path=None, remote_path='/tmp/lime.ko')¶ Upload LiME kernel module to remote host
Parameters:
-
wait_for_lime
(listen_port, listen_address='0.0.0.0', max_tries=20, wait=1)¶ Wait for lime to load unless max_retries is exceeded
Parameters:
-
-
margaritashotgun.remote_host.
process
(conf)¶
Remote Shell¶
-
class
margaritashotgun.remote_shell.
Commands
¶ -
__module__
= 'margaritashotgun.remote_shell'¶
-
kernel_version
= 'uname -r'¶
-
lime_check
= 'cat /proc/net/tcp'¶
-
lime_pattern
= '{0}:{1}'¶
-
load_lime
= 'sudo insmod {0} "path=tcp:{1}" format={2}'¶
-
mem_size
= "cat /proc/meminfo | grep MemTotal | awk '{ print $2 }'"¶
-
unload_lime
= 'sudo pkill insmod; sudo rmmod lime'¶
-
-
class
margaritashotgun.remote_shell.
RemoteShell
(max_async_threads=2)¶ -
-
__module__
= 'margaritashotgun.remote_shell'¶
-
cleanup
()¶ Release resources used during shell execution
-
connect
(auth, address, port, jump_host, jump_auth)¶ Creates an ssh session to a remote host
Parameters: - auth (
margaritashotgun.auth.AuthMethods
) – Authentication object - address (str) – remote server address
- port (int) – remote server port
- auth (
-
connect_with_auth
(ssh, auth, address, port, sock)¶
-
connect_with_key
(ssh, username, key, address, port, sock, timeout=20)¶ Create an ssh session to a remote host with a username and rsa key
Parameters:
-
connect_with_password
(ssh, username, password, address, port, sock, timeout=20)¶ Create an ssh session to a remote host with a username and password
Parameters:
-
decode
(stream, encoding='utf-8')¶ Convert paramiko stream into a string
Parameters: - stream – stream to convert
- encoding (str) – stream encoding
-
execute
(command)¶ Executes command on remote hosts
Parameters: command (str) – command to be run on remote host
-
execute_async
(command, callback=None)¶ Executes command on remote hosts without blocking
Parameters: - command (str) – command to be run on remote host
- callback (function) – function to call when execution completes
-
transport
()¶
-
Repository¶
-
class
margaritashotgun.repository.
Repository
(url, gpg_verify)¶ Lime-compiler repository client https://github.com/ThreatResponse/lime-compiler
-
__init__
(url, gpg_verify)¶ Parameters:
-
__module__
= 'margaritashotgun.repository'¶
-
check_signing_key
()¶ Check that repo signing key is trusted by gpg keychain
-
fetch
(kernel_version, manifest_type)¶ Search repository for kernel module matching kernel_version
Parameters:
-
fetch_module
(module)¶ Download and verify kernel module
Parameters: module (str) – kernel module path
-
get_manifest
(metadata)¶ Get latest manifest as specified in repomd.xml
Parameters: metadata (dict) – dictionary representation of repomd.xml
-
get_metadata
()¶ Fetch repository repomd.xml file
-
get_signing_key
()¶ Download a local copy of repo signing key for installation
-
init_gpg
()¶ Initialize gpg object and check if repository signing key is trusted
-
install_key
(key_data)¶ Install untrusted repo signing key
-
parse_manifest
(manifest_xml)¶ Parse manifest xml file
Parameters: manifest_xml (str) – raw xml content of manifest file
-
parse_metadata
(metadata_xml)¶ Parse repomd.xml file
Parameters: metadata_xml (str) – raw xml representation of repomd.xml
-
prompt_for_install
()¶ Prompt user to install untrusted repo signing key
-
unzip_manifest
(raw_manifest)¶ Decompress gzip encoded manifest
Parameters: raw_manifest (str) – compressed gzip manifest file content
-
verify_checksum
(data, checksum, filename)¶ Verify sha256 checksum vs calculated checksum
Parameters:
-
verify_data_signature
(signature_url, data_url, data)¶ Verify data against it’s remote signature
Parameters:
-
verify_file_signature
(signature_url, file_url, filename)¶ Verify a local file against it’s remote signature
Parameters:
-
SSH Tunnel¶
-
class
margaritashotgun.ssh_tunnel.
Forward
(local_port, remote_address, remote_port, transport)¶ -
__init__
(local_port, remote_address, remote_port, transport)¶ type: local_port: int param: local_port: local tunnel endpoint ip binding type: remote_address: str param: remote_address: Remote tunnel endpoing ip binding type: remote_port: int param: remote_port: Remote tunnel endpoint port binding type: transport:
paramiko.Transport
param: transport: Paramiko ssh transport
-
__module__
= 'margaritashotgun.ssh_tunnel'¶
-
forward_tunnel
(local_port, remote_address, remote_port, transport)¶
-
run
()¶ Method representing the thread’s activity.
You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.
-
stop
()¶
-
-
class
margaritashotgun.ssh_tunnel.
ForwardServer
(server_address, RequestHandlerClass, bind_and_activate=True)¶ -
__module__
= 'margaritashotgun.ssh_tunnel'¶
-
allow_reuse_address
= True¶
-
daemon_threads
= True¶
-
-
class
margaritashotgun.ssh_tunnel.
Handler
(request, client_address, server)¶ -
__module__
= 'margaritashotgun.ssh_tunnel'¶
-
handle
()¶
-
-
class
margaritashotgun.ssh_tunnel.
SSHTunnel
¶ -
__init__
()¶
-
__module__
= 'margaritashotgun.ssh_tunnel'¶
-
cleanup
()¶ Cleanup resources used during execution
-
configure
(transport, auth, address, port)¶ Connect paramiko transport
Parameters:
-
start
(local_port, remote_address, remote_port)¶ Start ssh tunnel
type: local_port: int param: local_port: local tunnel endpoint ip binding type: remote_address: str param: remote_address: Remote tunnel endpoing ip binding type: remote_port: int param: remote_port: Remote tunnel endpoint port binding
-
Workers¶
-
class
margaritashotgun.workers.
Workers
(conf, workers, name, library=True)¶ -
__init__
(conf, workers, name, library=True)¶
-
__module__
= 'margaritashotgun.workers'¶
-
cleanup
(terminate=False)¶
-
count
(workers, cpu_count, host_count)¶
-
cpu_count
= None¶
-
hosts
= None¶
-
progress_bar
= True¶
-
spawn
(desc, timeout=1800)¶
-
worker_count
= None¶
-
Architecture¶
Multiprocessing Overview¶
Coming Soon.
Kernel Module Repository¶
Kernel Modules can be automatically downloaded from a remote repository with the --repository
flag.
Repository Structure¶
Legacy repositories were dependent on the xml file listing provided by issuing a GET
request to the root of an amazon s3 bucket.
Metadata files have been introduced to remove s3 hosting as a requirement for repositories. The official Threat Response repository will continue to be hosted in s3
Note
Kernel modules will continue to be availible at the root of the Threat Response repository until at least the 0.4.0
release of Margarita Shotgun. GPG Signatures will not be provided for these modules.
The new repository structure introduces several new requirements.
- Optional The public portion of the GPG used for signing modules and metadata should present at the root of the repository with the filename
REPO_SIGNING_KEY.asc
. If the signing key is not present the--gpg-no-verify
flag must be used with Margarita Shotgun.
2. Optional A JSON file with key metadata including the key fingerprint. REPO_SIGNING_KEY.json.
Example:
`
{
"uids": ["Lime Signing Key (Threat Response Official Lime Signing Key) <security@threatresponse.cloud>"],
"fingerprint": "80DA92CB09161F241C8F9BC918BA980367172B17"
}
`
3. Required A folder must exist in at the path /repodata
which contains the following files.
- Required
repomd.xml
contains repository metadata including one or more manifests of kernel modules.- Optional
repomd.xml.sig
detached signature forrepomd.xml
. If not present in the repository the--gpg-no-verify
flag must be used with Margarita Shotgun.- Optional Manifest files. Techinally manifests can be stored at any relative path but it is recommended that they be stored in the
repodata
directory.
- Optional A
modules
directory is recommended which will contain the following files. Note the following files can have any location relative to the repository root, themodules
directory is simply best practice.- Required Compiled lime kernel modules. Module filenames are arbitrary as the files are explicitly listed in a manifest
- Optional Detached kernel module signatures. If signatures are not present the
--gpg-no-verify
flag must be used with Margarita Shotgun. The signature filename is arbitary as it is explicitly listed in a manifest.
Below is an example directory listing of the repository structure.
├── REPO_SIGNING_KEY.asc
├── REPO_SIGNING_KEY.json
├── modules
│ ├── lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko
│ ├── lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko.sig
│ ├── ...
│ ├── lime-4.4.8-20.46.amzn1.x86_64.ko
│ └── lime-4.4.8-20.46.amzn1.x86_64.ko.sig
└── repodata
├── a134928b6436bae3a9d9d1ddc47cb3c1539d4b559b266c84012ecb2e296b05a5-primary.xml.gz
├── repomd.xml.sig
└── repomd.xml
repomd.xml¶
The repomd.xml
file contains repository metadata required to resolve kernel modules.
The <revision></revision>
element contains the unix timestamp at which the metadata file was generated.
The structure of a <data/>
element is described in the following table.
type | metadata file type |
checksum | checksum of gzipped manifest file |
open_checksum | checksum of deflated manifest file |
location | href: path to gzipped manifest file |
timestamp | last modified timestamp of deflated file |
size | gzipped filesize |
open_size | deflated filesize |
Below is an example repository metadata listing.
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
<revision>1474605823</revision>
<data type="primary">
<checksum>8d9afbddd9041ed66d3da5db9b1d2d11fbdb5cf6a309d962ac4d1f66fb800551</checksum>
<open_checksum>a134928b6436bae3a9d9d1ddc47cb3c1539d4b559b266c84012ecb2e296b05a5</open_checksum>
<location href="repodata/a134928b6436bae3a9d9d1ddc47cb3c1539d4b559b266c84012ecb2e296b05a5-primary.xml.gz"/>
<timestamp>1474605823</timestamp>
<size>20726</size>
<open_size>143916</open_size>
</data>
</metadata>
Module manifest¶
A manifest consists of multiple module elements inside of a modules element. The module tag and it’s child elements are documented in the following table.
type | kernel module type |
name | kernel module friendly name |
arch | kernel module architecture |
checksum | kernel module checksum |
version | kernel version targeted by module |
packager | packager of kernel module |
location | href: path to module |
signature | href: path to module signature |
platform | the operating system this module targets |
Below is a truncated manifest.
<?xml version="1.0" encoding="UTF-8"?>
<modules>
<module type="lime">
<name>lime-2.6.32-358.11.1.el6.x86_64.ko</name>
<arch>x86_64</arch>
<checksum>1d7fc899a95b050a4f434c07012279e84bdd95234420648fbf348f5b4289e9e6</checksum>
<version>2.6.32-358.11.1.el6.x86_64</version>
<packager>lime-compiler info@threatresponse.cloud</packager>
<location href="modules/lime-2.6.32-358.11.1.el6.x86_64.ko"/>
<signature href="modules/lime-2.6.32-358.11.1.el6.x86_64.ko.sig"/>
<platform>linux</platform>
</module>
...
<module type="lime">
<name>lime-3.10.0-327.28.2.el7.x86_64.ko</name>
<arch>x86_64</arch>
<checksum>203e04dbe23ffb0c59d41760e7e8ebc55117e270de6ee17e149107345be6ed0d</checksum>
<version>3.10.0-327.28.2.el7.x86_64</version>
<packager>lime-compiler info@threatresponse.cloud</packager>
<location href="modules/lime-3.10.0-327.28.2.el7.x86_64.ko"/>
<signature href="modules/lime-3.10.0-327.28.2.el7.x86_64.ko.sig"/>
<platform>linux</platform>
</module>
</modules>
GPG Signatures¶
Unless explicitly disabled all kernel modules and metadata files will be checked agains their gpg signature in remote repositories. Failure to verify a signature, or lack of a signature for a given file is considered a fatal error and will result in a failed memory capture.
Note
Disable signature verification with --gpg-no-verify
.
Checksum verification cannot be disabled.
Build Kernel Modules¶
Kernel modules are build and signed by the lime-compiler. The source is availible and will soon be distributed as a ruby gem for use building private repositories.
Development¶
Tests¶
The test suite is written with pytest and can be run with py.test --cov=margaritashotgun
About¶
Margaritashotgun is a part of the Threat Response project.
License¶
Margarita Shotgun is distributed under the MIT License (MIT).