Overview¶
Vent is a library that includes a CLI designed to serve as a general platform for analyzing network traffic. Built with some basic functionality, Vent serves as a user-friendly platform to build custom plugins
on to perform user-defined processing on incoming network data. Vent is filetype-agnostic in that the plugins installed within your specific vent instance determine what type of files your instance supports.
Simply create your plugins
, point Vent to them, install them, and drop a file in Vent to begin processing!
Dependencies¶
docker >= 1.13.1
make (if building from source)
pip3
python3.6.x
Getting Set Up¶
There’s two ways to get Vent up and running on your machine:
Pip:
$ pip3 install vent
Clone the repo:
$ git clone --recursive https://github.com/CyberReboot/vent.git $ cd vent
Build from source (for sudo/root privileged users):
$ make
Users with limited permissions or require user-local installation can use the following:
$ sudo env "PATH=$PATH" make
Note
If you already have docker-py
installed on your machine, you may need to pip3 uninstall docker-py
first. vent
will install docker-py
as part of the installation process. However, there are known incompatibilities of docker-py
with older versions.
Once installed, it’s simply:
$ vent
Contributing to Vent¶
Want to contribute? Awesome! Issue a pull request or see more details here.
See this for a crash course on npyscreen: the GUI used by Vent!
Vent Quickstart Guide¶
Getting Vent set up is quick and easy.
1. First, use pip to install the latest stable version of Vent:
Using pip:
pip3 install vent && ventDeveloper versions of Vent are available but may not be entirely stable.
Using Docker:
docker pull cyberreboot/vent docker run -it vent_image_idIn order to avoid having to use sudo or run docker as root, adding your current user to the docker group is the recommended way to work.
Using Git:
git clone https://github.com/CyberReboot/vent cd vent && make && vent
2. Now that Vent has started, let’s add, build, and start the core tools.
- In the main menu, press
^X
to open the action menu- Select
Core Tools
or pressc
- Select
Add all latest core tools
or pressi
. Vent will now clone the core tools’ directories from CyberReboot/vent.- Select
Build core tools
from the core tools sub-menu and use the arrow keys and the Enter key to pressOK
. It’s possible to choose which core tools are built using the Space key. Boxes with anX
in them have been selected. Note that building the core tools takes a few minutes. Please be patient while Vent creates the images.- Once the tool images are built, go back to the core tools sub-menu from main action menu and select
Start core tools
and hitOK
. Much likeBuild core tools
, it’s possible to select which core tools are started.
3. The core tools’ containers are up and running. Next, let’s add some plugins.
- From the action menu, Select
Plugins
or pressp
.- Select
Add new plugin
or pressa
.- For this quick start guide, we will use one of the example plugins provided from CyberReboot/vent-plugins. So just hit
OK
on the form.- Press the Space key to choose the
master
branch and hitOK
.- Uncheck all the boxes except for
/tcpdump_hex_parser
and hitOK
. Depending on the plugin, add times may vary so it is not unusual for long plugin add times.
4. Now we have a plugin that can process files with the extension .pcap
.
- Now, at the Vent main menu, look for the field
File Drop
. This is the folder that Vent watches for new files.- Move or copy a
.pcap
file into the path. Vent will recognize this new file and starttcpdump_hex_parser
. Depending on the size of the.pcap
file, it could take anywhere from a few seconds to minutes. You should see thejobs running
counter increase by one and, after the plugin is finished running, thecompleted jobs
counter will increase by one.
5. Let’s look at the results of the plugin using elasticsearch.
- From the action menu, select
Services Running
and selectCore Services
.- Copy the address next to
elasticsearch
into the web browser of choice.- On the main page, there should be a section with
pcap
with the results of the plugin.
Congrats! Vent is setup and has successfully recognized the pcap file and ran a
plugin that specifically deals with pcaps. You can now remove the
tcpdump_hex_parser
via the Plugins
sub-menu and create and install your own
Custom Vent Plugins
Custom Vent Plugins¶
Building custom vent plugins is an easy process.
Each custom vent plugin needs at least a Dockerfile
and a vent.template
.
Read more about Dockerfile
here.
Vent.template Files¶
Vent template files are what Vent uses to build images into something it recognizes. Listed here are sections and their various options.
Look below for examples of a vent.template
file.
-docker¶
All possible options and their explanations are the same as the parameters for the python Docker container run command.
For example, if we wanted the plugin’s container to use the host network stack,
the following would be written in the vent.template
file:
[docker]
network_mode = host
-gpu¶
Vent plugins can run on solely on GPU if specified (extra on
). This section sets the
settings for GPU processing. At the moment, only NVIDIA GPUs are supported.
- dedicated
- Should the plugin be the only process running on the GPU? If it should be, set the
option to
yes
.no
by default. - device
- If you know the GPU device’s number, you can set it here.
- enabled
- Tell Vent to run the plugin on GPU
- mem_mb
- The amount of RAM(in MB) a plugin requires.
-info¶
All metadata about the custom plugin goes under this section.
- groups
- the group a plugin belongs to.
- name
- the plugin’s name.
-service¶
Appending info to an exposed port for a running service, multiple exposed ports can be done for the same service (up to 9) by incrementing the value of 1 at the end of each settings to correspond to the order of the ports exposed for that service.
- uri_prefix1
- Services a tool exposes that need a more specific URI at the beginning
- uri_postfix1
- Services a tool exposes that need a more specific URI at the end
- uri_user1
- Services a tool exposes that need a username
- uri_pw1
- Services a tool exposes that need a password
-settings¶
Options that specify how the plugin will run.
- ext_types
- Whatever this option is set to is the file extension that Vent will run this plugin on.
For example,
ext_types = jpg
means Vent will run this plugin if a.jpg
file is placed inFile Drop
. - priority
- Set the order in which tools get started. If a tool belongs to more than one group, it’s possible to specify priorities for each group. The priorities are comma separated and in the same order that the groups are specified.
- process_base
- Files put in
File Drop
by a user or process outside of Vent will get processed. This option is set toyes
by default. - process_from_tool
- Allows to specify if the tool should process files that are outputs from other tools specifically.
- instances
- Allows you to specify how many instantiations of a tool you want running. For example, you could have two instances of rq_worker running. This would create two rq_worker containers, which would be useful for scaling larger amounts of work.
Example Custom Vent Plugin¶
Let’s create a simple Vent plugin that uses cat
to output the contents of a
.example
file. Let’s create a vent_plugin_example
directory and enter it.
First, let’s create a simple bash
script that will cat
the contents of a
file.
#!bin/bash
cat $1
Next, a Dockerfile
is needed so let’s make it.
FROM ubuntu:latest
ADD cat.sh .
ENTRYPOINT ["/cat.sh"]
CMD [""]
Lastly, a vent.template
file is needed so Vent knows how to use the plugin:
[info]
name = example plugin
groups = example
[settings]
ext_types = example
Here’s an example of this plugin using GPUs to do work:
[info]
name = example plugin
groups = example
[settings]
ext_types = example
[gpu]
enabled = yes
mem_mb = 1024
dedicated = yes
We need to add this to either a git repo or the docker hub. Let’s use git.
Push the vent_plugin_example
into some repo.
Let’s now add the custom plugin to Vent. From the plugins sub-menu, select
Add new plugin
and enter the fields with whatever repo
vent_plugin_example
was pushed to. After, select the branch, commit and leave
build
to True
. Now select example_plugin
and hit OK
. Vent will
now build the custom plugin.
To test, let’s create a test file.
qwerty
Finally, with Vent and the plugin up and running and all core tools added, built,
and running, let’s drop test.example
into File Drop
. After a few
seconds, the job counter on the main menu of Vent will show that one job is
running, and it’ll finish soon after and show one completed job.
To check that the plugin worked and outputted qwerty
, let’s check the syslog
container using the command docker logs cyberreboot-vent-syslog-master | grep
qwerty
.
If you see this line, congrats! You have successfully built your first Vent plugin.
If the plugin did not function correctly, try rereading the tutorial or check the Troubleshooting guide.
Other examples of custom plugins can be found at CyberReboot/vent-plugins.
Custom Configurations of Plugins¶
Say your custom plugin, called example_plugin
, has a
example_plugin.config
file that it needs filled out before the plugin is
built into an image.
[Some_Section]
example_option1 = FILL IN
example_option2 = FILL IN
example_option3 = FILL IN
Vent can now fill in those values for you.
In your home directory, have a file called .plugin_config.yml
.
We’ll fill in .plugin_config.yml
with the necessary fields:
example_plugin
Some_Section:
example_option1 = 1
example_option2 = 2
example_option3 = 3
Vent will assume that, in the directory example_plugin
, there exists another
directory titled config
that has the actual config file.
The path of our example would be
example_plugin/config/example_plugin.config
.
Using this .plugin_config.yml
, you can fill in multiple tools’ config files
at the same time.
If you make a change to your config files and want to rebuild the images using
these new settings, clean and then build the plugin tools in the plugins
submenu. Vent will fill in the config files with the new values.
Vent Initialization¶
This page describes the different actions that vent takes upon initialization.
Auto-Install¶
Vent will detect if there are any vent images already running before an instance of itself is instantiated, and will automatically install the tools that correspond to that image without you having to do anything.
Startup¶
Before initialization you can specify a certain set of tools that you want vent to install once it starts up for the first time. You can do this by writing a .vent_startup.yml file in your home directory. If you do not know how YAML syntax works, watch this quick video for more info. The syntax that vent uses when parsing the .vent_startup.yml for tools is as follows:
repo_name:
tool_name:
<additional settings you want>
A thing to note, tool_name here means the name of the directory in which the Dockerfile and vent.template for the tool can be found. If you have multiple tools defined in the same directory, then use the syntax @tool_name to differentiate between the different tools in that directory. A lot of the additional settings you can configure are the same as what you would do for Custom Vent Plugins. These additional settings will be used to update the default settings for that tool (as defined in its vent.template), they will not overwrite the default settings. Besides those settings, there are some that are specific just to this startup file:
- build
- If you set build to yes, then the tool will be installed and then built immediately rather than just installed.
- start
- If you set start to yes, then the tool will be started immediately. Note, you have to set build to yes for a tool in order for you to be able to set start to yes as well.
- branch
- For tools installed from a repo, if you want to install the tools from a specific branch of the repo, specify it with the branch keyword. By default, the master branch will be where the tools are installed from.
- version
- For tools installed from a repo, if you want to install the tools from a specific commit version, specify it with the version keyword. By default, the version HEAD will be used to install tools.
Example Startup¶
With that said, here is an example .vent-startup.yml file that vent could use:
https://github.com/cyberreboot/vent:
rabbitmq:
build: yes
start: yes
elasticsearch:
rq_worker:
build: yes
start: yes
settings:
instances: 2
This would install the tools rabbitmq, elasticsearch, and rq_worker. Additionally, this would build and start rabbitmq and rq_worker, and it would set the number of instances in settings to 2 for rq_worker. Notice how with elasticsearch even though there were no additional parameters defined it still had the the colon at the end of its declaration. This is necessary otherwise a parsing error will occur with YAML.
Troubleshooting¶
Basic Troubleshooting¶
Something not working as expected within Vent? Not a problem! Let’s first get some basic possible errors out of the way.
Is Docker installed and is the daemon running? Vent uses Docker heavily so it is necessary to have the Docker installed and the daemon running.
Is this the latest version of Vent? Ways to get the latest Vent version:
Using pip:
pip3 install vent && vent
Pip installs the latest stable release of Vent. If a problem still persists, try using the latest developer build:
Using Docker:
docker pull cyberreboot/vent docker run -it vent_image_idUsing Git:
git clone https://github.com/CyberReboot/vent cd vent && make && vent
In-Depth Troubleshooting¶
Still not working? That’s fine! Let’s get into the nitty gritty and try to figure out what went wrong.
Is it Vent that’s causing the problem?¶
Firstly, let’s see if it’s Vent that’s causing the problems.
Go to the User Data
folder and open up vent.log
with your favorite
text editor. Let’s search the key term False
. Iterate through the
search results and look for Status of some_function: False
. This
tells us that one of Vent’s core functions is not performing as
expected. Next to it, there will be an error message explaining what
went wrong.
If there’s a problem with Vent’s implementation, please create an issue here.
Is it a custom plugin that’s causing the problem?¶
If there’s no obvious error messages within vent.log
, let’s check any
added plugin tools and their containers.
Run the command docker logs cyberreboot-vent-syslog-master
.
This will return all information about all plugin containers and any
information regarding the error should be displayed here.
Contributing to Vent¶
Want to hack on Vent? Awesome! Here are instructions to get you started. They are probably not perfect, please let us know if anything feels wrong or incomplete.
Contribution guidelines¶
Pull requests are always welcome¶
We are always thrilled to receive pull requests, and do our best to process them as fast as possible. Not sure if that typo is worth a pull request? Do it! We will appreciate it.
If your pull request is not accepted on the first try, don’t be discouraged! If there’s a problem with the implementation, hopefully you received feedback on what to improve.
We’re trying very hard to keep Vent lean and focused. We don’t want it to do everything for everybody. This means that we might decide against incorporating a new feature. However, there might be a way to implement that feature on top of Vent.
Create issues…¶
Any significant improvement should be documented as a github issue before anybody starts working on it.
…but check for existing issues first!¶
Please take a moment to check that an issue doesn’t already exist documenting your bug report or improvement proposal. If it does, it never hurts to add a quick “+1” or “I have this problem too”. This will help prioritize the most common problems and requests.
Conventions¶
Fork the repo and make changes on your fork in a feature branch.
Make sure you include relevant updates or additions to documentation and tests when creating or modifying features.
Pull requests descriptions should be as clear as possible and include a reference to all the issues that they address.
Code review comments may be added to your pull request. Discuss, then make the suggested modifications and push additional commits to your feature branch. Be sure to post a comment after pushing. The new commits will show up in the pull request automatically, but the reviewers will not be notified unless you comment.
Before the pull request is merged, make sure that you squash your commits into
logical units of work using git rebase -i
and git push -f
. After every
commit the test suite should be passing. Include documentation changes in the
same commit so that a revert would remove all traces of the feature or fix.
Commits that fix or close an issue should include a reference like Closes #XXX
or Fixes #XXX
, which will automatically close the issue when merged.
Add your name to the AUTHORS
file, but make sure the list is sorted and your
name and email address match your git configuration. The AUTHORS
file is
regenerated occasionally from the git commit history, so a mismatch may result
in your changes being overwritten.
Decision process¶
How are decisions made?¶
Short answer: with pull requests to the Vent repository.
All decisions affecting Vent, big and small, follow the same 3 steps:
- Step 1: Open a pull request. Anyone can do this.
- Step 2: Discuss the pull request. Anyone can do this.
- Step 3: Accept or refuse a pull request. A maintainer does this.
How can I become a maintainer?¶
- Step 1: learn the code inside out
- Step 2: make yourself useful by contributing code, bug fixes, support etc.
Don’t forget: being a maintainer is a time investment. Make sure you will have time to make yourself available. You don’t have to be a maintainer to make a difference on the project!
What are a maintainer’s responsibility?¶
It is every maintainer’s responsibility to:
- Deliver prompt feedback and decisions on pull requests.
- Be available to anyone with questions, bug reports, criticism, etc. regarding Vent.
New Vent Releases¶
Get the latest clone of Vent from https://github.com/CyberReboot/vent
Change the version number
setup.py
docs/source/conf.py
Edit
CHANGELOG.md
and include a list of changes that were made. Please follow previous formatting.Run list of authors and put in AUTHORS to make sure it is up to date:
git log --format='%aN <%aE>' | sort -f | uniq
Commit the changes, open a PR, and merge into
master
.Now let’s upload the release to pypi assuming there’s an account with admin access and a corresponding
.pypirc
:python3 setup.py sdist upload
Create a new github release. Tag and release title are the version number.
Finally, change the version number to the next version number with a
dev
tag. Eg:0.4.4.dev
. Commit the version change, make a PR, and merge tomaster
.
Core Tools and Plugins¶
Core Tool Explanations¶
There are currently nine core tools that Vent uses.
elasticsearch¶
Enables comprehensive text search of syslog.
file_drop¶
Watches the specified directory for any new files. If a new file is added, it is
added to a redis
queue.
network_tap¶
A container that will watch a specific nic using tcpdump
to output pcap
files based on what was monitored. Has an interface located in System
Commands -> Network Tap Interface
in the main action menu.
The interface has six available actions:
- Create: Create a new container with a specified nic, tag, interval (in seconds), filter, and iterations. The container is also automatically started on creation.
- Delete: Delete a specified network tap container. Containers must be stopped before they are able to be deleted.
- List: Show all network tap containers. Will return container’s ID, if the container is
running or not, and the tag provided in
create
. - NICs: Show all available network interfaces. Will return a list of the
names of the available NICs. Note for
Docker for Mac
it will show available network interfaces on the VM running the Docker daemon, not the network interface names on the Mac host. - Start: Start a network tap container if it is exited. Will run with the same
options given to the container in
create
. - Stop: Stop a network tap container.
- Update: Update the metadata of a network tap container.
rabbitmq¶
Formats messages received from syslog and sends them to rmq_es_connector.
redis¶
A key/value store that is used for the queuing system that file drop
sends to
and rq_worker
pulls out of.
rmq_es_connector¶
A gateway between the messaging system and elasticsearch
. This way, the message
formatting system is not locked to rabbitmq
.
rq_worker¶
The tool that takes files from the redis
queue and runs plugins that deal with
those file extensions.
rq_dashboard¶
Management console to look at rq_worker’s active queue.
syslog¶
Standard logging system that adheres to the syslog standard. All tool containers send their information to syslog. If there’s some unexpected outputs or a container isn’t running properly, all information will be in this tool’s container.
Access this tool’s container with the command:
docker logs cyberreboot-vent-syslog-master
Core Tool and Plugin Actions¶
Short explanations of all actions available in the core tools and plugins sub-menu.
Add all latest core/plugin tools¶
Clone the latest version of the tool. This will not update or remove any tools that have already been added.
By default, core tools are cloned from CyberReboot/vent and plugins, if no custom repo is specified, are cloned from CyberReboot/vent-plugins.
Build core/plugin tools¶
Build docker images from the Dockerfiles obtained from adding.
Clean core/plugin tools¶
Stop and remove the chosen tools’ containers.
Configure core/plugin tools¶
Edit a tool’s vent.template file found in the tool’s respective folder. Read more about Vent.template Files.
Disable core/plugin tools¶
Remove chosen tools from menus. For example, let’s say there were ten tools but only five were needed. Disabling the five unneeded tools would stop those tools from appearing on the other menus.
Enable core/plugin tools¶
Opposite of disable tools. Enables the tools so they can be seen again.
Inventory of core/plugin tools¶
Provides meta data regarding currently added core/plugin tools. It tells if a tool is built, enabled, the name of the image, and the if the tool is currently running.
Remove core/plugin tools¶
Remove a tool entirely. Any of that tool’s containers are also stopped and deleted. The tool must be added again if it is to be used.
Start core/plugin tools¶
Start the tools’ respective containers.
Stop core/plugin tools¶
Stop the tools’ respective containers.
Update core/plugin tools¶
Pulls the latest commit of the tool from its repo and builds it.
System Commands¶
This page describes what the different actions in the System Commands
sub-menu do.
Backup¶
Saves the vent.cfg
and the plugin_manifest
.
Change vent configuration¶
Edit the vent.cfg
in Vent. The vent.cfg
file can be found in the
directory listed next to User Data
in the main Vent menu.
Detect GPUs¶
Detect GPU devices on the current machine running Vent. Currently works with NVIDIA GPUs only .
Enable Swarm Mode¶
To Be Implemented.
Factory reset¶
Remove all Vent related images and containers. Removes the User Data
directory as well. Please back up any necessary information before activating this option.
Restore¶
Restores a backup.
Upgrade¶
To Be Implemented
Network Tap Interface¶
The interface in which to interact with the Network Tap
core tool. Read more
about network_tap. Each form will also have more information
regarding the specific action.
User Data¶
The User Data directory is listed on the Vent main menu. Here you can find five files and two directories here.
Files¶
plugin_manifest.cfg¶
Meta data about all built core and plugin tools.
status.json¶
All data regarding finished jobs is written to this JSON file. This file actually doesn’t exist until a job is processed and completed. Each tool will get it’s own JSON entry. If there were two tools that ran and finished, there will be one entry for each tool.
vent.cfg¶
A configuration file used to customize Vent and its processes.
-main¶
- files
- The directory of
File Drop
. For example, iffiles = /opt/vent_files
, Vent will monitor for any new files in that directory.- services_uri
Override the default POST name for all services. So, let’s say Vent has
Elasticsearch
at URL0.0.0.0:3772
.By adding:
services_uri = 5.5.5.5under the main section, the URL for
Elasticsearch
is now5.5.5.5:3772
.
-network-mapping¶
- nic_name
The option here can be whatever is desired. The value is the nic that the plugin tool
replay_pcap
will replay to.For example:
my_nic = enp0s25Now the plugin tool
replay_pcap
will mirror whatever given pcap files toenp0s25
.Currently, only one nic at a time is supported.
-nvidia-docker-plugin¶
- port
- Override port for the Nvidia docker plugin
- host
- Override the host for the Nvidia docker plugin
-external-services¶
Name of tool we want Vent to use externally rather than internally.
For example, if there is already an Elasticsearch
service that a user wants
Vent to use rather than Vent’s internal Elasticsearch
, we could write:
[external-services]
Elasticsearch = {"ip_address": "172.10.5.3", "port": "9200",
"locally_active": "yes"}
If a port is not specified, then it defaults to core tool’s default port. Also, you can toggle whether you want to use the local docker container that vent utilizes or the external service by switching locally_active between yes and no.
-build-options¶
- use_existing_images
- If set to
yes
it will not try to rebuild or pull images that already exist.
-groups¶
- start_order
The order in which the tool groups are started in csv format. For example:
core,pcap,replaymeans that core tools are started first, then any tool in the
pcap
group, and lastly any tool in thereplay
group. Within those groups, plugin priority will determine the start order of the tools.
vent.init¶
This file only tells Vent if this is the first time Vent is running so the tutorial can be the first menu listed rather than Vent’s main menu.
vent.log¶
Log file that keeps track of the statuses of Vent’s functionality. It lists what action is taking place and shows a list of steps taken to perform that action. So if something goes wrong, this file is a good place to start to find out what the problem is.
vent.api package¶
Submodules¶
vent.api.actions module¶
-
class
vent.api.actions.
Action
(**kargs)[source]¶ Bases:
object
Handle actions in menu
-
add
(repo, tools=None, overrides=None, version='HEAD', image=None, branch='master', build=True, user=None, pw=None, groups=None, version_alias=None, wild=None, remove_old=True, disable_old=True, update_repo=None)[source]¶ Add a new set of tool(s)
-
add_image
(image, link_name, tag=None, registry=None, groups=None)[source]¶ Add a new image from a Docker registry
-
backup
()[source]¶ Saves the configuration information of the current running vent instance to be used for restoring at a later time
-
build
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD')[source]¶ Build a set of tools that match the parameters given
-
clean
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD')[source]¶ Clean (stop and remove) a set of tools that match the parameters given, if no parameters are given, clean all installed tools on the master branch at verison HEAD that are enabled
-
disable
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD')[source]¶ Take an enabled tool and disable it
-
enable
(repo=None, name=None, groups=None, enabled='no', branch='master', version='HEAD')[source]¶ Take a disabled tool and enable it
-
get_configure
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD', main_cfg=False)[source]¶ Get the vent.template settings for a given tool by looking at the plugin_manifest
-
static
get_request
(url)[source]¶ Send a get request to the given url
- Args:
- url(str): url to send the data to. Eg: http://0.0.0.0:37728
- Returns:
- A tuple of success status and whatever the url is supposed to give after a GET request or a failure message
-
static
get_vent_tool_url
(tool_name)[source]¶ Iterate through all containers and grab the port number corresponding to the given tool name. Works for only CORE tools since it specifically looks for core
- Args:
- tool_name(str): tool name to search for. Eg: network-tap
- Returns:
- A tuple of success status and the url corresponding to the given tool name or a failure mesage. An example return url is http://0.0.0.0:37728. Works well with send_request and get_request.
-
static
post_request
(url, json_data)[source]¶ Send a application/json post request to the given url
- Args:
url(str): url to send the data to. Eg: http://0.0.0.0:37728 json_data(dict): json obj with data that will be sent to specified
urlaction(str): what is being done. Eg: ‘starting a container’
- Returns:
- A tuple of success status and whatever the url is supposed to give after a POST request or a failure message
-
prep_start
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD')[source]¶ Prep a bunch of containers to be started to they can be ordered
-
remove
(repo=None, namespace=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD', built='yes')[source]¶ Remove tools or a repo
-
restart_tools
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD', main_cfg=False, old_val='', new_val='')[source]¶ Restart necessary tools based on changes that have been made either to vent.cfg or to vent.template. This includes tools that need to be restarted because they depend on other tools that were changed.
-
save_configure
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD', config_val='', from_registry=False, main_cfg=False, instances=1)[source]¶ Save changes made to vent.template through npyscreen to the template and to plugin_manifest
-
start
(tool_d)[source]¶ Start a set of tools that match the parameters given, if no parameters are given, start all installed tools on the master branch at verison HEAD that are enabled
-
startup
()[source]¶ Automatically detect if a startup file is specified and stand up a vent host with all necessary tools based on the specifications in that file
-
stop
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD')[source]¶ Stop a set of tools that match the parameters given, if no parameters are given, stop all installed tools on the master branch at verison HEAD that are enabled
-
tool_status_checker
(tool_name)[source]¶ Reads from the plugin manifest. Checks to see if: 1. plugin manifest exists 2. if the given tool is built 3. if the given tool is running
- Args:
- tool_name(str): tool name. Checks plugin manifest option name
- Returns:
- A tuple of success status, and a tuple containing: bool describing if plugin manifest exists, bool describing if tool is built, bool describing if tool is running. eg: (True, (True, True, False))
-
tool_status_output
(tool_name)[source]¶ Function uses tool_status_checker to see tool status. Using that, it will return messages to output
- Args:
- tool_name(str): tool name
- Returns:
- A tuple of success status and a string to display
-
vent.api.plugin_helpers module¶
-
class
vent.api.plugin_helpers.
PluginHelper
(**kargs)[source]¶ Bases:
object
Handle helper functions for the Plugin class
-
available_tools
(path, version='HEAD', groups=None)[source]¶ Return list of possible tools in repo for the given version and branch
-
constraint_options
(constraint_dict, options)[source]¶ Return result of constraints and options against a template
-
prep_start
(repo=None, name=None, groups=None, enabled='yes', branch='master', version='HEAD')[source]¶ Start a set of tools that match the parameters given, if no parameters are given, start all installed tools on the master branch at verison HEAD that are enabled
-
start_containers
(container, tool_d, s_containers, f_containers)[source]¶ Start container that was passed in and return status
-
start_priority_containers
(groups, group_orders, tool_d)[source]¶ Select containers based on priorities to start
-
start_remaining_containers
(containers_remaining, tool_d)[source]¶ Select remaining containers that didn’t have priorities to start
-
vent.api.plugins module¶
-
class
vent.api.plugins.
Plugin
(**kargs)[source]¶ Bases:
object
Handle Plugins
-
add
(repo, tools=None, overrides=None, version='HEAD', image=None, branch='master', build=True, user=None, pw=None, groups=None, version_alias=None, wild=None, remove_old=True, disable_old=True, limit_groups=None, core=False, update_repo=None)[source]¶ Adds a plugin of tool(s) tools is a list of tuples, where the pair is a tool name (path to Dockerfile) and version tools are for explicitly limiting which tools and versions (if version in tuple is ‘’, then defaults to version) overrides is a list of tuples, where the pair is a tool name (path to Dockerfile) and a version overrides are for explicitly removing tools and overriding versions of tools (if version in tuple is ‘’, then tool is removed, otherwise that tool is checked out at the specific version in the tuple) if tools and overrides are left as empty lists, then all tools in the repo are pulled down at the version and branch specified or defaulted to version is globally set for all tools, unless overridden in tools or overrides branch is globally set for all tools build is a boolean of whether or not to build the tools now user is the username for a private repo if needed pw is the password to go along with the username for a private repo groups is globally set for all tools version_alias is globally set for all tools and is a mapping from a friendly version tag to the real version commit ID wild lets you specify individual overrides for additional values in the tuple of tools or overrides. wild is a list containing one or more of the following: branch, build, groups, version_alias the order of the items in the wild list will expect values to be tacked on in the same order to the tuple for tools and overrides in additional to the tool name and version remove_old lets you specify whether or not to remove previously found tools that match to ones being added currently (note does not stop currently running instances of the older version) disable_old lets you specify whether or not to disable previously found tools that match to ones being added currently (note does not stop currently running instances of the older version) limit_groups is a list of groups to build tools for that match group names in vent.template of each tool if exists Examples: - repo=fe: (get all tools from repo ‘fe’ at version ‘HEAD’ on branch ‘master’) - repo=foo, version=”3d1f”, branch=”foo”: (get all tools from repo ‘foo’ at verion ‘3d1f’ on branch ‘foo’) - repo=foo, tools=[(‘bar’, ‘’), (‘baz’, ‘1d32’)]: (get only ‘bar’ from repo ‘foo’ at version ‘HEAD’ on branch ‘master’ and ‘baz’ from repo ‘foo’ at version ‘1d32’ on branch ‘master’, ignore all other tools in repo ‘foo’) - repo=foo overrides=[(‘baz/bar’, ‘’), (‘.’, ‘1c4e’)], version=‘4fad’: (get all tools from repo ‘foo’ at verion ‘4fad’ on branch ‘master’ except ‘baz/bar’ and for tool ‘.’ get version ‘1c4e’) - repo=foo tools=[(‘bar’, ‘1a2d’)], overrides=[(‘baz’, ‘f2a1’)]: (not a particularly useful example, but get ‘bar’ from ‘foo’ at version ‘1a2d’ and get ‘baz’ from ‘foo’ at version ‘f2a1’ on branch ‘master’, ignore all other tools)
-
add_image
(**kwargs)¶
-
auto_install
()[source]¶ Automatically detects images and installs them in the manifest if they are not there already
-
builder
(**kwargs)¶
-
current_version
(name, namespace=None, branch='master')[source]¶ Return current version for a given tool
-
disable
(name, namespace=None, branch='master', version='HEAD')[source]¶ Disable tool at a specific version, default to head
-
enable
(name, namespace=None, branch='master', version='HEAD')[source]¶ Enable tool at a specific version, default to head
-
fill_config
(path)[source]¶ Will take a yml located in home directory titled ‘.plugin_config.yml’. It’ll then fill in, using the yml, the plugin’s config file
-
remove
(name=None, repo=None, namespace=None, branch='master', groups=None, enabled='yes', version='HEAD', built='yes')[source]¶ Remove tool (name) or repository, repository is the url. If no arguments are specified, all tools will be removed for the defaults.
-
state
(name, namespace=None, branch='master')[source]¶ Return state of a tool, disabled/enabled for each version
-
vent.api.templates module¶
-
class
vent.api.templates.
Template
(template=None)[source]¶ Bases:
object
Handle parsing templates
-
add_option
(**kwargs)¶
-
add_section
(**kwargs)¶
-
constrained_sections
(**kwargs)¶
-
del_option
(**kwargs)¶
-
del_section
(**kwargs)¶
-
option
(**kwargs)¶
-
options
(**kwargs)¶
-
section
(**kwargs)¶
-
sections
(**kwargs)¶
-
set_option
(**kwargs)¶
-
write_config
(**kwargs)¶
-
Module contents¶
vent.core package¶
Subpackages¶
vent.core.file_drop package¶
Submodules¶
vent.core.file_drop.file_drop module¶
-
class
vent.core.file_drop.file_drop.
GZHandler
(patterns=None, ignore_patterns=None, ignore_directories=False, case_sensitive=False)[source]¶ Bases:
watchdog.events.PatternMatchingEventHandler
Handles when an event on the directory being watched happens that matches the values in patterns
-
created_files
= {}¶
-
ignore_patterns
= ['*-miscellaneous*']¶
-
on_created
(event)[source]¶ Called when a file or directory is created.
Parameters: event ( DirCreatedEvent
orFileCreatedEvent
) – Event representing file/directory creation.
-
on_modified
(event)[source]¶ Called when a file or directory is modified.
Parameters: event ( DirModifiedEvent
orFileModifiedEvent
) – Event representing file/directory modification.
-
patterns
= ['*']¶
-
process
(event)[source]¶ - event.event_type
- ‘modified’ | ‘created’ | ‘moved’ | ‘deleted’
- event.is_directory
- True | False
- event.src_path
- path/to/observed/file
-
q
= Queue('default')¶
-
r
= Redis<ConnectionPool<Connection<host=redis,port=6379,db=0>>>¶
-
Module contents¶
vent.core.network_tap package¶
Subpackages¶
vent.core.network_tap.ncontrol package¶
-
class
vent.core.network_tap.ncontrol.paths.
CreateR
[source]¶ Bases:
object
This endpoint is for creating a new filter
-
class
vent.core.network_tap.ncontrol.paths.
DeleteR
[source]¶ Bases:
object
This endpoint is for deleting a network tap filter container
-
class
vent.core.network_tap.ncontrol.paths.
InfoR
[source]¶ Bases:
object
This endpoint is for returning info about this service
-
class
vent.core.network_tap.ncontrol.paths.
ListR
[source]¶ Bases:
object
This endpoint is for listing all filter containers
-
class
vent.core.network_tap.ncontrol.paths.
NICsR
[source]¶ Bases:
object
This endpoint is for listing all available network interfaces
-
class
vent.core.network_tap.ncontrol.paths.
StartR
[source]¶ Bases:
object
This endpoint is for starting a network tap filter container
Module contents¶
vent.core.rmq_es_connector package¶
Submodules¶
vent.core.rmq_es_connector.rmq_es_connector module¶
-
class
vent.core.rmq_es_connector.rmq_es_connector.
RmqEs
(es_host='elasticsearch', rmq_host='rabbitmq')[source]¶ Bases:
object
opens a connection to rabbitmq and receives messages based on the provided binding keys and then takes those messages and sends them to an elasticsearch index
-
callback
(ch, method, properties, body)[source]¶ callback triggered on rabiitmq message received and sends it to an elasticsearch index
-
channel
= None¶
-
connections
(wait)[source]¶ wait for connections to both rabbitmq and elasticsearch to be made before binding a routing key to a channel and sending messages to elasticsearch
-
es_conn
= None¶
-
es_host
= None¶
-
es_port
= 9200¶
-
queue_name
= None¶
-
rmq_host
= None¶
-
rmq_port
= 5672¶
-
vent.core.rmq_es_connector.test_rmq_es_connector module¶
-
class
vent.core.rmq_es_connector.test_rmq_es_connector.
Method
(routing_key='foo.bar')[source]¶ Bases:
object
create mock method object
-
routing_key
= None¶
-
-
vent.core.rmq_es_connector.test_rmq_es_connector.
test_rmq_es_connector_callback
()[source]¶ tests the callback function
Module contents¶
vent.core.rq_worker package¶
Submodules¶
vent.core.rq_worker.test_watch module¶
vent.core.rq_worker.watch module¶
Module contents¶
Module contents¶
vent.helpers package¶
Submodules¶
vent.helpers.meta module¶
-
vent.helpers.meta.
Containers
(vent=True, running=True, exclude_labels=None)[source]¶ Get containers that are created, by default limit to vent containers that are running
-
vent.helpers.meta.
Dependencies
(tools)[source]¶ Takes in a list of tools that are being updated and returns any tools that depend on linking to them
-
vent.helpers.meta.
Images
(vent=True)[source]¶ Get images that are build, by default limit to vent images
-
vent.helpers.meta.
Jobs
()[source]¶ Get the number of jobs that are running and finished, and the number of total tools running and finished for those jobs
-
vent.helpers.meta.
ParsedSections
(file_val)[source]¶ Get the sections and options of a file returned as a dictionary
-
vent.helpers.meta.
Services
(core, vent=True, external=False, **kargs)[source]¶ Get services that have exposed ports, expects param core to be True or False based on which type of services to return, by default limit to vent containers and processes not running externally, if not limited by vent containers, then core is ignored.