IRMA: Incident Response & Malware Analysis¶
Introduction¶
This publication is intended for advanced technical users of IRMA Enterprise. It assumes the reader has working knowledge of Systems Administration, the GNU/Linux operating system and basic Python.
IRMA (Incident Response and Malware Analysis) is a flexible content analysis orchestration platform. This guide will explain how to install, configure, use and customize it. This is an introductory chapter to IRMA. It describes IRMA’s overall purpose, architecture and process.
Purpose¶
IRMA provides its users with the ability to objectively assess whether content is malicious or not. Content may be delivered by various means as described in this document, and subsequently distributed to various configurable analysis engines (“probes”). After analysis, this information is then conveyed to the user.
In addition to this core functionality, IRMA provides an overview of the overall analysis process and incident.
Finally, IRMA is designed to be installed and maintained in self-contained on-premises environment. This enables discreet and self-contained analysis for organizations which do not wish to disclose potentially confidential files to third parties.
The ultimate purpose of IRMA is to orchestrate the entire analysis process and provide organizations with a flexible platform with which to manage and asses the content flowing through their organizations.
File Analysis Process¶
IRMA consists or three basic components. the Frontend, Brain and Probes. The basic functionality of frontend is to store results and host the API. Brain split analysis jobs on every probes involved, and Probes analyze files and returns results.
- An analysis begins when a user uploads files to the Frontend.
- Frontend checks for existing files and results in SQL. If needed, it stores the new files and calls asynchronously scan jobs on Brain.
- Brain worker sends as much subtasks to Probe(s) as needed.
- Probe workers process their jobs and send back results to Brain.
- Brain sends results to Frontend.
Supported Analyzers¶
Here is the list of analyzers that are bundled with IRMA.
Antiviruses¶
Probe Name | Anti-Virus Name | Platform |
---|---|---|
ASquaredCmdWin | Emsisoft Command Line | Microsoft Windows CLI |
AvastCoreSecurity | Avast Core Security | GNU/Linux CLI |
AVGAntiVirusFree | AVG | GNU/Linux CLI |
AviraWin | Avira | Microsoft Windows CLI |
BitdefenderForUnices | Bitdefender | GNU/Linux CLI |
ClamAV | ClamAV | GNU/Linux CLI |
ComodoCAVL | Comodo Antivirus for Linux | GNU/Linux CLI |
DrWeb | Dr.Web | GNU/Linux CLI |
EScan | eScan | GNU/Linux CLI |
EsetFileSecurity | Eset File Security | GNU/Linux CLI |
FProt | F-Prot | GNU/Linux CLI |
FSecure | F-Secure | GNU/Linux CLI |
GDataWin | G Data Antivirus | Microsoft Windows CLI |
Kaspersky | Kaspersky File Server | GNU/Linux CLI |
KasperskyWin | Kaspersky Internet Security | Microsoft Windows CLI |
McAfeeVSCL | McAfee VirusScan Command Line | GNU/Linux CLI |
McAfeeVSCLWin | McAfee VirusScan Command Line | Microsoft Windows CLI |
Sophos | Sophos | GNU/Linux CLI |
SophosWin | Sophos Endpoint Protection | Microsoft Windows CLI |
SymantecWin | Symantec Endpoint Protection | Microsoft Windows CLI |
VirusBlokAda | VirusBlokAda | GNU/Linux CLI |
Zoner | Zoner Antivirus | GNU/Linux CLI |
External analysis platforms¶
Probe Name | Analysis Platform | Description |
---|---|---|
ICAP | ICAP | Query an ICAP server |
VirusTotal | VirusTotal | Report is searched using the sha256 of the file which is not sent |
File database¶
Probe Name | Database | Description |
---|---|---|
NSRL | National Software Reference Library | collection of digital signatures of known, traceable software applications |
Metadata¶
Probe Name | Description |
---|---|
LIEF | PE/ELF File analyzer |
PEiD | PE File packer analyzer |
TrID | File type identification |
StaticAnalyzer | PE File analyzer adapted from Cuckoo Sandbox |
Yara | Checks if a file match yara rules |
Installation¶
This chapter describes the methods available to install IRMA using Ansible scripts.
Software requirements¶
- Ansible; You can see the requirement version of ansible in
ansible/requirements.txt
ansible==2.4.2.0
Hardware requirements¶
The IRMA platform is divided in three major components: the Frontend, the Brain and one or multiple Probes.
These three components can be installed on a unique host or on multiple hosts, according to the kind of probes that are being used.
The Frontend and the Brain must be installed on a GNU/Linux system [1]. Quarkslab recommends using a Debian Stable distribution which is supported and known to work.
According to the kind of probes and their dependencies, each analyzers can be installed on a separate hosts or share the same host as far as they do not interfere with each other [2]. Currently, only Debian Stable and Microsoft Windows 8 and 10 hosts have been tested.
Quarkslab does not provide any estimates regarding performance. However, the following configuration is known to provide reasonable performance for small deployments:
whole IRMA platform on a single machine by hosting it with multiple systems inside virtual machines: this setup gives fairly high throughput as long as it has reasonable IO (ideally, SSDs), and a good amount of memory (test setup was an i7 cpu with 16 GB ram on regular drives (at least 200 GB required),
For larger deployments, the following configuration is known to work: a single high-memory machine, with 16+ cores, and SSDs, could run IRMA platform and bear the workload load with reasonable response time.
Footnotes
[1] | Theorically, it should be possible, with some efforts, to make IRMA work on Microsoft Windows systems as most of the components used for the platform are known to work or to have equivalents on these systems. |
[2] | For instance, we managed to host several GNU/Linux anti-viruses on an unique probe by preventing it to launch daemons at startup. This is difficult for Microsoft systems on which it is not recommended to install multiple anti-viruses on a single host. |
[3] | with a limited set of probes |
Automated Installation¶
The IRMA platform is easily installed thanks to a set of ansible roles and playbooks. It permits a user to build, install or maintain different setups.
There are 2 different types of IRMA environment, and multiple setups for each environment:
- Development environment (sources rsync’d between host and vms)
allinone_dev
: everything installed in the same vmdev
: every component on its own vm
- Production environment (sources installed through generated archives, install on vms/physical servers)
allinone_prod
: everything installed in the same vm/physical server (default environment)prod
: every component on its own vm/physical server
For specific instructions on these 2 environments see the related section.
Note
Vagrant step is optional in production mode.
Environment file¶
IRMA installation uses ansible and optionally Vagrant, and supports a common configuration format that allows
launching of Vagrant and/or ansible. VagrantFile
automatically parses the configuration file to allow vagrant to launch required virtual machines, and irma-ansible.py
parses this same file to create an inventory and an extra variable (vars) file before launching ansible.
Format¶
For examples look at the files *.yml
in the ansible/environments
directory.
Whole IRMA infrastructure is described here:
servers:
- name: <hostname>
ip: <ip address>
ansible_groups: [list of ansible groups]
box: [vagrant box name]
cpus: [vagrant cpus (optional)]
memory: [vagrant memory (optional)]
shares: [vagrant share (optional)]
[...]
libvirt_config:
driver: kvm
# connect_via_ssh: true
# host:
# username:
# storage_pool_name:
# id_ssh_key_file:
ansible_vars:
key: value
[...]
servers
section both described ansible usage of the server and its vagrant configuration if needed.libvirt_config
section is a vagrant-only section for using libvirt hypervisor.ansible_vars
section is an ansible-only section for defining extra ansible variables.
Example of a development environment with vagrant:
servers:
- name: brain.irma
ip: 172.16.1.30
ansible_groups: [frontend, sql-server, brain, comodo, trid]
box: quarkslab/debian-9.0.0-amd64
cpus: 2
memory: 2048
shares:
- share_from: ../common
share_to: /opt/irma/irma-common/releases/sync
share_exclude:
- .git/
- venv/
- share_from: ../frontend
share_to: /opt/irma/irma-frontend/releases/sync
share_exclude:
- .git/
- venv/
- web/dist
- web/node_modules
- share_from: ../brain
share_to: /opt/irma/irma-brain/releases/sync
share_exclude:
- .git/
- venv/
- db/
- share_from: ../probe
share_to: /opt/irma/irma-probe/releases/sync
share_exclude:
- .git/
- venv/
libvirt_config:
driver: kvm
ansible_vars:
irma_environment: development
vagrant: true
And an example of an environment without vagrant:
servers:
- name: frontend.irma
ip: 172.16.1.30
ansible_groups: [frontend, sql-server]
- name: brain.irma
ip: 172.16.1.31
ansible_groups: [brain]
- name: avs-linux.irma
ip: 172.16.1.32
ansible_groups: [avast, avg, bitdefender, clamav, comodo, escan]
- name: mcafee-win.irma
ip: 172.16.1.33
ansible_groups: [mcafee-win]
windows: true
ansible_vars:
irma_environment: production
vagrant: true
irma_release: HEAD
Extra vars¶
It is possible to customize IRMA variables in section ansible_vars
(see irma_vars.yml.sample
for a full list of available vars).
Vagrant setup¶
Requirements¶
- Vagrant 1.9 or higher has to be installed
- a supported hypervisor:
- kvm/qemu (libvirt required, vagrant-libvirt plugin required)
- Virtualbox
Vagrant setup¶
(venv)$ export VM_ENV=dev
(venv)$ export VM_ENV=allinone_dev
(venv)$ export VM_ENV=prod
(venv)$ export VM_ENV=allinone_prod # (default)
Simply run in the Vagrantfile directory:
(venv)$ vagrant up (--provider=libvirt)
Vagrant will launch one/many VM(s).
Note
The basebox used in this project is provided by Quarkslab. The code source to build it is here.
Useful commands¶
Some useful commands with vagrant:
$ vagrant ssh <server_name> # login through ssh
$ vagrant halt <server_name> # shutdown the machine
$ vagrant reload <server_name> # restart the machine
$ vagrant up <server_name> # start the machine
$ vagrant destroy <server_name> # delete the machine
Ansible setup¶
Common requirements¶
- Ansible 2.0+ (see requirements.txt for version required)
(venv)$ pip install -r requirements.txt
Warning
Due to ansible breaking releases, the ansible version supported is now fixed
Ansible playbooks¶
IRMA Installation is split in playbooks (in ansible/playbooks directory):
- playbooks/provisioning.yml for dependencies setup
- playbooks/updating.yml for av update only
- playbooks/deployment.yml for irma code setup
- playbooks/playbook.yml (provisioning + updating + deployment)
Launch Ansible¶
Note
If your environment requires some virtual machines handled by vagrant, you must do this first.
To launch one of these playbook, the full command is:
# Dependencies setup
(venv)$ python irma-ansible.py environments/allinone_prod.yml playbooks/provisioning.yml
# AV update only
(venv)$ python irma-ansible.py environments/allinone_prod.yml playbooks/updating.yml
# IRMA code install
(venv)$ python irma-ansible.py environments/allinone_prod.yml playbooks/deployment.yml
# Full install (provisioning + updating + deployment)
(venv)$ python irma-ansible.py environments/allinone_prod.yml playbooks/playbook.yml
Last one will do the full install of IRMA. It can take a while (from 15 to 30 min) depending on the amount of RAM available on the machine and the hard disk drive I/O speed.
The default IRMA interface is available at http://172.16.1.30. According to your frontend server configuration.
References¶
Some roles from Ansible Galaxy used here:
- NodeJS role from JasonGiedymin/nodejs
- Nginx role from jdauphant/ansible-role-nginx
- OpenSSH role from Ansibles/openssh
- UFW role from weareinteractive/ansible-ufw
- Sudo role from weareinteractive/ansible-sudo
- Users role from mivok/ansible-users
Windows provisioning¶
Generate Windows base box¶
$ git clone https://github.com/boxcutter/windows
$ cd windows
$ make virtualbox/eval-win10x64-enterprise
Adding to Vagrant boxes¶
$ vagrant box add --name eval-win10x64-enterprise box/virtualbox/eval-win10x64-enterprise*.box
Creating an instance of the base box¶
$ VM_ENV=<your_env> vagrant up
Provisioning with ansible¶
In the config file don’t forget to add windows: true
in the server. Example:
servers:
- name: mcafee-win.irma
ip: 172.16.1.33
box: eval-win10x64-enterprise
ansible_groups: [mcafee-win]
windows: true
Provisioning a windows host is done the same way as other hosts:
(venv)$ python irma-ansible.py environments/allinone_prod.yml playbooks/playbook.yml
Production environment¶
IRMA will be installed on physical servers.
1. Prep servers¶
Create an account for ansible provisioning, or use one which has already been created. To speed up provisioning, you can:
- Authorize your SSH key for password-less authentication (optional):
*On your local machine*
$ ssh-copy-id user@hostname # -i if you want to select your identity file
- If you don’t want to have to type your password for
sudo
command execution, add your user to sudoers, usingvisudo
command (optional):
user ALL=(ALL) NOPASSWD: ALL
2. Configure the installation¶
Modify ansible extra_vars
especially the provisioning_ssh_key
section,
you’ll need to add private keys from user for password-less connection to the
default IRMA server user.
Warning
Be careful, you’ll need to change all passwords from this configuration files (password
variables for most of them).
You’ll need to create a configuration file and adapt it to your infrastructure.
Extras¶
Installation behind a corporate proxy¶
Thanks to the vagrant-proxyconf plugin, IRMA can be installed behind corporate proxy.
First, vagrant-proxyconf
has to be installed:
$ vagrant plugin install vagrant-proxyconf
Then, the vagrant-proxyconf
configuration has to be added to ansible/Vagrantfile.
Here is an example:
if Vagrant.has_plugin?("vagrant-proxyconf")
config.proxy.http = "http://corporate.proxy:3128"
config.proxy.https = "http://corporate.proxy:3128"
config.proxy.no_proxy = "localhost,127.0.0.1"
end
Finally, vagrant up
can be launched, as usual.
It has to be noted that using such mechanism has two limitations:
- it is not working with Windows based boxes
- it is not working with tools that are not able do deal with environment based
proxy definition (
http_proxy
andhttps_proxy
environment variables). For instance, AVG updater does not take into account such definition.
Use IRMA¶
There are 2 ways to use IRMA :
Web Interface¶
How to do a scan¶
First choose one or multiple files to scan by:
- Drop it in the select area
- Click on “Choose file” button
Now, you can see the selected files on the right.
To cancel a file selection, click on the red cross next to the filename.
By clicking on the “Display advanced settings”, you can see and determine scan parameters. Note that the defaults parameters are not reset by default after a scan.
In the scan parameters you can choose if the scans will be forced, meaning that the files are unconditionally scaned, even if there is a cached result. You can choose too which probes will be lauched.
When you are ready, launch the scan by clicking on “Scan for malwares” button.
Wait during the upload of your files.
By now, you are on the results page. At the top, the scan status is displayed :
- The progression rate
- The scan status (if the scan is running or finish)
- The link to download the scan report in csv format
- The scan Id, a unique id to identify this scan that you can share
- The number of probe tasks done on the total number of probe tasks for the scan.
Next, the page displays the list of scan’s files and their status. Click on the file’s name to display the detailed scan result of a file scan.
In the first part of detailed scan result page, you can obtain information about the scanned file: filename, size, mimetype, different hashes, date of the first scan and the last scan of this file.
In the second part, you can see the details of the differents probe tasks ranked by probe type.
Firstly, the antivirus. For each antivirus, the following information are given:
- The name and the platform used
- The name of threat if it exist
- The version of the antivirus
- The version of the virus database
- The duration of the task
Note there is a color code to quickly see the status of the probe : green if everything is ok, red if a threat was be founded or orange if there was a problem with the probe.
Then, it’s the metadata and external parts : each probes of those classes have different ways to display their results.
How to do a research¶
It’s possible to recover scan results in the “Search” section.
There are two ways to search scan : a research by name or a research by hash with a sha256. To this end, select in the scroll bar “By name” or “By hash” and effect your research : then a list of files’ results ranked by date is displayed.
Note
To add a filter tag, see the section “Playing with tags”
Playing with tags¶
Note
Tags are available in IRMA from version 1.3.0
Creating a tag¶
You could create tags by using the command line tools
>>> from irmacl.helpers import *
>>> tag_list()
[]
>>> tag_new("archive")
{u'text': u'archive', u'id': 1}
>>> tag_list()
[Tag archive [1]]:
or directly from your terminal by using curl and posting a json with ‘text’ key:
$ curl -H "Content-Type: application/json; charset=UTF-8" -X POST -d '{"text":"<your tag>"}' http://172.16.1.30/api/v1.1/tags
Note
There is currently no way to create a tag directly from the web IHM.
Tagging a File¶
Directly in web IHM, once you are on a file details page:
Just click the tag bar and you will see all available tags. You could add multiple tags.
It is also possible to add a tag through command line tools:
>>> from irmacl.helpers import *
>>> help(file_tag_add)
Signature: file_tag_add(sha256, tagid, verbose=False)
Docstring:
Add a tag to a File
:param sha256: file sha256 hash
:type sha256: str of (64 chars)
:param tagid: tag id
:type tagid: int
:return: No return
>>> file_tag_add("346ae869f7c7ac7394196de44ab4cfcde0d1345048457d03106c1a0481fba853",1)
Searching by tag¶
You could specify one or more tags while searching for files too:
choose your tag list then hit the search button:
or by command line:
>>> from irmacl.helpers import *
>>> file_search(tags=[1])
(1, [<irma.apiclient.IrmaResults at 0x7f079ca23890>])
Command Line Interface¶
For a use of IRMA by command line, use the command line tools
This api client is only made for IRMA API version 1.1.
Installation¶
$ python setup.py install
Configuration file contains the API endpoint (full url) and some optional paramters (max number and delay in second between retries)
[Server]
api_endpoint=http://172.16.1.30/api/v1.1
max_tries=3
pause=1
and is searched in these locations in following order:
- current directory
- environment variable (“IRMA_CONF”)
- user home directory
- global directory (“/etc/irma”)
Once you set up a working irma.conf settings file, you could run tests on your running IRMA server:
$ python setup.py test
Usage¶
>>> from irmacl.helpers import *
>>> probe_list()
[u'StaticAnalyzer', u'Unarchive', u'VirusBlokAda', u'VirusTotal']
>>> tag_list()
[Tag malware [1], Tag clean [2], Tag suspicious [3]]
>>> scan_files(["./irma/tests/samples/eicar.com"], force=True, blocking=True)
Scanid: ca2e8af4-0f5b-4a55-a1b8-2b8dc9ead068
Status: finished
Options: Force [True] Mimetype [True] Resubmit [True]
Probes finished: 2
Probes Total: 2
Date: 2015-11-24 15:43:03
Results: [<irma.apiclient.IrmaResults object at 0x7f3f250df890>]
>>> scan = _
>>> print scan.results[0]
Status: 1
Probes finished: 2
Probes Total: 2
Scanid: ca2e8af4-0f5b-4a55-a1b8-2b8dc9ead068
Scan Date: 2015-12-22 14:36:21
Filename: eicar.com
Filepath: ./irmacl/tests/samples
ParentFile SHA256: None
Resultid: 572f9418-ca3c-4fdf-bb35-50c11629a7e7
FileInfo:
None
Results: None
>>> print scan_proberesults("572f9418-ca3c-4fdf-bb35-50c11629a7e7")
Status: 1
Probes finished: 2
Probes Total: 2
Scanid: ca2e8af4-0f5b-4a55-a1b8-2b8dc9ead068
Scan Date: 2015-12-22 14:36:21
Filename: eicar.com
Filepath: ./irmacl/tests/samples
ParentFile SHA256: None
Resultid: 572f9418-ca3c-4fdf-bb35-50c11629a7e7
FileInfo:
Size: 68
Sha1: 3395856ce81f2b7382dee72602f798b642f14140
Sha256: 275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f
Md5: 44d88612fea8a8f36de82e1278abb02fs
First Scan: 2015-11-24 14:54:12
Last Scan: 2015-12-22 14:36:21
Id: 3
Mimetype: EICAR virus test files
Tags: []
Results: [<irmacl.apiclient.IrmaProbeResult object at 0x7f3f250b9dd0>, <irmacl.apiclient.IrmaProbeResult object at 0x7f3f250b9850>]
>>> fr = _
>>> print fr.probe_results[0]
Status: 1
Name: VirusBlokAda (Console Scanner)
Category: antivirus
Version: 3.12.26.4
Duration: 1.91s
Results: EICAR-Test-File
Searching for scans
>>> scan_list()
(89, [Scanid: ef0b9466-3132-40b7-990a-415f08377f09
Status: finished
Options: Force [True] Mimetype [True] Resubmit [True]
Probes finished: 1
Probes Total: 1
Date: 2015-11-24 15:04:27
[...]
Searching for files
>>> file_search(name="ei")
(1, [<irmacl.apiclient.IrmaResults at 0x7f3f250491d0>])
>>> (total, res) = _
>>> print res[0]
Status: 1
Probes finished: 1
Probes Total: 1
Scanid: 7ae6b759-b357-4680-8358-b134b564b1ca
Filename: eicar.com
[...]
>>> file_search(hash="3395856ce81f2b7382dee72602f798b642f14140")
(7,
[<irmacl.apiclient.IrmaResults at 0x7f3f250b96d0>,
<irmacl.apiclient.IrmaResults at 0x7f3f24fdc1d0>,
<irmacl.apiclient.IrmaResults at 0x7f3f24fdca90>,
<irmacl.apiclient.IrmaResults at 0x7f3f24fdcdd0>,
<irmacl.apiclient.IrmaResults at 0x7f3f24fdc690>,
<irmacl.apiclient.IrmaResults at 0x7f3f2504f390>,
<irmacl.apiclient.IrmaResults at 0x7f3f24fea350>])
>>> file_search(hash="3395856ce81f2b7382dee72602f798b642f14140", tags=[1,2])
(0, [])
# looking for an unexisting tagid raise IrmaError
>>> file_search(hash="3395856ce81f2b7382dee72602f798b642f14140", tags=[100])
IrmaError: Error 402
Objects (apiclient.py)¶
class irmacl.apiclient.IrmaFileInfo(id, size, timestamp_first_scan, timestamp_last_scan, sha1, sha256, md5, mimetype, tags)
Bases: “object”
IrmaFileInfo Description for class
- Variables:
- id – id
- timestamp_first_scan – timestamp when file was first scanned in IRMA
- timestamp_last_scan – timestamp when file was last scanned in IRMA
- size – size in bytes
- md5 – md5 hexdigest
- sha1 – sha1 hexdigest
- sha256 – sha256 hexdigest
- mimetype – mimetype (based on python magic)
- tags – list of tags
pdate_first_scan – property, humanized date of first scan
pdate_last_scan – property, humanized date of last scan
raw()
class irmacl.apiclient.IrmaProbeResult(**kwargs)
Bases: “object”
IrmaProbeResult Description for class
- Variables:
- status – int probe specific (usually -1 is error, 0 nothing found 1 something found)
- name – probe name
- type – one of IrmaProbeType (‘antivirus’, ‘external’, ‘database’, ‘metadata’…)
- version – probe version
- duration – analysis duration in seconds
- results – probe results (could be str, list, dict)
- error – error string (only relevant in error case when status == -1)
- external_url – remote url if available (only relevant when type == ‘external’)
- database – antivirus database digest (need unformatted results) (only relevant when type == ‘antivirus’)
- platform – ‘linux’ or ‘windows’ (need unformatted results)
to_json()
class irmacl.apiclient.IrmaResults(file_infos=None, probe_results=None, **kwargs)
Bases: “object”
IrmaResults Description for class
- Variables:
- status – int (0 means clean 1 at least one AV report this file as a virus)
- probes_finished – number of finished probes analysis for current file
- probes_total – number of total probes analysis for current file
- scan_id – id of the scan
- scan_date – date of the scan
- name – file name
- path – file path (as sent during upload or resubmit)
- result_id – id of specific results for this file and this scan used to fetch probe_results through file_results helper function
- file_infos – IrmaFileInfo object
- probe_results – list of IrmaProbeResults objects
to_json()
pscan_date – property, humanized date of scan date
class irmacl.apiclient.IrmaScan(id, status, probes_finished, probes_total, date, force, resubmit_files, mimetype_filtering, results=[])
Bases: “object”
IrmaScan Description for class
- Variables:
- id – id of the scan
- status – int (one of IrmaScanStatus)
- probes_finished – number of finished probes analysis for current scan
- probes_total – number of total probes analysis for current scan
- date – scan creation date
- force – force a new analysis or not
- resubmit_files – files generated by the probes should be analyzed or not
- mimetype_filtering – probes list should be decided based on files mimetype or not
- results – list of IrmaResults objects
is_finished()
is_launched()
pdate – property, printable date
pstatus – property, printable status
class irmacl.apiclient.IrmaTag(id, text)
Bases: “object”
IrmaTag Description for class
- Variables:
- id – id of the tag
- text – tag label
Helpers (helpers.py)¶
irmacl.helpers.file_download(sha256, dest_filepath, verbose=False)
Download file identified by sha256 to dest_filepath
- Parameters:
- sha256 (str of 64 chars) – file sha256 hash value
- dest_filepath (str) – destination path
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return tuple of total files and list of results for the given file
- Return type:
- tuple(int, list of IrmaResults)
irmacl.helpers.file_results(sha256, limit=None, offset=None, verbose=False)
List all results for a given file identified by sha256
- Parameters:
- sha256 (str of 64 chars) – file sha256 hash value
- limit (int) – max number of files to receive (optional default:25)
- offset (int) – index of first result (optional default:0)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- tuple(int, list of IrmaResults)
irmacl.helpers.file_search(name=None, hash=None, tags=None, limit=None, offset=None, verbose=False)
Search a file by name or hash value
- Parameters:
- name (str) – name of the file (‘name’ will be searched)
- hash (str of (64, 40 or 32 chars)) – one of sha1, md5 or sha256 full hash value
- tags (list of int) – list of tagid
- limit (int) – max number of files to receive (optional default:25)
- offset (int) – index of first result (optional default:0)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return tuple of total files and list of matching files already scanned
- Return type:
- tuple(int, list of IrmaResults)
irmacl.helpers.file_tag_add(sha256, tagid, verbose=False)
Add a tag to a File
- Parameters:
- sha256 (str of (64 chars)) – file sha256 hash
- tagid (int) – tag id
- Returns:
- No return
irmacl.helpers.file_tag_remove(sha256, tagid, verbose=False)
Remove a tag to a File
- Parameters:
- sha256 (str of (64 chars)) – file sha256 hash
- tagid (int) – tag id
- Returns:
- No return
irmacl.helpers.probe_list(verbose=False)
List availables probes
- Parameters:
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return probe list
- Return type:
- list
irmacl.helpers.scan_add_data(scan_id, data, filename, post_max_size_M=100, verbose=False)
Add files to an existing scan
- Parameters:
- scan_id (str) – the scan id
- data (str) – data to scan
- filename (str) – filename associated to data
- post_max_size_M (int) – POST data max size in Mb (multiple calls to the api will be done if total size is more than this limit, note that if one or more file is bigger than this limit it will raise an error)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the updated scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_add_files(scan_id, filelist, post_max_size_M=100, verbose=False)
Add files to an existing scan
- Parameters:
- scan_id (str) – the scan id
- filelist (list) – list of full path qualified files
- post_max_size_M (int) – POST data max size in Mb (multiple calls to the api will be done if total size is more than this limit, note that if one or more file is bigger than this limit it will raise an error)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the updated scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_cancel(scan_id, verbose=False)
Cancel a scan
- Parameters:
- scan_id (str) – the scan id
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_data(data, filename, force, post_max_size_M=100, probe=None, mimetype_filtering=None, resubmit_files=None, blocking=False,blocking_timeout=60, verbose=False)
Wrapper around scan_new / scan_add / scan_launch
- Parameters:
- data (str) – data to scan
- filename (str) – filename associated to data
- force (bool) – if True force a new analysis of files if False use existing results
- post_max_size_M (int) – POST data max size in Mb (multiple calls to the api will be done if total size is more than this limit, note that if one or more file is bigger than this limit it will raise an error)
- probe (list) – probe list to use (optional default: None means all)
- mimetype_filtering (bool) – enable probe selection based on mimetype (optional default:True)
- resubmit_files (bool) – reanalyze files produced by probes (optional default:True)
- blocking (bool) – wether or not the function call should block until scan ended
- blocking_timeout (int) – maximum amount of time before timeout per file (only enabled while blocking is ON)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_files(filelist, force, post_max_size_M=100, probe=None, mimetype_filtering=None, resubmit_files=None, blocking=False,blocking_timeout=60, verbose=False)
Wrapper around scan_new / scan_add / scan_launch
- Parameters:
- filelist (list) – list of full path qualified files
- force (bool) – if True force a new analysis of files if False use existing results
- post_max_size_M (int) – POST data max size in Mb (multiple calls to the api will be done if total size is more than this limit, note that if one or more file is bigger than this limit it will raise an error)
- probe (list) – probe list to use (optional default: None means all)
- mimetype_filtering (bool) – enable probe selection based on mimetype (optional default:True)
- resubmit_files (bool) – reanalyze files produced by probes (optional default:True)
- blocking (bool) – wether or not the function call should block until scan ended
- blocking_timeout (int) – maximum amount of time before timeout per file (only enabled while blocking is ON)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_get(scan_id, verbose=False)
Fetch a scan (useful to track scan progress with scan.pstatus)
- Parameters:
- scan_id (str) – the scan id
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_launch(scan_id, force, probe=None, mimetype_filtering=None, resubmit_files=None, verbose=False)
Launch an existing scan
- Parameters:
- scan_id (str) – the scan id
- force (bool) – if True force a new analysis of files if False use existing results
- probe (list) – probe list to use (optional default None means all)
- mimetype_filtering (bool) – enable probe selection based on mimetype (optional default:True)
- resubmit_files (bool) – reanalyze files produced by probes (optional default:True)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the updated scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_list(limit=None, offset=None, verbose=False)
List all scans
- Parameters:
- limit (int) – max number of files to receive (optional default:25)
- offset (int) – index of first result (optional default:0)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return tuple of total scans and list of scans
- Return type:
- tuple(int, list of IrmaScan)
irmacl.helpers.scan_new(verbose=False)
Create a new scan
- Parameters:
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return the new generated scan object
- Return type:
- IrmaScan
irmacl.helpers.scan_proberesults(result_idx, formatted=True, verbose=False)
- Fetch file probe results (for a given scan
- one scan <-> one result_idx
- Parameters:
- result_idx (str) – the result id
- formatted (bool) – apply frontend formatters on results (optional default:True)
- verbose (bool) – enable verbose requests (optional default:False)
- Returns:
- return a IrmaResult object
- Return type:
- IrmaResults
irmacl.helpers.tag_list(verbose=False)
List all available tags
- Returns:
- list of existing tags
- Return type:
- list of IrmaTag
irmacl.helpers.tag_new(text, verbose=False)
Create a new tag
- Parameters:
- text (str) – tag label (utf8 encoded)
- Returns:
- None
Administration¶
Environment configuration¶
Conf VMs with choice of probes
Components configuration¶
Frontend configuration¶
Configuration¶
The configuration file is located at config/frontend.ini
in the installation
directory.
Note
Detailed meaning of each field in config/frontend.ini
:
Section | Key | Type | Default | Description |
---|---|---|---|---|
log | syslog | integer |
0 | enable rsyslog (experimental) |
prefix | string |
irma-frontend: | prefix to append to rsyslog entries | |
debug | boolean |
False | enable Debug log | |
sql_debug | boolean |
False | enable SQL debug log | |
sqldb | username | string |
database username | |
password | string |
database password | ||
host | string |
database host | ||
port | integer |
database port | ||
dbname | string |
database name | ||
tables_prefix | string |
database tables prefix | ||
samples_storage | path | string |
Samples storage path | |
celery_brain | timeout | integer |
60 (sec) | time before considering that the brain has timed-out |
celery_frontend | timeout | integer |
30 (sec) | time before considering that the frontend has timed-out |
celery_options | concurrency | integer | 0 | number of concurrent workers (0 means nb of cores) |
soft_time_limit | integer |
300 (sec) | time limit before task soft interrupt | |
time_limit | integer |
1500 (sec) | time limit before task is killed | |
beat_schedule | string |
/var/irma/fronte nd_beat_schedule | celery beat schedule file | |
broker_brain | host | string |
hostname for the RabbitMQ server | |
port | integer |
5672 | port for the RabbitMQ server | |
vhost | string |
virtual host configured for brain | ||
username | string |
username used for brain on the RabbitMQ server | ||
password | string |
password used for brain on the RabbitMQ server | ||
queue | string |
queue to poll new tasks on the RabbitMQ server | ||
broker_frontend | host | string |
hostname for the RabbitMQ server | |
port | integer |
5672 | port for the RabbitMQ server | |
vhost | string |
virtual host configured for this frontend | ||
username | string |
username used for this frontend on the RabbitMQ server | ||
password | string |
password used for this frontend on the RabbitMQ server | ||
queue | string |
queue to poll new tasks on the RabbitMQ server | ||
ftp | protocol | string |
“sftp” | choose File Transfer Protocol (“sftp” or “ftps”) |
ftp_brain | host | string |
hostname for the FTP server | |
port | integer |
22 | port for the FTP server | |
auth | string |
“password” | SFTP authentication method (“password” or “key”) | |
key_path | string |
sftp private key absolute path | ||
username | string |
username used by this frontend on the FTP server | ||
password | string |
password used by this frontend on the FTP server | ||
|
|
string |
“0” | remove file when not scanned for givent time 0 means disabled (“1 hour”, “5 days”, “3w”, “1year”) |
|
string |
0 | cron hour settings | |
clean_fs_age _cron_minute | string |
0 | cron minute settings | |
clean_fs_age _cron_day_of_week | string |
* | cron day of week settings | |
|
|
string |
“0” | space’s maximum size dedicated to the file system (“100 Mb”, “512 Mb”, “1.5Gb”) |
|
string |
* | cron hour settings | |
clean_fs_size _cron_minute | string |
0 | cron minute settings | |
clean_fs_size _cron_day_of_week | string |
* | cron day of week settings | |
interprocess _lock | path | string |
/var/run/lock/ir ma-frontend.lock | Concurrency file lock |
ssl_config | activate_ssl | boolean |
False | Enable RabbitMQ ssl |
ca_certs | string |
RabbitMQ SSL certs | ||
keyfile | string |
RabbitMQ SSL keyfile | ||
certfile | string |
RabbitMQ SSL certfile |
Note
The default path for samples is /var/irma/samples/ make sure it exists with correct rights for irma user before launching your first scan.
Brain configuration¶
Configuration¶
The configuration file is located at config/brain.ini
in the installation
directory. Update it with your specific info.
Note
Detailed meaning of each field in config/brain.ini
:
Section | Key | Type | Default | Description | |
---|---|---|---|---|---|
log | syslog | integer |
0 | enable rsyslog (experimental) | |
prefix | string |
irma-brain: | prefix to append to rsyslog entries | ||
debug | boolean |
False | enable Debug log | |||
sql_debug | boolean |
False | enable SQL debug log | |||
celery_options | concurrency | integer | 0 | number of concurrent workers (0 means nb of cores) | |
soft_time_limit | integer |
300 (sec) | time limit before task soft interrupt | ||
time_limit | integer |
1500 (sec) | time limit before task is killed | ||
broker_brain | host | string |
hostname for the RabbitMQ server | ||
port | integer |
5672 | port for the RabbitMQ server | ||
vhost | string |
virtual host configured for brain | |||
username | string |
username used for brain on the RabbitMQ server | |||
password | string |
password used for brain on the RabbitMQ server | |||
queue | string |
queue to poll new tasks on the RabbitMQ server | |||
broker_probe | host | string |
hostname for the RabbitMQ server | ||
port | integer |
5672 | port for the RabbitMQ server | ||
vhost | string |
virtual host configured for probes | |||
username | string |
username used for probes on the RabbitMQ server | |||
password | string |
password used for probes on the RabbitMQ server | |||
queue | string |
queue to poll new tasks on the RabbitMQ server | |||
broker_frontend | host | string |
hostname for the RabbitMQ server | ||
port | integer |
5672 | port for the RabbitMQ server | ||
vhost | string |
virtual host configured for frontend | |||
username | string |
username used for frontend on the RabbitMQ server | |||
password | string |
password used for frontend on the RabbitMQ server | |||
queue | string |
queue to poll new tasks on the RabbitMQ server | |||
sqldb | dbms | string |
sqlite | dbapi engine | |
dialect | string |
sqlalchemy dialect | |||
username | string |
database username | |||
password | string |
database password | |||
host | string |
database host | |||
dbname | string |
/var/irma/ db/brain.db | database name | ||
tables_prefix | string |
database tables prefix | |||
ftp | protocol | string |
“sftp” | choose File Transfer Protocol (“sftp” or “ftps”) | |
ftp_brain | host | string |
hostname for the FTP server | ||
port | integer |
21 | port for the FTP server | ||
auth | string |
“password”| SFTP authentication method (“password” or “key”) | |||
key_path | string |
sftp private key absolute path
|
|||
username | string |
username used by probe on the FTP server
|
|||
password | string |
password used by the probe on the FTP server | |||
interprocess _lock | path | string |
/var/run/ lock/irma- brain.lock | Concurrency file lock | |
ssl_config | activate_ssl | boolean |
False | Enable RabbitMQ ssl | |
ca_certs | string |
RabbitMQ SSL certs | |||
keyfile | string |
RabbitMQ SSL keyfile | |||
certfile | string |
RabbitMQ SSL certfile |
Generate a SQLite database for scan tracking¶
You could easily generate the user database by running the following command. The path of the database is taken from the configuration file and the folder where the database is going to be stored must be created beforehand.
Note
The default path for the database is /var/irma/db/ make sure it exists before creating user database.
$ cd /opt/irma/irma-brain/current/
$ ./venv/bin/python -m scripts.create_user
usage: create_user <username> <rmqvhost> <ftpuser>
with <username> a string
<rmqvhost> the rmqvhost used for the frontend
<ftpuser> the ftpuser used by the frontend
example: create_user test1 mqfrontend frontend
To create an entry in the database for the frontend named frontend
and
which uses the mqfrontend
virtual host on the RabbitMQ server, simply run
the following commands:
$ ./venv/bin/python -m scripts.create_user frontend mqfrontend frontend
Note
There is a limitation due to SQLite. The folder where the database is stored, plus the database file must be writable by the user running the worker:
$ sudo chown irma:irma /var/irma/db/brain.db
$ sudo chmod a+w /opt/irma/irma-brain
Probe configuration¶
Configuration¶
The configuration file is config/probe.ini
located in the installation
directory.
Note
We recall in the following the meaning of each field in config/probe.ini
:
Section | Key | Type | Default | Description | |
---|---|---|---|---|---|
log | syslog | integer |
0 | enable rsyslog (experimental) | |
prefix | string |
irma-probe: | prefix to append to rsyslog entries | ||
celery_options | concurrency | integer | 0 | number of concurrent workers (0 means nb of cores) | |
soft_time_limit | integer |
300 (sec) | time limit before task soft interrupt | ||
time_limit | integer |
1500 (sec) | time limit before task is killed | ||
broker probe | host | string |
hostname for the RabbitMQ server | ||
port | integer |
5672 | port for the RabbitMQ server | ||
vhost | string |
virtual host configured for probes | |||
username | string |
username used for probes on the RabbitMQ server | |||
password | string |
password used for probes on the RabbitMQ server | |||
queue | string |
queue to poll new tasks on the RabbitMQ server | |||
ftp_brain | host | string |
hostname for the FTP server | ||
port | integer |
21 | port for the FTP server | ||
auth | string |
“password”| SFTP authentication method (“password” or “key”) | |||
key_path | string |
sftp private key absolute path
|
|||
username | string |
username used by probe on the FTP server | |||
password | string |
password used by the probe on the FTP server |
SSL settings¶
SSL is available for 5 services:
- for an https connection with an nginx configuration;
- for RabbitMQ;
- for PostgreSQL with an authentication by certificate;
In the nominal case, enabling SSL for at least one of these services generates a PKI made of a root CA and, for each mechanism, an intermediate CA and some other stuff.
Every CA and https certificate requires an openssl configuration file. These files are set in the appropriate directory in ./extras/pki/conf
: root.config
at the root, a <service>/ca.config
in the https
, rabbitmq
and psql
directories and configuration file corresponding to https clients in https
directory. The configuration files are copied in the corresponding directories during their generation.
The PKI is generated in the infra directory ./infras/<infra-name>/pki
where <infra-name>
is the ansible variable infra_name
in group_vars/all.yml
(defaults to “Qb”). The PKI is described in infras/<infra-name>/<infra-name>-infra.yml
. During the provisioning, ansible updates the PKI (or creates it) according to this file. To erase the PKI, delete the infra directory first.
HTTPS¶
Enable HTTPS¶
To enable SSL on the frontend server, edit group_vars/all.yml
with:
frontend_openssl: True
nginx_https_enabled: True # require frontend_openssl
nginx_https_client_enabled: True # require nginx_https_enabled
Note
HTTPS and HTTP connections can operate at the same time.
Note
nginx_https_enabled
[required] activates the server’s certificate verification.
nginx_https_client_enabled
[optional] activates the client’s certificate verification.
Generate certificates¶
The crypto objects for an https connection are generated in infras/<infra-name>/pki/https
. By default, these are:
- a CA (key, certificate, chained certificate, database and CRL);
- a server (key, certificate, chained certificate);
- a client (key, certificate, chained certificate).
$ tree infras/Qb/pki/https
infras/Qb/pki/https/
├── ca
│ ├── 01.pem
│ ├── 02.pem
│ ├── ca-chain.crt
│ ├── ca.config
│ ├── ca.crt
│ ├── ca.key
│ ├── db
│ │ ├── ca.crl.srl
│ │ ├── ca.crl.srl.old
│ │ ├── ca.crt.srl
│ │ ├── ca.crt.srl.old
│ │ ├── ca.db
│ │ ├── ca.db.attr
│ │ ├── ca.db.attr.old
│ │ └── ca.db.old
│ └── https.crl
├── clients
│ ├── client-chain.crt
│ ├── client.config
│ ├── client.crt
│ ├── client.key
│ └── revoked
└── server
├── server-chain.crt
├── server.config
├── server.crt
└── server.key
Add a client¶
To add a client:
edit
infras/<infra-name>/<infra-name>-infra.yml
with:--- infra: name: Qb https: clients: running: - name: client - name: new_client #there we indicate a the name of the new user revoked: []
add an openssl configuration file
./extras/pki/conf/https/<client-name>.config
corresponding to the new user.provision with ansible: it copies the previous file in clients directory.
Revoke a client¶
To revoke a client:
edit
infras/<infra-name>/<infra-name>-infra.yml
with:--- infra: name: Qb clients: running: - name: client revoked: - name: bad_user # the user is now in revoked list and not in running list
provision with ansible: it revokes the user with the user’s CA and moves its stuff in
clients/revoked/
.
RabbitMQ¶
Enable SSL on RabbitMQ¶
To enable SSL in RabbitMQ, edit group_vars/brain.yml
with:
rabbitmq_ssl: True
Note
If you are updating an already running no_ssl version, do the following on irma-brain RabbitMQ server:
$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudo rabbitmqctl start_app
# create again the RabbitMQ vhosts, usernames and passwords:
$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe probe mqprobe
$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain brain mqbrain
$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh frontend frontend mqfrontend
Certificates generation¶
The crypto objects for RabbitMQ with SSL are generated in infras/<infra-name>/pki/rabbitmq
. These are:
- a CA (key, certificate, chained certificate and database);
- a server brain (key, certificate);
- 3 clients for the entities frontend, brain and probe (key, certificate).
$ tree infras/Qb/pki/rabbitmq
infras/Qb/pki/rabbitmq/
├── ca
│ ├── 01.pem
│ ├── 02.pem
│ ├── 03.pem
│ ├── 04.pem
│ ├── ca-chain.crt
│ ├── ca.config
│ ├── ca.crt
│ ├── ca.key
│ └── db
│ ├── ca.crt.srl
│ ├── ca.crt.srl.old
│ ├── ca.db
│ ├── ca.db.attr
│ ├── ca.db.attr.old
│ └── ca.db.old
├── clients
│ ├── brain-client.crt
│ ├── brain-client.key
│ ├── frontend-client.crt
│ ├── frontend-client.key
│ ├── probe-client.crt
│ └── probe-client.key
└── server
├── brain.crt
└── brain.key
Note
In RabbitMQ case, only the CA needs a openssl configuration file.
Postgresql¶
Enable SSL on Postgresql¶
To activate SSL in PostgreSQL service, edit group_vars/brain.yml
with:
postgresql_ssl: True
Generate certificates¶
The crypto objects for PostgreSQL with SSL are generated in infras/<infra-name>/pki/psql
. These are:
- a CA (key, certificate, chained certificate, a CRL and database);
- a server (key, certificate);
- a client frontend (key, certificate).
$ tree infras/Qb/pki/psql
infras/Qb/pki/psql/
├── ca
│ ├── 01.pem
│ ├── 02.pem
│ ├── ca-chain.crt
│ ├── ca.config
│ ├── ca.crt
│ ├── ca.key
│ ├── db
│ │ ├── ca.crl.srl
│ │ ├── ca.crl.srl.old
│ │ ├── ca.crt.srl
│ │ ├── ca.crt.srl.old
│ │ ├── ca.db
│ │ ├── ca.db.attr
│ │ ├── ca.db.attr.old
│ │ └── ca.db.old
│ └── psql.crl
├── clients
│ ├── frontend.config
│ ├── frontend.crt
│ ├── frontend.key
│ └── revoked
└── server
├── server.config
├── server.crt
└── server.key
Revoke a client¶
To revoke a client:
edit
infras/<infra-name>/<infra-name>-infra.yml
with:--- infra: name: Qb psql: clients: revoked: - name: bad_user # bad_user is now in revoked list and no longer in running list
provision with ansible: it revokes the user with the user’s CA and moves its stuff in
clients/revoked/
.
External PKI¶
It is also possible to use an external PKI for one or more of these services, for the root entity or the whole Irma’s PKI. In this case, it is necessary to provide the corresponding cryptographic objects in PEM format.
To specify which PKI’s part are provided by an external PKI, edit group_vars/all.yml
:
root_external: False
pki_rabbitmq_external: False
pki_https_external: False
pki_psql_external: False
By default, the automatic generation of the whole PKI is activated and all variables for external PKI are set to False.
External root¶
To use a external root, edit group_vars/all.yml
with:
root_external: True
root_external_key: root_key.key
root_external_cert: root_cert.crt
Note
root_key.key
and root_external_cert
must contain the paths to respectively the key and the certificate of the external root entity.
The Irma’s PKI will be generated with this external root as authority.
External HTTPS PKI¶
To use an external PKI for HTTPS and disable the automatic generation of a new one, edit group_vars/all.yml
with:
pki_https_external: True
Provide the cryptographic objects and specify the paths editing group_vars/frontend.yml
:
frontend_openssl_certificates:
cert:
src: https_server.crt
dst: /etc/nginx/certs/{{ hostname }}.crt
key:
src: https_server.key
dst: /etc/nginx/certs/{{ hostname }}.key
ca:
src: https_ca_cert.crt
dst: /etc/nginx/certs/ca.crt
chain:
src: https_ca_chain.crt
dst: /etc/nginx/certs/ca-chain.crt
crl:
src: https_crl.crl
dst: /etc/nginx/certs/https.crl
Note
frontend_openssl_certificates.cert.src
is the path to the server’s certificate
frontend_openssl_certificates.key.src
is the path to the server’s private key
frontend_openssl_certificates.ca.src
is the path to the CA’s certificate
frontend_openssl_certificates.chain.src
is the path to the CA’s certification chain
frontend_openssl_certificates.crl.src
is the path to the CRL
External RabbitMQ PKI¶
To use an external PKI for RabbitMQ and disable the automatic generation of a new one, edit group_vars/all.yml
with:
pki_rabbitmq_external: True
Provide the cryptographic objects and specify the paths editing group_vars/all.yml
:
rabbitmq_cacert : ca-chain.crt
rabbitmq_server_key : server.key
rabbitmq_server_cert: server.crt
rabbitmq_frontend_key: frontend-client.key
rabbitmq_frontend_cert: frontend-client.crt
rabbitmq_brain_key: brain-client.key
rabbitmq_brain_cert: brain-client.crt
rabbitmq_probe_key: probe-client.key
rabbitmq_probe_cert: probe-client.crt
Note
rabbitmq_cacert
is the path to the CA’s certification chain
rabbitmq_server_key
is the path to the server’s private key
rabbitmq_server_cert
is the path to the server’s certificate
rabbitmq_frontend_key
is the path to the frontend’s private key
rabbitmq_fontend_cert
is the path to the frontend’s certificate
rabbitmq_brain_key
is the path to the brain’s private key
rabbitmq_brain_cert
is the path to the brain’s certificate
rabbitmq_probe_key
is the path to the probes’ private key
rabbitmq_probe_cert
is the path to the probes’ certificate
External PostgreSQL PKI¶
To use an external PKI for PostgreSQL and disable the automatic generation of a new one, edit group_vars/all.yml
with:
pki_psql_external: True
Provide the cryptographic objects and specify the paths editing group_vars/sql-server.yml
:
postgresql_ssl_cert_src_path: server.crt
postgresql_ssl_key_src_path: server.key
postgresql_ssl_ca_src_path: ca-chain.crt
postgresql_ssl_crl_src_path: psql.crl
Note
postgresql_ssl_cert_src_path
is the path to the server’s certificate
postgresql_ssl_key_src_path
is the path to the server’s private key
postgresql_ssl_ca_src_path
is the path to the CA’s certificate chain
postgresql_ssl_crl_src_path
is the path to the CRL
Database migration¶
IRMA uses Alembic to manage and perform databases migration.
Note
Alembic is a useful tool to manage migration, but can’t surpass local engine implementation
of SQL. As SQLite
doesn’t manage schema modifications such as ALTER_COLUMN
, the
whole migration system of IRMA won’t support it. The preferred database engine is
PostgreSQL
.
You can still use SQLite, but you will be on your own for migrations.
Warning
Please note that most of the manipulations on this can and sometimes will alter your data. If you are not sure about what you are doing, and even if you are sure, make backup.
Requirements¶
Content¶
Database migrations are managed in the frontend and brain IRMA components.
The files/directories used are:
alembic.ini
extras/migration/
+- env.py
+- script.py.mako
+- versions/
+- <revision_1>.py
+- <revision_2>.py
+- ...
Note
All the commands below will assert to be executed on top of this file system,
as Alembic needs the alembic.ini
configuration file.
You could also use the -c <path_to_conf_file>
.
Usage¶
Alembic manage a ‘revision’ for each database evolution. These revisions are used to upgrade or downgrade the database schema.
The command:
$ alembic current
… shows the current revision of the database.
The command to get the history of the latest alembic migrations is:
$ alembic history --verbose
Create database from scratch with Alembic¶
Configuration and creating database¶
Alembic will use the information in the [sqldb]
section of the configuration files
(respectively config/frontend.ini
or conf/brain.ini
for the repositories of
the frontend or the brain components). Make sure they are accurate.
The database must already exist. This step is quite simple, the SQL command usually being:
sql$ CREATE DATABASE <db_name>;
Update your schema with Alembic¶
If you use a virtualenv, activate it. Then enter:
$ alembic upgrade head
Alembic applies each revision one after the other. At the end of the process, if no error occurs, your database should be updated.
Note
You can update the database one revision at a time, or up to a specific revision. See the revisions section for further information.
If you already have a database WITHOUT Alembic¶
Alembic stores its current revision number in database. If your database doesn’t have this information, you are very likely to encounter errors when using Alembic, as it will try to create already existing tables.
The easiest solution is to destroy your database and go for a fresh install.
Although, if you don’t want to lose your data, you could update the Alembic information manually.
You will need to:
Get the exact current Alembic revision of your database. Each migration file has a
Revision ID
in its header. Investigate the successive revisions to know which one matches your current database state.Once you known your Alembic revision, run:
$ alembic stamp <your_alembic_revision_number>
Your database is now synchronized with Alembic! You should be able to use Alembic to upgrade/downgrade your database now. Be aware that if the revision number you provided is false, you could encounter massive errors while attempting to upgrade/downgrade your database.
Generating a new revision¶
Creating a new revision can be done with the command:
$ alembic revision -m <revision_message>
This command produces a new <hash>_<revision_message>.py
file in the extras/migration/versions/
directory. This file contains two functions upgrade
and downgrade
, respectively used
to upgrade the database to the revision, or downgrade from it. These two functions are empty
and must be completed with the desired modifications (see the
alembic documentation section ops).
A revision could be produced automatically, from database metadata defined in the IRMA SQL
objects description through sqlalchemy
, with the command:
$ alembic revision --autogenerate -m <revision_message>
These SQL objects are defined in:
frontend/models/sqlobjects.py
for the frontend,brain/models/sqlobjects.py
for the brain.
Alembic scripts in IRMA repositories are already configured to use metadata defined in these
files. You should be able to use the --autogenerate
option without further modifications.
Note
IRMA configuration allows to prefix table names through configuration. Our revision files
use the function <frontend_or_brain>/config/parser.py:prefix_table_name
to generate table
names rather than keeping alembic-generated plain string names. A good practice would be
to keep using this function in revision files.
Warning
Alembic easily detects changes such as adding/removing columns, but could be blind on thin, inner modifications. Re-reading the auto-generated script is a strongly recommended step before actually performing the migration.
See the alembic documentation section autogenerate for more information.
Warning
Database modifications using ALTER_COLUMN
(such as changing the type of
a column) can’t be performed on SQLite
databases. Be aware of this
limitation if you absolutely want to use migration scripts with this SQL
engine.
Migrating between revisions¶
Once the revision is properly described, the migration is performed with:
$ alembic upgrade head
Alembic allows to migrate the database to any revision, relatively to the current revision or absolutely. Several examples:
$ alembic upgrade +4
$ alembic downgrade base
$ alembic upgrade <revision_number>+3
Tips and tricks¶
Note
Don’t trust Alembic too much. It is nothing more than a tool, without any comprehension on the code. Cautiously read the revision scripts it generates.
Note
Database migration is hardly ever a painless step. Be sure to:
- save your data before performing a migration,
- test your application after the migration to ensure its compatibility with the new data schemes.
Note
With a PostgreSQL
database, the Float
type is tolerated but the real type name used
by the database is Real
. It means that SQL objects described in sqlalchemy
with
Float
columns will be properly applied in database, but at each autogenerate revision,
alembic
will see Real
type in database, against Float
type in the code metadata,
and so will perform each time a useless alter_column
from Real
to Float
.
This problem could be avoided (with PostgreSQL
) by declaring Real
instead of Float
.
See this page for more
information on PostgreSQL
numeric types.
Note
Alembic can’t directly deal with many somehow complex operations, such as type migration with no trivial cast. In these cases, the operation must be manually described with a raw SQL command (which could be database-dependent).
For instance, alembic can’t perform the migration from
real
todatetime
:> alembic.alter_column('table', 'column', existing_type=sqlalchemy.REAL(), type_=sqlalchemy.DateTime(), existing_nullable=False)
… because of an error a column "column" cannot be cast automatically to type timestamp
with time zone
.
A proper migration for PostgreSQL
would be (in Python
):
> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE TIMESTAMP WITHOUT TIME ZONE USING to_timestamp(column)')
And the reverse code to downgrade the migration could be:
> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE REAL USING extract(epoch from column)')
Note
Rather than managing migrations directly with Alembic, we could generate SQL migration revision to be used directly on database with the command:
$ alembic upgrade <revision> --sql > migration.sql
Note
Deleting a revision R is simple:
- downgrade the database to the revision before R-1 the revision you want to delete;
- if any, edit the script of the following revision R+1 and update the
down_revision
variable to match the revision number of revision R-1; - delete the script of the revision R you want to delete;
- upgrade your database.
The deleted revision want be applied any more.
Technical description¶
Each major component of the IRMA platform comes with their own python-based application. As the Brain is the nerve center of the whole platform, it is recommended to install it first before installing other components. One can then install either the Frontend or the Probes he wants.
The IRMA entrypoint is the web API hosted on frontend. File results are stored in PostgreSQL database. All files transfers are done through FTP (sftp server on brain). All tasks are executed by celery applications that consumes their own task queue on RabbitMQ server. For further details give a look at scan workflow part
API documentation¶
There is a dynamic documentation for IRMA API available on your instance
It allows you to read documentation but also try request and see server response.
You could see detailed information about one specific API route:
and by clicking on the Try it button, see the server response:
Frontend¶
The Frontend handles scan submission to the Brain, stores the results of the scanned files. These results can be displayed through a web graphical user interface or via the command line interface.
Installation¶
The Frontend must be installed on a GNU/Linux system. With some efforts, it should be possible to run it on a Microsoft Windows system, but this has not been tested yet.
This section describes how to get the source code of the application and to install it.
Architecture¶
Let us recall first the inner architecture of the Frontend. It uses multiple technologies with each a specific purpose:
- A client through which a user submits a file and get the analysis results. There are two clients bundled in the repository: a web user interface and a command-line client.
- A python-based restful API, served by a NGINX web server and a uWSGI application server. It gets the results of a file scan by querying a database.
- A worker that will handle scan submission to the Brain and store the results of analyzes scheduled by the Brain. The worker relies on Celery, a python-based distributed task queue.
- A database server (PostgreSQL) is used to store results of analyzes made on each file submitted either by the web graphical interface or the CLI client.
Brain¶
The Brain is a python-based application that only dispatches analysis requests from different frontends [1] to the available Probes. Analyses are scheduled by the Brain on Probes through Celery, an open source task.
Footnotes
[1] | This feature is not ready yet, we are currently working on its implementation. |
Installation¶
The Brain must be installed on a GNU/Linux distribution. With some efforts, it should be possible to run it on a Microsoft Windows system, but this has not been tested yet.
This section describes how to get the source code of the application for the Brain and to install it.
Architecture¶
Let us recall first the inner architecture of the Brain. It uses multiple technologies with a specific purpose each:
- a Celery worker that handles scan requests from Frontends and results returned by the Probes.
- a RabbitMQ server used by Celery as a backend and as a broker for task queues and job queues used to schedule tasks for Probes (for scan jobs) and the Frontend (for scan results).
- an SFTP server where files to be scanned are uploaded by Frontends and downloaded by Probes,
Nginx¶
In the Frontend, we use a nginx web server to serve the uWSGI application and the static web site that query the API in order to get results of scanned files and to present them to the user.
SQL server¶
The Frontend relies on a PostgreSQL database to keep track of all scans info.
Probe¶
The Probes are python-based application that host a single or multiple analyzers. Each analyzer listens on a specific work queue and waits for an analysis to be scheduled by the Brain through Celery, an open source task framework for Python. Python version should be at least 3.4 on linux, 3.5 on windows.
Architecture¶
Probes are mainly Celery workers that handle scan requests from Brain
Scan workflow¶
Frontend API Part (frontend_api/uwsgi+hug)¶
- A new scan object is created in PostgreSQL database.
- Files are uploaded to the WEB API, stored on Filesystem and registered in PostgreSQL database.
- Scan is launched, an asynchronous task is launched on Frontend celery.
Frontend Celery Part (frontend_app/celery)¶
- Used probes are filtered according to scan options (selected probes, mimetype filtering).
- Empty results are created in PostgreSQL database (one per probe per file).
- Each file is uploaded to SFTP server.
- For each file uploaded a scan task on Brain is launched with the file probelist (according to scan option force some results could already be present).
Brain Celery Part (scan_app/celery)¶
- A new scan object is created in SQLite database to track jobs (for canceling).
- Each file is send for analysis in sent for analysis in every probe selected (each time a probe is available in IRMA, it registers itself to the brain and open a RabbitMQ Queue named with its probe name, probe list is retrieved by listing active queues).
- Two callbacks are set on every probe scan tasks, one for success and the other for failure.
Probe Celery Part (probe_app/celery)¶
- Scan task is received with a file id.
- File is downloaded as temporary file.
- File is scanned by the probe.
- Results are sent back to Brain to one of the two callbacks set.
Brain Celery Part (result_app/celery)¶
- successful results are marked as completed in SQLite database.
- successful results are forwarded to Frontend.
- error are marked as completed in SQLite database.
- As there is no result, an error message is generated to tell the Frontend the particular job for the file and probe failed.
Frontend Celery Part (frontend_app/celery)¶
- Results is received for each file and probe.
- Results are updated in PostgreSQL database.
- If scan is finished, a scan flush task is launched on Brain to delete files on SFTP server.
Functional Testing¶
Only available on *dev* environments
On the frontend, to launch the functionals tests:
$ cd /opt/irma/irma-frontend/current/web
$ npm run functional-tests
It will launch the Javascript implementation of Cucumber. Cucumber.js will take a file that contain test scenarios, written using the Gherkin language). For each steps of a scenario, an action is perform by Cucumber.js, like accessing a web page and typing some texts in a form. In order to do that, IRMA project uses Puppeteer, a software that can launch and control a Chromium instance in headless mode.
IRMA scenarios can be found in frontend/web/tests/functionals/*.feature
and
actions used to perform these steps are available in the file
frontend/web/tests/functionals/support/steps.js
.
When an error occured, you will get a screenshot of the page where the scenario
ends. It will be available on the VM at frontend/web/error.jpeg
.
Debug¶
Using the headless mode of Chromium, it will be difficult to debug if an error occured.
- You can launch the test using a real Chromium instance on your host:
Install IRMA web interface devDependencies on your host
$ cd frontend/web $ npm install --only=dev
Update the
ROOT_URL
(see:frontend/web/tests/functionals/support/steps.js
) variable to the location of your IRMA web url (for example:const ROOT_URL="http://172.16.1.30"
) and toggle theHEADLESS
variable tofalse
(see:frontend/web/tests/functionals/support/hooks.js
)Run the tests:
$ npm run functional-tests
You can also use the power of X11 Forwarding through SSH to see a real browser launching the tests on the VM and getting the result on the host, without having to install NodeJS:
$ vagrant ssh # to connect as vagrant user/superuser
$ sudo apt-get install xorg # to install a X11 server to launch Chromium
$ sudo sed -i "s/^X11Forwarding .*/X11Forwarding yes/" /etc/ssh/sshd_config
$ sudo systemctl restart sshd
$ exit # disconnect from vagrant user
$ vagrant ssh -- -l deploy -X # to connect as deploy with XForwarding enable
$ cd /opt/irma/irma-frontend/current/web
$ sed -i "s/^const HEADLESS = true;/const HEADLESS = false;/" tests/functionals/support/hooks.js
$ npm run functional-tests
You should see an instance of a Chromium browser on your host machine, running the tests.
Take a look at the argument pass to the puppeteer.launch()
function in
frontend/web/tests/functionals/support/hooks.js
. For example, by modifying
the SLOW_MOTION_DELAY
you can force Puppeteer to slow down its operations.
Extending IRMA¶
Adding a new probe¶
Writing a Plugin for the probe¶
Note
To be a valid probe module, IRMA expects it to have a predefined structure. To save time, one can get a minimal working structure from the skeleton plugin. The new plugin is stored in the appropriate sub-directory of the directory probe/modules according to the type of the new probe (antivirus, metadata, external…).
For a probe that is not a antivirus¶
1. Copy the directory skeleton to the new module (appropriate localisation). Example with a module my_module with metadata type :
$ cp -r probe/modules/custom/skeleton/ probe/modules/metadata/my_module
- If there are packages to install, specify them in the file requirements.txt. Otherwise remove the file
- Adjust the file plugin.py according to the module :
- Adjust the class’s name with the name of your probe
- Fill in the fields of the class :- _plugin_name_ = [the plugin name]
- _plugin_display_name_ = [the field _name of the class of the probe]
- _plugin_version_ = [the version number]
- _plugin_category = [the type of the probe: IrmaProbeType.]
- _plugin_description = [quick description]
- _plugin_dependencies = [list of dependencies: platform, binary or/and file] => if used import from lib.plugins PlatformDependency, BinaryDependency or/and FileDependency
- _mimetype_regexp = [mimetype corresponding]
- Implement the functions corresponding to the type of the plugin
For an antivirus¶
In the case of an antivirus, it is a little different because an Antivirus class was created to avoid code’s duplication. You can use the skeleton below:
plugin.py:
#
# Copyright (c) 2013-2018 Quarkslab.
# This file is part of IRMA project.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License in the top-level directory
# of this distribution and at:
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# No part of the project, including this file, may be copied,
# modified, propagated, or distributed except according to the
# terms contained in the LICENSE file.
from .skeleton import Skeleton
from ..interface import AntivirusPluginInterface
from irma.common.plugins import PluginMetaClass
class SkeletonPlugin(AntivirusPluginInterface, metaclass=PluginMetaClass):
# =================
# plugin metadata
# =================
_plugin_name_ = "Skeleton"
_plugin_display_name_ = Skeleton._name
_plugin_author_ = "IRMA (c) Quarkslab"
_plugin_version_ = "1.0.0"
_plugin_category_ = "custom"
_plugin_description_ = "Plugin skeleton"
_plugin_dependencies_ = []
_mimetype_regexp = None
# ================
# interface data
# ================
module_cls = Skeleton
# If needed, overload the `verify` classmethod in order to check your class
# is instanciable. It should return if everything is alright, otherwise
# raise an exception. By default it checks that the module's attribute
# `self.scan_path` is an existing file (cf. `super()._chk_scanpath`)
#
# @classmethod
# def verify(cls):
# pass
The metaclass PluginMetaClass
handles the registering of the plugin to a plugin manager. It also checks that the class is instanciable thanks to the verify
method.
skeleton.py:
#
# Copyright (c) 2013-2018 Quarkslab.
# This file is part of IRMA project.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License in the top-level directory
# of this distribution and at:
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# No part of the project, including this file, may be copied,
# modified, propagated, or distributed except according to the
# terms contained in the LICENSE file.
import logging
# Choose the class you need to inherit from
from modules.antivirus.base import AntivirusUnix, AntivirusWindows
log = logging.getLogger(__name__)
# Inhererit from AntivirusUnix or AntivirusWindows according to your plateform
class Skeleton(Antivirus):
name = "Skeleton for Antivirus"
# ==================================
# Constructor and destructor stuff
# ==================================
def __init__(self, *args, **kwargs):
# class super class constructor
super().__init__(*args, **kwargs)
# do your initialization stuff
The recipe is the same, the files with the corresponding module name and differents fields need to be updated.
The attributes in Antivirus._attributes
are meant to be defined by the instanciation. One can either:
- leave it blank, in this case the super class will assign it a default value (eg.
"unavailable"
forself.version
);- define it directly (eg.
self.scan_path = Path("/opt/skeleton/skeleton")
);- define a function to be called to assign it (eg.
def get_scan_path(self): ...
), the super class will take care of calling it and handling exceptions.
Testing the new plugin¶
Before testing, module’s necessary stuff (binaries, files, etc) must be provisioned to the VM.
$ cd ansible
$ vagrant rsync
$ vagrant ssh
$ sudo su deploy
$ cd /opt/irma/irma-probe/current
$ venv/bin/python -m extras.tools.run_module
This last command lists available modules.
Now, if the new module is available, its launching can be done:
$ venv/bin/python -m extras.tools.run_module my_module file
Automatic provisioning¶
Creating a new role¶
Create a new directory with this structure:
cd ansible
tree roles/quarsklab.my_module
roles/quarkslab.my_module/
+-- defaults
| +-- main.yml
+-- tasks
+-- main.yml
tasks/main.yml is the default entry point for a role containing Ansible tasks. In this file, write the instruction to install the module. Add the file tasks/update.yml to write the informations for the update if necessary. In defaults/main.yml it is usual to store default variables for this role. If there are particular instructions, for example how to obtain a licence for a antivirus, add a README file.
Invoking the module role¶
Modify playbooks/provisioning.yml : add the module
-name : my_module
hosts: my_module
roles:
- { role: quarkslab.my_module, tags: 'my_module'}
If a task update was defined, add the module in playbooks/updating.yml :
-name : my_module
hosts: my_module
roles:
- { role: quarkslab.module, tags: 'my_module', task_from : update}
Defining hosts¶
Modify the environment to add the new probe.
For example for the allinone_dev :
$ cat environments/allinone_dev.yml
[ ... snip ... ]
virustotal:
- brain.irma
my_module:
- brain.irma
"probe:children":
- clamav
- comodo
- mcafee
- static-analyzer
- virustotal
- my_module
Troubleshooting¶
Check Celery configuration¶
Celery Workers¶
Before going further, you should check that the python applications manages to communicate with the RabbitMQ server through Celery. To ensure that, from the installation directory, execute both Celery workers:
On GNU/Linux:
$ cd /opt/irma/irma-brain/current
$ ./venv/bin/python -m brain.scan_tasks
-------------- celery@brain v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: scantasks:0x7fbd7ee4c350
- ** ---------- .> transport: amqp://brain:**@127.0.0.1:5672/mqbrain
- ** ---------- .> results: amqp://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> brain exchange=celery(direct) key=brain
[2016-07-15 15:00:36,155: WARNING/MainProcess] celery@brain ready.
This worker is responsible for splitting the whole scan job in multiples job per probe per file.
$ cd /opt/irma/irma-brain/current
$ ./venv/bin/python -m brain.results_tasks
-------------- celery@brain v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: resultstasks:0x7fa68f9aa590
- ** ---------- .> transport: amqp://probe:**@127.0.0.1:5672/mqprobe
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> results exchange=celery(direct) key=results
[2016-07-15 14:59:01,799: WARNING/MainProcess] celery@brain ready.
And this worker is responsible for collecting and tracking results.
If your Celery worker does not output something similar to the above output, you should check twice the parameters in the application configuration file you are using.
Verifying RabbitMQ configuration¶
We can verify that the RabbitMQ server has taken into account our modifications with some commands:
Checking for vhosts¶
$ sudo rabbitmqctl list_vhosts
Listing vhosts ...
mqbrain
/
mqfrontend
mqprobe
mqadmin
...done.
If the defined virtual host are not listed by the above command, please execute once more the script.
Checking for users¶
$ sudo rabbitmqctl list_users
Listing users ...
probe []
brain []
frontend []
...done.
If the defined users are not listed by the above command, please execute once more the script.
Changing password¶
If you do not remember the password you just typed, you can change it with
rabbitmqctl
command:
$ sudo rabbitmqctl change_password brain brain-rmq-password
Changing password for user "brain" ...
...done.
Restarting the service¶
You may want to restart the service. Thus, the following command can be done:
$ sudo invoke-rc.d rabbitmq-server restart
Check SFTP accounts¶
Try to login as frontend
and upload a sample file in home dir (should raise an error as
it is non writeable) then in uploads
dir.
$ sftp frontend@localhost
frontend@localhost's password:
Connected to localhost.
sftp> put test
Uploading test to /test
remote open("/test"): Permission denied
sftp> ls
uploads
sftp> cd uploads/
sftp> put test
Uploading test to /uploads/test
test 100% 10 0.0KB/s 00:00
FTP-TLS accounts¶
Additionnally, if you have configured IRMA to use FTP-TLS, you can check
whether the configured account is valid. On Debian, this can be done with the
ftp-ssl
package:
$ sudo apt-get install ftp-ssl
[...]
$ ftp-ssl <hostname of the brain>
Connected to brain.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 1 of 50 allowed.
220-Local time is now 18:55. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (brain:root): frontend-ftp
500 This security scheme is not implemented
234 AUTH TLS OK.
[SSL Cipher DHE-RSA-AES256-GCM-SHA384]
200 PBSZ=0
200 Data protection level set to "private"
331 User probe OK. Password required
Password: frontend-ftp-password
230 OK. Current directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Restful API¶
One can verify that the restful API is up and running by querying a specific route on the web server or by checking the system logs:
$ curl http://localhost/api/v1.1/probes
{"total": 9, "data": ["ClamAV", "ComodoCAVL", "EsetNod32", "FProt", "Kaspersky", "McAfeeVSCL", "NSRL", "StaticAnalyzer", "VirusTotal"]}
$ sudo cat /var/log/supervisor/frontend_api.log
[...]
added /opt/irma/irma-frontend/current/venv/ to pythonpath.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 3943)
spawned uWSGI worker 1 (pid: 3944, cores: 1)
spawned uWSGI worker 2 (pid: 3945, cores: 1)
spawned uWSGI worker 3 (pid: 3946, cores: 1)
spawned uWSGI worker 4 (pid: 3947, cores: 1)
mounting frontend/api/base.py on /api
mounting frontend/api/base.py on /api
mounting frontend/api/base.py on /api
mounting frontend/api/base.py on /api
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3945 (default app)
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3946 (default app)
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3944 (default app)
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3947 (default app)
Logs¶
How to debug¶
Collect debug files¶
An Ansible playbook is available in order to gather logs and other useful files.
The playbook is ansible/playbooks/collect_debug.yml
and it will allow you
to retrieve on each host:
- IRMA Files (located on the multiples hosts);
- Systemd logs;
- Application logs (Nginx, RabbitMQ, PostgreSQL).
After running the playbook, all the files are available in the directory
specified in the debug_directory
variable of the playbook. The files are
store in directories named after the host they where retrieve from
(<debug_directory>/<host_name>/<debug_files_or_directory>
).
Most of the files are plain text but Systemd logs are using a binary format.
To explore and read them, you’ll need the journalctl
command, for example:
$ journalctl -D debug/brain.irma/var/log/journal
Switch debug log on¶
Configuration file for frontend, brain and probe is located by default in the config
folder and
is named respectively frontend.ini
, brain.ini
and probe.ini
.
To turn on debug log just add the following line:
[log]
syslog = 0
debug = 1
and restart all related applications.
To turn on SQL debug log (warning: its verbose) just add the following line:
[log]
syslog = 0
debug = 1
sql_debug = 1
and restart all related applications.
Debug a probe¶
Open a session on the probe machine and change directory to the irma-probe location. Try the run_module tool on a file to see what analyzer is detected and what is its output on a file.
$ sudo su deploy
$ cd /opt/irma/irma-probe/current
$ ./venv/bin/python -m extras.tools.run_module
[...]
usage: run_module.py [-h] [-v]
{Unarchive,StaticAnalyzer,ClamAV,VirusTotal} filename
[filename ...]
run_module.py: error: too few arguments
Here 4 probes are automatically detected. Now try one on a file:
$ ./venv/bin/python -m extras.tools.run_module ClamAV requirements.txt
{'database': {'/var/lib/clamav/bytecode.cvd': {'ctime': 1458640823.285298,
'mtime': 1458640823.069295,
'sha256': '82972e6cc5f1204829dba913cb1a0b5f8152eb73d3407f6b86cf388626cff1a1'},
'/var/lib/clamav/daily.cvd': {'ctime': 1458640822.8932924,
'mtime': 1458640822.6692889,
'sha256': '9804c9b9aaf983f85b4f13a7053f98eb7cca5a5a88d3897d49b22182b228885f'},
'/var/lib/clamav/main.cvd': {'ctime': 1458640821.6972747,
'mtime': 1458640813.9771628,
'sha256': '4a8dfbc4c44704186ad29b5a3f8bdb6674b679cecdf83b156dd1c650129b56f2'}},
'duration': 0.0045299530029296875,
'error': None,
'name': 'Clam AntiVirus Scanner',
'platform': 'linux2',
'results': None,
'status': 0,
'type': 'antivirus',
'version': '0.99'}
And check the output.
Debug Ansible Provisioning¶
To debug errors while provisioning (same goes with deployment) with following typical command:
$ ansible-playbook --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml
Example output:
TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the
full variable syntax ('{{rabbitmq_users_definitions}}').
This feature will be removed in a future release. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user': u'brain'}) => {"failed": true, "item": {"password": "brain", "user": "brain", "vhost": "mqbrain"}, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 302, in <module>\r\n main()\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 274, in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 155, in get\r\n users = self._exec(['list_users'], True)\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 150, in _exec\r\n rc, out, err = self.module.run_command(cmd + args, check_rc=True)\r\n File \"/tmp/ansible_wKXoO5/ansible_modlib.zip/ansible/module_utils/basic.py\", line 1993, in run_command\r\n File \"/usr/lib/python2.7/posixpath.py\", line 261, in expanduser\r\n if not path.startswith('~'):\r\nAttributeError: 'list' object has no attribute 'startswith'\r\n", "msg": "MODULE FAILURE", "parsed": false}
You could first increase ansible verbosity by adding -vvv
option (-vvvv
on windows for winrm debug), it will help is the problem is linked to arguments.
$ ansible-playbook -vvv --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml
TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************
task path: /home/alex/repo/irma-ansible/roles/Mayeu.RabbitMQ/tasks/vhost.yml:13
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full
variable syntax ('{{rabbitmq_users_definitions}}').
This feature will be removed in a future release. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant
<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o 'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1468570550.09-211613386938202 `" && echo ansible-tmp-1468570550.09-211613386938202="` echo $HOME/.ansible/tmp/ansible-tmp-1468570550.09-211613386938202 `" ) && sleep 0'"'"''
<127.0.0.1> PUT /tmp/tmpiysJ6l TO /home/vagrant/.ansible/tmp/ansible-tmp-1468570550.09-211613386938202/rabbitmq_user
<127.0.0.1> SSH: EXEC sftp -b - -C -o ForwardAgent=yes -o Port=2222 -o 'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 '[127.0.0.1]'
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant
<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o 'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 -tt 127.0.0.1 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rbeeckncuxenewcwkayivqiwvarchlrd; LANG=fr_FR.UTF-8 LC_ALL=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1468570550.09-211613386938202/rabbitmq_user; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1468570550.09-211613386938202/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''
failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user': u'brain'}) => {"failed": true, "invocation": {"module_name": "rabbitmq_user"}, "item": {"password": "brain", "user": "brain", "vhost": "mqbrain"}, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 302, in <module>\r\n main()\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 274, in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 155, in get\r\n users = self._exec(['list_users'], True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 150, in _exec\r\n rc, out, err = self.module.run_command(cmd + args, check_rc=True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_modlib.zip/ansible/module_utils/basic.py\", line 1993, in run_command\r\n File \"/usr/lib/python2.7/posixpath.py\", line 261, in expanduser\r\n if not path.startswith('~'):\r\nAttributeError: 'list' object has no attribute 'startswith'\r\n", "msg": "MODULE FAILURE", "parsed": false}
In this particular case, verbose doesn’t add much information as the problem is linked to ansible scripts. Let’s go one level deeper so. Ansible output the temporary script executed on guest (highlighted in previous code block) but delete it just after execution. To further debug it we will set ansible to keep remote files and the debug session will now takes place inside the guest.
$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -vvv --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml
in debug log get the temporary ansible path to remote script:
/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/rabbitmq_user
Log in to remote machine and go to the temporary ansible dir. Explode the compressed script and run it locallly:
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls
rabbitmq_user
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python rabbitmq_user explode
Module expanded into:
/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls debug_dir/
ansible
ansible_module_rabbitmq_user.py
args
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python rabbitmq_user execute
Traceback (most recent call last):
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir/ansible_module_rabbitmq_user.py", line 302, in <module>
main()
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir/ansible_module_rabbitmq_user.py", line 274, in main
if rabbitmq_user.get():
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir/ansible_module_rabbitmq_user.py", line 155, in get
users = self._exec(['list_users'], True)
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir/ansible_module_rabbitmq_user.py", line 150, in _exec
rc, out, err = self.module.run_command(cmd + args, check_rc=True)
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir/ansible/module_utils/basic.py", line 1993, in run_command
args = [ os.path.expandvars(os.path.expanduser(x)) for x in args if x is not None ]
File "/usr/lib/python2.7/posixpath.py", line 261, in expanduser
if not path.startswith('~'):
AttributeError: 'list' object has no attribute 'startswith'
You could now add debug to source files and properly understand where the problem is. In our example case, it is an ansible problem related to module_rabbitmq_user present in 2.1.0.0 see github PR
References¶
Disclaimer¶
IRMA is distributed as it is, in the hope that it will be useful, but without any warranty neither the implied merchantability or fitness for a particular purpose.
Whatever you do with this tool is uniquely your own responsibility.
License¶
IRMA source code is licensed under Apache License, version 2.0.
The full license text can be found below (Apache License, version 2.0).
Apache License, version 2.0¶
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
Authors¶
IRMA is a project co-funded by the following actors:
- CEA DAM
- DCNS
- GOVCERT.LU (governmental CERT of Luxembourg)
- Airbus Group
- Quarkslab
- Orange Group IS&T
The PRIMARY AUTHORS are (and/or have been):
- Alexandre Quint - Lead Developer, Quarkslab
- David Carle - Quarkslab
- Guillaume Dedrie - Quarkslab
- Fernand Lone-Sang - Quarkslab
And here is an inevitably incomplete list of MUCH-APPRECIATED CONTRIBUTORS – people who have submitted patches, reported bugs, helped answer newbie questions, and generally made IRMA that much better:
- lpecheur
- y0ug
- mdeloitte
Resources¶
- Project website
- IRC (irc.freenode.net, #qb_irma)
- Twitter (@qb_irma)