ISA

This is the documentation of ISA (Independent Statistics Aggregator). Independent means that the measurements are made independent of each other. The measurement on one machine does barely influence the measurement on another machine. Statistical Aggregator means that it is capable of collecting statistics of multiple nodes.

Why ISA?

Compared to other modern monitoring packages, ISA is easy to setup. You do not even have to setup software on the nodes which will be monitored!

Features

In the next list, the features of ISA are summed up. - Measuring resource usage on a node or on a cluster of nodes - Easy to setup - Flexible configuration - Low interference with the application - Easy to start, pause and stop ISA

Comming up

These features are not implemented yet, but are of big concern in future releases of ISA. - Scalability - Maintain logs

Short introduction

Servers are specified in an INI file. The servers are monitored using plugins which execute code on each of the servers. When statistics are collected, they are written out to an output file.

Contents

Installation

You can call ISA by using pip:

pip install isa

Or you can install isa using setup.py:

python setup.py install

Usage

You can call ISA using the following code:

isa /path/to/nodes.ini

Where path/to/nodes.ini is an INI file specifying the nodes that should be monitored. The structure of the INI file is explained in the INI file chapter.

There are more arguments which can be specified. These are explained in this document. In the next example, a call with many arguments is shown and explained.

isa nodes.ini --out results.csv --timeout 3 --interval 1

In this example, nodes.ini is parsed for the node information. It tries to connect to all the specified servers with a maximum of 3 seconds. When the script is connected to all the nodes, it collects data every 1 second and writes the statistics to results.csv.

Example

In this section, an example is given with many flags. If some flags are not clear, these can be looked up in the Arguments section of this document.

isa nodes.ini --out results.csv -f -p collectl -t 5 -i 10 --interlogin 3 --max-nodes 2

This snippet reads the nodes.ini file for node configuration. Results are written to results.csv and since it ends on .csv, the output type is automatically set to csv. The -f flag makes sure to force the recreation of the output file even if the file already exists. The collectl plugin is used for gathering node information. The timeout is set to 5 seconds (-t 5), so nodes try to login for at most 5 seconds. The interval time is set to 10 seconds (-i 10), which makes sure that information is gathered every 10 seconds. The interlogin time is set to 3 seconds, so between consecutive logins of groups of 2 nodes (specified by –max-nodes 2) the script will wait 3 seconds (–interlogin 3). Notice that a mixture of shortcut arguments and normal arguments is used. It is up to you what you prefer.

Debugging

If there is not enough information visible, you can add the -v flag for seeing INFO logs (this is also the default if no -v or -vv flag is present). If this is still not enough, you can use the -vv flag (which stands for Very Verbose) to see DEBUG logs and INFO logs. This is illustrated by the following example:

isa nodes.ini --vv

Arguments

output (-o, –out, –output)
isa nodes.ini --output results.csv
isa nodes.ini -o results.csv

Optional argument, this specifies where the results will be stored. In this version of ISA, only files ending on .csv are allowed.

output_type (-x, –output-type)
isa nodes.ini --out output.csv --output-type csv

Optional argument, default standard_output, csv if –out ends on .csv. When this is set to csv, a CSV file will be filled with the output where the path is specified by the –output argument. When this is set to standard_output, all results are written to the standard output (this is useful for testing).

force (-f, –force)
isa nodes.ini --out output.csv --force

Optional argument. If this flag is present and if the file specified by the –output flag already exists, then this file will be removed and recreated.

max-nodes (-m, –max-nodes)
isa nodes.ini --interlogin 1 --max-nodes 5

Optional argument, in combination with interlogin. Specifies how many nodes can login simultaneously. After this number of nodes are logging in, the script pauses for interlogin amount of time before it logs in the next group of nodes (with a maximum of max-nodes again).

plugins (-p, –plugins)
isa nodes.ini --plugins iostat,myplugin

Optional argument, a comma separated list (without spaces) of plugins to use for monitoring the cluster.

timeout (-t, –timeout)
isa nodes.ini --timeout 10

Optional argument, this specifies the maximum amount of time (in seconds) in which the script tries to connect to a SSH server.

interval (-i, –interval)
isa nodes.ini --interval 5

Optional argument, this specifies the amount of time (in seconds) in which the script collects and processes the statistics on the servers. The shorter the time, the less accurate the statistics, the longer the time, the longer one has to wait on the statistics.

interlogin (-l, –interlogin)
isa nodes.ini --interlogin 1

Optional argument. Specify the time needed between two consecutive logins. If this is too small, then a server can deny a connection and errors will appear which are hard to debug. This is very useful if all connections go through one via node.

max-nodes (-m, –max-nodes)
isa nodes.ini --interlogin 1 --max-nodes 5

Optional argument, in combination with interlogin. Specifies how many nodes can login simultaneously. After this number of nodes are logging in, the script pauses for interlogin amount of time before it logs in the next group of nodes (with a maximum of max-nodes again).

skip-install (-s, –skip-install)
isa nodes.ini --skip-install

Optional argument. When set, installation of the packages is skipped and can save some amount of time.

verbose (-v, -vv)
isa nodes.ini -vv

Optional argument, sets the log level. For -v or when this is not set at all, the log level is set to INFO. This shows you the basic information that is available. If the -vv flag is set, then then log level is set to DEBUG. Not only INFO messages are shown, but also more detailed information is given.

INI file

Minimal example

[node1]
host = 127.0.0.1
enabled = False

[node2]
host = 127.0.0.2
port = 23

In this simplified example, there are two nodes. One of the nodes (node1) is disabled and will not be monitored. By default, nodes are enabled, so node2 will be monitored. Furthermore, the host (which can be either an hostname or IP address) of node1 is 127.0.0.1 and the host of node2 is 127.0.0.2. The port of node2 is 23. Looking at the structure of the example, each section corresponds to a unique node name and every section contains flags which can be set. Some flags are optional and other flags are required.

Flags

This section explains all flags that can be used within a node section.

host

Required flag. Specifies the hostname or IP address of a node.

username

Optional flag, default root. The username to use for an SSH connection to the specified host.

password

Optional flag, default None. The password to use for the specified username and the SSH connection to the specified host.

identity_file

Optional flag, default None. The identity file to use to set up an SSH connection to the specified host.

port

Optional flag, default 22. The port to use when connecting to the SSH server.

enabled

Optional flag, default True. Possible values are True and False where True means that the node will be monitored and False means that the node will not be monitored. This is useful for defining a via node which should not be included into the monitoring.

via

Optional flag, default None. This is useful for connecting to a node via another node. For example, look at the following configuration:

[node1]
host = 127.0.0.1
via = login

[node2]
host = 127.0.0.2
via = login

[login]
host = 127.0.0.3

In this configuration, an SSH connection to node node1 is made by connecting first to node login. Also for node2 holds that any connection is made via node login. The code will check whether a circular definition exists and throws a CircularException exception if that is the case.

Plugins

There are a lot of measurement tools for specific servers. Therefore, ISA has a plugin structure so that it is easy to add new monitors. You can call ISA with specifying plugins in the –plugins argument which is a comma separated list of plugins that will be used. Below you will find an example of in which multiple plugins are loaded. Note that there are no spaces in the list of plugins.

isa nodes.ini --plugins iostat,free

List of build-in plugins

iostat

A plugin which collects information (I/O) from the server. iostat must be installed on the node.

free

A module which uses the information from free -m. This free application must be available on the machine.

collectl

A module which uses the information from collectl. The collectl application must be available on the machine.

Creating a plugin

A plugin file is named as follows: plugin_pluginname.py. Then, you can call ISA with a list of plugins and you can add pluginname to the comma separated list of the plugins argument of ISA. It is very easy to create a new plugin. The plugin script should have the following structure:

def plugin_init(args):
   return MyPlugin(...)

class MyPlugin(Plugin):
   def install(self, server):
      server.execute(...)
   def collect(self, server):
      server.execute(...)

You see a collect method and a install method. The install method is called once after logging in to all the servers and its aim is to install software when it is not installed yet on the server. The collect method is used for collecting statistics on the server. The install method is not required! If you use the install method for installing a package, you can use the install_package method from the Plugin class. This is demonstrated in the next example:

def plugin_init(args):
   return MyPlugin(...)

class MyPlugin(Plugin):
   def install(self, server):
      self.install_package(server, 'sysstat')
   def collect(self, server):
      server.execute(...)

This example tries to install sysstat using different package managers (like yum and apt-get).

It is also possible to leave out the install method as is shown in the next example:

def plugin_init(args):
   return MyPlugin(...)

class MyPlugin(Plugin):
   def collect(self, server):
      server.execute(...)

The plugin is first initialized. You can use arguments from the original ISA call. For example, suppose ISA is called as follows:

isa nodes.ini --timeout 3

Then you can use args.timeout to refer to the timeout argument:

def plugin_init(args):
   return MyPlugin(args.timeout)

class MyPlugin(Plugin):
   def install(self, server):
      print("I assume my monitoring software is already installed on the server.")

   def __init__(self, timeout):
      print("I am called with timeout=%s" % timeout)

   def collect(self, server):
      server.execute(...)

The collect method should return either a list of dictionaries or a dictionary which specifies the collected data in key-value pairs:

def plugin_init(args):
   return MyPlugin()

class MyPlugin(Plugin):
   def install(self, server):
      pass

   def collect(self, server):
      return {
         'temperature': 25.0
      }

Or you can use a list:

def plugin_init(args):
   return MyPlugin()

class MyPlugin(Plugin):
   def install(self, server):
      pass

   def collect(self, server):
      return [{
         'sensor': 'A',
         'temperature': 15.0
      }, {
         'sensor': 'B',
         'temperature': 35.0
      }]

License

The MIT License (MIT)

Copyright (c) 2016 Kevin Jacobs

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Developers

Changelog

Version 1.2.4

  • Fixed double argument character bug

Version 1.2.3

  • Feature: skip installation
  • Bugfixing

Version 1.2.2

  • Test fixing

Version 1.2.1

  • Version change

Version 1.2.0

  • Version change

Version 1.1.10

  • New flags: output-type, force and max-nodes
  • Flags shortcuts added
  • Tests added for output handler
  • File creation in tests refactored

Version 1.1.9

  • Use fields in the CSV file such that the CSV file is consistent

Version 1.1.8

  • Fixed typos

Version 1.1.7

  • Bugfix: sometimes crashes randomly (solved by introducing intertime)
  • Bugfix: added interlogin time to overcome SSH security issues
  • Added port as parameter in the INI file

Version 1.1.6

  • Bugfix: ansering “yes” automatically to the quest “Are you sure you want to continue connecting?”
  • Removed retries for connecting

Version 1.1.5

  • Versioning fix

Version 1.1.4

  • Fix decoding bug

Version 1.1.3

  • Change version

Version 1.1.2

  • Change version

Version 1.1.1

  • Change version

Version 1.1.0

  • Final merge

Version 1.0.5

  • Set default verbose level to verbose
  • Install setup methods for automatically installing software on the external server
  • Changed documentation style
  • Better logging
  • Additional plugin settings

Version 1.0.4

  • Set Travis branch to master

Version 1.0.3

  • Added standard_output as default output method
  • Added the “v” and “vv” flags to the documentation
  • Added CTRL+C to get out the ISA application
  • Added the “free -m” plugin
  • Added the “collectl” plugin

Version 1.0.2

  • Changed version number
  • Fixed ReadTheDocs

Version 1.0.1

  • Changed version number

Version 1.0.0

  • Added plugin infrastructure
  • Added output handling
  • Added monitor class
  • Added input handling
  • Added SSH server connection

Version 0.2.0

  • Initial project code

Version 0.1.0

  • Project scaffolding