Welcome to infoset-ng’s documentation!¶
infoset-ng
is a lightweight Python 3 based REST API that stores
and/or retrieves timestamped data. It runs on Linux.
We recommend that you read as much as you can on installation, configuration, operation and the API to get a good understanding of the software.
The introduction will provide useful background information too.
Introduction¶
There’s a lot to know about infoset-ng
which we’ll summarize here.
Features¶
infoset-ng
has the following features:
- Open source.
- Written in python, a modern language.
- Easy configuration.
- Uses the well known Flask webserver for accepting data and responding to requests.
infoset-ng
has a number of fault tolerant features aimed at making it resilient in unstable computing environemnts.- MariaDB / MySQL database backend
- Database connection pooling to reduce database load.
- Ingestion of data supports parallel multiprocessing for maximum speed.
- The infoset-ng-ng API server can tolerate the loss of communication with its database by caching the data locally until the database returns online.
- The
infoset-ng
configuration is entirely stored in files. This allows it to collect data in the absense of a database, such as during maintenance or an outage. - Backups are simple. Just save the entire contents of the
infoset-ng
directory tree including hidden files, and save a copy of the database for your performance data.
We are always looking for more contributors!
Inspiration / History¶
The infoset-ng
project originally took inspiration from the
SourceForge based switchmap
project. switchmap
was written in
PERL and designed to create tabular representations of network
topologies. Early versions of infoset
eventually had expanded
features which included the polling of network devices for real time
performance data. This data was presented via a web interface. The code
became cumbersome and the original infoset
was split into three
componet parts.
infoset-ng
: An API for storing and retrieving real time data.garnet
: A network / server performance charting web application that uses various types of agents for collecting real time data.garnet
usesinfoset-ng
to store its data.switchmap-ng
A python 3 based feature equivalent version ofswitchmap
.
Each of these projects resides on the University of the West Indies Computing Society’s GitHub account.
Oversight¶
infoset-ng
is a student collaboration between:
- The University of the West Indies Computing Society. (Kingston, Jamaica)
- The University of Techology, IEEE Student Branch. (Kingston, Jamaica)
- The Palisadoes Foundation http://www.palisadoes.org
And many others.
Installation¶
This section outlines how to install and do basic configuration of infoset-ng
.
Dependencies¶
infoset-ng
has the following requirements:
- python >= 3.5
- python3-pip
- MySQL >= 5.7 OR MariaDB >= 10.0
It will not work with lower versions.
Ubuntu / Debian / Mint¶
The commands for installing the dependencies are:
$ sudo apt-get -y install python3 python3-pip python3-dev memcached
Select either of these commands to install MySQL server or MariaDB server
$ sudo apt-get -y install mysql-server
$ sudo apt-get -y install mariadb-server
Centos / Fedora¶
The commands for installing the dependencies are:
$ sudo dnf -y install python3 python3-pip python3-dev memcached
Select either of these commands to install MySQL server or MariaDB server
$ sudo dnf -y install mysql-server
$ sudo dnf -y install mariadb-server
Installation¶
Installation is simple. There are three basic steps.
- Clone the Repository
- Setup the Database
- Run the Installation Script
This will now be explained in more detail.
Clone the Repository¶
Now clone the repository and copy the sample configuration file to its final location.
$ git clone https://github.com/PalisadoesFoundation/infoset-ng
$ cd infoset-ng
Setup the Database¶
Next create the MySQL or MariaDB database. Make sure the database server is running.
$ mysql -u root -p
password:
mysql> create database infoset_ng;
mysql> grant all privileges on infoset_ng.* to infoset_ng@"localhost" identified by 'PASSWORD';
mysql> flush privileges;
mysql> exit;
Note Remember the value you select for PASSWORD
. It will be required when you edit the infoset-ng
configuration file later.
Run Installation Script¶
Run the installation script. There are two alternatives:
Installing as a regular user
There are some things to keep in mind when installing switchmap-ng as a regular user.
- Use this method if you don’t have
root
access to your system. - The
switchmap-ng
daemons will not automatically restart on reboot using this method.
To make switchmap-ng
run with your username, then execute this command:
$ maintenance/install.py
Installing as the “root” user
There are some things to keep in mind when installing switchmap-ng as the root user.
- The
switchmap-ng
daemons will automatically restart on reboot using this installation method. - Note: Do not run setup using
sudo
. Usesudo
to become theroot
user first.
To install switchmap-ng
as the root
user execute this command:
# maintenance/install.py
Next Steps¶
It is now time to review the various configuration options.
Configuration¶
It is important to have a valid configuration file in the etc/
directory before starting data collection. The installation automatically creates a default version that may need to be edited. This page explains the various configuration parameters.
The examples/etc
directory includes a sample reference file.
infoset-ng Configuration Example¶
In this example we explain each parameter in the configuration file.
The main
section governs the general operation of infoset-ng
.
main:
log_directory: /opt/infoset/log
log_level: debug
ingest_cache_directory: /opt/infoset/cache
ingest_pool_size: 20
interval: 300
listen_address: 0.0.0.0
bind_port: 6000
sqlalchemy_pool_size: 10
sqlalchemy_max_overflow: 10
memcached_hostname: localhost
memcached_port: 11211
db_hostname: localhost
db_username: infoset_ng
db_password: PASSWORD
db_name: infoset_ng
username: USERNAME
An explanation of these fields follows:
Parameter | Description |
---|---|
main: |
YAML key describing the server configuration. |
log_directory: |
The directory where infoset-ng places its log files |
log_level: |
Defines the logging level. debug level is the most verbose, followed by info , warning and critical |
ingest_cache_directory: |
Location where the agent data ingester will store its data in the event it cannot communicate with either the database or the server’s API |
ingest_pool_size: |
The maximum number of threads used to ingest data into the database |
interval: |
The expected interval in seconds between updates to the database from systems posting to the infoset API. Data retieved from the API will be spaced interval seconds apart. |
listen_address: |
IP address the API will be using. The default is 0.0.0.0 or all available IP addresses |
bind_port: |
The TCP port the API will be listening on |
sqlalchemy_pool_size: |
The SQLAlchemy pool size. This is the largest number of connections that infoset-ng will be keep persistently with the MySQL database |
sqlalchemy_max_overflow: |
The SQLAlchemy maximum overflow size. When the number of connections reaches the size set in sqlalchemy_pool_size , additional connections will be returned up to this limit. This is the floating number of additional database connections to be made available. |
memcached_hostname: localhost |
The hostname of our memcached cache server |
memcached_port: 11211 |
The port which memcached is running on |
db_hostname: |
The devicename or IP address of the database server. |
db_username: |
The database username |
db_password: |
The database password |
db_name: |
The name of the database |
username: |
The username that scripts should run as |
Logrotate Configuration¶
The examples/linux/logrotate/infoset-ng
file is a working logrotate
configuration to rotate the log files that infoset-ng
generates. The infoset-ng
log file data can be extensive and adding the logrotate file to your system
is highly recommended.
$ sudo cp examples/linux/logrotate/infoset-ng /etc/logrotate.d
Next Steps¶
It is time to test the operation of infoset-ng
.
Advanced Operation¶
The switchmap-ng
CLI is meant for ease of use. This page shows some advanced features.
Operating the Ingester as a System Daemon¶
This is the preferred mode of operation for production systems. This mode is automatically configured if you installed switchmap-ng
using the root
user.
Note: Sample systemd
files can be found in the examples/linux/systemd/
directory.
The ingester
can be started like this:
$ sudo systemctl start switchmap-ng-ingester.service
The ingester
can be stopped like this:
$ sudo systemctl stop switchmap-ng-ingester.service
You can get the status of the ingester
like this:
$ sudo systemctl status switchmap-ng-ingester.service
You can get the ingester
to automatically restart on boot like this:
$ sudo systemctl enable switchmap-ng-ingester.service
Operating the API as a System Daemon¶
This is the preferred mode of operation for production systems. This mode is automatically configured if you installed switchmap-ng
using the root
user.
Note: Sample systemd
files can be found in the examples/linux/systemd/
directory.
The API
can be started like this:
$ sudo systemctl start switchmap-ng-api.service
The API
can be stopped like this:
$ sudo systemctl stop switchmap-ng-api.service
You can get the status of the API
like this:
$ sudo systemctl status switchmap-ng-api.service
You can get the API
to automatically restart on boot like this:
$ sudo systemctl enable switchmap-ng-api.service
Command Line Interface (CLI)¶
This page outlines how to use the infoset-ng
command line interface (CLI)
Viewing infoset-ng
status¶
There are two important infoset-ng
daemons.
- ingester: Gets data from devices
- API: Displays device data on web pages
You can get the status of each daemon using the following CLI commands:
Ingester status¶
You can get the status of the ingester using this command:
$ bin/infoset-ng-cli show ingester status
API status¶
You can get the status of the API using this command:
$ bin/infoset-ng-cli show api status
Managing the infoset-ng
Daemons¶
You can manage the daemons using the CLI. Here’s how:
Ingester Management¶
The ingester can be started, stopped and restarted using the following commands. Use the --force
option only if the daemon may be hung.
$ bin/infoset-ng-cli ingester start
$ bin/infoset-ng-cli ingester stop
$ bin/infoset-ng-cli ingester stop --force
$ bin/infoset-ng-cli ingester restart
$ bin/infoset-ng-cli ingester restart --force
Note: You will need to do a restart whenever you modify a configuration parameter.
API Management¶
The API can be started, stopped and restarted using the following commands. Use the --force
option only if the daemon may be hung.
$ bin/infoset-ng-cli api start
$ bin/infoset-ng-cli api stop
$ bin/infoset-ng-cli api stop --force
$ bin/infoset-ng-cli api restart
$ bin/infoset-ng-cli api restart --force
Note: You will need to do a restart whenever you modify a configuration parameter.
Testing Infoset-NG¶
You will need to verify system operation. Here’s how.
$ bin/infoset-ng-cli test ingester
Viewing infoset-ng
logs¶
When troubleshooting it is a good practice to view the infoset-ng
log files.
Ingester logs¶
You can view the ingester logs using this command:
$ bin/infoset-ng-cli show ingester logs
Viewing the infoset-ng
Configuration¶
You can view the configuration using this command:
$ bin/infoset-ng-cli show configuration
Using the API¶
This section outlines how to use the API
Why Infoset-ng Expects UTC Timestamps¶
There is a good reason for this. According to the python datetime documentation page, The rules for time adjustment across the world are more political than rational, change frequently, and there is no standard suitable for every application aside from UTC.
We cannot guarantee the python timezone libraries will be always up to date, so we default to UTC as recommended.
Posting Data to the API¶
Posting data to the API is. Add the prefix http://SERVER_IP:6000
to
all the examples below to update data in your instance of infoset-ng
Route /infoset/api/v1/receive/<id_agent>
¶
JSON data needs to be posted to the http://SERVER_IP:6000/infoset/api/v1/receive/<id_agent>
URL where id_agent
is a unique identifier of the software script that is posting the data. This id_agent
must be unique and consistent for each script or application posting data to infoset-ng
. For example, if you have three data collection scripts running across two devices, then each script must report a unique id_agent
, three unique IDs in total. We suggest using a hash of a random string to generate your id_agent
. There is a 512 character limit on the size of the agent_id
.
The example below explains the expected JSON format:
{'agent': 'agent_name',
'timeseries': {'label_l': {'base_type': 1,
'data': [[1, 224.0, 'source_1']],
'description': 'description_1},
'label_2': {'base_type': 1,
'data': [[1, 1383.2, 'source_2']],
'description': 'description_2'}},
'devicename': '192.168.3.100',
'timestamp': 1474823400,
'id_agent': '8a6887228e33e3b433bd0da985c203904a48e2e90804ae217334dde2b905c57e'}
Where feasible, we will use Linux and networking related examples to make explanation easier.
Field | Descripton |
---|---|
agent |
Agent or application name. If your agent script is designed to collect server performance data, you could name it ‘server_performance’. Each server performance agent would therefore report the same agent value. |
timeseries |
TimeSeries data follows |
timeseries[label] |
A short label defining what the data is about. |
timeseries[label][base_type] |
Defines the type of data. The values are basesd on the SNMP standard. Values include: 0 for relatively unchanging alphanumeric data, which could include things like the version of an operating system; 1 for non-incremental, point-in-time numeric data such as temperature, speed, process count; 32 for numeric data that increments using a 32 bit counter such as bytes through a network interface since the device booted; 64 for 64 bit counter numeric data. |
timeseries[label][description] |
Description of the data, such as ‘temperature data’ |
timeseries[label][data] |
Data related to the labels. It is a list of lists. Each list has three fields [index, value, source]. The index value is a unique, unchangeable identifier for the source of the data, this is preferrably numeric such as an interface index number, but could also be string information such as an interface name or disk partition mount point. The value is the value of the data recorded. The source is a description of the source of the data to make it more recognizable when the data is eventually presented to your users. This could be interface eth0 versus a plain eth0 |
devicename |
Devicename of the device sending the data. For phone apps, this could be set to a phone number of SIM ID. |
timestamp |
Epoch UTC time when data was generated. This must be an integer. |
agent_id |
A unique, unchanging identifier for the application sending the data. |
How to Create Agents Using Python¶
When you installed infoset-ng
you probably ran the bin/tools/test_installation.py
test script. This script emulates an agent and uses the classess in the infoset/reference/reference.py
file to create the required JSON. Refer to this code when creating your agents.
Retrieving Data from the API¶
This section covers how to retrieve data from the API. First we cover some of the basics.
Overview¶
Retrieving data from infoset is easy. Add the prefix http://SERVER_IP:6000
to all the examples below to get data from your instance of infoset-ng
You can test each route using the command:
$ curl http://SERVER_IP:6000/route
The json fields in the results received from the API are explained in the infoset/db/db_orm.py
file.
Database Table Names¶
It is important to understand the purpose of each database table as they
are used in the routes. The structure of each table can be seen by
reviewing the db_orm.py
file in the infoset.db
module.
Table | Descripton |
---|---|
iset_agent |
Data on the agents that have posted information to the API |
iset_deviceagent |
The same agent could be installed on multiple devices. This table tracks which unique device and agent combination have posted information to the API |
iset_device |
Tracks all the devices that have posted information to the API |
iset_datapoint |
Stores metadata on the various datapoints that agents report on. A datapoint ID is unique throughout the system |
iset_data |
Stores the actual data for each datapoint |
iset_billcode |
Stores data on the billing code for datapoints. Useful for financial accounting. |
iset_department |
Stores data on the departments to which the billing code should be applied. Useful for financial accounting. |
Routes¶
Data is retrieved by making HTTP requests to well known URIs or routes
. These are covered next.
Route /infoset/api/v1/agents¶
This route will retreive data on all agents that have ever posted data to the API. It is returned in the form of a list of lists.
Field | Description |
---|---|
exists |
True if the agent exists, False if not |
enabled |
True if enabled, False if disabled |
id_agent |
The Agent ID |
idx_agent |
The unique index value of the agent in the database |
name |
The agent name |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/agents
[
{
"enabled": true,
"exists": true,
"id_agent": "ece739a93cca2c8e5444507990158b05b7d890d5798dc273578382d171bf6500",
"idx_agent": 2,
"last_timestamp": 1480570200,
"name": "linux_in"
},
{
"enabled": true,
"exists": true,
"id_agent": "1b3c081ba928d8a1ebb16084f23e55b972b0cda1737b0449853b591f4c84ad42",
"idx_agent": 3,
"last_timestamp": 1480570200,
"name": "_garnet"
},
]
Route /infoset/api/v1/agents/<idx_agent>
¶
This route retrieves information for a specific agent index value.
Field | Description |
---|---|
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
id_agent |
The unique Agent ID |
idx_agent |
The unique index of the agent in the database |
devicename |
Unique devicename in the infoset-ng database |
name |
The agent name |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/agents/3
{
"enabled": true,
"exists": true,
"id_agent": "70f2d9061f3ccc96915e19c13817c8207e2005d05f23959ac4c225b6a5bfe557",
"idx_agent": 3,
"last_timestamp": 1480611300,
"name": "linux_in"
}
$
Route /infoset/api/v1/agents?id_agent=<id_agent>¶
This route retrieves information for a specific id_agent
value as a list.
Field | Description |
---|---|
agent_label |
Label that the agent assigned to the datapoint |
agent_source |
The source of the data |
base_type |
Base type of the data |
billable |
True if billable, False if not. |
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
id_datapoint |
The unique datapoint ID |
idx_datapoint |
The unique datapoint index |
idx_agent |
The unique index of the agent that reported on this datapoint |
idx_billcode |
The index of the billing code to be applied to the datapoint |
idx_department |
The index value of the department to which the billing code should be applied |
idx_device |
The unique index of the device in the database |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
Example:
$ curl "http://SERVER_IP:6000/infoset/api/v1/agents?id_agent=70f2d9061f3ccc96915e19c13817c8207e2005d05f23959ac4c225b6a5bfe557"
[{
"enabled": true,
"exists": true,
"id_agent": "70f2d9061f3ccc96915e19c13817c8207e2005d05f23959ac4c225b6a5bfe557",
"idx_agent": 3,
"last_timestamp": 1480611600,
"name": "linux_in"
}]
$
Route /infoset/api/v1/deviceagents¶
The same agent could be installed on multiple devices. This route returns data that tracks each unique device and agent combination have posted information to the API. It is returned as a list of dicts.
Field | Description |
---|---|
idx_agent | The index value of the agent |
idx_device | The index value of the device |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/deviceagents
[
{
"idx_agent": 1,
"idx_device": 1
},
{
"idx_agent": 2,
"idx_device": 2
},
{
"idx_agent": 3,
"idx_device": 2
},
{
"idx_agent": 4,
"idx_device": 2
}
]
$
Route /infoset/api/v1/deviceagents/idx_deviceagent
¶
The same agent could be installed on multiple devices. This route
returns data that tracks each unique device and agent combination have
posted information to the API, filtered by idx_deviceagent
. It is returned as a list of dicts.
Field | Description |
---|---|
idx_agent | The index value of the agent |
idx_device | The index value of the device |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/deviceagents
[
{
"idx_agent": 1,
"idx_device": 1
},
{
"idx_agent": 2,
"idx_device": 2
},
{
"idx_agent": 3,
"idx_device": 2
},
{
"idx_agent": 4,
"idx_device": 2
}
]
$
Route /infoset/api/v1/devices/<idx_device>
¶
This route retrieves information for a specific device index value.
Field | Description |
---|---|
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
devicename |
Unique devicename in the``infoset-ng`` database |
idx_device |
The unique index of the device in the database |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/devices/2
{
"description": null,
"enabled": true,
"exists": true,
"devicename": "afimidis",
"idx_device": 2x
}
$
Route /infoset/api/v1/datapoints/<idx_datapoint>
¶
This route retrieves information for a specific datapoint index value value.
Please read section on the API’s /infoset/api/v1/receive
route for
further clarification of the field description in the table below.
Field | Description |
---|---|
agent_label |
Label that the agent assigned to the datapoint |
agent_source |
The source of the data |
base_type |
Base type of the data |
billable |
True if billable, false if not. |
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
id_datapoint |
The unique datapoint ID |
idx_datapoint |
The unique datapoint index |
idx_agent |
The unique index of the agent that reported on this datapoint |
idx_billcode |
The index of the billing code to be applied to the datapoint |
idx_department |
The index value of the department to which the billing code should be applied |
idx_device |
The unique index of the device in the database |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
timefixed_value |
Some datapoints may track unchanging numbers such as the version of an operating system. This value is placed here if the base_type is 0`` |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/datapoints/2
{
"agent_label": "cpu_count",
"agent_source": null,
"base_type": 1,
"billable": false,
"enabled": true,
"exists": true,
"id_datapoint": "fef5fb0c60f6ecdd010c99f14d120598d322151b9d942962e6877945f1f14b5f",
"idx_agent": 2,
"idx_billcode": 1,
"idx_datapoint": 2,
"idx_department": 1,
"idx_device": 2,
"last_timestamp": 1480611600,
"timefixed_value": null
}
$
Route /infoset/api/v1/datapoints?id_datapoint=<id_datapoint>¶
This route retrieves information for a specific id_datapoint value value.
Please read section on the API’s /infoset/api/v1/receive
route for
further clarification of the field description in the table below.
Field | Description |
---|---|
agent_label |
Label that the agent assigned to the datapoint |
agent_source |
The source of the data |
base_type |
Base type of the data |
billable |
True if billable, false if not. |
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
id_datapoint |
The unique datapoint ID |
idx_datapoint |
The unique datapoint index |
idx_agent |
The unique index of the agent that reported on this datapoint |
idx_billcode |
The index of the billing code to be applied to the datapoint |
idx_department |
The index value of the department to which the billing code should be applied |
idx_device |
The unique index of the device in the database |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
timefixed_value |
Some datapoints may track unchanging numbers such as the version of an operating system. This value is placed here if the base_type is 0`` |
Example:
$ curl "http://SERVER_IP:6000/infoset/api/v1/datapoints?id_datapoint=fef5fb0c60f6ecdd010c99f14d120598d322151b9d942962e6877945f1f14b5f"
{
"agent_label": "cpu_count",
"agent_source": null,
"base_type": 1,
"billable": false,
"enabled": true,
"exists": true,
"id_datapoint": "fef5fb0c60f6ecdd010c99f14d120598d322151b9d942962e6877945f1f14b5f",
"idx_agent": 2,
"idx_billcode": 1,
"idx_datapoint": 2,
"idx_department": 1,
"idx_device": 2,
"last_timestamp": 1480611600,
"timefixed_value": null
}
$
Route /infoset/api/v1/datapoints?idx_deviceagent=<idx_deviceagent>¶
This route retrieves information for a specific idx_deviceagent value value.
Please read section on the API’s /infoset/api/v1/receive
route for
further clarification of the field description in the table below.
Field | Description |
---|---|
agent_label |
Label that the agent assigned to the datapoint |
agent_source |
The source of the data |
base_type |
Base type of the data |
billable |
True if billable, false if not. |
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
id_datapoint |
The unique datapoint ID |
idx_datapoint |
The unique datapoint index |
idx_deviceagent |
The unique index of the device agent that reported on this datapoint |
idx_billcode |
The index of the billing code to be applied to the datapoint |
idx_department |
The index value of the department to which the billing code should be applied |
idx_device |
The unique index of the device in the database |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
timefixed_value |
Some datapoints may track unchanging numbers such as the version of an operating system. This value is placed here if the base_type is 0`` |
Example:
$ curl "http://SERVER_IP:6000/infoset/api/v1/datapoints?idx_deviceagent=2"
{
"agent_label": "cpu_count",
"agent_source": null,
"base_type": 1,
"billable": false,
"enabled": true,
"exists": true,
"id_datapoint": "fef5fb0c60f6ecdd010c99f14d120598d322151b9d942962e6877945f1f14b5f",
"idx_deviceagent": 2,
"idx_billcode": 1,
"idx_datapoint": 2,
"idx_department": 1,
"idx_device": 2,
"last_timestamp": 1480611600,
"timefixed_value": null
}
$
Route /infoset/api/v1/datapoints/all/summary¶
This route retrieves dummary information about all datapoints.
Please read section on the API’s /infoset/api/v1/receive
route for
further clarification of the field description in the table below.
Field | Description |
---|---|
agent_label |
Label that the agent assigned to the datapoint |
agent_source |
The source of the data |
devicename |
Unique devicename in the infoset-ng database |
id_agent |
The unique Agent ID |
id_datapoint |
The unique datapoint ID |
idx_datapoint |
The unique datapoint index |
idx_deviceagent |
The unique index of the deviceagent in the database |
name |
The agent name |
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/datapoints/all/summary
[
{
"agent_label": "system",
"agent_source": null,
"devicename": "palisadoes",
"id_agent": "f32eda632703ac9d94d80b43d5dd54d0198cd0dabf541dae97b94e5b75b851d5",
"idx_datapoint": 417,
"idx_deviceagent": 4,
"name": "remote_linux_passive"
},
{
"agent_label": "version",
"agent_source": null,
"devicename": "palisadoes",
"id_agent": "f32eda632703ac9d94d80b43d5dd54d0198cd0dabf541dae97b94e5b75b851d5",
"idx_datapoint": 418,
"idx_deviceagent": 4,
"name": "remote_linux_passive"
}
]
$
Route /infoset/api/v1/datapoints&id_datapoint=``<id_datapoint>``¶
This route retrieves information for a specific datapoint ID value value.
Please read section on the API’s /infoset/api/v1/receive
route for
further clarification of the field description in the table below.
Field | Description |
---|---|
agent_label |
Label that the agent assigned to the datapoint |
agent_source |
The source of the data |
base_type |
Base type of the data |
billable |
True if billable, false if not. |
enabled |
True if enabled, False if not |
exists |
True if the requested index value exists in the database |
id_datapoint |
The unique datapoint ID |
idx_datapoint |
The unique datapoint index |
idx_agent |
The unique index of the agent that reported on this datapoint |
idx_billcode |
The index of the billing code to be applied to the datapoint |
idx_department |
The index value of the department to which the billing code should be applied |
idx_device |
The unique index of the device in the database |
last_timestamp |
The UTC timestamp of the the most recent data posted by the agent to the API |
timefixed_value |
Some datapoints may track unchanging numbers such as the version of an operating system. This value is placed here if the base_type is 0`` |
Example:
$ curl "http://SERVER_IP:6000/infoset/api/v1/datapoints?id_datapoint=fef5fb0c60f6ecdd010c99f14d120598d322151b9d942962e6877945f1f14b5f"
{
"agent_label": "cpu_count",
"agent_source": null,
"base_type": 1,
"billable": false,
"enabled": true,
"exists": true,
"id_datapoint": "fef5fb0c60f6ecdd010c99f14d120598d322151b9d942962e6877945f1f14b5f",
"idx_agent": 2,
"idx_billcode": 1,
"idx_datapoint": 2,
"idx_department": 1,
"idx_device": 2,
"last_timestamp": 1480612500,
"timefixed_value": null
}
$
Route /infoset/api/v1/devices/<idx_device>
/agents¶
This route will retreive data on all the agents that have reported data from a specific device. The agent data returned are their index values, and the query is done based on the index of the device.
Example:
$ curl http://SERVER_IP:6000/infoset/api/v1/devices/2/agents
[
2,
3,
4
]
$
Route /infoset/api/v1/lastcontacts¶
This route will retreive all the most recently posted data values.
Data is queried starting from 10X the interval value in your configuration file seconds ago until the present.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl http://SERVER_IP:6000/infoset/api/v1/lastcontacts
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 60370900.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 60370900.0
},
...
...
...
...
...
...
{
"idx_datapoint": 417,
"timestamp": 1483629900,
"value": 60370900.0
},
{
"idx_datapoint": 418,
"timestamp": 1483629900,
"value": 60370900.0
}
]
Route /infoset/api/v1/lastcontacts?secondsago=<seconds>¶
This route will retreive all the most recently posted data values.
The query starts looking for contacts as of secondsago
seconds ago.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts
, which is the preferred method of getting this data.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts?secondsago=3600"
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 60370900.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 60370900.0
},
...
...
...
...
...
...
{
"idx_datapoint": 417,
"timestamp": 1483629900,
"value": 60370900.0
},
{
"idx_datapoint": 418,
"timestamp": 1483629900,
"value": 60370900.0
}
]
Route /infoset/api/v1/lastcontacts?ts_start=``timestamp``¶
This route will retreive all the most recently posted data values.
A starting UTC timestamp needs to be provided. Searches for contacts are made from starting at this time until the present.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts
, which is the preferred method of getting this data.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts?ts_start=0"
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 60370900.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 60370900.0
},
...
...
...
...
...
...
{
"idx_datapoint": 417,
"timestamp": 1483629900,
"value": 60370900.0
},
{
"idx_datapoint": 418,
"timestamp": 1483629900,
"value": 60370900.0
}
]
Route /infoset/api/v1/lastcontacts/id_agents¶
This route will retreive all the most recently posted data values.
Data is queried starting from 10X the interval value in your configuration file seconds ago until the present.
Field | Description |
---|---|
devicename |
The name of the device generating the data |
name |
The name of the agent generating the data |
id_agent |
The ID of the agent generating the data |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl http://SERVER_IP:6000/infoset/api/v1/lastcontacts/id_agents
[
{
"devicename": "name_1",
"id_agent": "bec9ba91e14804001e037fa4f52c94fb1ef027d04e1b86f6a74ab36e3b073609",
"name": "Agent_Name",
"timeseries": {
"agent_label_1": {
"timestamp": 1487377500,
"value": 1.8191
},
"agent_label_2": {
"timestamp": 1487377500,
"value": 1.8694
}
}
},
{
"devicename": "name_2",
"id_agent": "e33ce6311cf95c6264c6777323e9c717220b19ccad7b6da1877384e7fb3364e7",
"name": "Agent_Name",
"timeseries": {
"agent_label_1": {
"timestamp": 1487377500,
"value": 1.61078
},
"agent_label_2": {
"timestamp": 1487377500,
"value": 1.54421
}
}
}
]
Route /infoset/api/v1/lastcontacts/id_agents?secondsago=<seconds>¶
This route will retreive all the most recently posted data values.
The query starts looking for contacts as of secondsago
seconds ago.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts/id_agents
, which is the preferred method of getting this data.
Field | Description |
---|---|
devicename |
The name of the device generating the data |
name |
The name of the agent generating the data |
id_agent |
The ID of the agent generating the data |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts/id_agents?secondsago=3600"
[
{
"devicename": "name_1",
"id_agent": "bec9ba91e14804001e037fa4f52c94fb1ef027d04e1b86f6a74ab36e3b073609",
"name": "Agent_Name",
"timeseries": {
"agent_label_1": {
"timestamp": 1487377500,
"value": 1.8191
},
"agent_label_2": {
"timestamp": 1487377500,
"value": 1.8694
}
}
},
{
"devicename": "name_2",
"id_agent": "e33ce6311cf95c6264c6777323e9c717220b19ccad7b6da1877384e7fb3364e7",
"name": "Agent_Name",
"timeseries": {
"agent_label_1": {
"timestamp": 1487377500,
"value": 1.61078
},
"agent_label_2": {
"timestamp": 1487377500,
"value": 1.54421
}
}
}
]
Route /infoset/api/v1/lastcontacts/id_agents?ts_start=``timestamp``¶
This route will retreive all the most recently posted data values.
A starting UTC timestamp needs to be provided. Searches for contacts are made from starting at this time until the present.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts/id_agents
, which is the preferred method of getting this data.
Field | Description |
---|---|
devicename |
The name of the device generating the data |
name |
The name of the agent generating the data |
id_agent |
The ID of the agent generating the data |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts/id_agents?ts_start=0"
[
{
"devicename": "name_1",
"id_agent": "bec9ba91e14804001e037fa4f52c94fb1ef027d04e1b86f6a74ab36e3b073609",
"name": "Agent_Name",
"timeseries": {
"agent_label_1": {
"timestamp": 1487377500,
"value": 1.8191
},
"agent_label_2": {
"timestamp": 1487377500,
"value": 1.8694
}
}
},
{
"devicename": "name_2",
"id_agent": "e33ce6311cf95c6264c6777323e9c717220b19ccad7b6da1877384e7fb3364e7",
"name": "Agent_Name",
"timeseries": {
"agent_label_1": {
"timestamp": 1487377500,
"value": 1.61078
},
"agent_label_2": {
"timestamp": 1487377500,
"value": 1.54421
}
}
}
]
Route /infoset/api/v1/lastcontacts/<idx_deviceagent>
¶
Searches for contacts are made starting from an hour ago to the present. from a specific Device Agent combination. The query is done based on the device’s deviceagent index.
Data is queried starting from 10X the interval value in your configuration file seconds ago until the present.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl http://SERVER_IP:6000/infoset/api/v1/lastcontacts/2
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 4,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 5,
"timestamp": 1483629900,
"value": 9.0
}
]
Route /infoset/api/v1/lastcontacts/<idx_deviceagent>
?secondsago=``seconds``¶
This route will retreive the most recently posted data values from a specific Device Agent combination. The query is done based on the device’s deviceagent index.
Data is queried starting from 10X the interval value in your configuration file seconds ago until the present.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts
, which is the preferred method of getting this data.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts/2?secondsago=3600"
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 4,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 5,
"timestamp": 1483629900,
"value": 9.0
}
]
$
Route /infoset/api/v1/lastcontacts/<idx_deviceagent>
?ts_start=``timestamp``¶
This route will retreive the most recently posted data values from a specific Device Agent combination. The query is done based on the device’s deviceagent index.
A starting UTC timestamp needs to be provided. Searches for contacts are made from starting at this time until the present.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts
, which is the preferred method of getting this data.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts/2?ts_start=0"
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 4,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 5,
"timestamp": 1483629900,
"value": 9.0
}
]
$
Route /infoset/api/v1/lastcontacts/devicenames/<devicename>
/id_agents/<id_agent>
¶
Searches for contacts are made starting from an hour ago to the present. from a specific devicename
and id_agent
combination.
Data is queried starting from 10X the interval value in your configuration file seconds ago until the present.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl http://SERVER_IP:6000/infoset/api/v1/devicenames/_INFOSET_TEST_/id_agents/558bb0055d7b4299c2ebe6abcc53de64a9ec4847b3f82238b3682cad575c7749
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 4,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 5,
"timestamp": 1483629900,
"value": 9.0
}
]
Route /infoset/api/v1/lastcontacts/devicenames/<devicename>
/id_agents/<id_agent>
?ts_start=``timestamp`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This route will retreive the most recently posted data values from a specific devicename
and id_agent
combination.
A starting UTC timestamp needs to be provided. Searches for contacts are made from starting at this time until the present.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts/devicenames/<devicename>/id_agents/<id_agent>
, which is the preferred method of getting this data.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts/devicenames/_INFOSET_TEST_/id_agent/558bb0055d7b4299c2ebe6abcc53de64a9ec4847b3f82238b3682cad575c7749/?ts_start=0"
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 4,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 5,
"timestamp": 1483629900,
"value": 9.0
}
]
$
Route /infoset/api/v1/lastcontacts/devicenames/<devicename>
/id_agents/<id_agent>
?secondsago=``seconds``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This route will retreive the most recently posted data values from a specific devicename
and id_agent
combination.
A starting UTC timestamp needs to be provided. Searches for contacts are made from starting at this time until the present.
This route does not use the cache as efficiently as /infoset/api/v1/lastcontacts/devicenames/<devicename>/id_agents/<id_agent>
, which is the preferred method of getting this data.
Field | Description |
---|---|
idx_datapoint |
The datapoint index value |
timestamp |
UTC timestamp of the most recent contact |
value |
Value of the datapoint reading at the timestamp’s point in time |
$ curl "http://SERVER_IP:6000/infoset/api/v1/lastcontacts/devicenames/_INFOSET_TEST_/id_agent/558bb0055d7b4299c2ebe6abcc53de64a9ec4847b3f82238b3682cad575c7749/?secondsago=3600"
[
{
"idx_datapoint": 2,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 3,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 4,
"timestamp": 1483629900,
"value": 9.0
},
{
"idx_datapoint": 5,
"timestamp": 1483629900,
"value": 9.0
}
]
$
Troubleshooting¶
There are a number of ways you can troubleshoot the ingester
and API
. The most accessible ways are through the log files and the API test script.
Ingester Troubleshooting¶
This section covers the various ways you can troubleshoot ingester
operation.
Ingester Logging¶
It is always good to verify the operation of the ingester
by observing changes in its log file. It is a good source of troubleshooting information.
You can see these changes as they occur by using the tail -f
command as seen below:
$ tail -f /opt/infoset-ng/log/infoset-ng.log
The location of the log file is governed by the log_directory
parameter in the configuration.
Testing Ingester Operation¶
You can test the operation of the API by using the curl
command which is often used to test basic website functionality. The example below shows how. Replace SERVER_IP
with the IP address or fully qualified DNS name.
$ curl http://SERVER_IP:6000/infoset/api/v1/status
infoset-ng API Operational.
$
The curl
response should be infoset-ng API Operational
if
successful.
Invalid Agents¶
There is the possibility that agents may be posting incorrectly formatted JSON data to the API
. You can view the contents of these invalidated files in the failures/
sub-directory of the API
cache directory. The cache directory is defined in the ingest_cache_directory:
option of the configuration file.
API Troubleshooting¶
There are a number of ways you can troubleshoot the API
. The most accessible ways are through the log file.
API Logging¶
It is always good to verify the operation of the API
by observing changes in its log file. It is a good source of troubleshooting information.
You can see these changes as they occur by using the tail -f
command as seen below:
$ tail -f /opt/infoset-ng/log/api-web.log
The location of the log file is governed by the log_directory
parameter in the configuration.
Poor or Blocked Network Connectivity¶
It is possible that there could be firewalls or intermittent connectivity causing issues to your API
you should familarize yourself with the tcpdump
command to determine whether connections are coming through.
In this example we are testing to see whether we are receiving traffic from IP address 192.168.1.100 on TCP port 6000 which the API
uses
$ sudo tcpdump -ni tcp port 6000 and host 192.168.1.100
You can also use the basic telnet
command to determine whether the remote device or network can communicate with the API
. In this example we are testing to see whether we can communicate with the API
running on a server with IP address 192.168.1.200 on the default TCP port 6000.
$ telnet 192.168.1.200 6000
Trying 192.168.1.200...
Connected to 192.168.1.200.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
If you get no response, then you need. Try this approach on both the local and remote ends of the connection. In other words, use the same command on both the remote client and API
server. If there is response on the server, but none on the client, then there is probably a connectivity issue.
You can also determine whether the API
server is running at all. Use the netstat
command on the API
server itself to determine whether it is listening on port 6000. If there is no response, then the API isn’t running.
$ netstat -ant |grep 6000
tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN
$
You should also try to use the curl
examples in the API
guide to assist further.
Best Practices¶
There are a number of best practices to consider when implementing infoset-ng
.
Use a Web Proxy Server¶
infoset-ng
uses Gunicorn
as lightweight webserver. The Gunicorn
development team strongly recommends operating Gunicorn
behind a proxy server.
Nginx Configuration¶
Although there are many HTTP proxies available, the Gunicorn
team strongly advises that you use Nginx
.
According to their website: If you choose another proxy server you need to make sure that it buffers slow clients when you use default Gunicorn workers. Without this buffering Gunicorn will be easily susceptible to denial-of-service.
A sample configuration can be found in the examples/linux/nginx
directory
We also advise that you harden your nginx
installation to reduce security risks.
Apache Configuration¶
This is the less preferred option. Use Nginx
whenever possible.
A sample configuration can be found in the examples/linux/apache
directory
We also advise that you harden your nginx
installation to reduce security risks.
Community¶
Use these channels to communicate about the project.
Mailing list¶
The user mailing list is general discussion and support list for Gunicorn users.
- To subscribe, send an email to TBD
- To unsubscribe, send an email to TBD
- Finally, to post a message to the list use the address to TBD
The archive for this list can also be browsed online.
Issue Tracking¶
Bug reports, enhancement requests and tasks generally go in the Github issue tracker.
Security Issues¶
The security mailing list is a place to report security issues. Only developers are subscribed to it. To post a message to the list use the address to TBD.