Welcome to IIQTools’s documentation!

Contents:

Getting Started

Install and Upgrade

This section provides directions to install or upgrade the IIQTools package for your instance of InsightIQ.

Note

The --upgrade flag is automatically ignored if you’re performing an initial install

With Internet connection

To install or upgrade the IIQTools package to your instance of InsightIQ run:

$ sudo su
$ easy_install --upgrade iiqtools

Without Internet connection

1. Download the package from PyPI

2. Copy the package to the InsightIQ instance

3. Run these commands, replacing <package> with the actual package name:

   $ sudo su
   $ easy_install --upgrade <package>
   

Setting up a development environment

This section is only for settings up a development environment. Only follow these directions if you intend to contribute source code to the project.

If you have the know-how to build and test IIQTools on MacOS or Windows, we’d love to get a Pull Request from you updating this section!

Linux

These are non-Python libraries required to build and test IIQTools. Use the builtin packagemanger (i.e. apt-get, yum, dnf) to install these packages.

RHEL/CentOS/Fedora:

• make
• python-devel
• postgresql-devel
• postgresql
• redhat-rpm-config

Ubuntu:

• make
• python-dev
• postgresql
• libpq-dev

Before installing the Python dependencies, it’s highly encouraged to configure a Python Virtualenv for working with IIQTools.

Once you’ve got your virtualenv configured and activated, run this command to install the dependencies for building and testing IIQTools:

$ pip install -r requirements-dev.txt

Once you’ve installed those dependencies, you can run all the unit tests like this:

$ make test

And generate the docs with:

$ make docs

Or clean up everything with:

$ make clean

Scripts & Tools

This section goes over the scripts and tools added to your InsightIQ instance when you install the IIQTools package. Each section should have some examples of what running the script looks like, or what the tool does.

iiq_gather_info

The point of this script is to provide a convenient way for users to collect logs and configuration information about InsightIQ so that remote support can investigate any issues that they are having.

Examples (assuming your running as the root users)

Printing the help message:

[root@localhost ~]$ iiq_gather_info --help
usage: iiq_gather_info [-h] [--output-dir OUTPUT_DIR] --case-number
                     CASE_NUMBER

Generate a .tar file for debugging InsightIQ

optional arguments:
  -h, --help            show this help message and exit
  --output-dir OUTPUT_DIR
                        The directory to write the .tgz file to (default:
                        /home/administrator)
  --case-number CASE_NUMBER
                        The Service Request number. Used in naming tar file.
                        (default: None)

Generating a gather:

[root@localhost ~]$ iiq_gather_info --case-number 1234
2017-09-11 14:53:45,298 - INFO - Collecting config information
2017-09-11 14:53:46,123 - INFO - Collecting log files
2017-09-11 14:53:48,126 - INFO - Log gather complete
2017-09-11 14:53:48,126 - INFO - Created log file /home/administrator/IIQLogs-sr1234-1505166825.tgz

Forgetting to provide the --case-number argument:

[root@localhost ~]$ iiq_gather_info
usage: iiq_gather_info [-h] [--output-dir OUTPUT_DIR] --case-number
                       CASE_NUMBER
iiq_gather_info: error: argument --case-number is required

Outputting the tar file to a different directory:

[root@localhost ~]$ iiq_gather_info --case-number 1234 --output-dir /datastore
2017-09-11 14:55:44,195 - INFO - Collecting config information
2017-09-11 14:55:44,451 - INFO - Collecting log files
2017-09-11 14:55:45,957 - INFO - Log gather complete
2017-09-11 14:55:45,957 - INFO - Created log file /datastore/IIQLogs-sr1234-1505166944.tgz

Contributing

We love contributions from anyone! What’s a contribution you ask, well just about anything.

Feature Requests

Got an idea for a script or tool to add to IIQTools, but need some help making it? File an issue on Github. Please keep in mind that we can’t alter InsightIQ source code, so there’s literal zero chance of adding something to the InsightIQ UI.

Bugs

What is a bug? Well, anything that impacts your ability to use IIQTools. This includes things like:

• “the script generated this traceback”
• “your docs are wrong”
• “that script didn’t do what I thought it would”

but it’s not limited to that. This package exist to make your life easier, so if part of it is confusing, even some part of the process, let us know by filing an issue on Github.

Note

If it’s some sort of code bug, please provide the version of InsightIQ, IIQTools, and any tracebacks that were generated. We need that information to fix the bug.

Source code

Wow! Thanks for wanting to add some source code to IIQTools. The process is pretty standard Github stuff:

1. Fork the repo
2. Add code to your fork
3. Put up a Pull Request to our master branch

There are a few of rules to keep in mind:

1. Write some unit tests. We’re not demanding 100% coverage, but the closer the better.
2. Don’t incorporate any additional 3rd party libraries. Some of our users cannot access the internet, and adding 3rd party libs complicates the install process.
3. Follow our docstring format (look at existing code), and update the docs directory as necessary.

Source code docs

This is all auto-generated documentation from the IIQTools source code. It defines the different modules, their APIs, and some examples.

iiqtools.exceptions

Centralized location for all custom Exceptions

exception iiqtools.exceptions.CliError(command, stdout, stderr, exit_code, message='Command Failure')

Raised when an CLI command has a non-zero exit code.

Parameters:

• command (String) – The CLI command that was ran
• stdout (String) – The output from the standard out stream
• stderr (String) – The output from the standard error stream
• exit_code – The exit/return code from the command

exception iiqtools.exceptions.DatabaseError(message, pgcode)

Raised when an error occurs when interacting with the InsightIQ database

Attribute pgcode:  The error code used by PostgreSQL. https://www.postgresql.org/docs/9.3/static/errcodes-appendix.html Attribute message:  The error message

iiqtools.utils

The utils module for the IIQTools package. These are common-need/generic functions for making scripts/tools for use with InsightIQ.

cli_parsers

This module contains functions that parse stdout of a CLI command into a usable Python data structure.

iiqtools.utils.cli_parsers.df_to_dict(output)

Parse the output from the command df into a dictionary

Returns:Dictionary Parameters:output (String) – Required The pile of stuff outputted by running df

iiqtools.utils.cli_parsers.ifconfig_to_dict(ifconfig_output)

Parse the output from the command ifconfig into a dictionary

Returns:Dictionary Parameters:ifconfig_output (String) – Required The pile of stuff outputted by running ifconfig

iiqtools.utils.cli_parsers.iiq_version_to_dict(cli_output)

Parse the output from the command rpm -q isilon-insightiq into a dictionary

Returns:Dictionary Parameters:cli_output (String) – Required The pile of stuff outputted by running rpm -q isilon-insightiq

iiqtools.utils.cli_parsers.memory_to_dict(output)

Parse the output from the command free -m into a dictionary

Returns:Dictionary Parameters:output (String) – Required The pile of stuff outputted by running free -m

database

Utilities for interacting with the InsightIQ database

class iiqtools.utils.database.Column

A database column consiting of the column’s name, and it’s type

Type:

namedtuple

Parameters:

• name – The column’s name
• type – The database type for the given column (i.e. int, float, double)

class iiqtools.utils.database.Database(user='postgres', dbname='insightiq')

Simplifies communication with the database.

The goal of this object is to make basic interactions with the database simpler than directly using the psycopg2 library. It does this by reducing the number of API methods, providing handy built-in methods for common needs (like listing tables of a database), auto-commit of transactions, and auto-rollback of bad SQL transactions. This object is not indented for power users, or long lived processes (like the InsightIQ application); it’s designed for shorter lived “scripts”.

Parameters:

• user (String, default postgres) – The username when connection to the databse
• dbname (String, default insightiq) – The specific database to connection to. InsightIQ utilizes a different database for every monitored cluster, plus one generic database for the application (named “insightiq”).

close()

Disconnect from the database

cluster_databases()

Obtain a list of all the cluster databases

Returns:List

execute(sql, params=None)

Run a single SQL command

Returns:

Generator

Parameters:

• sql (String) – Required The SQL syntax to execute
• params (Iterable) – The values to use in a parameterized SQL query

This method is implemented as a Python Generator: https://wiki.python.org/moin/Generators This means you are suppose to iterate over the results:

db = Database()
for row in db.execute("select * from some_table;"):
    print row

If you want all the rows as a single thing, just use list:

db = Database()
data = list(db.execute("select * from some_table;")

But WARNING that might cause your program to run out of memory and crash! That reason is why this method is a generator by default ;)

To perform a parameterized query (i.e. avoid SQL injection), provided the parameters as an iterable:

db = Database()
# passing in "foo_column" alone would try and string format every
# character of "foo_column" into your SQL statement.
# Instead, make "foo_column" a tuple by wrapping it like ("foo_column",)
# Note: the trailing comma is required.
data = list(db.execute("select %s from some_table", ("foo_column",)))

executemany(sql, params)

Run the SQL for every iteration of the supplied params

This method behaves exactly like execute, except that it can perform multiple SQL commands in a single transaction. The point of this method is so you can retain Atomicity when you must execute the same SQL with different parameters. This method isn’t intended to be faster than looping over the normal execute method with the different parameters.

Returns:

Generator

Parameters:

• sql (String) – Required The SQL syntax to execute
• params (Iterable) – Required The parameterized values to iterate

isolation_level

Set the isolation level of your connnection to the database

primary_key(table)

Given a table, return the primary key

Note

If you supply a timeseries table that DOES NOT have an EPOC timestamp in the name, you will get zero results. For timeseries tables, supply a table that contains the EPOC timestamps to see the primary key.

Returns:Tuple of namedtuples -> (Column(name, type), Column(name, type)) Parameters:table (String) – Required The table to obtain the primary key from

table_schema(table)

Given a table, return the schema for that table

Returns:Tuple of namedtuples -> (Column(name, type), Column(name, type)) Parameters:table (String) – Required The table to obtain the primary key from

tables()

Obtain a list of all the tables for the database you’re connected to

Returns:List

generic

This module contains miscellaneous utility functions

iiqtools.utils.generic.printerr(message)

Just like print(), but outputs to stderr.

Returns:None Parameters:message (PyObject) – The thing to write to stderr

logger

iiqtools.utils.logger.get_logger(log_path=None, stream_lvl=0, file_lvl=20)

Factory for making logging objects

The verbosity of the logs are configurable as defined by the official Python documentation: https://docs.python.org/2/library/logging.html#logging-levels

Returns:

logging.Logger

Raises:

AssertionError on bad parameter input

Parameters:

• log_path (String) – Required The absolute file path to write logs to
• stream_lvl (Integer, default 20) – Set to print log messages to the terminal.
• file_lvl – How verbose the log file messages are. This value cannot be zero.

shell

This module reduces boilerplate when interacting with the command shell, i.e. BASH.

Example A:

>>> result = shell.run_cmd('ls')
>>> print result.stdout
README.txt
someOtherFile.txt
>>> print result.stderr

>>> result.exit_code
0

Example B:

>>> # run_cmd does not support the Unix Pipeline
>>> result = shell.run_cmd('cat foo.txt | grep "not supported"')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    raise CliError(cli_syntax, stdout, stderr, exit_code)
iiqtools.exceptions.CliError: Command Failure: cat foo.txt | grep "not supported"

class iiqtools.utils.shell.CliResult

The outcome from running a CLI command

Type:

collections.namedtuple

Parameters:

• command (String) – The CLI command that was ran
• stdout (String) – The output from the standard out stream
• stderr (String) – The output from the standard error stream
• exit_code – The exit/return code from the command

iiqtools.utils.shell.run_cmd(cli_syntax)

Execute a simple CLI command.

This function blocks until the CLI command returns and does not support the Unix pipeline.

Returns:CliResult - namedtuple Raises:CliError (when exit code is not zero) Parameters:cli_syntax (String) – The CLI command to run.

iiqtools.iiq_gather_info

This module contains all the business logic for collecting logs and configuration information about InsightIQ for remote troubleshooting.

iiqtools.iiq_gather_info.add_from_memory(the_tarfile, data_name, data)

Simplify adding in-memory information to the tar file

Returns:

None

Parameters:

• the_tarfile (tarfile.open) – The open tarfile object
• data_name (String) – The reference to the data; i.e. it’s name when you uncompress the file
• data (String) – The contents of the in-memory information

iiqtools.iiq_gather_info.call_iiq_api(uri)

Make an internal API call to the InsightIQ API

Returns:JSON String Parameters:uri (string) – The API end point to call

iiqtools.iiq_gather_info.check_path(cli_value)

Validate that the supplied path is an actual file system path

Returns:String Parameters:cli_value (String) – The value supplied by the end user

iiqtools.iiq_gather_info.cli_cmd_info(command, parser)

Standardizes the JSON format for any data collected via a CLI command

Returns:JSON String Parameters:command (String) – The CLI command to execute

iiqtools.iiq_gather_info.clusters_info()

Obtain a pile of data about all monitored clusters in InsightIQ

Returns:JSON String

iiqtools.iiq_gather_info.datastore_info()

Obtain data about the datastore for InsightIQ

Returns:JSON String

iiqtools.iiq_gather_info.get_tarfile(output_dir, case_number, the_time=None)

Centralizes logic for making tgz file for InsightIQ logs

Returns:

tarfile.TarFile

Parameters:

• output_dir (String) – Required The directory to save the tar file in
• case_number (String or Integer) – Required The SR that the logs are for; used in file name.
• the_time (EPOC time stamp) – An optional EPOC timestamp to use when naming the file. If not supplied, this function calls time.time().

iiqtools.iiq_gather_info.ifconfig_info()

Obtain data about the network interfaces on the host OS

Returns:JSON String

iiqtools.iiq_gather_info.iiq_version_info()

Obtain info about the version of InsightIQ installed on the host OS

Returns:JSON String

iiqtools.iiq_gather_info.ldap_info()

Obtain the config for LDAP in InsightIQ

Returns:JSON String

iiqtools.iiq_gather_info.main(the_cli_args)

Entry point for running script

iiqtools.iiq_gather_info.memory_info()

Obtain data about RAM on the host OS

Returns:JSON String

iiqtools.iiq_gather_info.mount_info()

Obtain data about mounted file systems on the host OS

Returns:JSON String

iiqtools.iiq_gather_info.parse_cli(cli_args)

Handles parsing the CLI, and gives us –help for (basically) free

Returns:argparse.Namespace Parameters:cli_args (List) – The arguments passed to the script

iiqtools.iiq_gather_info.reports_info()

Obtain info about any scheduled reports in InsightIQ

Returns:JSON String

Indices and tables

• Index
• Module Index
• Search Page