Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

Furtive¶

File Integrity Verification (furtive) aims to ensure long term data integrity verification for digital archival purposes. The idea is to create a manifest, or hash list, of all the files of which you wish to confirm integrity. Once a manifest has been created, a user can then confirm the integrity of files at any point in the future.

The documentation is available on Read The Docs at furtive.readthedocs.org

Contents¶

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install

CLI Reference¶

Command Line Interface (or Tool) reference.

Manage a Furtive manifest

usage: furtive [-h] [--basedir BASEDIR] [--manifest MANIFEST_PATH]
               [--log-level {debug,info,warn,error,critical}]
               [--exclude PATTERN] [--quiet] [--report-output FILE_NAME]
               [--version]
               {create,compare,check}

Positional arguments:

action

Which action to perform: compare - compare the current state of the files on the file system with the recorded state in the manifest file. Status code is 0 if the comparison was successful. check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. create - create a new manifest from the files inthe directory specified by the –basedir argument.

Possible choices: create, compare, check

Options:

`--basedir=.`	Directory containing files that will be checked. Default: .
`--manifest`	Location of the manifest file. Manifests may be located outside the directory indicated by –basedir. Must provide path and filename of the manifest file. Default: <basedir>/.manifest.yaml
`--log-level=info`
	verbosity of furtive Possible choices: debug, info, warn, error, critical
`--exclude=[]`	Patterns to exclude files and directories from manifest. Can have multiple occurances of this argument. Excludes are not stored in the manifest so it is up to the user to provide the same arguments every run. Patterns are evaluated as UNIX shell-style wildcard characters. See the [fnmatch documentation](https://docs.python.org/2/library/fnmatch.html)for more information. It is important to note that exclusions are not stored. Therefore ,they must be specified for every run of `furtive`. Otherwise, the files which were previously excluded will be included and will show up as files added to the manifest.
`--quiet=False`	Only print out critial error messages. Do not print a report at the end of a compare run. Using this argument will override the log-level and set it to “critical”. Only acceptions will be printed to terminal. The return code will be the only way to know if a Manifest has changed. This is useful for scripting such as a cron based manifest checks. Useful with the check command.
`--report-output=-`
	File to print the diff report to. - for stdout.This can be consumed by other scripts todetermine exactly what has changed within themanifestDefault: -
`--version`	show program’s version number and exit

API Reference¶

API Reference
- Furtive Class
- Sub-Modules
  - Hasher
  - Manifest

This document is for developers of furtive, it contains the API functions

Furtive Class¶

Furtive - File Integrity Verification System

class furtive.Furtive(base_dir, manifest_path, exclude=None)[source]¶

Bases: object

Furtive is an application which stores file state and allows users to verify the state in the future. Example use cases include file archives and file transport.

If the manifest file exists, it will be automatcally loaded. Calling create() will overwrite the existing manifest in memory as well as the file.

Parameters:	base_dir (str) – Base directory to use for the manifest. Can be a full or relative path. manifest_path (str) – Path to the manifest file. Can be a full or relative path. exclude (list) – list containing patterns to use to exclude files from the manifest.

compare()[source]¶

Compare the hashes in the database with the current hashes of files on the file system.

Returns:	Dictionary of added, deleted, and changed files.
Return type:	dict

create()[source]¶

Create and save a new manifest.

The contents of the new Manfiest() will be saved to manifest_path.

Returns:	None

Sub-Modules¶

Hasher¶

Manages the hashing of files

class furtive.hasher.HashDirectory(directory, exclude=None)[source]¶

Bases: object

Object to manage hashing files in a directory.

This object is responsible for walking the directory tree and adding each file to a list. Once the directory walk has compelted, each file path is passed to hash_task(). After each file has been hashed, this object will then create a Python dictionary of files with their associated hash.

Parameters:	directory (str) – Path to directory containing files exclude (list) – list containing patterns to use to exclude files from the manifest.
Returns:	Dictionary of file:hash
Return type:	dict

excluded(file_path)[source]¶

Should the file be excluded from the manifest?

Determines if a file should be excluded based on UNIX style pattern matching. Think *, ?, and [] sequences.

For matchers, see https://docs.python.org/2/library/fnmatch.html

Parameters:	file_path (str) – path of the file to match against.
Returns:	True or False indicating if the file should be excluded from the list of files containted within the manifest.
Return type:	bool

hash_files()[source]¶

Orchestrates the discovery and hashing of files.

Note: This method only supports the md5 hashing algorithm

furtive.hasher.hash_task(file_path, hash_algorithm='md5')[source]¶

Responsible for hashing a file.

This function reads in the file file_path in small chuncks the size of the hash algorithm’s block size in order to avoid running out of memory. This means that this function should be able to read any file irregardless of the size.

Parameters:	file_path (str) – path of file to hash hash_algorithm (str) – the hashing algorithm to use. All options available in hashlib.algorithms should work. See: https://docs.python.org/2/library/hashlib.html
Returns:	hash of file
Return type:	dict

furtive.hasher.initializer(terminating_)[source]¶: Method to make terminating a global variable so that it is inherited by child processes.

Manifest¶

Manifest of files and their hashes

class furtive.manifest.Manifest(directory, manifest_file, exclude=None)[source]¶

Bases: object

Manifest of files and the associated hashes.

Parameters:

directory – directory which will serve as the root for the manifest. All files under the directory will be hashed and added to or compared with the manifest.
type – str
manifest_file – file location of the manifest file. This is the path which will be used for the create() and compare() methods. If the file exists, the create() method will overwrite it.
exclude (list) – list containing patterns to use to exclude files from the manifest.

create()[source]¶: Creates a new manifest from the directory by calling furtive.hasher.HashDirectory() and placing the return dictionary in to Manifest.manifest.

is_empty()[source]¶

Determines if the manifest within memory is empty.

This simply checks to see if the manifest is None.

Returns:	True if manifest is empty, False otherwise.
Return type:	bool

load()[source]¶

Load a manifest from the manifest file.

This method will open the manfiest YAML file and load it in to the manifest object variable.

save()[source]¶

Save the manifest to the manifest file.

Open a YAML file and dump the contents of the manifest to it.

Requirements¶

Python 2.7, 3.4, or 3.5

Getting Started¶

To install furtive, run pip install furtive.

CLI Usage¶

See the CLI Reference for more information about available command line arguments.

Use Case Example¶

Suppose you have a million digital photos in a directory called my-photos that you have taken over the years. You would like to know if the files begin to decay due to hardware failure or something else. Alternatively, you may wish to have reassurance that your photos have not become corrupted while being stored in a cloud backup solution such as S3 or Glacier.

To record the current state of the files, run furtive --basedir my-photos create

This command creates the file .manifest.yaml in the my-photos/ directory. The location and name of this file can be changed by using the --manifest argument.

At this point, you can be sure that you will know if a file has changed. To check the files on the file system to the recorded state in the manifest, run furtive --basedir my-photos check. The application will output a list of files which have been added, removed, or changed. This output is YAML format so it should be easy to parse. Additionally, furtive will exit with 1 indicating the check failed. This command can be placed in a cron job and setup to send a notification if a file has changed.

Actions¶

There are a few actions which can be performed by furtive.

create - create a new manifest from the files in the directory specified by the --basedir argument.
compare - compare the current state of the files on the file system with the recorded state in the manifest file. A YAML based report will be created detailing which files have changed and which files have been added or removed. Status code is 0 if the comparison was successful.
check - check the integrity of files listed in the manifest. Same as compare but exits with status code 1 if there are changes to the files included in the manifest. That is, if any file hash changes or if files are added or removed, the application will exit with a status code of 1 to indicate there are changes. This action can be useful for scripting. For example, to run a nightly cron check of a manifest. A YAML based report will be generated as well.

Tests¶

This application comes with tests. To run them, ensure you have tox installed (pip install tox). Then you can run tox to run the tests.

To build the docs, run tox -e docs. The HTML docs will be generated in the .tox/docs/tmp/html/ directory.

Faster YAML¶

By default, furtive will install and use the full Python implementation of the YAML parser which is very slow. In a testing environment, the Python implementation of the YAML loader took 1 minute to parse a 187,000 line furtive manifest file. By contrast, when the LibYaml parser was used, the loader took only 5 seconds to parse the same file.

To install the faster parser, perform the following steps:

Follow the instructions from the LibYaml website to download and install the latest release of libyaml.
Reinstall the PyYAML package by downloading the latest tar from the PyYAML website and running python setup.py --with-libyaml install