Description

================================================================ ``pyfinity`` -- Python Counter Framework for Application Metrics ================================================================ **pyfinity** aims to provide a fast memory mapped, cross platform counter framework to aid the developer in implementing and extracting application counter metrics. It provides the ability to register and increment/decrement counters for a specified namespace without expensive locking. These counters are also stored in a memory mapped file for fast writing. These files can be access by readers as required. **This means processes outside the running Python VM's can also read counters.** Examples of readers (consumers) include SQLite loggers, JSON over UDP senders as well as Zabbix agent UserParameter scripts. The goals are:: - No locking for counter writes or reads - Counter write (increment/decrement) should have minimal overhead - Readers are decoupled from writers - Readers are aware of an unstable counter if write interleaving occurs Cross Platform ============== Supported Platforms:: Linux (Tested on Fedora 16) Windows (Tested on Windows 7) Darwin (Tested on Mac OS X Lion 10.7.2 (11C74)) Multiple Python Version Support =============================== Supported Python Versions:: Python 2.6 Python 2.7 Python 3.2 Performance =========== Typical overhead for incrementing a 64bit counter is < 4usec on Python 2.6 +-----------+--------+----------+ | Action | Type | Overhead | +===========+========+==========+ | increment | uint8 | 3.2 | +-----------+--------+----------+ | increment | uint16 | 3.5 | +-----------+--------+----------+ | increment | uint32 | 3.6 | +-----------+--------+----------+ | increment | uint64 | 3.7 | +-----------+--------+----------+ Installing ========== You can get ``pyfinity`` via ``git`` as follows:: git clone https://bitbucket.org/pymaximus/pyfinity.git And install it (say into a virtualenv sandbox) via:: cd pyfinity python setup.py install Pip can also be used to install:: pip install -e git+https://bitbucket.org/pymaximus/pyfinity/#egg=pyfinity pip install -e git+git@bitbucket.org:pymaximus/pyfinity.git#egg=pyfinity You can install an RPM via Yum as follows:: python setup.py bdist_rpm sudo yum install dist/pyfinity-0.1.0-rc.1.noarch.rpm Testing ======= **pyfinity** has an initial set of unit tests covering functionality, positive and negative test cases, performance and atomicity. To run unit tests after installing, do the following:: python -m pyfinity.tests.test_all All tests should pass if your installation is correct. You can also run specific test suites after installing pyfinity, as follows:: Positive Tests python -m pyfinity.tests.test_counter_manager_versioned Negative Tests python -m pyfinity.tests.test_counter_manager_versioned_negative Performance Tests python -m pyfinity.tests.test_counter_manager_versioned_performance Concurrency Tests python -m pyfinity.tests.test_counter_manager_versioned_concurrent_process python -m pyfinity.tests.test_counter_manager_versioned_concurrent_threads Atomicity Tests python -m pyfinity.tests.test_counter_manager_versioned_atomicity All Tests python -m pyfinity.tests.test_all Using pyfinity ============== After installing pyfinity, it can be used directly in your own code. An example is shown here:: # import counter implementation from pyfinity.counter_manager_versioned import register_user from pyfinity.counter_manager_versioned import COUNTER_TYPE_UINT64 from pyfinity.counter_manager_versioned import increment_counter_uint64 # register counter namespace, type and number of counters # In this case, we want storage for 10 individual counters, of type UINT64 register_user(10, COUNTER_TYPE_UINT64, counter_group='com.example.coolapp') # step counters as required places in your code base # this example increments a 64bit unsigned integer counter at index 6 in counter namespace # 'com.example.coolapp' increment_counter_uint64(6, counter_group='com.example.coolapp') Sample Consumers ================ Pyfinity comes with some sample consumers for you to play with. Some of the currently supported consumers are listed in the following table. +----------+---------------------------------------------------+ | Consumer | Description | +==========+===================================================+ | Stdout | Periodically dumps counters to stdout. | +----------+---------------------------------------------------+ | SQLite | Periodically dumps counters to SQLite database | +----------+---------------------------------------------------+ | UDP | Periodically sends JSON formatted counter via UDP | +----------+---------------------------------------------------+ | Zabbix | Zabbix agent external script via UserParameter | +----------+---------------------------------------------------+ Please see the pyfinity.consumers package directory for more examples. If you want to add another consumer, please send me a pull request. RoadMap ======= 1. Add better consumers of pyfinity counters, For now there are some simple examples [ongoing] 2. Tweak performance [ongoing] 3. Add support for Zabbix integration. This can be a backend consumer of the mmap data [started] 4. Improve documentation of API and consumers License ======= **pyfinity** is licensed under an MIT license. See LICENSE file for details. Send me an email if you find the software useful :-) Contact ======= You can email the author Frank Singleton via b17flyboy(echo 'cat' | sed s/c//)gmail.com. To Create Sphinx Documentation ============================== You can create documentation as follows:: [frank@localhost docs]$ pwd /home/frank/workspace/pyfinity/docs [frank@localhost docs]$ export PYTHONPATH=../src/ [frank@localhost docs]$ make html Obligatory Screen Shots ======================= The following links show capabilities of the sample consumers that are bundled with pyfinity. `Zabbix Item Integration <https://bitbucket.org/pymaximus/pyfinity/raw/ecfad8d66de0/docs/zabbix_pyfinity_items_example.png>`_ `Zabbix Graph Integration <https://bitbucket.org/pymaximus/pyfinity/raw/ecfad8d66de0/docs/pyfinity_zabbix_screen.png>`_ `JSON over UDP <https://bitbucket.org/pymaximus/pyfinity/raw/ecfad8d66de0/docs/pyfinity_json_over_udp.png>`_