sndict Documentation¶
Home¶
Nested Extensions to Python dictionaries
- Free software: MIT license
- Documentation: https://sndict.readthedocs.io
- Code: https://github.com/zphang/sndict
Introduction¶
This module provides extensions to dicts
in the python standard library, providing fast and clean manipulation of nested dictionary structures. This module exposes two new dict
-types:
NestedDict
/ndict
: A light-weight wrapper fordict
s that provides additional functionality for operations on nested dictionary structures.StructuredNestedDict
/sndict
: A heavy-weight datadict
-based structure for operating on hierarchical data with rich functionality for filtering and transformation across nested levels.
Both implementations are use OrderedDict
s under the hood.
No additional dependencies are required.
Features¶
NestedDict
/ndict
:- Iterating over flattened keys and values
- Nested getting/setting operations
- Applicable to dictionaries of arbitrary and unbalanced depth
StructuredNestedDict
/sndict
:flatten
/stratify
/rearrange
methods allow for powerful and rich operations across different levels of hierarchy- Nested getting/setting operations, including intelligent filtering via
ix
- Convenient data inspection via
dim
,unique_keys
, etc
Contents:¶
Home¶
Nested Extensions to Python dictionaries
- Free software: MIT license
- Documentation: https://sndict.readthedocs.io
- Code: https://github.com/zphang/sndict
Introduction¶
This module provides extensions to dicts
in the python standard library, providing fast and clean manipulation of nested dictionary structures. This module exposes two new dict
-types:
NestedDict
/ndict
: A light-weight wrapper fordict
s that provides additional functionality for operations on nested dictionary structures.StructuredNestedDict
/sndict
: A heavy-weight datadict
-based structure for operating on hierarchical data with rich functionality for filtering and transformation across nested levels.
Both implementations are use OrderedDict
s under the hood.
No additional dependencies are required.
Features¶
NestedDict
/ndict
:- Iterating over flattened keys and values
- Nested getting/setting operations
- Applicable to dictionaries of arbitrary and unbalanced depth
StructuredNestedDict
/sndict
:flatten
/stratify
/rearrange
methods allow for powerful and rich operations across different levels of hierarchy- Nested getting/setting operations, including intelligent filtering via
ix
- Convenient data inspection via
dim
,unique_keys
, etc
Examples¶
NestedDict / ndict¶
NestedDict
/ndict
s are designed to be light-weight wrappers around existing nested dictionaries, providing additional methods for nested operations.
-
class
sndict.nesteddict.
NestedDict
(*args, **kwargs)[source]¶ Extension of OrderedDict that exposes operations on nested dicts
args kwargs
-
convert
(dict_type=None, sort_keys=True, key=None, reverse=False)[source]¶ Convert all nested dictionaries to desired type.
- dict_type: [‘ndict’, ‘dict’, ‘odict’]
- Dict-type in string format
- sort_keys: bool
- Whether to sort keys
- key: function, optional
- Key function
- reverse: bool, optional
- Whether to sort in reverse
dict, OrderedDict or NestedDict
-
filter_values
(criteria, filter_out=False)[source]¶ Filter NestedDict values by criteria.
- The criteria used in the following ways, based on type:
- slice(None): Keep all
- function: Keep if function(key) is True
- list, set: Keep if key in list/set
- other: Keep if key==other
- criteria: See above
- Filter based on criteria
- filter_out: bool
- Whether to filter in or out
NestedDict
-
flatten
(max_depth=None)[source]¶ - Iterate over a flattened NestedDict. For each value, keys along the
- DFS are accumulated as a tuple
- max_depth: int, optional
- Maximum depth to flatten by, i.e. the maximum key-tuple length
- list
- List of (key-tuple, value) pairs
-
flatten_keys
(max_depth=None)[source]¶ Iterate over a flattened NestedDict, and expose only keys. Keys along the DFS are accumulated as a tuple
- max_depth: int, optional
- Maximum depth to flatten by, i.e. the maximum key-tuple length
- list
- List of key-tuples
-
flatten_values
(max_depth=None)[source]¶ Iterate over a flattened NestedDict, and expose only values.
- max_depth: int, optional
- Maximum depth to flatten by
- list
- List of values
-
classmethod
from_flat
(data, dict_type='ndict')[source]¶ Initialize from a dict keyed by tuples Each tuple-element is taken as a key for each level in the NestedDict
- data: dict, list
- Dictionary keyed by tuples
- dict_type: [‘ndict’, ‘dict’, ‘odict’]
- Dict-type in string format, for initializing dicts at depth if they don’t exist yet
NestedDict
-
has_nested_key
(key_list)[source]¶ Check if nested keys are valid
- key_list: list
- List of keys, one for each dict depth
bool
-
iterflatten
(max_depth=None)[source]¶ - Iterate over a flattened NestedDict. For each value, keys along the
- DFS are accumulated as a tuple
- max_depth: int, optional
- Maximum depth to flatten by, i.e. the maximum key-tuple length
- iterator
- Iterator of (key-tuple, value) pairs
-
iterflatten_keys
(max_depth=None)[source]¶ Iterate over a flattened NestedDict, and expose only keys. Keys along the DFS are accumulated as a tuple
- max_depth: int, optional
- Maximum depth to flatten by, i.e. the maximum key-tuple length
- iterator
- Iterator of key-tuples
-
iterflatten_values
(max_depth=None)[source]¶ Iterate over a flattened NestedDict, and expose only values.
- max_depth: int, optional
- Maximum depth to flatten by
- iterator
- Iterator of values
-
ix
¶ Indexer that allows for indexing by nested key list e.g.
my_ndict.ix[“key1”, “key2”, “key3”]Also supports:
my_ndict.ix[“key1”]If behavior is ambiguous, use ndict.ixkeys[key_list] and ndict[key] directly instead
Indexable
-
ixkeys
¶ Indexer that allows for indexing by nested key list e.g.
my_ndict.ix[“key1”, “key2”, “key3”]Note that for a single key, a tuple/list needs to be provided, e.g.
my_ndict.ix[“key1”,]Indexable
-
map_values
(val_func)[source]¶ Apply transformations to keys and values
- val_func: function
- Function to transform values
NestedDict
-
nested_get
(key_list)[source]¶ Get value at depth
- key_list: list
- List of keys, one for each dict depth
obj
-
nested_set
(key_list, value, dict_type='ndict')[source]¶ Set a value within nested dicts, creating dicts at depth if they don’t exist yet
- key_list: list
- List of keys, one for each dict depth
- value: object
- Value to set nested
- dict_type: [‘ndict’, ‘dict’, ‘odict’]
- Dict-type in string format, for initializing dicts at depth if they don’t exist yet
-
nested_setdefault
(key_list, default=None, dict_type='ndict')[source]¶ Nested version of dict.setdefault, where a value is set only if it doesn’t already exist, creating dicts at depth if they don’t exist yet
- key_list: list
- List of keys, one for each dict depth
- default: object
- Value to set nested
- dict_type: [‘ndict’, ‘dict’, ‘odict’]
- Dict-type in string format, for initializing dicts at depth if they don’t exist yet
-
nested_update
(other_dict)[source]¶ Nested version of dict.update. Changes NestedDict in-place
- other_dict: dict
- dict to update by.
-
StructuredNestedDict / sndict¶
StructuredNestedDict
/sndict
s are heavy-weight data structures hierarchical of dict
s. Unlike the light-weight ndict
, sndict
transforms all underlying dict
s to sndict
s and enforces a fixed dictionary depth internally.
In turn, these constraints allow for more powerful operations to be performed across the different levels of hierarchy.
-
class
sndict.structurednesteddict.
StructuredNestedDict
(*args, **kwargs)[source]¶ Extension of OrderedDict that exposes advanced operations on nested dicts of fixed depth
- data: dict, or list
- Nested dictionary
- levels: int
- Number of levels
- level_names: list
- List of level names
-
convert
(dict_type=None)[source]¶ Convert all nested dictionaries to desired type.
- dict_type: [‘sndict’, ‘ndict’, ‘dict’, ‘odict’]
- Dict-type in string format
dict, OrderedDict or NestedDict
-
dim
¶ Dimensions of whole StructuredNestedDict, up to defined level
- tuple:
- Tuple of widths of nested dictionaries, one per level
-
dim_dict
¶ Dimensions of whole StructuredNestedDict as dict, up to defined level
- dict
- Dimensions keyed by level name
-
filter_key
(criteria_ls, filter_out=False, drop_empty=False)[source]¶ Filter StructuredNestedDict by criteria.
- The criteria used in the following ways, based on type:
- slice(None): Keep all
- function: Keep if function(key) is True
- list, set: Keep if key in list/set
- other: Keep if key==other
- criteria_ls: list or dict
- Filter based on criteria
- filter_out: bool
- Whether to filter in or out
- drop_empty:
- Whether to drop empty nested dictionaries (nested dictionaries with all elements filtered out)
StructuredNestedDict
-
filter_values
(criteria, filter_out=False, level=None, drop_empty=False)[source]¶ Filter StructuredNestedDict values by criteria.
- The criteria used in the following ways, based on type:
- slice(None): Keep all
- function: Keep if function(key) is True
- list, set: Keep if key in list/set
- other: Keep if key==other
- criteria: See above
- Filter based on criteria
- filter_out: bool
- Whether to filter in or out
- drop_empty:
- Whether to drop empty nested dictionaries (nested dictionaries with all elements filtered out)
StructuredNestedDict
-
flatten
(levels=-1, named=True, flattened_name=None)[source]¶ Returns an StructuredNestedDict with multiple levels flattened
- WARNING:
- If there are empty dictionaries within the levels being flattened, their keys will be lost. This is consistent with the logic of flattening - those dictionaries contain no values for a (level-tuple)-keyed dictionary.
- Note:
- .flatten(levels=0) does nothing, .flatten(levels=1) compresses 1 level (i.e. keys will be 2-ples) .flatten(levels=-1) compresses all levels
- levels: int, default=1
- Number of levels to flatten by. Defaults to flattening one level
- named: bool
- Whether output key-tuples are namedtuples
- flattened_name: str
- Name of new flattened level. Defaults to original level names joined by “___”
StructuredNestedDict
-
flatten_keys
(levels=-1, named=True)[source]¶ Returns an list with of keys of flattened dict
- WARNING:
- If there are empty dictionaries within the levels being flattened, their keys will be lost. This is consistent with the logic of flattening - those dictionaries contain no values for a (level-tuple)-keyed dictionary.
- Note:
- .flatten(levels=0) does nothing, .flatten(levels=1) compresses 1 level (i.e. keys will be 2-ples) .flatten(levels=-1) compresses all levels
- levels: int, default=1
- Number of levels to flatten by. Defaults to flattening one level
- named: bool
- Whether output key-tuples are namedtuples
list
-
flatten_values
(levels=-1)[source]¶ Returns an list with of values of flattened dict
- WARNING:
- If there are empty dictionaries within the levels being flattened, their keys will be lost. This is consistent with the logic of flattening - those dictionaries contain no values for a (level-tuple)-keyed dictionary.
- Note:
- .flatten(levels=0) does nothing, .flatten(levels=1) compresses 1 level (i.e. keys will be 2-ples) .flatten(levels=-1) compresses all levels
- levels: int, default=-1
- Number of levels to flatten by. Defaults to flattening all levels.
list
-
get_named_tuple
(levels)[source]¶ Get namedtuple class for named keys
levels: Number of levels to construct named keys for
class
-
classmethod
groupby
(data, by, levels=None, level_names=None)[source]¶ Initialize by grouping elements from list
- data: list or dictionary
- List or dictionary to group by
- by: function
- Function applied to list elements to form key elements
- levels: int
- Number of levels
- level_names: list
- List of level names
StructuredNestedDict
-
has_nested_key
(key_list)[source]¶ Check if nested keys are valid
- key_list: list
- List of keys, one for each dict depth
bool
-
iterflatten
(levels=-1, named=True)[source]¶ Returns an iterator with multiple levels flattened
- WARNING:
- If there are empty dictionaries within the levels being flattened, their keys will be lost. This is consistent with the logic of flattening - those dictionaries contain no values for a (level-tuple)-keyed dictionary.
- Note:
- .flatten(levels=0) does nothing, .flatten(levels=1) compresses 1 level (i.e. keys will be 2-ples) .flatten(levels=-1) compresses all levels
- levels: int, default=1
- Number of levels to flatten by. Defaults to flattening one level
- named: bool
- Whether output key-tuples are namedtuples
StructuredNestedDict
-
iterflatten_keys
(levels=-1, named=True)[source]¶ Returns an iterator with of keys of flattened dict
- WARNING:
- If there are empty dictionaries within the levels being flattened, their keys will be lost. This is consistent with the logic of flattening - those dictionaries contain no values for a (level-tuple)-keyed dictionary.
- Note:
- .flatten(levels=0) does nothing, .flatten(levels=1) compresses 1 level (i.e. keys will be 2-ples) .flatten(levels=-1) compresses all levels
- levels: int, default=1
- Number of levels to flatten by. Defaults to flattening one level
- named: bool
- Whether output key-tuples are namedtuples
list
-
iterflatten_values
(levels=-1)[source]¶ Returns an iterator with of values of flattened dict
- WARNING:
- If there are empty dictionaries within the levels being flattened, their keys will be lost. This is consistent with the logic of flattening - those dictionaries contain no values for a (level-tuple)-keyed dictionary.
- Note:
- .flatten(levels=0) does nothing, .flatten(levels=1) compresses 1 level (i.e. keys will be 2-ples) .flatten(levels=-1) compresses all levels
- levels: int, default=-1
- Number of levels to flatten by. Defaults to flattening all levels.
list
-
ix
¶ Indexer that allows for indexing by nested key/criteria list e.g.
my_ndict.ix[“key1”, “key2”, “key3”]or with criteria such as
my_sndict.ix[“key1”, :, lambda _: “3” in _]Also supports:
my_sndict.ix[“key1”]If behavior is ambiguous, use sndict.ixkeys[key_list] and sndict[key] directly instead
Indexable
-
ixkeys
¶ Indexer that allows for indexing by nested key list e.g.
my_ndict.ix[“key1”, “key2”, “key3”]or with criteria such as
my_sndict.ix[“key1”, :, lambda _: “3” in _]Note that for a single key, a tuple/list needs to be provided, e.g.
my_sndict.ix[“key1”,]Indexable
-
level_names
¶ Names of levels. Defaults to [“level0”, “level1”, …] is no names are provided
list
-
levels
¶ Number of levels
int
-
map
(key_func=None, val_func=None, at_level=-1, warn=False)[source]¶ Apply transformations to keys and values
- key_func: function, optional
- Function to transform keys. Defaults to identity.
- val_func: function, optional
- Function to transform values. Defaults to identity.
- at_level: int
- Level to transform keys at
- warn: bool
- Warn if dimensions of dictionary have been changed
StructuredNestedDict
-
map_keys
(key_func, at_level=-1)[source]¶ Apply transformations to keys and values
- key_func: function
- Function to transform keys
- at_level: int
- Level to transform keys at
StructuredNestedDict
-
map_values
(val_func, at_level=-1)[source]¶ Apply transformations to keys and values
- val_func: function
- Function to transform values
- at_level: int
- Level to transform values at
StructuredNestedDict
-
nested_get
(key_list)[source]¶ Get value at depth
- key_list: list
- List of keys, one for each dict depth
obj
-
nested_set
(key_list, value)[source]¶ Set a value within nested dicts, creating StructuredNestedDict at depth if they don’t exist yet
Note: Only allowed to set up to level of StructuredNestedDict
- key_list: list
- List of keys, one for each dict depth
- value: object
- Value to set nested
-
nested_setdefault
(key_list, default=None)[source]¶ Nested version of dict.setdefault. Set a value within nested dicts, creating StructuredNestedDict at depth if they don’t exist yet
Note: Only allowed to set up to level of StructuredNestedDict
- key_list: list
- List of keys, one for each dict depth
- default: object, optional
- Value to set nested
-
rearrange
(level_ls=None, level_name_ls=None)[source]¶ Rearrange levels of StructuredNestedDict Only supply either level_ls or level_name_ls.
- Note: Whether supplying level_ls or level_name_ls, the supplied list
must cover all levels contiguously from the start. I.e.
level_ls=[2, 0, 1, 3]is valid but
level_ls=[2, 0]is not.
- level_ls: list
- List of level ints. If level_ls contains level_names instead, the arguments is passed on to level_name_ls
- level_name_ls: list
- List of level names
StructuredNestedDict
-
replace_data
(data)[source]¶ Return new StructuredNestedDict with different data but same metadata
- data: dict, or list
- Nested dictionary
StructuredNestedDict
-
replace_metadata
(**kwargs)[source]¶ Return new StructuredNestedDict with different metadata but same data
- levels: int
- Number of nested levels that StructuredNestedDict will work on
- level_names: list
- List of level names
StructuredNestedDict
-
sort_keys
(cmp=None, key=None, reverse=False)[source]¶ Sort keys of StructuredNestedDict (top-level only)
- cmp: function, optional
- Comparator function
- key: function, optional
- Key function
- reverse: bool, optional
- Whether to sort in reverse
StructuredNestedDict
-
sort_values
(key=None, reverse=False)[source]¶ Sort values of StructuredNestedDict (top-level only)
- key: function
- Key function
- reverse: bool
- Whether to sort in reverse
StructuredNestedDict
-
stratify
(levels=None, stratified_names=None)[source]¶ - Increases depth (nests) of StructuredNestedDict by splitting up
- keys in the top-most level
- levels: int, default=None
- Number of levels to stratify by. Defaults to length of first key. Note: levels must be <= length of all top-level keys
- stratified_names: default=None
- Names of newly created stratified levels. Must be same length as levels.
StructuredNestedDict
-
swap_levels
(level_a, level_b)[source]¶ Swap two levels in a StructuredNestedDict
Unlike sndict.rearrange, there’s no need to be contiguous
- level_a: int or str
- level or level_name
- level_b: int or str
- level or level_name
StructuredNestedDict
Applications¶
Installation¶
Stable release¶
To install sndict, run this command in your terminal:
$ pip install sndict
This is the preferred method to install sndict, as it will always install the most recent stable release.
If you don’t have pip installed, this Python installation guide can guide you through the process.
From sources¶
The sources for sndict can be downloaded from the Github repo.
You can either clone the public repository:
$ git clone git://github.com/zphang/sndict
Or download the tarball:
$ curl -OL https://github.com/zphang/sndict/tarball/master
Once you have a copy of the source, you can install it with:
$ python setup.py install
Usage¶
To use sndict in a project:
import sndict as snd
The two primary classes:
snd.ndict
snd.sndict
Contributing¶
Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
Types of Contributions¶
Report Bugs¶
Report bugs at https://github.com/zphang/sndict/issues.
If you are reporting a bug, please include:
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug.
Fix Bugs¶
Look through the GitHub issues for bugs. Anything tagged with “bug” and “help wanted” is open to whoever wants to implement it.
Implement Features¶
Look through the GitHub issues for features. Anything tagged with “enhancement” and “help wanted” is open to whoever wants to implement it.
Write Documentation¶
sndict could always use more documentation, whether as part of the official sndict docs, in docstrings, or even on the web in blog posts, articles, and such.
Submit Feedback¶
The best way to send feedback is to file an issue at https://github.com/zphang/sndict/issues.
If you are proposing a feature:
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that this is a volunteer-driven project, and that contributions are welcome :)
Get Started!¶
Ready to contribute? Here’s how to set up sndict for local development.
Fork the sndict repo on GitHub.
Clone your fork locally:
$ git clone git@github.com:your_name_here/sndict.git
Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:
$ mkvirtualenv sndict $ cd sndict/ $ python setup.py develop
Create a branch for local development:
$ git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
$ flake8 sndict tests $ python setup.py test or py.test $ tox
To get flake8 and tox, just pip install them into your virtualenv.
Commit your changes and push your branch to GitHub:
$ git add . $ git commit -m "Your detailed description of your changes." $ git push origin name-of-your-bugfix-or-feature
Submit a pull request through the GitHub website.
Pull Request Guidelines¶
Before you submit a pull request, check that it meets these guidelines:
- The pull request should include tests.
- If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst.
- The pull request should work for Python 2.6, 2.7, 3.3, 3.4 and 3.5, and for PyPy. Check https://travis-ci.org/zphang/sndict/pull_requests and make sure that the tests pass for all supported Python versions.
Credits¶
Development Lead¶
- Jason Phang <email@jasonphang.com>