dit: discrete information theory¶

Quickstart¶

The basic usage of dit corresponds to creating distributions, modifying them if need be, and then computing properties of those distributions. First, we import:

In [1]: import dit

Suppose we have a really thick coin, one so thick that there is a reasonable chance of it landing on its edge. Here is how we might represent the coin in dit.

In [2]: d = dit.Distribution(['H', 'T', 'E'], [.4, .4, .2])

In [3]: print(d)
Class:          Distribution
Alphabet:       ('E', 'H', 'T') for all rvs
Base:           linear
Outcome Class:  str
Outcome Length: 1
RV Names:       None

x   p(x)
E   0.2
H   0.4
T   0.4

Calculate the probability of \(H\) and also of the combination: \(H~\mathbf{or}~T\).

In [4]: d['H']
Out[4]: 0.40000000000000002

In [5]: d.event_probability(['H','T'])
Out[5]: 0.80000000000000004

Calculate the Shannon entropy and extropy of the joint distribution.

In [6]: dit.shannon.entropy(d)
Out[6]: 1.5219280948873621

In [7]: dit.other.extropy(d)
Out[7]: 1.1419011889093373

Create a distribution representing the \(\mathbf{xor}\) logic function. Here, we have two inputs, \(X\) and \(Y\), and then an output \(Z = \mathbf{xor}(X,Y)\).

In [8]: import dit.example_dists

In [9]: d = dit.example_dists.Xor()

In [10]: d.set_rv_names(['X', 'Y', 'Z'])

In [11]: print(d)
Class:          Distribution
Alphabet:       ('0', '1') for all rvs
Base:           linear
Outcome Class:  str
Outcome Length: 3
RV Names:       ('X', 'Y', 'Z')

x     p(x)
000   0.25
011   0.25
101   0.25
110   0.25

Calculate the Shannon mutual informations \(\I[X:Z]\), \(\I[Y:Z]\), and \(\I[X,Y:Z]\).

In [12]: dit.shannon.mutual_information(d, ['X'], ['Z'])
Out[12]: 0.0

In [13]: dit.shannon.mutual_information(d, ['Y'], ['Z'])
Out[13]: 0.0

In [14]: dit.shannon.mutual_information(d, ['X', 'Y'], ['Z'])
Out[14]: 1.0

Calculate the marginal distribution \(P(X,Z)\). Then print its probabilities as fractions, showing the mask.

In [15]: d2 = d.marginal(['X', 'Z'])

In [16]: print(d2.to_string(show_mask=True, exact=True))
Class:          Distribution
Alphabet:       ('0', '1') for all rvs
Base:           linear
Outcome Class:  str
Outcome Length: 2 (mask: 3)
RV Names:       ('X', 'Z')

x     p(x)
0*0   1/4
0*1   1/4
1*0   1/4
1*1   1/4

Convert the distribution probabilities to log (base 3.5) probabilities, and access its probability mass function.

In [17]: d2.set_base(3.5)

In [18]: d2.pmf
Out[18]: array([-1.10658951, -1.10658951, -1.10658951, -1.10658951])

Draw 5 random samples from this distribution.

In [19]: d2.rand(5)
Out[19]: ['01', '10', '00', '01', '00']

Enjoy!

Basic Notation¶

A random variable \(X\) consists of outcomes \(x\) from an alphabet \(\mathcal{X}\). As such, we write the entropy of a distribution as \(\H[X] = \sum_{x \in \mathcal{X}} p(x) \log_2 p(x)\), where \(p(x)\) denote the probability of the outcome \(x\) occuring.

Many distributions are joint distribution. In the absence of variable names, we index each random variable with a subscript. For example, a distribution over three variables is written \(X_0X_1X_2\). As a shorthand, we also denote those random variables as \(X_{0:3}\), meaning start with \(X_0\) and go through, but not including \(X_3\) — just like python slice notation.

If we ever need to describe an infinitely long chain of variables we drop the index from the side that is infinite. So \(X_{:0} = \ldots X_{-3}X_{-2}X_{-1}\) and \(X_{0:} = X_0X_1X_2\ldots\). For an arbitrary set of indices \(A\), the corresponding collection of random variables is denoted \(X_A\). For example, if \(A = \{0,2,4\}\), then \(X_A = X_0 X_2 X_4\). The complement of \(A\) (with respect to some universal set) is denoted \(\bar{A}\).

Furthermore, we define \(0 \log_2 0 = 0\).

Advanced Notation¶

When there exists a function \(Y = f(X)\) we write \(X \imore Y\) meaning that \(X\) is informationally richer than \(Y\). Similarly, if \(f(Y) = X\) then we write \(X \iless Y\) and say that \(X\) is informationally poorer than \(Y\). If \(X \iless Y\) and \(X \imore Y\) then we write \(X \ieq Y\) and say that \(X\) is informationally equivalent to \(Y\). Of all the variables that are poorer than both \(X\) and \(Y\), there is a richest one. This variable is known as the meet of \(X\) and \(Y\) and is denoted \(X \meet Y\). By definition, \(\forall Z s.t. Z \iless X\) and \(Z \iless Y, Z \iless X \meet Y\). Similarly of all variables richer than both \(X\) and \(Y\), there is a poorest. This variable is known as the join of \(X\) and \(Y\) and is denoted \(X \join Y\). The joint random variable \((X,Y)\) and the join are informationally equivalent: \((X,Y) \ieq X \join Y\).

Lastly, we use \(X \mss Y\) to denote the minimal sufficient statistic of \(X\) about the random variable \(Y\).

Numpy-based ScalarDistribution¶

ScalarDistribution specific stuff.

ScalarDistribution.__init__(outcomes, pmf=None, sample_space=None, base=None, prng=None, sort=True, sparse=True, trim=True, validate=True)¶

Initialize the distribution.

Parameters:

outcomes (sequence, dict) – The outcomes of the distribution. If outcomes is a dictionary, then the keys are used as outcomes, and the values of the dictionary are used as pmf instead. Note: an outcome is any hashable object (except None) which is equality comparable. If sort is True, then outcomes must also be orderable. pmf (sequence) – The outcome probabilities or log probabilities. If None, then outcomes is treated as the probability mass function and the outcomes are consecutive integers beginning from zero. sample_space (sequence) – A sequence representing the sample space, and corresponding to the complete set of possible outcomes. The order of the sample space is important. If None, then the outcomes are used to determine the sample space instead. base (float, None) – If pmf specifies log probabilities, then base should specify the base of the logarithm. If ‘linear’, then pmf is assumed to represent linear probabilities. If None, then the value for base is taken from ditParams[‘base’]. prng (RandomState) – A pseudo-random number generator with a rand method which can generate random numbers. For now, this is assumed to be something with an API compatible to NumPy’s RandomState class. This attribute is initialized to equal dit.math.prng. sort (bool) – If True, then the sample space is sorted before finalizing it. Usually, this is desirable, as it normalizes the behavior of distributions which have the same sample space (when considered as a set). Note that addition and multiplication of distributions is defined only if the sample spaces (as tuples) are equal. sparse (bool) – Specifies the form of the pmf. If True, then outcomes and pmf will only contain entries for non-null outcomes and probabilities, after initialization. The order of these entries will always obey the order of sample_space, even if their number is not equal to the size of the sample space. If False, then the pmf will be dense and every outcome in the sample space will be represented. trim (bool) – Specifies if null-outcomes should be removed from pmf when make_sparse() is called (assuming sparse is True) during initialization. validate (bool) – If True, then validate the distribution. If False, then assume the distribution is valid, and perform no checks.

Raises:

InvalidDistribution – If the length of values and outcomes are unequal. See validate() for a list of other potential exceptions.

Numpy-based Distribution¶

The primary method of constructing a distribution is by supplying both the outcomes and the probability mass function:

In [1]: from dit import Distribution

In [2]: outcomes = ['000', '011', '101', '110']

In [3]: pmf = [1/4]*4

In [4]: xor = Distribution(outcomes, pmf)

In [5]: print(xor)
Class:          Distribution
Alphabet:       ('0', '1') for all rvs
Base:           linear
Outcome Class:  str
Outcome Length: 3
RV Names:       None

x     p(x)
000   0.25
011   0.25
101   0.25
110   0.25

Another way to construct a distribution is by supplying a dictionary mapping outcomes to probabilities:

In [6]: outcomes_probs = {'000': 1/4, '011': 1/4, '101': 1/4, '110': 1/4}

In [7]: xor2 = Distribution(outcomes_probs)

In [8]: print(xor2)
Class:          Distribution
Alphabet:       ('0', '1') for all rvs
Base:           linear
Outcome Class:  str
Outcome Length: 3
RV Names:       None

x     p(x)
000   0.25
011   0.25
101   0.25
110   0.25

Distribution.__init__(outcomes, pmf=None, sample_space=None, base=None, prng=None, sort=True, sparse=True, trim=True, validate=True)¶

Initialize the distribution.

Parameters:

outcomes (sequence, dict) – The outcomes of the distribution. If outcomes is a dictionary, then the keys are used as outcomes, and the values of the dictionary are used as pmf instead. The values will not be used if probabilities are passed in via pmf. Outcomes must be hashable, orderable, sized, iterable containers. The length of an outcome must be the same for all outcomes, and every outcome must be of the same type. pmf (sequence, None) – The outcome probabilities or log probabilities. pmf can be None only if outcomes is a dict. sample_space (sequence, CartesianProduct) – A sequence representing the sample space, and corresponding to the complete set of possible outcomes. The order of the sample space is important. If None, then the outcomes are used to determine a Cartesian product sample space instead. base (float, None) – If pmf specifies log probabilities, then base should specify the base of the logarithm. If ‘linear’, then pmf is assumed to represent linear probabilities. If None, then the value for base is taken from ditParams[‘base’]. prng (RandomState) – A pseudo-random number generator with a rand method which can generate random numbers. For now, this is assumed to be something with an API compatibile to NumPy’s RandomState class. This attribute is initialized to equal dit.math.prng. sort (bool) – If True, then each random variable’s alphabets are sorted before they are finalized. Usually, this is desirable, as it normalizes the behavior of distributions which have the same sample spaces (when considered as a set). Note that addition and multiplication of distributions is defined only if the sample spaces are compatible. sparse (bool) – Specifies the form of the pmf. If True, then outcomes and pmf will only contain entries for non-null outcomes and probabilities, after initialization. The order of these entries will always obey the order of sample_space, even if their number is not equal to the size of the sample space. If False, then the pmf will be dense and every outcome in the sample space will be represented. trim (bool) – Specifies if null-outcomes should be removed from pmf when make_sparse() is called (assuming sparse is True) during initialization. validate (bool) – If True, then validate the distribution. If False, then assume the distribution is valid, and perform no checks.

Raises:

InvalidDistribution – If the length of values and outcomes are unequal. If no outcomes can be obtained from pmf and outcomes is None. See validate() for a list of other potential exceptions.

To verify that these two distributions are the same, we can use the is_approx_equal method:

In [9]: xor.is_approx_equal(xor2)
Out[9]: True

Distribution.is_approx_equal(other, rtol=None, atol=None)¶

Returns True is other is approximately equal to this distribution.

For two distributions to be equal, they must have the same sample space and must also agree on the probabilities of each outcome.

Parameters:	other (distribution) – The distribution to compare against. rtol (float) – The relative tolerance to use when comparing probabilities. See dit.math.close() for more information. atol (float) – The absolute tolerance to use when comparing probabilities. See dit.math.close() for more information.

Notes

The distributions need not have the length, but they must have the same base.

Marginal¶

Distribution.marginal(rvs, rv_mode=None)¶

Returns a marginal distribution.

Parameters:	rvs (list) – The random variables to keep. All others are marginalized. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of self._rv_mode is consulted.
Returns:	d – A new joint distribution with the random variables in rvs kept and all others marginalized.
Return type:	joint distribution

Conditional¶

Distribution.condition_on(crvs, rvs=None, rv_mode=None, extract=False)¶

Returns distributions conditioned on random variables crvs.

Optionally, rvs specifies which random variables should remain.

NOTE: Eventually this will return a conditional distribution.

Parameters:

crvs (list) – The random variables to condition on. rvs (list, None) – The random variables for the resulting conditional distributions. Any random variable not represented in the union of crvs and rvs will be marginalized. If None, then every random variable not appearing in crvs is used. rv_mode (str, None) – Specifies how to interpret crvs and rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random varible names. If None, then the value of self._rv_mode is consulted, which defaults to ‘indices’. extract (bool) – If the length of either crvs or rvs is 1 and extract is True, then instead of the new outcomes being 1-tuples, we extract the sole element to create scalar distributions.

Returns:

cdist (dist) – The distribution of the conditioned random variables. dists (list of distributions) – The conditional distributions for each outcome in cdist.

Examples

First we build a distribution P(X,Y,Z) representing the XOR logic gate.

>>> pXYZ = dit.example_dists.Xor()
>>> pXYZ.set_rv_names('XYZ')

We can obtain the conditional distributions P(X,Z|Y) and the marginal of the conditioned variable P(Y) as follows:

>>> pY, pXZgY = pXYZ.condition_on('Y')

If we specify rvs='Z', then only ‘Z’ is kept and thus, ‘X’ is marginalized out:

>>> pY, pZgY = pXYZ.condition_on('Y', rvs='Z')

We can condition on two random variables:

>>> pXY, pZgXY = pXYZ.condition_on('XY')

The equivalent call using indexes is:

>>> pXY, pZgXY = pXYZ.condition_on([0, 1], rv_mode='indexes')

Join¶

We can construct the join of two random variables:

Where \(\min\) is understood to be minimizing with respect to the entropy.

join(dist, rvs, rv_mode=None, int_outcomes=True)¶

Returns the distribution of the join of random variables defined by rvs.

Parameters:	dist (Distribution) – The distribution which defines the base sigma-algebra. rvs (list) – A list of lists. Each list specifies a random variable to be joined with the other lists. Each random variable can defined as a series of unique indexes. Multiple random variables can use the same index. For example, [[0,1],[1,2]]. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted. int_outcomes (bool) – If True, then the outcomes of the join are relabeled as integers instead of as the atoms of the induced sigma-algebra.
Returns:	d – The distribution of the join.
Return type:	ScalarDistribution

insert_join(dist, idx, rvs, rv_mode=None)¶

Returns a new distribution with the join inserted at index idx.

The join of the random variables in rvs is constructed and then inserted into at index idx.

Parameters:	dist (Distribution) – The distribution which defines the base sigma-algebra. idx (int) – The index at which to insert the join. To append the join, set idx to be equal to -1 or dist.outcome_length(). rvs (list) – A list of lists. Each list specifies a random variable to be met with the other lists. Each random variable can defined as a series of unique indexes. Multiple random variables can use the same index. For example, [[0,1],[1,2]]. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted.
Returns:	d – The new distribution with the join at index idx.
Return type:	Distribution

Meet¶

We can construct the meet of two random variabls:

Where \(\max\) is understood to be maximizing with respect to the entropy.

meet(dist, rvs, rv_mode=None, int_outcomes=True)¶

Returns the distribution of the meet of random variables defined by rvs.

Parameters:	dist (Distribution) – The distribution which defines the base sigma-algebra. rvs (list) – A list of lists. Each list specifies a random variable to be met with the other lists. Each random variable can defined as a series of unique indexes. Multiple random variables can use the same index. For example, [[0,1],[1,2]]. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted. int_outcomes (bool) – If True, then the outcomes of the meet are relabeled as integers instead of as the atoms of the induced sigma-algebra.
Returns:	d – The distribution of the meet.
Return type:	ScalarDistribution

insert_meet(dist, idx, rvs, rv_mode=None)¶

Returns a new distribution with the meet inserted at index idx.

The meet of the random variables in rvs is constructed and then inserted into at index idx.

Parameters:	dist (Distribution) – The distribution which defines the base sigma-algebra. idx (int) – The index at which to insert the meet. To append the meet, set idx to be equal to -1 or dist.outcome_length(). rvs (list) – A list of lists. Each list specifies a random variable to be met with the other lists. Each random variable can defined as a series of unique indexes. Multiple random variables can use the same index. For example, [[0,1],[1,2]]. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted.
Returns:	d – The new distribution with the meet at index idx.
Return type:	Distribution

Minimal Sufficient Statistic¶

This method constructs the minimal sufficient statistic of \(X\) about \(Y\): \(X \mss Y\):

Again, \(\min\) is understood to be over entropies.

mss(dist, rvs, about=None, rv_mode=None, int_outcomes=True)¶

Parameters:	dist (Distribution) – The distribution which defines the base sigma-algebra. rvs (list) – A list of random variables to be compressed into a minimal sufficient statistic. about (list) – A list of random variables for which the minimal sufficient static will retain all information about. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted. int_outcomes (bool) – If True, then the outcomes of the minimal sufficient statistic are relabeled as integers instead of as the atoms of the induced sigma-algebra.
Returns:	d – The distribution of the minimal sufficient statistic.
Return type:	ScalarDistribution

Examples

>>> d = Xor()
>>> print(mss(d, [0], [1, 2]))
Class:    ScalarDistribution
Alphabet: (0, 1)
Base:     linear
x   p(x)
0   0.5
1   0.5

insert_mss(dist, idx, rvs, about=None, rv_mode=None)¶

Inserts the minimal sufficient statistic of rvs about about into dist at index idx.

Parameters:	dist (Distribution) – The distribution which defines the base sigma-algebra. idx (int) – The location in the distribution to insert the minimal sufficient statistic. rvs (list) – A list of random variables to be compressed into a minimal sufficient statistic. about (list) – A list of random variables for which the minimal sufficient static will retain all information about. rv_mode (str, None) – Specifies how to interpret the elements of rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted.
Returns:	d – The distribution dist modified to contain the minimal sufficient statistic.
Return type:	Distribution

Examples

>>> d = Xor()
>>> print(insert_mss(d, -1, [0], [1, 2]))
Class:          Distribution
Alphabet:       ('0', '1') for all rvs
Base:           linear
Outcome Class:  str
Outcome Length: 4
RV Names:       None
x      p(x)
0000   0.25
0110   0.25
1011   0.25
1101   0.25

Complexity Profile¶

The complexity profile is simply the amount of information at scale \(\geq k\) of each “layer” of the I-diagram [BY04].

Consider example 1, which contains three independent bits. Each of these bits are in the outermost “layer” of the i-diagram, and so the information in the complexity profile is all at layer 1:

In [11]: ComplexityProfile(ex1).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-11-850d1430a4c3> in <module>()
----> 1 ComplexityProfile(ex1).draw()

NameError: name 'ComplexityProfile' is not defined

Whereas in example 2, all the information is in the center, and so each scale of the complexity profile picks up that one bit:

In [12]: ComplexityProfile(ex2).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-12-51ab6ae2e99f> in <module>()
----> 1 ComplexityProfile(ex2).draw()

NameError: name 'ComplexityProfile' is not defined

Both bits in example 3 are at a scale of at least 1, but only the shared bit persists to scale 2:

In [13]: ComplexityProfile(ex3).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-13-6311c54eb5f5> in <module>()
----> 1 ComplexityProfile(ex3).draw()

NameError: name 'ComplexityProfile' is not defined

Finally, example 4 (where each variable is the exclusive or of the other two):

In [14]: ComplexityProfile(ex4).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-14-d42e22fe7b80> in <module>()
----> 1 ComplexityProfile(ex4).draw()

NameError: name 'ComplexityProfile' is not defined

Marginal Utility of Information¶

The marginal utility of information (MUI) [ASBY14] takes a different approach. It asks, given an amount of information \(\I[d : \{X\}] = y\), what is the maximum amount of information one can extract using an auxilliary variable \(d\) as measured by the sum of the pairwise mutual informations, \(\sum \I[d : X_i]\). The MUI is then the rate of this maximum as a function of \(y\).

For the first example, each bit is independent and so basically must be extracted independently. Thus, as one increases \(y\) the maximum amount extracted grows equally:

In [15]: MUIProfile(ex1).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-15-a5377e16ac48> in <module>()
----> 1 MUIProfile(ex1).draw()

NameError: name 'MUIProfile' is not defined

In the second example, there is only one bit total to be extracted, but it is shared by each pairwise mutual information. Therefore, for each increase in \(y\) we get a threefold increase in the amount extracted:

In [16]: MUIProfile(ex2).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-16-da24b4abf7f6> in <module>()
----> 1 MUIProfile(ex2).draw()

NameError: name 'MUIProfile' is not defined

For the third example, for the first one bit of \(y\) we can pull from the shared bit, but after that one must pull from the independent bit, so we see a step in the MUI profile:

In [17]: MUIProfile(ex3).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-17-35d7522e2d62> in <module>()
----> 1 MUIProfile(ex3).draw()

NameError: name 'MUIProfile' is not defined

Lastly, the xor example:

In [18]: MUIProfile(ex4).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-18-e9bb2bcb7e49> in <module>()
----> 1 MUIProfile(ex4).draw()

NameError: name 'MUIProfile' is not defined

Schneidman Profile¶

Also known as the connected information or network informations, the Schneidman profile exposes how much information is learned about the distribution when considering \(k\)-way dependencies [SSB+03]. In all the following examples, each individual marginal is already uniformly distributed, and so the connected information at scale 1 is 0.

In the first example, all the random variables are independent already, so fixing marginals above \(k=1\) does not result in any change to the inferred distribution:

In [19]: SchneidmanProfile(ex1).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-19-0f650431dd1c> in <module>()
----> 1 SchneidmanProfile(ex1).draw()

NameError: name 'SchneidmanProfile' is not defined

In the second example, by learning the pairwise marginals, we reduce the entropy of the distribution by two bits (from three independent bits, to one giant bit):

In [20]: SchneidmanProfile(ex2).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-20-0844f957a1c0> in <module>()
----> 1 SchneidmanProfile(ex2).draw()

NameError: name 'SchneidmanProfile' is not defined

For the third example, learning pairwise marginals only reduces the entropy by one bit:

In [21]: SchneidmanProfile(ex3).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-21-45907c62856b> in <module>()
----> 1 SchneidmanProfile(ex3).draw()

NameError: name 'SchneidmanProfile' is not defined

And for the xor, all bits appear independent until fixing the three-way marginals at which point one bit about the distribution is learned:

In [22]: SchneidmanProfile(ex4).draw()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-22-5e149133a64d> in <module>()
----> 1 SchneidmanProfile(ex4).draw()

NameError: name 'SchneidmanProfile' is not defined

1.0.0 future

• [Feature]: Basic functionality.
• [Feature] #26: Add the Jensen-Shannon Divergence, a measure of distribution distance.
• [Feature] #5: Add the oft-used Total Correlation.
• [Feature] #10: Add the Co-Information.
• [Feature] #30: Add the Gács-Körner Common Information.
• [Feature] #7: Add the Residual Entropy.
• [Feature] #6: Add the Binding Information.
• [Feature] #2: Add the Extropy.
• [Feature] #33: Add the Perplexity.
• [Feature] #45: Add the Interaction Information.
• [Feature] #47: Add the TSE Complexity.
• [Feature] #16: Add the Channel Capacity.
• [Feature] #1: Add the Cumulative Residual Entropy.
• [Feature] #14: Add the creation of Minimal Sufficient Statistics.
• [Feature] #35: Add the cross entropy.
• [Feature] #34: Add the Kullback-Leibler Divergence.
• [Feature] #3: Add the Renyi entropy.
• [Feature] #4: Add the Tsallis entropy.
• [Feature] #87: Add Renyi, Tsallis, Hellinger, and alpha divergences.
• [Feature] #13: Add the Joint Minimal Sufficient Statistic.
• [Feature] #95: Add the complexity profile.
• [Feature] #96: Add the marginal utility of information.
• [Support] #32: Use six for python 2/3 compatibility.
• [Support] #40: Use an Enum for rv_mode.
• [Support] #75: Enable coveralls to display detailed coverage information.
• [Support] #77: Enable landscape.io to do static code analysis.
• [Support] #58: Alias Dual Total Correlation as Binding Information.