Mutual Information and Conditional Information

 mutual_information(X::CountData, Y::CountData, XY::CountData, estimator::Type{T}) where {T<:AbstractEstimator}
 mutual_information(joint::Matrix{I}, estimator::Type{T}) where {T<:AbstractEstimator, I<:Real}

\[I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x, y) \log \left(\frac{p_{X,Y}(x,y)}{p_X(x) p_Y(y)}\right)\]

But we use the identity

\[I(X;Y) = H(X) + H(Y) - H(X,Y)\]

where $H(X,Y)$ is the entropy of the joint distribution

The joint distribution, XY needs to be supplied by the user in the form of a CountData struct.

Example

Calculating mutual information directly over a matrix is the easiest way to use this function:

julia> m = Matrix([1 0; 0 1])
julia> i = mutual_information(m, MaximumLikelihood)
0.6931471805599453

julia> to_bits(i)
1.0

conditional_entropy(X::CountData, XY::CountData, estimator::Type{T}) where {T<:NonParameterisedEstimator}
conditional_entropy(joint::Matrix{R}, estimator::Type{NSB}; dim=1, guess=false, KJ=nothing, KX=nothing) where {R<:Real}
conditional_entropy(joint::Matrix{R}, estimator::Type{Bayes}, α; dim=1, KJ=nothing, KX=nothing) where {R<:Real}

Compute the conditional entropy of Y conditioned on X

\[H(Y \mid X) = - \sum_{x \in X, y \in Y} p(x, y) \ln \frac{p(x, y)}{p(x)}\]

Compute the estimated conditional entropy of Y given X, from counts of X, and (X,Y) and estimator

\[\hat{H}(Y \mid X) = \hat{H}(X, Y) - \hat{H}(X)\]

Example

julia> m = [4 9 4 5 8; 10 2 7 9 6; 3 5 6 9 6; 4 2 1 5 8; 4 5 43 8 3]
julia> X = DiscreteEntropy.marginal_counts(m, 1)

julia> conditional_entropy(from_data(X, Histogram), from_data,(m, Histogram), Zhang)
1.395955392163378

julia> conditional_entropy(m, Zhang)
1.395955392163378