In information theory, dual total correlation (Han 1978), information rate (Dubnov 2006), excess entropy (Olbrich 2008), or binding information (Abdallah and Plumbley 2010) is one of several known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation (Ay 2001).
Definition
![](https://web.archive.org/web/20220518091251im_/https://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/VennInfo3Var.svg/256px-VennInfo3Var.svg.png)
For a set of n random variables , the dual total correlation is given by
where is the joint entropy of the variable set and is the conditional entropy of variable , given the rest.
Normalized
The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value ,
Bounds
Dual total correlation is non-negative and bounded above by the joint entropy .
Secondly, Dual total correlation has a close relationship with total correlation, . In particular,
History
Han (1978) originally defined the dual total correlation as,
However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:
See also
References
- Han, Te Sun (1978). "Nonnegative entropy measures of multivariate symmetric correlations". Information and Control. 36 (2): 133–156. doi:10.1016/S0019-9958(78)90275-9.
- Fujishige, Satoru (1978). "Polymatroidal dependence structure of a set of random variables". Information and Control. 39: 55–72. doi:10.1016/S0019-9958(78)91063-X.
- Dubnov, Shlomo (2006). "Spectral Anticipations". Computer Music Journal. 30 (2): 63–83. doi:10.1162/comj.2006.30.2.63. S2CID 2202704.
- Olbrich, E.; Bertschinger, N.; Ay, N.; Jost, J. (2008). "How should complexity scale with system size?". The European Physical Journal B. 63 (3): 407–415. Bibcode:2008EPJB...63..407O. doi:10.1140/epjb/e2008-00134-9. S2CID 120391127.
- Abdallah, Samer A.; Plumbley, Mark D. (2010). "A measure of statistical complexity based on predictive information". arXiv:1012.1890v1 [math.ST].
- Nihat Ay, E. Olbrich, N. Bertschinger (2001). A unifying framework for complexity measures of finite systems. European Conference on Complex Systems. pdf.