From an analytic perspective, the dimension has played an important in information theory since its inception, mainly in connection with quantization. By partitioning $\Rr^d$ into cubes with vertexes in $\mathbb Z^d/n$, one might quantize a continuous probability measure $\rho$ into a measure $\rho_n$ with countable support, whose entropy satisfies \begin{equation}\label{eq:expansion_law} H(\rho_n) = D\ln n + h + o(1), \end{equation} where $D=d$ and $h=h(\rho)$ is the differential entropy of $\rho$ (Kolmogorov & Shiryayev, 1993). Renyi (Rényi, 1959) turned this into a definition: if $\rho$ is now a general law and the expansion \eqref{eq:expansion_law} holds for some constants $D,h\in \Rr$, one calls $D$ the information dimension of $\rho$ and $h$ its $d$-dimensional entropy. He wondered about the “topological meaning” of the entropic dimension, which might be noninteger.

In (Vigneaux, 2023) I introduced an asymptotic equipartition property for discrete-continuous mixtures or, more generally, of convex combination of rectifiable measures on $\Rr^d$. In particular, it gives an interpretation for the information dimension $D$ of one of these measures $\rho$: the product $(\Rr^d)^n$ naturally splits into strata of different dimensions, and the typical realizations of $\rho^{\otimes n}$ concentrate on strata of a few dimensions close to $nD$. I also obtained volume estimates (in terms of Hausdorff measures) for the typical realizations in each typical stratum. (A measure $\rho$ is $m$-rectifiable if there exists a set $E$, equal to a countable union of $C^1$ manifolds, such that $\rho$ has a density with respect to the restricted HAusdorff measure $\mathcal H^m|_E$, which is the natural notion of $m$-dimensional volume on $E$.)

Presentations:

Bibliography

  1. Kolmogorov, A. N., & Shiryayev, A. N. (1993). Selected Works of A. N. Kolmogorov. Volume III: Information Theory and the Theory of Algorithms. Kluwer Academic Publishers.
  2. Rényi, A. (1959). On the dimension and entropy of probability distributions. Acta Mathematica Academiae Scientiarum Hungarica, 10(1), 193–215.
  3. Vigneaux, J. P. (2023). Typicality for stratified measures. IEEE Transactions on Information Theory, 69(11), 6922–6940.