In “simple” terms, information topology regards a statistical system (a collection of interrelated observables) as a generalized topological space (a topos) and identifies Shannon entropy, along other important “measures of information” used in information theory, as a possible invariant associated to this space.

Toposes or topoi are an abstraction of topological spaces in the language of category theory and sheaves introduced by Grothendieck and his collaborators (Artin, Verdier,…). Toposes allow richer cohomology theories than set-theoretic topological spaces, and some of these theories (e.g. étale cohomology) play a key role in modern algebraic geometry. Moreover, these Grothendieck toposes are particular cases of elementary toposes, which are “nice” categories with properties analogous to those of the category of sets that play an important role in logic.

Baudot and Bennequin (Baudot & Bennequin, 2015) first identified Shannon’s discrete entropy as a toposic invariant of certain categories of discrete observables. My Ph.D. thesis (Vigneaux, 2019) and a series of articles extended their results in several directions. Namely, the general homological constructions were abstracted from the concrete setting of discrete variables via information structures (categories that encode the relations of refinement between observables), allowing seamless extensions and generalizations to other settings such as continuous vector-valued observables (Vigneaux, 2020).

When the information structure encodes discrete observables, the classical information functions—– Shannon entropy, Tsallis $\alpha$-entropy, Kullback-Leibler divergence—–appear as 1-cocycles; the corresponding “coefficients” of the cohomology are probabilistic functionals (i.e. functions of probability laws). There is also a combinatorial version of the theory (coefficients are functions of histograms) where the only 0-cocycle is the exponential function and the 1-cocycles are generalized multinomial coefficients (Fontené-Ward) (Vigneaux, 2023). There is an asymptotic relation between the combinatorial and probabilistic cocycles.

For information structures that contain continuous vector-valued observables (besides discrete ones), the only new degree-one cocycles are Shannon’s differential entropy entropy and the dimension (of the support of the measure) (Vigneaux, 2021). This constitutes a novel algebraic characterization of differential entropy.

Information cohomology has seen some advances in the last years. Marcolli and Manin (Manin & Marcolli, 2020) related information structures with other homotopy- and category-theoretic models of neural information networks. Similar perspectives have been developed more recently by Belfiore and Bennequin (Belfiore & Bennequin, 2021) to tackle the problem of interpretability of neural networks. They associate to each neural network a certain category equipped with a Grothendieck topology (determined by the connectivity of the neurons), and study the category of sheaves on it, which is a topos. Every topos has an internal logic, and they are linking this internal toposic logic with the classification capabilities that emerge in each layer of a trained neural network (these were previously studied in the experimental article (Belfiore et al., 2021)).

Presentations:

• “Cohomological Aspects of Information” [video], Topos Institute, 2024: I summarize the main results that I have obtained in this domain.
• “Information Cohomology of Classical Vector-valued Observables” [video], GSI2021: I provide details on the characterization of the differential entropy and the dimension as the only cohomology classes in degree 1 for systems of vector-valued observables.
• “Entropy under disintegrations” [video], GSI 2021: I explain how every disintegration of a reference measure $\lambda$ induces a chain rule for the generalized differential entropy $S(\rho) = -\int \log (\frac{d\rho}{d\lambda}) d\rho$, which gives a foundation to the extension of information cohomology to more general observables e.g. with values in locally compact topological groups.
• “Variations on Information Theory: Categories, Cohomology, Entropy” [video], IHES, 2016: an older presentation, aimed at probabilistists, where I introduce the notion of (de Rham) cohomology and it’s analogue in our theory.

Other references:

• “On the Structure of Information Cohomology”, Ph.D. thesis by Hubert Dubé (U. Toronto), which introduces the Mayer-Vietoris long exact sequence, Shapiro’s lemma and Hochschild-Serre spectral sequence in the framework of information cohomology, and provides some bounds on the cohomological dimension along with new cohomological computations.
• “Information cohomology and Entropy”, master thesis by Luca Castiglioni (University of Milan).

Bibliography

1. Baudot, P., & Bennequin, D. (2015). The Homological Nature of Entropy. Entropy, 17(5), 3253–3318.
2. Vigneaux, J. P. (2019). Topology of Statistical Systems: A Cohomological Approach to Information Theory [PhD thesis]. Université de Paris.
3. Vigneaux, J. P. (2020). Information structures and their cohomology. Theory and Applications of Categories, 35(38), 1476–1529.
4. Vigneaux, J. P. (2023). A characterization of generalized multinomial coefficients related to the entropic chain rule. Aequationes Mathematicae, 97(2), 231–255.
5. Vigneaux, J. P. (2021). Information cohomology of classical vector-valued observables. In F. "Nielsen & F. Barbaresco (Eds.), GSI 2021: Geometric Science of Information (Vol. 12829, pp. 537–546). Springer.
6. Manin, Y., & Marcolli, M. (2020). Homotopy Theoretic and Categorical Models of Neural Information Networks. ArXiv Preprint ArXiv:2006.15136.
7. Belfiore, J.-C., & Bennequin, D. (2021). Topos and stacks of deep neural networks. ArXiv Preprint ArXiv:2106.14587.
8. Belfiore, J.-C., Bennequin, D., & Giraud, X. (2021). Logical Information Cells I. ArXiv Preprint ArXiv:2108.04751.