Invariant integration for prior-knowledge enhanced deep learning architectures

Abstract

Incorporating prior knowledge to Deep Neural Networks is a promising approach to improve their sample efficiency by effectively limiting the search space the learning algorithm needs to cover. This reduces the amount of samples a network needs to be trained on to reach a specific performance. Geometrical prior knowledge is the knowledge about input transformations that affect the output in a predictable way, or not at all. It can be built into Deep Neural Networks in a mathematically sound manner by enforcing in- or equivariance.

Equivariance is the property of a map to behave predictably under input transformations. Convolutions are an example for a translation-equivariant map, where a translation of the input results in a shifted output. Group-equivariant convolutions are a generalization achieving equivariance towards more general transformation groups such as rotations or flips. Using group-equivariant convolutions within Neural Networks embeds the desired equivariance in addition to translations.

Invariance is a closely related concept, where the output of a function does not change when its input is transformed. Invariance is often a desirable property of a feature extractor in the context of classification. While the extracted features need to encode the information required to discriminate between different classes, they should be invariant to intra-class variations, i.e., to transformations that map samples within the same class subspace. In the context of Deep Neural Networks, the required invariant representations can be obtained with mathematical guarantees by applying group-equivariant convolutions followed by globally pooling among the group- and spatial domain. While pooling guarantees invariance, it also discards information and is thus not ideal.

In this dissertation, we investigate the transition from equi- to invariance within Deep Neural Networks that leverage geometrical prior knowledge. Therefore, we replace the spatial pooling operation with Invariant Integration, a method that guarantees invariance while adding targeted model capacity rather than destroying information. We first propose an Invariant Integration Layer for rotations based on the group average calculated with monomials. The layer can be readily used within a Neural Network and allows backpropagating through it. The monomial parameters are selected either by iteratively optimizing the least-squared-error of a linear classifier, or based on neural network pruning methods.

We then replace the monomials with functions that are more often encountered in the context of Neural Networks such as learnable weighted sums or self-attention. We thereby streamline the training procedure of Neural Networks enhanced with Invariant Integration.

Finally, we expand Invariant Integration towards flips and scales, highlighting the universality of our approach. We further propose a multi-stream architecture that is able to leverage invariance to multiple transformations at once. This approach allows us to efficiently combine multiple invariances and select the best-fit invariant solution for the specific problem to solve.

The conducted experiments show that applying Invariant Integration in combination with group-equivariant convolutions significantly boosts the sample efficiency of Deep Neural Networks improving the performance when the amount of available training data is limited.