Title: EEGAD! Everything Explained by a Generalized Aliasing Decomposition
Abstract: A central problem in science is to use potentially noisy samples of an unknown function to predict function values for unseen inputs. A standard tool for this is linear regression, used to fit a set of basis functions to data points. Two main questions about such models are How big will their error be on the unseen inputs? and What choices can we make to reduce that error?
In classical statistics we usually think about model complexity (number of parameters) in terms of a trade-off between models that are too simple (high bias) and models that are too complex (high variance). As a model increases in complexity, the prediction error typically follows a U-shaped error curve where the best model lies at a sweet spot that optimally balances bias and variance. However, modern machine learning has given us important examples of highly over-parameterized models, with many more parameters than data points, and these often exhibit counterintuitive behaviors such as double descent in which models of increasing complexity exhibit decreasing prediction error. These phenomena are not well explained by the classical bias-variance trade-off.
We introduce a new way of thinking about such models in general, and linear regression in particular, that we call the generalized aliasing decomposition. This decomposition explains both the classical U-shape and the counterintuitive behavior in the overparametrized regime. It does this by expressing the generalization error as a combination of an aliasing error (as occurs in Fourier analysis) and an invertibility failure
An especially valuable aspect of this new paradigm is that the generalized aliasing decomposition can be explicitly calculated from the relationship between model class and samples without seeing any data labels. This means it can answer questions related to experimental design and model selection before collecting data or performing experiments.
In this talk I'll explain the motivation and main ideas of the generalized aliasing decomposition and give several examples, including some classical regression problems and a cluster expansion model used in materials science.