On Uncertainty Calibration for Equivariant Functions
Edward Berman, Jacob Ginesin, Marco Pacini, Robin Walters
[Github Link] • [Preprint PDF]
[Joint "more details" post with Eddie]
At the time of writing of the paper, Eddie was sweating machinarium. Here is a scrapped figure 1 inspired from the game, authored in glorious Gimp.
While we worked together on a few projects before, both relating to machine learning, we found our paths aligning once again through our university's mathematics capstone course. We had previously taken a cuppa algebra classes together, which we also enjoyed ^-^b. The first hint that equivariance could be a neat direction for our next project came from a poster session hosted by the Boston Symmetry Group. We enjoyed a few posters / papers (vine boom and vine thud). Indeed, we went to a MauTed birthday dinner later that day.
Concurrently, Bermanator was working on the Biomarkers project with AstroAI at the CfA, so Equivariant ML and UQ were both top of mind. Bermanator and Jakester had capstone that semester, so it was a natural vehicle to pursue this project. The Berm sought to do some approximation and generalization error theory, whilst Jakey sought to do something that is not programming languages theory.
Chasing benchmarks is boring. Here is a relevant example in the equivariant ML space: Matbench has been saturated, and it has been for a long time. However, chasing architectures and state of the art performance is sexy, and so people still do it. It is a rat race. In domains like this, the real work is in evals, data curation, actual applications, and theory that illuminates actionable advice.
The last thrust on theory is especially enticing. As Andrew Gordon Wilson eloquently puts it: all good ML papers are at least implicitly position papers. Otherwise, what is the point?
And so! What is so enticing about EquiUQ is that it marries two seemingly disparate fields of study with actionable advice for practitioners in any domain that involves both symmetry and uncertainty. We elucidate precisely when equivariance is helpful to employ, using calibration error as our informant. For all intensive purposes, you can consider us symmetry skeptics.
See, an illustrative gif from a preliminary presentation on this work at a Robotics: Science and Systems conference workshop.
1. Estimating calibration errors in practice in high dimensional spaces
2. Analogies between extrinsic equivariance and covariate shift
2. Consume more swiss rolls
[Github Link] • [Preprint PDF]
[Joint "more details" post with Eddie]
Session 0: Prelude
This blog adds a bit of narrative and extra details about our work on EquiUQ. If you want details about this work, you can read the paper, or see this talk or these slides from when this project was in its early days. This blog is for the context around the authors.At the time of writing of the paper, Eddie was sweating machinarium. Here is a scrapped figure 1 inspired from the game, authored in glorious Gimp.
Session 1: Inception
While we worked together on a few projects before, both relating to machine learning, we found our paths aligning once again through our university's mathematics capstone course. We had previously taken a cuppa algebra classes together, which we also enjoyed ^-^b. The first hint that equivariance could be a neat direction for our next project came from a poster session hosted by the Boston Symmetry Group. We enjoyed a few posters / papers (vine boom and vine thud). Indeed, we went to a MauTed birthday dinner later that day.
Concurrently, Bermanator was working on the Biomarkers project with AstroAI at the CfA, so Equivariant ML and UQ were both top of mind. Bermanator and Jakester had capstone that semester, so it was a natural vehicle to pursue this project. The Berm sought to do some approximation and generalization error theory, whilst Jakey sought to do something that is not programming languages theory.
Session 2: Piece meal
Chasing benchmarks is boring. Here is a relevant example in the equivariant ML space: Matbench has been saturated, and it has been for a long time. However, chasing architectures and state of the art performance is sexy, and so people still do it. It is a rat race. In domains like this, the real work is in evals, data curation, actual applications, and theory that illuminates actionable advice.
The last thrust on theory is especially enticing. As Andrew Gordon Wilson eloquently puts it: all good ML papers are at least implicitly position papers. Otherwise, what is the point?
And so! What is so enticing about EquiUQ is that it marries two seemingly disparate fields of study with actionable advice for practitioners in any domain that involves both symmetry and uncertainty. We elucidate precisely when equivariance is helpful to employ, using calibration error as our informant. For all intensive purposes, you can consider us symmetry skeptics.
See, an illustrative gif from a preliminary presentation on this work at a Robotics: Science and Systems conference workshop.
Session 3: Future Lore
The point of the paper is to show how symmetry mismatch results in model miscalibration and that models that are over constrained by symmetry misattribute the source of their uncertainty. There are a few notable things we want to explore going forward:1. Estimating calibration errors in practice in high dimensional spaces
2. Analogies between extrinsic equivariance and covariate shift
2. Consume more swiss rolls