AIBRIEF
The HyperSAE project (a sparse autoencoder interpretability tool) trains an AI's internal concept dictionary in curved (hyperbolic) space, keeping the live model's forward pass in flat space.
Sparse autoencoders are the tool researchers use to break a language model's internal activity into individual, human-readable features. The new open-source project HyperSAE, released this month on GitHub and PyPI, runs that tool's fast path in flat (Euclidean) space and only pushes the feature dictionary into curved (hyperbolic) geometry during training. The bet: curvature is worth paying for only at the dictionary level, not on the forward pass.
The forward pass on token activations stays in bfloat16 Euclidean for speed, while a separate Poincaré ball projection is applied to the dictionary weights during optimization. Two optimizers run in parallel: AdamW on the fast path and RiemannianAdam on the slow path. A co-activation queue feeds an asymmetric Poincaré entailment loss on the dictionary's hyperbolic coordinates, giving the curvature signal to organize hierarchical features.
On Gemma-2-2B Layer 13 residual stream activations (FineWeb-Edu, NVIDIA L4 cluster), the project reports a visible benchmark row of 0.005 / 54.2 / 4.1232 / 78.9% / 6.1164 and a 9.8% reconstruction-MSE reduction with dead latents at 0.2%. Downstream, the model preserves 100% of GPQA Diamond accuracy and edges the MMLU-Pro baseline by +0.15% (16.26% vs 16.11%).
The catch: these numbers come from the author's own repository and a single configuration. No independent reproduction is visible in the current references, and the README's claim is reconstruction quality and feature utilization, not wall-clock inference speed.