Online softmax approximation
Numerical difference between the streaming running-max normalizer and the exact softmax denominator, across sequence lengths and dtypes.
Open appliedresearch
Technical notes on measurements from model internals
In progress
Research notes
draft
KDA geometry
Spectral properties of the gated delta-rule state matrix, measured for each head and position.
Quantized residuals vs unquantized signals
Per-direction quantization error in the residual stream, compared against the unquantized activations at the same layer.
Can Gemma 3n route matryoshka layers?
Test whether routing can select both layer depth and hidden width for each text feature.
When does RoPE preserve relative position?
Conditions where rotary embeddings preserve relative-position inner products, with measured errors where they do not.
Expert assignment in encoder-free architectures
Expert assignment for vision and audio inputs in models with no modality encoder.
2026
About
These notes report measurements from residual streams, attention states, projection boundaries, and positional constructions. They test which quantities remain invariant under changes of coordinates or precision and which depend on the selected representation.
Every fitted result states its training and evaluation split. In-sample results are labelled and do not serve as primary evidence.
Elsewhere