Experiments

Small probes on SAE features catch Gemma 3 drifting off topic while it writes. Raw hidden states do as well; the SAE buys a smaller detector whose inputs have names.

A shared transformer block applied up to four times, against a four-block Decision Transformer on Key-to-Door. A quarter of the parameters solved every layout, but so did a single block. Then a slightly bigger grid starved every model of successful data.

Four dense 1B checkpoints on 6B FineWeb tokens to test Kimi Delta Attention and LongCat n-gram embeddings, then grafting, pruning, 4-bit quantization and NER fine-tuning.

An autoencoder test of Mamba, two-direction Mamba, Hydra and Gated DeltaNet. Bidirectional encoders didn't help, five times more data closed most of the gap, and 4× compression broke every model.

Eleven speech recognisers on pure noise, then layer and neuron ablations inside Whisper. Larger models hallucinate more, the colour of the noise matters as much as the model, and one early-encoder neuron switches the behaviour off without hurting real speech.

Five architectures, one ceiling: jumps that differ only by blade edge are invisible to COCO-17 keypoints. Which class you predict matters about three times more than which model.

Can sparse autoencoder features replace a dedicated topic classifier in a guardrail stack, reading the topic from the model’s activations instead of its output?