RAVEL · Matryoshka advantage over BatchTopK
+1.22 pp at 50M / +1.16 pp at 500M
The overall score is higher at both budgets, with higher isolation scores.
Gemma-2-2b · Layer 12
BatchTopK and Matryoshka BatchTopK at 50M and 500M tokens. Compare sparse probing, RAVEL, and the final count of inactive latents.
RAVEL · Matryoshka advantage over BatchTopK
+1.22 pp at 50M / +1.16 pp at 500M
The overall score is higher at both budgets, with higher isolation scores.
Probing · 50M → 500M at probe k = 16
−0.004 pp BatchTopK / +0.152 pp Matryoshka
Mean test accuracy changes little in these runs. One training seed; no uncertainty intervals.
Mean score across 113 datasets · L1 feature selection · normal setting
| SAE · tokens | k = 1 | k = 2 | k = 4 | k = 5 | k = 16 |
|---|---|---|---|---|---|
| BatchTopK50M | 76.14 | 77.82 | 80.07 | 80.66 | 82.88 |
| Matryoshka50M | 76.14 | 78.07 | 80.08 | 80.52 | 82.90 |
| BatchTopK500M | 75.83 | 78.05 | 80.18 | 80.72 | 82.88 |
| Matryoshka500M | 76.57 | 78.60 | 80.23 | 80.79 | 83.05 |
Overall disentanglement, cause, and isolation scores
| SAE · tokens | Overall | Cause | Isolation |
|---|---|---|---|
| BatchTopK50M | 68.87 | 66.27 | 71.46 |
| Matryoshka50M | 70.08 | 65.79 | 74.38 |
| BatchTopK500M | 68.72 | 67.24 | 70.20 |
| Matryoshka500M | 69.88 | 67.22 | 72.55 |
Final dead-latent counts, recorded in each uploaded model’s configuration
0 / 16,384 dead
0.000% of latents
0 / 16,384 dead
0.000% of latents
5 / 16,384 dead
0.031% of latents
5 / 16,384 dead
0.031% of latents
“Dead” means no activation for more than 1,000 consecutive training batches, about 4.1M tokens. This is an end-of-training inactivity measure, not a claim that a latent can never activate. The recorded field is metadata.num_dead_features; W&B logs its history as sparsity/dead_features.
Training. Gemma-2-2b block output model.layers.12; input width 2,304; SAE width 16,384; BatchTopK k=100; Matryoshka widths [4,096, 16,384]. Dataset: monology/pile-uncopyrighted, streaming. Training seed 0.
Learning rate 3e-4; constant for 50M, with decay over the last 20% for 500M. Training used Hugging Face model execution and bfloat16 autocasting. SAELens 6.51.1.
The 50M run resumed from 5.45M tokens. Streaming resumption restored the document cursor but not the full activation shuffle buffer. Whole batches yield actual totals of 50,003,968 and 500,002,816 tokens.
Evaluation. sae-probes: 113 datasets, L1 selection, k ∈ {1, 2, 4, 5, 16}, evaluation seed 42. Accuracy, AUROC, and F1 are means across datasets. RAVEL: SAEBench configuration, evaluation seed 42, city and Nobel prize winner entity classes.
Interpretation. These are individual training/evaluation seeds. Small gaps are descriptive; repeated seeds or uncertainty estimates would be needed for a stronger comparison.
Exports. Model links are pinned to the uploaded revision. BatchTopK-trained models use SAELens’s learned-threshold inference export. CSV score columns use the 0–1 scale; this page displays percentages. The CSV includes all four models regardless of the filter.