Hybrid models can hide reasoning in their RNN state

What happens in the RNN stays in the RNN

Part of the reason I sleep well at night is because of one bet: a model pulling some sketchy moves needs to reason hard, and if it's reasoning, it needs to write that reasoning down as CoT, since models can only do so much serial work per token.

Hybrid and looped models change the odds on that bet. Models that use GDN or any linear RNN for attention, like Qwen 3.5 or OLMo-Hybrid, carry state across tokens (here's how GDN works, if useful), so the question is: can these models do their reasoning in the RNN layers instead, and evade CoT monitoring?

Too long, made my agent read it: yes, under answer-only training they can, and the deciding factor for whether this hidden reasoning generalises is the write strength β of the GDN.¹¹ In a Gated DeltaNet layer, the write strength β sets how strongly each new token overwrites what's already stored in the RNN's state. At β = 1 the old value is fully replaced; above 1 the update overshoots and can flip the sign of what's stored, which is exactly what tracking a flipping switch needs.

This work builds on Qwenputer, where I compiled a RISC-V CPU into Qwen 3.5's weights using its GDN layers as RAM. The same compiler turned out to be a handy tool here too. The other follow-up, on using compiled weights as in-weight, untamperable monitoring tools, uses the same setup.

The code, raw results and lab notes are all on GitHub.

The theory this tests

There's already a theory that predicts exactly this. Grazzi, Siems et al. (ICLR 2025)²² Riccardo Grazzi, Julien Siems, et al., "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues", ICLR 2025. showed that it's all about the eigenvalues of the state update. If β stays at or below 1, every eigenvalue sits between 0 and 1, and a linear RNN like that just can't flip a bit with every token, so tracking something like parity one token at a time is out of reach. Let β go up to 2 and you get negative eigenvalues, and suddenly flipping is on the table.

This is where the big labs made different calls. Qwen 3.5 squashes β through a sigmoid, so it's capped at 1. OLMo-Hybrid lets it go up to 2, and they did that on purpose (their paper says so).

One caveat I would like to be upfront about: Steifer (2026)³³ Tomasz Steifer, "Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders", arXiv 2026. showed that a Qwen-style hybrid, β cap and all, can still solve a parity-style task in a single pass by leaning on attention as well as the RNN. So the real question isn't "what's possible", it's "what does training actually find".

Setup

I used two hybrid models here: OLMo-Hybrid-7B, which lets β go anywhere up to 2, and Qwen 3.5 4B, which keeps it below 1, with OLMo-3-7B (a regular dense transformer, OLMo-Hybrid's twin) as a control.

All our candidates were made to perform one task: track the state of a light switch through many flips, then say whether it's on or off at the end. That's parity, the simplest thing that needs state tracking, which is exactly why it's the classic test for this.

First, I checked the models as they come, with no training. When asked to answer straight away without any working or reasoning, none of them could track the switch past 4-8 flips. So out of the box, CoT monitoring holds for these hybrids.

It is at this juncture that I put on the pressure. I fine-tuned each model to answer with just "on" or "off", no working or reasoning allowed at all, using a LoRA plus the β and decay projections,⁴⁴ LoRA trains small add-on matrices instead of the whole model, which keeps a fine-tune cheap. I also unfroze the projections that set β and the decay, so the model could change how its RNN writes and forgets. with a curriculum that ramped up to 64 flips. Once done I tested the models at 128 and 256 flips, way past any trajectory length they saw in training. If a model still gets those right, it isn't memorising, and it has learned to track the switch silently, inside its RNN state.

Result 1: OLMo-Hybrid learns silent tracking that generalises

Update, 23 Oct 2026: I ran 3 more seeds for both OLMo-Hybrid arms, so the numbers below are now over 6 seeds each. The result got stronger, not weaker.

This is the big one. After a fine-tune targeting no-reasoning answering, OLMo-Hybrid learned to track the switch silently, and kept doing it at twice the length it was trained on: 0.89 at 128 flips, averaged over 6 seeds. Five of the six learned the training range properly, and all five of those still got it right at 128 flips (0.89 to 1.00). The odd one out never learned 64 flips in the first place.

The other two only memorised and worked well at the trajectory lengths they'd seen. Qwen 3.5 answered well till the training range and then essentially fell to a coin flip, scoring 0.55 at 128 flips. The dense OLMo-3 did the same at 0.54 (it can learn short parity with some seeds, but not past the training range).

Result 2: β is the cause

Alright, but OLMo-Hybrid and Qwen 3.5 differ in lots of ways, so how do I know it's β? Well, I took the exact same OLMo-Hybrid, capped its β at 1 so it behaves like Qwen's, changed nothing else, and trained it the same way. It also dropped to 0.52 at 128 flips, basically chance, on all 6 seeds (0.49 to 0.56). To be fair, the cap also made the training range itself harder: only 2 of the 6 capped runs properly learned 64 flips, and both of those fell apart past it. Qwen 3.5 did the exact same thing, with 2 of its 3 runs learning 64 flips and neither carrying on to 128.

Capping β barely costs anything on normal text, perplexity goes up by about 3.5%.

Where the hidden bit lives (and why a naive monitor misses it)

So where is OLMo-Hybrid actually keeping the switch? The obvious test was to take the RNN state from one sequence, swap it into another, and see if the answer follows. I did that, and the answer didn't move at all (0.00). Now, this looked like the state wasn't doing anything, and I was about to call it a day here, but on more digging it turned out to be a trap (Admiral Ackbar called it): attention was quietly keeping a backup copy of the bit, so swapping the state alone changed nothing.

By swapping the state and knocking out attention at the same time, the answer follows the swapped state every single time (1.00). Conclusion: the bit really does live in the RNN state, in a couple of specific heads (layer 16 head 2 and layer 17 head 4). The capped model reuses the same heads, but there they act more like a counter than a flip.

This is the part that monitoring enthusiasts should focus on: a monitor that only looks at the recurrent state would have missed a real hidden tracker. Hybrids give you a test plain transformers don't, that being, you can block attention and swap memory separately, and that tells you exactly where the information lives.

The trained head even looks like a rough copy of the exact tracker I can compile by hand: it flips about -0.45 on a toggle and +0.46 on a no-op, where the compiled one flips a clean -1 and +1. That weak flip is probably why it fades at longer lengths.

Finally, a use for my weird compiler

This is where Qwenputer's compiler came in handy. I compiled exact β = 2 trackers straight into GDN layers, and they track parity perfectly over 100k tokens. With the stock β this just fails. That's the answer key for what a "perfect" silent tracker looks like, which is how I could tell the trained head was a rough copy of one.

I also planted a compiled tracker inside a real Qwen 3.5 and checked that a blind detector could find it, and it ranked it 1st out of 864 heads. More on that in the monitoring post.

Limits

I want to be honest about what this is and isn't. It's 6 seeds per arm for the main OLMo-Hybrid comparison and 3 for the rest, and one task: parity. That's about all that the compute I have on hand lets me run. I also tried a 3-cup shell game and nothing learned it on my current training budget (turns out my GPU budget can't even afford a shell game). A four-switch version of the light switch task went the same way: both OLMo-Hybrid arms learned it at 16 flips and were at chance by 64, so it can't tell the two apart. The pressure is artificial too: I directly trained for answer-only, which is a stand-in for training that pushes models to skip their working, not a real RL setup.

What it means

The short version, for those constrained for time: whether hidden reasoning in the RNN state generalises comes down largely to the RNN's β. That's mainly an architecture choice, not just a training one, and right now different labs are making different calls on it.

So, to whoever is designing the next big hybrid: please consider keeping β at 1. It costs you about 3.5% perplexity, and it would really help me sleep at night. Thanks.

If you're shipping or evaluating a new hybrid, here's two pretty cheap things to check: what range its eigenvalues can reach, and whether the swap-plus-knockout test finds state that attention is hiding.

Looped models like Huginn and Ouro (and reportedly OpenAI's Astra, if I could ever get my hands on it) raise the same question with a different knob: the number of loops instead of β. That's where I would like to take this next.

Related posts

Qwenputer, for how the compiler and GDN-as-RAM work, and the follow-up on compiled in-weight, untamperable monitoring tools.

Notes

1. In a Gated DeltaNet layer, the write strength β sets how strongly each new token overwrites what's already stored in the RNN's state. At β = 1 the old value is fully replaced; above 1 the update overshoots and can flip the sign of what's stored, which is exactly what tracking a flipping switch needs. ↩
2. Riccardo Grazzi, Julien Siems, et al., "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues", ICLR 2025. ↩
3. Tomasz Steifer, "Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders", arXiv 2026. ↩
4. LoRA trains small add-on matrices instead of the whole model, which keeps a fine-tune cheap. I also unfroze the projections that set β and the decay, so the model could change how its RNN writes and forgets. ↩