"How concerned should we be about OpenAI’s recurrent architecture rumors?" by Rauno Arike cover art

"How concerned should we be about OpenAI’s recurrent architecture rumors?" by Rauno Arike

"How concerned should we be about OpenAI’s recurrent architecture rumors?" by Rauno Arike

Listen for free

View show details

This Season's Biggest Deal | $0.99/mo for 3 months + $20 Audible bonus

Auto-renews at $8.99/mo. after 3 months - terms apply.
Summary
Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth.

What architecture is Astra likely to have?

The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across [...]

---

Outline:

(00:41) What architecture is Astra likely to have?

(02:14) How bad is this?

(06:23) Will looped transformers be scaled up in the future?

(09:40) What serial depth warrants neuralese concerns?

(14:04) Additional speculation about the architecture

(15:29) Some open questions

(16:54) Conclusion

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concerned-should-we-be-about-openai-s-recurrent

---



Narrated by TYPE III AUDIO.

No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.