"How concerned should we be about OpenAI’s recurrent architecture rumors?" by Rauno Arike
Failed to add items
Sorry, we are unable to add the item because your shopping cart is already at capacity.
Add to basket failed.
Please try again later
Add to Wish List failed.
Please try again later
Remove from Wish List failed.
Please try again later
Follow podcast failed
Unfollow podcast failed
-
Narrated by:
-
By:
Summary
What architecture is Astra likely to have?
The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across [...]
---
Outline:
(00:41) What architecture is Astra likely to have?
(02:14) How bad is this?
(06:23) Will looped transformers be scaled up in the future?
(09:40) What serial depth warrants neuralese concerns?
(14:04) Additional speculation about the architecture
(15:29) Some open questions
(16:54) Conclusion
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 2nd, 2026
Source:
https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concerned-should-we-be-about-openai-s-recurrent
---
Narrated by TYPE III AUDIO.
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.