// NATURE NEWS — SPAZIO & SCIENZA
This AI ‘mechanic’ can peer inside the black box of other neural networks
Search author on:
PubMed
Google Scholar
Some AI models behave like mechanics — fixing issues in other models that they neither designed nor fully understand. Credit: Mint Images/Getty
Researchers have used an artificial-intelligence model to read and tinker with the internal workings of another AI network — offering a new way to peek into the ‘black box’ of AI.
Computer scientists usually improve models by training them in tiny increments or by building them from scratch. A team at the Massachusetts Institute of Technology (MIT) in Cambridge has demonstrated how an AI model called GPT-6 Astra can interpret the raw internal values of some other models well enough to predict what different parts of the systems do. It can then edit the models to make improvements — including boosting their mathematical abilities. The authors reported their findings in a preprint1 on 2 November.
ChatGPT is a black box: how AI research can break it open
Frontier AI models are “showing early sparks” of an ability to treat neural networks like files that they can read, reason about and change, says co-author Rikiya Takehi, a machine-learning researcher at MIT.
The results do not mean that Astra, developed by OpenAI in San Francisco, California, can explain or understand how a given AI network works. But the findings suggest that general-purpose AI models — such as ChatGPT or Claude — might be able to modify a network in useful ways, like a mechanic who can identify and fix issues in an engine that they neither designed nor fully understand.
Researchers already try to interpret how different components of a network create its outputs. The preprint shows that it is possible to automate some of these ‘interpretability’ methods, says Fazl Barez, an AI-safety researcher at the University of Oxford, UK.
Researchers hope that better understanding models in this way will help to create safer AI systems, although Barez cautions that this assumption is “largely untested”.
It remains unclear if Astra is going beyond applying information and techniques learned in its training. Barez says he would be surprised if Astra is able to read and interpret models in ways that go beyond existing human abilities.