Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more
Anthropic CEO Dario Amodei made an urgent push in April for the need to understand how AI models think.
This comes at a crucial time. As Anthropic battles in global AI rankings, it’s important to note what sets it apart from other top AI labs. Since its founding in 2021, when seven OpenAI employees broke off over concerns about AI safety, Anthropic has built AI models that adhere to a set of human-valued principles, a system they call Constitutional AI. These principles ensure that models are “helpful, honest and harmless” and generally act in the best interests of society. At the same time, Anthropic’s research arm is diving deep to understand how its models think about the world, and why they produce helpful (and sometimes harmful) answers.
Anthropic’s flagship model, Claude 3.7 Sonnet, dominated coding benchmarks when it launched in February, proving that AI models can excel at both performance and safety. And the recent release of Claude 4.0 Opus and Sonnet again puts Claude at the top of coding benchmarks. However, in today’s rapid and hyper-competitive AI market, Anthropic’s rivals like Google’s Gemini 2.5 Pro and Open AI’s o3 have their own impressive showings for coding prowess, while they’re already dominating Claude at math, creative writing and overall reasoning across many languages.
If Amodei’s thoughts are any indication, Anthropic is planning for the future of AI and its implications in critical fields like medicine, psychology and law, where model safety and human values are imperative. And it shows: Anthropic is the leading AI lab that focuses strictly on developing “interpretable” AI, which are models that let us understand, to some degree of certainty, what the model is thinking and how it arrives at a particular conclusion.
Amazon and Google have already invested billions of dollars in Anthropic even as they build their own AI models, so perhaps Anthropic’s competitive advantage is still budding. Interpretable models, as Anthropic suggests, could significantly reduce the long-term operational costs associated with debugging, auditing and mitigating risks in complex AI deployments.
Sayash Kapoor, an AI safety researcher, suggests that while interpretability is valuable, it is just one of many tools for managing AI risk. In his view, “interpretability is neither necessary nor sufficient” to ensure models behave safely — it matters most when paired with filters, verifiers and human-centered design. This more expansive view sees interpretability as part of a larger ecosystem of control strategies, particularly in real-world AI deployments where models are components in broader decision-making systems.
The need for interpretable AI
Until recently, many thought AI was still years from advancements like those that are now helping Claude, Gemini and ChatGPT boast exceptional market adoption. While these models are already pushing the frontiers of human knowledge, their widespread use is attributable to just how good they are at solving a wide range of practical problems that require creative problem-solving or detailed analysis. As models are put to the task on increasingly critical problems, it is important that they produce accurate answers.
Amodei fears that when an AI responds to a prompt, “we have no idea… why it chooses certain words over others, or why it occasionally makes a mistake despite usually being accurate.” Such errors — hallucinations of inaccurate information, or responses that do not align with human values — will hold AI models back from reaching their full potential. Indeed, we’ve seen many examples of AI continuing to struggle with hallucinations and unethical behavior.
For Amodei, the best way to solve these problems is to understand how an AI thinks: “Our inability to understand models’ internal mechanisms means that we cannot meaningfully predict such [harmful] behaviors, and therefore struggle to rule them out … If instead it were possible to look inside models, we might be able to systematically block all jailbreaks, and also characterize what dangerous knowledge the models have.”
Amodei also sees the opacity of current models as a barrier to deploying AI models in “high-stakes financial or safety-critical settings, because we can’t fully set the limits on their behavior, and a small number of mistakes could be very harmful.” In decision-making that affects humans directly, like medical diagnosis or mortgage assessments, legal regulations require AI to explain its decisions.
Imagine a financial institution using a large language model (LLM) for fraud detection — interpretability could mean explaining a denied loan application to a customer as required by law. Or a manufacturing firm optimizing supply chains — understanding why an AI suggests a particular supplier could unlock efficiencies and prevent unforeseen bottlenecks.
Because of this, Amodei explains, “Anthropic is doubling down on interpretability, and we have a goal of getting to ‘interpretability can reliably detect most model problems’ by 2027.”
To that end, Anthropic recently participated in a $50 million investment in Goodfire, an AI research lab making breakthrough progress on AI “brain scans.” Their model inspection platform, Ember, is an agnostic tool that identifies learned concepts within models and lets users manipulate them. In a recent demo, the company showed how Ember can recognize individual visual concepts within an image generation AI and then let users paint these concepts on a canvas to generate new images that follow the user’s design.
Anthropic’s investment in Ember hints at the fact that developing interpretable models is difficult enough that Anthropic does not have the manpower to achieve interpretability on their own. Creative interpretable models requires new toolchains and skilled developers to build them
Broader context: An AI researcher’s perspective
To break down Amodei’s perspective and add much-needed context, VentureBeat interviewed Kapoor an AI safety researcher at Princeton. Kapoor co-authored the book AI Snake Oil, a critical examination of exaggerated claims surrounding the capabilities of leading AI models. He is also a co-author of “AI as Normal Technology,” in which he advocates for treating AI as a standard, transformational tool like the internet or electricity, and promotes a realistic perspective on its integration into everyday systems.
Kapoor doesn’t dispute that interpretability is valuable. However, he’s skeptical of treating it as the central pillar of AI alignment. “It’s not a silver bullet,” Kapoor told VentureBeat. Many of the most effective safety techniques, such as post-response filtering, don’t require opening up the model at all, he said.
He also warns against what researchers call the “fallacy of inscrutability” — the idea that if we don’t fully understand a system’s internals, we can’t use or regulate it responsibly. In practice, full transparency isn’t how most technologies are evaluated. What matters is whether a system performs reliably under real conditions.
This isn’t the first time Amodei has warned about the risks of AI outpacing our understanding. In his October 2024 post, “Machines of Loving Grace,” he sketched out a vision of increasingly capable models that could take meaningful real-world actions (and maybe double our lifespans).
According to Kapoor, there’s an important distinction to be made here between a model’s capability and its power. Model capabilities are undoubtedly increasing rapidly, and they may soon develop enough intelligence to find solutions for many complex problems challenging humanity today. But a model is only as powerful as the interfaces we provide it to interact with the real world, including where and how models are deployed.
Amodei has separately argued that the U.S. should maintain a lead in AI development, in part through export controls that limit access to powerful models. The idea is that authoritarian governments might use frontier AI systems irresponsibly — or seize the geopolitical and economic edge that comes with deploying them first.
For Kapoor, “Even the biggest proponents of export controls agree that it will give us at most a year or two.” He thinks we should treat AI as a “normal technology” like electricity or the internet. While revolutionary, it took decades for both technologies to be fully realized throughout society. Kapoor thinks it’s the same for AI: The best way to maintain geopolitical edge is to focus on the “long game” of transforming industries to use AI effectively.
Others critiquing Amodei
Kapoor isn’t the only one critiquing Amodei’s stance. Last week at VivaTech in Paris, Jansen Huang, CEO of Nvidia, declared his disagreement with Amodei’s views. Huang questioned whether the authority to develop AI should be limited to a few powerful entities like Anthropic. He said: “If you want things to be done safely and responsibly, you do it in the open … Don’t do it in a dark room and tell me it’s safe.”
In response, Anthropic stated: “Dario has never claimed that ‘only Anthropic’ can build safe and powerful AI. As the public record will show, Dario has advocated for a national transparency standard for AI developers (including Anthropic) so the public and policymakers are aware of the models’ capabilities and risks and can prepare accordingly.”
It’s also worth noting that Anthropic isn’t alone in its pursuit of interpretability: Google’s DeepMind interpretability team, led by Neel Nanda, has also made serious contributions to interpretability research.
Ultimately, top AI labs and researchers are providing strong evidence that interpretability could be a key differentiator in the competitive AI market. Enterprises that prioritize interpretability early may gain a significant competitive edge by building more trusted, compliant, and adaptable AI systems.
