Harsh Verma, Principal Software Engineer in AI at Palo Alto Networks, IEEE Senior Member, and Stanford Distinguished Scholar, has published a body of technical analysis across HackerNoon, and Forbes that, taken together, outlines what can be understood as a unified framework for the next phase of enterprise AI: one that connects the rise of multi-agent architectures, the emergence of orchestration as a core engineering function, the breakdown of traditional trust models, and a growing gap in how organisations observe and govern the AI systems they have already deployed.
Verma’s framework addresses a challenge that is becoming more visible as enterprise AI deployments mature. Organisations have invested heavily in building capable AI systems. Many can now monitor what those systems are doing. What they increasingly cannot do is explain why.
That distinction, between observing outputs and understanding reasoning, is where Verma’s analysis begins, and it is the distinction he argues will define which organisations are able to scale AI adoption responsibly and which will find themselves managing risk they cannot fully see or quantify.
The Black Box Operations Problem
“Logs can tell you what happened. They cannot tell you why it happened,” Verma says. “Systems are running in production that even their creators cannot fully interpret.”
Verma’s core argument is that conventional monitoring tools can answer what happened within an AI system, but not why. An organisation can observe that a model produced a particular output, that a system experienced downtime, or that a workflow completed successfully or unsuccessfully. What those tools cannot reliably explain is why the AI made a specific decision, why its behaviour changed between similar tasks, what specifically caused a failure, or which step in a multi-stage reasoning process produced a given result.
This gap, in Verma’s framing, creates what he terms black box operations: a situation in which the teams that built and deployed an AI system cannot fully explain how it is actually functioning in production. Verma points to developments at OpenAI during 2025, where the expansion of agentic systems capable of navigating tools and operating across environments with increasing independence surfaced a specific challenge for developers: tracing how an agent arrived at a particular decision across long, multi-step operations, particularly where the agent had adjusted its own plan or reinterpreted its objective along the way.
Why This Matters for Accountability and Regulation
Verma identifies three specific categories of operational risk that emerge from this observability gap. The first is non-reproducibility: the same input may not produce the same reasoning path on a subsequent run, making it difficult to verify or validate system behaviour. The second is audit gaps: reconstructing the chain of decisions that led to a particular outcome becomes genuinely difficult when the reasoning process itself was not captured in a structured way. The third is accountability: in regulated environments, organisations may be unable to justify or explain outcomes that their own AI systems produced.
Verma’s position is that AI observability is moving beyond traditional logging into a distinct set of capabilities: prompt tracing, monitoring of reasoning paths, tracking of memory states across multi-step tasks, behavioural analytics, and visibility into how an AI system interacts with external tools. “In software, the code documents the app,” Verma says, citing a framing used by LangChain that has shaped his thinking. “In AI, the traces do.”
From Engineering Concern to Governance Requirement
A central theme in Verma’s analysis is that observability is increasingly being treated not purely as an engineering concern but as a component of AI governance itself. As organisations deploy AI systems capable of independent reasoning across business-critical workflows, the ability to explain and reconstruct system behaviour becomes a prerequisite for operating those systems responsibly, particularly in contexts where decisions need to be justified to regulators, auditors, or affected parties.
“My view on this is that AI structures and assets will stop being limited to intelligence alone,” Verma says. “It will be defined by how visible the processes are. Enterprises that solve the observability problem early will lead the race because they can grow with confidence while others continue to manage unclear risks.”
What Verma Believes Will Define the Next Phase of AI Adoption
Verma’s broader view is that the trajectory of enterprise AI has shifted. The initial focus across the industry was on capability and speed, building systems that could operate independently across an increasing range of tasks. What organisations are now confronting is that independent systems whose internal processes cannot be examined represent a significant operational risk regardless of how capable those systems are.
“Organizations that master that layer, the visibility layer, will define the next era of AI,” Verma says.
The result, in his assessment, is a shift toward behavioural monitoring, cognitive telemetry, and the auditing of reasoning itself, disciplines that did not meaningfully exist in conventional software observability and that he believes will increasingly distinguish organisations that scale AI adoption with confidence from those that continue to manage largely unquantified risk.