Last year, my company decided to rebuild our enterprise AI platform, which our customers had been using for over a decade. But instead of simply updating it, we started over with a different ambition: to build an AI-native platform using AI.

Using AI to rebuild a platform of that size gave us first-hand experience with AI economics. We employed AI throughout the entire engineering process to write code, review changes, generate tests and accelerate development. It helped us reduce years of engineering to just a few months but also forced us to keep close watch over every single token we consumed and truly consider why we were consuming it.

This experience challenged one of our key assumptions: It turns out that token consumption is influenced as much by architectural decisions as by choice of model.

What token sprawl actually looks like

Indeed, we found that most of the token consumption waste wasn’t due to complex engineering tasks, but the architecture around them; this is something that is referred to as the architecture tax in enterprise AI.

Recursive agents kept spawning subtasks and subagents, and although they technically completed every task, each handoff introduced new prompts and context. Furthermore, frontier reasoning models were being wasted on routine classification tasks that deterministic code could have completed at a fraction of the cost. And, because we hadn’t clearly defined ‘success,’ retry loops continued firing and burning tokens to chase marginal gains, instead of improving business outcomes.  Finally, we experienced context inflation, with workflows carrying information from earlier steps long after they had stopped contributing to the decision being made.

None of these decisions seemed expensive in isolation. Things like a few extra prompts, an unnecessary reasoning step or a larger context window barely impacted the cost of a single workflow. Across millions of executions, however, they grew into a level of token consumption that no model upgrade alone could offset.

Designing for token efficiency

Once we figured out where all our tokens were going, we had to determine how to spend them more deliberately.

First, we accepted that not every step in an AI workflow needs the same amount of intelligence. Some tasks legitimately require reasoning, but others are structural, deterministic or repeatable, and are better handled through software, business logic or tools that are reused rather than reasoned through every execution.

Even when a model is required, not all tasks need the same amount of reasoning. This concept is better known as Reasoning Effort Optimization. If you’re designing systems, exploring alternatives or making difficult decisions, high reasoning effort has value. However, once you’ve established the approach, you can usually produce the same outcome with far less reasoning.

It’s comparable to organizations that structure work as a pyramid of expertise. The most experienced engineers tackle the most difficult problems, make important architectural decisions and handle any ambiguity that arises. Then, once their work is complete, they split it into small, well-defined tasks that other engineers and machines can execute consistently.

Allocation is only part of why this works, though; persistence is equally important. A senior engineer’s judgment gets built into interfaces, schemas and conventions, allowing everyone downstream to build on those decisions without having to derive them again. The same applies to AI systems. If you fail to capture their reasoning as a durable artifact like a graph, specification or compiled skill, the system repays for it every single time it runs.

We apply the same principle to AI systems. Advanced reasoning should only be used when judgment is required. Smaller models can handle well-defined implementation work, and deterministic software and reusable tools can execute tasks where the outcome is already known. The goal is to apply the right level of intelligence to each stage of the workflow.

Architect once, inherit everywhere

So, how much of an AI system should teams rebuild for every new workflow?

In our experience, very little. Architectural capabilities like evaluation, orchestration, guardrails, context management and auditability aren’t unique to a single AI application; every enterprise AI system eventually needs them. Rebuilding them for each workflow is a waste of time and money that forces every AI system to solve the same infrastructure problems independently.

It’s much more efficient to move these capabilities into a common harness. Orchestration, governance and evaluation can then be developed, tested and improved at the platform level rather than separately inside every workflow. Each new AI system can reuse this foundation while teams focus on the business logic that makes the application distinct.

This architectural discipline extends to the execution layer. The amount of computation an AI system performs before producing an outcome depends heavily on the context provided, how its instructions are designed and how its agents are structured.

A 2026 study found that consolidating three or four agents into a single call reduced token consumption by 53.7% and latency by 49.5%, with comparable accuracy. Most of these savings came from not repeatedly passing the same context between agents.

The study also demonstrated that, as the number of tasks assigned to each agent increased, the agent became less reliable at choosing the best way to approach each task. However, increased routing helped recover much of that performance. In short, AI efficiency depends on how the work is structured and routed, not just how many agents are involved.

Optimize throughout the lifecycle

Building the right architecture creates the foundation for continuous optimization.

Model selection, for example, shouldn’t be a one-time decision. As models improve and workloads evolve, it’s worth re-evaluating every AI workflow to determine whether the same outcome can be achieved with less reasoning and lower cost. Demonstrating that system design can outweigh raw model capability, Andrew Ng, a computer scientist, AI entrepreneur and Stanford University adjunct professor, highlighted that while zero-shot GPT-3.5 scored 48.1% on the HumanEval coding benchmark and GPT-4 scored 67.0%, wrapping GPT-3.5 in an iterative agentic loop pushed its score to 95.1%.

The same applies to cost visibility. Token economics should be visible across every workflow, model and task so teams can identify ways to reduce costs before they accumulate.

AI is much less expensive than it used to be, but lower costs encourage more use. Organizations can run more workloads and take on more ambitious problems, so it’s even more important to know where the AI is actually creating value.

Allocate intelligence by the value it creates

Most organizations start with fixed token budgets, because they are easy to administer. However, not every AI task has the same value.

Think about AI as an engineering resource, rather than a shared utility. The teams that consistently choose the right model for the right task and avoid unnecessary reasoning should get more compute because their tokens are producing better outcomes.

Over time, I expect enterprises to manage AI the same way they manage any scarce engineering resource — by using it where it has the greatest impact.

Working with large financial institutions, I’ve watched enterprise data consumption change drastically. The old model might have had 1,000 people using the same carefully designed application. These 1,000 users could reliably and predictably work from this application, and data access would typically occur in the same way.

That change was already underway as dashboards, APIs, notebooks, microservices and specialized applications multiplied the number and variety of consumers reaching the same underlying data. Agentic AI is turning this construction on its head. Those 1,000 people could now be working from 1,000 individualized dashboards, applications or agents. That creates more paths, data combinations and entitlement decisions, and each requires reconstruction.

What gets easier and more individualized at the application layer gets harder underneath it: infrastructure and InfoSec teams must support more dynamic demand, more access patterns and more control decisions. And that challenge is becoming more pressing as agentic AI broadens: 23% of organizations report scaling agentic systems, and an additional 39% are experimenting with agents.

The result has accelerated a broader shift from relatively predictable data consumption to fluid, individualized and difficult-to-govern access patterns across the same underlying data.

In capital markets, tolerance for inconsistencies is nonexistent. This environment has been an early proving ground for architectures designed to coordinate changing consumer patterns. The lesson is not that every workload should be centralized or routed through one platform, but rather that a governed data consumption layer can provide the leverage to reduce repeated integration, protect sensitive source systems and enforce consistent access, audit and runtime controls.

I use “governed data consumption layer” here to describe the consumption-facing role of a broader data fabric. The fabric connects and contextualizes data across enterprise systems, maintains reusable state and controls and distributes that information through governed interfaces. The consumption layer is where those capabilities meet the applications, users and agents requesting the data.

However, CIOs need to evaluate where this approach may prove valuable and where direct access may remain the better choice, if they wish to manage how a growing number of human and machine consumers access, combine and act on data. The architectural question is what changes when the consumer, its query pattern and its data requirements can no longer be known in advance.

Where a data fabric creates leverage

A data fabric can create significant leverage when growing numbers of consumers would otherwise have to independently reconstruct the same context, establish their own access paths and operate their own controls. A consumer may need a current business view that combines live activity with persistent context. A credit-card authorization, for example, may need to be evaluated against available credit, merchant history, fraud scores and card status. If multiple consumers independently reconstruct that same contextualized view, they duplicate processing and create more opportunities to arrive at inconsistent answers.

A data fabric separates the primary workload from data delivery in these cases. It allows the same working set to be reused, reducing pressure on source systems and making latency more predictable. This layer is also helpful when each subsequent application requires different views of the same information, based on their fields, histories, update rates or interfaces. The objective is to deliver data at the level of freshness each workflow requires; not every consumer needs to be real-time.

That separation also preserves optionality. A new application or AI workflow can gain governed access without requiring the underlying system to be replaced, migrated or exposed directly to every new consumer. Reuse solves only one part of the challenge. As consumption becomes less predictable, the layer also must govern what each consumer receives.

One pattern I describe as the “fire-hose approach” is sending consumers a broad stream of information and expecting them to discard what they do not need. That becomes harder to justify as consumers multiply and request different fields, filters and time windows. At the consumption layer, the fabric can instead tailor delivery more precisely to what each consumer requires, reducing unnecessary movement and downstream processing.

That precision also improves governance. If the request itself defines exactly what information a consumer receives, the organization maintains a more granular record of who requested what instead of knowing only that a user subscribed to a much broader stream.

Additionally, entitlements and audit requirements would need to remain consistent across these dashboards, APIs, notebooks, services and agents that request information. As access paths multiply, observability and governance practices must scale with them. As of writing, only one-third of surveyed organizations rate themselves at least moderately mature in AI strategy, governance and agentic governance.

I’m already seeing AI change the practical meaning of entitlement. An operations employee may legitimately hold broad credentials to perform a technical role but historically lack the time or expertise to use that access to derive sensitive business information. With an AI interface, the same employee may be able to ask for something like an entire desk’s P&L in a few keystrokes. The entitlement has not changed, but what the user can accomplish with it has. Once multiple consumers depend on that shared layer, governance is not solely an access-control problem. The layer itself must be operated as critical infrastructure.

High-consequence architecture demands more than a simple check to confirm it’s running, which only confirms the pipeline is alive. CIOs must know whether the state it is distributing is current, correct and reconstructable. As consumption becomes more individualized, visibility into which consumers are driving load, what queries are becoming common and how those patterns are changing over time is also required.

An enterprise needs to be able to explain which data, state, transformation and permissions produced a result. That means making sure the information is current, explainable and traceable. Here, the leverage of the fabric once again proves valuable. It can supply evidence of what was requested, which state was used, what transformation occurred and what was delivered.

With this foundation, organizations are better equipped to enforce governance at the point where access is granted, rather than leaving it until a final review checkpoint. This includes identity, access scope, consumption limits, tool permissions and audit. As consumers become capable of doing more than retrieving information, that governance also has to follow the progression from what they may access, to what operations they may perform with it, to what actions they may execute.

In higher-risk workflows, that final stage may require a human override. I’ve seen banks use kill switches that allow humans monitoring outliers to halt automated trading when something appears wrong. As automated consumers take on more decision-making, preserving that ability for intervention becomes more important.

When governance becomes part of how the organization operates, and not just how it says it should operate, later applications can reuse connectivity, mappings, state, controls and recovery procedures and trust that they align with the overall business view, which prevents every new consumer from requiring another independent integration, entitlement model and operational path.

The fabric becomes most useful when growing numbers of consumers need the same underlying data in different and increasingly difficult-to-predict ways. It helps provide this data while maintaining oversight and protecting source systems. However, that doesn’t mean that it’s the best choice for every organization.

When predictable consumption still makes direct access the better choice

The decision comes down to a practical tradeoff: a data fabric earns its place when the cost and risk of repeatedly integrating, contextualizing, governing and operating access for individual consumers exceeds the cost and risk of introducing shared infrastructure.

This fabric concentrates responsibility. If the layer can’t process requests, update its working state or distribute information quickly enough, it can become a bottleneck. And with multiple consumers relying on it, an outage, stale working state, faulty transformation or policy error can have a larger blast radius. Under these conditions, the layer can shift into a mechanism for propagating stale working state, bad mappings or entitlement errors.

That tradeoff matters because direct access remains sensible when the consumer is bounded, demand is predictable and the source was designed to serve the workload. The case is even stronger when the data requires little transformation or enrichment, entitlements are straightforward and there is limited expectation that later consumers will reuse the connection. Under those conditions, introducing another layer can add latency, complexity and another failure dependency without creating enough shared value to justify it.

This argues for incremental adoption rather than an enterprise-wide mandate. Start with a bounded workflow where shared access, state or governance can create measurable value, establish reusable connectivity and controls, then expand only when later consumers can reuse those assets. The shared layer should earn a broader role through demonstrated reuse rather than architectural preference alone.

Rather than architectural uniformity, the goal is to encourage optionality while creating shared consumption and governance capabilities where reuse genuinely creates leverage. CIOs do not need to predict every future consumer, but their architecture should govern the conditions under which new consumers access, combine and act on enterprise data.

Brands like Snickers, M&M’s, and Twix are familiar to most consumers, but Mars Inc. doesn’t just produce snacks. The family-owned company, with a revenue of approximately $65 billion, is also one of the largest manufacturers of pet food and ready meals, and its more than 100 production facilities operate around the clock. Of course, this places considerable demands on its IT.

“Our team must ensure that every system, including production lines, runs at maximum performance so we can continuously deliver the products and services our customers value,” says Luciano Batista, the company’s VP of enterprise services delivery.

However, Batista and his team realized that the existing data infrastructure could no longer reliably support operations, especially during peak periods such as Halloween and the pre-Christmas shopping season. So with the support of hybrid, multi-cloud data storage service Everpure, Mars is rebuilding its data and IT infrastructure.

“The Everpure platform met all our requirements,” says Batista. “It’s a scalable platform that futureproofs our operations and integrates seamlessly with our hybrid cloud infrastructure.”

Unified storage environment 

Mars initially consolidated its complex network of storage systems for business-critical databases like Oracle and applications like SAP onto a single Everpure Flash Array system. These software-defined, all-flash storage arrays are available in versions for different workloads, and typical use cases include databases, virtualized environments, SAP applications, and AI and analytics applications. 

Mars has since expanded its flash array infrastructure and now supports mixed workloads, including VMware, Windows, and Linux in areas of production, development, and quality assurance. It also uses Everpure Flash Blade as the basis for the global SAP file system. And while Flash Array is optimized for structured data, the scale-out systems of the Flash Blade series are designed for unstructured information.

“At peak times, Everpure supports up to 300,000 IOPS without any performance degradation,” says Lincoln Silva, product owner for Linux and on-prem storage at Mars. From his perspective, another point speaks favorably of the new platform in that he estimates his team saves approximately three months of planning time thanks to the Evergreen subscription model. This is because the vendor provides regular updates for the storage platform’s hardware and software. As a result, Mars’ IT professionals can focus on more critical tasks. 

Basis for hybrid cloud strategy

Mars also works with choice vendors to implement its approach to cloud. Dedicated local storage capabilities, for instance, are being integrated into Microsoft Azure cloud workloads, which simplifies restore processes and increases resilience.

Snapshots from the local environment can be replicated to the cloud, too. Recovery point objectives (RPEs) of four to 24 hours are available, depending on system priority. “Our success is also the success of our partners,” Batista says. “We embrace a spirit of reciprocity to get the most out of our collaboration.”

The hybrid cloud allows Mars to run VMware workloads and extend its IT infrastructure to the cloud as needed. And the company aims to expand its use of cloud-native applications via Microsoft Azure at a lower cost.

“We’re seeing a data reduction ratio of 18 to one. That’s nine times the expected compression rate,” Batista adds. “This puts us on track to save up to 50% on cloud storage costs. We can now work more efficiently and make better decisions thanks to intelligent solutions and automation.”

Fewer racks and lower power consumption

By consolidating on the flash platform, Mars has also reduced the space requirements and power consumption of its data centers so they only use one sixth of the power, and the number of racks has decreased significantly.

“We’re shaping a sustainable future by changing the way we work,” says Batista. “The decisions we make today will impact the world we leave behind, and Everpure aligns with our commitment to thinking in generations, not just business quarters.”

Enterprises have built their data systems for humans, but AI agents need a whole new infrastructure.

Separate research from Cloudera and Google/MIT found that, not surprisingly, there is fervent enterprise interest in AI agents, but underlying infrastructure struggles to keep up. Deployments continue to be hampered, sometimes even abandoned, largely due to issues with data access, context, and governance.

“Enterprise adoption of agentic AI is on the cusp of an extraordinary acceleration,” the Google/MIT report noted. “As organizations look to scale agentic AI across the enterprise, they cannot ignore their data systems.”

Resolving data bottlenecks, then, should be an immediate priority.

Projects delayed, inaccessible data

Cloudera’s report, created in partnership with Wakefield Research, describes the need for a “great AI re-architecture.”

Of the 1,500 enterprise architects and cloud infrastructure leads surveyed, a stunning 95% said they had delayed or cancelled AI projects, in some cases six or more, in the past year, due to issues with data governance, compliance, or regulatory issues.

A wide majority also reported that AI integrations have changed their data storage and architecture practices, AI workloads have increased infrastructure costs, and current data architecture requires a “significant overhaul” to meet AI goals.

“Even if enterprises are ready to use AI, many are coming to the realization that the foundational infrastructure it relies on is not,” the report noted.

Similarly, more than half of the 300 IT execs and heads of product, IT, data, and AI responding to the Google/MIT survey said they have paused or delayed the deployment of AI agents to address foundational data issues, such as siloes or lack of context. Further, more than half reported that legacy data systems are preventing them from scaling agentic AI and are having a “significant negative impact” on their AI ROI. High latency has also hampered AI from making decisions at “high velocity.”

“To make good decisions and take effective action, agentic systems need a data foundation that is multimodal, context-aware, and instantly available,” the report noted. “Legacy data systems struggle to meet these demands, ultimately compromising AI trustworthiness.”

Google and MIT identified several reasons that enterprises struggle to deploy AI agents, most notably:

• Entrenched siloes: Data sits in disconnected systems, or is “pocketed away” in different departments with no integration layer; it could also be in outdated formats, old management platforms or logs, or on obsolete IoT devices.
• Difficult-to-access data: “Dark” or unstructured data is contained in different formats like images, video, or PDFs.
• Insufficient access to real-time data: Legacy batch processing architectures can make in-time action a challenge.
• Lack of context: Agents often receive basic metadata rather than enterprise-specific semantics, so they struggle to make relevant connections or suggestions.

“Without this deep understanding, data cannot be highly relevant to specific use cases,” the report noted.

‘Data leaders’ versus ‘data laggards’

This is not to say that enterprises aren’t deploying AI; quite the contrary. Nearly all respondents (98%) to the Google/MIT survey are already using agentic AI or plan to soon. One in 10 is using it widely and nearly three quarters have deployed it in a limited fashion.

The most common uses for AI agents right now are in customer service (routing requests and resolving issues), IT systems management (managing user access and incident response), and IT security (anomaly detection and threat scanning). In the near future, the survey said, enterprises also plan to use AI agents in HR, finance, and supply chains.

“It is easy to understand why companies are eager to put AI agents to work,” the report noted. Agents can supercharge productivity and efficiency so employees can turn to more strategic work.

The enterprises seeing the most success are what Google and MIT refer to as “data leaders,” those that give AI systems access to more than 70% of their data. “Data laggards,” by contrast, share just 30% or less of their data with AI. Interestingly, 100% of data leaders say their agents make “mostly” or “consistently” accurate and relevant decisions, while just 22% of data laggards say they have that trust.

Agents need what the report calls “frictionless access” to operating systems, multimodal data, and context, helping them understand how data maps to different teams’ goals. “The most important initiative to enable scaling among all respondents is improving access to structured and unstructured data for AI agents.”

To successfully scale agents, the report recommended that enterprise leaders prioritize several data initiatives. First, improve access to data; discover, inventory, and classify data, both structured and “dark”/unstructured. After governance principles are applied, information can then be extracted and connected to AI agents.

Next, “put a premium on context” by giving agents enterprise-specific data. Replace batch processing with streaming and event-driven pipelines as well.

Finally, think AI-native. “AI-native systems are designed to operate in an AI environment and built from the outset to leverage AI for data management and decision-making,” the report noted.  

AI needs multimodal cloud environments

AI needs a lot of data that is often spread across on-premises systems and cloud, SaaS, and edge environments. In fact, 97% of respondents to Cloudera’s survey said they move data between environments monthly, and nearly one-third do so daily.

However, nearly three-quarters (73%) said AI integration makes data governance more complex.

“Enterprises had good governance systems when humans were the only ones accessing data manually,” the report noted. “But when thousands of agents provide support to employees, customers, or partners the story changes and governance requires another dimension.”

This makes private AI and data sovereignty critical, Cloudera said. Enterprises should have a data foundation that is unified and provides control over 100% of organizational data, wherever it resides. They must also think about where AI workloads run, while still maintaining control over sensitive data and keeping cost controls and flexibility in mind.

One notable trend Cloudera uncovered is a “resurgence” of on-premises and private cloud environments. Over the last 12 months, 66% of respondents moved AI workloads from public cloud back to on-premises or private cloud environments. And 25% said they plan to place more emphasis on a hybrid-first approach, 24% said they are planning to increase on-premises spend, and 22% plan to increase edge spend.

“IT leaders are putting more emphasis on moving workloads to the environments that they are best suited for, based on performance, cost, latency, governance, availability, and accessibility,” the report stated.