Visualização de leitura

From tokens to terabytes: Building reactive generative media pipelines

For the first three years of the generative AI wave, the output of a model was a string. You called an API, you got tokens back, you rendered them in a chat window or wrote them to a row in Postgres. The economics of that pipeline were dominated by inference cost. Storage was a rounding error.

That era is over. The output of a modern generative pipeline is an asset: a 4K video clip, a stem-separated audio track, a 50-megapixel product render, a 3D mesh with PBR textures. Generative AI has gone from text-centric to asset-centric, and the architectural center of gravity has moved with it. The teams building durable advantages in generative media right now are the ones treating their storage layer as a pipeline component rather than a destination.

This is a good problem. It is the problem you get when your pipeline works.

Asset-centric changes the shape of the system

Text pipelines are stateless in practice. A prompt goes in, a response comes out and the interesting state lives in a database. You can rebuild almost any artifact by re-running the call.

Media pipelines are not like that. Every stage produces a large, opaque binary that the next stage consumes. A single finished deliverable might traverse a dozen of them: prompt expansion, base generation, upscale, frame interpolation, color pass, audio generation, mix, mux, transcode to delivery formats, thumbnail extraction. Each stage writes an intermediate. Each intermediate is expensive enough to regenerate that you keep it.

The result is a system where the objects are the state. Your object store stops being a place you put things when you are finished and becomes the substrate the pipeline runs on.

Adoption is past experimentation, and the volume is in production

Advertising has the clearest numbers. IAB’s 2026 Digital Video Ad Spend and Strategy Report finds that nearly two in three digital video buyers now use generative AI for creative, up from half in 2025. A third of their ad assets are expected to involve generative AI this year, up from a quarter in 2025, with buyers projecting 43 percent by 2027. That is happening inside a U.S. digital video ad market IAB projects will pass $80 billion in 2026, growing 11 percent year over year, nearly 20 percent faster than the total ad market.

The interesting detail for architects is what the creative is used for. IAB’s prior-year data showed buyers reaching for generative AI specifically to produce audience-specific versions of an ad, visual style variations and contextually adapted cuts. That is not one asset per campaign. That is a matrix.

Games are the instructive counterexample. GDC’s 2026 State of the Game Industry puts generative AI use at 36 percent of industry professionals and 30 percent at game studios specifically, but the usage breakdown is dominated by language models rather than media generation: research and brainstorming at 81 percent, code assistance and routine writing at 47 percent each, prototyping at 35 percent. The most-used tools are ChatGPT, Gemini and Copilot. Sentiment is sharply negative, with 52 percent saying generative AI is having a negative impact on the industry, rising to 64 percent among visual and technical artists. Asset-centric pipelines have not landed in games the way they have in advertising, and the constraint is as much workforce and provenance as it is tooling.

Elsewhere, the pattern holds even where the survey data is thinner: e-commerce teams generating on-model imagery per SKU per segment, localization pipelines producing dubbed and lip-synced variants per market, previsualization work that used to require an art department.

What the adopting categories have in common is that none of them produce one asset per request. They produce a set. The pipeline is judged on how many viable options it surfaces, which means a better pipeline is, definitionally, a pipeline that writes more bytes.

Reactive architecture, because the model layer will not hold still

The model landscape resets on a cadence measured in weeks. A new video model ships with better temporal coherence. A new audio model handles multilingual prosody properly. A new image model finally gets text rendering right. If your pipeline requires an engineering sprint to adopt a new model, you are structurally behind teams whose pipelines do not.

Reactive architecture is the answer, and it means two specific things.

  1. Model-agnostic stages. Each stage of the pipeline should express a contract in terms of inputs and outputs, not in terms of a vendor. A generation stage takes a prompt and conditioning assets and produces a video at a declared resolution and duration. Which model backs it is configuration. Swapping providers should be a config change and an eval run, not a refactor.
  2. Event-driven orchestration. Polling-based orchestration couples your stages to a scheduler and makes each new stage a change to the control plane. Event-driven orchestration inverts it: a stage completes, it writes its output, the write itself is the signal that the next stage should start. Adding a stage means subscribing to an event, not modifying a DAG definition that six other teams depend on.

This is where storage stops being passive. Object storage that emits events on write lets your bucket act as the message bus for the pipeline. B2 Event Notifications send a signed HTTP POST to a webhook endpoint when objects are created, updated or deleted, with rules scoped per bucket and filterable by prefix. That prefix filter is the part that matters architecturally: if your bucket is organized by stage, a rule on stage/upscale/ is a subscription to “upscale finished” without any code knowing what upscale is. Custom headers on the notification carry auth tokens or context to the target, so the endpoint can be a queue, a serverless function or a workflow platform rather than a service you had to build.

A completed upscale triggers the color pass. A completed mux triggers the transcode fan-out. A completed transcode triggers the CDN warm and the catalog write. The storage layer sequences the work, which removes an entire class of orchestration glue from your codebase and removes polling latency along with it.

Quality improvements arrive as file size increases

Every generation of media models improves along axes that all cost bytes. Resolution goes up. Frame rate goes up. Duration limits extend. Bit depth and color fidelity improve. Audio moves from mono to multi-channel. Compression artifacts that were acceptable at 720p are not acceptable at 4K, so teams move to higher bitrates and, for anything entering a post pipeline, to intermediate codecs.

The arithmetic is worth doing explicitly. A 10-second clip in a delivery-grade H.264 4K encode at 50 Mbps is roughly 60 MB. The same ten seconds as a ProRes 422 HQ intermediate, which Apple targets at 884 Mbps for 3840×2160 at 30p, is 1.1 GB. That is roughly 18 times the size, and intermediates are exactly what you keep between stages. Now assume your pipeline generates eight candidates per brief because your creative director wants options, and each candidate produces four intermediates before final. That is one brief consuming tens of gigabytes.

Nobody plans for that in a proof of concept. Everybody encounters it in month four of production.

The iteration multiplier

Here is the part that surprises teams: robustness and storage growth are the same curve.

A fragile pipeline produces one output per request because that is all it can manage. A robust pipeline produces candidates, keeps the rejects for training and eval, versions every asset so a creative decision can be reverted, retains intermediates so a late note does not require regenerating from the prompt, and derives proxies, thumbnails and per-platform cuts from every approved master.

Each of those is the correct engineering decision. Together they mean that improving your pipeline increases your storage footprint superlinearly relative to your output volume. If your unit economics assume storage scales with delivered assets, they are wrong. Storage scales with attempts multiplied by stages multiplied by versions multiplied by derivatives.

This is why storage strategy has to be a design input rather than a line item you discover on an invoice. The two things that turn it from a manageable cost into a structural problem are egress pricing and the absence of a lifecycle policy. Egress hurts most in the exact architecture described above because a multi-stage pipeline repeatedly reads its own intermediates, and a distribution layer constantly reads masters. When every read carries a metered charge, the pipeline design that produces the best creative output is also the one that produces the worst bill, and teams start making architectural compromises to protect margin. Lifecycle policy hurts by omission: if you never decide what an intermediate is worth after 30 days, you pay to keep all of them forever.

What to put in place now

If generative media is core to what you are building, four decisions determine whether your storage layer accelerates you or constrains you:

  1. Choose a storage economic model that does not penalize reads. Understand your egress terms before your architecture depends on them. A pipeline that reads its own outputs at every stage is a read-heavy workload, and pricing that assumes write-once, read-rarely does not fit it. Model the ratio you actually expect: egress as a multiple of stored volume, not as an absolute. That ratio is the number to design against.
  2. Make writes trigger work. Use object-level event notifications as the pipeline’s signaling mechanism. This buys you loose coupling, lower latency between stages and the ability to add a stage without touching the orchestrator.
  3. Attach metadata at write time. Model version, prompt hash, parent asset, generation parameters, approval state. Metadata written at generation time is nearly free. Reconstructing provenance across a million objects later is not; provenance is what makes your rejected candidates usable as eval data and a training signal.
  4. Define lifecycle policy per artifact class. Masters, approved derivatives, intermediates and rejects have different retention values. Encode that as policy on day one rather than as a cleanup project in year two.

Which points at the useful way to think about the storage layer: in a pipeline where everything else churns, it is the constant. Models turn over every few weeks. Stages get swapped, added and reordered around them. Output volume compounds with every quality improvement. The one layer absorbing all of that without being redesigned is the one holding the assets, so it is worth choosing based on the characteristics that stay true while the rest moves. That is what we built B2 for. Always hot, so no stage waits on a rehydration to read what the last one wrote. No retention minimum or file size floor, so intermediates that were always disposable cost what they used. Egress scales to what you store rather than metered per read, so a pipeline that reads its own output is not penalized for being good at its job. The architecture above it should change every quarter. The storage underneath it should not have to.

The opportunity

The teams that will win in generative media are not the ones with privileged access to a model. Model access is converging toward commodities. The advantage is in the pipeline: how fast you can adopt a better model, how many candidates you can afford to generate, how much history you retain to evaluate and fine-tune against, and how cheaply you can move all of it.

Every one of those is a storage architecture question. Treat the storage layer as an active participant in the workflow and it becomes the thing that lets you iterate faster than your competition. Treat it as a bucket you dump finished files into, and it becomes the ceiling on how good your pipeline is allowed to get.

The assets are the product now. Architect accordingly.

Dell’s $95B AI backlog shows the infrastructure crunch is far from over

Dell Technologies is acknowledging that infrastructure and storage supply still can’t keep up with agentic AI’s insatiable appetite for resources.

The company this week reported a “record” AI backlog, with $95 billion in orders waiting to be filled. This dovetails with quarterly earnings reflecting a more than 50% year-over-year increase in AI demand.

On an earnings call, Dell COO Jeff Clarke acknowledged that supply constraints start with servers and storage, and span the stack to “just about every product going through a leading node.”

“We are doing everything we can to get more supply,” he said. “In today’s environment, that’s a very difficult task.”

A glimpse of infrastructure demands ahead

Dell reported that, in its financial quarter ending July 31, its revenue was $47 billion, reflecting 58% year-over-year growth. Moreover, revenue in its Dell Infrastructure Solutions Group (ISG) increased 89% to a record $31.8 billion.

Much of this growth is in servers, notably traditional central processing unit (CPU)-based servers that are increasingly supporting agentic AI workloads. Demand is “exceptionally strong” in this area, with earnings up 122% year-over-year.

Perhaps most tellingly when it comes to the ongoing demand, the company booked nearly $61 billion in AI server orders in the three months ending July 31; all told, over the last 12 months, it has inked more than $130 billion in AI server orders.

Clarke reported that Dell converted $131.7 billion of demand into orders over the last year, and that demand is broadening across enterprise customers, neoclouds, and sovereign cloud providers. To illustrate his point, he noted that the number of customers using Dell AI Factory, the company’s platform built to support AI workflows, has surpassed 6,500, and of those, 3,300 signed on in the last three quarters. Clarke pointed out that, by contrast, it took the company two years to sign on the first 3,200 after debuting Dell AI Factory in May 2024.

“Agentic demand is reshaping the data center,” Clarke said. Inference is “pure demand in our industry.” In fact, Dell anticipates that 3,600 quadrillion tokens will be in use by 2030, representing an 87x increase from today. Further, over that same period, training demand is predicted to grow to 850 zettaflops, a 5x jump.

“Enterprise agentic AI is expected to be the single largest workload by 2028,” Clarke said, and by 2030 will account for 75% of all data center demand.

Enterprises clamor for traditional servers

Dell is seeing a growing trend of customers requiring “meaningful CPU compute capacity” to support AI and agentic workflows. As evidence of this demand, in just its last two financial quarters, it has generated nearly as much revenue from traditional servers and networking as it has in any prior full year in company history.

Most of this growth comes from existing customers accelerating their investments in traditional IT environments to refresh, modernize, and bolster performance, efficiency, and resiliency. Dell anticipates “significant and durable” refreshes ahead, and heightened security and resiliency requirements are also increasing demand.

“AI requires modern, disaggregated architectures that keep data accessible and in motion across compute, storage, and networking,” Clarke noted. It is much more than assembling and delivering components; AI deployments require significant engineering, design, and deployment expertise. Some customer engagements, in fact, require upwards of 50 unique designs as enterprises optimize for workload performance, power, cooling and the data center environment, he claimed.

Enterprises want new servers with more cores, more dynamic random-access memory (DRAM), and more storage. However, the constraints remain the same: “DRAM, DRAM, DRAM, followed by NAND, NAND, NAND [flash memory],” Clarke said. There are “spotty” CPU and disk drive shortages, and constraints all the way down the supply chain, from microcontrollers to drives to transistors.

Large enterprises and multinational corporations across the globe “would prefer to have products now if we had the supply,” he said. “We are supply constrained in the sense of what we can build in any given quarter.”

This has led Dell to plan accordingly and optimize configurations with what “bits and bytes” they do have coming in to maximize outputs, with a focus on “getting it out the door,” Clarke said. There are associated lead times that the company is working through, but they’ve been able to “realize greater shipments.”

“We’ll continue to focus on trying to get more supply, and take the supply we have and optimize the output,” he said.

Reflecting increased need for storage as enterprises prep, manage, and protect huge volumes of data, Dell has also seen strong growth across its PowerFlex, PowerStore, PowerProtect, and PowerVault products.

“Demand remains broad based; enterprises continue to modernize their storage environments as data growth increases the importance of keeping data available and secure,” Clarke said.

How customers respond to shortages

Clarke acknowledged that modernization is driving higher core counts, more DRAM, and more storage. Those configurations “cost more than they did last quarter, and the quarter before, and the quarter before.”

Customers are adjusting to these price increases, he noted, deferring purchases because they are unable to sufficiently flex existing budget dollars. In other cases, enterprises are placing orders further in advance to ensure they have access to constrained supplies. “Large, sophisticated customers are acting, first and foremost,” Clarke said. Some are collaboratively planning with Dell to gain a view of their needs further into the future.

“That is a new phenomenon,” he said. “We are working through this demand environment that’s well ahead of supply, helping customers manage.”

This article originally appeared on Network World.

Mars consolidates complex data infrastructure in hybrid cloud

Brands like Snickers, M&M’s, and Twix are familiar to most consumers, but Mars Inc. doesn’t just produce snacks. The family-owned company, with a revenue of approximately $65 billion, is also one of the largest manufacturers of pet food and ready meals, and its more than 100 production facilities operate around the clock. Of course, this places considerable demands on its IT.

“Our team must ensure that every system, including production lines, runs at maximum performance so we can continuously deliver the products and services our customers value,” says Luciano Batista, the company’s VP of enterprise services delivery.

However, Batista and his team realized that the existing data infrastructure could no longer reliably support operations, especially during peak periods such as Halloween and the pre-Christmas shopping season. So with the support of hybrid, multi-cloud data storage service Everpure, Mars is rebuilding its data and IT infrastructure.

“The Everpure platform met all our requirements,” says Batista. “It’s a scalable platform that futureproofs our operations and integrates seamlessly with our hybrid cloud infrastructure.”

Unified storage environment 

Mars initially consolidated its complex network of storage systems for business-critical databases like Oracle and applications like SAP onto a single Everpure Flash Array system. These software-defined, all-flash storage arrays are available in versions for different workloads, and typical use cases include databases, virtualized environments, SAP applications, and AI and analytics applications. 

Mars has since expanded its flash array infrastructure and now supports mixed workloads, including VMware, Windows, and Linux in areas of production, development, and quality assurance. It also uses Everpure Flash Blade as the basis for the global SAP file system. And while Flash Array is optimized for structured data, the scale-out systems of the Flash Blade series are designed for unstructured information.

“At peak times, Everpure supports up to 300,000 IOPS without any performance degradation,” says Lincoln Silva, product owner for Linux and on-prem storage at Mars. From his perspective, another point speaks favorably of the new platform in that he estimates his team saves approximately three months of planning time thanks to the Evergreen subscription model. This is because the vendor provides regular updates for the storage platform’s hardware and software. As a result, Mars’ IT professionals can focus on more critical tasks. 

Basis for hybrid cloud strategy

Mars also works with choice vendors to implement its approach to cloud. Dedicated local storage capabilities, for instance, are being integrated into Microsoft Azure cloud workloads, which simplifies restore processes and increases resilience.

Snapshots from the local environment can be replicated to the cloud, too. Recovery point objectives (RPEs) of four to 24 hours are available, depending on system priority. “Our success is also the success of our partners,” Batista says. “We embrace a spirit of reciprocity to get the most out of our collaboration.”

The hybrid cloud allows Mars to run VMware workloads and extend its IT infrastructure to the cloud as needed. And the company aims to expand its use of cloud-native applications via Microsoft Azure at a lower cost.

“We’re seeing a data reduction ratio of 18 to one. That’s nine times the expected compression rate,” Batista adds. “This puts us on track to save up to 50% on cloud storage costs. We can now work more efficiently and make better decisions thanks to intelligent solutions and automation.”

Fewer racks and lower power consumption

By consolidating on the flash platform, Mars has also reduced the space requirements and power consumption of its data centers so they only use one sixth of the power, and the number of racks has decreased significantly.

“We’re shaping a sustainable future by changing the way we work,” says Batista. “The decisions we make today will impact the world we leave behind, and Everpure aligns with our commitment to thinking in generations, not just business quarters.”

❌