Why Your Next AI Decision Is a Networking Decision

I'll be honest, I didn't expect much from HPE Discover. I've watched HPE for years and never fully thought we'd see the industry be where it is today. I felt that HPE didn't have a clear path forward.

Then I saw the announcements and the hardware up close. It became clear to me that the way enterprises run AI is about to put pressure on the network. That's the conversation I want to have.

The thing nobody is talking about

There is no debate about going cloud-first. We don't see much infrastructure on-premises unless it's a datacenter. The conversation at HPE Discover told a different story.

Compute is coming back on-premises. Not for everything. Low-risk workloads will stay in the cloud, because for those, inference is cheap and the convenience is worth it. But sensitive data, agentic AI workloads, and anything that touches internal information is moving back into private data centers.

We may be at the beginning of a real shift, and the drivers aren't what most people assume. It's not just data privacy and security. Those definitely matter. But also the economics of tokens.

A chatbot answers a question and stops. An agent doesn't. Agentic systems reason in loops, call tools, retry, and leverage workflows, and every one of those steps burns tokens. When you run that continuously against your operational data, public inference models stop looking like a convenience and start looking like a huge burn in your budget. The math that made cloud obvious for a single API call changes when the workload runs all day, every day, and at scale. At that point, owning the GPUs and running inference on-premises becomes the cheaper answer.

So the question every CIO should be asking isn't "cloud or on-prem."

It's "at what volume does my agentic workload justify bringing inference in-house?"

Pulling AI workloads back on-premises forces a network infrastructure upgrade. Moving large AI workloads in-house means your local network has to handle data transfer most enterprise networks were never designed for. If you're running aging switching infrastructure, that's your next project, and it's a budget line you haven't thought of yet.

The system becomes continuous

One slide at the CTO keynote stuck with me. It read: "The system becomes continuous." The idea is that agentic AI doesn't run as a request and a response. It runs as a loop. Observe, reason, act, validate, and then it does it all over again.

That sounds like a software story but it's also a networking story.

When an agent observes, reasons, acts, and validates continuously against your data, it generates a constant stream of traffic between compute, storage, and the data it's reasoning over. It's not a batch job. It's high-volume data moving as long as the agent is running.

HPE sees this coming. The announcements that mattered weren't about AI features, they were about the plumbing. The Alletra Storage MP X10000 was pitched as built for inference at scale. The ProLiant Compute DL380 Gen 12 is the on-prem horsepower to run the workloads. The building blocks for running agentic AI in your own data center are here.

Here's the part that's easy to miss. You can buy the fastest inference storage and the compute on the market, and your network becomes the bottleneck the moment those agents go live. If your switching fabric was sized for people accessing cloud infrastructure and video meetings, it was never designed to carry AI data movement between racks. Buy the compute and the storage, and you've made a networking decision too. Most people don't realize that until later.

That's the point I keep coming back to. Most organizations will buy the compute and the storage first, then discover the network is a constraint.

The honest operator take

HPE is pushing to be at the forefront of on-premises AI infrastructure, including what they're doing with NVIDIA, and that's directionally the correct push for where enterprise IT is going.

But the gap between the vendor vision and production is still significant. Most organizations don't have the AI governance frameworks in place to deploy agentic infrastructure responsibly. Most don't have a clear model of their projected token spend, which means they can't actually run the on-prem versus cloud cost comparison that should be driving the decision. And most lean IT teams don't have the capacity to operationalize a platform shift of this scale without an implementation partner.

The technology is ready before the organizations are. That's the reality of every major infrastructure shift we've seen.

What a CIO should do with this

This is the time to carefully pilot the technology, not something to act on immediately. It's worth having a serious look at the on-prem AI shift. If you aren't leveraging any AI, then you're going to be left behind.

So do three things before you sign anything. Look into a token cost model for your agentic workloads for the cloud vs on-prem discussions. Do the AI governance work, because the vendors won't do it for you. And be honest about what your team can operationalize without help.

Then buy the infrastructure that supports the outcome. Don't let the demo become your deployment plan. That's the lesson I keep learning.