Why The AI Infrastructure Race Is Moving Beyond GPUs

The AI revolution may be powered by algorithms, yet its progress increasingly depends on something far more physical: how many computers companies can install, power, cool and afford. #Sponsored #AMDpartnership

For the past few years, much of the infrastructure conversation has focused on graphics processing units, or GPUs. Demand for the chips used to train and run large AI models has surged, turning computing capacity into a strategic resource and reshaping the technology industry.

However, AI systems are becoming more complex. Organizations must continue running databases, enterprise resource planning systems, virtual machines, websites, customer applications and countless other workloads. They are now adding AI training, inference, retrieval systems and autonomous agents to that already crowded environment.

This creates an infrastructure challenge extending well beyond the purchase of more GPUs.

Alexander Troshin, who leads technical business development for AMD data center CPU deployments across Europe, believes the answer begins with a return to basics.

“There’s no one-size-fits-all solution,” he told me during our recent conversation. “Make sure you know what you want to do and make sure you know how you want to do it, and if you don’t, make sure to ask the right questions.”

It sounds obvious. In practice, plenty of AI investments still begin with a technology purchase in search of a business problem.

Start With The Result, Then Build The Infrastructure

When generative AI first exploded into the mainstream, organizations raced to experiment. Speed was understandable. Nobody wanted to watch a major technological shift unfold from the sidelines.

The economics change when experiments become production systems. A promising pilot used by 50 employees can turn into a very different proposition when it reaches 50,000 employees, serves millions of customers or operates continuously across multiple markets.

Infrastructure decisions affect performance, latency, security, resilience, energy consumption and total cost of ownership. A poor decision can leave expensive GPUs waiting for data, create software licensing costs that were never anticipated or force an organization to rebuild parts of its data center sooner than expected.

Troshin described a familiar organizational trap: “We need to do AI, we want to do AI, but we don’t really know what we want to do.”

The better starting point is a clearly defined result. What task will the system perform? How many people will use it? How quickly must it respond? Where does the relevant data reside? What privacy restrictions apply? How much demand is expected at peak times? How expensive would failure or downtime become?

These questions help determine the most appropriate infrastructure. Some AI inference workloads can run effectively on existing CPU-based systems. Training large foundation models generally requires specialized GPU infrastructure. Larger models, high user volumes and demanding latency targets can also make accelerators more attractive for inference.

Troshin describes the CPU as a “Swiss knife” because it can support many kinds of work across databases, virtualization, enterprise software and AI inference. GPUs offer highly parallel processing for specialized, compute-intensive tasks. The practical challenge lies in establishing the right balance for a specific workload.

“There is no one size fits all solution, which makes it incredibly challenging,” he said. His advice is wonderfully unglamorous: “Testing, testing, testing, measuring.”

That principle should sit at the heart of every serious AI infrastructure strategy.

Agentic AI Changes The Computing Equation

The rise of agentic AI adds another layer of complexity.

A conventional chatbot receives a prompt, generates a response and waits for the next request. An AI agent can interpret a goal, build a plan, retrieve information, call applications, check permissions, execute code, update systems and evaluate the outcome. It may repeat this process several times before completing one assignment.

Each step creates computing work around the model. The agent must coordinate processes, access memory, communicate with databases, interact with conventional business applications and manage several tasks at once. Much of this orchestration relies heavily on CPUs.

One employee might also deploy several agents. Those agents may create sub-agents or collaborate with agents belonging to other departments. The number of active digital workers could therefore grow far faster than the number of human users.

Troshin explained that agentic AI creates a scaling challenge because agents can multiply the number of active workloads, placing greater demands on infrastructure as they operate and coordinate with each other.

This changes how leaders should think about AI capacity. Model performance remains important, although the surrounding workflow can become the real constraint. A powerful GPU delivers limited value if data arrives too slowly, orchestration services become overloaded or network bottlenecks leave the accelerator sitting idle.

The emerging architecture is likely to feature dense GPU systems for model execution alongside substantial CPU capacity for orchestration, tool use, security checks, data processing and conventional enterprise applications. Memory capacity, storage, networking and software will have equally important roles.

Buying the most powerful individual component does not guarantee the best overall system. Leaders should optimize the complete workflow.

Making Space And Power Available For AI

Energy may become one of the defining constraints on enterprise AI.

Data centers have finite supplies of electricity, cooling capacity and physical space. An organization cannot assume that additional computing power will always be available at the required location. Planning applications, power connections and new facilities can take years.

This makes infrastructure efficiency a strategic business concern. Every watt consumed by an outdated server is a watt unavailable for an AI workload. Every rack occupied by underused equipment limits the capacity available for future services.

Modernizing conventional IT can free space and power before an organization expands its data center footprint. Higher core density can allow companies to consolidate workloads onto fewer servers, helping reduce hardware requirements, software licenses and cooling demand.

Troshin says some legacy environments can achieve consolidation ratios of seven or nine older systems to one current system, depending on the workload and configuration. Such claims require careful testing against an organization’s real applications, of course. The underlying idea is sound: AI infrastructure planning should include the systems a company can remove, consolidate or repurpose.

AMD positions its EPYC™ server processors around three priorities: scalability, performance and energy efficiency. Its EPYC 9006 Series processors are designed for general enterprise computing, cloud services, AI inference and GPU-hosting systems.

Troshin argues that improving the CPU layer can increase the performance and efficiency of a GPU-heavy system, even when the GPU carries much of the model workload. He also sees CPU capacity becoming increasingly relevant as enterprises adopt agentic AI.

“AI transformation does not equal accelerators alone,” he said. “It equals overall system performance or platform, or the right thing for the right job.”

This may be the most useful lesson for business leaders. AI infrastructure is a system-level challenge. Processing power, memory, networking, storage, cooling, software and data architecture must work together.

Treat AI Infrastructure As A Business Portfolio

The next phase of enterprise AI will require a more disciplined approach to investment.

Organizations should view their infrastructure as a portfolio of computing resources. General-purpose CPUs can handle enterprise applications, orchestration and suitable inference workloads. GPUs can support model training and demanding inference. Cloud services can provide flexibility when demand fluctuates. On-premises infrastructure can offer greater control over data, latency and predictable workloads.

The best combination will vary by use case, industry and risk profile. It may also change as models become smaller, software improves and new processor generations arrive.

Total cost of ownership provides a more useful measure than the purchase price alone. Leaders need to account for energy, cooling, space, networking, software licensing, utilization, maintenance and the expected life of the hardware. They should also calculate the cost of overprovisioning capacity that sits idle.

Troshin recommends revisiting these assumptions continually because workloads and technical options keep changing. “It’s basically being able to revisit the fundamentals and being more agile from the technology perspective, from governance, from everything within the company,” he said.

That combination of business clarity and technical flexibility will become essential. Companies cannot build a fixed data center for an AI future whose precise shape remains uncertain. They can build an adaptable foundation, measure how real workloads behave and expand capacity in response to evidence.

The organizations that approach AI as a shopping exercise risk acquiring expensive islands of computing power. Those that begin with outcomes, understand their workloads and design the complete system will have a much stronger chance of turning infrastructure spending into business value.

The smartest AI strategy may therefore start with a surprisingly simple question: What exactly are we trying to make this machine do?

Learn more about AMD EPYC™ processors here: https://www.amd.com/en/resources/epyc-tools.html?utm_medium=social&utm_source=youtube&utm_campaign=q126_077pi&utm_content=bernardmarr

#Sponsored #AMDPartnership