AI infrastructure survey reveals GPU challenges

18
computing AI artificial intelligence code
Photo: Pexels/Pixabay

ClearML has announced the findings from a global AI survey recently conducted with FuriosaAI and the AI Infrastructure Alliance (AIIA).

The new report, The State of AI Infrastructure at Scale 2024: Unveiling Future Landscapes, Key Insights, and Business Benchmarks, includes responses from AI/ML and technology leaders at 1,000 companies of various sizes across North America, Europe, and Asia Pacific.

The report focuses on: how executives are building their AI infrastructure; the critical benchmarks and key challenges they face; and how they rank priorities when evaluating AI infrastructure solutions against their business use cases.

The report dives into respondents’ current scheduling, compute, and AI/ML needs for training and deploying models and AI framework plans for 2024-2025.

One of the primary drivers propelling hypergrowth in the AI infrastructure market is the realization among organizations of how AI can drive their operational efficiency and workforce productivity as the leading business use case. Companies are recognizing the need for Gen AI solutions to extract actionable insights from their internal knowledge bases and plan to deploy Gen AI to boost their competitive edge, enhance knowledge worker productivity, and impact their bottom line.

As companies navigate the AI infrastructure market, they are seeking clarity, peer insights and reviews, as well as industry benchmarks on AI/ML platforms and compute. To understand executives’ biggest pain points in moving AI/ML to production, this survey examined not only model training, but also model serving and inference.

“Our research shows that while most organizations are planning to expand their AI infrastructure, they can’t afford to move too fast in deploying Generative AI at scale at the cost of not prioritizing the right use cases,” said Noam Harel, ClearML’s CMO and GM, North America.

“We also explore the myriad challenges organizations face in their current AI workloads and how their ambitious plans for the future signal a need for highly performant, cost-effective ways to optimize GPU utilization (or find alternatives to GPUs), and harness seamless, end-to-end AI/ML platforms to drive effective, self-serve compute orchestration and scheduling with maximum utilization.”

“There are lots of claims about how businesses are addressing their rapidly evolving AI infrastructure resource needs and incorporating Generative AI into their products. This report provides hard data to answer these questions,” said June Paik, FuriosaAI’s CEO.

“In particular, it shows how businesses are actively looking for new, cost-effective options for inference compute. We’re excited to see that our second-gen product, launching later this year, directly addresses one of the top concerns cited in the report.”

Key insights

The report shows 96% of respondents plan to expand their AI compute infrastructure with availability, cost, and infrastructure challenges weighing on their minds, with 40% considering more on-premise and 60% considering more cloud, and they are looking for flexibility and speed. The top concern for cloud compute is wastage/idle costs.

It also reveals that 95% of executives reported that having and using Open Source technology is important for their organization. 96% are focused on customizing Open Source models. PyTorch is their framework of choice.

The findings also include that 74% of companies are dissatisfied with their current job scheduling and orchestration tools and face compute resource on-demand allocation and team productivity constraints. As such, 74% of respondents see value in having compute and scheduling functionality as part of a single, unified AI/ML platform (instead of cobbling together an AI infrastructure tech stack of stand-alone point solutions), but only 19% of respondents actually have a scheduling tool that supports the ability to view and manage jobs within queues and effectively optimize GPU utilization. Also, 93% of surveyed executives believe that AI team productivity would substantially increase if compute resources could be self-served.

Optimizing GPU utilization and GPU partitioning are major concerns, according to the report, with the majority of GPUs underutilized during peak times. It shows that 40% of respondents, regardless of company size, are planning to use orchestration and scheduling technology to maximize their current compute investments of their existing AI infrastructure. Only 42% of companies have the ability to manage Dynamic MiG/GPU partitioning capabilities to optimize GPU utilization.

Cost is the key buying factor for inference compute. To address GPU scarcity, 52% of respondents reported actively looking for cost-effective alternatives to GPUs for inference in 2024 as compared to 27% for training. Twenty percent were interested in cost-effective alternatives to GPU but were not aware of existing alternatives. This indicates cost is a key buying factor for inference solutions. While industries are still in early days for inference, the demand for cost-efficient inference compute will grow.

The biggest challenges for compute were latency, followed by access to compute and power consumption. More than half of respondents plan to use language models (like LLama), followed by embedding models (BERT and family) in their commercial deployments. Mitigating compute challenges will be essential in their plans.

Jim Cornall is editor of Deeptech Digest and publisher at Ayr Coastal Media. He is an award-winning writer, editor, photographer, broadcaster, designer and author. Contact Jim here.