Industry leaders said businesses must focus on infrastructure to keep AI chips running efficiently. While companies spend crores on processors, poor storage and network speeds often cause bottlenecks. Sharad Sanghi noted, “Buying the latest GPUs is not enough.” Without supporting systems, expensive hardware fails to deliver the expected financial returns.
Synopsis
Industry leaders said businesses must pay closer attention to the systems around their processors, which actually decide the return on their investments.
Companies can spend crores on the latest artificial intelligence (AI) chips but not building the surrounding infrastructure around them can stymie their optimal utilisation, especially as storage and network bottlenecks push up AI costs, according to industry leaders.
As India expands its AI computing capacity, executives across cloud services, networking and power infrastructure suggest businesses to pay closer attention to the systems around their processors, which actually decide the return on their investments.
For Sharad Sanghi, co-founder and chief executive of AI cloud company Neysa, the problem begins when companies focus on buying powerful chips without investments to keep them working efficiently.
“One thing enterprise customers sometimes miss is that buying the latest GPUs is not enough. They may not invest in the right storage and networking, so storage or input/output becomes the bottleneck and GPU [graphics processing units] utilisation remains low,” Sanghi had told ET AI.
Neysa has 12,500 GPUs live and plans to add another 8,000 to serve Indian businesses and global customers, Sanghi said.
Live Events
The cost of waiting for data
Sanghi said companies need to measure how much of their GPU capacity they actually use, since the lack of supporting systems can prevent expensive hardware from delivering the returns they expect.
“We help customers improve utilisation because that is how they get a return on their investment. If they use only 30% of their GPU capacity, they will not get the returns they should, and AI will look expensive,” he said.
Gautam Billa, vice-president of solutions engineering at Ciena, described how the problem can arise when companies train AI models across GPU clusters in different locations.
The clusters must exchange data as training progresses; a slow connection can leave processors waiting and reduce the amount of work they complete, Billa told ET AI.
“So, basically what really happens is your GPU idle time goes up. The maximum investment that you make is in GPUs and you want them to work as optimally as possible, as efficiently as possible. That’s the reason the network speed is extremely important,” Billa said.
AI agents add to that pressure because they can work continuously and exchange information about their tasks, requiring network operators to prepare for sustained traffic, he explained.
Preparing systems for future chips
Keeping chips productive also means planning for the electricity needs of future hardware, according to Greg Funk, vice-president of modular power converters at Vertiv.
AI servers concentrate power consumption and heat within smaller spaces, Funk said, making the power requirements of each rack, which houses the servers, an important consideration.
He said higher-voltage distribution starts to make sense as rack requirements approach 300 kilowatts, because supplying increasingly powerful equipment at lower voltages requires impractical amounts of copper.
Raising the voltage reduces the current needed to deliver the same power, allowing operators to use less copper, he said.
Operators could introduce 800-volt direct current in stages by converting selected rows of servers and expanding the system as their requirements grow, Funk added.
He said he had not seen AI projects in India get delayed due to power constraints, but operators were already deciding on the infrastructure for newer chips.
“Data centre operators are asking what investments they need to make now to be ready in a year or two, when these chips arrive and they want to deploy the servers. The last thing they want is to be sitting on millions of dollars’ worth of GPUs they cannot turn on,” he had told ET AI.
Matching the chip to the job
Sanghi said companies can also improve returns by choosing hardware around a specific task, since older GPUs or processors may meet their current needs.
He said businesses do not need Nvidia’s Blackwell chips for every workload, and an H100 can suffice if it accommodates the model and delivers the required performance.
Sanghi cited an unnamed payments company that runs AI jobs in batches on spare CPU servers, using existing capacity that does not require the latest GPUs.
“I am not saying companies should avoid the latest GPUs. They should invest if they need them. Buying the latest hardware without understanding how it will be used or calculating the actual return on investment leads to mistakes. That is when companies conclude that AI is too expensive,” Sanghi said.
US mid-term elections – Part 2: What does it mean for Trump to lose control of Congress?
The ABCD of the US mid-term elections – Part 1: Why it matters across the globe
Curbing rupee bets: What the RBI’s new rules mean for the hedge-now-or-wait question