Skip to main content
Exploring what’s next in tech – Insights, information, and ideas for today’s IT and business leaders

Accelerating AI is not a one size fits all solution

AI performance is about to hit the wall. Hardware accelerators can help.

Marc Andreessen may have been right with his observation that software was eating the world. But today, it might be better said that AI is eating the world. Just as software forever changed how business is conducted, AI is opening up new possibilities that didn't exist a decade ago. This means an idea that was conceived in the 1950s and has endured multiple periods where it was nearly dormant is now running damn near everything. And there's a good chance you're interacting with some sort of AI on a regular basis.

More than 4 billion devices now employ an intelligent voice assistant like Alexa or Siri, a number that's likely to double by 2024. Each of the 86 million passenger vehicles rolling off assembly lines in 2022 will feature AI-powered cameras and other intelligent sensors. The global market for AI products is projected to surge from $36 billion in 2020 to $360 billion in 2028—growing nearly tenfold in just eight years.

According to McKinsey, half of all enterprises have adopted AI for at least one business function, whether it's generating new product ideas, predicting equipment failures, modeling investment risks, identifying cyberattacks, flagging fraudulent transactions, or delivering more personalized customer services.

AI is clearly here to stay. But as the algorithms get more sophisticated and the volume of data they're ingesting continues to grow at an exponential rate, bottlenecks are starting to emerge. We are reaching the performance, cost, and energy limits of conventional processors for training and implementing machine learning models.

Increasingly, AI applications need help, in the form of dedicated hardware accelerators.

What are AI hardware accelerators?

As the name implies, accelerators are silicon optimized to speed up common processes used by AI models.

The first AI hardware accelerators were graphics processing units (GPUs) originally built to render 3D images inside games. It turns out, the matrix math operations and massive parallelism used to generate a realistic Lara Croft are also great at performing the billions of simultaneous calculations that power deep neural networks.

Please read: The do's and don'ts of machine learning

GPUs changed how quickly machine learning models could be trained, while at the same time boosting their accuracy. And as data scientists tackled more complex problems, they could simply deploy more GPUs—at a cost, of course.

But this approach has started to reach the point of diminishing returns, note researchers at MIT's Computer Science and Artificial Intelligence Lab. The low-hanging fruit has already been picked; trying to model more sophisticated scenarios may require so much computing power that they're too expensive to solve, as well as environmentally disastrous.

For example, in 2017, researchers at Google Brain were able to create a model that cut the error rate of the first GPU-generated neural networks by half—but it required 1,000 times more computing power to achieve that milestone. This has driven a rapid rise in alternatives to traditional GPUs in generating machine learning models.

Why would your company need an AI accelerator?

We are in the middle of a Cambrian explosion of AI accelerators, says Sreenivas Rangan Sukumar, distinguished technologist in the Office of the CTO at Hewlett Packard Enterprise.

The newest breed of hardware accelerators is application-specific silicon tuned to handle a particular type of workload or work with specific algorithms. For example, Google's Tensor Processing Unit is designed to work with the open source TensorFlow machine learning platform.

These specialized chips can be more energy efficient and deliver better results at lower costs. But more important, using the right accelerator can save precious time, notes Paolo Faraboschi, HPE Fellow, AI research, at Hewlett Packard Labs. You don't want your very expensive, increasingly hard-to-find data scientists waiting days for a model to finish grinding through a billion calculations before they can analyze the results.

"In the right scenarios, accelerators can open up opportunities you didn't think were possible, like training a model in 20 minutes instead of three days," says Faraboschi. "So now, your data scientists can start evaluating a new drug right away. To me, that's a lot more interesting than someone who says their accelerator will give you X per-cent improvement in performance per dollar."

How do you pick the right accelerator?

More than 50 vendors are currently marketing AI accelerator hardware, each with its own specialties. And just as an 18-wheeler is ideal for hauling goods long distances but terrible for delivering items door to door, certain accelerators are better at some jobs than others.

But picking the right one is not a simple matter of applying accelerator A to machine learning problem B, says Sukumar. For example, do you need to accelerate the speed and accuracy of training the AI models, or do you want to boost how quickly the trained model makes inferences and predictions?

The choice of the right accelerator depends on a wide range of parameters to consider, including the type of workload, the algorithms being used, the architecture of the neural network, the size of the dataset, memory and bandwidth requirements, electricity needs, cost, and where the chips are going to be deployed.

Cloud-based accelerators can't provide the low latency required by assembly line robots or vehicles traveling 70 miles an hour. And accelerators that fit perfectly into a training application in a data center are unlikely to meet the power and cost requirements of an edge device.

Please read: The road to machine learning success is paved with failure

"If you want an AI engine to do visual recognition on a drone, it's not just about getting the best performance," says Faraboschi. "It's about getting the best performance at under 50 watts and 50 bucks."

Is one type of accelerator enough?

The more broadly enterprises deploy machine learning models, the more likely they'll need different accelerators for different applications, notes Sukumar. Heterogeneous processor architectures are likely to become the norm.

"If you're running hundreds of models and you want to ensure the greatest accuracy for each of them, you're not going to get away with just having one type of accelerator," he says.

But choosing the correct accelerator is not a simple process. One of the ways HPE helps customers is to step them through the decision tree—where is the data coming from, how big is the model, how fast do customers expect to train a model, what are their budget requirements, and so on—and then make recommendations, Sukumar adds.

"You can see a 20 to 200 times improvement by choosing the right accelerator, but it's not a trivial problem to solve," he says. "You need a trusted partner to not only help you pick the right hardware for the AI model but also make sure the acceleration applies to as many workloads as possible."

This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.