Fraunhofer IIS Walks the Line for Edge AI
April 2, 2025
Pat Brans
The institute’s development projects strike a balance between hardware and software to optimize AI model performance on small processors.
Advertisement
Before edge AI can deliver a full range of new applications, there is work to be done to improve the way models operate in a constrained environment—without sacrificing accuracy. As an applied research institute in the field of integrated circuits, Fraunhofer IIS develops small-form–factor AI processors and optimized AI software to address the challenges posed by the edge environment.
“When it comes to hardware, two research paths are given the highest priority,” said Nicolas Witt, machine intelligence department lead and leader of the edge AI special interest group at Fraunhofer IIS.
Fraunhofer IIS’s Nicolas Witt
One is an analog deep neural network accelerator that uses in-memory computing based on analog signals from electric current. “We have our Adelia accelerator, which accelerates neural networks and does the processing in a very low-energy manner,” Witt said. “We need up to 1,000× less energy than what’s required by microcontrollers, because Adelia only uses power when it really computes.”
The other path is around spiking neural networks, which are inspired by what we know about the human brain.“We have accelerators for these types of neural networks too,” Witt said. “Having specialized hardware for that enables us to get to even smaller form factors and devices [with] less energy consumption.”
Fraunhofer’s Adelia accelerator (Source: Adobe Stock/Fraunhofer IIS)
Hardware-aware deep compression of AI models
In addition to its work on hardware, Fraunhofer IIS develops software tools to optimize AI models for use in constrained edge devices without losing too much accuracy. The small devices have constrained computational and memory resources, and power consumption must be minimized. But thermal factors are also a big concern, because heat cannot be dissipated easily on densely packed hardware.
“If you generate a lot of heat through processing, you hit a heat wall, which becomes a major problem,” Witt said. “We’re researching ways of scaling down processing at the edge, especially for sensor applications.”
Fraunhofer researchers use multi-objective optimization to address several metrics at once, working to maximize AI performance (for example, speed and accuracy) while minimizing the number of operations and the amount of memory needed to carry out AI functions. “After the multi-objective optimization, you get AI models that are small but also capable when it comes to processing data,” Witt said.
“One technique we use to reduce the size of a model is deep compression,” he said. “Deep compression denotes pruning, getting rid of redundant compute paths in a neural network. That makes the whole model smaller.
“Another technique is quantization, restricting the resolution of all the numbers. Typically, quantization is done down to 16- or 8-bit [resolution], but it can get down even lower than that. There are networks that can be processed with 1-bit resolution. You have to do some computation tricks, but it’s possible.”
Losses are generally low with quantization because it involves only a change in resolution, Witt said, but pruning a model often results in non-negligible loss of functionality. Fraunhofer and other institutes are conducting research on the redundancies in the neural networks and what gets lost when they’re removed. The work in this area is similar to research in explainable AI, which tries to map network structures to specific behaviors. Witt identified “the fundamental question” here as, “What is the minimal AI model that delivers the functionality and accuracy my application needs?” Finding out is an active field of research.
While these deeper questions remain unanswered, Fraunhofer IIS experts still use pruning to build prototypes, mitigating the potential loss of functionality on a case-by-case basis. “We usually start with an oversized network and apply deep compression to get it on smaller hardware,” Witt said. “And then you have to test quite carefully to find out which functionality is lost. Of course, the test functions have to be specific to the application.”
Fraunhofer IIS casts a large net in its AI research and applies its know-how to a range of projects with industrial partners. One application area is audio compression, transmission, and processing on wireless headsets. Another involves processing 5G data on small devices to determine position. Fraunhofer also worked on a demo involving cameras that can scan a crowd and count the people directly on the camera itself, without sending images to another device for further processing. And in the retail sector, the institute has worked on seamless shopping, which would enable shoppers to pay for their purchases without having to wait in a register line. Rather, a camera system would detect all of the items in a customer’s trolley, calculate the total, and complete the payment transaction wirelessly.
Perhaps the most unusual project Fraunhofer IIS has worked on involves the deployment of swarms of edge devices to monitor and detect disease in wildlife populations. Cameras are mounted on vultures, which congregate around the carcasses of other animals to feed. The cameras record video that is processed locally by AI agents, and then swarm processing is used to complete the vision tasks that would ordinarily require bigger neural networks.
Swarms of edge devices can be deployed to monitor wildlife populations. (Source: Fraunhofer IIS)
While models built for edge AI are more efficient than the big AI models, Witt said work must be done to better understand how humans and other living creatures learn, because those biological learning processes are far more energy-efficient than today’s machine-learning tools. “Our brains process information with far less energy than we currently do with AI,” he said. “It’s very important to invest in research to get AI functions to use much less energy.”
Further refining the interplay between hardware and software
According to Witt, one caveat for current edge AI projects is that developers must make decisions about the hardware on which the models will run before the models can be built. Research is needed to help developers match the intended AI functionality with the appropriate hardware more efficiently. Another line of research is around automating the development process, so that once the hardware is decided, most of the steps needed to make a model work for that platform can be performed with little or no human intervention.
Another hurdle is education. Data scientists and AI developers are highly trained to optimize the accuracy of AI models. Hardware and firmware developers are equally skilled at optimizing memory use and energy consumption. The challenge, Witt said, is “to find people knowledgeable in both areas, or even to get these two types of people to work together on edge AI.”
www.linkedin.com
..
