Good place to be because Australia is about to be flooded by a tsunami of cash from BRN.Hey Bravo,
Nice post !.......exactly the point Peter made when he was the acting Interim CEO, he produced at report regarding energy demand that was
going to be needed to "keep the lights on" at ALL Data Centre's, Server Farms and Ice Caves
Peter promoted "Beneficial AI" and "Green AI" and while we still adhere to those titles, we added "Essential AI"...all part of the suggestive
sales tool to promote what Brainchip truly stands for, to make this planet a better place for ALL, a selfless company attempting to make
disruptive change for the better.
I'm now back in New Zealand for 6-8 months, or maybe longer, just see how things play out.
I have to share this, I've only been at my property for 2 days and this afternoon I run into a couple of guys wondering around the area that
I co own, I asked if I could help them with something, well as the conversation expanded I got around to my favourite subject, that being AI
and Brainchip, these guys were IT specialists checking an issue with the Chinese owners Wi-Fi, anyway, one of the guys knew about Brainchip!
I asked, are you a shareholder ? answer no...I asked, how did you hear about us then ?...I read about it on the internet.....THEN he tells me
his friend works for Mercedes Benz!...my ears prick up and I say, what, in Germany ? and he replies, no in Wellington, he is in the Corporate
Division.
I couldn't help myself, did he say anything about Brainchip?.....answer no, but did say they (Mercedes) are definitely working with AI with Lidar
etc....which we already knew....but what are the odds, been here 2 days, some random person I have never met before knows of Brainchip,
isn't a current shareholder and blurts out the words Mercedes Benz....time to visit the Casino in Auckland (joke)
Regards....Tech (Karikari Peninsula) NZ
BRN Discussion Ongoing
- Thread starter TechGirl
- Start date
Tony Coles
Regular
https://www.embedded.com/top-5-reasons-why-cpu-is-the-best-processor-for-ai-inference/
Link above from ARM twitter X
29 Oct 2024 / 7:49 am
Ronan Naughton
4 min read
0
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misconception is that these AI workloads can only be processed in the cloud and data center. In fact, the majority of AI inference workloads, which are cheaper and faster to run than training, can be processed at the edge – on the actual devices.
Partner Content
Partner Content
20 Sep 2024
Real-Time Processing in High-Voltage Testing: Insights from HighVolt and Red Pitaya
By: Red Pitaya
Partner Content
3 Sep 2024
GigaDevice Semiconductor expands its Arm MCU product roadmap through Arm Total Access
By: GigaDevice Semiconductor Inc.
Partner Content
12 Jun 2024
IoT Innovations: Qualcomm debuts new technologies at Embedded World 2024
By: Qualcomm
The availability and growing AI capabilities of the CPU across today’s devices are helping to push more AI inference processing to the edge. While heterogeneous computing approaches provide the industry with the flexibility to use different computing components – including the CPU, GPU, and NPU – for different AI use cases and demands, AI inference in edge computing is where the CPU shines.
With this in mind, here are the top five reasons why the CPU is the best target for AI inference workloads.
ADVERTISEMENT
SCROLL DOWN TO CONTINUE READING
The availability of smaller LLMs, like the new Llama 3.2 1B and 3B releases, is critical to enabling AI inference at scale. Recently, Arm demonstrated that running the Llama 3.2 3B LLM on Arm-powered mobile devices through the Arm CPU-optimized kernels leads to a 5x improvement in prompt processing and a 3x improvement in token generation.
We are already seeing developers write more compact models to run on low-power processors and even microcontrollers, saving time and costs. Plumerai, which provides software solutions for accelerating neural networks on Arm Cortex-A and Cortex-M systems-on-chip (SoCs), runs just over 1MB of AI code on an Arm-based microcontroller that performs facial detection and recognition. Keen to preserve user privacy, all inference is done on the chip, so no facial features or other personal data are sent to the cloud for analysis.
Link above from ARM twitter X
29 Oct 2024 / 7:49 am
Top 5 Reasons why CPU is the Best Processor for AI Inference
Ronan Naughton
4 min read
0
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misconception is that these AI workloads can only be processed in the cloud and data center. In fact, the majority of AI inference workloads, which are cheaper and faster to run than training, can be processed at the edge – on the actual devices.
Partner Content
Partner Content
20 Sep 2024
Real-Time Processing in High-Voltage Testing: Insights from HighVolt and Red Pitaya
By: Red Pitaya
Partner Content
3 Sep 2024
GigaDevice Semiconductor expands its Arm MCU product roadmap through Arm Total Access
By: GigaDevice Semiconductor Inc.
Partner Content
12 Jun 2024
IoT Innovations: Qualcomm debuts new technologies at Embedded World 2024
By: Qualcomm
The availability and growing AI capabilities of the CPU across today’s devices are helping to push more AI inference processing to the edge. While heterogeneous computing approaches provide the industry with the flexibility to use different computing components – including the CPU, GPU, and NPU – for different AI use cases and demands, AI inference in edge computing is where the CPU shines.
With this in mind, here are the top five reasons why the CPU is the best target for AI inference workloads.
ADVERTISEMENT
SCROLL DOWN TO CONTINUE READING
The benefits of AI inference on the CPU
Efficiency at the edge
AI processing at the edge is important to the tech industry because the more AI processing at the edge, the more power is saved by not having to send data traveling to and from the cloud. This leads to significant energy and cost savings. The user also benefits from quicker, more responsive AI inference experiences, as well as greater privacy since data is processed locally. These are particularly important for power-constrained devices and edge applications, such as drones, smart wearables, and smart home devices, where power efficiency, latency, and security are paramount. In this context, the CPU plays a crucial role because it’s able to handle these AI inference tasks in the most efficient way possible.Versatility for various AI inference tasks
The CPU’s versatility allows it to handle a wide range of AI inference tasks, especially for applications and devices requiring quick responses and reliable performance. For example, real-time data processing tasks, like predictive maintenance, environmental monitoring, or autonomous navigation, are handled more efficiently and quickly on the CPU. In industrial IoT applications, this ensures that systems can respond to their environment, or any changes in its environment, in milliseconds. This is crucial for safety and functionality.Great performance for smaller AI Models
CPUs support a wide range of AI frameworks, like Meta’s PyTorch and ExecuTorch and Google AI Edge’s MediaPipe, making it easy to deploy large language models (LLMs) for AI inference. These LLMs are evolving at a rapid rate, with exceptional user experiences being unlocked by smaller compact models with an ever-decreasing number of parameters. The smaller the model, the more efficient and effective it runs on the CPU.The availability of smaller LLMs, like the new Llama 3.2 1B and 3B releases, is critical to enabling AI inference at scale. Recently, Arm demonstrated that running the Llama 3.2 3B LLM on Arm-powered mobile devices through the Arm CPU-optimized kernels leads to a 5x improvement in prompt processing and a 3x improvement in token generation.
We are already seeing developers write more compact models to run on low-power processors and even microcontrollers, saving time and costs. Plumerai, which provides software solutions for accelerating neural networks on Arm Cortex-A and Cortex-M systems-on-chip (SoCs), runs just over 1MB of AI code on an Arm-based microcontroller that performs facial detection and recognition. Keen to preserve user privacy, all inference is done on the chip, so no facial features or other personal data are sent to the cloud for analysis.
Greater flexibility and programmability for developers
The software community is actively choosing the CPU as the preferred path for targeting their AI workloads due to its flexibility and programmability. The greater flexibility of CPUs means developers can run a broader range of software in a greater variety of data formats without requiring developers to build multiple versions of their code. Meanwhile, every month there are new models with different architectures and quantization schemes emerging. As the CPU is highly programmable, these new models can be deployed on the CPU in a matter of hours.The architecture foundation for AI innovation
This developer innovation is built on the foundation of the CPU architecture, which continuously adds new features and instructions to process more advanced AI workloads. The ubiquity of the CPU means developers can then access these capabilities to accelerate and innovate AI-based experiences even further. In fact, the ongoing evolution of the CPU architecture has directly corresponded with the evolution of applications that are now faster and more intelligent.Why CPUs for AI inference are indispensable
CPUs are not just a component of system-on-chip (SoC) designs, they enable AI to be practical, efficient, and accessible across a wide variety of edge applications and devices. Offering a unique blend of efficiency, versatility, and accessibility, CPUs are indispensable for AI inference. They help reduce energy consumption and latency by processing AI tasks at the edge while delivering faster, more responsive AI experiences for the end user. As AI continues to evolve and permeate every aspect of technology, the role of CPUs in processing AI inference workloads will only grow, ensuring that AI can be deployed widely and sustainably across industries.https://www.embedded.com/top-5-reasons-why-cpu-is-the-best-processor-for-ai-inference/
Link above from ARM twitter X
29 Oct 2024 / 7:49 am
Top 5 Reasons why CPU is the Best Processor for AI Inference
View attachment 72794
Ronan Naughton
4 min read
0
Advanced artificial intelligence (AI), like generative AI, is enhancing all our smart devices. However, a common misconception is that these AI workloads can only be processed in the cloud and data center. In fact, the majority of AI inference workloads, which are cheaper and faster to run than training, can be processed at the edge – on the actual devices.
Partner Content
Partner Content
20 Sep 2024
Real-Time Processing in High-Voltage Testing: Insights from HighVolt and Red Pitaya
By: Red Pitaya
Partner Content
3 Sep 2024
GigaDevice Semiconductor expands its Arm MCU product roadmap through Arm Total Access
By: GigaDevice Semiconductor Inc.
Partner Content
12 Jun 2024
IoT Innovations: Qualcomm debuts new technologies at Embedded World 2024
By: Qualcomm
The availability and growing AI capabilities of the CPU across today’s devices are helping to push more AI inference processing to the edge. While heterogeneous computing approaches provide the industry with the flexibility to use different computing components – including the CPU, GPU, and NPU – for different AI use cases and demands, AI inference in edge computing is where the CPU shines.
With this in mind, here are the top five reasons why the CPU is the best target for AI inference workloads.
ADVERTISEMENT
SCROLL DOWN TO CONTINUE READING
The benefits of AI inference on the CPU
Efficiency at the edge
AI processing at the edge is important to the tech industry because the more AI processing at the edge, the more power is saved by not having to send data traveling to and from the cloud. This leads to significant energy and cost savings. The user also benefits from quicker, more responsive AI inference experiences, as well as greater privacy since data is processed locally. These are particularly important for power-constrained devices and edge applications, such as drones, smart wearables, and smart home devices, where power efficiency, latency, and security are paramount. In this context, the CPU plays a crucial role because it’s able to handle these AI inference tasks in the most efficient way possible.
Versatility for various AI inference tasks
The CPU’s versatility allows it to handle a wide range of AI inference tasks, especially for applications and devices requiring quick responses and reliable performance. For example, real-time data processing tasks, like predictive maintenance, environmental monitoring, or autonomous navigation, are handled more efficiently and quickly on the CPU. In industrial IoT applications, this ensures that systems can respond to their environment, or any changes in its environment, in milliseconds. This is crucial for safety and functionality.
Great performance for smaller AI Models
CPUs support a wide range of AI frameworks, like Meta’s PyTorch and ExecuTorch and Google AI Edge’s MediaPipe, making it easy to deploy large language models (LLMs) for AI inference. These LLMs are evolving at a rapid rate, with exceptional user experiences being unlocked by smaller compact models with an ever-decreasing number of parameters. The smaller the model, the more efficient and effective it runs on the CPU.
The availability of smaller LLMs, like the new Llama 3.2 1B and 3B releases, is critical to enabling AI inference at scale. Recently, Arm demonstrated that running the Llama 3.2 3B LLM on Arm-powered mobile devices through the Arm CPU-optimized kernels leads to a 5x improvement in prompt processing and a 3x improvement in token generation.
We are already seeing developers write more compact models to run on low-power processors and even microcontrollers, saving time and costs. Plumerai, which provides software solutions for accelerating neural networks on Arm Cortex-A and Cortex-M systems-on-chip (SoCs), runs just over 1MB of AI code on an Arm-based microcontroller that performs facial detection and recognition. Keen to preserve user privacy, all inference is done on the chip, so no facial features or other personal data are sent to the cloud for analysis.
Greater flexibility and programmability for developers
The software community is actively choosing the CPU as the preferred path for targeting their AI workloads due to its flexibility and programmability. The greater flexibility of CPUs means developers can run a broader range of software in a greater variety of data formats without requiring developers to build multiple versions of their code. Meanwhile, every month there are new models with different architectures and quantization schemes emerging. As the CPU is highly programmable, these new models can be deployed on the CPU in a matter of hours.
The architecture foundation for AI innovation
This developer innovation is built on the foundation of the CPU architecture, which continuously adds new features and instructions to process more advanced AI workloads. The ubiquity of the CPU means developers can then access these capabilities to accelerate and innovate AI-based experiences even further. In fact, the ongoing evolution of the CPU architecture has directly corresponded with the evolution of applications that are now faster and more intelligent.
Why CPUs for AI inference are indispensable
CPUs are not just a component of system-on-chip (SoC) designs, they enable AI to be practical, efficient, and accessible across a wide variety of edge applications and devices. Offering a unique blend of efficiency, versatility, and accessibility, CPUs are indispensable for AI inference. They help reduce energy consumption and latency by processing AI tasks at the edge while delivering faster, more responsive AI experiences for the end user. As AI continues to evolve and permeate every aspect of technology, the role of CPUs in processing AI inference workloads will only grow, ensuring that AI can be deployed widely and sustainably across industries.
Hi Tony,
The "Real Ace" that Brainchip has up it's sleeve is Native SNN's which has to this point, never been brought to the surface.
The performance level increases dramatically from my understanding in all aspects, but we have to date, only ever engaged
with clients whom have used the CNN2SNN tool...but maybe I am behind the times, since TENN's raised it's beautiful head ?
We were always talking about battery operated devices (handheld) for the healthcare sector, which is an area I personally
think we will excel in, just as with the space industry, but Native SNN's was always the end game.
But as I say, maybe I'm not up to speed with what Tony and Sean have planned ?....where's Peter when I need to have a chat
Kind regards....Tech
P.S. Thanks for your support all these years Tony
supersonic001
Regular
…..and boom
the detective dot joining continues, with 3 familiar Monika’s in the Partner column , go Frangi…..Getupthere
Regular
Edge data is critical to AI — here's how Dell is helping enterprises unlock its value — VentureBeat
It’s anticipated that by next year, more than 50% of enterprise data will be created and processed outside traditional data centers or clouds.
Don't get me started on Cerence @IloveLamp!!!!!!
PYMNTS: In-Car Voice Makes Itself Heard at CES
Before CES even opened to the industry, in-car voice technology was making itself heard. Learn how, read more.
brainchip.com
Boab
I wish I could paint like Vincent
Report from Wevolver.
Had a brief look through it. Substantial piece on what Syntiant are doing which look very similar to what we are about?
bearing in mind that Syntiant was one of the sponsors of this report???
EdgeAITechnologyReportGenerativeAIattheEdgeEdition
Edit......Looks like it may be too big?
Perhaps visit Wevolver site if you're keen.
Had a brief look through it. Substantial piece on what Syntiant are doing which look very similar to what we are about?
bearing in mind that Syntiant was one of the sponsors of this report???
EdgeAITechnologyReportGenerativeAIattheEdgeEdition
Edit......Looks like it may be too big?
Perhaps visit Wevolver site if you're keen.
Tuliptrader
Regular
IMHO, a good usage for akida LLMs at the edge using RAG could be replacing/enhancing the owners manual in new cars. No need for cloud connectivity. It would definitely be faster and more efficient than flicking through a physical copy or downloading one to your phone.
TT
TT
Sotherbys01
Regular
Been holding since AZK days....sold enough to become a free carry during the spike of Jan 2022.
Like many other LTH I am tiring of the lack of news and the promises (like a hockey stick) that have been served up to investors.
Sean has stated that there will be deals done etc etc before the end of this year.....well Mariah Carey is starting to play on the radio......
Like many other LTH I am tiring of the lack of news and the promises (like a hockey stick) that have been served up to investors.
Sean has stated that there will be deals done etc etc before the end of this year.....well Mariah Carey is starting to play on the radio......
FiveBucks
Regular
It's certainly been
Hear, hear!
Smoothsailing
Regular
6 weeks till Christmas, hurry up Santa
Odin Shen on LinkedIn: #endpoint #taiwan #tinyml #cortexm55 #ethosu55
I'm thrilled by the variety of AI #endpoint applications and look forward to collaborating with the brand new EDGE AI FOUNDATION. The #Taiwan industry has…
FiveBucks
Regular
SharesForBrekky
Regular
Sorry, just some useless but fascinating thing I just noticed, when looking at the 5 year SP chart of BRN and AXE, they're almost identical.
I don't really follow AXE, so not sure of it's history though, but found it quite interesting nonetheless.
Same algo messing with the sp no doubt
Food4 thought
Regular
Another dumping of a day
When will this turn around Sean
When will this turn around Sean
Tothemoon24
Top 20
( Sean has mentioned streaming & video conference technology)
Attendees will also be able to see Atlas NV-CMM (Non-Volatile CXL Memory Module) showing low-latency persistent memory performance.
Unigen will present a range of high-performance computing solutions at booth 764, including DDR5 Memory Modules, NVDIMMs, and high-capacity SSDs, highlighting their capabilities in handling multiple live video streams and low-latency memory performance.
The company specializes in OEM products and electronics manufacturing services, serving various industries such as automotive, medical, and IoT.
Unigen recently expanded AI production with a new Malaysian manufacturing hub.
Similar threads
