Hi Doz,ReLU strikes back …..
View attachment 87380
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language...
Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for...openreview.net
Great research,
According to the abstract, GELU and SiLu are more processor intensive.
The abstract compares ReLU with lemons, but we are using cherries.
If anything, this abstract presents a stronger case for LUTs compared with alternatives.