Can't recall whether it was Tony or Jonathan, but they mentioned the use of Look-up Tables (LUT) for the activation function as a great time/processing saver in place of doing the ReLU calculation.
Well ARM U85 seems to do something similar, for 8 & 16 bits:
Lookup table Arm Ethos-U85 NPU Technical Overview
The Activation Output (AO) unit supports lookup through a programmable table, where the activation acts as an index in the LookUp Table (LUT). This LUT can be used as a generic activation function or to reduce a series of layers of pointwise operations into a single fused lookup.
The AO unit supports the following modes for LUTs:
Direct 8-bit to 8-bit lookup from 8 tables of 256 bytes each
Direct signed 8-bit to signed 16-bit lookup from 4 tables of 512 bytes each
Direct signed 8-bit to signed 32-bit lookup from 2 tables of 1024 bytes each
Piecewise linear approximation for signed 16-bit to signed 16-bit
The table entries for piecewise linear approximation consists of a 16-bit base and a 16-bit slope. The lookup is made from the high nine bits of the 16-bit activation so a single 2048 bytes table of 512 32-bit entries is available. The interpolation is made from the low seven bits of the 16-bit activation.
Although the direct modes enable bit-exact support for any activation or pointwise function, the linear approximation available for 16-bit activations likely diverges from a bit-exact representation.
Akida 2 still does 4-bit, so not identical, but, by golly, it's hard to tell them apart at a distance.
