Megalodon also uses “chunk-wise attention,” which divides the input sequence into fixed-size blocks to reduce the complexity of the model from quadratic to linear. Read Full Article
Originally appeared on: TheSpuzz
/cdn.vox-cdn.com/uploads/chorus_asset/file/25403518/ursa_minicom.jpg)
/cdn.vox-cdn.com/uploads/chorus_asset/file/25362056/STK_414_AI_CHATBOT_R2_CVirginia_A.jpg)
/cdn.vox-cdn.com/uploads/chorus_asset/file/25429267/hime.jpg)
Megalodon also uses “chunk-wise attention,” which divides the input sequence into fixed-size blocks to reduce the complexity of the model from quadratic to linear. Read Full Article
