Doubling down on flash attention (my interpretation of this) is quite risky, as there are more efficient attention mechanisms seeping into bigger and bigger models.
Deepseek’s MLA is a start. Jamba is already doing hybrid GQA/Mamba attention, and a Qwen3 update is rumored to be using something exotic as well. And Google’s been doing it with Gemini for some time, though we can only guess what since it’s closed source.
In English, this seems like they’re selling the idea of the software architecture not changing much, when that doesn’t seem to be the case.
Any favorites? What do you think about state space models?
Jamba (hybrid transformers/space state) is a killer model folks are sleeping on. It’s actually coherent at long context, fast, has good world knowledge, even/grounded, and is good at RAG. Its like a straight up better Cohere model IMO, and a no brainer to try for many long context calls.
TBH I didn’t try Falcon H1 much when it seemed to break at long context for me. I think most folks (at least publicly) are sleeping on hybrid SSMs because support in llama.cpp is janky at best, hence they’re not getting any word-of-mouth. For instance, context caching does not work. And Jamba’s janky commercial licensing (unless you pay them) does not help.
…Not sure about others, toy models aside. There really aren’t too many to try.
…TBH, Deepseek is the only non-bog-standard transformers grouped-query-attention model folks have mostly played with. The big trainers seem to be risk-averse architecture wise (hence no big bitnet model attempts yet), which is what Nvidia is betting on I guess.
ty, appreciate this
It’ll all come crashing down soon enough.
