Collected sources and patterns will appear here. Add from search, explore, or the patterns library.
Tensor<HiddenStates> -> Tensor<HiddenStates>
Route token representations to a combination of statically active shared experts and dynamically routed specialized experts.
Problem it solves
Dynamic MoE models waste capacity when multiple specialized experts independently redundant-learn common context or general knowledge.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.