Collected sources and patterns will appear here. Add from search or the patterns library.
Tensor<Tokens> -> Tensor<RoutedTokens>
Eliminate token-dropping in Mixture-of-Experts (MoE) layers by minimizing the expert-parallelism dimension and utilizing intra-node-only routing.
Problem it solves
Traditional expert parallelism drops tokens when expert capacity is exceeded, leading to training instability and model quality degradation.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.