Collected sources and patterns will appear here. Add from search or the patterns library.
ModelConfig -> TrainedSparseMoEModel
Initialize Feed-Forward Networks as replicated experts, then train only the gating mechanism and newly added expert weights while keeping the projection layer and base attention layers frozen.
Problem it solves
Jointly training sparse experts and multi-modal projectors from scratch leads to representational instability and slow convergence.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.