Collected sources and patterns will appear here. Add from search or the patterns library.
LargePrefillRequest -> List<TokenChunk>
Divide long prompt prefill sequences into smaller token chunks to interleave them with active decoding tasks.
Problem it solves
Large prefills block active decoding iterations, causing high latency spikes (jitter) for active requests.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.