Collected sources and patterns will appear here. Add from search or the patterns library.
Queue<InferenceRequest> -> Batch<TokenTask>
Schedule the execution of prompt prefills and token generations at the individual iteration boundary rather than processing whole sequences.
Problem it solves
Static batching forces requests to wait for the longest sequence in the batch to complete, wasting GPU compute cycles.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.