Collected sources and patterns will appear here. Add from search or the patterns library.
Stream<InferenceRequest> -> DynamicTokenBatch
Iteratively insert incoming generation requests into the current execution loop at token boundaries rather than waiting for whole batches to finish.
Problem it solves
Static batching causes newer requests to stall waiting for active long-generation requests to complete.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.