Collected sources and patterns will appear here. Add from search, explore, or the patterns library.
UnsortedRecordBatch -> SortedFileChunks
Sort incoming record batches in memory by primary key, spilling sorted chunks to disk when memory usage exceeds a configured threshold before final compaction.
Problem it solves
Sorting massive datasets with highly cardinality keys in memory during batch writes leads to out-of-memory (OOM) exceptions.
Consumes
Emits
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.