Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index

Infini-gram mini addresses a critical pain point in the LLM era: the inability to efficiently search the massive, multi-terabyte datasets (like RedPajama or Pile) used for training. While the underlying FM-index and BWT algorithms are standard in bioinformatics (e.g., Bowtie/BWA), applying them to the engineering constraints of petabyte-scale text with arbitrary n-gram queries is a significant engineering feat. The project's defensibility is currently low (score 4) because, while the engineering is non-trivial, it lacks a surrounding ecosystem or 'data gravity' in its current form—it is a reference implementation of a research paper. The quantitative signals (0 stars, 5 forks) suggest this is a specialized research artifact rather than a production-grade tool with broad adoption. The primary threat comes from platform providers like Hugging Face or large-scale data curators who are likely to implement similar search capabilities as a service (Platform Domination Risk: High). If a frontier lab or a major data host (like Common Crawl) integrates this or a similar suffix-array-based search, the standalone utility of this repository diminishes. However, for researchers performing data contamination studies or attribution, this remains a valuable, niche algorithmic contribution.