AISysRev -- LLM-based Tool for Title-abstract Screening

Defensibility (score=2/10) - Quantitative signals are extremely weak: ~0 stars and ~5 forks with ~0 observed velocity and age of 1 day. This indicates the project is either newly created, not yet discoverable, or not gaining adoption. - Functionality described is largely a standard “LLM-for-title-abstract screening” workflow: ingest titles/abstracts (CSV), accept inclusion/exclusion criteria, run an LLM to label or rank papers, and assist human reviewers. That pattern is already well-explored in systematic review informatics and is straightforward to replicate. - The only potentially distinguishing factor mentioned is “containerized web application.” Containerization helps usability/deployment but is not a moat; it’s easily copied. - There is no evidence (from the provided info) of a novel model, dataset, evaluation benchmark, active user community, or proprietary dataset/labeling pipeline. Without these, there is minimal switching cost. Moat assessment (or lack thereof) - No measurable ecosystem effects: stars near zero and no velocity suggest no user gravity. - No technical defensibility signals: no mention of training, fine-tuning, retrieval-augmented generation with curated corpora, calibration, uncertainty quantification, or domain-specific heuristics that would be hard to replicate. - No reference to rigorous evaluation/benchmarks (e.g., screening performance metrics like recall/precision at inclusion/exclusion thresholds) that could make it “the” trusted tool. Frontier-lab obsolescence risk (high) - Frontier labs are very likely to provide adjacent functionality inside their existing AI platforms (e.g., “upload CSV, define criteria, auto-screen studies” as a workflow template or tool). - Because the problem is generic (criteria-based document screening) and the implementation is containerized around LLM calls, the marginal effort for a platform to replicate the capability is low. - As LLM platforms add more structured outputs and evaluation-aware prompting/function-calling, this project is likely to be absorbed as a feature or a template. Three-axis threat profile 1) Platform domination risk: HIGH - Who could absorb/replace it: OpenAI/Anthropic/Google (and likely major LLM vendors via agents/workflow tooling) could implement a “systematic review screening” agent using their function calling / structured extraction. - Why: the core technical approach is not specialized infrastructure; it’s a thin application layer around LLM inference. Containerized delivery is not a barrier. - Timeline logic: once templates/agents mature, the project can be functionally obsolete quickly. 2) Market consolidation risk: HIGH - Likely consolidation: systematic review automation tends to consolidate around a few general-purpose AI workflow providers plus a handful of domain-specific incumbents. - Reason: generic screening workflows are easy to bundle into larger “research assistant” platforms. Specialist tools without unique datasets or evaluation-grade reliability typically get displaced or become integrations. 3) Displacement horizon: 6 months - Given age = 1 day, stars = 0, and no velocity, there’s no evidence of rapid entrenchment. - LLM tooling iteration cycles are fast; within ~6 months, platform-level agents and structured-output workflows can match this capability with lower user friction (no container setup, better UI, better model access). Key opportunities - If the authors publish rigorous evaluation (benchmark results, inter-annotator agreement proxies, active learning loops, audit trails) and demonstrate consistent improvements over prompt-only baselines, they could increase defensibility from “prototype” toward “beta.” - Adding calibrated outputs (confidence/uncertainty), human-in-the-loop review controls, and traceable rationales (with evidence spans from abstracts) could create partial switching cost if users trust the process. Key risks - Highly substitutable: other repos and commercial tooling can replicate the same interface with similar prompts/LLM calls. - Platform absorption: frontier labs can provide the same workflow as a template/agent. - Lack of adoption signals: with essentially no stars/velocity, there is no momentum to defend against commoditization. Adjacent competitors / alternatives (conceptual) - Generic systematic review automation tools (screening/ranking) that already support inclusion/exclusion workflows. - “LLM research assistant” products that can ingest batches and output structured labels. - Existing open-source LLM tooling frameworks (agents + structured extraction) that make it trivial to implement criteria-based screening. Bottom line: AISysRev appears to be an early-stage, LLM-based screening application. Without demonstrated traction, novel technical contribution, or hard-to-replicate assets, its defensibility is very low and its frontier obsolescence risk is high.