Attested Discovery Ledger (ADL)

# SPEC.md — Attested Discovery Ledger (ADL)

> Generated by the Gerolamo compose engine (mode=compose) from meta molecule
> `1505999d-1478-4515-8391-c23221ce1e1a`. Architecture, stack, and structure are
> from the engine; the Build Plan, Milestones, and Risks sections were authored
> to complete the spec (the engine's output was capped before the build plan).

## Project Overview

The Attested Discovery Ledger (ADL) is a cryptographic provenance infrastructure layer for autonomous (self-driving) laboratory systems that binds every step of a closed-loop discovery workflow — hypothesis generation, experiment execution, instrument measurement, and Bayesian model update — into a tamper-evident, independently verifiable lineage graph. It exists because autonomous labs are beginning to produce novel materials, compounds, and biological constructs without continuous human oversight, creating a critical gap: downstream users (regulators, licensing partners, defense contractors, peer reviewers) have no mechanism to verify that a discovered artifact was actually produced by the process claimed. ADL solves this by cryptographically anchoring each workflow node to its producing hardware (via TEE remote attestation), chaining nodes into a content-addressed directed acyclic graph (DAG), and signing the graph edges with Ed25519 keys that are rooted in instrument-level identity — making retroactive falsification computationally infeasible without physical access to the attesting hardware.

---

## Architecture

### System Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                     ADL System Boundary                         │
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌───────────────────┐  │
│  │  Instrument  │    │  Orchestrat- │    │  Bayesian Active  │  │
│  │  Telemetry   │───▶│  ion Engine  │───▶│  Learning Agent   │  │
│  │  Signing     │    │  (Lab Loop)  │    │  (Hypothesis Gen) │  │
│  └──────┬───────┘    └──────┬───────┘    └────────┬──────────┘  │
│         │                  │                      │             │
│         ▼                  ▼                      ▼             │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              Attestation & Signing Layer                │    │
│  │  ┌─────────────┐  ┌──────────────┐  ┌───────────────┐  │    │
│  │  │ TEE Enclave │  │ Ed25519 Key  │  │ Payload Hash  │  │    │
│  │  │ (SGX/TrustZ)│  │ Management  │  │ (BLAKE3/SHA3) │  │    │
│  │  └─────────────┘  └──────────────┘  └───────────────┘  │    │
│  └──────────────────────────┬──────────────────────────────┘    │
│                             │                                   │
│                             ▼                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │            Content-Addressed Provenance DAG             │    │
│  │  ┌──────────┐   ┌──────────┐   ┌──────────┐            │    │
│  │  │ Node:    │──▶│ Node:    │──▶│ Node:    │            │    │
│  │  │ Hypothesis│  │ Synthesis│  │ Analysis │            │    │
│  │  │ [CID]    │  │ Run [CID]│  │ [CID]    │            │    │
│  │  └──────────┘   └──────────┘   └──────────┘            │    │
│  └──────────────────────────┬──────────────────────────────┘    │
│                             │                                   │
│         ┌───────────────────┼────────────────────┐              │
│         ▼                   ▼                    ▼              │
│  ┌─────────────┐   ┌───────────────┐   ┌──────────────────┐     │
│  │  Storage    │   │  Verification │   │  REST / gRPC     │     │
│  │  Backend    │   │  Service      │   │  API Gateway     │     │
│  │  (IPFS/S3)  │   │               │   │                  │     │
│  └─────────────┘   └───────────────┘   └──────────────────┘     │
└─────────────────────────────────────────────────────────────────┘
```

### Artifact-to-Component Mapping

| Artifact Element | ADL Component | Role |
|---|---|---|
| Ed25519 signatures | `adl-signing` library | Signs all provenance node payloads; key identity bound to instrument |
| TEE remote attestation | `adl-attestation` service | Generates hardware-rooted trust anchors; verifies enclaves are unmodified |
| Content-addressed provenance graph | `adl-dag` core | DAG node creation, CID computation, edge linking, graph traversal |
| Laboratory orchestration software | `adl-orchestrator` adapter | Intercepts experiment lifecycle events and emits signed provenance nodes |
| Bayesian active learning | `adl-agent` module | Generates hypothesis nodes; its acquisition function outputs become signed graph entries |
| Instrument telemetry signing | `adl-instrument` SDK | Per-instrument signing shim; captures raw telemetry and wraps in attested payloads |

### Integration Points & Data Flow

```
Instrument HW
    │
    │ raw telemetry (serial/USB/TCP)
    ▼
adl-instrument SDK
    │ signs payload with instrument Ed25519 key
    │ attaches TEE attestation quote
    ▼
adl-orchestrator adapter   ◀──── lab orchestration events (webhooks/gRPC)
    │ creates ProvenanceNode (type=TELEMETRY)
    │ computes BLAKE3(canonical_payload) → CID
    │ links to parent node CID
    ▼
adl-dag core
    │ stores node to content-addressed backend (IPFS / S3 + Merkle index)
    │ emits node_committed event
    ▼
adl-agent (Bayesian AL)
    │ reads prior node CIDs as observation inputs
    │ computes acquisition function, selects next experiment
    │ emits signed hypothesis node (type=HYPOTHESIS)
    ▼
adl-dag core  (cycle back to orchestrator)
    │
    ▼
adl-verification service
    │ exposes REST/gRPC: verify_lineage(artifact_cid) → VerificationReport
    │ walks DAG, re-checks all signatures, verifies TEE quotes against Intel/AMD CA
    ▼
API Gateway  ──── consumed by: regulators, auditors, licensing partners
```

### API Boundaries

- **adl-instrument SDK → adl-orchestrator**: gRPC streaming (`InstrumentTelemetryService`)
- **adl-orchestrator → adl-dag**: internal Python library call + event bus (Redis Streams)
- **adl-dag → storage**: IPFS HTTP API or S3-compatible (pluggable backend interface)
- **adl-verification → external**: REST JSON API (`GET /v1/lineage/{cid}/verify`)
- **adl-agent → adl-dag**: Python library call (`dag.commit_node(...)`)

---

## Tech Stack

### Core Languages
- **Python 3.11** — orchestration adapter, Bayesian agent, DAG core, verification service
- **Rust 1.78** — instrument signing SDK (performance-critical, memory-safe cryptography)
- **TypeScript 5.4** — API gateway, web audit dashboard

### Cryptography
- **PyNaCl 1.5.0** — Ed25519 signing/verification (Python layer)
- **ed25519-dalek 2.1.1** (Rust crate) — instrument-side signing
- **blake3 1.5.1** (Rust crate) + **blake3 0.3.8** (Python) — content addressing / CID computation
- **Intel SGX SDK 2.22** or **ARM TrustZone via OP-TEE 3.22** — TEE enclave runtime
- **sgx-ra-tls** — TEE remote attestation TLS extension (RA-TLS)

### Storage & Graph
- **py-ipfs-http-client 0.8.0a2** — IPFS node interaction (primary content-addressed store)
- **boto3 1.34.x** — S3-compatible fallback storage backend
- **networkx 3.3** — in-memory DAG traversal and validation
- **pyarrow 16.0.0** — columnar serialization of telemetry payloads

### Bayesian Active Learning
- **botorch 0.10.0** — Gaussian Process surrogate models, acquisition functions
- **gpytorch 1.12** — GP backend
- **torch 2.3.0** — tensor operations
- **scipy 1.13.0** — optimization utilities

### API & Services
- **FastAPI 0.111.0** — REST API gateway and verification service
- **grpcio 1.64.0** + **grpcio-tools 1.64.0** — gRPC for instrument telemetry streaming
- **protobuf 5.27.0** — schema definitions
- **redis 5.0.4** (Python client) — event bus (Redis Streams)
- **pydantic 2.7.0** — data validation and schema enforcement

### Infrastructure
- **Docker 26.x** + **Docker Compose v2** — local development
- **pytest 8.2.0** — test framework
- **hypothesis 6.103.0** — property-based testing for cryptographic invariants
- **mypy 1.10.0** — static typing

---

## Project Structure

```
attested-discovery-ledger/
├── SPEC.md
├── CLAUDE.md
├── README.md
├── pyproject.toml
├── docker-compose.yml               # Local dev: IPFS, Redis, mock SGX
├── .env.example
│
├── proto/
│   ├── telemetry.proto              # InstrumentTelemetryService, TelemetryPayload
│   ├── provenance.proto             # ProvenanceNode, ProvenanceEdge, DAGRoot
│   └── verification.proto           # VerificationRequest, VerificationReport
│
├── adl-core/                        # Python: shared types, DAG engine
│   ├── pyproject.toml
│   ├── adl/
│   │   ├── __init__.py
│   │   ├── types.py                 # ProvenanceNode, NodeType enum, CID type
│   │   ├── dag.py                   # DAG construction, CID computation, storage I/O
│   │   ├── signing.py               # Ed25519 key management, sign/verify wrappers
│   │   ├── attestation.py           # TEE quote parsing, verification against CA
│   │   ├── storage/
│   │   │   ├── base.py              # Abstract StorageBackend interface
│   │   │   ├── ipfs.py
│   │   │   └── s3.py
│   │   └── serialization.py         # Canonical JSON / Arrow serialization
│   └── tests/
│
├── adl-instrument/                  # Rust: instrument signing SDK
│   ├── Cargo.toml
│   ├── src/
│   │   ├── lib.rs                   # Public FFI + Python bindings (PyO3)
│   │   ├── signer.rs
│   │   ├── telemetry.rs
│   │   ├── tee/{mod.rs, sgx.rs, mock.rs}
│   │   └── grpc/client.rs
│   └── tests/
│
├── adl-orchestrator/                # Python: lab lifecycle adapter
│   ├── orchestrator/
│   │   ├── adapter.py
│   │   ├── event_bus.py             # Redis Streams publisher/subscriber
│   │   ├── grpc_server.py
│   │   ├── hooks/{base.py, generic_webhook.py, mock_lab.py}
│   │   └── node_factory.py
│   └── tests/
│
├── adl-agent/                       # Python: Bayesian active learning loop
│   ├── agent/
│   │   ├── loop.py                  # observe → model → acquire → commit
│   │   ├── surrogate.py             # GP surrogate (BoTorch)
│   │   ├── acquisition.py           # EI, UCB, qNEHVI
│   │   ├── hypothesis.py
│   │   └── observation.py
│   └── tests/
│
├── adl-verification/                # Python: lineage verification service
│   ├── verification/
│   │   ├── service.py               # FastAPI app
│   │   ├── verifier.py              # walk DAG, re-verify signatures + TEE quotes
│   │   ├── report.py
│   │   └── router.py
│   └── tests/
│
└── adl-gateway/                     # TypeScript: public API gateway + audit dashboard
    ├── package.json
    ├── src/
    │   ├── index.ts
    │   ├── routes/{lineage.ts, artifacts.ts}
    │   ├── dashboard/               # React audit dashboard (Vite + react-flow)
    │   │   ├── App.tsx
    │   │   ├── components/{LineageGraph.tsx, VerificationBadge.tsx}
    │   │   └── hooks/useLineage.ts
    │   └── types/provenance.ts
    └── tests/
```

---

## Build Plan

> Authored to complete the spec (the compose engine was capped before this point).
> Phases are ordered so each one is independently verifiable against a mock before
> any real hardware (TEE / instrument) is required.

### Phase 1 — Core Types, Signing, and Content Addressing
**Objective:** A signed, content-addressed `ProvenanceNode` that round-trips through a storage backend, with all cryptography exercised against a software key (no TEE yet).

**Tasks:**
1. Implement `adl-core/adl/types.py`: `ProvenanceNode` (fields: `node_type`, `payload`, `parent_cids: list[CID]`, `producer_id`, `timestamp`, `signature`, `attestation_quote: Optional`), `NodeType` enum (`HYPOTHESIS`, `SYNTHESIS`, `TELEMETRY`, `ANALYSIS`, `MODEL_UPDATE`), and a `CID` newtype.
2. Implement `adl-core/adl/serialization.py`: deterministic canonical serialization (sorted-key canonical JSON for the signed view; Arrow for bulk telemetry payloads). Canonicalization must be byte-stable across processes.
3. Implement `adl-core/adl/signing.py`: `KeyManager` (generate/load Ed25519 keys via PyNaCl), `sign(node) -> bytes`, `verify(node) -> bool`. Signature covers `BLAKE3(canonical_payload || parent_cids || producer_id || timestamp)`.
4. Implement `adl-core/adl/dag.py`: `compute_cid(node) -> CID` (BLAKE3 of canonical bytes), `commit_node(node)`, `get_node(cid)`, `iter_ancestors(cid)`.
5. Implement `adl-core/adl/storage/{base,ipfs,s3}.py` behind a `StorageBackend` ABC; default to a local filesystem/SQLite Merkle index for dev.

**Verification:**
```bash
pytest adl-core/tests/ -v
# Property tests (hypothesis): re-serialization is byte-identical; any single-bit
# payload mutation invalidates the signature; CID is collision-stable.
```

### Phase 2 — Attestation Layer (mockable)
**Objective:** Bind a node's signature to a hardware-rooted quote, verifiable against a CA, with a mock enclave so the loop runs in CI.

**Tasks:**
1. Implement `adl-core/adl/attestation.py`: `Quote` model, `verify_quote(quote, ca_bundle) -> AttestationResult`. Support SGX DCAP and a `MockAttestor` that signs quotes with a test CA.
2. Extend `signing.py` so a node's producer key is certified by an attestation quote (key-to-measurement binding).
3. Wire `MockAttestor` into the dev `docker-compose.yml`.

**Verification:**
```bash
pytest adl-core/tests/test_attestation.py -v
# A node carrying a valid mock quote verifies; a tampered measurement fails.
```

### Phase 3 — Instrument SDK (Rust) and Orchestrator Adapter
**Objective:** A real per-instrument signing shim streaming signed telemetry into the DAG via the orchestrator.

**Tasks:**
1. Implement `adl-instrument` (Rust): `Signer` (ed25519-dalek), `TelemetryPayload`, BLAKE3 CID, `tee::mock`, gRPC streaming client. Expose PyO3 bindings.
2. Implement `adl-orchestrator/adapter.py`: gRPC server receiving telemetry, `node_factory.py` building typed nodes, `event_bus.py` (Redis Streams) emitting `node_committed`.
3. Provide `hooks/mock_lab.py`: a simulated closed loop that emits hypothesis → synthesis → telemetry → analysis nodes on a timer.

**Verification:**
```bash
docker compose up -d redis ipfs
pytest adl-orchestrator/tests/ -v
# Mock lab produces a connected DAG; every edge verifies end to end.
```

### Phase 4 — Bayesian Agent Closed Loop
**Objective:** The agent reads committed observation nodes, proposes the next experiment, and commits a signed hypothesis node, closing the loop.

**Tasks:**
1. Implement `adl-agent/surrogate.py` (BoTorch GP) and `acquisition.py` (qNEHVI for multi-objective discovery).
2. Implement `adl-agent/loop.py`: subscribe to `node_committed`, build the observation set from ancestor analysis nodes, fit the surrogate, select the next point, emit a signed `HYPOTHESIS` node.

**Verification:**
```bash
pytest adl-agent/tests/ -v
# On a synthetic 2-objective benchmark the loop advances the Pareto front;
# every hypothesis node is signed and parented to the observations it used.
```

### Phase 5 — Verification Service and Gateway
**Objective:** A third party verifies a discovered artifact's full lineage from a single CID.

**Tasks:**
1. Implement `adl-verification/verifier.py`: given an artifact CID, walk all ancestors, re-verify every signature and every attestation quote against the CA bundle, return a `VerificationReport` (pass/fail per node + summary).
2. Implement `adl-verification/router.py`: `GET /v1/lineage/{cid}/verify`, `GET /v1/lineage/{cid}/graph`.
3. Implement `adl-gateway` (TS): proxy routes + a react-flow lineage visualizer with per-node verification badges.

**Verification:**
```bash
pytest adl-verification/tests/ -v
curl localhost:8000/v1/lineage/<cid>/verify   # → {"verified": true, "nodes": N, ...}
# Negative test: swap one stored node's payload → report flags exactly that node.
```

---

## Milestones

- **M1 (Phases 1–2):** Signed, content-addressed, attestation-bound nodes round-trip in CI with mocks. *Demo:* tamper a byte, watch verification fail.
- **M2 (Phase 3):** Real Rust instrument SDK streams into a live DAG via the orchestrator and a mock lab loop.
- **M3 (Phase 4):** Full autonomous loop runs on a synthetic discovery benchmark, every node attested.
- **M4 (Phase 5):** One-CID third-party verification + lineage dashboard. *This is the licensable artifact.*

## Risks & Open Questions

- **Attestation portability.** SGX DCAP, TDX, and SEV-SNP quote formats differ; the `Attestor` abstraction must not leak a single vendor. Treat the CA bundle as configuration.
- **Instrument key custody.** Many lab instruments cannot run a TEE. Fallback: a TEE-backed gateway box that attests the *capture* of an otherwise-untrusted instrument, and the threat model must state this honestly.
- **Throughput.** High-rate telemetry (spectra, images) should be Arrow-batched and signed per batch, not per sample; choose a batch boundary that preserves the provenance semantics you actually need to prove.
- **What is being proven.** ADL proves *process integrity and lineage*, not that the science is correct. Keep that boundary explicit in the verification report so it is not oversold.