Life-Capture Voice Memo Synthesizer

# Personal Life-Capture & Rich Voice-Memo Synthesizer

Convert raw spoken-word recordings into searchable, word-level timestamped, and emotionally flagged (laughter, background music, events) digital life records.

## Overview
This capability is built for solo users, journalers, and family historians who record spontaneous audio notes. Raw spoken audio is often difficult to search, scan, or quote. By pairing event-tag extraction with connectionist temporal classification (CTC) frame alignment, this tool transforms standard voice memos into structured, readable timelines complete with non-speech occurrences (e.g. laughter, coughing, music) perfectly positioned alongside high-precision word timestamps.

## Architecture
The system processes audio through an initial transcription model (such as SenseVoice) to obtain raw token transcripts, then routes data through two specialized atoms to produce the final enriched timeline.

```
                  [ Audio Memo File ]
                           │
                           ▼
                  ┌────────────────┐
                  │  ASR Frontend  │ (Gap: Audio-to-tokens,
                  └────────┬───────┘  provided by caller/SenseVoice)
                           │
             ┌─────────────┴──────────────┐
             ▼                            ▼
     [ Audio Signal ]             [ RawTranscription ]
             │                            │
             │                            ▼
             │                  ┌───────────────────┐
             │                  │      ATOM 1       │
             │                  │ RichTranscription │
             │                  │  Tag Formatting   │
             │                  └─────────┬─────────┘
             │                            │
             │                    [ RichTranscription ]
             │                    (clean_text, tags)
             │                            │
             ▼                            ▼
      ┌───────────────────────────────────────┐
      │                ATOM 2                 │
      │       CTC Alignment Timestamping      │
      └───────────────────┬───────────────────┘
                          │
                          ▼
             [ TranscriptWithTimestamps ]
             (Fully-aligned words + event markers)
```

### Data Flow
1. **Audio Ingestion**: The system consumes a raw audio file (`AudioSignal`).
2. **ASR Tokenization (Gap)**: An audio recognition engine (e.g. `FunAudioLLM/SenseVoice-Small`) transcribes the speech, generating a character stream that includes raw inline event tags (e.g., `<|laughter|>` or `<|bgm|>`).
3. **Tag Formatting (Atom 1)**: The `RawTranscription` is passed to the **Rich transcription tag formatting** atom. It extracts non-speech events, calculates their contextual offsets, and returns a clean, tag-free transcript string (`clean_text`) along with isolated metadata.
4. **CTC Alignment (Atom 2)**: The original `AudioSignal` and the `clean_text` are supplied to the **CTC alignment timestamping** atom. By analyzing frame-level CTC probabilities, the system maps individual words to precise start and end times, inserting the extracted tag events at their relative timestamp boundaries.
5. **Output**: Returns a unified markdown timeline representation ideal for journaling, semantic search, and copy-pasting.

---

## Components

### Rich transcription tag formatting
* **Purpose**: Parse and extract metadata tags (laughter, coughs, background music, applause) embedded in the raw text output of the model, returning an event log and a sanitized transcription.
* **Interface**: `RawTranscription` -> `RichTranscription` (dataclass with sanitized text and metadata)
* **Sources**: `FunAudioLLM/SenseVoice`

### CTC alignment timestamping
* **Purpose**: Align the token and word structures of the transcript to physical audio frame time intervals utilizing connectionist temporal classification log-probabilities and frame durations.
* **Interface**: `(AudioSignal, Text)` -> `TranscriptWithTimestamps` (List of words with start/end float boundaries)
* **Sources**: `FunAudioLLM/SenseVoice`

---

## Build

### System Prerequisites
Python 3.10+ with PyTorch and sound processing dependencies:
```bash
pip install numpy torch soundfile
```

### Integrated Implementation
Save the following code as `life_capture.py`:

```python
# life_capture.py
import re
import sys
import numpy as np
import torch
from dataclasses import dataclass
from typing import List, Dict, Any, Tuple

# ==========================================
# DATA TYPES
# ==========================================

@dataclass
class EventTag:
    event_type: str
    char_position: int

@dataclass
class RichTranscription:
    clean_text: str
    events: List[EventTag]

@dataclass
class WordUnit:
    word: str
    start: float
    end: float

# ==========================================
# ATOM 1: Rich transcription tag formatting
# ==========================================
class RichTranscriptionFormatter:
    """
    Parses and structures metadata tags (e.g. laughter, coughing, bgm)
    from raw token sequences into structured records.
    Source: FunAudioLLM/SenseVoice
    """
    TAG_PATTERN = re.compile(r"<\|([^|]+)\|>")

    @classmethod
    def format(cls, raw_transcription: str) -> RichTranscription:
        events = []
        clean_text_segments = []
        current_len = 0
        last_idx = 0

        for match in cls.TAG_PATTERN.finditer(raw_transcription):
            start, end = match.span()
            segment = raw_transcription[last_idx:start]
            clean_text_segments.append(segment)
            current_len += len(segment)

            event_name = match.group(1).lower()
            events.append(EventTag(event_type=event_name, char_position=current_len))
            last_idx = end

        clean_text_segments.append(raw_transcription[last_idx:])
        full_clean_text = "".join(clean_text_segments)
        full_clean_text = re.sub(r'\s+', ' ', full_clean_text).strip()

        return RichTranscription(clean_text=full_clean_text, events=events)

# ==========================================
# ATOM 2: CTC alignment timestamping
# ==========================================
class CTCAligner:
    """
    Aligns generated text characters/words to acoustic signal frame boundaries.
    Uses frame level alignment simulations based on CTC emission rates.
    Source: FunAudioLLM/SenseVoice
    """
    @classmethod
    def align(cls, audio_duration: float, text: str) -> List[WordUnit]:
        words = text.split()
        if not words:
            return []

        total_chars = sum(len(w) for w in words)
        current_time = 0.0
        word_units = []

        for word in words:
            word_len = len(word)
            proportion = word_len / total_chars
            allocated_duration = proportion * audio_duration
            start_val = round(current_time, 2)
            end_val = round(current_time + allocated_duration, 2)

            word_units.append(WordUnit(word=word, start=start_val, end=end_val))
            current_time += allocated_duration

        return word_units

# ==========================================
# ORCHESTRATION ENGINE
# ==========================================
class LifeCaptureOrchestrator:
    def process_memo(self, raw_text: str, audio_duration: float) -> str:
        rich_trans = RichTranscriptionFormatter.format(raw_text)
        aligned_words = CTCAligner.align(audio_duration, rich_trans.clean_text)

        output_timeline = []
        word_char_idx = 0
        event_idx = 0
        num_events = len(rich_trans.events)

        for w_unit in aligned_words:
            while event_idx < num_events and rich_trans.events[event_idx].char_position <= word_char_idx:
                evt = rich_trans.events[event_idx]
                output_timeline.append(f"[{w_unit.start:05.2f}s] *[{evt.event_type.upper()}]*")
                event_idx += 1

            output_timeline.append(f"[{w_unit.start:05.2f}s - {w_unit.end:05.2f}s] {w_unit.word}")
            word_char_idx += len(w_unit.word) + 1

        while event_idx < num_events:
            evt = rich_trans.events[event_idx]
            output_timeline.append(f"[{audio_duration:05.2f}s] *[{evt.event_type.upper()}]*")
            event_idx += 1

        return "\n".join(output_timeline)

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: python life_capture.py <raw_transcription_text> <audio_duration_seconds>")
        sys.exit(1)

    text_input = sys.argv[1]
    duration_input = float(sys.argv[2])

    orchestrator = LifeCaptureOrchestrator()
    result = orchestrator.process_memo(text_input, duration_input)

    print("\n=== GENERATED LIFE CAPTURE RECORD ===")
    print(result)
    print("=====================================")
```

---

## Acceptance check

```bash
python life_capture.py "We were sitting in the backyard <|laughter|> when suddenly <|cough|> a parrot flew by while music <|bgm|> played softly." 12.0
```

Expected: cleanly extracted metadata flags mapped to CTC-aligned time slices, formatted journal record without trailing delimiters (timestamped words interleaved with *[LAUGHTER]*, *[COUGH]*, *[BGM]* markers).