Metadata-Version: 2.4
Name: kdenlive-transcript-sync
Version: 0.1.dev1+gfced68a17
Summary: Align multi-cam video clips on a Kdenlive timeline using multi-method audio sync
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: faster-whisper
Provides-Extra: gpu
Requires-Dist: onnxruntime-gpu<1.23,>=1.18; extra == "gpu"
Provides-Extra: cpu
Requires-Dist: onnxruntime>=1.18; extra == "cpu"

# kdenlive-transcript-sync (`kts`)

Tooling to rebuild a **Kdenlive project** from a raw multi-cam / external-recorder
shoot: figure out which external audio recording belongs to which camera clip,
how far apart they are in time, and which part of the script each take covers —
then arrange all of that on a timeline with the audio aligned to the video.

Typical setup it is built for:

- one or more cameras producing video files with (poor) on-camera sound,
- a separate audio recorder producing `.wav` files (e.g. a mic track `…M.wav`
  and an internal/scratch track `…I.wav` per recording group),
- a text script of what is supposed to be said, split into scenes.

## Status

The project currently ships **two CLIs**, which are the two steps of the
pipeline:

| Command | Role |
|---|---|
| `kts-match`   | Step 1 — match each video clip to an audio group and compute the sync offset |
| `kts-arrange` | Step 2 — match each take against the script and group takes by scene |

The final step — writing the `.kdenlive` XML timeline from `arrange.json` — is
the goal of the project and is not wired up in this branch yet. Everything
before it (matching, offsets, scene grouping) works and is cached as JSON.

## Install

```bash
pip install .          # or: pip install -e .
```

Arch users can build the package with the provided `PKGBUILD`.

External dependencies:

- `ffmpeg` — audio decoding for the peaks / envelope / whisper methods
- `fpcalc` (`chromaprint`) — for the chroma fingerprint method
- `faster-whisper`, `numpy`

## Configuration

Config is TOML, read from (later wins):

1. `/etc/kts.conf`
2. `$XDG_CONFIG_HOME/kts.conf` (fallback `~/.config/kts.conf`)

Command-line flags always override the config file. See `conf_example.toml`:

```toml
video-dir        = "video/"
audio-dir        = "audio/"
script           = "script.txt"
whisper-language = "fr"
```

Recognised keys: `video-dir`, `audio-dir`, `script`, `cache-dir`,
`whisper-language`, `whisper-model`, `whisper-cache-dir`.

All intermediate results are written into `cache-dir` (default `.kts/`):
`match.json` and `arrange.json`.

## Step 1 — `kts-match`

```bash
kts-match --video-dir video/ --audio-dir audio/
# with a config file, simply:
kts-match
```

It pairs every video with every audio group and runs up to four independent
sync methods:

| Method | What it does | Speed |
|---|---|---|
| `peaks`    | transient peak-constellation Hough matching | fast (~1 s) |
| `envelope` | RMS-envelope cross-correlation | fast (~5 s) |
| `chroma`   | Chromaprint fingerprint sliding-window match | medium (~30 s) |
| `whisper`  | word-level transcript alignment | slow (minutes) |

A fusion layer votes across methods to pick the winning audio group and a
weighted-mean consensus offset with a confidence level.

Output `.kts/match.json`:

```json
{
  "MVI_8113.MP4": {
    "group": "4CH001",
    "offset": 2.347,
    "confidence": "HIGH",
    "method": "envelope,whisper",
    "votes": 4,
    "details": { "peaks": {}, "envelope": {} }
  }
}
```

Useful flags: `--skip-peaks/--skip-envelope/--skip-chroma/--skip-whisper`,
`--max-s`, `--delta-max`, `--whisper-model`, `--whisper-language`, `--force`.

Audio files are grouped by name: `NAME` + role letter (`M` = mic, `I` =
internal) + `.wav`, e.g. `4CH001M.wav` / `4CH001I.wav` → group `4CH001`.
A group needs at least an `M` track.

## Step 2 — `kts-arrange`

```bash
kts-arrange --script script.txt --video-dir video/
```

Reads `.kts/match.json`, transcribes each clip with Whisper (cached), and
fuzzy-matches the transcript against the script to determine which scene(s)
each take covers. Takes covering the same scene are grouped so you can pick
the best one.

### Script format

```
## Scene 1
MARIE: Bonjour, tu vas bien ?
PIERRE: Très bien, merci.
[pause — ils se regardent]
## Scene 2
...
```

- `##` lines are scene markers.
- `[...]` lines are stage directions (kept, but never matched).
- `NAME: text` gives the speaker.

Output `.kts/arrange.json`:

```json
{
  "scenes": [
    {
      "scene": 1,
      "script_lines": [{"index": 0, "speaker": "MARIE", "text": "Bonjour…"}],
      "takes": [
        {
          "video": "MVI_8114.MP4",
          "audio_group": "4CH002",
          "offset_s": 1.23,
          "in_s": 4.5,
          "out_s": 18.2,
          "n_matched": 5,
          "n_script": 6,
          "anchors": []
        }
      ]
    }
  ]
}
```

Useful flags: `--min-score` (fuzzy threshold, default `0.45`), `--min-matched`,
`--all-takes` (also report takes with no match), `--force-retranscribe`.

## Pipeline overview

```
video/*.mp4   audio/*M.wav,*I.wav          script.txt
      │              │                          │
      └──── kts-match (peaks/envelope/          │
            │          chroma/whisper + fusion) │
            ▼                                   │
      .kts/match.json  (group + offset/clip)    │
            │                                   │
            └──── kts-arrange (whisper + fuzzy match)
                        │
                        ▼
              .kts/arrange.json  (scenes → takes)
                        │
                        ▼
              [planned] project.kdenlive
```

## Project layout

```
src/kdenlive_transcript_sync/
    config.py        TOML config loader (/etc/kts.conf, ~/.config/kts.conf)
    match_cli.py     kts-match entry point
    arrange_cli.py   kts-arrange entry point
    fuse.py          cross-method voting / consensus offset
    align.py         script parsing + fuzzy transcript matching
    discover.py      file discovery helpers
    models.py        dataclasses (ScriptLine, TranscriptSegment, Anchor, …)
    methods/
        peaks.py     transient peak constellation
        envelope.py  RMS envelope cross-correlation
        chroma.py    Chromaprint fingerprints (fpcalc)
        whisper.py   faster-whisper transcription + alignment
```

## Tuning tips

- Bad audio→video pairings: raise `--delta-max` only if offsets really can be
  large, and prefer `--max-s 0` (full file) on long takes.
- Too many false script matches: raise `--min-score`; too many skipped lines:
  lower it.
- Set `whisper-language` in the config — auto-detect is noticeably worse on
  short or noisy clips.
- Use a bigger `--whisper-model` (`small`, `medium`) for noisy audio; the
  transcripts are cached in `--whisper-cache-dir`, so it only costs once.
