kdenlive-transcript-sync (kts)

Tooling to rebuild a Kdenlive project from a raw multi-cam / external-recorder shoot: figure out which external audio recording belongs to which camera clip, how far apart they are in time, and which part of the script each take covers — then arrange all of that on a timeline with the audio aligned to the video.

Typical setup it is built for:

• one or more cameras producing video files with (poor) on-camera sound,
• a separate audio recorder producing .wav files (e.g. a mic track …M.wav and an internal/scratch track …I.wav per recording group),
• a text script of what is supposed to be said, split into scenes.

Status

The project ships three CLIs, one per pipeline step:

Command	Role
kts-match	Step 1 — match each video clip to an audio group and compute the sync offset
kts-arrange	Step 2 — match each take against the script and group takes by scene
kts-build	Step 3 — write a .kdenlive timeline from arrange.json

Steps 1 and 2 cache their results as JSON in .kts/; kts-build consumes that cache and writes a Generation-5 Kdenlive project (vertical 1080x1920 by default) with video and synced external audio placed on the timeline. Remaining work: proxy setup.

Install

pip install .          # or: pip install -e .

Arch users can build the package with the provided PKGBUILD.

External dependencies:

• ffmpeg — audio decoding for the peaks / envelope / whisper methods
• fpcalc (chromaprint) — for the chroma fingerprint method
• faster-whisper, numpy

Configuration

Config is TOML, read from (later wins):

1. /etc/kts.conf (system-wide)
2. $XDG_CONFIG_HOME/kts.conf (user-level, fallback ~/.config/kts.conf)
3. ./kts.conf (project-local, highest priority)

Command-line flags always override the config file. See conf_example.toml:

video-dir        = "video/"
audio-dir        = "audio/"
script           = "script.txt"
whisper-language = "fr"

Recognised keys: video-dir, audio-dir, script, cache-dir, whisper-language, whisper-model, whisper-cache-dir.

All intermediate results are written into cache-dir (default .kts/): match.json and arrange.json.

Step 1 — kts-match

kts-match --video-dir video/ --audio-dir audio/
# with a config file, simply:
kts-match

It pairs every video with every audio group and runs up to four independent sync methods:

Method	What it does	Speed
peaks	transient peak-constellation Hough matching	fast (~1 s)
envelope	RMS-envelope cross-correlation	fast (~5 s)
chroma	Chromaprint fingerprint sliding-window match	medium (~30 s)
whisper	word-level transcript alignment	slow (minutes)

A fusion layer votes across methods to pick the winning audio group and a weighted-mean consensus offset with a confidence level.

Output .kts/match.json:

{
  "MVI_8113.MP4": {
    "group": "4CH001",
    "offset": 2.347,
    "confidence": "HIGH",
    "method": "envelope,whisper",
    "votes": 4,
    "details": { "peaks": {}, "envelope": {} }
  }
}

Useful flags: --skip-peaks/--skip-envelope/--skip-chroma/--skip-whisper, --max-s, --delta-max, --whisper-model, --whisper-language, --force.

Audio files are grouped by name: NAME + role letter (M = mic, I = internal) + .wav, e.g. 4CH001M.wav / 4CH001I.wav → group 4CH001. A group needs at least an M track.

Step 2 — kts-arrange

kts-arrange --script script.txt --video-dir video/

Reads .kts/match.json, transcribes each clip with Whisper (cached), and fuzzy-matches the transcript against the script to determine which scene(s) each take covers. Takes covering the same scene are grouped so you can pick the best one.

Script format

## Scene 1
MARIE: Bonjour, tu vas bien ?
PIERRE: Très bien, merci.
[pause — ils se regardent]
## Scene 2
...

• ## lines are scene markers.
• [...] lines are stage directions (kept, but never matched).
• NAME: text gives the speaker.

Output .kts/arrange.json:

{
  "scenes": [
    {
      "scene": 1,
      "script_lines": [{"index": 0, "speaker": "MARIE", "text": "Bonjour…"}],
      "takes": [
        {
          "video": "MVI_8114.MP4",
          "audio_group": "4CH002",
          "offset_s": 1.23,
          "in_s": 4.5,
          "out_s": 18.2,
          "n_matched": 5,
          "n_script": 6,
          "anchors": []
        }
      ]
    }
  ]
}

Useful flags: --min-score (fuzzy threshold, default 0.55), --min-matched, --all-takes (deprecated — unmatched takes are always reported), --force-retranscribe.

Step 3 — kts-build

kts-build --cache-dir .kts -o project.kdenlive

Reads .kts/arrange.json and writes a Generation-5 Kdenlive project. Every take contributes one video clip (source in/out = in_s/out_s) and one audio clip per audio role of its group (M = mic, I = internal), shifted by offset_s so external audio stays in sync with the picture.

Two layouts are available:

• --layout stacked (default) — the best take of each scene is the reference and starts at the scene position. The other takes of the scene are shifted by the median of their shared-anchor offsets so matched script lines line up across takes, each take on its own track. Use --no-anchor-align for the old behaviour where all takes of a scene start at the same position.
• --layout sequential — takes are placed one after another, packed onto as few tracks as possible.

Useful flags: --width, --height, --fps auto|25|30000/1001, --video-tracks, --audio-tracks, --audio-roles MI, --gap 1.0, --rotate cw|ccw|180, --include-unmatched, --empty.

Pipeline overview

video/*.mp4   audio/*M.wav,*I.wav          script.txt
      │              │                          │
      └──── kts-match (peaks/envelope/          │
            │          chroma/whisper + fusion) │
            ▼                                   │
      .kts/match.json  (group + offset/clip)    │
            │                                   │
            └──── kts-arrange (whisper + fuzzy match)
                        │
                        ▼
              .kts/arrange.json  (scenes → takes)
                        │
                        ▼
              kts-build (timeline layout)
                        │
                        ▼
                 project.kdenlive

Project layout

src/kdenlive_transcript_sync/
    config.py        TOML config loader (/etc/kts.conf, ~/.config/kts.conf, ./kts.conf)
    match_cli.py     kts-match entry point
    arrange_cli.py   kts-arrange entry point
    build.py         kts-build entry point (timeline XML writer)
    fuse.py          cross-method voting / consensus offset
    align.py         script parsing + fuzzy transcript matching
    discover.py      file discovery helpers
    models.py        dataclasses (ScriptLine, TranscriptSegment, Anchor, …)
    methods/
        peaks.py     transient peak constellation
        envelope.py  RMS envelope cross-correlation
        chroma.py    Chromaprint fingerprints (fpcalc)
        whisper.py   faster-whisper transcription + alignment

Tuning tips

• Bad audio→video pairings: raise --delta-max only if offsets really can be large, and prefer --max-s 0 (full file) on long takes.
• Too many false script matches: raise --min-score; too many skipped lines: lower it.
• Set whisper-language in the config — auto-detect is noticeably worse on short or noisy clips.
• Use a bigger --whisper-model (small, medium) for noisy audio; the transcripts are cached in --whisper-cache-dir, so it only costs once.