kdenlive-transcript-sync (0.6.0)
Installation
pip install --index-url About this package
Align multi-cam video clips on a Kdenlive timeline using multi-method audio sync
kdenlive-transcript-sync (kts)
Tooling to rebuild a Kdenlive project from a raw multi-cam / external-recorder shoot: figure out which external audio recording belongs to which camera clip, how far apart they are in time, and which part of the script each take covers — then arrange all of that on a timeline with the audio aligned to the video.
Typical setup it is built for:
- one or more cameras producing video files with (poor) on-camera sound,
- a separate audio recorder producing
.wavfiles (e.g. a mic track…M.wavand an internal/scratch track…I.wavper recording group), - a text script of what is supposed to be said, split into scenes.
Status
The project currently ships two CLIs, which are the two steps of the pipeline:
| Command | Role |
|---|---|
kts-match |
Step 1 — match each video clip to an audio group and compute the sync offset |
kts-arrange |
Step 2 — match each take against the script and group takes by scene |
The final step — writing the .kdenlive XML timeline from arrange.json — is
the goal of the project and is not wired up in this branch yet. Everything
before it (matching, offsets, scene grouping) works and is cached as JSON.
Install
pip install . # or: pip install -e .
Arch users can build the package with the provided PKGBUILD.
External dependencies:
ffmpeg— audio decoding for the peaks / envelope / whisper methodsfpcalc(chromaprint) — for the chroma fingerprint methodfaster-whisper,numpy
Configuration
Config is TOML, read from (later wins):
/etc/kts.conf$XDG_CONFIG_HOME/kts.conf(fallback~/.config/kts.conf)
Command-line flags always override the config file. See conf_example.toml:
video-dir = "video/"
audio-dir = "audio/"
script = "script.txt"
whisper-language = "fr"
Recognised keys: video-dir, audio-dir, script, cache-dir,
whisper-language, whisper-model, whisper-cache-dir.
All intermediate results are written into cache-dir (default .kts/):
match.json and arrange.json.
Step 1 — kts-match
kts-match --video-dir video/ --audio-dir audio/
# with a config file, simply:
kts-match
It pairs every video with every audio group and runs up to four independent sync methods:
| Method | What it does | Speed |
|---|---|---|
peaks |
transient peak-constellation Hough matching | fast (~1 s) |
envelope |
RMS-envelope cross-correlation | fast (~5 s) |
chroma |
Chromaprint fingerprint sliding-window match | medium (~30 s) |
whisper |
word-level transcript alignment | slow (minutes) |
A fusion layer votes across methods to pick the winning audio group and a weighted-mean consensus offset with a confidence level.
Output .kts/match.json:
{
"MVI_8113.MP4": {
"group": "4CH001",
"offset": 2.347,
"confidence": "HIGH",
"method": "envelope,whisper",
"votes": 4,
"details": { "peaks": {}, "envelope": {} }
}
}
Useful flags: --skip-peaks/--skip-envelope/--skip-chroma/--skip-whisper,
--max-s, --delta-max, --whisper-model, --whisper-language, --force.
Audio files are grouped by name: NAME + role letter (M = mic, I =
internal) + .wav, e.g. 4CH001M.wav / 4CH001I.wav → group 4CH001.
A group needs at least an M track.
Step 2 — kts-arrange
kts-arrange --script script.txt --video-dir video/
Reads .kts/match.json, transcribes each clip with Whisper (cached), and
fuzzy-matches the transcript against the script to determine which scene(s)
each take covers. Takes covering the same scene are grouped so you can pick
the best one.
Script format
## Scene 1
MARIE: Bonjour, tu vas bien ?
PIERRE: Très bien, merci.
[pause — ils se regardent]
## Scene 2
...
##lines are scene markers.[...]lines are stage directions (kept, but never matched).NAME: textgives the speaker.
Output .kts/arrange.json:
{
"scenes": [
{
"scene": 1,
"script_lines": [{"index": 0, "speaker": "MARIE", "text": "Bonjour…"}],
"takes": [
{
"video": "MVI_8114.MP4",
"audio_group": "4CH002",
"offset_s": 1.23,
"in_s": 4.5,
"out_s": 18.2,
"n_matched": 5,
"n_script": 6,
"anchors": []
}
]
}
]
}
Useful flags: --min-score (fuzzy threshold, default 0.45), --min-matched,
--all-takes (also report takes with no match), --force-retranscribe.
Pipeline overview
video/*.mp4 audio/*M.wav,*I.wav script.txt
│ │ │
└──── kts-match (peaks/envelope/ │
│ chroma/whisper + fusion) │
▼ │
.kts/match.json (group + offset/clip) │
│ │
└──── kts-arrange (whisper + fuzzy match)
│
▼
.kts/arrange.json (scenes → takes)
│
▼
[planned] project.kdenlive
Project layout
src/kdenlive_transcript_sync/
config.py TOML config loader (/etc/kts.conf, ~/.config/kts.conf)
match_cli.py kts-match entry point
arrange_cli.py kts-arrange entry point
fuse.py cross-method voting / consensus offset
align.py script parsing + fuzzy transcript matching
discover.py file discovery helpers
models.py dataclasses (ScriptLine, TranscriptSegment, Anchor, …)
methods/
peaks.py transient peak constellation
envelope.py RMS envelope cross-correlation
chroma.py Chromaprint fingerprints (fpcalc)
whisper.py faster-whisper transcription + alignment
Tuning tips
- Bad audio→video pairings: raise
--delta-maxonly if offsets really can be large, and prefer--max-s 0(full file) on long takes. - Too many false script matches: raise
--min-score; too many skipped lines: lower it. - Set
whisper-languagein the config — auto-detect is noticeably worse on short or noisy clips. - Use a bigger
--whisper-model(small,medium) for noisy audio; the transcripts are cached in--whisper-cache-dir, so it only costs once.