Plain-language explanation + options with price tags · measured 2026-09-14 · base numbers: 6,666,998 clips, 3,793,720 repaired (56.9%)
What it looks like. The transcript (what was said) contains tokens the script does not have in the same spelling — or the other way round. Two common shapes:
Transcript: ... Mi'ali [Low Mumble] Merith ... Script: ... Mi'ali (Low Mumble) Merith ... ← same burst, different brackets AND case Transcript: ... Änderungsanträge . Script: ... Änderungsanträge ← trailing " ." lives only in the transcript
Why it happens. Almost all of these are synthetic clips (voice profiles, VC, drama box). Their transcripts were written by a different pipeline than the real-speech one, and that pipeline embeds non-verbal tokens ([Low Mumble] [yawn] [UH]) directly into the transcript text, while the script builder put them in round parens — or dropped a trailing period. The repair refuses to "translate" between the two conventions on its own, because a wrong translation silently corrupts training text.
| Option | What changes | Cost (basis below) | Difficulty |
|---|---|---|---|
| A0 — accept (current default) | Nothing. These clips stay untimed; the 20k/50k/100k recipes work unchanged (the old format still trains). | 0 | Easy |
| A1 — transcript convention + re-pass | Decide once, in writing: embedded tokens are transcript words (then the repair keeps them as words and only times the rest) or tags (then they are stripped from the transcript too, with WER-eval updated to match). Then ~half a day of dev to encode the rule + one more repair pass. | Dev ~4 h · compute ~0.5 node-hours (measured repair speed: 6.7M clips ≈ 10 node-minutes on 2 booster nodes) | Medium — the code is easy, the decision has eval consequences |
| A2 — re-transcribe synth with the real-speech pipeline | Run the real ASR + sentence splitter over all synth audio so transcripts and scripts come from one pipeline. Cleanest data, but it rewrites ~2M transcripts (WER baselines shift) and needs word-level timestamps for everything. | GPU ASR ≈ 50–150 node-hours for ~2,000 h of synth audio (rough: 0.1–0.3× realtime per GPU, 4 GPUs/node) + 1–2 days plumbing | Hard |
What it looks like. The record has audio, text, and scores — but the words[] list is empty, so there is nothing to measure a sentence duration from. Concentrated in synth sources whose audio was generated without running the aligner.
| Option | What changes | Cost | Difficulty |
|---|---|---|---|
| B0 — accept (current default) | Nothing; these clips stay untimed. | 0 | Easy |
| B1 — run the aligner afterwards | Force-align the existing transcript to the existing audio (Parakeet-TDT path already used in this project), store word times, re-run the repair. Fills the exact gap, touches nothing else. | Same GPU math as A2 but smaller (≈600k short clips ≈ 1,500 h audio → ≈ 40–110 node-hours) + ~1 day plumbing | Medium — pipeline exists, needs sharding + QA |
What it looks like. After sentences, pauses, and bursts are all printed, up to ~0.8 s of the clip is still unaccounted for — usually leading/trailing recording padding or a sentence the splitter missed. The repair folds at most 0.30 s into the last tag; anything bigger is kept untimed rather than inventing time.
| Option | What changes | Cost | Difficulty |
|---|---|---|---|
| C0 — accept (current default) | Nothing. | 0 | Easy |
| C1 — audit a sample, then decide | Listen to/measure 200 of these (where in the clip is the gap? music? silence? missed sentence?). Most likely outcome: raise the fold cap slightly or fix the sentence splitter for one missed pattern — a targeted fix, not a rebuild. | CPU-only, ~2–3 h analysis + small code change + re-pass ~0.5 node-hours | Easy |
| C2 — re-trim audio to words | Cut leading/trailing silence from the audio itself so clips start/end at speech. Changes audio and MOSS frames and every manifest — the most invasive option, only worth it if the audit shows systematic padding. | CPU re-cut + full re-encode of affected shards + manifest rebuild ≈ 1–2 days + downstream re-validation | Hard |
Now (free): keep repaired 56.9% + untimed 43% side by side — training works on both shapes. Next (hours): C1 sample audit (tells us whether C is padding or missed sentences). Then (decision): A1 transcript convention — unlocks most of the 26% in A. Later (only if needed): B1 aligner backfill for the 9% without word times. Only on audit evidence: C2 re-trim; A2 full re-transcription (most expensive, last resort).
Cost basis, honestly labelled. Repair-pass speed is measured (job 1787315: 6.67M clips, 2 booster nodes, ~5 min → ≈ 0.2 node-hours per full pass). GPU ASR/align numbers are rough estimates from typical Parakeet-TDT throughput (0.1–0.3× realtime per GH200 GPU); the real figure comes from a 1-shard pilot, which is how every job above should start. Billing reminder: booster nodes bill whole (288 cores × walltime), so all CPU passes run sharded like the repair job, never single-threaded.
Evidence: repair reports publish/scaling_ladder_repaired/data/*/*/tier*/*.report.json (2,682) · script code/m2_multitask/repair_prompts.py · driver repair_driver.py · job sacct 1787315 (COMPLETED 0:0).