Ladder datasets: inventory + prompt-repair plan

Where every dataset lives, how big it is, and how the timed-prompt repair runs over it · measured 2026-09-14 · English, plain language

The short version. All ladder data is built the same way (same packer, same untimed prompt shape), so one repair script fixes all of it. The script is written, tested on 4 shards, and ready: 95% repaired on real speech, ~32% on synth (the rest is kept untouched with a logged reason — never guessed). Small sets and the 20k ladder can be repaired right now. The 50k/100k stages should be repaired after the running pack job finishes, as a second pass over finished shards. No repacking, no new scoring, no stopping of running jobs.

1 · Inventory — what exists and where

Dataset	Location (SC=/e/scratch/reformo/schuhmann1_moss)	Size (measured)	Status
Published ladder (the 20k source) Tier 0–7 + heldout, per-tar .mp3 + .moss.npy + .json	publish/scaling_ladder/data/<lang>/<origin>/{tier0..7,heldout}/ index in publish/scaling_ladder/index/… (202 cols)	20,081 h total = German 10,000 h + English 10,000 h (ladder.json: tiers/lange [50…10000], 2,694 shards, 6,692,637 elements). Tiers are nested: Tier7 already contains Tier0–6.	shipped, frozen (M2-20k trained on it)
2 × 10k (German 10k + English 10k)	same as above, Tier7 per language	10,000 h per language (≈3.3M clips each)	shipped (= Tier7 slices)
Tier0–6 (smaller subsets)	same ladder, lower tiers (nested prefixes of Tier7)	50 / 125 / 250 / 500 / 1,250 / 2,500 / 5,000 h per language	shipped
2 × 50k stages (in build)	out/scaling_stages/sel/… (selected.parquet) → pack/out/…	12,500 h × 4 cells = 50,000 h; Tier7 shards hard-linked/reused, rest packed fresh	packing (was 5,247/11,034, now ~7,980 tars)
100k stages (in build)	out/scaling_stages/… (same tree)	100,000 h total as built: DE 17,355 + 32,645 + EN 25,000 + 25,000 (selection tables in reports/STATE_scaling_stages.md). Note: this is 50k German + 50k English, not 2×100k — see §4.	packing (same job)
Mini + eval + smoke subsets	out/scaling_stages/mini/ (2×8 clips) · out/m2_20k_score/eval_holdout50 + eval/prompts.json (50) · out/m2_20k_score/{training_data_smoke,runtime_smoke,ddp_benchmark,continuation_smoke_v1}	hours to minutes	done, for tests
M2-20k training manifest	out/m2_20k_score/training_data/ (records.jsonl 24.8 GB + offsets/order)	6,666,998 rows (one per target)	frozen
M2-multitask manifest	out/m2_multitask/training_data/ (order/tasks .npy, no record dup)	29,975,019 presentations (5 tasks × targets, refs where verified)	trained (step 7319 COMPLETE)
Reference backfill (reusable)	out/m2_multitask/training_data/ref_backfill.parquet	3,307,664 verified same-voice refs	ready

2 · What the repair does (no repacking)

Script: code/m2_multitask/repair_prompts.py. Input: finished packed shards. It reads the timing that is already inside each record (words[start,end], sentences[start,end,dur], bursts[start,end,label]) and rewrites only the prompt string to the scheme format:

BEFORE (shipped):  (very smooth, very formal) Es riecht nach Sonnencreme.
AFTER  (repaired): [0.66 seconds pause]
                   (very smooth, very formal) [1.52 seconds duration] Es riecht nach Sonnencreme.
                   [1.22 seconds pause]

Sentence tags [N.NN seconds duration] from first-word onset to last-word offset; pauses [N.NN seconds pause] for every gap ≥ 0.20 s (leading/trailing included); bursts as (label, N.NN seconds) at the nearest word gap (two decimals so the printed sum lands within ~0.01 s; a bounded ≤0.30 s remainder is folded into the last tag and logged).
GENERAL's last word re-derived from measured spans: with-bursts iff the clip has ≥1 kept burst span, else burst-free — always emitted, so it agrees with the script by construction. Direction cues are kept word-identical (digits forbidden).
Spoken fillers ([UH] [UM]) and redactions ([company name]) are transcript words and are never stripped; bare burst spellings ([yawn], (Chuckle)) are removed and re-emitted timed from spans.
Original prompt is kept as prompt_untimed; anything that cannot be verified keeps the original prompt plus a logged reason (never guessed).

Independent verification per record (re-parse with no access to build internals): durations+pauses sum to clip length within 0.05 s · transcript after tag-stripping byte-identical · direction parens digit-free · every burst paren carries a duration · GENERAL token matches span presence.

3 · Test results (measured, 4 shards)

Cell (all tiers 0–7, full pass 2026-09-14, jobs 1787315+1788325, timed-v2)	Records	Repaired	Kept untimed (reason)
German real	1,554,476	89.3%	10.7% transcript-vs-script mismatch in source data
English real	1,565,277	91.0%	9.0% mismatch
German synth (voice profiles, VC, drama box)	1,833,801	43.1%	42.8% embedded burst-tokens in transcripts ([Low Mumble] etc.) · 14.1% no word times
English synth	1,713,444	38.1%	42.8% embedded tokens · 19.1% no word times
Total: 2,682 shards, 6,666,998 records		4,254,840 (63.8%)	1,824,174 transcript convention (accepted) · 587,421 no word times (accepted) · 563 fold edge cases

History: first pass (timed-v1, job 1787315) reached 56.9%; the C1 audit showed the large-gap residues were accumulated sub-0.20 s word gaps, so timed-v2 folds them into sentence boundaries per the scheme ("folded into the neighbouring speech") — large-gap skips went from 559,962 to zero. Per user decision: A (transcript convention) accepted as-is, B (no word times) left untimed, C fixed. Details in ~/PROTOKOLL_m2_repair.md.

Speed on login-node CPU: ~2,500 records in ~4 min single-threaded (≈10/s; the full 6.7M-record ladder ≈ 8 CPU-days single-threaded, i.e. hours on a 30–70-way sharded CPU job — same pattern as the running pack job). Reports land next to each repaired tar (.report.json).

4 · Two things you should decide

A. The 100k question. The built selection is 100,000 h total (German 50k + English 50k), per reports/STATE_scaling_stages.md — not 2×100k = 200,000 h. The German-real pool only holds ~17,400 h, so 100k German alone is impossible from current pools (German-synth topped it to 50k). If you want true 2×100k, new source pools are needed first; otherwise the "2×100k" runs as 2×50k-by-language. B. Synth transcripts with embedded tokens ([Low Mumble] inside the transcript text). The repair skips these (safe). Options: accept the skip (~40% of synth stays untimed), or define the convention (tokens stay as transcript words → keep bare spelling, or map to timed bursts).

5 · Rollout plan

READY NOW (small + 20k, no dependency on running jobs):
[1] Repair mini + eval_holdout50 (minutes) — done pattern proven 8/8
[2] Repair published ladder Tier7 (20k source): ~6.7M records, sharded CPU array (~70-way,
    same layout as pack job 1785746); outputs to publish/scaling_ladder_repaired/ (originals untouched)
[3] Rewrite training manifests to the repaired prompts (base rows keep uid/order; only prompt
    strings change) → retrain (retraining is no problem per your OK: same 32-node recipe)
AFTER the stages pack job finishes (do NOT stop it):
[4] Same repair pass over out/scaling_stages/pack/out (50k + 100k), then verify + membership checks
[5] Update consumers together with the data (same commit):
    - m2_20k_score/packing.py score_prompt() verifier for timed shapes
    - m2_multitask/tasks.py bracket_contents_concat() for timed shapes
    - eval renderers (they print prompts)
EFFORT: repair itself ~1.5–2 days to verified data (script done + tested; remaining: shard driver
sbatch + full-pass + verify). Retraining on top as usual (7 h / 32 nodes per 20k-scale run).

Evidence: ladder/code/pack_worker.py:49–91 · procedural-voice-captions/burst_captions.py:881–969 · caption.py:592–603 · prompt_schema_2026-08-25.html §3 · repair script code/m2_multitask/repair_prompts.py · test reports out/m2_multitask/repair_test/*.report.json · pack job sacct 1785746 (60/60 COMPLETED) · ~/ARBEITSPROTOKOLL.md §11:00.