How LAPSE works
Read the full report (PDF, 62 pages)
Abstract
LAPSE moves a subtitle file onto the speech in a video. It corrects a constant offset, the linear drift that a framerate conversion leaves behind, and the piecewise offsets of recuts, advert breaks and files that hold two parts, and it does so without reading the language of either side. This edition describes version 2.5.0.
Speech is detected with Silero VAD, or with libfvad when the neural model is missing, and turned into weighted spans that start 150 ms before the detector fires. Offsets are found with a two channel FFT cross correlation on speech blocks and speech onsets, with a running mean removed from both channels. Under a doubly stochastic point process model of speech onsets, that removal keeps 95 to 96% of the energy of the peak at the correct offset and a fraction 47/10080 (W/τ)3 of the slow component that dense dialogue adds, with W the width of the running mean and τ the correlation time of the dialogue density. Candidates are refined on an exact overlap score. That score is a sum of truncated distances to plateaus, so its local maximiser is a weighted interval median, and its asymptotic efficiency under Gaussian timing noise is 2/π. Candidates are ranked by a lock statistic that compares each offset with the same file at random offsets. Drift is searched over the standard framerate pairs and then, if none fits, by a grid and golden section search over ratios between 0.94 and 1.06. Files that need splitting are aligned by penalised dynamic programming on an irregular grid of offsets, with an unconstrained jump at twice the penalty for scenes that are missing from the video. The verdict that decides whether the original file is overwritten is computed on the file that the split leaves behind.
On 39 feature films chosen for being hard to synchronise, LAPSE 2.5.0 passed 37 in the shift and drift test, against 31 for alass 2.0.0 and 33 for ffsubsync 0.5.1. With cuts injected it passed 34 when it had to find the cuts by itself, up from 19 for LAPSE 2.0.0 (exact McNemar p = 6.1 × 10−5), and 37 with splitting forced on, against 28 for alass and 18 for ffsubsync's split mode. On the 27 films where verdicts were recorded, all 22, 16 and 17 of its solid verdicts in the three tests were correct. A first run on all 39 films took 373 s, 27% less than 2.0.0. A run from the speech cache is about a fifth slower than in 2.0.0, because a wide search for distant parts of the file runs every time.
Contents
- Introduction
- Background and related work
- Formal model
- Architecture and libraries
- Speech detection
- Searching for one offset
- Refinement and the lock statistic
- Drift
- Split alignment
- Choosing the correction and judging it
- A worked example
- Evaluation
- Development history
- Threats to validity and limitations
- Conclusion and future work
Appendices: constants, per film results, and running the engine.
Read the full report (PDF, 62 pages)