Designing Sentence-Level Navigation for a Subtitle-Based Language Learning Player

When you build a video player for language learners, the timeline scrubber quickly turns out to be the wrong tool. Learners do not think in seconds; they think in sentences. "Play that line again" is the most common action, and a seek bar makes it surprisingly hard. Here are the design decisions that mattered most when we built sentence-level navigation into a subtitle-driven player.

1. Treat subtitle cues as the primary index

Every subtitle cue (start, end, text) becomes a navigable unit. Instead of currentTime, the UI state revolves around currentCueIndex. Keyboard shortcuts map to cue operations:

• previous / next sentence

• replay current sentence

• loop current sentence N times

• pause automatically at the end of each sentence ("auto-pause" mode)

Finding the active cue on timeupdate is a binary search over sorted start times, which stays cheap even for long transcripts.

2. Fix the cue boundaries before you trust them

Speech-to-text output and many downloaded subtitle files do not align neatly to sentences. Cues often split mid-sentence or merge two speakers. A few practical rules:

• Merge cues shorter than ~0.8 s into a neighbour.

• Split cues at sentence-final punctuation when a cue is longer than ~7 s.

• Pad the start by 100–200 ms when looping, because ASR timestamps tend to clip the first consonant.

3. Bilingual subtitles need a stable pairing

If you show a translation under the original line, pair translations to cue IDs, not to timestamps. Re-timing the original (step 2) should never desync the translation.

4. Word-level interaction

Click-to-look-up means tokenising each cue. For space-separated languages a regex tokenizer is fine; for Chinese and Japanese you need a segmenter. Store the cue ID alongside any saved word so a flashcard can replay the exact sentence clip later.

5. Keep media handling simple

Long videos and podcasts are where learners spend most time. Process audio in chunks for transcription, stream playback normally, and persist progress and saved words locally so a refresh never loses a session.

Where this ended up

These ideas power LangScribe's AI subtitles for language learning: it generates subtitles in about 99 languages, translates them into bilingual subtitles, and builds the whole player around sentence-by-sentence study, click-to-look-up words, and vocabulary cards with the original clip.

If you are building something similar, start with the cue index. Once "sentence" is a first-class concept in your state, almost every learner-facing feature becomes easier to build.