1. Treat subtitle cues as the primary index

Every subtitle cue (start, end, text) becomes a navigable unit. Instead of currentTime, the UI state revolves around currentCueIndex. Keyboard shortcuts map to cue operations:

• previous / next sentence

• replay current sentence

• loop current sentence N times

• pause automatically at the end of each sentence ("auto-pause" mode)

Finding the active cue on timeupdate is a binary search over sorted start times, which stays cheap even for long transcripts.

2. Fix the cue boundaries before you trust them

Speech-to-text output and many downloaded subtitle files do not align neatly to sentences. Cues often split mid-sentence or merge two speakers. A few practical rules:

• Merge cues shorter than ~0.8 s into a neighbour.

• Split cues at sentence-final punctuation when a cue is longer than ~7 s.

• Pad the start by 100–200 ms when looping, because ASR timestamps tend to clip the first consonant.

3. Bilingual subtitles need a stable pairing

If you show a translation under the original line, pair translations to cue IDs, not to timestamps. Re-timing the original (step 2) should never desync the translation.

4. Word-level interaction

Click-to-look-up means tokenising each cue. For space-separated languages a regex tokenizer is fine; for Chinese and Japanese you need a segmenter. Store the cue ID alongside any saved word so a flashcard can replay the exact sentence clip later.

5. Keep media handling simple

Long videos and podcasts are where learners spend most time. Process audio in chunks for transcription, stream playback normally, and persist progress and saved words locally so a refresh never loses a session.

Where this ended up

These ideas power LangScribe's AI subtitles for language learning: it generates subtitles in about 99 languages, translates them into bilingual subtitles, and builds the whole player around sentence-by-sentence study, click-to-look-up words, and vocabulary cards with the original clip.

If you are building something similar, start with the cue index. Once "sentence" is a first-class concept in your state, almost every learner-facing feature becomes easier to build.

Stack notes: Vue 3 + Vite; hls.js for HLS; native media for MP4/WebM and local File; Pinia; Dexie/IndexedDB for signed-out libraries. Cloudflare Workers (Hono) serve assets and API.

Cue list as the timeline: jump previous/next cue, replay current, loop current, A-B loop. Bilingual text is two strings on the active cue. Clicking a cue seeks; seeking highlights the row. Hard parts are HLS quirks, VFR, and uneven ASS timing.

Subtitles from three sources: import SRT/VTT/ASS; AI transcription audio chunks to Worker to timed cues; translation tracks. Word lookup is a separate cached call so it does not block playback.

Shipping lesson: if cue interactions feel slow, users never care about STT cleverness. Optimize the loop first.

I use this shape in LangScribe for local-file language study; play the film locally; optional AI on audio and text only.