Live Speaker Transcript

Live Speaker Transcript combines real-time face tracking with browser-native speech recognition to produce a transcript that's colour-attributed to whichever visible face was most likely speaking. Everything runs client-side — no video, audio, or transcript text is ever sent to a server.

Faces are detected with MediaPipe FaceLandmarker (v0.10.3), which returns 478 normalised landmarks per face at roughly 30 frames per second. Every inference frame, a mouth-openness score is computed per face from the vertical lip-gap distance, normalised by inter-eye distance so it doesn't depend on distance from the camera, and compared against that face's own openness over the last ~400 ms. Scoring the range of recent movement, rather than the single-frame change, is what separates an actively talking mouth from ordinary landmark jitter on a still face. A short debounce (default 400 ms, adjustable) smooths only the on-screen glow/label so it doesn't flicker — it does not affect which speaker an utterance is ultimately logged under. Separately, a Web Audio analyser measures microphone volume; either volume above threshold or live speech-recognition text already in progress counts as "someone is talking." With only one face in frame, mouth-comparison is skipped entirely — there's no one else it could be. If no face stands out, or the room is quiet, the utterance is attributed to unknown.

The transcript itself comes from the browser's SpeechRecognition API (webkitSpeechRecognition in Chrome/Edge — this tool requires one of those browsers, and a secure (HTTPS or localhost) origin, since neither the microphone nor speech recognition are available otherwise). Interim results update the current line live as they're refined; once a result is finalised, the line is locked in and coloured using whichever speaker was credited most often while that utterance was in progress — using the same per-frame mouth-activity signal, not the debounced on-screen one, so a brief flicker in the glow doesn't bias the vote. Because recognition's first interim result typically lags true speech onset by a few hundred milliseconds, each new utterance's vote is seeded with a rolling ~1 second buffer of mouth activity recorded just before recognition caught up, so the opening of an utterance isn't attribution-blind. Chrome's continuous recognition mode also restarts itself periodically (silence timeouts, an internal time cap) — when that happens mid-sentence, whatever was still in progress is committed as a best-effort finalisation first, so an engine hiccup drops at most a fragment rather than silently losing the line; the status badge briefly shows "Reconnecting…" when this happens.

Why live transcription has no model choice, unlike the uploaded-video path (below): SpeechRecognition is effectively the only option that gives genuinely real-time, word-by-word live transcription — an alternative like Whisper doesn't run as a streaming model at all, so using it live would mean continuously buffering short chunks of audio and running a full inference pass on each one, then stitching the partial results back together. That's a fundamentally different, much heavier engineering effort (and a real accuracy/latency tradeoff even once built) rather than a simple dropdown swap, so it isn't offered here. One honest tradeoff worth knowing about SpeechRecognition itself: in Chrome, it isn't guaranteed to be fully on-device — the browser's default implementation has historically sent audio to Google's servers for recognition, which sits in tension with the rest of this tool's fully client-side design. There's no user-facing setting to force on-device recognition today; if that matters for your use case, the uploaded-video path's Whisper transcription (fully local, your choice of model size below) is the fully client-side alternative.

Up to four faces are tracked and colour-coded (red, blue, green, orange) by their left-to-right position on screen; a 5th+ face is drawn in grey and is never eligible for speaker attribution.

Head-tracking model is a live-only choice under Settings → Face detection: MediaPipe only (the default) runs a single detection pass per frame — fast enough for real-time use, but it's the same short-range detector described in the limitations below, so small, distant, or steeply-angled faces can go undetected. RetinaFace + MediaPipe swaps in the same two-stage pipeline built for uploaded video (a dedicated face-finder scans the whole frame first, then MediaPipe gets a tight crop per face) — measurably better at catching those same small/distant/angled faces, but at a real cost: it hasn't been validated for continuous real-time use the way the upload path has, and doing extra inference work every frame can introduce visible lag, especially with several people in frame at once. Try it if MediaPipe alone is missing someone; switch back if the tool starts stuttering.

Uploading a video works too, but through a different pipeline, because SpeechRecognition manages its own internal microphone capture and can't be pointed at an arbitrary media file. You only ever watch and control one video — an ordinary player with a normal scrub bar — but under the hood a second, hidden copy of the same file does the actual analysis. That split exists because the analysis genuinely can't tolerate being scrubbed: it detects who's talking by comparing mouth movement over a short rolling window of recent frames, and it records audio in the order it plays, so jumping around mid-analysis would compare frames that aren't really adjacent and capture audio out of order, corrupting the transcript. Keeping the two copies independent means the player you interact with is never at risk of any of that — scrubbing it is always safe, at any time.

Uploading doesn't start analyzing immediately. After the video loads, it pauses on the visible player, and you can scrub freely to find where you want analysis to begin — useful for skipping a boring intro, or focusing on one part of a longer recording. This is also your chance to optionally draw a crop box directly on the video, narrowing which part of the frame gets searched for faces — handy if your group only occupies part of a wide shot, or to deliberately exclude bystanders. It's optional, though, because uploaded video runs through a different, stronger detection pipeline than the live webcam path: a dedicated face-finder model (RetinaFace, ~1.8 MB, downloaded and cached the first time it's used) scans the frame (or your crop region, if drawn) for every face at once, and only then hands each one, individually and tightly cropped, to MediaPipe for the actual mouth/pose landmarks — automatically doing for every detected face what the crop tool previously required drawing by hand for one. When you're ready, clicking "Start analysis" begins the real-time pass from wherever you've scrubbed to. If you scrub the visible player again while that's running, analysis stops immediately — nothing durable has been written yet at that point, so there's nothing to salvage — and you're back to picking a starting point; click "Start analysis" again to run it from the new spot. Once a run reaches the end of the video uninterrupted, its captured audio is handed to an entirely separate, still fully client-side, speech-to-text model — OpenAI Whisper, running in-browser via Transformers.js (unquantized weights, needed for reliable loading with the current transformers.js version — bigger than a quantized build would be, but that's what actually loads reliably today). Settings → Uploaded video (models) lets you pick which size: Tiny (~150 MB, fastest, least accurate), Base (~290 MB, balanced), or Small (~970 MB, slowest, most accurate) — each downloaded and cached separately the first time it's selected. The same settings panel also lets you turn off the RetinaFace face-detection pass described below, falling back to MediaPipe alone for faster, lighter analysis when you don't need the small-face improvement. Whichever model you pick transcribes the whole clip in one pass — using your browser's GPU via WebGPU when it's available, for meaningfully faster transcription, and quietly falling back to the ordinary CPU/WASM path otherwise (WebGPU support isn't universal yet, and a couple of the same browsers/setups already noted as unstable for Sharingan are worth watching here too). Each resulting transcript chunk is then attributed to whichever face's mouth activity dominated that chunk's time range (using the activity recorded during playback), and committed through the exact same pipeline a live utterance uses — same colouring, same stats, same CSV rows, with timestamps mapped back to the original video's timeline regardless of where analysis started. Because transcription only happens after the video finishes playing, an uploaded video's transcript, timeline, and metrics only finish populating a short while after the video ends — there's a load bar and status line to track progress, but it isn't the same instant experience as the live webcam mode. There's also no live face-tracking view for uploads — analysis runs invisibly in the background, since the only video worth watching is the one you already control. Attribution is also coarser than live: it's decided per Whisper chunk (a sentence or so) rather than per utterance, so a mid-chunk speaker change can be missed.

Three honest limits worth knowing. With a single shared microphone, this is mouth-movement-based attribution, not true audio diarization — it can't separate two people talking over each other, since there's only one volume signal for the whole room. Because a face's on-screen left-to-right position (not a face identity/embedding) is what "Person 1"–"Person 4" refers to, if two people swap which side of the frame they're on, their colour and running stats swap with them.

The third is more involved, and the story behind it is worth telling honestly. The live webcam path still relies solely on MediaPipe's own built-in face detector, which is tuned for faces reasonably close to and facing the camera — small, distant, angled, or downward-tilted faces (someone reading or looking at a table rather than the lens) can go undetected there, and no confidence setting recovers it; that's a real limitation of the live path with no workaround today. The uploaded-video path no longer has this problem nearly as badly, because of testing driven directly by a real failure report: a user's own footage — two people, moderate distance, heads tilted down toward documents — was measured at just 2.7% frame-detection with the old single-detector approach, and cropping (which fixes size) didn't help, because the real problem was head angle, not size. Swapping in a dedicated face-finder model (RetinaFace) as a first pass — scanning the whole frame for every face before MediaPipe ever looks at it, then handing each one a tight, individual crop — was tested against that exact same footage and measured at 100% frame detection in isolation, and roughly 72–94% end-to-end through the full app (two faces found in most sampled frames, at least one in nearly all of them, running at real-time speed with no crashes). One further real bug surfaced and fixed along the way: feeding MediaPipe a rapidly alternating sequence of different people's face crops on its normal "video" mode — built for one continuous camera feed with smooth frame-to-frame change — intermittently confused its internal tracking; switching that step to MediaPipe's stateless "single image" mode (the semantically correct choice anyway, since each crop really is an unrelated still image) fixed it. None of this is a complete solve — a difficult-enough angle or a badly compressed, very low-resolution source can still go undetected, and this was validated on one real case (downward head tilt) rather than exhaustively across every possible pose — but it's a measured, large improvement over relying on MediaPipe's detector alone, specific to uploaded video. The manual crop tool still exists and still helps (narrowing the search area, or deliberately excluding bystanders), but is no longer required the way it once was for this problem.

One more real bug this testing caught: a video with no audio track at all (silent, or video-only) used to still get transcribed, because Whisper — like most speech-to-text models — can hallucinate short phrases (things like "you" or "Thank you.") from pure silence rather than returning nothing. The upload pipeline now checks the actual signal level of the captured audio, not just how long it is, before deciding whether there's anything worth transcribing — a silent or audio-less video now correctly produces an empty transcript instead of a fabricated line.

Per-speaker turn count and word count are tracked alongside the face boxes as basic participation metrics, and shown again as a per-speaker breakdown in the session summary once recording stops.

The interaction timeline, next to the transcript, shows the last 30 seconds of activity as a rolling, continuously-scrolling Gantt chart: one lane per person, time flowing downward, newest at the bottom — deliberately drawn as a tall, narrow strip rather than a wide one, since time is the axis actually worth reading here. Unlike the transcript's speaker attribution — which always picks a single "most likely" winner per utterance — each lane here is scored independently against its own mouth-activity threshold, so two lanes can legitimately light up at the same time. That makes this the one place in the tool where genuine overlap is actually visible, alongside gaps (silence) and each person's speaking rhythm, all at a glance. The currently-still-talking segment in each lane renders at full opacity; finished segments are slightly dimmed.

Body orientation is approximated from head pose: MediaPipe's per-face transformation matrix gives the 3D rotation of the head, from which a horizontal (yaw) angle is derived. This only means something with two or more people on screen — with one face there's no "group" to face, so no orientation indicator is shown. With two or more, each person's head direction is compared against where the other detected faces actually are, not the camera: a small turn (within the configurable threshold, 35° by default) is treated as still facing the group regardless of exact direction, and a larger turn only counts as "facing" if it's directed toward where a groupmate is rather than off to the side. This assumes head orientation tracks body orientation closely enough in casual conversation to serve as a proxy — it can miss someone who turns their torso but keeps their head facing the group.

The data is exported as a CSV file structured as follows:

speaker_rank is the 0-indexed left-to-right position of the attributed face (blank if unknown); speaker_color is the hex colour drawn on screen for that face (grey if unknown). One row is written per finalised utterance while recording is active.

References:
Google. MediaPipe Face Landmarker. developers.google.com/mediapipe/solutions/vision/face_landmarker
MDN. SpeechRecognition API. developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition
Radford, A. et al. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). github.com/openai/whisper
Hugging Face. Transformers.js. github.com/huggingface/transformers.js

Source Code

• https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition