This is the second entry in the Digital Companion Build Journal and records the project’s first major abandonment: local real-time lip sync. Three open-source routes were evaluated—wav2lip through LiveTalking, MuseTalk v15 through the same engine, and LatentSync offline. The conclusion first: none crossed the quality threshold required for a photoreal companion, and the gap could not be closed through tuning.
Test environment and method
The machine ran native Windows 11 with an RTX 5070 Ti 16GB (Blackwell, sm_120) and PyTorch 2.9.1+cu128. The source was a photoreal 1904×1072 video.
Validation did not rely on visual impression alone. Two scripts measured whether lip motion was actually occurring. The first read the true mouth region from the face-coordinate file and compared frame-to-frame change during speech and silence. The second subtracted silent motion from speaking motion pixel by pixel to confirm that change was concentrated around the mouth rather than spread as full-face noise.
That method immediately exposed its own trap. The first result said the mouth barely moved, with a ratio of 0.97×. Investigation showed that the two backends stored their coordinate fields in opposite orders—one y1,y2,x1,x2 and the other x1,y1,x2,y2—with no comments. Reading the intuitive order silently measures an area beside the face without raising an error. After correction, the ratio became 1.52× and the motion peak landed exactly on the mouth.
Measured results
The same source, audio, and machine were used throughout:
| Metric | wav2lip256 | MuseTalk v15 |
|---|---|---|
| Rendering frame rate | 24.9–25.1 fps | 24.9–25.1 fps |
| Audio-to-mouth latency | 217 ms | 443 ms |
| VRAM at startup | 6.2 GB | 9.4 GB |
| Batch size | 16 (default) | Had to fall to 4 |
| Mouth motion (speech/silence) | 0.941 | 1.626 |
The quality comparison was decisive. wav2lip barely opened the mouth and left a grey rectangular patch around it with a visible edge. MuseTalk produced a genuinely open mouth, visible teeth, and no obvious compositing boundary. If forced to choose, MuseTalk won—but there was no reason to accept the forced choice.
Seven quiet failures
This section may be more reusable than the performance table. Every failure shared one property: nothing reported an error; the system failed while looking normal.
- VRAM fallback without an OOM. High-resolution face detection plus the resident service exceeded 16GB. Instead of raising an exception, the Windows NVIDIA driver fell back to system memory. The retry-on-OOM code never ran; the task simply became hundreds of times slower and remained on its first batch after fifteen minutes.
- The same MuseTalk trap at high resolution. A batch size of 16 filled memory at startup. Idle playback stayed at a perfect 25 fps, while inference fell to 1 fps and accumulated more than ten seconds of audio, with zero exceptions.
- Duplicated-frame stutter. Converting 24 to 25 fps with
-r 25duplicates one frame each second. The freeze is regular and obvious, and once baked into the avatar frames the renderer cannot repair it. Both server and browser still report a perfect 25.0 fps with no dropped frames. Motion interpolation removed the duplicates. - Successful completion with zero frames. When the video path could not be opened, the extraction tool reported no error and marked the job complete at 100%. Only playback revealed a divide-by-zero failure in another thread.
- Dirty-directory contamination. Regenerating an avatar did not clear its old frame directory. When the new source was shorter, the tail of the previous version remained—precisely after the carefully prepared seamless join—and destroyed the loop.
- LatentSync’s path trap. Its internal ffmpeg command did not quote paths containing spaces. Intermediate files disappeared silently, followed by a stack trace that looked like face-detection failure; the true cause was two layers away.
- The coordinate-order trap described above.
Why the route was abandoned
The models were not abandoned because they failed to run. All three ran, and the real-time routes held their nominal frame rate. They were abandoned because:
- The quality difference was qualitative, not incremental. Compared with mature commercial systems, pasted boundaries, missing micro-expression, and the uncanny-valley effect were not parameter problems but a model-generation gap.
- The route forms a trilemma. Photoreal real-time expression, lip sync, and motion control currently demand either VRAM beyond a consumer machine, seconds of latency from a heavier offline pipeline, or a major quality compromise from a lighter model. If all three concessions are unacceptable, the route is a wall at this point in time.
For this project, the escape was to redefine the problem: stop matching the mouth to every word and use a pool of high-quality prerendered clips with an independent voice layer. Image and voice quality are preserved separately. The cost is unsynchronised lips, which proved far less damaging in use than the uncanny valley.
What remained useful
The abandoned route still left reusable work: forward playback and a 25fps interpolation tool modified under the upstream licence, a zero-frame guard, dirty-directory cleanup, and the validation method documented here. Its most important legacy is a working rule: anything that looks normal deserves one more verification with data.