This is the fourth entry in the Digital Companion Build Journal. She needed a body: first a browser tab, then a borderless window, and eventually the desktop wallpaper itself. Implementing three independent versions would make maintenance unmanageable. This article records the decision that allowed all three to coexist and the three infrastructure battles required to make it real.
Principle: the page is a display, not the application
There is one rule: every state, decision, and side effect belongs to the local server. The page receives events through WebSocket, sends requests over HTTP, and stores no source of truth. The result:
- all three hosts load one URL and serve as mutual fallbacks;
- the system continues when every page is closed—voice and hotkeys still work;
- every host-specific defect can be bypassed at the server, the common theme of the next three sections.
Battle one: moving the ears into the server
Speech recognition first used the browser Web Speech API. In wallpaper mode it met an unsolvable interaction: the microphone permission bubble is native UI, while the wallpaper host forwards the mouse only to page content. The bubble could never be clicked, and WebView2 speech services were unreliable.
The solution was to remove the page from the problem. The microphone belongs to the machine, not the page. The server records directly, transcribes with a local whisper model—about 0.7 seconds for 45 seconds of audio on the 16GB GPU—and sends text through the same route as typed input. No host now needs microphone permission, and private-context audio never leaves the machine.
Two supporting designs matter.
Self-listening suppression. With continuous recognition, her own speaker output reaches the microphone and can make her converse with herself. Playback therefore reports a speaking window, stored by the server as an absolute deadline rather than a Boolean flag. If the reporter dies, the deadline expires naturally; a flag could remain stuck forever. Recognition segments beginning inside that window are dropped. Push-to-talk bypasses suppression because holding the key expresses intent to interrupt.
Missing CUDA libraries appear only at inference. Successful model construction does not prove the model works. Without cuBLAS, construction succeeds and the first transcription fails. Startup must perform a half-second silent warm-up and fall back to CPU on failure. Another trap: ordinary LoadLibrary search observes PATH; Python’s add_dll_directory alone is insufficient.
Battle two: moving the voice into the server
The trigger was a wallpaper host that freezes the wallpaper process when another window gains focus. She stopped mid-sentence. After several rounds fighting host pause settings, the project returned to its rule: sound belongs to the machine, not the page.
The server now decodes audio and writes directly to the system audio stream. The benefits were broader than expected: host freezes no longer interrupt speech; multiple open hosts cannot duplicate playback because there is one source; the browser’s click-to-unlock overlay disappears; interruption accuracy improves to roughly 40 milliseconds because the stream is written in stoppable chunks; and the self-listening window is known precisely by the server.
One environment trap killed the entire voice route during this work. Under a Japanese locale, a log stream redirected to a file fell back to cp932; printing one simplified-Chinese sentence raised an exception. That print sat directly on the required path after transcription. The repair forced UTF-8 and made logging non-throwing. The lesson is simple: a log must not be able to kill what it observes.
Battle three: hands
The first global shortcuts used Win+letter combinations and collided broadly with system bindings. A programmable macro pad replaced them, sending F13–F19—key codes present in USB HID but absent from physical keyboards, so they do not conflict. The listener uses the native RegisterHotKey API. After discovering that the planned third-party hotkey utility had never installed correctly, the listener was rewritten in Python and added to the startup process.
The real opponent was switch bounce. One physical press could emit several events and broke three layers, each requiring a different repair:
| Location | Symptom | Repair |
|---|---|---|
| Push-to-talk key | Repeated start/stop while held | Release debounce: a new press within 120ms of release counts as bounce |
| Mode key | Closing one password dialog immediately opens another | Serialise with a mutex and add cooldown after close |
| Button inside the wallpaper page | Toggle appears not to respond because two clicks cancel | Capture-layer dedupe: a second click on the same target within 250ms is an echo |
One hardware defect required three solutions at three abstraction layers. That is perhaps the most representative engineering lesson in this entry.
Reusable principles
- The more hosts there are, the more logic should move downward. The return from “the page is a display” grows with every host.
- Capabilities belonging to the machine—microphone, speakers, and hotkeys—should not travel through a page. Every host limitation otherwise becomes yours.
- Timeouts and absolute deadlines are safer than Boolean flags. When an owner dies, time heals the former and leaves the latter stuck.
- Treat successful construction with suspicion; validate with the smallest real call.