Building a voice assistant that works for one very specific person

This post is about the engineering behind Neo, a voice companion for disabled and elderly users. It's currently running at one deployment in Vienna with a handful of users. I'm writing this because the problems turned out more interesting than I expected, and almost nobody writes about them.

Most voice assistants are built for a median user and then adapted outward. You build for someone who speaks clearly, sits upright, and can repeat themselves when misheard. Then you add accessibility features.

I started from the other end, mostly by accident. My co-founder uses a wheelchair and depends on mouthpiece ventilation. He's the primary user. That means every decision gets tested against someone the median-user assumptions don't describe. It's been the most useful engineering constraint I've ever worked under — it breaks things early and unambiguously. Here's what broke.

Ventilator noise sits in the speech band

The user takes air from a mouthpiece near their face — which means it's also near the mic. The result is continuous irregular turbulence: air rushing, pressure changes, mechanical cycling.

Standard noise suppression assumes noise is spectrally distinct from speech and roughly stationary. Ventilator noise fails both: it's non-stationary and its energy overlaps the speech band. Subtracting an averaged profile removes consonants along with the air.

What's worked better is non-stationary spectral gating (noisereduce with prop_decrease=0.85), which re-estimates continuously. Backing off from full suppression matters — at 1.0 the output sounds cleaner but transcribes worse, because aggressive gating eats plosives and fricatives. Sounding clean and being legible to an STT model are different objectives.

I don't consider this solved. It's better than off-the-shelf, and still the weakest link.

The microphone problem that wasn't the problem I thought

Before the interesting audio work, I lost embarrassing time to dumber issues. The USB mic captures at 3–15% of full scale — just quiet. Whisper on that input produces confident garbage. Fix: gain it 10x in software. Ugly, works.

The genuinely annoying one: the wake word detector ran fine from terminal but couldn't enumerate the mic when launched as a systemd service. Same binary, same user. Under systemd, pipewire wouldn't expose the device. Fix: Environment="XDG_RUNTIME_DIR=/run/user/1000". Without it, enumeration silently returns an empty list. Also: MIC_INDEX isn't stable between contexts. Test both. Assume nothing.

And don't share an audio device between pvrecorder and arecord in the same process. On Pi 5 ARM64 you get an ALSA mutex crash — hard segfault. Tear down the first recorder completely before starting the second.

Whisper hallucinating on its own echo

The system speaks via TTS. The mic picks that up. Whisper transcribes it. The system treats its own voice as user input. Worse: Whisper hallucinates on near-silence — feeds back plausible invented text from training artifacts.

Fix: below an RMS threshold, skip transcription entirely. Silence should be handled before it reaches the model, not by the model.

Language detection was also tricky: users speak German and English, often mixed. We run tiny first, use its language guess to decide whether to escalate to base for German. We also inject an Austrian-German vocabulary list as initial_prompt — measurably helps on regional terms.

Fall detection assumes bodies that stand up

This is where the median-user assumption fails hardest. Pose-based fall models are trained on people who stand, walk, and fall from standing. Our actual users: wheelchair transfers, walker users, people who may be horizontal and still for long stretches normally. Standard models flag all of these systematically. False alarm rate was unusable.

Two decisions: pose runs on-sensor (Sony AITRIOS IMX500). Raw frames never leave the camera — only pose metadata. That was a privacy decision, but it also frees the Pi's CPU for audio. Second: the alarm path is completely decoupled from the LLM. Fall detection to caregiver notification is deterministic. No model call, no network dependency. If the LLM is down, the alarm still fires.

I feel strongly about this: a safety-critical function that depends on a language model's output is not safety-critical. The companion side can be probabilistic and wrong occasionally. The "someone is on the floor" side cannot.

We're targeting under 0.5 false alarms per day per user. We're not there yet — wheelchair transfers remain our worst false-positive category.

What "local" actually means

Stays local always: raw audio, raw video. STT and TTS run on the Pi. Pose runs on the sensor. Nothing leaves the home architecturally.

Leaves the home: a routing tier. Routine queries handled locally. Complex ones escalate to a self-hosted EU model, pseudonymized. Fall metadata goes to a caregiver dashboard. Health logging is server-side. All in the DPA with our pilot.

Separately: the mobile app is cloud-backed — lighter-weight for family, different privacy properties. Calling both "Neo" may be a naming decision I regret.

Routing target: ≥60% of recurring queries local. Takes inference cost from ~€4.50 to under €0.70 per user per month.

What the constraint bought

Building for one person with unusual requirements produced a system better for everyone — not for the usual "accessibility helps all" reason. It's more specific: hard constraints eliminate lazy design.

If STT has to work through ventilator noise, it works through a noisy kitchen. If fall detection can't assume standing, you think harder about what the sensor sees. If the user can't repeat themselves, you can't paper over failures with "say that again." If onboarding must work without settings, you build something that needs none.

Every one of those started as an accommodation and ended as a general improvement — because it removed an escape hatch.

Where this is

One deployment at a disability services organization in Vienna. Small number of users, daily use. TRL 6. ~€200 in hardware per install (Pi 5 + AITRIOS camera).

Things I don't have good answers for yet: false positives on wheelchair transfers (best idea: per-user calibration); long-horizon memory drift (we keep care-critical facts write-protected, but the boundary is hand-drawn); whether any of this survives contact with more than one deployment.

Happy to answer questions about the audio pipeline or on-sensor work.

Neo Companion — VXNeo Labs, Vienna. Mobile app free on iOS and Android (companion features only; fall detection requires the edge device).

• iOS: https://apps.apple.com/app/id6768433086
• Android: https://play.google.com/store/apps/details?id=com.vxneolabs.neo_companion