Wake-word detection: always-listening smart devices

“Always listening” sounds more alarming than most wake-word hardware needs to be. The microphone is active, yes, but the device can process short audio frames locally, keep no recording, and discard them unless a small model hears the trigger phrase.

That boundary matters. A wake-word detector is not speech-to-text and should not behave like an open microphone to a server. I think of it as a low-power gate: it answers one narrow question — “did this sound like the wake phrase?” — before the heavier voice pipeline is allowed to start.

For this article, I used an Apache-2.0 training framework that produces TensorFlow Lite Micro models for constrained devices. ESPHome implements its on-device runtime, which makes it a practical reference for an ESP32-S3 build.

What is actually listening?

The normal loop looks like this:

1. An I2S microphone produces a continuous PCM stream.
2. Firmware slices it into short overlapping frames and extracts compact spectral features.
3. Voice activity detection (VAD) can reject frames that do not look like speech.
4. The wake-word model returns a probability for each new window.
5. Firmware smooths several results and compares them with a threshold.
6. Below the threshold, the audio is overwritten. Above it, the device lights an indicator and starts command capture.

No transcript is produced in that first path. On a privacy-first device, neither raw frames nor features leave the enclosure before detection.

The always-on path remains on the device. Only a confirmed wake event opens the command-capture path.

A practical ESP32-S3 baseline

I would start with an ESP32-S3 board with PSRAM, an I2S MEMS microphone, a small amplifier and speaker if the device must reply, and a physical switch that cuts microphone power. The switch should not merely set a software flag; someone looking at the device should be able to know the microphone cannot work.

ESPHome keeps the wake-word configuration short:

micro_wake_word:
  microphone:
    microphone: i2s_mic
    channels: 0
    gain_factor: 4
  vad:
  models:
    - model: okay_nabu
      id: wake_model
      probability_cutoff: 97%
      sliding_window_size: 5
  on_wake_word_detected:
    then:
      - voice_assistant.start:
          wake_word: !lambda return wake_word;

The microphone pins and audio output remain board-specific. I would first verify clean 16 kHz mono capture, then add the model, and only then connect the voice assistant. Debugging I2S, inference, networking, and playback at once is a good way to make every failure look mysterious.

A useful first prototype is small: an ESP32-S3, one I2S microphone, a speaker, and a switch that physically removes microphone power.

Thresholds trade one annoyance for another

Two errors matter:

• A false reject means you said the wake phrase and nothing happened.
• A false accept means TV audio, conversation, or noise activated the device.

Raising probability_cutoff usually reduces false accepts but increases false rejects. A larger sliding_window_size asks for more consistent evidence, often at the cost of latency. VAD helps reject non-speech noise, but it cannot protect against a television clearly saying something similar to the wake phrase.

There is no universal best number. Microphone placement, enclosure openings, room echo, speaker volume, accent, and speaking distance all change the result. Tune on the final enclosure, not on a bare board ten centimeters from your mouth.

Test the room, not one clean recording

My test sheet would include:

Test	What I record
Intended phrase at 0.5, 2, and 4 meters	Detected attempts / total attempts and response time
Quiet speech, normal speech, calling from another room	False rejects by condition
Several hours of TV, podcasts, music, kitchen and fan noise	False accepts per hour
Different household voices and accents	Whether one person is consistently missed
Device playing its own response	Whether echo re-triggers the detector

Run these tests with the real gain, speaker, enclosure, and mounting position. Also log only the event time, model score, and test label when possible. Keeping every false-trigger audio clip creates a privacy problem of its own.

Accuracy is a room-level property, so test distance, TV speech, music, noise, and the voices that will actually use the device.

Custom wake words take more work than the demo suggests

A good phrase is distinctive, comfortable to repeat, and not common in television or everyday conversation. For Vietnamese, I would test multiple regional pronunciations rather than assume one synthetic voice represents everyone.

The microWakeWord repository includes a basic training notebook and synthetic sample generation, but its own documentation warns that the notebook output is only a starting point. A usable model needs varied positive samples, large negative datasets, augmentation for noise and reverberation, and testing on speakers the training set did not include.

If the detector runs on Linux or a Raspberry Pi instead of a microcontroller, openWakeWord is another open-source path with ONNX/TFLite models and a Python runtime. I would not force the larger stack onto an ESP32 when microWakeWord already fits that constraint.

The privacy promise should be visible

For an always-listening product, I would make these behaviors part of the specification:

• Before detection, inference stays local, and audio is not stored or streamed.
• A visible LED turns on when command capture begins, not just when the device replies.
• A physical control cuts microphone power.
• The device exposes no hidden “listen live” endpoint.
• Event logs are optional and contain no raw audio by default.
• After timeout, cancellation, or command completion, capture stops and the device returns to the small local detector.

The wake-word model is only one component. The product earns trust when its electronics, firmware, indicators, logs, and documentation all tell the same story.

References and image sources

• GitHub - microWakeWord
• ESPHome - Micro Wake Word
• GitHub - ESPHome micro wake word models
• GitHub - openWakeWord
• Cover source: the blowup on Unsplash, customized for this article.
• Hardware source: Harrison Broadbent on Unsplash, customized for this article.
• Test-bench source: Kit on Unsplash, customized for this article.