Silent speech, from muscle to text.

SETU-EMG · Technical summary

8 sEMG channels · 1,000 Hz · 63,817,766 parameters · EMG-only at inference

System

SETU-EMG is an end-to-end neural system for decoding deliberately mouthed silent speech directly from surface electromyography (sEMG), enabling sentence-level text decoding without acoustic input at inference time. The system operates on eight channels of articulatory muscle activity sampled at 1,000 Hz and learns a direct mapping from multichannel physiological signals to character sequences. The neural network comprises 63,817,766 trainable parameters and was trained for 200 epochs on a single NVIDIA H100 GPU, with model selection based on silent-speech validation word error rate (WER).

Training

To preserve phonetic information across modalities, training combines silent and voiced EMG with speech audio. Cross-modal representation learning and latent dynamic time warping establish temporal correspondence between EMG and acoustic latent sequences despite timing differences, while connectionist temporal classification (CTC) provides sequence-level supervision for character recognition without requiring manually aligned EMG-to-character labels.

Inference and results

At inference time, the recognition pathway is strictly EMG-only. The model receives processed electrophysiological waveforms and has no access to synchronized audio, reference transcripts, target phoneme sequences, filenames, or reference-dependent metadata. Input signals pass through a deterministic preprocessing pipeline consisting of filtering, resampling, temporal framing, cropping, and amplitude transformation before neural inference. The resulting character probabilities can be decoded directly via greedy CTC or combined with a fixed statistical language model through beam search. On the official held-out test partition of the public silent-speech benchmark (comprising 8 facial EMG channel data of 50+ hours across people), 4-gram beam-search decoding with a beam width of 5,000 achieves a 16.16% WER and a 10.91% character error rate (CER), reducing total word errors from 586 to 266 and increasing exact-transcript matches from 7 to 17 out of 99 utterances, while greedy CTC decoding produces 35.60% WER and 14.35% CER on the same test partition.

Controls

This primary result was obtained from a single neural checkpoint without model ensembling, generative text correction, reference-conditioned reranking, confidence-based rejection, manual editing, or sample exclusion; all test utterances received predictions. A separate validation experiment examined whether higher-level linguistic inference could recover additional information from the model's decoding distribution: ten candidate transcriptions were generated for each of 100 validation utterances (1,939 reference words), after which a frozen language model selected among the candidates without reference transcript access, reducing validation WER from 20.42% to 17.77% and increasing exact matches from 11 to 16. Implementation-level sanity checks were conducted using controlled non-speech inputs: ten zero-valued inputs produced no non-empty outputs, and one out of ten amplitude-matched noise waveforms produced a non-empty transcription.

Latency and evaluation note

Runtime profiling of neural inference followed by greedy and beam-search decoding on an H100 produced a median latency of 2.14 seconds and a 95th-percentile latency of 9.61 seconds. The official benchmark test partition was encountered during broader development and is reported as a fixed benchmark evaluation rather than a previously inaccessible cohort; all model selection and training decisions remained grounded in the designated training and validation partitions.

Personalization

Separately from the benchmark system, SETU-EMG includes an experimental user-personalization capability built around a lightweight 24,576-parameter residual feature adapter while keeping the underlying MONA-derived backbone frozen. The calibration workflow supports training the adapter from labelled, compatible eight-channel recordings, saving and reloading personalized checkpoints, running EMG-only inference with the adapted model, and reverting to the unadapted base model without retraining the backbone. This personalization module was not used to obtain the reported 16.16% test WER. Its software workflow has been implemented and validated, while quantitative improvements for previously unseen users, alternative electrode placements, and the proposed near-ear wearable configuration remain to be demonstrated experimentally.

Next

The next stage extends this decoding framework to proprietary wearable recordings collected across multiple users, sessions, and controlled sensor placements to enable direct measurement of cross-user adaptation, calibration requirements, longitudinal stability, physiological false activations, channel-count sensitivity, and robustness to electrode displacement.