Machines that can read the room.
MundiSense hears a voice, watches a face, and holds back when it isn't sure.
Watch it work.
Eight real cases, with the model's saved outputs.
Three models. One decision.
Fusion, calibration, and a gate that can say “not sure.”
Drag the gate
Below the line, it abstains instead of guessing.
Voice and face, fused.
Illustrative output for one utterance.
Every number keeps its denominator.
Sentiment transfer
Pearson r · 95% CI 0.756–0.829 · 976 of 1,034 clips
Coverage at a fixed threshold
Abstained cases stay in the F1 denominator.
Stream latency
3 h recorded replay · one RTX 5090 workstation
Measurement notes
Models. H3 = Q_AV (Qwen2.5-Omni-7B + LoRA) + A3 (Chinese HuBERT) + V2 (CLIP ViT-L/14), frozen fusion and calibration.
CH-SIMS v2. Sentiment labels, not seven emotions. Test n = 1,034; 976 with media. Pearson r 0.792, cluster-bootstrap 95% CI [0.756, 0.829]. All-clip r 0.761.
EmotionTalk. 1,929 test utterances, four speakers. Macro-F1 0.645 (no rejection); 0.624 with automatic segmentation at 87.66% coverage.
Latency. Utterance end → last event includes waiting for merged or split segments; it is not time-to-first-feedback. No claim for mobile, robot hardware or long-duration use.
Sense first. Then remember. Then express.
Sense
Hear voice, read faces, report doubt.
ReleasedRemember
Keep context, with consent.
PlannedExpress
A response layer each robot can make its own.
Planned