Modes Came Back: Voice Assistants and Hidden State

IDEAS · MODES CAME BACK: VOICE ASSISTANTS AND HIDDEN STATE

November 8, 2018

A mode, in Jef Raskin’s definition, is any state in which the same user action produces a different result than it would in another state. He spent much of his career arguing that modes are a primary, predictable, and avoidable cause of user error: the user performs an action based on a reasonable belief about the system’s state, and the system does something other than what the user intended. The user is not at fault. The interface failed by allowing a false belief about state to persist.

For most of the graphical-interface era, the mode problem was a problem of the visible: a dialog box that captured keystrokes, a Caps Lock toggle, an insert-versus-overtype switch buried in a word processor. The state was at least nominally inspectable, even when the inspection was inadequate. The arrival of voice assistants — Apple’s Siri in 2011, Amazon’s Alexa in 2014, Google Assistant in 2016 — has reintroduced the mode problem in a form Raskin did not live to analyze. The state is now almost entirely hidden, and the consequences of misjudging it are unusually opaque.

The Wake Word Is a Mode Boundary

A voice assistant spends most of its life in a listening-but-not-acting state. Say the wake word — “Alexa,” “Hey Siri,” “OK Google” — and it crosses into a different state, one in which your subsequent speech is captured, interpreted, and acted upon. The same utterance produces wildly different results depending on which side of that boundary you are on. Spoken before the wake word, “play the news” is ignored. Spoken after, it triggers an action.

This is a mode in the strictest Raskin sense. And it is a mode with no reliable continuous indicator. A small light ring may glow, or a chime may sound, but the user frequently cannot see the device, is in another room, or is mid-sentence and not attending to the indicator. The result is the now-familiar comedy of the assistant either ignoring a command the user believed it heard, or — more unsettlingly — acting on speech the user did not intend as a command. Both are mode errors. The user held a false belief about whether the device was listening.

Hidden State Stacked on Hidden State

Raskin’s canonical complaint about Caps Lock was that the mode indicator — an LED on the keyboard — was in the wrong place. Users look at their documents, not at the indicator light. The voice-assistant case is worse, because there is often no place to look at all. The state of a conversational agent is distributed across at least three invisible layers.

First, there is the listening state already described. Second, there is conversational context: many assistants attempt to carry a thread across turns, so that “what about tomorrow?” is interpreted relative to a previous query about the weather. The user cannot see what context the system is holding, and the same follow-up question yields different answers depending on a history the user cannot inspect. Third, there is the matter of which device or which account is responding. In a household with several assistant-enabled devices, an ambiguous command can be answered by an unexpected unit, drawing on a different user profile.

Each of these is a layer of state. None of them is visible. Stacked together, they make the assistant’s behavior unpredictable in precisely the way Raskin warned against: the user forms a model of what the system will do, acts on that model, and is surprised. He would have recognized the pattern immediately, because it is the same pattern he identified in modal software in 1979. The technology is new; the failure is old.

Why This Is Not Merely Inconvenient

It is tempting to treat assistant mode errors as trivial — the device misheard, the user repeats the command, life continues. Raskin’s argument against treating mode errors lightly was that the aggregate cost is large even when each individual incident is small, and that the errors are uniquely demoralizing because they feel like the user’s fault when they are the interface’s fault.

With voice assistants the stakes are slightly different, and in one respect higher. Because the listening boundary is invisible, the mode error has a privacy dimension that keyboard modes never did. A device that acts when the user believed it was dormant, or that captures speech the user did not intend to direct at it, is not just an annoyance — it is a breach of the user’s model of when they are and are not being heard. The 2018 discourse around smart speakers and accidental recordings is, at bottom, a discourse about an unobservable mode boundary. Raskin’s framework explains why the problem is structural rather than incidental.

The Quasimode Was the Answer All Along

Raskin’s proposed remedy for modes was the quasimode: a state that requires continuous user action to maintain and therefore cannot be silently forgotten. The shift key is the canonical example. You hold it to get capitals; you cannot forget you are holding it, because forgetting means releasing it, which ends the quasimode. The physical demand of the action is the state indicator.

Push-to-talk is the voice-assistant equivalent, and it is instructive that the most error-resistant voice interfaces are the ones built around it. A walkie-talkie has never had a wake-word problem, because transmitting requires holding a button. The push-to-talk button on a car’s steering wheel, or the press-and-hold invocation of an assistant on a phone, are quasimodes in Raskin’s exact sense: the listening state exists only while the user is doing something to sustain it. There is no ambiguity about whether the system is listening, because listening and holding are the same act.

The industry has largely chosen the opposite design — always-listening, wake-word-triggered — for reasons of convenience and ambient availability. Those are real benefits. But they are benefits purchased at the cost of an invisible mode boundary, and Raskin’s analysis predicts exactly the class of errors that result. A press-and-hold assistant is less magical and more honest about its state.

The Argument Still Holds

The voice assistant is among the most genuinely novel interface categories to emerge since Raskin wrote The Humane Interface in 2000. It is striking, then, how completely his decades-old analysis of modes applies to it. The mechanism is new — speech recognition, natural-language parsing, cloud inference — but the design failure is the one he named: a state the user cannot observe, a belief about that state that turns out to be false, and an action the system executes faithfully against the user’s intent.

Raskin held that every mode error is a design failure rather than a user failure, and that the interface could almost always have been built to make the error impossible. The voice assistant could indicate its state continuously and unambiguously, or could require a sustained action to listen, and largely does neither. Thirty-nine years after he first made the argument, the most advanced interfaces being shipped have rediscovered the mode problem and have not yet rediscovered his solution.