You say: «Book me a flight to Lisbon, sometime next week, not too early in the morning.»
The agent works. A few seconds. Then the confirmation: Wednesday, 11:40, direct flight, paid.
You lean back. Done. Good decision.
Was it a decision?
Between your sentence and the booked flight lies a chain of choices you never saw. Which airline. Which price counted as «not too expensive». What «not too early» means — seven, nine, eleven. Whether a layover was allowed. Whether the cancellation terms mattered. You answered none of these questions. Someone answered them in your name and laid the result in front of you like a choice of your own.
And it feels like your choice.
* * *
Ich baue solche Systeme. Seit zehn Jahren gestalte ich Interfaces, in denen Menschen Geld bewegen, Kredite aufnehmen, Verträge abschliessen. Ein guter Teil meiner Arbeit besteht darin, den Moment zu formen, in dem jemand auf «Bestätigen» drückt und denkt: Das habe ich entschieden.
Often they didn’t decide. They agreed. That’s not the same thing — but from the inside it feels identical.
* * *
1975. Ellen Langer, a psychologist at Harvard, sells lottery tickets to office workers. One dollar a ticket. One half gets to pick their ticket. The other is handed one.
Just before the draw, Langer asks everyone whether they’d sell their ticket back. And for how much.
Those who had been handed a ticket asked for just under two dollars on average. Those who had chosen their own asked for almost nine.
Same odds. Same draw. The only difference: one group had taken the ticket in hand and chosen it themselves. That single act — irrelevant to the outcome, a lottery knows no skill — produced the feeling of greater influence.
Langer called it the Illusion of Control. The feeling of control doesn’t come from control. It comes from involvement.
She found several conditions under which the illusion reliably appears. One is free choice. Another is active involvement. When you give an agent a task, both are met: you formulated it, you triggered it. You were involved. So it feels like control.
Whether you had control is a different question.
* * *
There’s a second experiment that goes deeper. Daniel Wegner, also Harvard, late nineties.
Two people jointly hold a computer mouse that moves a cursor across a screen full of small images. One of the two is in on it. Through headphones, the test subject hears words, supposedly just a distraction. In truth, the headphones sometimes name the exact object the cursor is about to stop on — because the insider is steering it there.
If the subject hears the matching word just beforehand, she’s convinced: I stopped. I wanted to go there.
She didn’t stop. The other person steered. But the thought came before the result, and the thought matched the result. That’s enough.
Wegner called it the Illusion of Conscious Will. We experience an action as our own when the thought comes before the deed, fits the deed, and no other cause is visible.
With an agent, all of it comes together. You think the flight. You speak it. The flight appears. No other cause is visible — the whole apparatus of searching, filtering, weighing runs where you can’t see it.
You didn’t book it. You said a sentence. Your brain logs it as an act.
I call this feeling the borrowed will. The sense of having acted, resting on an execution that wasn’t yours.
* * *
So far this sounds harmless. A flight is booked, so what.
Now raise the stakes.
June 1, 2009. Air France 447, from Rio to Paris, an Airbus over the Atlantic. The airspeed sensors ice over. The autopilot does what it’s supposed to do when it no longer trusts its data: it switches off and hands the aircraft back to the pilots.
And the pilots, used for years to the automation flying, are handed an aircraft they can no longer control in that moment. A few minutes later the plane hits the water. 228 people.
The sensors were clear again shortly after. The aircraft was flyable the whole time. What was missing was the humans’ ability to take over — because they’d gone too long without having to.
The research knows the pattern. Parasuraman and Riley, 1997: the more reliable a system, the less we control it. Not despite its reliability. Because of it. Someone who has seen the autopilot be right a hundred times no longer checks on the hundred-and-first.
The most dangerous agent isn’t the bad one. It’s the one that’s almost always right.
And it needn’t be an aircraft. August 1, 2012, Knight Capital, a stock trader in New Jersey. Faulty software goes live. Over the next forty-five minutes the system buys and sells shares in a loop no one wanted. Loss: around 440 million dollars. In three quarters of an hour.
No one at Knight had placed that order. Someone had built a system that acted in their name — faster than a human could intervene. The very speed it was built for made it uncontrollable.
You want your agent to be fast. That’s the whole point. But speed and control pull in different directions, and you can’t maximize both.
* * *
Here’s the place where I have to correct myself.
The problem isn’t that agents make mistakes. Machines have always made mistakes, and so have people. If that were all, the fix would be simple: better agents.
Better agents don’t solve the problem. That’s the uncomfortable part. Because the feeling of control you have while booking is independent of how good the agent is. It arises the moment you speak the sentence — through involvement, not through oversight.
Even the perfect agent would trigger the same feeling as the bad one. You’d feel equally secure — without being able to check whether the security is warranted.
A year ago I wrote about sham understanding. About the feeling of having understood something without having understood it. That was the precondition. Whoever doesn’t understand what an agent does can’t judge whether it’s doing it right.
Now comes the next layer. You don’t understand what happens in the gap between your sentence and the result — and you feel that you’re steering anyway.
Fake news is the problem everyone talks about. Sham understanding is the problem no one notices. Sham control — the problem that’s just emerging.
* * *
Last week I dictated an email to the assistant, addressed to a designer. Tell her I need the mockups by Friday. The draft appeared — polite, precise. Send. I didn’t press. My eye caught on the name. She had written to me the day before that she was sick. The agent didn’t know that.
Had I pressed, I’d have been the careless one. That was cheap. An embarrassing email, no damage. But the pause wasn’t a method. I got lucky that my eye caught on a word.
An embarrassing email you undo like a wrong flight. A wire transfer, a termination, you don’t. There the gap between your sentence and the result is no longer convenience. There it’s the risk.
What helps is no grand rule. A small question, just before it gets expensive: What would be the worst mistake I couldn’t take back? And would I even have a chance to spot it before it happens?
The agent booked the flight. You said a sentence.
Knowing the difference is the whole work.