The journalist Karen Hao calls the big AI companies «empires». In her reading, they amass data, land, energy and labour on a historically unusual scale and withdraw these resources from the communities they come from. One need not follow her on the scale to take the core: the logic behind it is not confined to AI. The same mechanism runs, far more quietly, in the ordinary digital product.
The logic is extraction. Value is drawn from the user without return and without consent: attention, data, behaviour. It needs no data centres; the pattern sits in the detail. In concrete terms: engagement placed above the user’s interest. Defaults that work for the provider and disguise themselves as convenience. Nudges with no way out. Friction deliberately set between the user and cancellation. Cross-sell that poses as a recommendation. On its own each pattern looks harmless. Together they do what Hao accuses the empires of, only on a smaller scale.
There is rarely ill intent behind this. Most of these patterns are a by-product of a single, barely questioned optimization toward the next funnel step. That explains their persistence.
Hao calls for regulation, and rightly so. At one point, though, her argument stops. She concedes that Europe regulates but has no vision of its own: no model for what should take the empires’ place. That is the decisive gap. Resistance is not design. Whoever only regulates draws the limits of the permissible and leaves the building to those who are building anyway. Bans create no alternative, at most a tamer variant of the same system. The more productive question is therefore not how to ban extraction, but what to put in its place. There is a design answer to that.
* * *
The most common objection to ethical design runs: less steering, weaker conversion. It rests on an error. Choice architecture begins with an insight from the discipline itself: there is no neutral option. Every screen, every default, every ordering decides for the user before the user decides. Steering happens regardless; the only open question is where to, and in whose favour. Ethical design therefore does not forgo steering. It makes steering transparent and reversible. That becomes concrete at three points.
The default is among the most powerful decisions in a product, because the fewest users actively change it. It turns extractive when it works for the provider and disguises itself as convenience: the pre-ticked insurance, the silent opt-in. The test is simple. Would the user choose this default if they saw the reasoning behind it? Where the answer is no, the product works against them. This is measurable at once, through the correction rate: how many change the default once they understand it. It takes no year; it shows in the session.
Equally telling is the asymmetry of exit costs. One click to sign up, five screens to cancel. Entry is designed, exit is friction. Ethical architecture makes the two equal: the way out is as clear as the way in. This sounds like forfeited retention, but it actually holds the more valuable bond. Whoever keeps users only because they cannot leave is holding dead conversion. This too can be checked today, with an audit of the click paths: steps to sign up against steps to cancel.
What weighs most is who is left with the value in the end. The same nudge serves both directions. A nudge toward saving creates value the user keeps; a nudge toward an overdraft creates value the bank skims off. Same mechanics, opposite ethics. The decisive factor is solely who is left with the gain. Ethical design chooses the variant in which some of it stays with the user. This criterion is not new. The Value-Irritant Matrix by Price and Jaffe, well established in Swiss banking, locates the worthwhile dialogue precisely where bank and customer benefit at the same time. It is just usually read as a cost question, rarely as an ethical one. That is exactly where the blank space lies.
Then comes the objection that cuts deepest: who defines the user’s interest? If the bank does, that is no counter-model but merely the next presumption, called care. The answer lies in the first principle. The yardstick is not what the bank deems right, but what the user would confirm if they saw the reasoning. Transparency and an open exit do not patronize; they give the user the means to override any steering. An architecture that ships with its own correction protects the user’s autonomy rather than skimming it off.
How sharp this yardstick is shows in the current debate around agentic AI in customer service. It promises to fully automate routine contacts and, in the same dialogue, to place cross- and upselling prompts, embedded so deftly that the customer does not perceive them as a nuisance. By the measure of irritation, that is a success. By the transparency test, it is the opposite. A sale the user does not notice passes the one test and fails the other. Non-irritation is not consent, and it is precisely this confusion that separates orchestration from extraction.
With that the common myth collapses. Ethics here is not the soft option but the more demanding one. The operational proof is short-loop and falsifiable: correction rate, exit symmetry, complaints on the cancellation path, all measurable within the quarter rather than after one’s tenure. The lifetime of the customer relationship is the thesis behind it, not the KPI by which you steer it. Extraction optimizes the moment, choice architecture the relationship. The difference shows up sooner than most assume.
* * *
The strategic point does not hold for the whole bank, only for a particular part of its business. Start with the objection that could topple the entire lever. In retail banking, supposedly, it is not trust that holds the relationship but inertia: switching costs, account integration, standing orders. The cancellation rate barely responds to a pre-ticked box. That is true, but only for the existing book. Inertia holds what is already there.
But it holds only that. The dormant account, not primary-bank status. The low-margin base, not the valuable growth: share of wallet, the mortgage mandate and the investment portfolio do move; they hang on offer and trust. And inertia does not protect the next generation at all. Whoever chooses their first primary bank at 25 carries hardly any switching costs; that decision turns almost entirely on trust and experience. This makes the calculation precise. Extraction is not expensive everywhere. On the existing book it is cheap, because the customer stays anyway. It turns expensive exactly where the bank has to grow: in acquisition, in share of wallet, in the generation still choosing. Inertia carries the book, trust contests the growth. That is the narrow version of the argument, and the more defensible one.
That leaves the uncomfortable part, which should not be smoothed away. An unticked box can be copied overnight; there is no advantage in it. And in a commodity market one may lose share before any trust premium sets in, if it sets in at all. Whoever conceals this is selling ethics as a free lunch. What is hard to copy is not the surface but the calculation behind it. A bank that measures relationship quality instead of funnel conversion and pays its teams accordingly builds something a competitor does not adopt overnight, because it cannibalizes that competitor’s running revenue logic. The real moat lies in the operating model, not in the screen. Even the sober optimization literature concedes as much: without a target operating model built for it, even the pure efficiency potential goes unrealized, let alone the matter of trust. The lead is real, but slow. Whoever builds it does so not out of virtue but out of a different discount rate: less weight on the next quarter, more on the relationship that still carries margin in ten years. A temporal argument, then, not a moral one. And vulnerable the moment the bank’s time horizon is too short.
Hao’s most important sentence is not the one about the empires, but the quieter one beside it. In paraphrase: it is not enough to refuse what we do not want; we must also say what we build instead. For the financial industry the answer is more concrete than it sounds. We build products that steer the user, which is unavoidable, but in their favour and traceable for them. We measure operational success short-loop and honestly, and we treat the lifetime of the relationship as a goal, not as an excuse for absent numbers. And we admit that this path changes little on the existing book. It counts where inertia no longer protects: in growth.
The empires optimize the moment; that is their strength and their limit at once. Whoever bets on trust builds nothing sacred, but makes a sober wager on the longer time horizon. It is not without risk. But it aims at the thing that is hardest to extract.
* * *
Read more:
