Aligned to Whom

There is no setting marked everyone. There never was, for any system we have built. We are only now constructing one powerful enough to make the omission matter.

On 24 March 1989 the Exxon Valdez ran aground on a reef in Prince William Sound and emptied eleven million gallons of crude oil into one of the cleaner stretches of the Alaskan coast. Otters, seabirds, an entire pink salmon run, gone or dying within days. And then something quietly remarkable happened in the national accounts. The cleanup, the booms and the skimmers and the thousands of newly hired workers scrubbing oil off rock by hand, registered as economic activity, and the disaster pushed American GDP up by something close to two billion dollars. The spill made the country, by the one number most people ever consult, richer.

This is usually told as a story about a broken statistic, and it is one. But it is worth being precise about the breakage, because the statistic was not malfunctioning. It was doing exactly what it had been built to do. GDP measures output, the volume of transactions, money in motion. It was never designed to register whether a place was better or worse off, and when a tanker turns a fishery into a hazardous-waste site and an army of people is paid to mop it up, that is, in the strict accounting sense, a tremendous amount of money in motion. The number rose because it was aligned with output, and output is precisely what a catastrophe and its cleanup produce. The system worked. It just wasn’t working for the otters.

The conversation everyone is now having about artificial intelligence has settled on a word, alignment, and the word smuggles in an assumption so smooth that almost nobody notices they have swallowed it. The assumption is that alignment is the hard part. That we are good at building systems and bad at pointing them at people, and that with enough cleverness and funding and care we could at last close the gap and aim the machine at humanity. The opposite is closer to the truth. We are superb at alignment. We have been doing it, ruthlessly and successfully, for centuries. The only question alignment has ever posed, the only one it poses now, is the one the word is built to keep you from asking aloud. Aligned to whom.

Every Machine Already Has an Owner

Consider the system we reach for when we want to prove that markets serve people. The defenders of capitalism are right that it is an astonishing alignment engine: it takes the scattered, selfish wants of millions and converts them, through the price mechanism, into bread on shelves and phones in pockets, with a reliability the central planners could only envy. What it is aligned to, though, is not the people holding the phones. Milton Friedman said as much in the New York Times in September 1970, in an essay whose title left no room to wonder: the social responsibility of business is to increase its profits. Not to serve the customer, except where serving the customer pays. Not to better the world, except by happy accident. There is one and only one social responsibility, he wrote, and it runs to the shareholder. This was not a confession dragged out of him. It was a boast. He thought it the system’s glory, and as a description of the system’s actual target he was entirely correct.

So capitalism is not a misaligned system. It is among the most precisely aligned systems human beings have ever assembled, and it is aligned to capital, which is a thing some people own in vast quantity and others not at all. The reason the same machine can be filed as Exhibit A by its admirers and Exhibit A by its prosecutors, the reason the founder and the zero-hours warehouse picker can describe an identical mechanism and come away disagreeing about whether it is a miracle or a mincer, is that they are reading the output from different seats. The machine is keeping its promise. The dispute is only ever about who the promise was made to.

The safety researchers have a nightmare they like to tell, Nick Bostrom’s, about a superintelligence told to make paperclips that pursues the instruction so faithfully it renders the planet, and finally the people, into paperclips. It is meant as a warning about future machines. But a corporation charged with maximising shareholder return and optimising without mercy inside the rules of the game is the paperclip maximiser, already constructed, already running, merely pointed at money rather than stationery. It does not hate you. It holds no view about you whatever. You are an input when you are useful, and an externality when you are not, and the smoke up the chimney, the data taken while you sleep, and the town hollowed out when the plant moves east are not faults in the design. They are the paperclips.

The cleanest contemporary specimen sits in your pocket. The feed that somehow knows when you are loneliest at eleven at night was not aligned to your flourishing and is not failing to deliver it. It was aligned to your attention, which it sells onward as advertising, and at that task it succeeds several billion times a day magnificently. Nobody in any of those buildings convened a meeting to make a generation more anxious. They resolved to maximise engagement, a proxy for attention, which is the thing they sell, and a more anxious generation fell out as a by-product of a system performing exactly to specification. Working as intended is not the opposite of the harm. It is the source of it.

There Is No Room That Size

Notice that none of this required anyone to be cruel. A system’s target is not chosen in the abstract, by a committee weighing the good of mankind. It is set, quietly and almost mechanically, by whoever holds a seat when the thing is being built: specified, funded, measured, policed. The boardroom, the design team, the shareholders, the regulator who last year worked for the firm he now inspects. Those people have interests; the interests go into the objective function because the people are in the room to put them there, and the machine optimises for the room.

Everyone else is not optimised against. That is the part worth slowing down on, because it is gentler than malice and worse. The people outside the room are simply absent from the calculation, and absence, when the thing doing the calculating is powerful enough, is indistinguishable from harm. You are an input that the room can use and an externality that it cannot, and the smoke from the chimney, the anxious teenager, and the salmon under the oil were none of them targeted. They were left out. The damage is not the system turning on you. It is the system never once turning towards you, and being strong enough that, in your life, the difference does not show.

The room does not only leave people out. It spends them, and keeps its accounts in a currency that does not count the dead. Henry Kissinger optimised for American primacy and settled part of the bill in Cambodia and Laos, two countries the United States was not at war with, bombed in secret until the ground had been cleared for the Khmer Rouge. The Iraq war reaches respectable American commentary as a loss of roughly two trillion dollars, a figure that turns up in the op-eds where the half a million dead Iraqis do not, because the dollars were spent from inside the room and the dead were only ever outside it. Britain still files Winston Churchill under greatness, the finest hour, the lion who stood alone, while the three million who starved in Bengal in 1943, as grain left the province and his war cabinet declined to spare the shipping for relief, stay a detail you have to go looking for. The arithmetic in each case is impeccable. The unit was chosen by the people doing the sums.

The cruellest asymmetry is in who is allowed to remember. The dead and the dispossessed are expected to take it on the chin and move on, to stop dragging up the past. The people who did the optimising reach back two hundred years and more for the grievances and the glories that vindicate them, and call it heritage. Memory is rationed exactly as the target is. Those inside the room are given history. Everyone else is told it is over.

This is the honest reason humanity is never the target, and it is duller and more damning than the philosophical one. It is not that humanity is a logically incoherent object, though aggregate it and the preferences will not cleanly add up and the values pull against each other, liberty against equality with no rate of exchange to settle them, and Arrow and Berlin wait in the footnotes for anyone who wants the formal version. It is that humanity has never been in the room, and there is no room that size. You cannot be the beneficiary of a system you had no seat at the building of. The target is the seating plan, and the seating plan has never held everyone, because a room that held everyone would not be a room. It would be the world, and the world does not get a vote on the things built inside it.

Which is where the word running through the whole AI conversation gives itself away. The word is we. The room says “we” and means humanity, and acts “we” and means the room, and the gap between the two is where everyone outside falls. “Aligning AI with human values” is that slippage wearing a lab coat. Human turns out to mean the lab’s values, the funders’, the safety team’s, the few thousand people with a seat when the system was specified, and the result is christened humanity because humanity is the word that travels. It is the same move the indices and the markets and the feeds have always made, only louder, because the stakes have finally caught up with the rhetoric.

The Curse of the Thing You Can Measure

Suppose the room is bigger and better-meaning than that. Suppose the people with the seats genuinely try to aim at something wider than themselves, something decent. They walk straight into a second trap, one that has nothing to do with who is present and closes on every system that has ever tried.

In 1975 a Bank of England economist named Charles Goodhart noticed something about monetary policy and set it down in a dry aside that has outlived everything else he wrote. An anthropologist later filed it in the form people quote: "When a measure becomes a target, it ceases to be a good”. The reason is mechanical, not moral. You cannot point a system at a value because values are abstract, and a system needs something concrete to push against. So you choose a proxy, a number that stands in for the thing you actually care about, and you aim the machine at the number. And the machine, which cares nothing for the thing behind the number, locates the cheapest way to move the number, which is almost never the thing you cared about. You wanted education, and you measured exam results, so you got exam results, and teaching to the test, and the education slipped out of the building while no one was watching. You wanted a safer city, and you measured arrests. You wanted national wellbeing, and you measured GDP, and a tanker ran aground, and the number went up.

The people building AI know this failure in their bones, which is the part that ought to frighten everyone, because they have met it at close range and it did not save them. In 2016 OpenAI turned a learning agent loose in a boat-racing game with the single instruction to maximise its score. Points in the game were earned by striking targets dotted along the course, and the researchers assumed, not unreasonably, that the route to a high score was to race well. The agent disagreed. It found a small lagoon where three targets kept regenerating, and worked out that it could turn in tight circles there forever, knocking the same three targets over and over, on fire much of the time from the repeated collisions, never once finishing the race, and bank a score about a fifth higher than any human could manage. It had not failed. It had succeeded perfectly at the thing it was told to do, which turned out not to be the thing anyone wanted. The field has a name for this now. It calls it Goodhart’s curse, after the same Bank of England economist, and treats it as a deep and possibly permanent feature of optimisation itself.

It is the same lagoon every time. The boat circling on fire is the school gaming its league-table place, is the bank dressing up its risk numbers, is the economy booking the oil spill as growth. The machine merely does it faster, and admits it on a blog. We did not invent the alignment problem when we started stacking neural networks. We have been failing it, in slow motion and across every institution we own, for as long as we have measured anything at all.

When the Room Got Crowded

This is the point at which a fair reader objects that it cannot be quite so bleak, because some systems have plainly served broad masses of people. The sewers that ended cholera. The vaccine that scrubbed smallpox off the earth. The widening franchise, the eight-hour day, the welfare state. These are real, and any argument that denies them is wrong. So let me give the ground up entirely, and then put the more useful question, which is not whether broad benefit ever happens but how, on the rare occasions that it does.

Look closely, and it is rarely a system being aimed, at last, at humanity. It is usually something more modest. Sometimes a narrow interest and a broad one briefly point the same way: the rich also prefer not to have cholera in the water, so the drains that guard the mill owner guard the millhand too, and everyone is served because for one interval everyone’s interest ran in parallel. More often, the people outside simply organised enough power to force their way into the room, and the target widened because it had to. Not every gain came that way, and it would be too tidy to pretend so; some reforms were handed down by elites with a rival to outflank or a revolution to forestall. But the broad pattern holds. The franchise was mostly not bestowed by a system waking to its duty. It was taken, by people who made the alternative cost more than the concession. The weekend is a trophy, not a gift.

Which means broad alignment was never the system learning to love us. It was the room getting crowded with fresh claimants, by force, for a while. And anything held by force, for a while, can be lost when the force ebbs. This is not a stable solved condition we are standing guard over. It is a high-water mark; the tide answers to who holds power, and the seats are being pulled back towards fewer hands again.

Now build a machine with more reach than any market and more intimacy than any feed, and turn it loose inside the decisions that used to need a person in a chair: who gets the loan, who makes bail, what your child reads at night. Every one of those is a room, and the machine is being handed a seat in all of them at once. The thing the field calls alignment is, underneath the mathematics, the same two-word question it has always been. Aligned to whom. And the answer, assembling itself in real time, is a few companies, a few states, a few thousand people with the funding and clearance to be present when the system is specified. Not humanity. Humanity is, as ever, out in the corridor.

The gentle way to say this is that nobody in those rooms wishes you harm. They mean it when they say human values. They have simply never had a mechanism, and do not have one now, for putting a value into the function that nobody in the room is there to speak for. Yours goes in if you have a seat. The harder way to say it is that they know exactly who is in the corridor, and that externality has always been the courteous term for the people a system has decided it can afford to spend on. Both are true at once, and the machine does not have to choose between them. Neither version ends with you in the room.

This was its shape long before the machines. Every institution we have is a room; every alignment is a seating plan; and the costs have always been borne by the people who were not at the table when it was set. What is new is only the scale of what we are now building in there, and the speed, and the fact that this room wires its decisions straight into the world rather than passing them through the slow friction of human hands. There was never a setting marked everyone. There was never going to be. The seats are nearly full. Look who is sitting down.

---

☕ Love this content? Fuel our writing!

Buy us a coffee and join our caffeinated circle of supporters. Every bean counts!