Why Any Rational Agent Must Maximize Utility

What does it mean to be rational, or to be a rational agent? If you look this question up on Google, the answer you’ll likely get is something along the lines of, “A rational agent is an agent that analyzes a situation, considers the range of actions available to it, and chooses whichever action leads to the best result.” A more formal definition of a rational agent might go something like this: A rational agent is an agent who, within any given situation:

1. Understands the range of actions it can take.

2. Assigns accurate probabilities to what will happen in the world, contingent on it taking a particular action.

3. Using some utility function, assigns a real-numbered value to each possible scenario.

4. Calculates the expected utility of each action that it could take. (That is, for each possible action, it adds up the utility of each scenario times the odds that that scenario will happen, conditional on taking the action.)

5. Takes whichever action maximizes expected utility.

Admittedly, this definition is pretty dense, and it’s not immediately clear why this is the definition of a rational agent. So I’ll try to break it down into pieces. Point 1, understanding its range of possible actions, seems pretty intuitive. It makes sense that for an agent to be called “rational”, it would have to know enough about itself and its environment to understand what actions are available for it to take. Part 2, assigning probabilities, also strikes me as intuitive. In a world of limited / imperfect information, it’s impossible to know with certainty the consequences of your actions. So, in order to better anticipate all possible outcomes, it helps to think in terms of probabilities. (For a more detailed defense of why rational agents should use probabilities, see Dutch Book Arguments or Joe Carlsmith’s essay on the subject.)

But what about Points 3-5? Why should a rational agent have a utility function? Why do we calculate expected value by adding probabilities together? And why should a rational agent bother to maximize expected value? The reasons for doing this were proven in the 1940s by the inventors of game theory, John von Neumann and Oskar Morgenstern. Their namesake von Neumann-Morgenstern (VNM) utility theorem shows that for any agent following four basic axioms, that agent must be maximizing some utility function. In this article, I’ll give a basic rundown of the VNM utility theorem and why it is relevant.

But first, a caveat: There’s a subtle but crucial difference in the way that economists / game theorists use the term utility, and the way that utilitarian philosophers use the term utility. To a utilitarian, utility is typically defined as pleasure minus suffering. But to an economist / game theorist, utility could mean almost any set of preferences that an agent has. For instance, we can imagine somebody with the ability to either buy 1,000 back rubs from trained masseuses, or buy a classic painting by Monet. This person knows that they would receive more pleasure from the back rubs, but they choose to buy the painting anyway, because they care more about art / status than raw pleasure. To an economist, this is a perfectly valid set of preferences. Moreover, when a utilitarian philosopher talks about maximizing utility, they typically mean maximizing the utility of all conscious beings across the entire universe. There’s a normative implication to it (a moral agent should maximize total utility). Whereas, when an economist / game theorist talks about maximizing utility, they typically mean maximizing personal utility, regardless of how it affects others. Here, there’s no normative implication (a rational agent would maximize its own utility, but the economist makes no claims about whether an agent should try to be rational). The VNM utility theorem uses the term “utility” in the way an economist would, not in the way an ethicist would.

But okay, how does the VNM utility theorem actually work? I’ll attempt to explain it in a way that is mathematically rigorous, while also being understandable to somebody without much background in advanced math.

Definitions

Let X be the set of anything you might have preferences for. X could be something simple like “the set of fruits”, or something more complicated like “the set of all possible worlds”. It doesn’t really matter what X represents; the proof works no matter what.

A lottery is some way of assigning probabilities to the elements of X, where the sum of these probabilities is 100%. For instance, let’s imagine that X contains three elements: apples, bananas, and cantaloupes. One possible lottery would be “30% chance of apples, 20% chance of bananas, 50% chance of cantaloupes.” It’s possible for a lottery to be degenerate, meaning it only puts non-zero odds on a single element. For instance, the degenerate apple lottery would be “100% chance of apples, 0% chance of anything else.”

If we take two lotteries p and q, and some number α between 0% and 100%, we can form a mixed lottery (p, q, α). This means a lottery with an α chance of p, and a (1-α) chance of q. For instance, if p represents “100% chance of apples, 0% chance of anything else,” and q represents “100% chance of bananas, 0% chance of anything else,” then (p, q, 30%) would represent, “30% chance of apples, 70% chance of bananas, 0% chance of anything else.” It can be shown that the mixed lottery of any two lotteries is itself a lottery.

For any lotteries p and q, we’ll say that p ≻ q means “I prefer p over q”, p ~ q means “I am indifferent between p and q”, and p ≽ q means “I either prefer p over q, or I’m indifferent between them”.

The Four VNM Axioms

Here is where we formally define what it means to be a “rational agent”. According to this theorem, a rational agent must display at least four qualities. These are not necessarily the only qualities that a rational agent must display, but they are the four qualities that are sufficient to prove the theorem.

Axiom 1: Completeness

• For all lotteries p and q, either p ≻ q, p ≺ q, or p ~ q.

In other words, a rational agent must make a choice between outcomes (even if that choice is “I’m indifferent”). You cannot simply throw your hands up and refuse to decide.

Axiom 2: Transitivity

• For all lotteries p, q, and r, if p ≻ q and q ≻ r, then p ≻ r.

(Read: If you prefer p over q and q over r, then you must prefer p over r.)

In other words, a rational agent’s preferences can’t be circular. You can’t say, “I prefer apples over bananas, bananas over cantaloupes, and cantaloupes over apples.” Why can’t a rational agent say this? Well, among other reasons, it opens you up to being “money-pumped”. Since you prefer apples over bananas, you might be willing to pay $1 to trade your banana for my apple. Then, since you prefer cantaloupes over apples, you might be willing to pay $1 to trade your newly-acquired apple for my cantaloupe. Finally, since you prefer bananas over cantaloupes, you might be willing to pay $1 to trade your newly-acquired cantaloupe for your old banana. But then look at what’s just happened! A bunch of fruit has changed hands, you’ve swapped a banana for a banana, and now you’re $3 poorer than when you started. That doesn’t seem like something a rational agent would do.

Axiom 3: Independence

• For all lotteries p, q, and r, and any probability α, if p ≽ q then (p, r, α) ≽ (q, r, α).

Put another way: If you prefer p over q, then you should prefer some chance of p over an equal chance of q. For instance, if you prefer apples over bananas, then you should also prefer (a 40% chance of apples and a 60% chance of oranges) over (a 40% chance of bananas and a 60% chance of oranges). Again, this seems like the kind of property that a rational agent would have.

Axiom 4: Continuity

• For all lotteries p, q, and r, if p ≽ q ≽ r, then there exists some probability α such that q ~ (p, r, α).

In other words, if you like p better than q and q better than r, then there ought to be some probability α such that you’d be indifferent between a 100% chance of q and (an α chance of p plus a (1 - α) chance of r). For instance, maybe you like pomegranates slightly better than quandongs, and you like both pomegranates and quandongs much more than raspberries. Then, you might be indifferent between a 100% chance of quandongs and (a 95% chance of pomegranates plus a 5% chance of raspberries). Admittedly, this is probably the most arbitrary and hardest to justify of the four axioms.

---

Maybe you’re skeptical of these four axioms. Maybe you think these are totally arbitrary conditions that have no bearing on what it means to be a “rational agent”. I won’t defend the four axioms here, though if you’re curious, Joe Carlsmith offers some arguments in their defense in his essay on VNM.

But for now, let’s take it as a given that any rational agent has preferences which are complete, transitive, independent, and continuous. On this basis alone, we can prove that such an agent is acting in accordance with a utility function.

The Theorem

(Note that for simplicity, I’ll be glossing over some of the more technical details. But for those who want the complete, formal version, I recommend this paper by Jingyuan Li.)

We wish to build a function U that takes in any lottery over X, and outputs a real number between 0 and 1. We’ll start by considering just the degenerate lotteries — the lotteries that put 100% odds on one element, and 0% odds on all other elements. First, pick out the most and least preferred elements from X.¹ We’ll call the degenerate lottery for the most preferred element m, and we’ll call the degenerate lottery for the least preferred element d. Then, we’ll say that U(m) = 1, and U(d) = 0. For instance, maybe your favorite fruits are mangos, and your least favorite fruits are durians. So we’ll say that “100% chance of mangos” has a utility value of 1, and “100% chance of durians” has a utility value of 0.

Completeness and transitivity ensure that every other degenerate lottery p falls somewhere between m and d. Per our example above, no matter what fruit you choose, a 100% chance of that fruit is no better than a 100% chance of mangos, and no worse than a 100% chance of durians. Because m ≽ p ≽ d, by the continuity axiom, there must be some number α between 0 and 1 such that p ~ (m, d, α). So, let U(p) = α. For instance, let’s say that you’re indifferent between a 100% chance of pomelos and (an 80% chance of mangos plus a 20% chance of durians). Then, we would say that U(pomelos) = 0.8.

Then, we can note that every non-degenerate lottery is just a weighted sum of multiple degenerate lotteries. So, we’ll say that the utility of a lottery is the weighted sum of the utilities of its degenerate base parts.²³ For instance, let’s say that we’re evaluating the lottery q, which represents “30% chance of apples, 20% chance of bananas, 50% chance of cantaloupes.” Then, we can say: U(q) = 0.3 x U(apples) + 0.2 x U(bananas) + 0.5 x U(cantaloupes).

Now here’s the critical part: For any lotteries p and q, p ≻ q if and only if U(p) > U(q). In other words, our agent will prefer p over q if and only if p yields a higher number than q on the utility function. This means that for a rational agent, satisfying its preferences is synonymous with maximizing expected utility on some utility function.

Not only does our utility function U have the nice property that it matches our agent’s preferences exactly; it can be proven that U is the only possible function which does this.⁴

Implications

We should be precise about what the VNM utility theorem does and does not prove. The theorem does not prove that a rational agent should maximize utility (after all, mathematics is not about making normative claims about what agents ought to do). Rather, the theorem proves that any rational agent, so long as they are acting to fulfill their preferences, already is maximizing utility, according to some utility function.

Part of what makes this theorem so powerful is how few assumptions we had to make. We made no assumptions about the kind of agents who are making these decisions. The agents in question could be human beings, but they could just as easily be companies, government agencies, extraterrestrial aliens, or AI-powered robots. We also made no assumption about the kinds of choices being made. The agents could be choosing between different kinds of fruit, different romantic partners, different planets to inhabit — really, any set of elements for which it is possible to have a preference is fair game for being utility-maxxed. The agents in question need not be competent at fulfilling their preferences (the world is certainly full of incompetent agents). The agents in question need not be conscious of the fact that they have a utility function (it’s possible for an agent to maximize expected utility, while being completely unaware that that’s what it is doing). The only assumptions that the VNM utility theorem makes are the four starting axioms.

The reason why economists, game theorists, and rationalists are so obsessed with utility maximization is not (just) because we are nerds who want to put numbers on everything. Rather, it is because utility functions are a fundamental part of the universe, and they apply regardless of time, space, or species. Any rational agent that acts to fulfill its preferences, whether that agent realizes it or not, must be an expected utility maximizer.

1. One can reasonably ask: How do we know that such a most / least preferred element exists?

If we assume that X is finite, as many versions of the VNM theorem do, then existence is easy. Every fully ordered finite set has a greatest / least element, so existence is guaranteed by completeness and transitivity.

If we assume that X is allowed to be infinite, then the theorem can still work, but it will require some additional axioms. For alternative theories of expected utility based on different axioms, see McCarthy et al.

2. The reason we’re able to do this is because of the independence axiom. Because of independence, swapping any outcome in a lottery for something the agent is indifferent to leaves the whole lottery equally preferred, so we can replace each degenerate outcome with its indifferent (m, d, α) mixture until the lottery becomes a compound gamble over nothing but m and d. That compound reduces to a single lottery (m, d, α*) where α* is the weighted sum of the individual utilities, which means the weighted sum matches the value that the agent’s preferences already assign.

3. A simple sum is sufficient if we’re only dealing with lotteries that put non-zero odds on a finite number of elements. If we allow for lotteries that put non-zero odds on an infinite number of elements, then the concept can still work, but we’d have to replace summation with some form of integration.

4. More precisely, it is the only possible function, modulo positive affine transformations. This means that you can still scale U by a positive number, or you can add a real number to it, without altering the relative order of the lotteries. For instance, we can define an “enhanced” utility function U* which multiplies all the outputs of U by 100, so anything with a utility of 0.2 becomes 20, and anything with a utility of 0.65 becomes 65. But this change isn’t really meaningful; it gives us no new information, and it changes nothing about our ordinal preferences. Similarly, we could define a function U** which subtracts 0.5 from all the outputs of U, so anything with a utility of 0.2 becomes -0.3, and anything with a utility of 0.65 becomes 0.15. Again, this change gives us no new information and changes nothing about our ordinal preferences. So, even though there are ways we can modify U while respecting our agent’s preferences, we can nonetheless say that U is unique.