Also, can you link a more detailed write-up on Harsayni's argument? The more I think about what you have here, the less I'm convinced by it. I'll be convinced when the following scenario is addressed:
let's assume there are two utilitarians, Alice and Bob, arguing over how to order events which affect them both, X and Y. Since they are both utilitarians, they agree that VNM applies to them individually, so they each have utility functions \(A\) and \(B\) which give them the following values:
For Bob, \(B(X) = 1\) and \(B(Y) = 5\)
For Alice, \(A(X) = 10\) and \(A(Y) = 5\)
Now they move on to the problem of finding \(C(A, B)\), the combined utility of the two of them (what you are calling script U).
First, we distinguish between pure and degenerate lotteries -- really, \(A\) and \(B\) are functions on only degenerate lotteries (which we will call events from now on). To find the utility over non-degenerate lotteries, we use linearity of expectation, and in general use \(\mathbb{E}_A\) and \(\mathbb{E}_B\) to find utilities.
Alice claims that by Harsayni's argument, they should use \(C(\mathbb{E}_A, \mathbb{E}_B) = \mathbb{E}_A + \mathbb{E}_B\) (in general there could be some k_i's, but for now lets assume they are all 1). Bob claims that they should use \(C(\mathbb{E}_A, \mathbb{E}_B) = \sqrt{ \mathbb{E}_A } + \sqrt{ \mathbb{E}_B }\).
Alice: let's start by showing that \(C\) must be homogeneous: to do this, we will use the fact that \(C\) is linear over pure (i.e. non-degenerate) lotteries --
Bob: well I don't agree! My proposed \(C\) isn't linear over pure lotteries -- if \(W\) and \(Z\) are events and \(p\) and \(q\) are probabilities, we have \([\sqrt{ \mathbb{E}_A } + \sqrt{ \mathbb{E}_B }](pW + qZ) = \sqrt{ pA(W) + qA(Z) } + \sqrt{ pB(W) + qB(Z) }\).
Alice: Don't you know about the Von-Neumann Morgenstern Utility theorem? \(C\) *has* to be linear over pure lotteries.
Bob: In order to apply VNM, we need to agree on a total ordering on events -- but we do not! I think \(C(x) < C(Y)\), but you disagree.
Alice: ????
Care to help me out here?
@p-norm Pete
If you want to apply VNM and actually get your utility function from your preferences you'll need to agree on a total ordering but if you only want to know that one exists you just have to assume a total ordering exists. That's something we implicitly assume in the proof in assumption 2. It seems like a nice "rationality" assumption and if our collective doesn't have it it will either be dutchbookable or not take action in some scenarios. (In a practical situation a collective will probably have one of those properties instead of having total preferences).
It's true there is no objective way for Alice and Bob to agree on weights. I think in practice if there is such a scenario it's better to think about game theory and bargaining power than this theorem. I kind of think this theorem is only relevant if you are a government or philanthropist already committed to representing the collective.
Here's a possible continuation:
Alice: Indeed, I got ahead of myself. Let's agree on a few more key axioms: 1) Societal events have an ordering which satisfies all the VNM assumptions. We may disagree for the moment on what that ordering looks like, but we have to agree that it exists.
Bob: Yeah, that seems about as reasonable as my belief that my own preferences have a similar structure.
Alice: 2) If every individual is indifferent between societal outcomes, then society should be indifferent between the outcomes.
Bob: Sure, why not.
Alice: Finally, in order to do any actual calculation, we need to agree on the societal order of events with utility \((1, 0)\) (this even we'll call \(N\)) and \((0, 1)\) (This event we'll call M). Let's agree on these being equal to society for now.
Bob: I guess that seems fair.
Alice: Okay, we can now establish for certain that your proposed order leads to a contradiction. One way would be to just go through the proof of Harsayni's argument again, but we can also show the contradiction directly.
Bob: Let's do the direct contradiction.
Alice: Suppose we have the ordering \(Y > X > N \sim M > 0\) as you want. Let's consider two lotteries. L_1 is the lottery where \(M\) occurs \(25\%\) of the time, and the null event \(0\) occurs \(75\%\) of the time. \(L_2\) has event \(N\ 20\%\) of the time, and event \(0\ 80\%\) of the time.
Bob: So society must prefer \(L_1\), since \(N\) and \(M\) are equal to society.
Alice: Yup. Now if we add event \(N\) with probability \(5\%\) to both lotteries (taking over some of the probability of event \(0\)), we should both agree that our preferences to \(L_1\) and \(L_2\) haven't changed, right?
Bob: Right. So society still ranks \(L_1\) over \(L_2\). Writing things out, we have \(L_1\) is \((25\%\ M,\ 5\%\ N,\ 70\%\ 0)\) and \(L_2\) is \((25\%\ N,\ 75\%\ 0)\).
Alice: Ok, then similarly add event \(M\) with probability \(25\%\) to both lotteries, and again society ranks \(L_1\) over \(L_2\). But now \(L_1\) is \((50\%\ M, 5\%\ N,\ 45\%\ 0)\), which neither of us have a preference over \((5\%\ X,\ 95\%\ 0)\), and \(L_2\) is \((25\%\ M,\ 25\%\ N,\ 50\%\ 0)\), which neither of us have a preference over \((Y\ 5\%,\ 95\%\ 0)\). So now we have the contradiction, since we cant have both \((5\%\ X,\ 95\%\ 0) > (5\%\ Y,\ 95\%\ 0)\) and \(Y > X\).
Bob: Bah. I guess you're right.
Fun article! I also enjoyed that Eric Hoel blog post -- though I'm a little confused as to what precisely he's arguing in his example with the trillions of hiccups vs violent shark death. He seems to be implying both that one should prefer one outcome to another (which I agree with!) and also that some outcomes are incomparable (which I also agree with! and it would render VNM inapplicable), but I don't see how those two claims are related. In trying to come up with examples of incomparable preferences, somehow the best I've come up with so far is from the story of Abraham and Isaac (odd, since I'm not religious) -- but I feel like there's something there about how a devoutly religious person can't decide on a preference between obeying god and not harming his son.