To me, the most plausible justification for assigning higher probability to S1 than S2 is that we ought to have priors that penalize more complex laws. More generally, it seems to me that we should be specifying priors at the level of theories / mechanistic models / etc, from which we then derive our priors about propositions like S1, S2, N-bad, N-good, “value concordance”. As opposed to directly consulting our intuitions about the latter.
So in the case of nuclear war, our priors over the long-run welfare consequences should be derived from our priors over the parameters of mechanistic models that we would use to predict how the world evolves conditional on nuclear war vs no nuclear war. And it seems much less clear that there will be a privileged prior over these parameters and that this prior will favor N-bad. (It seems plausible that the appropriate response would be to have imprecise priors over these parameters, and that this would lead to an indeterminate judgement about the total welfare consequences of nuclear war.)
I don't think penalizing complexity is enough to escape radical skepticism in general. Consider the "universe popped into existence (fully-formed) 5 minutes ago" hypothesis. It's not obvious that this is more complex than the alternative hypothesis that includes the past five minutes PLUS billions of years before that. One could try to argue for this claim, but I don't think that our confidence in history should be *contingent* on that extremely contentious philosophical project working out successfully!
I would indeed try to argue for something like that claim (along the lines sketched here [1]) but I agree that one should be uncertain about such arguments.
In the end, if our most compelling principles imply we ought to give significant weight to skeptical hypotheses, then I think we should bite the bullet and do so. And I personally don’t think “give greater prior weight to value-concordant worlds” or "give low prior weight to skeptical scenarios" are compelling principles. They look ad-hoc. By contrast, I think that Occam’s razor and the principle of indifference (under some conditions) are good principles, as they embody a kind of anti-ad-hocness.
I'm not claiming that these are *principles*. I just think they're obviously true claims about what we ought to believe. (It remains an open question *why*.) As I put it in the OP:
The question is, what kind of prior is most reasonable:
(i) One that builds in a default assumption of value concordance across different time-frames, e.g. taking the immense near- and medium-term harms of global nuclear war to be some reason to expect worse long-term outcomes;
or
(ii) One that takes value discordance to be just as likely, and so sees no reason whatsoever to expect global nuclear war to be overall bad.
It just seems very obvious to me that (i) is more reasonable, and I hope that most readers will agree (just as I hope that most will agree that we should assign higher credence to S1 than S2, above). But if you don’t, I probably don’t have much more to offer, besides an incredulous stare.
> I'm not claiming that these are *principles*. I just think they're obviously true claims about what we ought to believe. (It remains an open question *why*.)
Ok, sorry about that. I don’t understand yet, though. Is the idea that
1) it’s just a brute normative fact that our prior should have that property? (I guess when I said “principle” I meant to include this kind of view, probably I’m not using standard terminology.)
2) you have the strong intuition that there are more fundamental, as-yet-unidentified normative facts that imply that property?
Thanks for the always helpful and interesting engagement, Richard!
I'd like to clarify the Nuclear War argument a bit. I am claiming that we are clueless about whether a nuclear war in the near future would overall have good vs bad consequences over a billion-years-plus time frame continuing at least to the heat death of the universe. I do think a nuclear war would be bad for humanity! The way you summarize my claim, which depends on a certain way of thinking about what is "bad for humanity", makes my view sound more sharply in conflict with common sense than I think it actually is.
Clarifying "N-Bad" as *that* claim, it's not clear to me that denying it is commonsensical or that it should have a high prior.
(I do also make a shorter-term claim about nuclear war: That if we have a nuclear war soon, we might learn an enduring lesson about existential risk that durably convinces us to take such risks seriously, and if this even slightly decreases existential risk, then humanity would be more likely to exist in 10,000 years than without nuclear war. My claim for this argument is only that it is similar in style to and as plausible as other types of longtermist arguments; and that's grounds for something like epoche (skeptical indifference) regarding arguments of this sort.)
Hi Eric, thanks for continuing the conversation! I remain confused by your suggestion that there are multiple ways of thinking about what is "bad for humanity" in this context. You mentioned in our previous discussion that you think that *evaluation* (and not just decision) needs to build in something like temporal discounting. But this would seem to imply that it is "worse for humanity" for (i) one human to die of malaria today, and trillions of humans to flourish a billions years hence, than for (ii) nobody to die of malaria today, but the entire population of humanity, trillions strong, to be enslaved and tortured a billion years hence.
But that's surely just a misuse of language. There's no reasonable usage on which (i) can be sensibly described as "worse for humanity" than (ii). So you cannot coherently claim that nuclear war is "bad for humanity" unless you believe that it is bad for humanity *overall*, and not just in a local time period.
[Edited to add: You could coherently claim that (i) is worse than (ii), on grounds that we should care more about current people than about humanity at large. I'd argue (another day) that such a verdict is substantively misguided, but for this post meant to focus more on the epistemic issues, hence the stipulation that we're evaluating these prospects "for humanity" at large, not relative to other possible values, such as more partial ones. I could judge that nuclear war would be bad *because bad for my family* -- even if it averted future extinction -- but it would misdescribe my verdict for me to describe this as "bad for humanity", because there's more to humanity than just my family.]
I do think we should care more about current people than people a billion years in the future, on a combination of the grounds of their existence, our closer causal/social relationships, and maybe also pure temporal discounting. I’m not sure we can cleanly separate badness from caring, but I won’t press on that.
Maybe part of the issue here I that I’m unsure whether to include people existing a billion years in the future as part of “humanity”. Is it important to you to phrase it in that way? I feel that phrasing it in terms of what is overall timelessly good for whatever entities happen to exist avoids that definitional problem and is also closer to the consequentialist heart of the matter, assuming that consequentialism doesn’t privilege “humanity”.
It's true that utilitarian concern is more general, but that also raises extraneous complications about whether the extinction of humanity might be judged better for other species. To make the epistemic judgment as straightforward as possible, I'm wanting here to just focus on the interests of humanity and its descendants. That's enough to suggest that we aren't *completely* clueless about reasonable long-term expectations. We can always argue more another day about whether we have good reason to expect that what's good for humanity is also a good thing more broadly :-)
I'm okay with that rephrasing, but I don't think that the question of whether the extinction of humanity is good for other species is extraneous. There is a sense in which I have the harder path dialectically: My argument requires that there is *no* action currently available to us that we can justifiably evaluate as likelier to be good vs bad overall for the billion-year-plus future. So I'm committed to a negative existential generalization. But I don't think it's dialectically correct to make this path still harder by adding qualifications like good specifically *for humanity and its descendants* unless those qualifications are very well justified. By choosing nuclear war as an example, I'm trying to choose one of the hardest cases for my cluelessness view, to see if it still works. Let's not make it still harder by adding qualifications that my opponents would not actually endorse. I assume that my dialectical opponent here is more interested in the overall goodness of the outcome than in the goodness of the outcome specifically for humans and their descendants.
Here's an analogy, but one that doesn't take advantage of that rephrasing. A space alien arrives at informs me that in a billion years planet Earth will host a quadrillion Gorks. The alien now asks me whether I think that nuclear war would be good for a group composed of humans over the next several generations, plus Gorks, considered as a group. Call this artificial group the HuGork Group. Since I know basically nothing about Gorks, I have no idea and decline to speculate on this question. But I do know that nuclear war would be bad for humans over the next several generations, and I'm happy to say that.
Regarding the HuGork Group, I'd say one of the below, in increasing order of preference:
(1.) We should just accept being 50/50 on its goodness/badness for the HuGork Group.
(2.) If we expect nuclear war to have total value negative-X for humans over the next several generations, we should also expect it to have total value negative-X for the HuGork Group.
(3.) We should treat this as a question where mathematical models are more likely to mislead than illuminate.
So: I think nuclear war is bad for the future as far as we can foresee. But I don't think we can foresee its badness vs goodness overall, or its badness vs goodness for HuGorks overall. I'm not sure why I need to have an opinion about its badness considered in terms of timeless overall consequences to have an opinion about its badness.
This thought might help. Suppose baby Hitler had drowned. That event would have been bad, even if from an omniscient perspective one could foresee that it had overall good consequences. The family and neighbors knew that it was bad even if they couldn't know its consequences. Similarly, if some child I happen to know dies, I know that the death is bad even if I don't know what the child's ultimate impact on the world would have been.
Perhaps you should call this shorter-term claim the "Ozymandias argument", after the character in Watchmen who causes a disaster that kills millions in NYC in order to get the countries of the world to unite, averting a greater disaster from all-out nuclear war.
I think they're largely unrelated. The paralysis argument rather trades on something that is highly predictable: that among the many long-term causal consequences of our actions will be very significant harms and benefits. And while these might largely cancel each other out in expectation, if you give extra weight (as many deontologists do) to our reasons not to cause harms, then it seems every positive action will end up being prohibited (with the possible exception of longtermist-recommended actions, as then the positive expectation might be sufficiently great to outweigh the deontological asymmetry).
P.S. I'm not aware of any formatting options for comments, so I think you just have to write out the link (as you did). Though note that you can usually delete everything after a '#' or '?' in a url (including the symbol itself).
To me, the most plausible justification for assigning higher probability to S1 than S2 is that we ought to have priors that penalize more complex laws. More generally, it seems to me that we should be specifying priors at the level of theories / mechanistic models / etc, from which we then derive our priors about propositions like S1, S2, N-bad, N-good, “value concordance”. As opposed to directly consulting our intuitions about the latter.
So in the case of nuclear war, our priors over the long-run welfare consequences should be derived from our priors over the parameters of mechanistic models that we would use to predict how the world evolves conditional on nuclear war vs no nuclear war. And it seems much less clear that there will be a privileged prior over these parameters and that this prior will favor N-bad. (It seems plausible that the appropriate response would be to have imprecise priors over these parameters, and that this would lead to an indeterminate judgement about the total welfare consequences of nuclear war.)
I don't think penalizing complexity is enough to escape radical skepticism in general. Consider the "universe popped into existence (fully-formed) 5 minutes ago" hypothesis. It's not obvious that this is more complex than the alternative hypothesis that includes the past five minutes PLUS billions of years before that. One could try to argue for this claim, but I don't think that our confidence in history should be *contingent* on that extremely contentious philosophical project working out successfully!
I would indeed try to argue for something like that claim (along the lines sketched here [1]) but I agree that one should be uncertain about such arguments.
In the end, if our most compelling principles imply we ought to give significant weight to skeptical hypotheses, then I think we should bite the bullet and do so. And I personally don’t think “give greater prior weight to value-concordant worlds” or "give low prior weight to skeptical scenarios" are compelling principles. They look ad-hoc. By contrast, I think that Occam’s razor and the principle of indifference (under some conditions) are good principles, as they embody a kind of anti-ad-hocness.
[1] https://substack.com/home/post/p-156868796
I'm not claiming that these are *principles*. I just think they're obviously true claims about what we ought to believe. (It remains an open question *why*.) As I put it in the OP:
The question is, what kind of prior is most reasonable:
(i) One that builds in a default assumption of value concordance across different time-frames, e.g. taking the immense near- and medium-term harms of global nuclear war to be some reason to expect worse long-term outcomes;
or
(ii) One that takes value discordance to be just as likely, and so sees no reason whatsoever to expect global nuclear war to be overall bad.
It just seems very obvious to me that (i) is more reasonable, and I hope that most readers will agree (just as I hope that most will agree that we should assign higher credence to S1 than S2, above). But if you don’t, I probably don’t have much more to offer, besides an incredulous stare.
> I'm not claiming that these are *principles*. I just think they're obviously true claims about what we ought to believe. (It remains an open question *why*.)
Ok, sorry about that. I don’t understand yet, though. Is the idea that
1) it’s just a brute normative fact that our prior should have that property? (I guess when I said “principle” I meant to include this kind of view, probably I’m not using standard terminology.)
2) you have the strong intuition that there are more fundamental, as-yet-unidentified normative facts that imply that property?
3) something else?
Sounds like a false dichotomy. You could also have option (iii): A set of priors that regards it as indeterminate whether value concordance or value discordance is more likely. (I defend this kind of perspective here: https://forum.effectivealtruism.org/posts/NKx8sHcAyCiKT723b/should-you-go-with-your-best-guess-against-precise)
Thanks for the always helpful and interesting engagement, Richard!
I'd like to clarify the Nuclear War argument a bit. I am claiming that we are clueless about whether a nuclear war in the near future would overall have good vs bad consequences over a billion-years-plus time frame continuing at least to the heat death of the universe. I do think a nuclear war would be bad for humanity! The way you summarize my claim, which depends on a certain way of thinking about what is "bad for humanity", makes my view sound more sharply in conflict with common sense than I think it actually is.
Clarifying "N-Bad" as *that* claim, it's not clear to me that denying it is commonsensical or that it should have a high prior.
(I do also make a shorter-term claim about nuclear war: That if we have a nuclear war soon, we might learn an enduring lesson about existential risk that durably convinces us to take such risks seriously, and if this even slightly decreases existential risk, then humanity would be more likely to exist in 10,000 years than without nuclear war. My claim for this argument is only that it is similar in style to and as plausible as other types of longtermist arguments; and that's grounds for something like epoche (skeptical indifference) regarding arguments of this sort.)
Hi Eric, thanks for continuing the conversation! I remain confused by your suggestion that there are multiple ways of thinking about what is "bad for humanity" in this context. You mentioned in our previous discussion that you think that *evaluation* (and not just decision) needs to build in something like temporal discounting. But this would seem to imply that it is "worse for humanity" for (i) one human to die of malaria today, and trillions of humans to flourish a billions years hence, than for (ii) nobody to die of malaria today, but the entire population of humanity, trillions strong, to be enslaved and tortured a billion years hence.
But that's surely just a misuse of language. There's no reasonable usage on which (i) can be sensibly described as "worse for humanity" than (ii). So you cannot coherently claim that nuclear war is "bad for humanity" unless you believe that it is bad for humanity *overall*, and not just in a local time period.
[Edited to add: You could coherently claim that (i) is worse than (ii), on grounds that we should care more about current people than about humanity at large. I'd argue (another day) that such a verdict is substantively misguided, but for this post meant to focus more on the epistemic issues, hence the stipulation that we're evaluating these prospects "for humanity" at large, not relative to other possible values, such as more partial ones. I could judge that nuclear war would be bad *because bad for my family* -- even if it averted future extinction -- but it would misdescribe my verdict for me to describe this as "bad for humanity", because there's more to humanity than just my family.]
I do think we should care more about current people than people a billion years in the future, on a combination of the grounds of their existence, our closer causal/social relationships, and maybe also pure temporal discounting. I’m not sure we can cleanly separate badness from caring, but I won’t press on that.
Maybe part of the issue here I that I’m unsure whether to include people existing a billion years in the future as part of “humanity”. Is it important to you to phrase it in that way? I feel that phrasing it in terms of what is overall timelessly good for whatever entities happen to exist avoids that definitional problem and is also closer to the consequentialist heart of the matter, assuming that consequentialism doesn’t privilege “humanity”.
How about "humanity and its descendants"?
It's true that utilitarian concern is more general, but that also raises extraneous complications about whether the extinction of humanity might be judged better for other species. To make the epistemic judgment as straightforward as possible, I'm wanting here to just focus on the interests of humanity and its descendants. That's enough to suggest that we aren't *completely* clueless about reasonable long-term expectations. We can always argue more another day about whether we have good reason to expect that what's good for humanity is also a good thing more broadly :-)
I'm okay with that rephrasing, but I don't think that the question of whether the extinction of humanity is good for other species is extraneous. There is a sense in which I have the harder path dialectically: My argument requires that there is *no* action currently available to us that we can justifiably evaluate as likelier to be good vs bad overall for the billion-year-plus future. So I'm committed to a negative existential generalization. But I don't think it's dialectically correct to make this path still harder by adding qualifications like good specifically *for humanity and its descendants* unless those qualifications are very well justified. By choosing nuclear war as an example, I'm trying to choose one of the hardest cases for my cluelessness view, to see if it still works. Let's not make it still harder by adding qualifications that my opponents would not actually endorse. I assume that my dialectical opponent here is more interested in the overall goodness of the outcome than in the goodness of the outcome specifically for humans and their descendants.
Here's an analogy, but one that doesn't take advantage of that rephrasing. A space alien arrives at informs me that in a billion years planet Earth will host a quadrillion Gorks. The alien now asks me whether I think that nuclear war would be good for a group composed of humans over the next several generations, plus Gorks, considered as a group. Call this artificial group the HuGork Group. Since I know basically nothing about Gorks, I have no idea and decline to speculate on this question. But I do know that nuclear war would be bad for humans over the next several generations, and I'm happy to say that.
Regarding the HuGork Group, I'd say one of the below, in increasing order of preference:
(1.) We should just accept being 50/50 on its goodness/badness for the HuGork Group.
(2.) If we expect nuclear war to have total value negative-X for humans over the next several generations, we should also expect it to have total value negative-X for the HuGork Group.
(3.) We should treat this as a question where mathematical models are more likely to mislead than illuminate.
So: I think nuclear war is bad for the future as far as we can foresee. But I don't think we can foresee its badness vs goodness overall, or its badness vs goodness for HuGorks overall. I'm not sure why I need to have an opinion about its badness considered in terms of timeless overall consequences to have an opinion about its badness.
This thought might help. Suppose baby Hitler had drowned. That event would have been bad, even if from an omniscient perspective one could foresee that it had overall good consequences. The family and neighbors knew that it was bad even if they couldn't know its consequences. Similarly, if some child I happen to know dies, I know that the death is bad even if I don't know what the child's ultimate impact on the world would have been.
Perhaps you should call this shorter-term claim the "Ozymandias argument", after the character in Watchmen who causes a disaster that kills millions in NYC in order to get the countries of the world to unite, averting a greater disaster from all-out nuclear war.
How do Schwitzgebel's points about the long-term unpredictability of our actions relate to McAskill and Mogensen's "paralysis argument"?
https://globalprioritiesinstitute.org/summary-the-paralysis-argument/#:~:text=In%20%E2%80%9CThe%20Paralysis%20Argument%2C%E2%80%9D,improving%20the%20long%2Drun%20future.
BTW is it possible to embed links in the comments?
I think they're largely unrelated. The paralysis argument rather trades on something that is highly predictable: that among the many long-term causal consequences of our actions will be very significant harms and benefits. And while these might largely cancel each other out in expectation, if you give extra weight (as many deontologists do) to our reasons not to cause harms, then it seems every positive action will end up being prohibited (with the possible exception of longtermist-recommended actions, as then the positive expectation might be sufficiently great to outweigh the deontological asymmetry).
P.S. I'm not aware of any formatting options for comments, so I think you just have to write out the link (as you did). Though note that you can usually delete everything after a '#' or '?' in a url (including the symbol itself).