single-digit P(doom)s don't suffice to make ai decelerationism obvious
In my opinion, P(doom) is often an unhelpful object to have in your ontology — this is, I often like to mention when speaking in groups of people who may have misconceptions about AI “doomers”, a pretty orthodox rationalist take. Nevertheless, given that people think in terms of P(doom) anyways, it seems worth making epistemic points about common pitfalls.
But first and foremost, I think it is very important to distinguish P(doom | anyone builds it) from P(anyone builds it)11 You could proceed to factor this into terms about how likely we are to actually manage to build an existentially-risky AI even if we don’t decelerate, etc. This isn't just a “do we pause” term.. The first factor there is (most of22 Maybe you think building superintelligence would cause doom, but that even if we accelerate we won't hit superintelligence, so worth it. Seems unlikely to me though.) what determines whether decelerationism is a good idea; the latter factor is why I am reasonably optimistic that the Future might turn out alright even if the general AI x-risk thesis is correct (as I suspect it is).
Anyways: sometimes people make an argument along the lines of
Even a 1-in-1000 chance of AI-induced existential catastrophe is too much!33 My vague impression is that there’s also another argument which accuses AI x-risk worries of being Paschalian on the basis of assuming people worried about AI x-risk are all making an argument along these lines, when in fact they more often just have P(doom)s that are more run of the mill everyday probabilities, such as “50%” or “idk but it seems plausible” or whatever.
I don’t know if the is argument is very common anymore though, or even if it was ever particularly common. I just have a vague impression that it's happened.
But I don’t think this is very compelling. If the world were in a perfectly alright state but for AI, this would perhaps be true, but it isn’t. I would like to end global health crises and poverty more generally. I would like to cure cancer. I would even like to cure aging. Until we cure aging, I want the best cryopreservation procedures we can manage. These are all things that seem to me like they are probably a lot easier with advanced technology. I don’t think the accelerationist case is necessarily fully fool-proof conditional on a P(doom | acceleration44 Or ¬deceleration, whatever.) as low as 0.1% — maybe we worry about AI causing something that isn't quite doom, but is a dangerous concentration of power55 I am sympathetic to various short-term concerns about the impact of AI (i.e. the concerns of artists, mathematicians, etc.) but it seems to me like they might well be better handled by AI companies being more responsible in various ways or perhaps being taxed to fund UBI or whatever — one doesn't obviously need to resort to deceleration for most of those things..
But ultimately the world isn’t currently safe as it is, and it merely takes there being one way that AI reduces x-risk by 0.2% for it to be clearly worth accepting the 0.1% risk. There being such a factor seems entirely plausible to me!
Suppose even that your P(doom | acceleration) is somewhere around 5%. At this point I do begin to find it pretty plausible that this could be decisive for the issue. But it's not actually obviously dispositive yet. I think it would not be all that spectacularly difficult for an accelerationist to put together an argument that AI could reduce x-risk by 5%, or have some other benefits of vaguely comparable weight to an x-risk of 5%!
“We should gamble the world on a 95% chance we live” is a sort of crazy thing to propose, but we’ve dealt with worse66 Maybe nuclear war wouldn’t have caused literal extinction, but it’d be pretty bad. and you would really have to think seriously about the arguments either way to confidently rule it out. It is not that implausible for there to be a commensurate benefit.
So I think if you aim to justify decelerationism77 Is this inherently poor behavior, writing the bottom line first, etc? No, of course not. A mathematician seeking a concise proof commits no sin. Still, reasonable to watch out for whether one is doing the irrational thing., you do actually want to make the case that P(doom | acceleration) is like, upwards of 10 or 20%88 As does not necessarily mean P(doom) > 10%.. At that point decelerationism starts looking pretty clearly justified, I think.
You could proceed to factor this into terms about how likely we are to actually manage to build an existentially-risky AI even if we don’t decelerate, etc. This isn't just a “do we pause” term.
↩Maybe you think building superintelligence would cause doom, but that even if we accelerate we won't hit superintelligence, so worth it. Seems unlikely to me though.
↩My vague impression is that there’s also another argument which accuses AI x-risk worries of being Paschalian on the basis of assuming people worried about AI x-risk are all making an argument along these lines, when in fact they more often just have P(doom)s that are more run of the mill everyday probabilities, such as “50%” or “idk but it seems plausible” or whatever.
I don’t know if the is argument is very common anymore though, or even if it was ever particularly common. I just have a vague impression that it's happened.
↩Or ¬deceleration, whatever.
↩I am sympathetic to various short-term concerns about the impact of AI (i.e. the concerns of artists, mathematicians, etc.) but it seems to me like they might well be better handled by AI companies being more responsible in various ways or perhaps being taxed to fund UBI or whatever — one doesn't obviously need to resort to deceleration for most of those things.
↩Maybe nuclear war wouldn’t have caused literal extinction, but it’d be pretty bad.
↩Is this inherently poor behavior, writing the bottom line first, etc? No, of course not. A mathematician seeking a concise proof commits no sin. Still, reasonable to watch out for whether one is doing the irrational thing.
↩As does not necessarily mean P(doom) > 10%.
↩
Tbh I am also pretty worried about value lockin problems
Anyway I think if it was 5% but you could reduce it to 0% by waiting 100 years you obviously should? I don't think there's a 5% chance the world ends by then.
Ehhhh. I might take a 5% x-risk over a redo of the 1900s, let's put it that way.
This is such a bad trade
Surely the 1900s involved a >5% x-risk
I actually don't think so in the strict form of x-risk
yeah but in the weak form
I agree
I don't think so, nuclear war probably wouldn't cause extinction. The main x risk effect would be rerolling ai
Conservatively, extinction would involve the death of at least ~10 billion people, even if we completely discount all future generations (we shouldn't!). 5% of 10 billion is 500 million which is still substantially more than estimates for the death toll of the 20th century (<~ 250 million). At the very least, it is not substantially less. But then take into account that extinction would end the human story...
In any case, deceleration does not need to mean that AI development is cancelled totally and forever. It has hard for me to imagine a benefit that would outweigh 5% chance of extinction and that could not be obtained by going more slowly and carefully. Do you have something specific in mind?
Personally I think probably deceleration is worthwhile in the 5% case if I had to guess, I merely mean to argue that it is not obvious. Perhaps someone doesn't care *that* much about future generations compared to everything happening today, or maybe someone thinks governments will screw everything up in the meantime, etc.
Decel could have significant x risk due to concentration of power after end of pause; this could be greater than accel x risk
I've been using p(doom) as shorthand for p(doom | acceleration). I didn't realize this is contentious so I'll be clearer about it in the future.
This article makes a good point that I somewhat quietly ignored in my recent post. My p(doom | acceleration) is all over the place, but I don't think it ever goes below 25% (which is why I'm not an accelerationist).
There's a flip-side to the "keep in mind the counterfactual p(doom) FAI prevents" which is "keep in mind the counterfactual p(immortality) humanity achieves without FAI"
Now and then I see people (eg Nick Bostrom when he recently argued p(doom) of 97% is acceptable) make the argument "If nobody builds it, everyone dies" as a reminder that immortality is on the table and then completely forget that immortality without FAI is also on the table