The prime futurist fear is that humanity will create some advanced technology with an ostensibly positive purpose, but it will buck our control and undo the world as it pursues some twisted version of the ends it was programmed to achieve. Quiet Babylon writes that this artificially-sentient oppressor has already arrived:
… Read the rest
One of my favorite recurring tropes of AI speculation/singulatarian deep time thinking is meditations on how an evil AI might destroy us.
Here’s an example: The scenario imagined is where there is a button that humans push if the AI gets an answer right and the AI wants to get a lot of button presses, and eventually it realizes that the best way to get button presses is to kill all the humans and institute a rapid fire button-pressing regime.
You would have this thing that behaves really well, until it has enough power to create a technology that gives it a decisive advantage — and then it would take that advantage and start doing what it wants to in the world.