I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.

Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth.

Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.

> I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.

Lots of humans have pretty despicable goals, including some very intelligent ones.

I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons.

Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.