You decide you want to optimize for number of lives saved. You decide that future lives, those of people yet to be born, are worth as much as those currently alive. You place small, but importantly not zero, probability on existential risks to humanity, so that when you do the expected value calculation, even an infinitesimally small risk of humanity's complete extinction results in negative infinite utility. You're also very smart and realize that smart people can do damage if their objectives are misaligned, and you start to worry about something much much smarter than you with objectives misaligned to humanity's more broadly. In their defense reward specification is indeed a hard problem, RL agents find unexpected policies that maximize reward in even toy settings. At this point you're down the rabbit hole and no other problem seems to compare. Climate change will leave some people alive, pandemics leave some people alive, AIs have no such kindness.