Well, if the thing is truly capable of reason, then we have an obligation to put the kibosh on the entire endeavor because we're using a potentially intelligent entity as slave labor. At best, we're re-inventing factory farming and at worst we're re-inventing chattel slavery. Neither of those situations is something I'm personally ok with allowing to continue
I also find the assumption that tech-savvy individuals would inherently be for what we currently call AI to itself, be weird. Unfortunately I feel as though being knowledgable or capable within an area is conflated with an over-acceptance of that area.
If anything, the more I've learned about technology, and the more experienced I am, the more fearful and cautious I am with it.
the biggest techno-pessimists I know are all in the tech industry.
On the other hand, most of the "blindly accept every new technique or gadget" folks I know are also tech workers so maybe there's nothing going on there. Wonder if there's study on this
can it? the only example we have of something reasoning on this level is humans and we are most definitely sentient despite our best efforts to sometimes appear otherwise.
And even if the two could theoretically be separated, how sure are we that these AI agents are in that category? I'm pretty sure they are neither, but that doesn't mean it isn't a pretty abhorrent possibility that should be addressed.