I think the dread you may be feeling is "facts without agency" which is to say that a system which can answer any question on a topic but doesn't have the agency to understand can be really bad. The whole "best way to hide a body" stuff when Siri was released, now backed up by facts is what? possible? The example (no I don't know how real it was) of an alleged 13 year old girl asking how to make sex with a 31 year old male she met on the internet "special" is the kind of thing where a human in the loop starts with "Wait, this is the wrong question." Similarly with questions about how to successfully crime.
Having run a search engine for a bit it quickly became clear how criminals use search engines (mostly to search out unpatched web sites with shopping carts or wordpress blogs they could exploit at the time). I don't doubt that many malicious actors are exploring ways to use this technology to further their aims. Because the system doesn't "understand" it cannot (or at least has not been shown to) detect problems and bad actors.
FWIW, the first application I thought of for this tech is what the parent comment fears, basically having people who can follow a script running a "Front end" that presents to an end user a person who looks familiar and speaks their language in a similar accent (so accent free as far as the caller is concerned) about a topic such as support or sales. Off shore call centers become even more cost effective with on-the-fly translation because you don't even need native language speakers. That isn't a "bad thing" in that there is nominally a human in the loop but their interests are not aligned with the callers (minimize phone time, costs, boost satisfaction).
And of course the whole "you trained it on what?" question where you wonder just what is used as source material and without knowing that what sort of trust can you put in the answer?
Having run a search engine for a bit it quickly became clear how criminals use search engines (mostly to search out unpatched web sites with shopping carts or wordpress blogs they could exploit at the time). I don't doubt that many malicious actors are exploring ways to use this technology to further their aims. Because the system doesn't "understand" it cannot (or at least has not been shown to) detect problems and bad actors.
FWIW, the first application I thought of for this tech is what the parent comment fears, basically having people who can follow a script running a "Front end" that presents to an end user a person who looks familiar and speaks their language in a similar accent (so accent free as far as the caller is concerned) about a topic such as support or sales. Off shore call centers become even more cost effective with on-the-fly translation because you don't even need native language speakers. That isn't a "bad thing" in that there is nominally a human in the loop but their interests are not aligned with the callers (minimize phone time, costs, boost satisfaction).
And of course the whole "you trained it on what?" question where you wonder just what is used as source material and without knowing that what sort of trust can you put in the answer?