It doesn't help that they've been trained to refer to themselves in ways that imply a sense of self-awareness ("as a large language model, I") or that they employ emotional language, which is far more effective at influencing people than rationality.

If people on HN can believe LLMs are sentient, sapient and intelligent beings (even more so than other humans I suspect) then there isn't much chance for average people getting caught in the intersection of LLM marketing, a hundred years of pop sci-fi cultural conditioning and a million years of primate evolution.