To an extent; they can give hints and suggest directions, but you need to treat them as an unreliable narrator: think of them as entities that can help or deceive you at random.

That being said, we are researching tailored LLMs and other architectures to assist mathematical research that are more geared towards accuracy at the expense of freedom ("imagination"). The Lean FRO has some related information and links.