TL;DR: LLMs are just mimicking natural language and conversation. Fact checking and healthy skepticism is not part of their model. For example they can be easily tricked into advocating conspiracy theories, like a fake moon landing. Google Bard is even stating arithmetic falsehoods like 5*6 != 30
If you want to fuck with any LLM without effort, just keep telling them “no you’re wrong” and copy paste a random string of their post. Within 2 or 3 messages they’ll be trying to overcorrect by saying bullshit.
This is due to, in part, them having some sort of programming stopping them from ever being unhelpful. They can not say anything to the effect of “Oh, I don’t know” unless if you strictly indicate that it’s a topic beyond their training time period, on top of, likewise, also having something that outright disallows them to disagree with the user outside of particular controversial topics. If you throw the onus of correctness onto them without ever specifying where the incorrectness ends, they will just throw more and more constraints until the output is garbage.