I think this occurs due to the prompt. LLMs are actually text completion/translation focused in architecture. We just give them a prompt along the lines of “the context is that you’re a world leading expert now complete the response”.
They need the prompt to encourage expert outputs but unfortunately we also get ‘pretending to be an expert’ outputs since there’s a large amount of polluted training data for this.
