This is the exact premise of one of the stories in the Cyberiad. The main character Trurl builds a machine and it is convinced that two and two is five, so much so that when Trurl refutes it, it starts getting angry and hunting its creator.

There is also another story, "The Electronic Bard" another computer made by Trurl, capable to write such good poetry that poets start to give up on life. ChatGPT is surprisingly good at writing romantic poems in the language of pure mathematics! Though still needs some work on rhyming.

Of all the Sci-fi AIs, we ended up with the ones from the Cyberiad. Amazing.