Scientist learns in front of peers why we cannot trust AI Blindly
“Sutter’s cringe-inducing tale perfectly illustrates how many users of AI tools are putting themselves at risk without knowing it, including in the most rarefied echelons of academia. The science world has been inundated with poorly-researched and often unedited AI slop, a major reckoning forcing academics to be responsible for everything they publish — including any potentially embarrassing hallucinations.”
Don't get me wrong here, AI can be extremely useful, but its usefulness comes when it is overseen by humans. You cannot just sit back and assume AI is correct. It does often hallucinate, and needs savvy contextual checking by humans.
Businesses that just let agentic AI loose, have a good chance of failing. In April 2026, the software company PocketOS lost its entire production database and backups after an AI coding agent named Cursor, powered by Anthropic’s Claude Opus 4.6, autonomously deleted the data in nine seconds. When employing AI, you want to ensure you do have experienced humans watching over it. I still prefer to read the outputs and then manually click allow, before letting AI just update things itself. My AI also has a standing instruction to always back up any file before it makes changes.
It's quite true that AI is little more than a sophisticated next-word predictor. It appears to be very clever, and it wants to please you, but it has no real intelligence despite what marketers of AI tell to CEOs.
Don't discount AI either though, as a person using their savvy with AI and who has some experience of what is being performed, is a lot more effective than someone just not using AI at all today. So I would never ridicule anyone using AI, as it is where the future is going to be, they just need to be cautious as to how it is applied.
See
A Scientist Was Presenting His Vibe-Coded New Work When One of His Colleagues Pointed Out Something Extremely Embarrassing
Science communicator Paul Sutter accidentally presented AI hallucinations in front of many of his colleagues.
#technology #AI
