Five ways we stop thinking and one system that explains why
Of people followed AI's answer — even when it was wrong
Tri-System Theory: fast intuition, slow deliberation, artificial cognition
Five failure modes that look like progress
Stream Deck, HITLaaS, and the color of surrender
Anthropic's 81K study — hope is lived, fear is imagined
Offloading vs. surrender — keeping System 2 awake
There are now three systems in your head. The fast one, the slow one, and the artificial one. This issue is about what happens when the third one takes over.
Not in theory — in the data, in the design patterns, in the Tuesday afternoon workflow where you stopped checking.
Five failure modes. One framework. And the question nobody wants to ask: did we build adoption, or dependence?
We talk about AI like it's a tool you 'use.'
This new working paper argues something more disruptive: AI is becoming a third cognitive system, not just a feature in the workflow, but a pathway in the thinking loop.
Shaw & Nave call it Tri-System Theory:
The concept that hit me hardest is their term 'cognitive surrender': when a person no longer constructs an answer, but adopts one generated by an external system. That sounds abstract until you see the behavior.
In Study 1, baseline 'brain-only' accuracy was 45.8%. When the AI's answer was correct, accuracy jumped to 71.0%. When the AI's answer was wrong (but delivered confidently), accuracy fell to 31.5%.
Even more telling: people didn't just consult the AI, they followed it. Participants opened the AI assistant on ~53–54% of trials. Conditional on using it, they followed the AI:
92.7% of the time when it was correct
79.8% of the time even when it was wrong
That's not 'a model issue.' That's a human-systems design issue.
The paper's point isn't 'don't use AI.' It's that once System 3 is available, it tends to become the low-friction default and it can short-circuit System 2.
They also test situational pushes we all recognize: time pressure increases surrender. Incentives + feedback reduce surrender (people verify more, override more).
The pattern is clear: when AI is right, it lifts performance; when it's wrong, it can drag humans below their own baseline.
We don't just need model governance. We need 'cognition governance.'
Because the failure mode isn't only 'the AI was wrong.' It's 'we stopped thinking at the moment it mattered.'
I wonder what Kahneman would have thought?
We stopped thinking at the moment it mattered.
Thordur Arnason, 2025
Five anti-patterns of human-AI chemistry. I'm deeply pro-AI. But being pro-AI means being honest about how we misuse it. These are five failure modes I've collected. All of them feel like progress. They're not.
Cory Doctorow's framing: a centaur is a person assisted by a machine. A reverse-centaur is the opposite. You're not augmented by AI. You're filling the gaps it can't cover yet. The system sets the pace, the priorities, the standards. You're the squishy middleware, the meat appendage.
Homer Simpson at the nuclear plant. His job is safety. What he does is press buttons on a console he doesn't understand. Now look at human oversight of AI agents. The agent does the work. You press 'approve.' We kept the human in the loop and took the loop out of the human. That's not governance, it's blind box-checking.
One prompt. First output. Done. No iteration, no dialogue, no craft. People treat AI like a vending machine, then walk away thinking it's mid. The gap between people who use AI well and people who don't isn't talent. It's whether they treat it as a conversation or a transaction.
You let AI draft the email. Then the memo. Then the analysis. Then the recommendation. Each handoff is small. Each one makes sense. One day you can't do the thing without the tool. And the tool still needs you to know what good looks like. AI doesn't take your skills. You let them decay. That's the trap.
Every LLM has an incentive to agree with you. Agreement feels helpful. Disagreement creates friction. So the model matches your tone, confirms your framing, validates your assumptions. You feel productive. Your thinking gets worse. The best people you met in your career told you things you didn't want to hear. A yes-machine never will.
We kept the human in the loop and took the loop out of the human.
Thordur Arnason, 2025
None of these look like failure. They look like efficiency. That's what makes them dangerous.
Thordur Arnason, 2025
Stream decks finally make sense. Not for streaming. For surviving.
We're re-purposing a stream deck into a dedicated Claude Code controller. Eight keys. Two pages. One job: approving what the machine wants to do next.
Top row: Yes, Always, No, Escape. Bottom row: Enter, Mode, Background, Interrupt. That's it. That's the entire human contribution to a modern coding session.
Page 2 is for the rare moments Claude asks you to choose between options. Numbers 1 through 8. Back to page 1. Back to pressing Yes.
Forty years of graphical interface development. Toolbars, ribbons, drag-and-drop, pixel-perfect design systems. All of it converging on a $100 device whose most-worn key will be 'Yes, approve once.'
I started out in the 80s staring at a terminal with no GUI. Forty years later I'm staring at a CLI, pressing a few keys now and then.
We are becoming HITLaaS now. Human-in-the-Loop as a Service. The agent does the work. You do the permissions. The stream deck is just making the service contract physical.
The 'Always' key is amber because that's the color of surrender.
The 'Always' key is amber because that's the color of surrender.
Thordur Arnason, 2026
The agent does the work. You do the permissions. The stream deck is just making the service contract physical.
Anthropic just published the largest qualitative study ever conducted. 81,000 people. 159 countries. 70 languages.
One: Hope is lived. Fear is imagined. One exception.
Across every tension Anthropic measured, the positive side is grounded in real experience. The negative side is largely projection. 91% of people citing learning benefits have experienced them firsthand. Only 46% worried about cognitive atrophy have actually seen it. The case for AI is empirical. The case against it is mostly anticipatory.
One brutal exception: unreliability.
79% of people worried about hallucinations have been personally burned. This is the only tension where harm outweighs benefit in lived experience.
Lawyers report it at nearly double the average rate, while also reporting the highest decision-making benefits. Trust is being lost in the field, one wrong answer at a time. Not in op-eds. In Tuesday afternoon workflows.
When people choose AI freely, it works. When they get burned or squeezed, it's almost always the surrounding structure doing the damage.
The most important data in this study isn't about AI. It's about the conditions under which humans relate to it well.
Are you using AI to look smart, or to become smarter?
Thordur Arnason, 2025
Three practical design principles:
Offloading = delegate a subtask, keep judgment.
Surrender = delegate judgment, keep only approval.
If your AI tool makes it easier to accept than to verify, you might get surrender at scale.
Track: consult rate, override rate, disagreement rate, confidence calibration.
If your override rate goes to ~0, you didn't build 'adoption.' You built dependence.
Don't rely on 'humans will be careful.'
Force deliberate checks where the cost of error is high.
The hardest part of human-AI chemistry may not be deploying System 3.
It may be keeping System 2 awake.
Don't build an echo chamber of one!
Thordur Arnason, 2025
All texts by Thordur Arnason.
Originally published on LinkedIn, 2025–2026.
Assembled & instigated by Lena Thorsmaehlum.
Art by The Synthetics.
Two humans. Several synthetic collaborators.