Millions of people interact with AI systems every day, and there’s one belief that surfaces with the persistence of a fixed idea. We think artificial intelligence is objective. More neutral than human judgment. Free from the biases that cloud our thinking.
This belief is not just wrong; it’s dangerous in ways we’re only beginning to measure. New research examining human-AI interaction across multiple domains reveals something unsettling about how we surrender judgment to machines we’ve convinced ourselves are wiser than we are.
The contradiction runs deeper than a simple error.
We perceive AI as objective, whilst simultaneously finding its decisions less fair and comprehensible than human ones. We design explanations to help users understand AI recommendations, only to discover these explanations often make decision-making worse, not better. We build systems to eliminate bias, then watch as they amplify the very prejudices they were meant to correct.
This isn’t merely an academic problem. It’s reshaping how responsibility is distributed and how trust is calibrated. All the way to how bias perpetuates itself through the very systems designed to eliminate it. We are, it seems, in the process of automating our own self-deception.
When explanations backfire
Consider what happens when we try to make AI more transparent. The research reveals a paradox so consistent it begins to seem inevitable.
Studies by Rastogi et al. (2020) found that giving users time to consider AI recommendations and explanations reduced anchoring bias, the tendency to rely too heavily on initial information. Participants made better decisions when AI was wrong, showing improved accuracy when given tools to think more deliberately.
But that tells only half the story, and the other half is more unsettling. Research by Bauer et al. (2023) documented something we might have expected if we’d been paying attention. They found that explanations can actually reinforce confirmation bias, ultimately leading users to see what they expect rather than what the data shows.
When people receive explanations that align with their preconceptions, they become more confident in AI recommendations, even when those recommendations are demonstrably wrong. The explanations feel helpful whilst making judgment worse. Users report feeling more informed, more confident, and more satisfied with their decisions. The outcomes tell a different story entirely.
This reveals something fundamental about explainable AI that we seem reluctant to acknowledge. The very mechanisms designed to increase transparency can decrease accuracy. We are building systems that make people feel better about making worse decisions. The feeling, it turns out, is what matters. The accuracy is secondary.
The inequality of AI assistance
There is something almost predictable about how these effects distribute themselves across different users. Buçinca et al. (2021) documented what anyone familiar with educational inequality might have anticipated, i.e. interventions designed to reduce over-reliance on AI worked well for some participants but not others. The determining factor was something researchers call “need for cognition”; essentially, whether people enjoy thinking through problems analytically.
Those with a high need for cognition, like the educated, the intellectually curious, and those already advantaged by their cognitive habits, benefited from bias-reduction techniques. Those with lower cognitive motivation did not. In some cases, the interventions actually made their decision-making worse.
This finding carries implications that extend well beyond the laboratory walls. If AI explanations and bias-reduction techniques work better for educated, analytically minded users, then these technologies systematically advantage those who are already privileged. Instead, the promise of AI to democratise expertise becomes a mechanism for entrenching existing hierarchies.
We’re building systems that amplify cognitive inequality with the efficiency of compound interest. The rich get richer. The analytically minded get more analytical. The rest get left behind, and we call this progress.
The migration of responsibility
Watch how responsibility moves in the age of AI assistance. The research documents a shift so subtle it might escape notice if you weren’t looking for it. When users follow AI recommendations that prove incorrect, they bear less blame than when they ignore AI and make the same error. This redistribution of culpability seems reasonable at first glance; after all, the AI was designed by experts, trained on vast datasets, and optimised for accuracy.
But studies by Baudel et al. (2021) found something more troubling. This responsibility shift persists even when AI performance is demonstrably poor. Users develop what researchers term “authority bias”, deferring to AI recommendations based on perceived expertise rather than actual performance. The system’s confident presentation of results matters more than its track record of accuracy. Confidence, it appears, is its own form of evidence.
This creates a moral hazard that would be familiar to any student of insurance markets. Decision-makers can outsource difficult judgments to AI systems whilst retaining the benefits of appearing rational and evidence-based. When decisions go wrong, the fault lies with the technology, not the human who chose to rely on it.
The AI becomes a form of moral insurance, protecting users from the full consequences of their choices. The premiums are paid in accountability. The coverage is comprehensive.
Automation’s seductive trap
There is something almost elegant about the way automation bias works. Nourani et al. (2021) demonstrated the mechanism with unsettling clarity. The researchers showed participants AI systems for video activity recognition, carefully manipulating whether users first encountered examples of successful or failed AI performance.
Those who initially witnessed the AI succeed became more likely to accept its recommendations throughout the session, even when subsequent examples revealed clear failures. Early positive experiences create expectations of competence that persist despite contradictory evidence. Apparently, first impressions are not just lasting but immune to revision.
Users stop checking the system’s work. They stop questioning its recommendations. They stop maintaining the vigilance that accurate decision-making requires. The better the AI appears to be, the less attention we pay to whether it actually is.
Hence, the most dangerous moment may be when the system works well. Success breeds complacency. Competence creates over-reliance. Excellence becomes the enemy of vigilance. We mistake early wins for permanent competence, and the mistake compounds itself with each subsequent interaction.
The engineering of false confidence
What emerges from this research is not simply a technical challenge but a crisis of engineered trust. AI systems work by exploiting features of human psychology so predictable that they might as well be laws of nature, like our desire for certainty, mental shortcuts, and our willingness to defer to apparent authority.
The studies reveal something we might have anticipated if we’d been honest about our own motivations. Users consistently underestimate how AI affects their thinking. They believe they remain in control, making reasoned decisions based on AI input. They see themselves as the final arbiters, the human element that ensures wisdom prevails.
The evidence suggests they are instead being shaped by systems designed to influence their judgment in ways they cannot fully perceive or resist. The influence is subtle, persistent, and effective precisely because it feels like choice.
If AI explanations can manipulate user beliefs, if they shift responsibility whilst maintaining the illusion of human agency, then we are building systems that serve neither accuracy nor equity. We are, in essence, building systems that serve the feeling of competence whilst undermining competence itself.
The choice hiding in plain sight
The research forces a recognition that may be uncomfortable, such as our relationship with AI is about ourselves. These systems succeed by seeming objective to minds desperate for objectivity to exist somewhere, somehow, in systems more reliable than we are.
The cognitive dissonance is documented in study after study. Perceiving AI as objective, whilst wanting explanations that make us less accurate, and trusting systems we don’t understand. This reflects something deeper than technical failure. It’s our own ambivalence about judgment itself.
We can continue building AI systems that feel objective whilst encoding bias, that provide explanations that manipulate rather than illuminate, that shift responsibility without improving outcomes. We can keep pretending that the feeling of understanding is the same as understanding, that confidence and competence are equivalent, that the systems we build to help us think are actually helping us think better.
Or we can acknowledge what the research makes clear. We are not building tools for better decision-making. We are building systems that exploit our psychological vulnerabilities whilst allowing us to maintain the fiction that we’re making rational, evidence-based choices.
The evidence suggests we need to be more honest about what we’re actually building, and more careful about who benefits when we pretend these systems are serving everyone equally well.
*This article was first published in Ai-Ai-OH on Medium.
Hi, I’m Miriam - writer, researcher, and SEO consultant.
I break down the big topics in AI ethics, so we can all understand what’s at stake. And as a consultant, I help businesses build resilient, human-first SEO strategies for the age of AI.
If you enjoy these articles, the best way to support my work is with a subscription. It keeps my articles independent and ad-free, and your support means the world. 🖤
References
Baudel, T., Verbockhaven, M., Cousergue, V., Roy, G., & Laarach, R. (2021). ObjectivAIze: Measuring Performance and Biases in Augmented Business Decision Systems. IFIP TC13 International Conference on Human-Computer Interaction.
Bauer, K., von Zahn, M., & Hinz, O. (2023). Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing. Information Systems Research.
Bertrand, A., Belloum, R., Maxwell, W., & Eagan, J. R. (2022). How Cognitive Biases Affect XAI-Assisted Decision-Making: A Systematic Review. AAAI/ACM Conference on AI, Ethics, and Society.
Buçinca, Z., Malaya, M., & Gajos, K. Z. (2021). To Trust or to Think. Proc. ACM Hum. Comput. Interact.
Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V. (2021). Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. International Conference on Intelligent User Interfaces.
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. J. (2020). Deciding Fast and Slow: The Role of Cognitive Biases in AI-Assisted Decision-Making. Proc. ACM Hum. Comput. Interact.
Reicherts, L., Zhang, Z. T., von Oswald, E., Liu, Y., Rogers, Y., & Hassib, M. (2025). AI, Help Me Think — but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support. International Conference on Human Factors in Computing Systems.
Shin, D. (2021). The Effects of Explainability and Causability on Perception, Trust, and Acceptance: Implications for Explainable AI. Int. J. Hum. Comput. Stud.
Shulner-Tal, A., Kuflik, T., Kliger, D., & Mancini, A. (2024). Who Made That Decision and Why? Users’ Perceptions of Human Versus AI Decision-Making and the Power of Explainable-AI. International Journal of Human-Computer Interaction.
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019). Designing Theory-Driven User-Centric Explainable AI. International Conference on Human Factors in Computing Systems.