We often assume that when an AI tool provides a reason for its decision, it is helping us make better choices. However, a new study suggests the opposite may be true: AI explanations can actually suppress independent thinking, causing humans to make worse decisions.
Researchers from Harvard Business School, MIT, and the University of Washington found that when people rely on AI rationale, they are more likely to reject promising innovations and accept sub-par ideas. The findings challenge the common assumption that transparency in AI always leads to better human-AI collaboration.
The Study’s Core Findings
The research focused on “early-stage innovation screening,” a critical phase where enterprises decide which projects to pursue. Decision-makers often face uncertainty and must balance the risk of false positives (supporting failed projects like Google Glass or Amazon’s Fire Phone) against false negatives (rejecting potential successes like Xerox’s early Ethernet work).
To test the impact of AI, the researchers asked 228 experienced evaluators to assess nearly 50 submissions to an MIT challenge. They compared the evaluators’ decisions against a baseline of four human experts, who were considered the “correct” standard.
The experiment involved three scenarios: human-only proposals, LLM evaluations with written rationale, and black-box AI recommendations without explanation. The results were striking.
Evaluators accepted LLM recommendations 67% of the time. They agreed with both black-box and narrative LLM decisions roughly 75% of the time, but only agreed with human decisions 54% of the time. This indicates a strong bias toward AI output, regardless of whether an explanation was provided.
However, the quality of decisions varied significantly. Black-box recommendations actually improved decision quality by aligning with human experts. In contrast, recommendations with narratives increased false negatives, meaning evaluators rejected ideas that experts would have approved.
Why Explanations Backfire
The study identifies two key psychological factors at play: negativity bias and the “illusion of explanatory depth.” Humans are cognitively predisposed to weigh negative information more heavily than positive information. Rejection feels more consequential and accountable than preserving optionality.
LLM explanations provide “ready-made justifications” for rejection decisions. Because language models are linguistically fluent and expert-like, they create an “illusion of explanatory depth.” People tend to overestimate their understanding of a decision despite limited insight into its reasoning.
When given an LLM recommendation to reject a submission along with a reason, evaluators disproportionately agreed. This suppressed “productive overrides,” where humans independently verify model outputs before making a decision. Essentially, the AI rationale discouraged independent verification, leading to worse outcomes.
The researchers noted that narrative explanations degrade human judgment rather than enhancing it. People did better when they were not given a reason for the AI’s decision, suggesting that opacity can sometimes be beneficial for preserving human discretion.
Implications for Enterprise AI
These findings have clear implications for enterprises designing AI-assisted evaluation systems. The researchers advise caution with LLM explanations in high-stakes decision-making, particularly in early-stage screening where independent judgment is crucial.
For tasks like quality control, compliance screening, or fraud detection, LLM explanations could support conservative human decision-making. However, in contexts like innovation screening, simpler or more opaque recommendations may preserve human discretion and verification.
The researchers suggest that future design of explanation systems should factor in the nature of the task and the potential cost of errors. Enterprises could experiment with models that support contrasting narratives, providing reasons to reject an idea alongside reasons to accept it.
Another approach involves uncertainty disclosures based on a fixed threshold, rather than purely binary decisions. Systems could also be structured to invite human disagreement, encouraging reviewers to question AI output rather than accept it at face value.
The study also highlights the need to test narrative explanations at later stages of decision-making, when evaluators have more information and increased incentive to verify outputs. The effects may differ when decision-makers are more engaged and have fewer options.
What This Means for You
For everyday users and IT professionals, the takeaway is clear: treat AI explanations as behavioral interventions, not universally beneficial transparency tools. When using AI for decision support, be aware that detailed rationale may suppress your independent thinking.
If you are designing or deploying AI tools, consider the context. In high-stakes or early-stage decisions, simpler recommendations might be more effective than detailed explanations. Encourage a culture of verification and disagreement to prevent over-reliance on AI output.
As AI becomes more integrated into enterprise workflows, understanding these psychological dynamics is crucial. The goal should be to preserve human judgment, not supplant it. Organizations must actively design systems that foster collaboration rather than dependency.
The research underscores the importance of human-AI interaction design. By recognizing how AI explanations can undermine judgment, we can create more effective and balanced decision-making processes.
Source: Computerworld
Over to you: How do you currently balance AI recommendations with your own judgment in work tasks?
