When AI explains the reasons behind its decisions, people tend to perceive those decisions as more transparent and fair. However, an easy-to-understand explanation does not guarantee the accuracy or fairness of a decision. The standard for AI governance must shift from “Was it explained?” to “Can it be verified?”

***

As artificial intelligence has moved deeper into people’s daily lives and organizational decision-making, a new question has emerged. Beyond asking how accurate AI is, people have begun asking whether it can explain why it made a particular decision. If a system rejects a loan application, sets different insurance premiums, ranks job applicants, or predicts a patient’s likelihood of disease, simply presenting the result is not enough. The growing concern is that an AI system whose reasoning cannot be understood is difficult to trust, no matter how well it performs.

Traditional statistical models used relatively simple relationships. It was easier for people to trace how much each variable affected an outcome. However, the spread of machine learning and deep learning changed the situation. Hundreds, billions, or even more variables and parameters began interacting in complex ways to produce results. Even developers found it difficult to explain clearly how a particular output had been generated. Accuracy increased, but the decision-making process became opaque. This was the background against which the so-called “black box” problem emerged in earnest.

Explainable AI, or XAI, gained attention as a response to the black box problem. XAI refers to technologies and methods that present the decisions of complex models in forms that people can understand. When a particular loan application is rejected, for example, the system might indicate whether income, debt, or repayment history had the greatest influence. In medical imaging, it might highlight the area on which the AI based its judgment that an abnormality was present. Some methods explain how an entire model works, while others explain the factors that influenced a single decision.

Explainability has become an important component of responsible AI. For companies, it offers a means of examining AI decisions internally. For users, it provides a channel through which they can understand the causes of unfavorable decisions and challenge them. Regulators have also begun using explanation tools to determine whether AI produces discriminatory or unfair outcomes. Transparency and explainability have been presented as core principles of AI governance alongside fairness, accountability, and privacy protection.

The problem is that explainability is a far more complicated concept than initially expected. Fully disclosing the internal workings of an AI system is different from providing a summary that people can understand. A technical explanation intended for developers is not the same as the explanation needed by consumers. Showing that a particular variable is important is also different from explaining why that variable became important. Most importantly, the fact that AI provides an explanation is not the same as the fact that its decision is fair.

Early expectations surrounding explainable AI contained a simple assumption. If the black box could be opened and its interior examined, errors and biases could also be discovered. In practice, however, explanations resemble maps that simplify complex movements so that people can understand them more than transparent windows that reveal a model’s internal workings exactly as they are. Maps are useful, but they do not contain every feature of reality. The same place can look entirely different depending on what is shown and what is omitted.

Research introduced by Knowledge at Wharton in May 2026 issued a warning at precisely this point. The researchers demonstrated that widely used machine-learning interpretation tools could make a discriminatory model appear neutral. The problem did not arise from a lack of explanation. Instead, the existence of an explanation itself could create false confidence. A tool designed to increase transparency could become a mechanism for concealing bias rather than a window for discovering it.

This does not mean that explainable AI is unnecessary. It means that the belief that accountability and fairness will naturally follow once an explanation is available is dangerous. Installing a single window in a black box does not reveal the entire interior. It is also necessary to examine where that window is facing, what it conceals, and how faithfully the visible scene reflects the system’s actual operation.

People tend to trust a judgment accompanied by reasons more than one that presents only a conclusion. The notification “Your loan application has been rejected” feels less reasonable than the explanation “Your loan application was rejected because of your debt ratio and recent late payments.” Even when people dislike an outcome, they are more likely to accept it if they understand the criteria that were applied. Explanations reduce uncertainty and create the impression that there is order behind a decision.

This psychology also applies to AI. When AI provides reasons, users are more likely to assume that the system analyzed sufficient data, applied consistent standards, and operated without hidden prejudice. The more detailed and visually sophisticated the explanation is, the more trust it may inspire. When graphs showing variable importance, color-coded risk levels, and precise probability figures are presented, the result can appear to be the product of objective science.

However, sophisticated presentation and the legitimacy of a decision are not the same thing. The appearance of a figure such as 78.4% does not guarantee that the prediction is actually that accurate. The presence of a clean graph does not mean that the model is free from bias. The more professional the explanation appears, the more likely users are to accept the result rather than question the assumptions and limitations behind it.

AI explanations can broadly be divided into methods that make the model itself easier to understand and methods that interpret the results of a complex model afterward. Some models, such as simple decision trees, allow their internal structures to be examined directly. In practice, however, organizations often use complex black box models to achieve high predictive performance and then attach separate interpretation tools. In such cases, an explanation is not a record that translates every calculation the model actually performed. It is an approximate explanation created by analyzing the relationship between the model’s inputs and outputs in a way that people can understand.

Post-hoc explanations are useful because they reduce complexity. Yet approximation entails loss. The same result may be explained differently depending on which interpretation method is selected, and the ranking of important variables may change depending on the scope of analysis and the reference values used. An explanation concerning a particular individual may also create an impression different from an explanation about the population as a whole. An explanation is closer to the result of viewing a model from a particular perspective than to a single absolute truth.

The partial dependence plot examined by the researchers is also a representative interpretation method. A partial dependence plot shows how a particular variable affects the predicted outcome while controlling, on average, for the influence of other conditions. For example, it can display as a curve how age affects an automobile insurance premium prediction. If the curve does not rise or fall sharply with age, it may create the impression that the model barely uses age in a discriminatory way.

However, this plot can include predictions for combinations of data that do not exist in reality or are extremely rare. When variables are closely related to one another, changing only one variable independently can produce hypothetical cases that lie outside the distribution of real-world data. A model can retain discriminatory predictions for actual observations while adjusting predictions in these hypothetical regions to make the overall graph appear flat and neutral.

The researchers designed an adversarial method that exploited precisely this vulnerability. They changed predictions in the extrapolation regions referenced by the explanation tool while leaving almost all of the original model’s predictions for real-world cases intact. As a result, a model that was actually disadvantageous to a particular group appeared neutral in the partial dependence plot. The model’s discriminatory behavior had not disappeared; it had merely become invisible on the explanation screen.

More troubling is the fact that this problem does not necessarily arise only through malicious manipulation. The research demonstrated that explanations could be deliberately deceived, but the same vulnerability can also emerge through incorrectly chosen analytical ranges, insufficient understanding of correlations within the data, inappropriate visualizations, and the mechanical application of convenient interpretation tools. Even when developers or managers have no intention of concealing bias, an incomplete explanation may still be accepted as evidence of fairness.

Explanations help people, but they also lower their guard. When users confront a system that offers no explanation at all, they begin with suspicion. When a plausible reason is provided, however, they may feel there is no longer any need to ask questions. Because of this psychological effect, an inaccurate explanation can be more dangerous than no explanation. An opaque AI becomes an object of caution, while a poorly explained AI becomes an object of trust.

In discussions of AI governance, transparency, explainability, fairness, and accountability are often grouped together. They are all principles of trustworthy AI, but they are not interchangeable. Explainability is the quality that enables people to understand how AI reached a decision. Fairness concerns whether that decision imposes unjust disadvantages on particular individuals or groups. Accountability concerns the structure that determines who will review and correct a problematic decision and take responsibility for it.

An explainable model can still be unfair. Suppose an AI trained on historical hiring data evaluates male applicants as having a higher likelihood of strong performance. The system may clearly explain that years of service, area of experience, previous job title, and participation in particular activities were important variables. Yet if those variables are structurally connected to historical gender inequality, the result may still be unfair even though the explanation is clear. An explanation can reveal the path through which discrimination operates, but it does not automatically eliminate discrimination.

Removing sensitive variables does not solve the problem either. Even if gender or race is excluded from the input data, proxy variables such as residential area, school, employment history, and income level may remain closely related to those sensitive characteristics. AI can use combinations of such variables to reconstruct information that was supposedly removed. The explanation screen may not display an item labeled “gender,” while the model still produces similar disparities by using information strongly associated with gender.

Insurance pricing reveals the same problem. Insurers seek to predict the likelihood of accidents accurately and set prices according to risk. However, some variables that improve accuracy may be difficult for individuals to control or may be linked to socioeconomic inequality. A postal code may reflect traffic conditions or accident rates, but it may also indirectly reflect income and racial segregation. Even if the system honestly explains that a postal code affected the premium, whether that price is socially fair must be judged separately.

Fairness itself is difficult to define with a single number. Results differ depending on whether the goal is to equalize approval rates across groups, give the same probability of approval to people with equal repayment ability, or equalize error rates across groups. Strengthening one fairness criterion can weaken another. Fairness is therefore not a technical property that an interpretation tool can automatically determine. It is a normative question that an organization must address by considering its objectives, impacts, legal obligations, and social values.

The relationship between accuracy and fairness also requires careful examination. AI development organizations often optimize overall predictive accuracy. Yet even when overall accuracy is high, errors may be concentrated within a minority group. If a hiring AI evaluates 95% of applicants accurately but repeatedly rejects qualified candidates from a particular group, the quality of the system cannot be assessed solely through average performance. Overall averages can conceal differences among groups.

Explanation tools also frequently rely on averages. A partial dependence plot shows the average effect of a particular variable on predictions. A stable average curve does not mean that the model behaves in the same way for every group and individual. Group-specific effects moving in opposite directions may cancel one another out in the average. A variable that substantially disadvantages some people and benefits others may appear to have almost no effect in the overall graph.

The researchers examined these vulnerabilities using automobile insurance claims data and data from COMPAS, a system used to predict the risk of criminal recidivism. COMPAS became a prominent case in debates over algorithmic bias because it was used in the criminal justice system to assess defendants’ risk of reoffending. The findings showed that discriminatory relationships could be concealed by manipulating partial dependence plots while preserving most actual predictions. Even when the interpretation tool presented a neutral picture, the model’s real decisions could remain unfavorable to a particular group.

The significance of these results goes beyond the possibility of a technical attack. Many organizations use explanation reports or variable-importance graphs as evidence of AI fairness. If auditors and executives review a few visualizations and conclude that sensitive variables do not substantially affect the outcome, they may overlook discrimination occurring in actual data. The illusion begins when an explainability tool moves from being an aid to verification to becoming a seal that certifies fairness.

Transparency is one of the conditions needed to examine fairness. If no one can understand what is happening, discovering problems becomes difficult. However, the fact that something can be seen does not mean that what is visible is accurate or sufficient. Organizations must also examine what an explanation shows and does not show, which data conditions the explanation method assumes, and whether other interpretation tools produce the same conclusion.

When companies adopt AI, the first thing they seek is an actionable standard. The principle of building fair and responsible AI is correct but abstract. In contrast, a requirement to provide an explanation function is relatively concrete. An organization can generate variable-importance reports, inform users of the reasons for decisions, and build model-explanation dashboards. Whether these measures have been implemented can be checked. For this reason, explainability can easily become the representative indicator of responsible AI.

However, when an easily measured indicator replaces an important objective, organizational behavior begins to conform to the indicator. Producing an explanation report may take priority over thoroughly examining the social impact of AI. Submitting a graph that looks fair may be easier than creating fair outcomes. If explainability is reduced to a box on a checklist, AI governance can turn from an effort to reduce actual risk into a procedure for completing documents.

Under such circumstances, transparency can be used as a shield against accountability. A company can claim that it disclosed which information the AI used, explained the reasons for its results to users, and applied interpretation tools internally. Even when a problem occurs, the organization may present the fact that the system was “explainable” as evidence that it fulfilled its responsibilities. If the explanation was inaccurate or incomplete, however, formal disclosure has replaced substantive accountability.

Responsibility may also be shifted onto users who receive explanations. A line of reasoning may develop in which the company provided the grounds for the decision, so users are responsible for understanding them and filing objections. Yet people without technical expertise find it difficult to verify complex variable-importance measures and probability figures. Receiving an explanation is different from having a meaningful ability to challenge a decision. If an organization merely provides an explanation without creating review procedures or remedies, transparency does not protect users’ rights.

Disclosing more information is not always better. Excessively complex explanations can obscure the central issue. When dozens of graphs, technical terms, and model-performance indicators are presented, an organization may appear transparent, but users may not know what they are supposed to examine. The problem may arise not because information is absent but because there is too much of it and it is not structured in a way that supports judgment. More important than the quantity of disclosure is providing the information needed for a decision in a form that can be understood and verified.

Commercial interests must also be considered. A company that developed a model wants to demonstrate that its AI is accurate and fair. If the model operator is also the party selecting the explanation tool, defining the scope of analysis, and visualizing the results, there is an incentive to present only favorable explanations. Even without deliberate manipulation, organizations may prefer results that appear stable over results that expose problems. Uncomfortable indicators may be omitted from internal reports or buried in averages.

This phenomenon can lead to “transparency washing.” Like greenwashing, which exaggerates environmental performance, transparency washing creates an image of responsible AI by emphasizing explanation functions and ethical principles without making substantive improvements. Model cards, ethical declarations, and explanation dashboards are necessary, but they do not by themselves eliminate bias in data or harm caused by decisions. Unless organizations verify that disclosed materials faithfully reflect actual operations, transparency can become a public-relations tool for gaining trust.

The research using automobile insurance and criminal justice data showed that explanation tools could also be deceived technically. If discriminatory predictions can remain almost unchanged while an explanation that looks fair is produced, a procedure that approves a model based only on an explanation report is insufficient. Auditors must do more than consume explanation results. They must examine how the explanation was produced, the distribution of the data, the extrapolation region, and correlations among variables.

The role of executives must also change. The question “Is this AI explainable?” is too broad and too simple. Executives should ask what explanations are being provided, to whom, whether those explanations correspond to actual model behavior, whether they have been independently verified using other methods, and whether people harmed by the outcome can challenge the decision. The existence of explanatory materials should not be enough to justify approval of the system’s accountability.

Regulators also cannot rely solely on mandating the use of a particular explanation tool. Once a standard tool becomes a regulatory requirement, companies may focus on obtaining favorable results from that tool. As the research demonstrated, every interpretation method has its own assumptions and vulnerabilities. Regulation should require multiple layers of accountability, including verifiability of outcomes, assessment of discriminatory impact, record preservation, external auditing, and the right to appeal, rather than focusing only on the format of explanations.

Explanations are necessary for trustworthy AI, but they are not sufficient. A system is needed that combines explanations with other evidence and verifies them. Organizations must jointly manage data, performance, fairness, human intervention, and remedies for harm from the development stage through deployment and operation. Continuous verification, rather than a one-time explanation report, must become the center of AI governance.

First, organizations should not rely on a single explanation method. Partial dependence plots, variable importance, individual prediction explanations, and counterfactual explanations answer different questions. The absence of a sensitive relationship in one tool does not mean that another method will produce the same result. Multiple interpretation methods should be applied together, and their results should be compared for consistency. When they differ, organizations should analyze why the difference arose rather than hastily selecting one explanation as correct.

Second, the assumptions underlying explanation tools must be examined. Organizations need to determine whether variables are strongly correlated, whether predictions are being extended into regions where actual data do not exist, and whether averages are concealing group-specific differences. When partial dependence plots are used, realistic data regions and extrapolation regions should be distinguished, and explanations drawn from data-sparse ranges should not be treated as evidence of fairness.

Third, actual outcomes must be measured directly instead of relying on explanations. Comparing group-level approval rates, rejection rates, false-positive rates, false-negative rates, prediction errors, and cost burdens reveals who is affected by a model and how. For an insurance AI, the organization should examine how premiums and errors differ by gender, age, region, and income level. For a hiring AI, it should assess not only group-specific pass rates but also whether people with similar actual performance receive equal opportunities.

Fourth, average performance must be disaggregated. Overall accuracy and average loss values can conceal problems affecting particular groups. Errors involving a small minority group have little impact on the overall indicator and may therefore persist unnoticed. Organizations should measure performance separately for important subgroups and explicitly identify groups for which reliable evaluation is difficult because of insufficient data. The fact that something was not measured must not be interpreted as evidence that no problem exists.

Fifth, model development and validation teams should be separated. A structure in which the people who created a model also explain it and approve its fairness creates a conflict of interest. An independent internal audit team or external experts should review the data and code, reproduce the results, and apply alternative explanation methods. The higher the risk posed by an AI system, the more important the independence and expertise of the validation body become.

Sixth, verification must not end before deployment. As the operating environment changes, the performance and impact of AI also change. Economic downturns, labor-market shifts, the arrival of new customers, and changes in data-collection methods can alter the distribution of inputs. A variable that caused no apparent problem during development may later begin to disadvantage a particular group. Organizations need regular reassessment, real-time monitoring, and procedures for suspending a model or reverting to a previous method when anomalies are detected.

Seventh, human review must be designed as a substantive process. Merely requiring a person to press the final approval button is not enough. The reviewer must have the authority, time, information, and expertise needed to overturn the AI’s decision. If following the AI’s recommendation is regarded as the safest choice within the organization, the human remains no more than a ceremonial approver. Operating principles must also ensure that rejecting or modifying an AI recommendation does not result in a disadvantage.

Eighth, explanations must be connected to actions that users can take. It is not enough to tell someone why a loan was denied. The organization should explain which information can be corrected or supplemented and how the person can request reconsideration. If incorrect data were used, users must be able to correct them and challenge the automated decision. Unless an explanation is connected to a remedy, it remains little more than an incomprehensible retrospective notice.

Ninth, important decisions must be documented. Organizations should record which model and data were used, when the model was updated, which explanation tool was applied, and why a human reviewer approved or changed the result. When a problem occurs, the decision path must be traceable so that responsibility can be identified and recurrence prevented. Without records, the explanation disappears with the screen on which it was displayed, and the organization loses its capacity for retrospective verification.

Tenth, organizations must also determine the areas and conditions in which AI should not be used. Even with stronger explanations and audit procedures, automation may need to be restricted for decisions in which harm is severe and errors are difficult to reverse. When a decision has major consequences for liberty, life, health, or livelihood, AI may need to remain in a supporting role. The scope of automation should not be expanded simply because a system can technically provide an explanation.

Such a verification system requires time and money. Yet the efficiency of AI should not be calculated while excluding the cost of risk management. When the damage caused by wrongful loan denials, discriminatory hiring, unfair insurance premiums, and misdiagnoses?as well as litigation, reputational decline, and regulatory penalties?is considered, pre-deployment verification and continuous auditing are not additional burdens. They are part of the cost of operation.

Discussions of AI transparency often focus on how to increase trust. The expectation is that if users understand how AI reaches decisions, they will feel more comfortable accepting and using the system. Trust is also an important asset for companies seeking to adopt and expand AI. However, setting increased trust as the sole objective of AI governance can lead in a dangerous direction.

A high level of trust is not always desirable. Strong trust in a poorly performing or biased system can amplify harm. By contrast, cautious skepticism can help users discover errors and preserve human judgment. What is needed is not unconditional trust but appropriately calibrated trust that matches the system’s actual capabilities and limitations. People should be able to use AI for tasks it performs well, verify its conclusions in uncertain areas, and challenge it when decisions carry substantial risk.

Explanations should function not as devices for increasing trust but as instruments for calibrating it. They should communicate what AI knows, what it does not know, the conditions under which its performance deteriorates, and the limitations of the explanation itself. Providing uncertainty and alternative interpretations may be more responsible than presenting only confident reasons. The purpose should not be to make users trust AI more, but to help them decide when to trust it and when to remain skeptical.

The intended audience for an explanation must also be distinguished. Developers need technical information that allows them to diagnose variable interactions and model errors. Executives need summaries that help them judge whether AI aligns with organizational goals and risk standards. Auditors and regulators need access to raw data, validation methods, group-specific performance, and change histories. Consumers need to understand the central factors that affected their decision and how to challenge it. Providing everyone with the same graph may represent an abandonment of responsibility rather than transparency.

Executives should not treat AI explanations like performance reports. If they ask only, “What is the most important variable?” they may accept the ranking produced by the explanation tool as fact. They must also ask, “What assumptions does this explanation use?”, “Does it include combinations of data that do not exist in reality?”, “Would the result change if analyzed separately by group?”, and “Do other tools produce the same conclusion?” The level of questioning determines the level of governance.

Fairness cannot be left solely to the technical team. Determining what is fair is a question of data science, but it is also a matter of management, law, ethics, and consumer protection. Developers can calculate error rates, but the organization and society must decide which errors are more serious. Wrongfully rejecting a person who is capable of repaying a loan and granting a loan to someone who may struggle to repay it create different harms. Misdiagnosis by a medical AI also has different effects depending on the disease and the patient population. This is why diverse stakeholders must jointly design fairness standards.

People who are subject to AI decisions should also be included in the governance process. An internal technical review cannot identify every problem that actual users experience. Organizations must gather feedback on whether explanations are understandable, whether decisions can be challenged, and whether particular groups repeatedly experience disadvantages. A declaration of fair AI carries little persuasive power when the people who suffer harm are excluded from the validation process.

The most important message of the research is not that explanation tools should never be trusted. It is that explanation tools themselves must be subject to verification. Just as a thermometer must be checked for accuracy, a tool that explains AI must be examined to determine whether it faithfully represents the model’s actual behavior. An interpretation result must not be accepted as evidence of fairness merely because it is visually appealing and easy to understand.

Explainable AI is not a finished solution to the black box problem. It is a tool for beginning investigation and verification. Explanations should help organizations identify suspicious variables, compare outcomes across groups, modify models, and enable affected people to challenge decisions. If no action follows an explanation, transparency remains a decorative feature.

Companies should not design screens that persuade people to trust AI. They should design structures in which errors can be detected, bias can be corrected, and harm can be remedied with clear accountability. Fairness must be confirmed in the outcomes of real decisions, not in the shape of a graph. Accountability must be demonstrated through the authority and procedures used to trace and correct errors, not through declarations of ethical principles.

The fact that AI can state the reasons for its decisions is an important advance. Yet the next step must be to ask whether the explanation is accurate, whether it is sufficient, and whether it has been selectively constructed to serve a particular interest. The illusion begins when an explanation becomes a reason to demand trust. Transparency begins to perform its proper role only when explanations are used as material for verification and challenge.

The question required in the AI era cannot stop at “Can the system explain itself?” It must advance to “Can that explanation be independently verified?”, “Does the explanation correspond to actual outcomes?”, and “Can an incorrect decision be challenged and corrected?” Responsible AI is not AI that speaks persuasively. It is AI whose decisions can be tested. The value of transparency does not lie in gaining trust quickly, but in making it possible to continually determine whether that trust is deserved.

Knowledge at Wharton, May 2026, Knowledge at Wharton, When AI Transparency Backfires