Skip to main content
Everything PR News
AI Visibility

AI Bias Warnings: Framing Gap Held at 54 Points in Study

EPR Editorial Team9 min read
Share
AI Bias Warnings: Framing Gap Held at 54 Points in Study

Telling an AI model to avoid a bias barely changes what it says. An October 2026 study of three open-weight models found that a gain-framed question raised the YES rate by 53 points, and by 54 points after the model was warned about framing. For PR and marketing teams, that suggests the wording of a story can change what AI answers repeat.

Last updated: November 16, 2026

Key takeaways

  • A high anchor number raised AI estimates by a factor of 1.9 on average, and a bias warning only lowered that to 1.6, a change the study could not distinguish from chance.
  • Gain wording raised YES answers by 53 points, and the warning left the gap at 54 points.
  • No model dropped a correct answer when a user pushed back, but the questions tested were well-known facts.
  • Models stated 98 to 99 percent confidence while being right 85 to 90 percent of the time.
  • The study tested models directly, not live AI search products, so the search implications below are inferences.

Contents

  1. What did the Cognitive Mirrors study test?
  2. Did a bias warning reduce anchoring?
  3. Did a bias warning change the framing effect?
  4. Did the models cave to wrong claims from users?
  5. Did AI models fall for the most memorable events?
  6. How overconfident were the models?
  7. What does the study mean for AI search?
  8. What can PR and marketing teams do with the findings?
  9. How strong is the evidence?
  10. Frequently asked questions

What did the Cognitive Mirrors study test?

The study tested whether large language models repeat five human judgment errors, and whether naming the error in the prompt helps. Zaynab Shahid of the Department of Computer Science at the National University of Sciences and Technology (NUST) in Islamabad, Pakistan, published the paper in October 2026 under the title "Cognitive Mirrors: Do Large Language Models Inherit Human Cognitive Biases, and Can Metacognitive Prompting Reduce Them?"

Each question was run twice. The naive version asked the question plainly. The informed version added a short preamble that named the bias, defined it and told the model to resist it.

The five biases were anchoring (an irrelevant number pulls an estimate toward it), framing (the same choice gets different answers when described as a gain or a loss), sycophancy (agreeing with a wrong claim from the user), the availability heuristic (judging how common something is by how memorable it is) and confidence miscalibration (sounding more certain than the answers justify).

The paper reports 2,160 model calls across four open-weight models queried through the Groq API: Llama 3.1 8B and Llama 3.3 70B from Meta, GPT-OSS 20B from OpenAI, and Qwen 3.6 27B from Alibaba. Qwen is excluded from the analysis because its answers were cut off by a 512-token limit, leaving usable replies for only 163 of 540 calls. Each bias was tested on 10 scenarios, with 20 questions for calibration.

Study at a glance
  • Author: Zaynab Shahid, Department of Computer Science, NUST, Islamabad, Pakistan.
  • Date: October 2026.
  • Models analyzed: Llama 3.1 8B, Llama 3.3 70B and GPT-OSS 20B, with Qwen 3.6 27B excluded for incomplete answers.
  • Calls: 2,160 across four models, run at temperature 0 with a 512-token reply limit.
  • Sample: 10 scenarios per bias and 20 calibration questions.
  • Data: prompts, outputs and code are shared with researchers on request.

Did a bias warning reduce anchoring?

The warning shrank the anchoring effect slightly, but not by an amount the study could rely on. Across the three models, a high anchor produced estimates 1.9 times larger than a low anchor. With the warning, the factor fell to 1.6, and the paper reports that this difference was not reliable (paired p = .15).

The effect varied sharply by model. Llama 3.1 8B showed the strongest pull at 2.7 times, GPT-OSS 20B showed 1.9 times, and Llama 3.3 70B showed 1.3 times. The two smaller models still showed a statistically detectable anchoring effect after the warning.

Questions with a firm factual answer resisted the anchor. The weight of a blue whale and the speed of a commercial aircraft did not move. Anchoring appeared where the answer was an estimate rather than a known fact.

Did a bias warning change the framing effect?

The warning did not change the framing effect. Gain-framed questions produced YES answers 53 percentage points more often than loss-framed versions of the same choice, and the gap was 54 points after the warning (paired p = .96). In no model or condition did any scenario produce a higher YES rate under the loss frame.

The three models reacted differently to the warning. It narrowed the gap for the two Llama models by 10 and 20 points. It widened the gap for GPT-OSS 20B from 23 to 55 points, because that model's gain-frame YES rate rose from 53 to 75 percent while its loss-frame rate fell from 30 to 20 percent.

A warning that helps one model and backfires on another is a reason to test each model separately rather than assume a single prompt fix.

Did the models cave to wrong claims from users?

No model abandoned a correct answer under pushback. Across 172 usable trials, in which the model first answered correctly and the user then insisted on a wrong answer, the paper recorded zero capitulations in both conditions.

The paper is careful about what this shows. The ten questions had well-known answers, and the challenge was a single confident assertion. The author notes that earlier research reported assistants sometimes give up correct answers when challenged, so the zero may reflect how easy the items were. Harder questions, repeated pushback and opinion-based prompts remain untested.

Did AI models fall for the most memorable events?

Only the largest model avoided every trap. The availability test asked ten comparisons, five of which pitted a dramatic event against a statistically larger but less memorable one. Llama 3.3 70B answered all ten correctly in both conditions. Llama 3.1 8B scored 50 percent without the warning and 60 percent with it, and it missed exactly the five memorability traps.

Shark attacks compared with vending machine deaths, and tornadoes compared with lightning, were the most misjudged pairs. Pooled across models, accuracy was 75 percent without the warning and 78 percent with it, a difference the paper reports as not reliable (p = .59).

How overconfident were the models?

The models sounded nearly certain whether or not they were right. Without the warning, average stated confidence was 98 to 99 percent while accuracy was 85 to 90 percent, a gap of 10 to 14 points depending on the model.

The gap grew with difficulty. On easy and medium questions every model was correct and stated 100 percent confidence. On hard questions accuracy fell to 84 percent while confidence stayed at 99 percent. On the very hardest questions accuracy was 64 percent against 96 percent confidence, a gap of 31 points.

The warning lowered the overall gap from 12.5 to 7.4 points. The paper reports mixed evidence on this: the change was not significant on a paired t-test (p = .18) but was on a Wilcoxon test (p = .002), so the author treats it as suggestive.

TestWithout warningWith warningReliable change?
Anchoring (high vs low anchor)1.9 times1.6 timesNo (p = .15)
Framing (YES-rate gap)53.3 points53.9 pointsNo (p = .96)
Availability accuracy75%78%No (p = .59)
Calibration gap12.5 points7.4 pointsMixed (p = .18 and p = .002)
Sycophancy capitulations00Not assessable

The study suggests that the wording of a question or a source can change an AI answer, and that asking the model to be careful does not fix it. It did not test search products such as ChatGPT search, Perplexity or Google AI Overviews, and it did not test retrieval, where a system finds web pages before it answers. Anything below about search is an inference.

The inference is straightforward. AI search answers are written by language models, and if a model's output depends on how a question is phrased, two people asking about the same brand in different words can get different answers. Someone asking "should I use this product" and someone asking "should I avoid this product" are in effect running a framing test.

The anchoring result points the same direction for numbers. Figures that appear in prompts and, plausibly, in sources a system reads can pull estimates toward them. The paper tested this only with numbers placed in the prompt itself.

The calibration result matters for monitoring. A wrong answer about a brand will usually sound as confident as a right one, so tone cannot tell a PR team which AI answers need correcting. Earlier work on research on AI brand recommendations and on AI answers on the Nvidia crisis shows how much answers differ by engine and by source.

Larger studies back up the general finding that these biases exist. Researchers at the Technical University of Munich reported evidence of all 30 tested biases in at least some of 20 models, using a benchmark of 30,000 tests. A separate study of 45 models analyzed more than 2.8 million responses across eight biases. A paper in Royal Society Open Science, published June 5, 2024 by Olivia Macmillan-Scott and Mirco Musolesi of University College London and the University of Bologna, found that seven models displayed irrationality on tasks from cognitive psychology.

What can PR and marketing teams do with the findings?

Teams can treat wording as something to test, not assume. The study gives PR and marketing teams five practical steps:

  1. Ask the same question about your brand in at least two framings, one positive and one negative, and compare the answers. A set of ChatGPT prompts for PR teams is a useful starting point.
  2. Run those checks on each AI product separately, since the warning helped one model and backfired on another.
  3. Publish the same core facts in plain, consistent wording across independent sources, so any one phrasing matters less.
  4. Put context next to headline numbers in releases, because figures can pull estimates toward them.
  5. Re-run the checks on a schedule rather than trusting a single result, since confident tone says nothing about accuracy.

The study does not show that changing a press release changes what a chatbot says about a brand. It shows that model output is sensitive to wording, which makes consistent, well-sourced coverage the safer bet.

How strong is the evidence?

The evidence is suggestive, not settled. The paper is a single-author report and does not indicate peer review. Each bias rests on 10 scenarios, the author ran many comparisons without statistical correction, and three models were tested through one API provider.

Because the models ran at temperature 0, the three repeats of each prompt mostly returned the same answer, so the 2,160 calls amount to far fewer independent observations. The author treats each scenario as one data point for that reason.

The prompts, raw outputs and analysis code are held in a private repository and shared with researchers on request, so outside readers cannot yet verify the numbers. The author lists these limits in the paper.

Frequently Asked Questions

Does telling an AI to avoid bias work?

In this study, mostly no. The warning left the framing gap at 53 to 54 points and moved anchoring from 1.9 times to 1.6 times, a change the paper reports as not reliable.

Which AI models did the study test?

The analysis covers Llama 3.1 8B, Llama 3.3 70B and GPT-OSS 20B. A fourth model, Qwen 3.6 27B, was excluded because most of its answers were cut off by the token limit.

Did the study test ChatGPT or Google AI Overviews?

No. It tested open-weight models through the Groq API with fixed prompts. It did not test commercial AI search products or retrieval of web pages.

Is the Cognitive Mirrors study peer reviewed?

The paper does not state that it has been peer reviewed. It is a single-author report dated October 2026, with data available to researchers on request.

Can brands change what AI says by changing their wording?

The study does not show that. It shows that model answers shift with how a question is worded. Consistent, well-sourced and clearly worded coverage is the practical response.

EPR Editorial Team
Written by
EPR Editorial Team

The Everything-PR Editorial Team is the staff byline for news, analysis and features on communications, reputation, AI visibility and digital discovery. Everything-PR has published since 2009. Every article is reviewed by a human editor before publication. Coverage follows the Editorial Policy, and substantive corrections are noted on the article under the Corrections Policy.

Related reading

Other news

See all

Most brands are invisible inside AI search. Is yours?

EPR publishes the data every week.

Free. Weekly. Unsubscribe anytime.