How we test whether AI answers cite coverage
The questions, services, test dates and evidence behind every Earned Answers case, and what a case can and can't show.
Published
A case starts with one piece of earned coverage and a few buyer questions that don’t name the business. We put each question to several AI services, several times, on the day the case starts, a week later and four weeks later.
We count the answers that link to the coverage, apart from answers that only repeat its facts. Every answer is saved, and each case reports its counts and dates, including results we couldn’t reproduce.
What counts as a case
The coverage is an article, interview, survey write-up or listing published by someone other than the business. The business’s own website and paid placements don’t count.
| Term | Meaning |
|---|---|
| Citation | The answer links the coverage as a source |
| Reflection | The answer repeats a fact or phrase from the coverage without linking it |
We report reflections separately and never count them as citations.
The questions
We write the questions before the first test and don’t change them afterward. A new wording is a new question with its own ID.
Each question is phrased the way a buyer, journalist or researcher would ask it, without naming the outlet or the coverage. A question that names the business is labeled branded and reported apart from the others.
The services
We use each AI service only in ways its terms allow: through the provider’s API, or by reading Google’s AI results by hand in a clean, signed-out browser. Every result is labeled API or manual, and the two are never added together.
An API answer can differ from what the same company’s app shows people, and each case says which route every result came from. For each answer we record the service, the model it reported, the country and language settings, and the time in UTC. A service we can’t test within its terms is listed as not tested.
Test days and results
We test on day 0, day 7 and day 28 after a case starts. Each case shows how many answers cited the coverage on each date, for example 3 of 5, next to one of these labels:
| Label | Meaning |
|---|---|
| Reproduced | Cited on day 28 |
| Partly reproduced | Cited on day 7 but not on day 28 |
| Not reproduced | Cited on day 0 only |
| Not cited | No answer cited the coverage on day 0 |
A coverage piece that was never cited can still be worth publishing as a finding.
The evidence we keep
- Every answer in full, its cited links in order and the raw response, plus a screenshot for answers read by hand.
- An archived copy of the coverage, made the day we log it, with its publication date as printed.
- How the coverage came about, from public sources only. Anything we can’t find stays “unknown”.
What a case can’t show
One set of tests from one country is a sample. People elsewhere, signed in, or asking on another day can see different answers.
A citation shows that an answer used the coverage. It doesn’t show that the coverage changed anyone’s decision.
Corrections
If a business, an outlet or a reader shows that a case is wrong, we correct the page, add a dated note and keep the original records. Our corrections policy explains how to report one.
Changes to this method
- : First version.