Measure a campaign's effect on AI answers
A before-and-after workbook that shows whether coverage changed what AI answers say and cite, and when a change is too small to call.
Published
Ask the same questions before the campaign and again 7 and 28 days after the main coverage appears, then mark each answer for three checks. The workbook gives each change a 95% range, so a shift that everyday variation could explain doesn’t read as a win.
Download the campaign evaluation workbook (Excel workbook, 60 KB). It uses only functions that Excel, Google Sheets and LibreOffice share, and we tested it in LibreOffice.
What counts as an effect on AI answers
| Check | Passes when the answer |
|---|---|
| Brand named | names your brand |
| Cites your site | links to one of your websites |
| Cites your coverage | links to coverage you earned in this campaign |
Run three rounds
Run the Baseline round in the week before the campaign launches, then the Day 7 and Day 28 rounds after the main coverage appears. Change nothing between rounds except the date. Use the services whose terms allow it, as listed in the guide to finding the outlets AI answers cite.
Plan enough answers
Small samples can’t show small changes. If 1 in 10 answers cited your coverage at the Baseline, these are the smallest later results the workbook calls a clear rise:
| Answers per round | Baseline | Smallest clear rise |
|---|---|---|
| 12 | 1 of 12 | 6 of 12 |
| 30 | 3 of 30 | 10 of 30 |
| 60 | 6 of 60 | 15 of 60 |
| 120 | 12 of 120 | 23 of 120 |
Ten questions on three services, with two runs each, gives 60 answers a round.
Read the results
The Results sheet shows every check in every round:
| Column | What it shows |
|---|---|
| Share | Share of answers that pass, with a 95% range (Wilson score interval) |
| Change | Points up or down from Baseline, with a 95% range (Newcombe method 10) |
| Verdict | Clear rise or fall when the range doesn’t include zero. Otherwise, Can’t tell yet. |
Below that, the same comparison of Baseline and Day 28 for each question and each service.
Limits
- The ranges treat each answer as independent. Answers to one question tend to move together, so treat a borderline result with care.
- A rise shows the answers changed. It can’t show why: other coverage, model updates and competitors move answers too.
- Answers from a provider’s API can differ from what people see in its app.
Sources
- Interval estimation for the difference between independent proportions: comparison of eleven methods, Statistics in Medicine (Robert G. Newcombe, 1998). Accessed . The change range uses its method 10. We cite it for the method and haven't read the full paper.
- ProportionsDiffCIsNewcombe SAS macro, McGill University. Accessed . Its worked example (56 of 70 against 48 of 80) comes from the paper; our test suite checks the same example.