Measure a campaign's effect on AI answers
A before-and-after workbook that shows whether coverage changed what AI answers say and cite, and when a change is too small to call.
Published
Ask the same questions before the campaign and again 7 and 28 days after the main coverage appears, then mark each answer for three checks. The workbook gives each change a 95% range, so a shift that everyday variation could explain doesn’t read as a win.
Download the campaign evaluation workbook (Excel workbook, 60 KB). It uses only functions that Excel, Google Sheets and LibreOffice share, and we tested it in LibreOffice.
What it measures
| Check | Passes when the answer |
|---|---|
| Brand named | names your brand |
| Cites your site | links to one of your websites |
| Cites your coverage | links to coverage you earned in this campaign |
Run three rounds
Run the Baseline round in the week before the campaign launches, then the Day 7 and Day 28 rounds after the main coverage appears. Change nothing between rounds except the date. Use the services listed in the source-opportunity log, for the same reasons.
Plan enough answers
Small samples can’t show small changes. If 1 in 10 answers cited your coverage at the Baseline, these are the smallest later results the workbook calls a clear rise:
| Answers per round | Baseline | Smallest clear rise |
|---|---|---|
| 12 | 1 of 12 | 6 of 12 |
| 30 | 3 of 30 | 10 of 30 |
| 60 | 6 of 60 | 15 of 60 |
| 120 | 12 of 120 | 23 of 120 |
Ten questions on three services, with two runs each, gives 60 answers a round.
Read the results
The Results sheet shows every check in every round:
| Column | What it shows |
|---|---|
| Share | Share of answers that pass, with a 95% range (Wilson score interval) |
| Change | Points up or down from Baseline, with a 95% range (Newcombe method 10) |
| Verdict | Clear rise or fall when the range doesn’t include zero. Otherwise, Can’t tell yet. |
Below that, the same comparison of Baseline and Day 28 for each question and each service.
Limits
- The ranges treat each answer as independent. Answers to one question tend to move together, so treat a borderline result with care.
- A rise shows the answers changed. It can’t show why: other coverage, model updates and competitors move answers too.
- Answers from a provider’s API can differ from what people see in its app.
Sources
- Interval estimation for the difference between independent proportions: comparison of eleven methods, Statistics in Medicine (Robert G. Newcombe, 1998). Accessed . The change range uses its method 10. We cite it for the method and haven't read the full paper.
- ProportionsDiffCIsNewcombe SAS macro, McGill University. Accessed . Its worked example (56 of 70 against 48 of 80) comes from the paper; our test suite checks the same example.