1. What we measure
EvidenceSignal measures how AI engines — ChatGPT, Perplexity, Claude, and Gemini — describe and recommend supplement brands in response to real shopper queries. For each query, we capture: which brands each engine cites, the exact language used, the position in the answer, and the substantive claims attributed to each product.
We then check every claim the engines make against the largest indexed corpus of supplement regulatory enforcement in the category, classify it under the DSHEA structure-function vs disease taxonomy, and surface any precedent that suggests it may be challenged.
2. The regulatory corpus
| Source | Volume | Refresh |
|---|---|---|
| FDA Warning Letters | 6,717 total · 222 confirmed supplement | Weekly |
| NAD / BBB National Programs | 2,202 cases · 290 supplement | Weekly |
| FTC enforcement actions | 271 cases · 115 supplement-flagged | Weekly |
| NIH DSLD label database | 214,780 labels (pipeline built) | Monthly |
| TINA.org consumer callouts | 2,808 articles + 6,744 class actions (pipeline built) | Weekly |
3. Classification — the DSHEA taxonomy
- STRUCTURE_FUNCTION — permitted with disclaimer. "supports a healthy immune response"
- DISEASE_CLAIM — prohibited absent new-drug approval. "prevents UTIs"
- QUALIFIED_HEALTH_CLAIM — specific FDA-authorized language
- NUTRIENT_CONTENT_CLAIM — quantified nutrient claim with %DV/RDI
- GRAY_ZONE — ambiguous; requires regulatory-counsel review
Classification is performed by a prompt-based LLM classifier (v1) calibrated against a 26-case manual gold set. We run it against two engines (Gemini and GPT). On the gold set, agreement is 96.2%. Disagreement cases are flagged as gray-zone candidates for human review.
Every cited_claim entry in our database is annotated with the classification model, version, and disagreement flag. Replication tooling is available to anyone who wants to audit our scoring.
4. The Index
The Supplement AI Shelf Index aggregates citation share across all four engines for the two hundred most-prompted supplement shopper queries. For each query we record which brands each engine cites, position in the answer, and substantive context. We compute a per-query citation-share-weighted score (position decay: 1.0 / 0.7 / 0.5 / 0.35 / 0.25 / 0.18), aggregate equal-weight across engines, normalize to a category baseline (100.0 on April 1, 2026). Rebased nightly at 11:59 PM ET.
What the Index does not measure
- Shopper conversion or revenue impact
- Social-channel discovery or paid-search performance
- Compliance risk directly (that's per-claim, not Index-level)
- Engines we don't track (Microsoft Copilot, Brave Search AI, You.com)
5. Cross-validation
- Two-engine agreement is our gold standard. A claim is "confirmed" only when both extraction engines agree on classification.
- Replication is open. Extraction prompts, gold set, classification logs, and code are linkable on request.
6. Limitations — what we cannot do
- We cannot predict novel claim categories. Our classifier learns from historical enforcement.
- We cannot detect pre-publication enforcement. Form 483s, FTC investigations before consent-order publication, NAD complaints before decision — none of these are in our corpus.
- We cannot replace your General Counsel. We flag risk; we don't make legal calls.
- We cannot guarantee a litigation timeline. Our 30-day-ahead alert is actionable lead time, not prediction.
- We cannot measure private AI surfaces. Internal RAG, paid enterprise LLM deployments, customer-specific fine-tunes are outside scope.
7. Replication
We publish per-source extract code, classification prompts (v1), and our gold set on request. Contact us for the bundle.
Three reproducibility commitments: every chart shows source + date range; every methodology change bumps the version number; disagreement cases are reported alongside confident calls.
8. Version
Methodology v0.1 · published May 24, 2026.