NLI/LLM Funnel Report

Narrowing down citing works that support or contradict a Background Message

Published

August 18, 2026

SUPPORTS Funnel — IAS

Methodology

Every work discovered by the snowball search under a Background Message (BM) is put through a 3-level sieve, each level a subset of the one before it, counted as distinct citing works per (assessment, km, bm) — a work cited under two different BMs is two separate findings, not one, so per-BM counts sum to the assessment-wide total shown below.

  1. Snowball corpus — every citing work discovered for that BM (the forward/citing direction of the snowball search).
  2. NLI SUPPORTS (certain) — of those, works with at least one claim scored label == "SUPPORTS" with uncertain == FALSE by the zero-shot NLI model (Phase 1 — see TD_BM_NLI_approach).
  3. LLM-confirmed SUPPORTS — of those, works where OpenRouter’s LLM review (Phase 2 — see TD_NLI_LLM_two_phase) independently agreed with the SUPPORTS verdict (llm_agrees == TRUE).

There is deliberately no further “+ sufficient evidence” level: Phase 2’s own verification logic forces its llm_label to NOT_ENOUGH_INFO whenever it judges the evidence insufficient, so an LLM verdict of SUPPORTS already implies sufficient evidence — a separate level here would always contain exactly the same works as level 3.

Claim granularity: complete_bm. sub_bm (the default) segments each BM’s bm_description/bm_label into evidence-delimited sub-claims (2-17 per BM) at the sentence level. complete_bm instead treats each field as a single, whole claim (2 per BM, or 1 if bm_label/bm_description are identical) — cheaper to score (~71% fewer NLI pairs measured across both assessments), at the cost of losing which specific part of a longer BM a citing work actually supports or contradicts, and of the resulting hypothesis text sometimes exceeding the NLI model’s own token limit (see TD_BM_NLI_approach.qmd for the measured numbers and caveats). Level 2/3 counts below reflect whichever granularity this report was rendered for; the two are not directly comparable work-for-work, since they’re scored against different claim text.

Phase 2 (LLM verification) is a partial, ongoing process — OpenRouter credit has run out mid-run at least once already, so level 3 below reflects only the pairs reviewed so far, not the full SUPPORTS-certain set from level 2. This number will grow as llm_verification_parquet gets more OpenRouter runs; re-render this report after a rerun rather than treating today’s snapshot as final.

Show the code 
if (isTRUE(x$empty)) {
  cat("No data available yet for this assessment — the snowball, NLI, or LLM verification stage hasn't produced output here.\n\n")
  knitr::knit_exit()
}

No data available yet for this assessment — the snowball, NLI, or LLM verification stage hasn’t produced output here.