NLI/LLM Funnel Report

Narrowing down citing works that support or contradict a Background Message

Published

August 18, 2026

SUPPORTS Funnel — GA1

Methodology

Every work discovered by the snowball search under a Background Message (BM) is put through a 3-level sieve, each level a subset of the one before it, counted as distinct citing works per (assessment, km, bm) — a work cited under two different BMs is two separate findings, not one, so per-BM counts sum to the assessment-wide total shown below.

  1. Snowball corpus — every citing work discovered for that BM (the forward/citing direction of the snowball search).
  2. NLI SUPPORTS (certain) — of those, works with at least one claim scored label == "SUPPORTS" with uncertain == FALSE by the zero-shot NLI model (Phase 1 — see TD_BM_NLI_approach).
  3. LLM-confirmed SUPPORTS — of those, works where OpenRouter’s LLM review (Phase 2 — see TD_NLI_LLM_two_phase) independently agreed with the SUPPORTS verdict (llm_agrees == TRUE).

There is deliberately no further “+ sufficient evidence” level: Phase 2’s own verification logic forces its llm_label to NOT_ENOUGH_INFO whenever it judges the evidence insufficient, so an LLM verdict of SUPPORTS already implies sufficient evidence — a separate level here would always contain exactly the same works as level 3.

Claim granularity: sub_bm. sub_bm (the default) segments each BM’s bm_description/bm_label into evidence-delimited sub-claims (2-17 per BM) at the sentence level. complete_bm instead treats each field as a single, whole claim (2 per BM, or 1 if bm_label/bm_description are identical) — cheaper to score (~71% fewer NLI pairs measured across both assessments), at the cost of losing which specific part of a longer BM a citing work actually supports or contradicts, and of the resulting hypothesis text sometimes exceeding the NLI model’s own token limit (see TD_BM_NLI_approach.qmd for the measured numbers and caveats). Level 2/3 counts below reflect whichever granularity this report was rendered for; the two are not directly comparable work-for-work, since they’re scored against different claim text.

Phase 2 (LLM verification) is a partial, ongoing process — OpenRouter credit has run out mid-run at least once already, so level 3 below reflects only the pairs reviewed so far, not the full SUPPORTS-certain set from level 2. This number will grow as llm_verification_parquet gets more OpenRouter runs; re-render this report after a rerun rather than treating today’s snapshot as final.

Show the code 
if (isTRUE(x$empty)) {
  cat("No data available yet for this assessment — the snowball, NLI, or LLM verification stage hasn't produced output here.\n\n")
  knitr::knit_exit()
}

Overall numbers

Show the code 
knitr::kable(
  x$funnel_overall,
  col.names = c("Level", "Description", "Distinct works")
)
Level Description Distinct works
level1 Snowball corpus 290740
level2 NLI SUPPORTS (certain) 25295
level3 LLM-confirmed SUPPORTS 0
Show the code 
if (length(fig_overall)) cat(sprintf("![](%s)\n\n", fig_overall))

Per-BM numbers

Show the code 
knitr::kable(
  x$funnel_by_bm,
  col.names = c(
    "KM", "BM", "Corpus", sprintf("NLI %s", params$label), "LLM-confirmed",
    "% 2 of 1", "% 3 of 2"
  )
)
KM BM Corpus NLI SUPPORTS LLM-confirmed % 2 of 1 % 3 of 2
A. A1 23272 989 0 4.2 0
A. A2 28825 1979 0 6.9 0
A. A3 4891 86 0 1.8 0
A. A4 1915 129 0 6.7 0
A. A5 30856 2019 0 6.5 0
A. A6 33600 3420 0 10.2 0
A. A7 5904 57 0 1.0 0
A. A8 4546 577 0 12.7 0
A. A9 1992 76 0 3.8 0
B. B10 38022 3698 0 9.7 0
B. B11 365 28 0 7.7 0
B. B3 30477 380 0 1.2 0
B. B4 8409 415 0 4.9 0
B. B5 5858 964 0 16.5 0
C. C2 5325 424 0 8.0 0
C. C4 4904 101 0 2.1 0
C. C6 1670 30 0 1.8 0
C. C7 346 10 0 2.9 0
C. C8 977 28 0 2.9 0
D. D1 11477 1439 0 12.5 0
D. D10 1287 50 0 3.9 0
D. D11 9020 1946 0 21.6 0
D. D12 11288 1867 0 16.5 0
D. D2 9801 2164 0 22.1 0
D. D3 1715 509 0 29.7 0
D. D5 1194 197 0 16.5 0
D. D6 3139 230 0 7.3 0
D. D7 4903 800 0 16.3 0
D. D8 2659 451 0 17.0 0
D. D9 2103 232 0 11.0 0
Show the code 
if (length(fig_by_bm)) cat(sprintf("![](%s)\n\n", fig_by_bm))

Normalized so every BM’s own snowball corpus reads 1 (dashed line) — lets BMs be compared on conversion rate regardless of raw corpus size.

Show the code 
if (length(fig_by_bm_norm)) cat(sprintf("![](%s)\n\n", fig_by_bm_norm))

Level 3: LLM-confirmed SUPPORTS

Every citing work where the LLM’s own independent review agreed with NLI’s SUPPORTS verdict (and, by construction, judged the evidence for it sufficient — see Methodology). Use the BM dropdown in the filter row below the header to narrow to one Background Message.

Show the code 
if (length(table_l3)) {
  cat(sprintf(
    '<iframe src="%s" width="100%%" height="800px" style="border:none;"></iframe>\n\n',
    table_l3
  ))
}