NLI/LLM Funnel Report

Narrowing down citing works that support or contradict a Background Message

Published

August 18, 2026

SUPPORTS Funnel — IAS

Methodology

Every work discovered by the snowball search under a Background Message (BM) is put through a 3-level sieve, each level a subset of the one before it, counted as distinct citing works per (assessment, km, bm) — a work cited under two different BMs is two separate findings, not one, so per-BM counts sum to the assessment-wide total shown below.

  1. Snowball corpus — every citing work discovered for that BM (the forward/citing direction of the snowball search).
  2. NLI SUPPORTS (certain) — of those, works with at least one claim scored label == "SUPPORTS" with uncertain == FALSE by the zero-shot NLI model (Phase 1 — see TD_BM_NLI_approach).
  3. LLM-confirmed SUPPORTS — of those, works where OpenRouter’s LLM review (Phase 2 — see TD_NLI_LLM_two_phase) independently agreed with the SUPPORTS verdict (llm_agrees == TRUE).

There is deliberately no further “+ sufficient evidence” level: Phase 2’s own verification logic forces its llm_label to NOT_ENOUGH_INFO whenever it judges the evidence insufficient, so an LLM verdict of SUPPORTS already implies sufficient evidence — a separate level here would always contain exactly the same works as level 3.

Claim granularity: sub_bm. sub_bm (the default) segments each BM’s bm_description/bm_label into evidence-delimited sub-claims (2-17 per BM) at the sentence level. complete_bm instead treats each field as a single, whole claim (2 per BM, or 1 if bm_label/bm_description are identical) — cheaper to score (~71% fewer NLI pairs measured across both assessments), at the cost of losing which specific part of a longer BM a citing work actually supports or contradicts, and of the resulting hypothesis text sometimes exceeding the NLI model’s own token limit (see TD_BM_NLI_approach.qmd for the measured numbers and caveats). Level 2/3 counts below reflect whichever granularity this report was rendered for; the two are not directly comparable work-for-work, since they’re scored against different claim text.

Phase 2 (LLM verification) is a partial, ongoing process — OpenRouter credit has run out mid-run at least once already, so level 3 below reflects only the pairs reviewed so far, not the full SUPPORTS-certain set from level 2. This number will grow as llm_verification_parquet gets more OpenRouter runs; re-render this report after a rerun rather than treating today’s snapshot as final.

Show the code 
if (isTRUE(x$empty)) {
  cat("No data available yet for this assessment — the snowball, NLI, or LLM verification stage hasn't produced output here.\n\n")
  knitr::knit_exit()
}

Overall numbers

Show the code 
knitr::kable(
  x$funnel_overall,
  col.names = c("Level", "Description", "Distinct works")
)
Level Description Distinct works
level1 Snowball corpus 177357
level2 NLI SUPPORTS (certain) 10340
level3 LLM-confirmed SUPPORTS 0
Show the code 
if (length(fig_overall)) cat(sprintf("![](%s)\n\n", fig_overall))

Per-BM numbers

Show the code 
knitr::kable(
  x$funnel_by_bm,
  col.names = c(
    "KM", "BM", "Corpus", sprintf("NLI %s", params$label), "LLM-confirmed",
    "% 2 of 1", "% 3 of 2"
  )
)
KM BM Corpus NLI SUPPORTS LLM-confirmed % 2 of 1 % 3 of 2
KM-A1 A1 7151 483 0 6.8 0
KM-A2 A2 9599 1397 0 14.6 0
KM-A2 A3 7125 491 0 6.9 0
KM-A3 A4 9813 359 0 3.7 0
KM-A3 A5 5444 127 0 2.3 0
KM-A4 A5 5444 127 0 2.3 0
KM-A4 A6 2943 43 0 1.5 0
KM-A5 A7 8613 277 0 3.2 0
KM-A5 A8 2401 54 0 2.2 0
KM-B1 B11 5302 51 0 1.0 0
KM-B1 B12 5179 200 0 3.9 0
KM-B1 B14 6085 623 0 10.2 0
KM-B1 B9 4608 128 0 2.8 0
KM-B2 B10 929 19 0 2.0 0
KM-B3 B11 5302 51 0 1.0 0
KM-B3 B12 5179 200 0 3.9 0
KM-B3 B14 6085 623 0 10.2 0
KM-B3 B9 4608 128 0 2.8 0
KM-B4 B10 929 19 0 2.0 0
KM-B4 B13 5228 630 0 12.1 0
KM-B4 B14 6085 623 0 10.2 0
KM-C1 C15 602 63 0 10.5 0
KM-C1 C16 2203 134 0 6.1 0
KM-C1 C17 2580 74 0 2.9 0
KM-C1 C18 1823 69 0 3.8 0
KM-C1 C22 2359 342 0 14.5 0
KM-C1 C23 2413 74 0 3.1 0
KM-C2 C15 602 63 0 10.5 0
KM-C2 C17 2580 75 0 2.9 0
KM-C2 C18 1823 69 0 3.8 0
KM-C3 C19 2170 49 0 2.3 0
KM-C4 C20 1576 119 0 7.6 0
KM-C5 C21 3676 390 0 10.6 0
KM-C6 C23 2413 74 0 3.1 0
KM-C6 C24 884 25 0 2.8 0
KM-D1 D25 1768 92 0 5.2 0
KM-D2 D26 2500 68 0 2.7 0
KM-D2 D30 2739 12 0 0.4 0
KM-D3 D27 1768 24 0 1.4 0
KM-D3 D28 1098 14 0 1.3 0
KM-D4 D26 2500 68 0 2.7 0
KM-D4 D33 1423 150 0 10.5 0
KM-D5 D31 2903 278 0 9.6 0
KM-D5 D32 3802 267 0 7.0 0
KM-D6 D29 3170 131 0 4.1 0
KM-D6 D31 2903 278 0 9.6 0
KM-D6 D32 3802 267 0 7.0 0
KM-D7 D32 3802 268 0 7.0 0
KM-D7 D33 1423 150 0 10.5 0
Show the code 
if (length(fig_by_bm)) cat(sprintf("![](%s)\n\n", fig_by_bm))

Normalized so every BM’s own snowball corpus reads 1 (dashed line) — lets BMs be compared on conversion rate regardless of raw corpus size.

Show the code 
if (length(fig_by_bm_norm)) cat(sprintf("![](%s)\n\n", fig_by_bm_norm))

Level 3: LLM-confirmed SUPPORTS

Every citing work where the LLM’s own independent review agreed with NLI’s SUPPORTS verdict (and, by construction, judged the evidence for it sufficient — see Methodology). Use the BM dropdown in the filter row below the header to narrow to one Background Message.

Show the code 
if (length(table_l3)) {
  cat(sprintf(
    '<iframe src="%s" width="100%%" height="800px" style="border:none;"></iframe>\n\n',
    table_l3
  ))
}