NLI/LLM Funnel Report

Narrowing down citing works that support or contradict a Background Message

REFUTES Funnel — IAS

Methodology

Every work discovered by the snowball search under a Background Message (BM) is put through a 3-level sieve, each level a subset of the one before it, counted as distinct citing works per (assessment, km, bm) — a work cited under two different BMs is two separate findings, not one, so per-BM counts sum to the assessment-wide total shown below.

1. Snowball corpus — every citing work discovered for that BM (the forward/citing direction of the snowball search).
2. NLI REFUTES (certain) — of those, works with at least one claim scored label == "REFUTES" with uncertain == FALSE by the zero-shot NLI model (Phase 1 — see TD_BM_NLI_approach).
3. LLM-confirmed REFUTES — of those, works where OpenRouter’s LLM review (Phase 2 — see TD_NLI_LLM_two_phase) independently agreed with the REFUTES verdict (llm_agrees == TRUE).

There is deliberately no further “+ sufficient evidence” level: Phase 2’s own verification logic forces its llm_label to NOT_ENOUGH_INFO whenever it judges the evidence insufficient, so an LLM verdict of REFUTES already implies sufficient evidence — a separate level here would always contain exactly the same works as level 3.

Claim granularity: sub_bm. sub_bm (the default) segments each BM’s bm_description/bm_label into evidence-delimited sub-claims (2-17 per BM) at the sentence level. complete_bm instead treats each field as a single, whole claim (2 per BM, or 1 if bm_label/bm_description are identical) — cheaper to score (~71% fewer NLI pairs measured across both assessments), at the cost of losing which specific part of a longer BM a citing work actually supports or contradicts, and of the resulting hypothesis text sometimes exceeding the NLI model’s own token limit (see TD_BM_NLI_approach.qmd for the measured numbers and caveats). Level 2/3 counts below reflect whichever granularity this report was rendered for; the two are not directly comparable work-for-work, since they’re scored against different claim text.

Phase 2 (LLM verification) is a partial, ongoing process — OpenRouter credit has run out mid-run at least once already, so level 3 below reflects only the pairs reviewed so far, not the full REFUTES-certain set from level 2. This number will grow as llm_verification_parquet gets more OpenRouter runs; re-render this report after a rerun rather than treating today’s snapshot as final.

if (isTRUE(x$empty)) {
  cat("No data available yet for this assessment — the snowball, NLI, or LLM verification stage hasn't produced output here.\n\n")
  knitr::knit_exit()
}

Overall numbers

knitr::kable(
  x$funnel_overall,
  col.names = c("Level", "Description", "Distinct works")
)

Level	Description	Distinct works
level1	Snowball corpus	177357
level2	NLI REFUTES (certain)	9327
level3	LLM-confirmed REFUTES	704

if (length(fig_overall)) cat(sprintf("![](%s)\n\n", fig_overall))

Per-BM numbers

knitr::kable(
  x$funnel_by_bm,
  col.names = c(
    "KM", "BM", "Corpus", sprintf("NLI %s", params$label), "LLM-confirmed",
    "% 2 of 1", "% 3 of 2"
  )
)

KM	BM	Corpus	NLI REFUTES	LLM-confirmed	% 2 of 1	% 3 of 2
KM-A1	A1	7151	768	74	10.7	9.6
KM-A2	A2	9599	586	161	6.1	27.5
KM-A2	A3	7125	196	10	2.8	5.1
KM-A3	A4	9813	1499	127	15.3	8.5
KM-A3	A5	5444	109	20	2.0	18.3
KM-A4	A5	5444	109	0	2.0	0.0
KM-A4	A6	2943	64	0	2.2	0.0
KM-A5	A7	8613	101	5	1.2	5.0
KM-A5	A8	2401	126	8	5.2	6.3
KM-B1	B11	5302	113	10	2.1	8.8
KM-B1	B12	5179	65	5	1.3	7.7
KM-B1	B14	6085	381	62	6.3	16.3
KM-B1	B9	4608	418	18	9.1	4.3
KM-B2	B10	929	25	2	2.7	8.0
KM-B3	B11	5302	113	0	2.1	0.0
KM-B3	B12	5179	65	0	1.3	0.0
KM-B3	B14	6085	381	0	6.3	0.0
KM-B3	B9	4608	418	0	9.1	0.0
KM-B4	B10	929	25	0	2.7	0.0
KM-B4	B13	5228	93	7	1.8	7.5
KM-B4	B14	6085	381	0	6.3	0.0
KM-C1	C15	602	35	20	5.8	57.1
KM-C1	C16	2203	107	19	4.9	17.8
KM-C1	C17	2580	169	7	6.6	4.1
KM-C1	C18	1823	36	1	2.0	2.8
KM-C1	C22	2359	350	19	14.8	5.4
KM-C1	C23	2413	29	4	1.2	13.8
KM-C2	C15	602	35	0	5.8	0.0
KM-C2	C17	2580	169	0	6.6	0.0
KM-C2	C18	1823	36	0	2.0	0.0
KM-C3	C19	2170	109	14	5.0	12.8
KM-C4	C20	1576	50	6	3.2	12.0
KM-C5	C21	3676	483	29	13.1	6.0
KM-C6	C23	2413	29	0	1.2	0.0
KM-C6	C24	884	67	14	7.6	20.9
KM-D1	D25	1768	17	0	1.0	0.0
KM-D2	D26	2500	70	10	2.8	14.3
KM-D2	D30	2739	4	0	0.1	0.0
KM-D3	D27	1768	5	0	0.3	0.0
KM-D3	D28	1098	29	3	2.6	10.3
KM-D4	D26	2500	69	0	2.8	0.0
KM-D4	D33	1423	30	4	2.1	13.3
KM-D5	D31	2903	168	6	5.8	3.6
KM-D5	D32	3802	257	32	6.8	12.5
KM-D6	D29	3170	227	7	7.2	3.1
KM-D6	D31	2903	168	0	5.8	0.0
KM-D6	D32	3802	256	0	6.7	0.0
KM-D7	D32	3802	257	0	6.8	0.0
KM-D7	D33	1423	30	0	2.1	0.0

if (length(fig_by_bm)) cat(sprintf("![](%s)\n\n", fig_by_bm))

Normalized so every BM’s own snowball corpus reads 1 (dashed line) — lets BMs be compared on conversion rate regardless of raw corpus size.

if (length(fig_by_bm_norm)) cat(sprintf("![](%s)\n\n", fig_by_bm_norm))

Level 3: LLM-confirmed REFUTES

Every citing work where the LLM’s own independent review agreed with NLI’s REFUTES verdict (and, by construction, judged the evidence for it sufficient — see Methodology). Use the BM dropdown in the filter row below the header to narrow to one Background Message.

if (length(table_l3)) {
  cat(sprintf(
    '<iframe src="%s" width="100%%" height="800px" style="border:none;"></iframe>\n\n',
    table_l3
  ))
}