NLI/LLM Funnel Report

Narrowing down citing works that support or contradict a Background Message

REFUTES Funnel — GA1

Methodology

Every work discovered by the snowball search under a Background Message (BM) is put through a 3-level sieve, each level a subset of the one before it, counted as distinct citing works per (assessment, km, bm) — a work cited under two different BMs is two separate findings, not one, so per-BM counts sum to the assessment-wide total shown below.

1. Snowball corpus — every citing work discovered for that BM (the forward/citing direction of the snowball search).
2. NLI REFUTES (certain) — of those, works with at least one claim scored label == "REFUTES" with uncertain == FALSE by the zero-shot NLI model (Phase 1 — see TD_BM_NLI_approach).
3. LLM-confirmed REFUTES — of those, works where OpenRouter’s LLM review (Phase 2 — see TD_NLI_LLM_two_phase) independently agreed with the REFUTES verdict (llm_agrees == TRUE).

There is deliberately no further “+ sufficient evidence” level: Phase 2’s own verification logic forces its llm_label to NOT_ENOUGH_INFO whenever it judges the evidence insufficient, so an LLM verdict of REFUTES already implies sufficient evidence — a separate level here would always contain exactly the same works as level 3.

Claim granularity: sub_bm. sub_bm (the default) segments each BM’s bm_description/bm_label into evidence-delimited sub-claims (2-17 per BM) at the sentence level. complete_bm instead treats each field as a single, whole claim (2 per BM, or 1 if bm_label/bm_description are identical) — cheaper to score (~71% fewer NLI pairs measured across both assessments), at the cost of losing which specific part of a longer BM a citing work actually supports or contradicts, and of the resulting hypothesis text sometimes exceeding the NLI model’s own token limit (see TD_BM_NLI_approach.qmd for the measured numbers and caveats). Level 2/3 counts below reflect whichever granularity this report was rendered for; the two are not directly comparable work-for-work, since they’re scored against different claim text.

Phase 2 (LLM verification) is a partial, ongoing process — OpenRouter credit has run out mid-run at least once already, so level 3 below reflects only the pairs reviewed so far, not the full REFUTES-certain set from level 2. This number will grow as llm_verification_parquet gets more OpenRouter runs; re-render this report after a rerun rather than treating today’s snapshot as final.

if (isTRUE(x$empty)) {
  cat("No data available yet for this assessment — the snowball, NLI, or LLM verification stage hasn't produced output here.\n\n")
  knitr::knit_exit()
}

Overall numbers

knitr::kable(
  x$funnel_overall,
  col.names = c("Level", "Description", "Distinct works")
)

Level	Description	Distinct works
level1	Snowball corpus	290740
level2	NLI REFUTES (certain)	8852
level3	LLM-confirmed REFUTES	1045

if (length(fig_overall)) cat(sprintf("![](%s)\n\n", fig_overall))

Per-BM numbers

knitr::kable(
  x$funnel_by_bm,
  col.names = c(
    "KM", "BM", "Corpus", sprintf("NLI %s", params$label), "LLM-confirmed",
    "% 2 of 1", "% 3 of 2"
  )
)

KM	BM	Corpus	NLI REFUTES	LLM-confirmed	% 2 of 1	% 3 of 2
A.	A1	23272	604	34	2.6	5.6
A.	A2	28825	846	21	2.9	2.5
A.	A3	4891	67	5	1.4	7.5
A.	A4	1915	42	7	2.2	16.7
A.	A5	30856	237	18	0.8	7.6
A.	A6	33600	875	86	2.6	9.8
A.	A7	5904	79	8	1.3	10.1
A.	A8	4546	615	117	13.5	19.0
A.	A9	1992	654	31	32.8	4.7
B.	B10	38022	1697	352	4.5	20.7
B.	B11	365	11	1	3.0	9.1
B.	B3	30477	964	73	3.2	7.6
B.	B4	8409	86	1	1.0	1.2
B.	B5	5858	221	51	3.8	23.1
C.	C2	5325	150	15	2.8	10.0
C.	C4	4904	41	7	0.8	17.1
C.	C6	1670	214	17	12.8	7.9
C.	C7	346	19	1	5.5	5.3
C.	C8	977	171	10	17.5	5.8
D.	D1	11477	379	47	3.3	12.4
D.	D10	1287	48	21	3.7	43.8
D.	D11	9020	209	20	2.3	9.6
D.	D12	11288	360	65	3.2	18.1
D.	D2	9801	111	1	1.1	0.9
D.	D3	1715	4	1	0.2	25.0
D.	D5	1194	10	4	0.8	40.0
D.	D6	3139	23	9	0.7	39.1
D.	D7	4903	35	6	0.7	17.1
D.	D8	2659	40	7	1.5	17.5
D.	D9	2103	40	9	1.9	22.5

if (length(fig_by_bm)) cat(sprintf("![](%s)\n\n", fig_by_bm))

Normalized so every BM’s own snowball corpus reads 1 (dashed line) — lets BMs be compared on conversion rate regardless of raw corpus size.

if (length(fig_by_bm_norm)) cat(sprintf("![](%s)\n\n", fig_by_bm_norm))

Level 3: LLM-confirmed REFUTES

Every citing work where the LLM’s own independent review agreed with NLI’s REFUTES verdict (and, by construction, judged the evidence for it sufficient — see Methodology). Use the BM dropdown in the filter row below the header to narrow to one Background Message.

if (length(table_l3)) {
  cat(sprintf(
    '<iframe src="%s" width="100%%" height="800px" style="border:none;"></iframe>\n\n',
    table_l3
  ))
}