Fact Checker DRAFT

Fact Checker of BMs of IPBES Assessments

Author

Affiliation

Rainer M Krug

University of Zürich

Published

September 18, 2026

Doi

xxxxxxxxxxxxxx4

Abstract

The assessments Linked Open Data is used to conduct analysis and searches to discover new knowledge related to the assessments.

Introduction

IPBES assessments were converted / enhanced into Linked Open Data (LOD) using the ttl format.

Three assessments available in ttl format are currently:

the First Global Assessment (GA1)
the Invasive Alien Species Assessment (IAS).

Methods

Methodology is documented separately in the TD_* design documents. Note: these are currently stale relative to the implementation — treat them as a record of the design rationale rather than an up-to-date specification of the pipeline as it stands today.

TD_targets — the targets pipeline architecture
TD_BM_NLI_approach — the active NLI scoring approach
TD_LLM_approach — the earlier LLM-comparison approach (superseded, source removed)
TD_NLI_LLM_two_phase — the two-phase NLI → LLM design (implemented; not yet wired into this report’s tables/figures)
TD_NLI_training — NLI fine-tuning plan (planned, not yet implemented)
TD_formatting — archived report design system

The figures and tables in the Results section below are all produced by targets pipeline targets — see _targets.R and the corresponding R/build_*.R files for the exact computation behind each one.

Results

The dataset contains 8656 papers referenced in the background messages. Out of these, 9946 were usable for the snowball search and also yielded results.

#|

all_papers |>
  dplyr::summarise(
    n = dplyr::n(),
    n_key_paper = sum(oa_input, na.rm = TRUE),
    n_non_key_paper = sum(!oa_input, na.rm = TRUE),
    n_nkp_pre_2018 = sum(
      (!oa_input) & (publication_year <= 2018),
      na.rm = TRUE
    ), # Count of citing papers with publication_year <= 2018
    n_nkp_post_2018 = sum(
      (!oa_input) & (publication_year > 2018),
      na.rm = TRUE
    ), # Count of citing papers with publication_year > 2018
    .by = c(km, bm)
  ) |>
  dplyr::arrange(km, bm) |>
  IPBES.R::table_dt()

Publication types after 2018

Here we show the different types of publication after 2018 as identified by the snowbal search. All Sub-background messages are grouped.

type	n
article	19024594
book-chapter	2652317
preprint	1408244
conference-paper	629523
review	511872
dissertation	373250
book	255372
peer-review	194374
paratext	164570
other	145412
reference-entry	138619
editorial	97436
report	91288
data-paper	29044
dataset	13914
book-review	13775
conference-abstract	5028
letter	3837
erratum	3050
software-paper	2344
retraction	1489
supplementary-materials	1116
software	81
standard	57

This table shows the number of work types per sub-background message. It allows filtering as well as sorting and downloading.

Publications per year

Overlap

In this section we will look at the number of background messages in which a given paper, as identified by their OpenAlex id. We willlook at that separatr for key-papers and non key-papers.

In the following tables only the papers which occur in more than one background message are included and only papers which were published after 2018.

This overlap can have different causes. Either there is an overlap of the key-papers, ehich results in an overlap of the new publications, or there is a topical overlap betweent the new paper themselves.

Overlap of Key Papers in Background Messages

The first table shows the overlap of the key-papers only regarding background messages.

Only papers occurring in more than 1 background message are shown.

Overlap of Papers after 2018

Temporarily removed — needs review

The two tables that used to appear here — overlap of papers after 2018 across sub messages, and across background messages — are currently disabled, so this section is incomplete.

They were withdrawn on 2026-09-18 because building them collected roughly 39 GB in a single step (doi, title and abstract for all 25.7 million post-2018 work × background-message combinations, before reducing to the papers actually cited more than five times) and exhausted the machine’s memory.

Making that query efficient is straightforward. What needs a decision first is what these tables should show: the two most recent versions disagree with each other (6,727 vs 805,028 rows) even though both group the data the same way, and the “more than 5” threshold dates from when the citing corpus was a small fraction of its current 2.7 million works — so it no longer selects a meaningful subset.

This is on the to-do list to look into. The key-paper overlap table above is unaffected and current.

NLI Alignment Scores

Each citing work is scored against its partition’s Background Message (BM) as SUPPORTS, REFUTES, or NOT_ENOUGH_INFO using a zero-shot NLI model. See TD_BM_NLI_approach for methodology.

for (data_path in nli_overview_data_paths) {
  x <- readRDS(data_path)
  nli_active_this <- x$nli_active %||% "deberta_zeroshot"
  gran_this <- x$granularity %||% "naive_bm"
  cat(sprintf("\n### %s (%s, %s)\n\n", x$assessment, nli_active_this, gran_this))

  if (isTRUE(x$empty)) {
    cat("No NLI scores available yet for this assessment/model/granularity combination.\n\n")
    next
  }

  # Exact suffix match (not a loose substring grepl) -- filenames now carry
  # both a model suffix ("" for deberta_zeroshot, "_<nli_active>" otherwise)
  # and a granularity suffix (always "_<granularity>", explicit for every
  # value including naive_bm), so a bare "_<assessment>\\.png$" anchor
  # would silently only ever match one branch and miss the others.
  model_suffix_this <- if (identical(nli_active_this, "deberta_zeroshot")) "" else paste0("_", nli_active_this)
  gran_suffix_this <- paste0("_", gran_this)
  suffix_this <- paste0(model_suffix_this, gran_suffix_this)

  cat(sprintf(
    "- **%s** works scored\n- **%d** background messages\n- **%d** key messages\n- **%s%%** uncertain (confidence < 0.60)\n\n",
    format(x$n_total, big.mark = ","),
    x$n_bm,
    x$n_km,
    x$pct_unc
  ))

  fig_this <- nli_overview_figures_paths[
    grepl(paste0("_", x$assessment, suffix_this, "\\.png$"), nli_overview_figures_paths)
  ]
  plot_overall <- fig_this[grepl("plot_overall", fig_this)]
  plot_km <- fig_this[grepl("plot_km", fig_this)]
  plot_bm <- fig_this[grepl("plot_bm", fig_this)]
  plot_conf <- fig_this[grepl("plot_conf", fig_this)]
  plot_aln <- fig_this[grepl("plot_aln", fig_this)]

  cat("#### Label distribution\n\n")
  cat(
    knitr::kable(
      x$label_overall |>
        dplyr::mutate(
          label = factor(
            label,
            levels = c("SUPPORTS", "NOT_ENOUGH_INFO", "REFUTES")
          )
        ) |>
        dplyr::arrange(label),
      col.names = c("Label", "Works", "%")
    ),
    sep = "\n"
  )
  cat("\n\n")
  if (length(plot_overall)) {
    cat(sprintf("![](%s)\n\n", plot_overall))
  }

  cat("#### Interactive BM Explorer\n\n")
  cat(
    "Select a background message from the dropdown and a minimum confidence ",
    "from the slider. All works are always shown; the slider instead splits ",
    "each bar into a **solid** segment (confidence at or above the ",
    "threshold) and a **hollow/outlined** segment on top (below the ",
    "threshold, stacked) — in the label-distribution panel (each bar also ",
    "labelled `n / N`: works at/above the threshold vs. all works with that ",
    "label), the confidence panel (with a dashed line marking the ",
    "threshold), and the alignment panel. Mean/median confidence and the % ",
    "meeting the threshold are reported above the panels. Below the panels, ",
    "a second dropdown (default **REFUTES**) selects which label the ",
    "drill-down table lists — the actual works for the current BM + ",
    "threshold + label, sorted by confidence descending and capped at 50 ",
    "rows (the caption reports the true total when capped). Each work links ",
    "to its DOI where one exists (OpenAlex otherwise). **Download table ",
    "(CSV)** is not limited to the 50 rows shown — it exports up to 5,000 ",
    "matching works for the current BM + threshold + label (sorted by ",
    "confidence descending, flagged with a trailing comment line if even ",
    "that limit truncates the true match count), including the assessment ",
    "and BM as explicit columns.\n\n"
  )
  explorer_this <- nli_bm_explorer_paths[
    grepl(paste0("_", x$assessment, suffix_this, "\\.html$"), nli_bm_explorer_paths)
  ]
  if (length(explorer_this)) {
    cat(sprintf(
      '<iframe src="%s" width="100%%" height="1020px" style="border:none;"></iframe>\n\n',
      explorer_this
    ))
  }

  cat("#### Per Key Message\n\n")
  cat(
    knitr::kable(
      x$label_km,
      col.names = c("KM", "SUPPORTS %", "NEI %", "REFUTES %")
    ),
    sep = "\n"
  )
  cat("\n\n")
  if (length(plot_km)) {
    cat(sprintf("![](%s)\n\n", plot_km))
  }

  cat("#### Per Background Message\n\n")
  if (length(plot_bm)) {
    cat(sprintf("![](%s)\n\n", plot_bm))
  }
  cat(
    knitr::kable(
      x$table_bm,
      col.names = c("KM", "BM", "SUPPORTS %", "NEI %", "REFUTES %")
    ),
    sep = "\n"
  )
  cat("\n\n")

  cat("#### Confidence by label\n\n")
  cat(
    knitr::kable(
      x$conf_table,
      col.names = c("Label", "n", "Mean", "Median", "P25", "P75")
    ),
    sep = "\n"
  )
  cat("\n\n")
  if (length(plot_conf)) {
    cat(sprintf("![](%s)\n\n", plot_conf))
  }

  cat("#### Alignment score per BM\n\n")
  cat(
    knitr::kable(
      x$align_table,
      col.names = c(
        "KM",
        "BM",
        "n",
        "Mean aln",
        "SD",
        "SUPP %",
        "REF %",
        "NEI %",
        "Unc %"
      )
    ),
    sep = "\n"
  )
  cat("\n\n")
  if (length(plot_aln)) cat(sprintf("![](%s)\n\n", plot_aln))

  # AI Based Interpretation — last subsection of each assessment. Hand-
  # written commentary, deliberately scoped to the two signals of interest
  # (SUPPORTS and REFUTES) rather than the dominant NOT_ENOUGH_INFO label.
  # Recomputed against the current scored data (below) — will go stale as
  # scoring continues or the NLI config changes, so re-derive the numbers
  # from x$table_bm / x$label_km before trusting this after a rerun.
  cat("#### AI Based Interpretation\n\n")
  if (identical(x$assessment, "GA1")) {
    cat(r"----(
::: {.callout-note}
## About this section
Focused specifically on the SUPPORTS/REFUTES signals — NOT_ENOUGH_INFO (~71% of GA1 pairs) is set aside here as the least informative label. Written against the current scored data (2026-07-06); re-derive the numbers below from the tables above before trusting this after a rerun.
:::

**Key finding — KM D is the only reliable signal.** GA1 overall SUPPORTS (16.1%) edges out REFUTES (13.0%), but that gap is almost entirely driven by KM D (34.7% SUPPORTS vs 10.9% REFUTES). KMs A–C sit close to parity or worse — A: 15.0% vs 12.5%, B: 10.0% vs 14.2% (REFUTES-inflated), C: 14.5% vs 20.9% (REFUTES-inflated) — so the argmax label there carries little information on its own.

Within KM D, four BMs are unambiguous: **D3** (62.6% SUPPORTS vs 4.4% REFUTES), **D2** (44.0% vs 6.5%), **D1** (40.8% vs 10.3%), and **D12** (40.7% vs 11.1%) — all over 40% SUPPORTS at single-digit-to-low-teens REFUTES. The one exception inside KM D is **D10**, where REFUTES (15.2%) exceeds SUPPORTS (11.6%) — an outlier against its own KM's pattern, worth treating differently from its D-series siblings.

Likely reason: D-series BMs are probably more concrete and empirically testable — biodiversity metrics, species trends, ecosystem services — the kind of claims abstracts engage with directly.

::: {.callout-warning}
## REFUTES inflation — KMs A, B, C

The BMs with the widest REFUTES-over-SUPPORTS gap:

| BM | SUPPORTS | REFUTES | Gap (REFUTES − SUPPORTS) |
|----|---------|---------|---------|
| C6  |  5.2 % | 28.6 % | +23.4 |
| A9  | 13.8 % | 30.7 % | +16.9 |
| C8  |  8.3 % | 24.8 % | +16.5 |
| B11 | 15.4 % | 28.4 % | +13.0 |
| C7  |  7.1 % | 16.1 % |  +9.0 |
| B3  |  3.3 % | 12.0 % |  +8.7 |

Citing papers of an assessment being scored as majority-REFUTES at these margins is not credible. This is the well-documented zero-shot NLI failure mode: the hypothesis *"This paper refutes the following claim: …"* draws spurious entailment when the BM contains complex, multi-part, or politically nuanced language. The model fires on surface-level topical tension rather than genuine contradiction.
:::

::: {.callout-note}
## Closest to parity — weakest signal

A handful of BMs have SUPPORTS and REFUTES within a couple of points of each other — the argmax label there is close to a coin flip:

- **A1** 12.9 % vs 11.0 %
- **B10** 12.7 % vs 14.7 %
- **A4** 9.7 % vs 11.8 %

For these, only the continuous alignment score (`p_supports − p_refutes`) carries any real information.
:::

##### Notable exception in KM A

**A2** (20.5% SUPPORTS, 11.0% REFUTES) is the clearest positive signal in KMs A–C outside of KM D — a nearly 10-point SUPPORTS lead, well ahead of its KM-A neighbours. Its claim is likely more concrete or empirically specific than the rest of KM A — worth inspecting manually as a calibration reference.

##### Operational implications

1. **Do not act on raw REFUTES labels for KMs A–C** without Phase-2 LLM review. NLI here is a coarse NEI filter, not a judgment.
2. **KM D SUPPORTS can be treated as the most reliable signal in the assessment** — D3, D2, D1, D12 in particular (all >40% SUPPORTS, single-digit-to-low-teens REFUTES). D10 is the one KM D BM that does NOT fit this pattern.
3. **Use `p_supports − p_refutes` for ranking** rather than argmax labels, especially for the near-parity BMs above.
4. **`max_length: 256` is probably hurting KMs A–C.** Truncating abstract tails removes findings — the part that matters most for support/refute judgements on conceptual BMs. The base-model-at-512 comparison (`deberta_zeroshot_v3`) is the highest-priority next experiment.
)----")
    cat(sprintf(
      "5. **Fine-tuning on BM citations** ([TD_NLI_training](%s)) would most help KMs A–C — the support/refute axis that zero-shot is fumbling. D-series BMs are already working.\n\n",
      td_doc_link("TD_NLI_training")
    ))
  } else if (identical(x$assessment, "IAS")) {
    cat(r"----(
::: {.callout-note}
## About this section
Focused specifically on the SUPPORTS/REFUTES signals — NOT_ENOUGH_INFO (~81% of IAS pairs) is set aside here as the least informative label. Written against the current scored data (2026-07-06); re-derive the numbers below from the tables above before trusting this after a rerun.
:::

**Key finding — IAS is weaker and more REFUTES-skewed than GA1.** Overall, IAS REFUTES (11.1%) actually *exceeds* SUPPORTS (8.4%) — the opposite of GA1's overall SUPPORTS lean (16.1% vs 13.0%). Unlike GA1, no IAS BM reaches D-series levels of confidence; the strongest SUPPORTS signal is markedly weaker.

::: {.callout-tip}
## Strongest SUPPORTS signal

**D33** (25.9% SUPPORTS vs 9.5% REFUTES, a 16-point lead) is the clearest positive signal in IAS, appearing under both KM-D4 and KM-D7. A handful of others lean the same way but at much lower absolute rates: **B13** (19.4% vs 10.8%), **D30** (8.7% vs 3.0%), **D27** (7.5% vs 1.8%), and **D25** (12.5% vs 7.2%). None of these approach GA1's D3/D2/D1/D12 margins.
:::

::: {.callout-warning}
## REFUTES inflation

The BMs with the widest REFUTES-over-SUPPORTS gap — the same zero-shot failure mode documented for GA1's KMs A–C:

| BM | SUPPORTS | REFUTES | Gap (REFUTES − SUPPORTS) |
|----|---------|---------|---------|
| A1  |  7.5 % | 22.6 % | +15.1 |
| A8  |  3.6 % | 16.7 % | +13.1 |
| D32 | 22.5 % | 33.5 % | +11.0 |
| A4  |  7.6 % | 17.6 % | +10.0 |
| C19 |  5.0 % | 14.3 % |  +9.3 |
| D28 |  8.1 % | 14.7 % |  +6.6 |

**D32** is a special case: both SUPPORTS and REFUTES are elevated (22.5% and 33.5%) rather than REFUTES alone — it appears under three different KMs (KM-D5, KM-D6, KM-D7), which may mean it's a genuinely contested or multi-faceted claim rather than a pure model artifact, and is worth a manual look before writing it off as noise.
:::

##### Operational implications

1. **Do not act on raw REFUTES labels anywhere in IAS** without Phase-2 LLM review — the overall REFUTES-over-SUPPORTS skew makes this an even higher priority here than for GA1's KMs A–C.
2. **D33 is the one IAS BM currently worth treating as a real signal** — everything else is either near parity or REFUTES-leaning.
3. **Use `p_supports − p_refutes` for ranking** rather than argmax labels; IAS's overall skew means a plain SUPPORTS-label count will systematically undercount real support.
4. **D32 deserves manual review before either accepting or discarding it** — high confidence in both directions, across multiple KMs, is a different failure mode from the single-direction REFUTES inflation seen elsewhere.
)----")
  } else {
    cat("No AI-based interpretation has been written yet for this assessment.\n\n")
  }
}

GA1 (bge_m3_zeroshot_naive_bm, naive_bm)

No NLI scores available yet for this assessment/model/granularity combination.

GA1 (bge_m3_zeroshot_complete_bm, complete_bm)

594,034 works scored
30 background messages
4 key messages
70.5% uncertain (confidence < 0.60)

Label distribution

Label	Works	%
SUPPORTS	375165	63.2
NOT_ENOUGH_INFO	145999	24.6
REFUTES	72870	12.3

Interactive BM Explorer

Select a background message from the dropdown and a minimum confidence from the slider. All works are always shown; the slider instead splits each bar into a solid segment (confidence at or above the threshold) and a hollow/outlined segment on top (below the threshold, stacked) — in the label-distribution panel (each bar also labelled n / N: works at/above the threshold vs. all works with that label), the confidence panel (with a dashed line marking the threshold), and the alignment panel. Mean/median confidence and the % meeting the threshold are reported above the panels. Below the panels, a second dropdown (default REFUTES) selects which label the drill-down table lists — the actual works for the current BM + threshold + label, sorted by confidence descending and capped at 50 rows (the caption reports the true total when capped). Each work links to its DOI where one exists (OpenAlex otherwise). Download table (CSV) is not limited to the 50 rows shown — it exports up to 5,000 matching works for the current BM + threshold + label (sorted by confidence descending, flagged with a trailing comment line if even that limit truncates the true match count), including the assessment and BM as explicit columns.

Per Key Message

KM	SUPPORTS %	NEI %	REFUTES %
A.	62.7	22.8	14.5
B.	53.7	31.6	14.8
C.	54.1	34.2	11.7
D.	80.3	16.3	3.5

Per Background Message

KM	BM	SUPPORTS %	NEI %	REFUTES %
A.	A1	78.3	4.7	16.9
A.	A2	55.8	40.3	3.8
A.	A3	74.3	20.5	5.2
A.	A4	63.1	30.6	6.3
A.	A5	71.4	14.7	13.8
A.	A6	50.3	23.6	26.1
A.	A7	54.2	42.0	3.8
A.	A8	80.4	17.1	2.5
A.	A9	11.8	53.0	35.2
B.	B10	66.6	26.3	7.1
B.	B11	82.3	6.2	11.5
B.	B3	31.9	40.7	27.5
B.	B4	61.4	26.7	12.0
B.	B5	72.2	26.2	1.6
C.	C2	44.4	39.1	16.6
C.	C4	56.4	36.9	6.8
C.	C6	69.5	18.9	11.6
C.	C7	40.0	34.4	25.6
C.	C8	74.3	20.0	5.7
D.	D1	69.0	28.0	3.0
D.	D10	93.4	5.1	1.4
D.	D11	94.2	5.3	0.5
D.	D12	81.6	13.1	5.3
D.	D2	78.8	19.0	2.2
D.	D3	79.3	17.6	3.1
D.	D5	81.9	16.9	1.2
D.	D6	69.2	19.3	11.5
D.	D7	86.1	11.3	2.6
D.	D8	82.6	10.2	7.2
D.	D9	73.3	23.3	3.4

Confidence by label

Label	n	Mean	Median	P25	P75
SUPPORTS	375165	0.575	0.548	0.468	0.666
NOT_ENOUGH_INFO	145999	0.518	0.504	0.435	0.590
REFUTES	72870	0.445	0.437	0.396	0.482

Alignment score per BM

KM	BM	n	Mean aln	SD	SUPP %	REF %	NEI %	Unc %
A.	A1	46544	0.253	0.257	78.3	16.9	4.7	68.5
A.	A2	57650	0.274	0.295	55.8	3.8	40.3	47.7
A.	A3	9782	0.224	0.158	74.3	5.2	20.5	88.2
A.	A4	3830	0.213	0.214	63.1	6.3	30.6	65.3
A.	A5	64678	0.176	0.207	71.4	13.8	14.7	82.1
A.	A6	70256	0.057	0.146	50.3	26.1	23.6	93.0
A.	A7	11808	0.256	0.291	54.2	3.8	42.0	49.5
A.	A8	9092	0.315	0.234	80.4	2.5	17.1	50.8
A.	A9	3984	-0.070	0.152	11.8	35.2	53.0	82.5
B.	B10	79520	0.202	0.194	66.6	7.1	26.3	78.1
B.	B11	730	0.381	0.307	82.3	11.5	6.2	45.9
B.	B3	64010	0.007	0.185	31.9	27.5	40.7	85.0
B.	B4	16818	0.269	0.306	61.4	12.0	26.7	51.1
B.	B5	11716	0.500	0.351	72.2	1.6	26.2	44.9
C.	C2	10650	0.122	0.215	44.4	16.6	39.1	77.4
C.	C4	9808	0.177	0.207	56.4	6.8	36.9	66.0
C.	C6	3340	0.200	0.197	69.5	11.6	18.9	86.0
C.	C7	692	0.071	0.164	40.0	25.6	34.4	94.8
C.	C8	1954	0.284	0.224	74.3	5.7	20.0	71.1
D.	D1	22954	0.259	0.191	69.0	3.0	28.0	73.2
D.	D10	2574	0.538	0.220	93.4	1.4	5.1	30.1
D.	D11	18040	0.601	0.220	94.2	0.5	5.3	20.9
D.	D12	22576	0.273	0.193	81.6	5.3	13.1	77.5
D.	D2	19602	0.384	0.219	78.8	2.2	19.0	49.4
D.	D3	3430	0.371	0.238	79.3	3.1	17.6	57.6
D.	D5	2388	0.400	0.188	81.9	1.2	16.9	62.7
D.	D6	6278	0.240	0.231	69.2	11.5	19.3	78.6
D.	D7	9806	0.395	0.254	86.1	2.6	11.3	54.5
D.	D8	5318	0.504	0.339	82.6	7.2	10.2	36.4
D.	D9	4206	0.406	0.284	73.3	3.4	23.3	42.0

AI Based Interpretation

About this section

Focused specifically on the SUPPORTS/REFUTES signals — NOT_ENOUGH_INFO (~71% of GA1 pairs) is set aside here as the least informative label. Written against the current scored data (2026-07-06); re-derive the numbers below from the tables above before trusting this after a rerun.

Key finding — KM D is the only reliable signal. GA1 overall SUPPORTS (16.1%) edges out REFUTES (13.0%), but that gap is almost entirely driven by KM D (34.7% SUPPORTS vs 10.9% REFUTES). KMs A–C sit close to parity or worse — A: 15.0% vs 12.5%, B: 10.0% vs 14.2% (REFUTES-inflated), C: 14.5% vs 20.9% (REFUTES-inflated) — so the argmax label there carries little information on its own.

Within KM D, four BMs are unambiguous: D3 (62.6% SUPPORTS vs 4.4% REFUTES), D2 (44.0% vs 6.5%), D1 (40.8% vs 10.3%), and D12 (40.7% vs 11.1%) — all over 40% SUPPORTS at single-digit-to-low-teens REFUTES. The one exception inside KM D is D10, where REFUTES (15.2%) exceeds SUPPORTS (11.6%) — an outlier against its own KM’s pattern, worth treating differently from its D-series siblings.

Likely reason: D-series BMs are probably more concrete and empirically testable — biodiversity metrics, species trends, ecosystem services — the kind of claims abstracts engage with directly.

REFUTES inflation — KMs A, B, C

The BMs with the widest REFUTES-over-SUPPORTS gap:

BM	SUPPORTS	REFUTES	Gap (REFUTES − SUPPORTS)
C6	5.2 %	28.6 %	+23.4
A9	13.8 %	30.7 %	+16.9
C8	8.3 %	24.8 %	+16.5
B11	15.4 %	28.4 %	+13.0
C7	7.1 %	16.1 %	+9.0
B3	3.3 %	12.0 %	+8.7

Citing papers of an assessment being scored as majority-REFUTES at these margins is not credible. This is the well-documented zero-shot NLI failure mode: the hypothesis “This paper refutes the following claim: …” draws spurious entailment when the BM contains complex, multi-part, or politically nuanced language. The model fires on surface-level topical tension rather than genuine contradiction.

Closest to parity — weakest signal

A handful of BMs have SUPPORTS and REFUTES within a couple of points of each other — the argmax label there is close to a coin flip:

A1 12.9 % vs 11.0 %
B10 12.7 % vs 14.7 %
A4 9.7 % vs 11.8 %

For these, only the continuous alignment score (p_supports − p_refutes) carries any real information.

Notable exception in KM A

A2 (20.5% SUPPORTS, 11.0% REFUTES) is the clearest positive signal in KMs A–C outside of KM D — a nearly 10-point SUPPORTS lead, well ahead of its KM-A neighbours. Its claim is likely more concrete or empirically specific than the rest of KM A — worth inspecting manually as a calibration reference.

Operational implications

Do not act on raw REFUTES labels for KMs A–C without Phase-2 LLM review. NLI here is a coarse NEI filter, not a judgment.
KM D SUPPORTS can be treated as the most reliable signal in the assessment — D3, D2, D1, D12 in particular (all >40% SUPPORTS, single-digit-to-low-teens REFUTES). D10 is the one KM D BM that does NOT fit this pattern.
Use p_supports − p_refutes for ranking rather than argmax labels, especially for the near-parity BMs above.
max_length: 256 is probably hurting KMs A–C. Truncating abstract tails removes findings — the part that matters most for support/refute judgements on conceptual BMs. The base-model-at-512 comparison (deberta_zeroshot_v3) is the highest-priority next experiment.
Fine-tuning on BM citations (TD_NLI_training) would most help KMs A–C — the support/refute axis that zero-shot is fumbling. D-series BMs are already working.

GA1 (bge_m3_zeroshot_atomic_bm, atomic_bm)

2,429,848 works scored
30 background messages
4 key messages
76.4% uncertain (confidence < 0.60)

Label distribution

Label	Works	%
SUPPORTS	872506	35.9
NOT_ENOUGH_INFO	1232027	50.7
REFUTES	325315	13.4

Interactive BM Explorer

Per Key Message

KM	SUPPORTS %	NEI %	REFUTES %
A.	34.6	51.1	14.3
B.	27.7	56.0	16.2
C.	44.7	44.5	10.8
D.	58.9	37.9	3.2

Per Background Message

KM	BM	SUPPORTS %	NEI %	REFUTES %
A.	A1	28.9	51.3	19.8
A.	A2	22.7	68.7	8.6
A.	A3	32.4	59.3	8.3
A.	A4	37.0	46.2	16.8
A.	A5	39.3	46.0	14.7
A.	A6	45.1	36.3	18.6
A.	A7	32.3	55.3	12.4
A.	A8	56.8	30.7	12.5
A.	A9	36.4	52.9	10.7
B.	B10	25.0	58.1	17.0
B.	B11	51.1	45.7	3.2
B.	B3	23.9	59.6	16.5
B.	B4	50.9	25.2	24.0
B.	B5	50.8	42.6	6.6
C.	C2	43.6	43.7	12.8
C.	C4	39.9	47.9	12.2
C.	C6	50.9	40.7	8.4
C.	C7	39.3	50.5	10.2
C.	C8	53.7	41.1	5.2
D.	D1	34.9	61.5	3.6
D.	D10	68.8	29.0	2.2
D.	D11	67.9	31.0	1.1
D.	D12	59.4	36.7	3.9
D.	D2	77.6	19.7	2.6
D.	D3	61.5	36.9	1.6
D.	D5	81.5	17.4	1.1
D.	D6	51.6	39.0	9.4
D.	D7	72.8	25.3	1.9
D.	D8	64.5	33.5	2.1
D.	D9	63.5	31.8	4.7

Confidence by label

Label	n	Mean	Median	P25	P75
SUPPORTS	872506	0.516	0.490	0.430	0.579
NOT_ENOUGH_INFO	1232027	0.545	0.529	0.451	0.627
REFUTES	325315	0.439	0.426	0.391	0.472

Alignment score per BM

KM	BM	n	Mean aln	SD	SUPP %	REF %	NEI %	Unc %
A.	A1	186176	0.062	0.210	28.9	19.8	51.3	76.4
A.	A2	403550	0.068	0.186	22.7	8.6	68.7	65.0
A.	A3	34237	0.097	0.175	32.4	8.3	59.3	77.1
A.	A4	13405	0.071	0.179	37.0	16.8	46.2	84.3
A.	A5	194034	0.104	0.176	39.3	14.7	46.0	86.8
A.	A6	386408	0.088	0.180	45.1	18.6	36.3	90.7
A.	A7	41328	0.083	0.186	32.3	12.4	55.3	81.1
A.	A8	45460	0.192	0.243	56.8	12.5	30.7	78.0
A.	A9	19920	0.111	0.217	36.4	10.7	52.9	70.0
B.	B10	318080	0.042	0.169	25.0	17.0	58.1	77.2
B.	B11	4015	0.218	0.194	51.1	3.2	45.7	75.1
B.	B3	320050	0.036	0.185	23.9	16.5	59.6	78.4
B.	B4	33636	0.105	0.222	50.9	24.0	25.2	81.7
B.	B5	52722	0.202	0.229	50.8	6.6	42.6	68.8
C.	C2	21300	0.134	0.184	43.6	12.8	43.7	83.9
C.	C4	24520	0.161	0.256	39.9	12.2	47.9	63.2
C.	C6	15030	0.192	0.228	50.9	8.4	40.7	75.3
C.	C7	2076	0.112	0.165	39.3	10.2	50.5	81.0
C.	C8	6839	0.211	0.193	53.7	5.2	41.1	78.3
D.	D1	57385	0.138	0.158	34.9	3.6	61.5	71.1
D.	D10	6435	0.309	0.211	68.8	2.2	29.0	65.9
D.	D11	54120	0.394	0.251	67.9	1.1	31.0	48.8
D.	D12	90304	0.241	0.216	59.4	3.9	36.7	71.8
D.	D2	19602	0.377	0.223	77.6	2.6	19.7	50.5
D.	D3	10290	0.305	0.199	61.5	1.6	36.9	69.6
D.	D5	2388	0.411	0.190	81.5	1.1	17.4	59.2
D.	D6	15695	0.193	0.220	51.6	9.4	39.0	76.0
D.	D7	19612	0.390	0.237	72.8	1.9	25.3	51.3
D.	D8	18613	0.337	0.234	64.5	2.1	33.5	60.9
D.	D9	12618	0.284	0.223	63.5	4.7	31.8	66.4

AI Based Interpretation

About this section

Likely reason: D-series BMs are probably more concrete and empirically testable — biodiversity metrics, species trends, ecosystem services — the kind of claims abstracts engage with directly.

REFUTES inflation — KMs A, B, C

The BMs with the widest REFUTES-over-SUPPORTS gap:

BM	SUPPORTS	REFUTES	Gap (REFUTES − SUPPORTS)
C6	5.2 %	28.6 %	+23.4
A9	13.8 %	30.7 %	+16.9
C8	8.3 %	24.8 %	+16.5
B11	15.4 %	28.4 %	+13.0
C7	7.1 %	16.1 %	+9.0
B3	3.3 %	12.0 %	+8.7

Closest to parity — weakest signal

A handful of BMs have SUPPORTS and REFUTES within a couple of points of each other — the argmax label there is close to a coin flip:

A1 12.9 % vs 11.0 %
B10 12.7 % vs 14.7 %
A4 9.7 % vs 11.8 %

For these, only the continuous alignment score (p_supports − p_refutes) carries any real information.

Notable exception in KM A

Operational implications

Do not act on raw REFUTES labels for KMs A–C without Phase-2 LLM review. NLI here is a coarse NEI filter, not a judgment.
KM D SUPPORTS can be treated as the most reliable signal in the assessment — D3, D2, D1, D12 in particular (all >40% SUPPORTS, single-digit-to-low-teens REFUTES). D10 is the one KM D BM that does NOT fit this pattern.
Use p_supports − p_refutes for ranking rather than argmax labels, especially for the near-parity BMs above.
max_length: 256 is probably hurting KMs A–C. Truncating abstract tails removes findings — the part that matters most for support/refute judgements on conceptual BMs. The base-model-at-512 comparison (deberta_zeroshot_v3) is the highest-priority next experiment.
Fine-tuning on BM citations (TD_NLI_training) would most help KMs A–C — the support/refute axis that zero-shot is fumbling. D-series BMs are already working.

VA (bge_m3_zeroshot_naive_bm, naive_bm)

No NLI scores available yet for this assessment/model/granularity combination.

VA (bge_m3_zeroshot_complete_bm, complete_bm)

No NLI scores available yet for this assessment/model/granularity combination.

VA (bge_m3_zeroshot_atomic_bm, atomic_bm)

No NLI scores available yet for this assessment/model/granularity combination.

TCA (bge_m3_zeroshot_naive_bm, naive_bm)

No NLI scores available yet for this assessment/model/granularity combination.

TCA (bge_m3_zeroshot_complete_bm, complete_bm)

No NLI scores available yet for this assessment/model/granularity combination.

TCA (bge_m3_zeroshot_atomic_bm, atomic_bm)

No NLI scores available yet for this assessment/model/granularity combination.

IAS (bge_m3_zeroshot_naive_bm, naive_bm)

No NLI scores available yet for this assessment/model/granularity combination.

IAS (bge_m3_zeroshot_complete_bm, complete_bm)

355,300 works scored
33 background messages
22 key messages
54.3% uncertain (confidence < 0.60)

Label distribution

Label	Works	%
SUPPORTS	208839	58.8
NOT_ENOUGH_INFO	120754	34.0
REFUTES	25707	7.2

Interactive BM Explorer

Per Key Message

KM	SUPPORTS %	NEI %	REFUTES %
KM-A1	61.7	32.3	6.1
KM-A2	33.9	36.4	29.7
KM-A3	43.1	42.0	14.9
KM-A4	51.6	41.2	7.2
KM-A5	71.5	26.1	2.4
KM-B1	65.5	29.3	5.2
KM-B2	52.6	39.5	7.9
KM-B3	65.5	29.3	5.2
KM-B4	68.8	26.0	5.2
KM-C1	60.6	37.5	1.9
KM-C2	60.0	36.3	3.7
KM-C3	55.9	40.8	3.2
KM-C4	71.7	20.0	8.2
KM-C5	68.5	30.9	0.6
KM-C6	48.6	45.6	5.8
KM-D1	68.8	30.6	0.7
KM-D2	59.6	36.8	3.6
KM-D3	56.5	42.4	1.1
KM-D4	64.5	34.1	1.5
KM-D5	62.7	36.2	1.1
KM-D6	59.9	39.2	0.9
KM-D7	66.4	32.4	1.2

Per Background Message

KM	BM	SUPPORTS %	NEI %	REFUTES %
KM-A1	A1	61.7	32.3	6.1
KM-A2	A2	23.5	32.3	44.2
KM-A2	A3	48.0	41.9	10.2
KM-A3	A4	38.8	42.0	19.2
KM-A3	A5	50.8	42.2	7.1
KM-A4	A5	50.8	42.2	7.1
KM-A4	A6	53.1	39.5	7.4
KM-A5	A7	76.2	22.7	1.1
KM-A5	A8	54.6	38.3	7.1
KM-B1	B11	56.2	35.6	8.3
KM-B1	B12	67.0	31.1	1.8
KM-B1	B14	72.7	19.6	7.7
KM-B1	B9	65.0	33.0	2.1
KM-B2	B10	52.6	39.5	7.9
KM-B3	B11	56.2	35.6	8.3
KM-B3	B12	67.0	31.1	1.8
KM-B3	B14	72.7	19.6	7.7
KM-B3	B9	65.0	33.0	2.1
KM-B4	B10	52.6	39.5	7.9
KM-B4	B13	67.3	30.9	1.9
KM-B4	B14	72.7	19.6	7.7
KM-C1	C15	59.9	38.3	1.8
KM-C1	C16	64.3	35.2	0.5
KM-C1	C17	42.7	51.6	5.7
KM-C1	C18	84.6	13.9	1.5
KM-C1	C22	58.8	40.3	0.8
KM-C1	C23	60.0	39.4	0.6
KM-C2	C15	59.9	38.3	1.8
KM-C2	C17	42.7	51.6	5.7
KM-C2	C18	84.6	13.9	1.5
KM-C3	C19	55.9	40.8	3.2
KM-C4	C20	71.7	20.0	8.2
KM-C5	C21	68.5	30.9	0.6
KM-C6	C23	60.0	39.4	0.6
KM-C6	C24	17.4	62.6	20.0
KM-D1	D25	68.8	30.6	0.7
KM-D2	D26	60.5	38.1	1.4
KM-D2	D30	58.7	35.7	5.6
KM-D3	D27	56.5	42.6	0.9
KM-D3	D28	56.6	42.0	1.4
KM-D4	D26	60.5	38.1	1.4
KM-D4	D33	71.4	27.1	1.5
KM-D5	D31	60.3	38.6	1.1
KM-D5	D32	64.5	34.4	1.1
KM-D6	D29	54.0	45.6	0.5
KM-D6	D31	60.3	38.6	1.1
KM-D6	D32	64.5	34.4	1.1
KM-D7	D32	64.5	34.4	1.1
KM-D7	D33	71.4	27.1	1.5

Confidence by label

Label	n	Mean	Median	P25	P75
SUPPORTS	208839	0.640	0.632	0.501	0.785
NOT_ENOUGH_INFO	120754	0.561	0.548	0.462	0.649
REFUTES	25707	0.457	0.431	0.391	0.496

Alignment score per BM

KM	BM	n	Mean aln	SD	SUPP %	REF %	NEI %	Unc %
KM-A1	A1	14378	0.177	0.151	61.7	6.1	32.3	94.0
KM-A2	A2	19414	-0.052	0.234	23.5	44.2	32.3	71.8
KM-A2	A3	14314	0.147	0.254	48.0	10.2	41.9	55.9
KM-A3	A4	19802	0.062	0.170	38.8	19.2	42.0	76.4
KM-A3	A5	10898	0.205	0.361	50.8	7.1	42.2	32.0
KM-A4	A5	10898	0.205	0.361	50.8	7.1	42.2	32.0
KM-A4	A6	5886	0.171	0.236	53.1	7.4	39.5	73.6
KM-A5	A7	17246	0.425	0.257	76.2	1.1	22.7	42.2
KM-A5	A8	4802	0.158	0.175	54.6	7.1	38.3	86.1
KM-B1	B11	10604	0.122	0.189	56.2	8.3	35.6	79.0
KM-B1	B12	10358	0.377	0.311	67.0	1.8	31.1	45.7
KM-B1	B14	12170	0.313	0.301	72.7	7.7	19.6	56.7
KM-B1	B9	9216	0.376	0.326	65.0	2.1	33.0	37.6
KM-B2	B10	1858	0.133	0.187	52.6	7.9	39.5	73.7
KM-B3	B11	10604	0.122	0.189	56.2	8.3	35.6	79.0
KM-B3	B12	10358	0.377	0.311	67.0	1.8	31.1	45.7
KM-B3	B14	12170	0.313	0.301	72.7	7.7	19.6	56.7
KM-B3	B9	9216	0.376	0.326	65.0	2.1	33.0	37.6
KM-B4	B10	1858	0.133	0.187	52.6	7.9	39.5	73.7
KM-B4	B13	10470	0.461	0.354	67.3	1.9	30.9	44.7
KM-B4	B14	12170	0.313	0.301	72.7	7.7	19.6	56.7
KM-C1	C15	1204	0.362	0.370	59.9	1.8	38.3	44.9
KM-C1	C16	4406	0.400	0.294	64.3	0.5	35.2	39.8
KM-C1	C17	5160	0.125	0.297	42.7	5.7	51.6	47.4
KM-C1	C18	3646	0.412	0.244	84.6	1.5	13.9	51.0
KM-C1	C22	4718	0.388	0.301	58.8	0.8	40.3	38.6
KM-C1	C23	4826	0.341	0.250	60.0	0.6	39.4	43.9
KM-C2	C15	1204	0.362	0.370	59.9	1.8	38.3	44.9
KM-C2	C17	5160	0.125	0.297	42.7	5.7	51.6	47.4
KM-C2	C18	3646	0.412	0.244	84.6	1.5	13.9	51.0
KM-C3	C19	4340	0.221	0.224	55.9	3.2	40.8	72.9
KM-C4	C20	3152	0.227	0.195	71.7	8.2	20.0	84.1
KM-C5	C21	7352	0.498	0.316	68.5	0.6	30.9	33.0
KM-C6	C23	4826	0.341	0.250	60.0	0.6	39.4	43.9
KM-C6	C24	1768	0.041	0.141	17.4	20.0	62.6	73.5
KM-D1	D25	3536	0.424	0.284	68.8	0.7	30.6	36.6
KM-D2	D26	5000	0.342	0.261	60.5	1.4	38.1	46.8
KM-D2	D30	5478	0.223	0.241	58.7	5.6	35.7	64.7
KM-D3	D27	3536	0.306	0.259	56.5	0.9	42.6	40.7
KM-D3	D28	2196	0.319	0.287	56.6	1.4	42.0	44.1
KM-D4	D26	5000	0.342	0.261	60.5	1.4	38.1	46.8
KM-D4	D33	2846	0.312	0.228	71.4	1.5	27.1	66.5
KM-D5	D31	5806	0.360	0.285	60.3	1.1	38.6	46.0
KM-D5	D32	7604	0.369	0.309	64.5	1.1	34.4	40.2
KM-D6	D29	6340	0.419	0.376	54.0	0.5	45.6	20.2
KM-D6	D31	5806	0.360	0.285	60.3	1.1	38.6	46.0
KM-D6	D32	7604	0.369	0.309	64.5	1.1	34.4	40.2
KM-D7	D32	7604	0.369	0.309	64.5	1.1	34.4	40.2
KM-D7	D33	2846	0.312	0.228	71.4	1.5	27.1	66.5

AI Based Interpretation

About this section

Focused specifically on the SUPPORTS/REFUTES signals — NOT_ENOUGH_INFO (~81% of IAS pairs) is set aside here as the least informative label. Written against the current scored data (2026-07-06); re-derive the numbers below from the tables above before trusting this after a rerun.

Key finding — IAS is weaker and more REFUTES-skewed than GA1. Overall, IAS REFUTES (11.1%) actually exceeds SUPPORTS (8.4%) — the opposite of GA1’s overall SUPPORTS lean (16.1% vs 13.0%). Unlike GA1, no IAS BM reaches D-series levels of confidence; the strongest SUPPORTS signal is markedly weaker.

Strongest SUPPORTS signal

D33 (25.9% SUPPORTS vs 9.5% REFUTES, a 16-point lead) is the clearest positive signal in IAS, appearing under both KM-D4 and KM-D7. A handful of others lean the same way but at much lower absolute rates: B13 (19.4% vs 10.8%), D30 (8.7% vs 3.0%), D27 (7.5% vs 1.8%), and D25 (12.5% vs 7.2%). None of these approach GA1’s D3/D2/D1/D12 margins.

REFUTES inflation

The BMs with the widest REFUTES-over-SUPPORTS gap — the same zero-shot failure mode documented for GA1’s KMs A–C:

BM	SUPPORTS	REFUTES	Gap (REFUTES − SUPPORTS)
A1	7.5 %	22.6 %	+15.1
A8	3.6 %	16.7 %	+13.1
D32	22.5 %	33.5 %	+11.0
A4	7.6 %	17.6 %	+10.0
C19	5.0 %	14.3 %	+9.3
D28	8.1 %	14.7 %	+6.6

D32 is a special case: both SUPPORTS and REFUTES are elevated (22.5% and 33.5%) rather than REFUTES alone — it appears under three different KMs (KM-D5, KM-D6, KM-D7), which may mean it’s a genuinely contested or multi-faceted claim rather than a pure model artifact, and is worth a manual look before writing it off as noise.

Operational implications

Do not act on raw REFUTES labels anywhere in IAS without Phase-2 LLM review — the overall REFUTES-over-SUPPORTS skew makes this an even higher priority here than for GA1’s KMs A–C.
D33 is the one IAS BM currently worth treating as a real signal — everything else is either near parity or REFUTES-leaning.
Use p_supports − p_refutes for ranking rather than argmax labels; IAS’s overall skew means a plain SUPPORTS-label count will systematically undercount real support.
D32 deserves manual review before either accepting or discarding it — high confidence in both directions, across multiple KMs, is a different failure mode from the single-direction REFUTES inflation seen elsewhere.

IAS (bge_m3_zeroshot_atomic_bm, atomic_bm)

686,020 works scored
7 background messages
7 key messages
79.1% uncertain (confidence < 0.60)

Label distribution

Label	Works	%
SUPPORTS	236379	34.5
NOT_ENOUGH_INFO	349319	50.9
REFUTES	100322	14.6

Interactive BM Explorer

Per Key Message

KM	SUPPORTS %	NEI %	REFUTES %
KM-A1	49.2	40.3	10.5
KM-A2	27.4	51.9	20.7
KM-A3	29.6	55.6	14.8
KM-A5	28.8	60.9	10.3
KM-B1	43.7	45.1	11.2
KM-B3	43.7	45.1	11.2
KM-B4	43.7	45.1	11.2

Per Background Message

KM	BM	SUPPORTS %	NEI %	REFUTES %
KM-A1	A1	49.2	40.3	10.5
KM-A2	A2	30.4	48.2	21.4
KM-A2	A3	22.1	58.6	19.4
KM-A3	A4	28.2	57.4	14.4
KM-A3	A5	32.2	52.2	15.6
KM-A5	A7	28.8	60.9	10.3
KM-B1	B14	43.7	45.1	11.2
KM-B3	B14	43.7	45.1	11.2
KM-B4	B14	43.7	45.1	11.2

Confidence by label

Label	n	Mean	Median	P25	P75
SUPPORTS	236379	0.507	0.485	0.426	0.568
NOT_ENOUGH_INFO	349319	0.534	0.518	0.445	0.611
REFUTES	100322	0.440	0.425	0.389	0.473

Alignment score per BM

KM	BM	n	Mean aln	SD	SUPP %	REF %	NEI %	Unc %
KM-A1	A1	64701	0.135	0.192	49.2	10.5	40.3	85.0
KM-A2	A2	126191	0.043	0.224	30.4	21.4	48.2	79.3
KM-A2	A3	71570	0.005	0.202	22.1	19.4	58.6	75.2
KM-A3	A4	108911	0.046	0.193	28.2	14.4	57.4	77.4
KM-A3	A5	54490	0.053	0.182	32.2	15.6	52.2	78.6
KM-A5	A7	77607	0.074	0.204	28.8	10.3	60.9	72.7
KM-B1	B14	60850	0.113	0.194	43.7	11.2	45.1	82.2
KM-B3	B14	60850	0.113	0.194	43.7	11.2	45.1	82.2
KM-B4	B14	60850	0.113	0.194	43.7	11.2	45.1	82.2

AI Based Interpretation

About this section

Strongest SUPPORTS signal

REFUTES inflation

The BMs with the widest REFUTES-over-SUPPORTS gap — the same zero-shot failure mode documented for GA1’s KMs A–C:

BM	SUPPORTS	REFUTES	Gap (REFUTES − SUPPORTS)
A1	7.5 %	22.6 %	+15.1
A8	3.6 %	16.7 %	+13.1
D32	22.5 %	33.5 %	+11.0
A4	7.6 %	17.6 %	+10.0
C19	5.0 %	14.3 %	+9.3
D28	8.1 %	14.7 %	+6.6

Operational implications

Do not act on raw REFUTES labels anywhere in IAS without Phase-2 LLM review — the overall REFUTES-over-SUPPORTS skew makes this an even higher priority here than for GA1’s KMs A–C.
D33 is the one IAS BM currently worth treating as a real signal — everything else is either near parity or REFUTES-leaning.
Use p_supports − p_refutes for ranking rather than argmax labels; IAS’s overall skew means a plain SUPPORTS-label count will systematically undercount real support.
D32 deserves manual review before either accepting or discarding it — high confidence in both directions, across multiple KMs, is a different failure mode from the single-direction REFUTES inflation seen elsewhere.

BBA (bge_m3_zeroshot_naive_bm, naive_bm)

No NLI scores available yet for this assessment/model/granularity combination.

BBA (bge_m3_zeroshot_complete_bm, complete_bm)

No NLI scores available yet for this assessment/model/granularity combination.

BBA (bge_m3_zeroshot_atomic_bm, atomic_bm)

No NLI scores available yet for this assessment/model/granularity combination.

Funnel Reports

Companion per-assessment reports narrow REFUTES- and SUPPORTS-labelled citing works down through NLI certainty and LLM re-verification. See each report’s own Methodology section for the full 3-level sieve definition.

# Plain relative links, not iframes/copy_into_report_files(): each funnel
# report is a full standalone page meant to be opened on its own, and both
# it and this report sit as flat siblings both in the project root and in
# output/reports/ (build_report_output_dir()), so a relative link resolves
# in both places with no copying needed here.
label_funnel_html_paths <- rendered_report_html(
  "^IPBES_(REFUTES|SUPPORTS)_Report_.*\\.html$"
)
# Assessment ids are alphanumeric (GA1, IAS); the trailing blob is
# "_<model>" (optional -- omitted only for the deberta_zeroshot default)
# and "_<granularity>" (always present -- every granularity, including
# naive_bm, is explicit now). Both segments can contain underscores
# themselves, so the combined trailing blob is split by checking whether it
# ENDS in one of the fixed, known granularity values (same list
# _targets.R's own nli_granularities uses) rather than a naive underscore
# split, which would be ambiguous for e.g. "bge_m3_zeroshot_complete_bm".
known_granularities <- c("naive_bm", "complete_bm", "atomic_bm")
for (p in label_funnel_html_paths) {
  m <- regmatches(
    basename(p),
    regexec("^IPBES_([A-Z]+)_Report_([A-Za-z0-9]+)(?:_(.+))?\\.html$", basename(p))
  )[[1]]
  label <- m[[2]]
  id <- m[[3]]
  suffix <- m[[4]]
  gran <- "naive_bm"
  model <- "deberta_zeroshot"
  if (nzchar(suffix)) {
    hit <- known_granularities[vapply(known_granularities, function(g) grepl(paste0("(^|_)", g, "$"), suffix), logical(1))]
    if (length(hit)) {
      gran <- hit[[1]]
      suffix <- sub(paste0("_?", gran, "$"), "", suffix)
    }
    if (nzchar(suffix)) model <- suffix
  }
  cat(sprintf("- [%s %s Funnel Report (%s, %s)](%s)\n", id, label, model, gran, basename(p)))
}

QA Reports

Per-assessment QA check for how nli_ready_evidence_parquet actually split each Background Message into claims under the currently active granularity setting — not a scoring result. See the report’s own Methodology section for details.

# Same plain-relative-link convention as the Funnel Reports section above.
bm_split_html_paths <- rendered_report_html("^QA_BM_Split_Report_.*\\.html$")
for (p in bm_split_html_paths) {
  m <- regmatches(
    basename(p),
    regexec("^QA_BM_Split_Report_([A-Za-z0-9]+)(?:_(.+))?\\.html$", basename(p))
  )[[1]]
  id <- m[[2]]
  gran <- if (nzchar(m[[3]])) m[[3]] else "naive_bm"
  cat(sprintf("- [%s QA BM Split Report (%s)](%s)\n", id, gran, basename(p)))
}

Per-(assessment, granularity) QA check on Phase 1 (zero-shot NLI) scoring itself — claims x citing works with title/abstract/DOI and label/ confidence, plus a label x confidence-decile matrix. Not a scoring result; see the report’s own Methodology section for details.

# Same suffix-splitting logic as the Funnel Reports section above --
# model_suffix and gran_suffix are both optional and concatenated in the
# filename, so known_granularities (defined there) disambiguates them here
# too.
nli_scores_qa_html_paths <- rendered_report_html("^QA_NLI_Scores_Report_.*\\.html$")
for (p in nli_scores_qa_html_paths) {
  m <- regmatches(
    basename(p),
    regexec("^QA_NLI_Scores_Report_([A-Za-z0-9]+)(?:_(.+))?\\.html$", basename(p))
  )[[1]]
  id <- m[[2]]
  suffix <- m[[3]]
  gran <- "naive_bm"
  model <- "deberta_zeroshot"
  if (nzchar(suffix)) {
    hit <- known_granularities[vapply(known_granularities, function(g) grepl(paste0("(^|_)", g, "$"), suffix), logical(1))]
    if (length(hit)) {
      gran <- hit[[1]]
      suffix <- sub(paste0("_?", gran, "$"), "", suffix)
    }
    if (nzchar(suffix)) model <- suffix
  }
  cat(sprintf("- [%s QA NLI Scores Report (%s, %s)](%s)\n", id, model, gran, basename(p)))
}

Per-(assessment, llm_config) QA check on Phase 2 (LLM verification) — independent LLM review of NLI-flagged claim/work pairs, plus a full audit of every key paper regardless of its own NLI label/confidence. Not a scoring result; see the report’s own Methodology section for details.

# Filename is always "..._<assessment>_<llm_config>.html" -- llm_config is
# always present (no optional-default-unsuffixed convention here, unlike
# nli_model_suffix()'s historical exception for "deberta_zeroshot").
llm_verification_qa_html_paths <- rendered_report_html(
  "^QA_LLM_Verification_Report_.*\\.html$"
)
for (p in llm_verification_qa_html_paths) {
  m <- regmatches(
    basename(p),
    regexec("^QA_LLM_Verification_Report_([A-Za-z0-9]+)_(.+)\\.html$", basename(p))
  )[[1]]
  id <- m[[2]]
  cfg <- m[[3]]
  cat(sprintf("- [%s QA LLM Verification Report (%s)](%s)\n", id, cfg, basename(p)))
}

GA1 QA LLM Verification Report (openrouter_cheap)

Reuse

CC BY 4.0

Citation

BibTeX citation:

@report{m_krug2026,
  author = {M Krug, Rainer},
  title = {Fact {Checker} {DRAFT}},
  date = {2026-09-18},
  doi = {xxxxxxxxxxxxxx},
  langid = {en},
  abstract = {The assessments Linked Open Data is used to conduct
    analysis and searches to discover new knowledge related to the
    assessments.}
}

For attribution, please cite this work as:

M Krug, Rainer. 2026. Fact Checker DRAFT. https://doi.org/xxxxxxxxxxxxxx.