Two-Phase Alignment Scoring: NLI → LLM

Technical Design Document

Published

August 19, 2026

Two-Phase Alignment Scoring: NLI → LLM

Status: implemented. The llm_verification_parquet target (R/build_llm_verification_parquet.R) runs Phase 2 below on top of the live Phase 1 NLI scoring chain (nli_scores_by_claim_evidence, see TD_BM_NLI_approach). Its architecture is ported from the sibling Categorisation_Literature project’s LLM epistemology classifier (R/llm_epistemology.R there) — see Ported architecture below for what came across unchanged and what didn’t.

Overview

A two-phase pipeline that combines the speed and scale of NLI with the nuance and explanatory power of an LLM. NLI handles all pairs cheaply; the LLM only processes the remainder where a richer judgement is needed.

This is pure inference — no model training is involved. The LLM acts as a classifier and explainer on top of the NLI output.

Phase 1 — NLI (all pairs)

Run the existing nli_scores_by_claim_evidence pipeline across all citing works × BM claims. The NLI model returns label + p_supports / p_refutes / p_nei / confidence / uncertain for every pair.

Routing after Phase 1 — originally a single fixed filter (label == "REFUTES" | uncertain), now per-config: each llm_verification config picks its own routed set via two fields, read by select_llm_verification_candidates() in R/build_llm_verification_parquet.R:

  • nli_labels — null/omitted (any label, no filter) or one/more labels, e.g. [REFUTES], [REFUTES, SUPPORTS].
  • nli_certainty — null/omitted (any), or one/more of "certain" (uncertain == FALSE) / "uncertain" (uncertain == TRUE), e.g. certain, or [certain, uncertain] (equivalent to null, spelled out).

The two combine with AND to select candidates. This can’t express the original rule’s mixed per-label logic (“REFUTES at any confidence, OR any other label if uncertain”) in a single config — that’s a deliberate trade for simplicity; define a separate config per bucket instead of widening one config to cover several.

Listing several labels or certainties in one config does not merge their rows into one bucket, though: the hive partition each row lands in (nli_route, see Output schema) is derived per row from that row’s own actual outcome, not from the filter that let it through. So nli_labels: [REFUTES, SUPPORTS], nli_certainty: certain routes both labels in one config call, but still writes them to two separate partitions, nli_route=REFUTES-certain and nli_route=SUPPORTS-certain — never a merged REFUTES_SUPPORTS-certain.

Measured against the real evidence-scored corpus (3,547,231 scored pairs, of which 2,357,718 / 66.5% would be routed under the original combined rule — far above the original design’s 10-20% estimate, the bulk being NOT_ENOUGH_INFO pairs NLI itself wasn’t confident about), the four buckets that combined rule was built from break down as:

nli_labels nli_certainty pairs (of 3,547,231 scored) ~cost (gpt-4o-mini)
REFUTES certain 26,861 ~$4
SUPPORTS certain 52,061 ~$7.75
REFUTES uncertain 403,400 ~$60
SUPPORTS uncertain 388,554 ~$58
NOT_ENOUGH_INFO uncertain 1,538,903 ~$231

REFUTES + certain was the original first step; SUPPORTS + certain was added once the REFUTES funnel report (IPBES_Label_Funnel_Report.qmd) was generalized to a SUPPORTS counterpart — all four shipped configs now route both (nli_labels: [REFUTES, SUPPORTS]), 78,922 pairs combined, ~$11.75 at gpt-4o-mini rates. SUPPORTS is nearly 2x the size of REFUTES at the same certainty band (52,061 vs. 26,861) — GA1’s NLI scores skew SUPPORTS-heavy overall (37,371 of the 52,061 SUPPORTS-certain pairs are GA1’s), consistent with what the main report’s own “AI Based Interpretation” section already says about GA1’s KM D. The remaining two uncertain buckets and NOT_ENOUGH_INFO stay available to route later, by defining additional configs or widening these two fields on existing ones — nothing else in the pipeline needs to change to pick them up.

Phase 2 — LLM (remainder only)

Ported architecture

Rather than build Phase 2 from scratch, its plumbing is ported from Categorisation_Literature/R/llm_epistemology.R (an LLM classifier scoring papers against epistemological traditions in a sibling project), because that code already solves the reliability problems this task hits:

Ported as-is Why
Resumable per-item JSON cache, keyed on a hash of (system prompt, user template, output schema) A crashed or interrupted run does not restart from zero, and editing either prompt or the schema correctly starts a fresh cache namespace instead of silently reusing stale answers
Verbatim-quote verification (quote_is_verbatim()) The LLM is required to cite a quote supporting its verdict; this checks the quote actually occurs (normalised: lowercased, punctuation-stripped, whitespace-collapsed, ellipsis-split into fragments) in the premise text it was shown, and demotes the verdict to NOT_ENOUGH_INFO if not. Measured on the sibling project’s corpus: 2-4% of otherwise well-formed responses cited a fabricated or reconstructed quote
Required-boolean evidence gate (sufficient_evidence) rather than a nullable score Sidesteps inconsistent JSON-schema null support across OpenRouter models; abstention is a first-class, always-present field rather than an edge case
Fail-loud on structured-output non-compliance (frac >= 0.5 → stop(), else warning()) If a model ignores JSON-schema mode, every call fails to parse and the run would otherwise look like 100% honest NOT_ENOUGH_INFO rather than a broken pipeline
Retry pass for failed calls Provider errors (empty body, finish_reason: error) are usually transient; a blind retry recovers most of them without cost beyond the retry itself

What didn’t port as-is: the sibling project’s schema scores an array of N traditions per paper in one call (assessments = type_array(...)), which needs unwrapping a list-column ellmer returns for nested types (a genuine footgun there — see that project’s normalise_assessment() comment). Phase 2 here scores exactly one (claim, work) pair per call, so llm_verification_output_type() is a flat object with no nested field, and normalise_llm_verification() has no list-column to unwrap. Partitioning combines both projects’ conventions: llm_config=<name> (the sibling’s per-config separation, so switching active or comparing two named configs never overwrites either one’s scored rows) nested with assessment/km/bm (this project’s own convention, matching nli_scores_evidence).

One more difference worth naming: Phase 1 (score_one_claim.R) dispatches across a fixed pool of RunPod hosts via file locks, because each host is a dedicated resource to load-balance across. Phase 2 talks to OpenRouter, a shared multi-tenant endpoint — there is nothing to load-balance, so ellmer::parallel_chat_structured()’s own max_active concurrency is sufficient and no crew/file-lock machinery was needed. One target call per assessment loops over its own routed candidates internally.

Candidate scoping (subset)

Status: parked. This section describes a second, independent narrowing lever, orthogonal to the nli_labels/nli_certainty routing above — still implemented and tested, but not the active cost lever right now (the routing fields are doing that job instead, via REFUTES + certain). subset: "sm" remains available on any config whenever it’s worth revisiting.

The original combined routing rule (REFUTES at any confidence, OR any label if uncertain) alone left 66.5% of all NLI-scored pairs in scope for Phase 2 — far more than practical to run for real. This lever addresses that structurally rather than by tightening thresholds: IPBES’s own evidence-reference structure. Each evidence-segmented claim ends with a brace group like {5.4.1, 5.4.2} naming the specific sub-chapter(s) it draws on, and refs_parquet’s sm column already links specific references (dois) to those same sub-chapter identifiers. Chaining sm -> seed doi -> seed OpenAlex work id -> citing work (via the existing snowball edges) gives, per claim, an allow-list of citing works actually tied to that claim’s own evidentiary basis, rather than every citing work found anywhere under the whole BM.

This is implemented as its own target, llm_candidate_scope_parquet (R/build_llm_candidate_scope_parquet.R), feeding into llm_verification_parquet — not folded into the LLM-stage builder itself, so the scoping logic and its target-level identity stay separately inspectable and independently cacheable. It reads only already-existing, unmodified targets (key_messages_parquet, refs_parquet, works_parquet, the snowball edges dataset, and — for a drift sanity check only — nli_ready_evidence_parquet) and writes to its own output root. Nothing about it edits download_works.R, build_snowball_parquet.R, build_works_citing_parquet.R, build_nli_ready_evidence_parquet.R, or the NLI scoring chain — targets invalidation only flows forward from something that actually changes, and adding a new consumer of an existing target’s output does not retroactively invalidate it.

One deliberate non-reuse: extract_claim_evidence_tokens() in the new file duplicates the sentence-splitting/terminal-brace/buffer logic from segment_bm_by_evidence() (R/build_nli_ready_evidence_parquet.R) rather than calling it, so that the two stay behaviorally identical (verified directly: run against the real BM text, both functions produce exactly the same segment count and boundaries in every test case, including a segment with multiple inline brace groups). Calling the original instead would work too — until someone edits it: targets hashes function bodies as dependencies, so even a behavior-preserving change there would mark nli_ready_evidence_parquet, and everything downstream of it including the NLI scoring chain, outdated. build_llm_candidate_scope_parquet() also cross-checks its own derived claim_ids against nli_ready_evidence_parquet’s real ones at build time and warning()s (not stop()s) on any mismatch, so a future edit to either segmentation implementation that silently drifts out of sync gets caught rather than silently mis-attributing evidence.

Exposed per llm_verification config via subset::

subset Behavior
"all" Every citing work found under the claim’s BM (today’s default; the 66.5%/2.36M figure above)
"sm" Only citing works tracing back to a seed reference whose sm matches the claim’s own evidence sub-chapter(s)

Two policy decisions, made deliberately rather than defaulted into:

  • A claim with no evidence braces at all falls back to unrestricted (keeps every one of its routed candidates) rather than being excluded — no claim is ever silently dropped from Phase 2 review just because it lacks a brace. The same fallback applies if the whole scope target found nothing to write for an assessment (e.g. an unexpected data gap): absence of scope data is never read as “review nothing.”
  • Sub-chapter token matching is prefix-based, not exact, after normalizing (lowercase; strip box/table/spm table prefixes; strip (...) parenthetical annotations; split on comma/semicolon). The sm field is messy free text (311 distinct values observed: comma- and semicolon-separated lists, inconsistent box/Box casing, trailing letter suffixes like .a, occasional malformed strings with unbalanced parentheses). Prefix matching handles the common cases cleanly — "5.4.2.1" matches "5.4.2.1.a", a bare "4" matches any "4.x.y" — but is deliberately not exhaustive; the malformed cases are an accepted, disclosed gap. subset: "all" remains available as the exhaustive fallback whenever "sm"’s narrower recall is a concern for a given analysis.

Prompt design

Two files, tracked as format = "file" targets so editing either invalidates the cache namespace and the target:

The NLI result is passed as a prior for two reasons: it focuses the LLM on the borderline or surprising case at hand, and it gives the LLM something concrete to agree with, refine, or override, rather than scoring cold.

Model choice

input/config.yaml’s llm_verification: block follows the same active + named configs structure as nli: above — switch tiers by changing active to a different key, without touching any other config:

llm_verification:
  active: openrouter_cheap

  configs:
    openrouter_cheap:
      model: "openai/gpt-4o-mini"
      temperature: 0
      max_tokens: 8000
      max_active: 24
      max_retries: 2
      subset: all
      nli_labels: [REFUTES, SUPPORTS]
      nli_certainty: certain

    openrouter_cheap_sm:
      model: "openai/gpt-4o-mini"
      temperature: 0
      max_tokens: 8000
      max_active: 24
      max_retries: 2
      subset: sm
      nli_labels: [REFUTES, SUPPORTS]
      nli_certainty: certain

    openrouter_midtier:
      model: "google/gemini-2.5-flash"
      temperature: 0
      max_tokens: 8000
      max_active: 24
      max_retries: 2
      subset: all
      nli_labels: [REFUTES, SUPPORTS]
      nli_certainty: certain

    openrouter_toptier_gpt5:
      model: "openai/gpt-5"
      temperature: 0
      max_tokens: 8000
      max_active: 24
      max_retries: 2
      subset: all
      nli_labels: [REFUTES, SUPPORTS]
      nli_certainty: certain

The model tiers are carried over directly from the sibling Categorisation_Literature project’s epistemology: config block (same model choices, same max_active/max_tokens values — see below for why those two specifically were kept rather than re-derived). subset and nli_labels/nli_certainty are new here — see Candidate scoping above and Routing after Phase 1 for what they do; all four configs currently route both REFUTES + certain (the original first step) and SUPPORTS + certain (added once the REFUTES funnel report was generalized to a SUPPORTS counterpart — see IPBES_Label_Funnel_Report.qmd), and differ only in model/subset.

Config Model Subset Notes
openrouter_cheap openai/gpt-4o-mini all Default; good balance of quality and cost for a first pass
openrouter_cheap_sm openai/gpt-4o-mini sm Same model, narrowed to the sm-derived candidate scope — parked (see Candidate scoping)
openrouter_midtier google/gemini-2.5-flash all Step up when the cheap tier is topic-matching rather than judging the claim — spot-check quote/explanation
openrouter_toptier_gpt5 openai/gpt-5 all For a final, citable verification pass; a reasoning model

max_active: 24 is the concurrency the sibling project actually ran at ~2,800-call scale against OpenRouter without hitting rate limits — kept as the proven value rather than a smaller guess.

max_tokens: 8000 is a cap, not a reservation (unused tokens are not billed). It looks oversized for this schema’s four short scalar fields on a non-reasoning model, but the reasoning tiers (gpt-5 here, gemini-2.5-pro as a possible fourth) spend hidden thinking tokens against the same budget before any visible JSON appears — too low a cap truncated responses mid-JSON in the sibling project even on a larger per-tradition-array schema. Lower it per-config later only if measured usage on a specific non-reasoning model justifies it; there’s no cost pressure to pre-tune it.

max_tokens must be set explicitly in every config — left to the provider default, long responses can truncate mid-JSON on some models (the exact failure mode the sibling project’s epistemology classifier hit on google/gemini-2.5-flash at max_tokens: 4000, which is why that project settled on 8000 for every tier).

Output schema

One row per LLM-reviewed pair, written to output/llm_verification/scores/llm_config=<config>/subset=<subset>/assessment=<id>/nli_route=<route>/km=<km>/bm=<bm>/. <route> is nli_route_label()’s hive-safe encoding of that row’s own NLI outcome, e.g. REFUTES-certain or SUPPORTS-uncertain — not a single value per config call. So one build_llm_verification_parquet() call can write several nli_route= subdirectories under the same llm_config/subset/assessment (one per distinct label/certainty combination actually present among its routed candidates), and any two configs that happen to route the same outcome land on the same nli_route value regardless of their own names. llm_config/subset/assessment are the fixed prefix fully deleted and rewritten on each call; nli_route/km/bm vary within it based on the data:

tibble(
  llm_config          = character(), # selected input/config.yaml llm_verification.configs entry name
  subset              = character(), # "all" or "sm" -- which candidate scoping this config used
  nli_config          = character(), # NLI profile used in Phase 1
  assessment          = character(),
  nli_route           = character(), # this ROW's own outcome, e.g. "REFUTES-certain" -- see nli_route_label()
  km                  = character(),
  bm                  = character(),
  claim_id            = character(),
  work_id             = character(),
  claim               = character(), # the BM claim text, carried through for convenience
  nli_label           = character(), # original NLI label
  uncertain           = logical(),   # NLI's own confidence < uncertain_threshold flag
  nli_confidence      = double(),
  p_supports          = double(),
  p_refutes           = double(),
  p_nei               = double(),
  llm_model           = character(),
  llm_label           = character(),   # SUPPORTS / REFUTES / NOT_ENOUGH_INFO
  llm_agrees          = logical(),     # TRUE if llm_label == nli_label
  sufficient_evidence = logical(),     # FALSE forces llm_label = NOT_ENOUGH_INFO
  quote               = character(),   # verbatim quote backing llm_label
  quote_verbatim      = logical(),     # NA if not applicable, FALSE if fabricated (then demoted)
  explanation         = character()    # 1-2 sentence free-text explanation
)

quote_verbatim = FALSE rows have already been demoted to llm_label = "NOT_ENOUGH_INFO" / sufficient_evidence = FALSE by the time they reach this table — the column is kept so a reader can distinguish “the LLM abstained” from “the LLM claimed evidence that wasn’t real,” which is a meaningfully different failure to audit.

Caching

Two layers, doing different jobs:

  • Resumable per-pair cache (the one that matters for cost and crash safety): one JSON file per (claim_id, work_id) pair under output/llm_verification/raw/model=<model>/prompt=<hash>/, where the hash covers the system prompt, the user template, and the output schema (adding a field to the schema silently reusing cached responses that cannot contain it was a real bug in the sibling project before the schema was added to the hash — ported here from the start instead of rediscovered).
  • Provider-side automatic prefix caching (OpenAI, Google) — a side benefit of prompt ordering (system → fixed instructions → variable per-pair content), not something explicitly engineered here since Phase 2 has no long shared prefix across pairs the way Phase 1’s truth-document design once did (see TD_LLM_approach.qmd for that now-removed design).

Using LLM Output as Training Data

The LLM explanations are high-quality labeled examples for fine-tuning the NLI model (see TD_NLI_training.qmd):

  • llm_agrees = TRUE → confirms the NLI label; strong training signal
  • llm_agrees = FALSE → NLI error with explanation; gold-standard correction
  • REFUTES cases reviewed by LLM → rare but valuable negative examples

This dataset now exists as a side effect of running llm_verification_parquet — no separate collection step is needed to start using it.

Estimated Cost (Phase 2 only)

Per-1k-pair unit costs, average prompt ~400 tokens, completion ~150 tokens:

Model Input cost Output cost Total per 1k pairs
gpt-4o-mini $0.15/1M $0.60/1M ~$0.15
gpt-4o $2.50/1M $10/1M ~$2.50
claude-haiku-4-5 $0.80/1M $4/1M ~$0.52

Scaled to each nli_labels/nli_certainty bucket, measured on the real evidence-scored corpus (3,547,231 pairs total):

Bucket Pairs gpt-4o-mini gpt-5-tier
REFUTES + certain (original first step) 26,861 ~$4 ~$65
SUPPORTS + certain (added alongside the SUPPORTS funnel report) 52,061 ~$7.75 ~$126
REFUTES + SUPPORTS, certain (current default, all 4 configs) 78,922 ~$11.75 ~$191
REFUTES + uncertain 403,400 ~$60 ~$960
SUPPORTS + uncertain 388,554 ~$58 ~$925
NOT_ENOUGH_INFO + uncertain 1,538,903 ~$231 ~$3,660
All labels, both certainty bands (the original fixed-rule design) 2,357,718 (66.5% of scored) ~$375 ~$5,900+
sm-narrowed variant of the combined set 1,792,007 (24% smaller) ~$269 ~$4,480

openrouter_cheap (gpt-4o-mini) is the configured default, currently routing REFUTES + SUPPORTS, both certain, across all four shipped configs — the smallest, cheapest, highest-signal bucket for each label, and the confirmed first step. The remaining uncertain/NOT_ENOUGH_INFO buckets stay available by defining additional configs or widening nli_labels/nli_certainty on existing ones. The resumable per-pair cache means cost is paid once per (prompt hash, pair), not once per tar_make() — and a pair’s cached answer is valid under any combination of routing/subset that happens to select it again later.

Where this sits in the pipeline

llm_verification_parquet and llm_candidate_scope_parquet are both live, active targets (not commented out) — tar_make() runs them. llm_candidate_scope_parquet is always computed regardless of which llm_verification config is active — it makes no paid API calls of its own (pure local joins over already-downloaded parquet), so subset: "all" configs simply never read its output rather than the target being conditionally skipped.

report_fact_checker is no longer safe to render blind. Originally neither Phase 2 target was a dependency of it, so rendering the report never forced an OpenRouter call. That changed once the label funnel reports (refutes_funnel_data/supports_funnel_data → report_refutes_funnel_html/report_supports_funnel_html, both folded into report_fact_checker’s dependency list) started reading llm_verification_parquet directly — a plain tar_make() or tar_make(names = "report_fact_checker") will now rebuild llm_verification_parquet first if it’s outdated, which means real OpenRouter spend can happen as a side effect of “just rendering the report.” This is exactly what happened when SUPPORTS was added to nli_labels: that edit alone makes llm_verification_parquet outdated, so the next non-shortcut tar_make() touching report_fact_checker would try to review the new ~52,061-pair SUPPORTS-certain backlog (~$7.75 at gpt-4o-mini rates) automatically. To render the report (or the funnel reports) against currently on-disk Phase 2 data without triggering a fresh Phase 2 run, use targets::tar_make(names = ..., shortcut = TRUE) — it uses stored metadata for upstream targets instead of re-checking them. Run Phase 2 explicitly and deliberately with targets::tar_make(names = "llm_verification_parquet") when you actually want to spend the money. Phase 2’s output is not yet consumed by the report’s own overview tables/figures or by nli_overview_data — that merge (llm_label overriding nli_label where available) is a deliberate next step, not an oversight; see Recommended Workflow.

See Also