LLM Alignment Scoring Approach

Technical Design Document

Published

August 19, 2026

LLM Alignment Scoring Approach

Status: superseded, source removed. This was the original LLM scoring design (truth/citing JSON documents, a −5..+5 work_alignement scale) and, at the time this document was first written, the intended shape of Phase 2 of the two-phase pipeline. Phase 2 was ultimately implemented differently — a flat SUPPORTS/REFUTES/NOT_ENOUGH_INFO label per pair, ported from a sibling project’s LLM classifier rather than built out from this design — see TD_NLI_LLM_two_phase.qmd for what actually shipped. R/build_prompts_truth_parquet.R, R/build_prompts_citing_parquet.R, R/build_alignement_scores_parquet.R, R/alignement_schema.R, and input/prompts/system_prompt.md/truth_wrapper.md/citing_wrapper.md have all been deleted; the source-code links below are kept as plain paths (not links) for that reason. This document is kept as a record of the design that was considered and the reasoning for not building it — in particular the Shortcomings section below directly motivated the simpler design that replaced it.

Overview

This document describes the LLM-based alignment scoring approach that was considered: feeding structured representations of an IPBES Background Message (the truth) and a citing paper (the candidate) to an LLM via OpenRouter/ellmer, and receiving a structured judgement of whether the paper supports, contradicts, or is neutral toward the BM.

Prompt Architecture

The LLM receives two documents per scoring decision, each preceded by a short wrapper:

[System prompt]          ← task definition, output schema, rules
[Truth wrapper]          ← names the reference document, explains its schema
[Truth JSON]             ← structured BM + SM + source passages
[Citing wrapper]         ← names the candidate paper, explains its schema
[Citing JSON]            ← title, abstract, work_id, doi, publication_year

Why JSON, not Markdown

BM source passages contain literal # characters, parenthetical citations, and table-derived text. Wrapping in Markdown breaks the outer document structure. JSON treats each source as opaque text and round-trips cleanly through the LLM.

Prefix caching

The (system + truth wrapper + truth JSON + citing wrapper) prefix is identical for all candidate papers scored against the same BM. OpenRouter/OpenAI auto-cache identical prefixes — the BM portion costs full price once per BM, and only the citing JSON suffix is the variable cost per candidate. Group calls by BM to exploit this.

Truth Document Structure

One document per (assessment, KM, BM). Was built by R/build_prompts_truth_parquet.R (removed).

{
  "assessment": "GA1",
  "km": "A.",
  "km_label": "...",
  "km_description": "...",
  "bm": "A1",
  "bm_label": "...",
  "bm_description": "...",
  "bm_well_established": "...",
  "bm_established_incomplete": "...",
  "sub_messages": [
    {
      "sm_id": "A1.1",
      "sm_description": "...",
      "sm_well_established": "...",
      "sm_established_incomplete": "...",
      "sources": [
        {
          "section": "...",
          "subsection": "...",
          "content": "raw passage text from the assessment chapter"
        }
      ]
    }
  ]
}

Note: the same source passage may appear under multiple SubMessages — this is accurate to the underlying LOD and not a bug.

Citing Document Structure

One document per citing work. Was built by R/build_prompts_citing_parquet.R (removed).

{
  "assessment": "GA1",
  "km": "A.",
  "bm": "A1",
  "work_id": "W2741809807",
  "doi": "10.1038/...",
  "publication_year": 2021,
  "relation": "citing",
  "title": "...",
  "abstract": "..."
}

Output Schema

Was defined as an ellmer structured type in R/alignement_schema.R (removed). One row per (KM, BM, work_id).

Field Type Description
lm_id string Must match km from the truth document
work_id string Must match work_id from the citing document
km_summary string KM distilled to ≤ 20 words
work_alignement integer −5 to +5 (see rubric below)
confidence number 0–1 confidence in the score
evidence string Supporting excerpt from title/abstract, ≤ 100 words
justification string Explanation of the score, ≤ 100 words

Alignment rubric

Score Meaning
+5 Strong support — findings directly corroborate the BM
+3 Moderate support
+1 Slight support
0 Neutral / not enough information
−1 Slight contradiction
−3 Moderate contradiction
−5 Strong contradiction — findings directly oppose the BM

System Prompt (as designed)

Was located at input/prompts/system_prompt.md (removed). Key rules:

  • Use only the information in the two JSON documents — no outside knowledge
  • Do not invent quotations or claims not present in the abstract
  • If the abstract is missing or very short, say so explicitly and lower confidence
  • Return the JSON object directly — no markdown fences, no preamble

Implementation (as designed, never fully built out)

The scoring pipeline lived in R/build_alignement_scores_parquet.R (removed):

  1. Load truth prompt for the (assessment, KM, BM)
  2. Load citing prompts for the same partition (optionally capped at n_citing)
  3. Build user prompts: truth_wrapper + truth_json + citing_wrapper + citing_json
  4. Call ellmer::parallel_chat_structured() with max_active concurrent requests
  5. Validate structured output; retry individually on failure
  6. Write to output/alignement_scores/assessment=<id>/run_id=<id>/km=<km>/bm=<bm>/model=<model>/replicate=<n>/

Run configuration (model, KMs to score, n_citing, replicates, temperature) was defined per run in an analysis: block of input/config.yaml, since replaced by the simpler llm_verification: block the implemented Phase 2 actually uses.

Shortcomings

Cost at scale

At 330k pairs × ~550 tokens average prompt → ~180M input tokens. At gpt-4o-mini rates ($0.15/1M) that is ~$27 in input tokens alone, plus output. Acceptable for a targeted subset; prohibitive for the full pair space.

Speed

Even with max_active = 8 parallel calls, scoring 330k pairs takes days, not hours. NLI does the same in ~75 minutes.

No probability distribution

The LLM returns a point estimate (work_alignement) and a self-reported confidence. Neither is calibrated. The NLI model returns a proper probability distribution over three classes, which is more useful for downstream thresholding.

Prompt sensitivity

Small changes to the system prompt or wrapper text can shift scores significantly. The NLI model is not affected by prompt wording.

Abstract-only

Both approaches are limited to title + abstract. Full-text classification would require chunking and aggregation and is not yet implemented.

Multi-part BMs

A paper may support one sub-message of a BM while contradicting another. The LLM returns a single score for the whole BM. Splitting BMs into atomic sub-claims before scoring would improve precision but multiply the number of calls.

Relation to the Two-Phase Pipeline (what actually shipped)

TD_NLI_LLM_two_phase.qmd kept the routing idea from this design — NLI uncertain and REFUTES cases go to the LLM, high-confidence SUPPORTS/NOT_ENOUGH_INFO do not — but implemented Phase 2 independently rather than adapting this document’s pipeline:

  • The work_alignement −5..+5 scale was dropped in favour of a flat SUPPORTS/REFUTES/NOT_ENOUGH_INFO llm_label, matching the NLI label scheme it’s reviewing rather than introducing a second, differently-shaped scale to reconcile downstream.
  • The truth/citing structured-JSON document design (SubMessages, source passages) was dropped for a single BM claim + one paper’s title/abstract per call — Phase 2 reviews already-routed (claim, work) pairs one at a time, so there was no need for the fuller per-BM document this design used to batch candidates.
  • The implementation itself was ported from a sibling project’s LLM classifier rather than built out from R/build_alignement_scores_parquet.R — see TD_NLI_LLM_two_phase.qmd for what was reused and why.

The Shortcomings above (cost at scale, no calibrated probability distribution, prompt sensitivity) still apply to any LLM-based approach, and are exactly why Phase 2 only reviews the NLI-flagged subset rather than attempting the full pair space this document’s n_citing-capped design would have needed to bound cost on.

See Also

  • TD_BM_NLI_approach.qmd — Phase 1 design
  • TD_NLI_LLM_two_phase.qmd — the implemented two-phase pipeline
  • TD_NLI_training.qmd — fine-tuning the NLI model
  • R/build_alignement_scores_parquet.R, R/alignement_schema.R, input/prompts/system_prompt.md/truth_wrapper.md/citing_wrapper.md — all removed; this document is their only remaining record