targets Pipeline Documentation

Technical Design Document

Published

August 18, 2026

targets Pipeline Documentation

This document describes the targets pipeline for the IPBES BM Fact Checker project.

Note: this document predates the reporting/explorer layer and is stale in that respect — it does not describe the nli_ready_evidence_parquet → nli_claim_units_evidence → nli_scores_by_claim_evidence chain, or the downstream nli_overview_data, nli_overview_figures, nli_bm_explorer_html, td_doc_html, and report_fact_checker targets, all of which exist and run today. See _targets.R and CLAUDE.md’s R-files table for the current, authoritative target list.

Overview

The pipeline fetches IPBES assessment data from the IPBES Linked Open Data (LOD) repository in RDF/Turtle format, queries the assessments via SPARQL, and produces per-assessment Parquet datasets for refs, sections, key-messages, Zotero items, OpenAlex works, snowball results, and NLI alignment scores.

The active scoring approach is NLI (nli_scores_by_claim / nli_scores_by_claim_evidence): each citing work is classified as SUPPORTS / REFUTES / NOT_ENOUGH_INFO against its partition’s Background Message using a zero-shot DeBERTa NLI model served on RunPod (see external/runpod/docker/nli-runpod/ and TD_BM_NLI_approach.qmd).

Phase 2 LLM verification (llm_verification_parquet) reviews whichever slice of NLI scores each config’s nli_labels/nli_certainty fields select (currently REFUTES + certain on every shipped config — the confirmed first step, ~27k of 3.5M scored pairs), one ellmer::chat_openrouter() call per (claim, work) pair, optionally narrowed further by llm_candidate_scope_parquet (per-claim citing-work scoping via IPBES’s own sub-chapter evidence references — parked for now) — see TD_NLI_LLM_two_phase.qmd for the design and R/build_llm_verification_parquet.R/R/build_llm_candidate_scope_parquet.R for the implementation. The earlier truth/citing-document LLM design it replaced (prompts_truth_parquet → prompts_citing_parquet → alignement_scores_parquet) has been removed; see TD_LLM_approach.qmd for that record.

Fuseki startup and shutdown are handled inside the parquet builders rather than as separate targets. When sparql_url: fuseki, each builder boots an in-memory Fuseki, POSTs the TTL into the named graph http://ontology.ipbes.net/report/<id> via the Graph Store Protocol, runs its query, and tears Fuseki down.

Running the Pipeline

# Run all outdated targets
targets::tar_make()

# Visualise the dependency graph
targets::tar_visnetwork()

# Check which targets are outdated
targets::tar_outdated()

# Load a target into your session
targets::tar_load(refs_parquet)

Pipeline Diagrams

Rendered Workflow (NLI approach)

Workflow

Pipeline

Auto-generated pipeline flow based on _targets.R definition:

Pipeline

Pipeline DAG

config_file (file)
    ├── sparql_url                              ← SPARQL backend string
    ├── nli_active                              ← active NLI profile name
    ├── nli_config                              ← resolved NLI profile (host, model, etc.)
    ├── assessments_list
    │       └── assessment (list, branched)
    │               ├── ttl_path (file)                    ← Target 1
    │               ├── refs_sparql (file) ──────────────── queries/refs.sparql
    │               ├── refs_parquet (file)                ← Target 2a
    │               ├── sections_sparql (file) ──────────── queries/sections.sparql
    │               ├── sections_parquet (file)            ← Target 2b
    │               ├── key_messages_sparql (file) ──────── queries/key_messages.sparql
    │               ├── key_messages_parquet (file)        ← Target 2b2
    │               ├── zotero_parquet (file)              ← Target 2c
    │               ├── works_parquet (file)               ← Target 2d
    │               ├── snowball_parquet (file)            ← Target 2e
    │               └── works_citing_parquet (file)        ← Target 2f
    └── nli_config ─────────────────────────────────────────┐
        assessment ─────────────────────────────────────────┤
        key_messages_parquet ───────────────────────────────┤→ nli_scores_parquet ← Target 2g
        works_citing_parquet ───────────────────────────────┘

Phase 2 — LLM verification + candidate scoping (both active targets):

assessment ───────────────────────────────────┐
key_messages_parquet ─────────────────────────┤
refs_parquet ──────────────────────────────────┤→ llm_candidate_scope_parquet ← Target 2h4a
works_parquet ─────────────────────────────────┤
snowball_parquet ──────────────────────────────┤
nli_ready_evidence_parquet ────────────────────┘

nli_active ──────────────────────────────────┐
assessment ──────────────────────────────────┤
nli_ready_evidence_parquet ──────────────────┤
llm_verification_active ─────────────────────┤→ llm_verification_parquet ← Target 2h4b
llm_verification_config ─────────────────────┤
llm_verification_system_prompt_file ─────────┤
llm_verification_user_prompt_file ───────────┤
llm_candidate_scope_parquet ──────────────────┤
nli_scores_by_claim_evidence (DAG dep only) ──┘

config_file ── llm_verification_active ── llm_verification_config   (mirrors nli_active → nli_config)

llm_candidate_scope_parquet reads only already-existing, unmodified targets (key_messages_parquet, refs_parquet, works_parquet, snowball_parquet, plus nli_ready_evidence_parquet for a drift sanity check) and makes no network/API calls of its own — adding it does not invalidate Phase 1’s NLI chain or any download/snowball step upstream of it. It always runs regardless of which llm_verification config is active; subset: "all" configs simply never read its output.

Configuration (input/config.yaml)

sparql_url: fuseki

assessments:
  - id: GA1
    ttl_url: https://...
  - id: IAS
    ttl_url: https://...

nli:
  active: deberta_zeroshot   # which entry under configs: is active

  configs:
    deberta_zeroshot:
      scheme: https
      host: <pod-id>-8080.proxy.runpod.net
      port: null
      model: MoritzLaurer/deberta-v3-large-zeroshot-v2.0
      candidate_labels: ["supports", "refutes", "is not relevant to"]
      hypothesis_template: "This paper {} the following claim: %s"
      batch_size: 128
      uncertain_threshold: 0.60
      nei_threshold: 0.90

Config is split into fine-grained targets — each reads only its own section from config_file directly (no intermediate config target). Changing one section re-runs only that target; if the extracted value is unchanged, nothing downstream cascades.

Config key Target Invalidates
sparql_url sparql_url All SPARQL build targets
assessments assessments_list → assessment Assessment branches and their outputs
nli.active nli_active → nli_config nli_scores_parquet only
nli.configs.<active>.* nli_config nli_scores_parquet only
llm_verification.active llm_verification_active → llm_verification_config llm_verification_parquet only
llm_verification.configs.<active>.* (incl. subset) llm_verification_config llm_verification_parquet only

Target 0: assessment — Assessment Specs

Source: _targets.R

Creates a named list of assessment specifications from assessments_list. Each element gains an index field used for deterministic Fuseki port assignment. Branch names are the assessment IDs — adding a new assessment creates a new branch without invalidating existing ones.

Target 1: ttl_path — Download TTL File

Source: R/download_ttls.R → download_ttl(assessment)

For each assessment branch: 1. Creates output/LoD/ if it does not exist. 2. Queries the GitHub Contents API for the current SHA of the remote TTL file. 3. Skips the download if a local copy with a matching SHA already exists. 4. Otherwise downloads to output/LoD/<id>.ttl and writes the SHA next to it.

Returns a character vector of local TTL file paths (format = "file").

Even when sparql_url is a remote URL, TTL files are still downloaded as a declared pipeline dependency. The download is cheap once cached.

Target 2a: refs_parquet — DB1

Source: R/write_refs_parquet.R → build_refs_parquet(sparql_url, assessment, ttl_path, refs_sparql, "output/refs")

Builds the refs dataset for one assessment branch and writes to output/refs/assessment=<id>/. When sparql_url: fuseki, starts a local Fuseki session, POSTs the TTL into the named graph, runs the SPARQL query, and stops Fuseki on exit.

Reading DB1

arrow::open_dataset("output/refs") |>
  dplyr::filter(assessment == "GA1", km == "A.", bm == "A1") |>
  dplyr::collect()

Schema — DB1

Column Type Description
assessment chr Assessment ID (partition)
km chr Key Message identifier
bm chr Background Message identifier
sm chr Sub-Message identifier
doi chr DOI string — NA if absent
description chr Citation text from the LOD
citation chr Zotero-key citation string like [P62TQUG2]
zotero_group chr Zotero group id
zotero_key chr Raw Zotero item key
zotero chr Zotero URL (owl:sameAs)

Target 2b: sections_parquet — DB2

Source: R/write_sections_parquet.R → build_sections_parquet(sparql_url, assessment, ttl_path, sections_sparql, "output/sections")

Builds the sections dataset for one assessment branch. Each SubChapter source row carries the SubMessage that referenced it (sm column). Written to output/sections/assessment=<id>/.

Reading DB2

arrow::open_dataset("output/sections") |>
  dplyr::filter(assessment == "GA1", km == "A.", bm == "A1") |>
  dplyr::collect()

Schema — DB2

Column Type Description
assessment chr Assessment ID (partition)
km chr Key Message identifier
bm chr Background Message identifier
sm chr Sub-Message identifier
section chr Chapter identifier
subsection chr SubChapter identifier
content chr SubChapter description text

Target 2b2: key_messages_parquet — DB3

Source: R/write_key_messages_parquet.R → build_key_messages_parquet(sparql_url, assessment, ttl_path, key_messages_sparql, "output/key_messages")

Builds the key/background/sub-messages dataset for one assessment branch. Written to output/key_messages/assessment=<id>/.

Reading DB3

arrow::open_dataset("output/key_messages") |>
  dplyr::filter(assessment == "GA1", km == "A.", bm == "A1") |>
  dplyr::collect()

Schema — DB3

Column Type Description
assessment chr Assessment ID (partition)
km chr Key Message identifier
km_label chr KM headline text
km_description chr KM detail text
bm chr Background Message identifier
bm_label chr BM headline text
bm_description chr BM detail text — used as NLI hypothesis
bm_well_established chr BM confidence flag
bm_established_incomplete chr BM confidence flag
sm_id chr Sub-Message identifier
sm_description chr SM statement text
sm_well_established chr SM confidence flag
sm_established_incomplete chr SM confidence flag

Target 2c: zotero_parquet — Zotero Group Items

Source: R/download_zotero.R → download_zotero(assessment, refs_parquet)

Reads the refs branch for one assessment, infers the Zotero group id from the zotero column, downloads all top-level Zotero items page by page, and writes to output/zotero/assessment=<id>/ partitioned by group_id and page.

Target 2d: works_parquet — OpenAlex Works

Source: R/download_works.R → download_works(assessment, zotero_parquet, refs_parquet, workers = 8)

Reads DOIs from the Zotero branch, fetches OpenAlex works with openalexPro::pro_fetch(), then joins with refs_parquet on normalised DOI to attach km and bm columns. Written to output/works/assessment=<id>/ partitioned by assessment, km, and bm.

Reading DB4

arrow::open_dataset("output/works") |>
  dplyr::filter(assessment == "GA1", km == "A.", bm == "A1") |>
  dplyr::select(id, doi, title) |>
  dplyr::collect()

Schema — DB4 (selected columns)

Column Type Description
assessment chr Assessment ID (partition)
km chr Key Message identifier (partition)
bm chr Background Message identifier (partition)
id chr OpenAlex work ID URL
doi chr DOI string
title chr Work title
publication_year int Year of publication
… … 51 OpenAlex columns total

Target 2f: works_citing_parquet — Citing Works

Source: R/build_works_citing_parquet.R → build_works_citing_parquet(assessment, snowball_parquet, "output/works_citing")

Copies snowball nodes with relation == "citing" into output/works_citing/assessment=<id>/km=<km>/bm=<bm>/. File-copy approach is required because OpenAlex’s nested struct columns can have inconsistent schemas across km/bm partitions.

Reading

arrow::open_dataset("output/works_citing") |>
  dplyr::filter(assessment == "GA1", km == "A.", bm == "A1") |>
  dplyr::select(id, doi, title, publication_year) |>
  dplyr::collect()

Schema

Same columns as snowball nodes minus relation (always citing in this dataset).

Target 2g: nli_scores_parquet — NLI Alignment Scores

Source: R/build_nli_scores_parquet.R → build_nli_scores_parquet(assessment, key_messages_parquet, works_citing_parquet, nli_config, nli_active, "output/nli_scores")

For each (assessment, km, bm) partition in works_citing_parquet:

  1. Reads the BM description from key_messages_parquet as the NLI hypothesis.
  2. Cleans each citing work’s title + abstract as the premise.
  3. POSTs batches to the zero-shot NLI server (see external/runpod/docker/nli-runpod/).
  4. Stores the full probability distribution (p_supports, p_refutes, p_nei) plus predicted label and confidence.

Output is written to output/nli_scores/nli_config=<cfg>/assessment=<id>/km=<km>/bm=<bm>/, partitioned by nli_config so scores from different model profiles are stored separately and comparable.

See TD_BM_NLI_approach.qmd for full design rationale.

Reading

arrow::open_dataset("output/nli_scores") |>
  dplyr::filter(nli_config == "deberta_zeroshot", assessment == "GA1", bm == "A1") |>
  dplyr::collect()

Schema

Column Type Description
nli_config chr Active NLI profile name (partition)
assessment chr Assessment ID (partition)
km chr Key Message identifier (partition)
bm chr Background Message identifier (partition)
work_id chr OpenAlex work ID URL
doi chr DOI string
title chr Paper title
publication_year int Publication year
label chr Predicted label: SUPPORTS, REFUTES, or NOT_ENOUGH_INFO
p_supports dbl Model probability for SUPPORTS
p_refutes dbl Model probability for REFUTES
p_nei dbl Model probability for NOT_ENOUGH_INFO
confidence dbl max(p_supports, p_refutes, p_nei)
uncertain lgl TRUE if confidence < uncertain_threshold

Target 2h4a: llm_candidate_scope_parquet — per-claim candidate scoping

Source: R/build_llm_candidate_scope_parquet.R

Chains refs_parquet’s sm (sub-chapter id) → seed doi → seed OpenAlex work id → citing work (via the existing snowball edges) to produce, per evidence-segmented claim, an allow-list of citing works tied to that claim’s own evidentiary sub-chapter rather than the whole BM’s — the input llm_verification_parquet’s subset: "sm" configs narrow against. Reads only already-existing, unmodified targets (key_messages_parquet, refs_parquet, works_parquet, snowball_parquet, and nli_ready_evidence_parquet for a drift sanity check) and makes no network/API calls of its own. Deliberately does not call segment_bm_by_evidence() (R/build_nli_ready_evidence_parquet.R) — even a behavior-preserving edit there would mark nli_ready_evidence_parquet, and the NLI scoring chain downstream of it, outdated, since targets hashes function bodies as dependencies. Instead duplicates the small amount of segmentation logic needed, verified byte-for-byte identical in behavior against the original. See TD_NLI_LLM_two_phase.qmd for the full design, including the fallback and prefix-matching policy decisions. Writes to output/llm_candidate_scope/assessment=<id>/km=<km>/bm=<bm>/ (assessment, km, bm, claim_id, work_id — one row per allowed pair). Always computed regardless of which llm_verification config is active.

Target 2h4b: llm_verification_parquet — Phase 2 LLM verification

Source: R/build_llm_verification_parquet.R

One ellmer::chat_openrouter() call per (claim, work) pair selected by the active config’s nli_labels/nli_certainty fields (see TD_NLI_LLM_two_phase.qmd for the full design and its ported-architecture rationale) — select_llm_verification_candidates() filters on label %in% nli_labels AND the certainty band, both optional (NULL/absent = no filter on that dimension); currently every shipped config sets nli_labels: [REFUTES], nli_certainty: certain. Optionally narrowed further by llm_candidate_scope_parquet when the active config’s subset is "sm" (parked for now). Resumable per-pair JSON cache under output/llm_verification/raw/model=<model>/prompt=<hash>/; final output — nli_label, llm_label, llm_agrees, a required verbatim quote (demoted to NOT_ENOUGH_INFO if fabricated), explanation — written to output/llm_verification/scores/llm_config=<config>/subset=<subset>/assessment=<id>/nli_route=<route>/km=<km>/bm=<bm>/. nli_route is nli_route_label()’s hive-safe encoding of that row’s own NLI outcome (e.g. REFUTES-certain), computed per row from its actual label/uncertain values — not a single value per config call, so a config listing several labels/certainties (e.g. nli_labels: [REFUTES, SUPPORTS]) still writes one nli_route= subdirectory per distinct outcome actually present, never a merged string. llm_config/subset/assessment form the fixed prefix fully deleted and rewritten each call.

Configured via input/config.yaml’s llm_verification: block — active + named configs (model, temperature, max_tokens, max_active, max_retries, subset, nli_labels, nli_certainty per config), mirroring nli:’s structure exactly. Four configs ship by default (openrouter_cheap/openrouter_cheap_sm/openrouter_midtier/openrouter_toptier_gpt5; only openrouter_cheap_sm uses subset: "sm"), the model tiers carried over from the sibling Categorisation_Literature project’s epistemology: config, including its proven max_active: 24/max_tokens: 8000 values. This replaced an earlier truth/citing-document LLM design (prompts_truth_parquet → prompts_citing_parquet → alignement_scores_parquet, driven by an analysis: config block) — all of that source has been removed; see TD_LLM_approach.qmd for why.

Target 2h4c: {refutes,supports}_funnel_{data,figures,tables} — label funnels

Source: R/build_label_funnel_data.R, R/build_label_funnel_figures.R, R/build_label_funnel_tables.R

A 3-level sieve of distinct citing works per (km, bm), each level a subset of the previous: (1) the snowball corpus (works_citing_parquet), (2) NLI <label> + certain (nli_scores_evidence), (3) of those, llm_agrees == TRUE in llm_verification_parquet (re-filtered to nli_route == paste0(target_label, "-certain") defensively, in case a future config routes other outcomes into the same output tree). A 4th level, sufficient_evidence == TRUE, was considered for the original REFUTES-only version and dropped: build_llm_verification_parquet.R’s own parser forces llm_label to NOT_ENOUGH_INFO whenever sufficient_evidence is FALSE, so llm_agrees == TRUE already implies sufficient_evidence == TRUE for every row — a 4th level would always equal the 3rd, never narrowing anything further.

build_label_funnel_data(assessment, target_label, ..., granularity) is one shared implementation called twice from _targets.R — once with target_label = "REFUTES" (target refutes_funnel_data), once with "SUPPORTS" (target supports_funnel_data) — rather than two near-identical copies. Each branches over pattern = cross(map(assessment, works_citing_parquet, llm_verification_parquet), nli_granularities) — a combinatorial product against a fixed nli_granularities <- c("sub_bm", "complete_bm") (not read from config; both are always rendered, one falling back to the empty-data state until a complete_bm NLI config is actually run), NOT a plain map(), since nli_granularities is an independent dimension from assessment, not zipped 1:1 with it — 4 branches per target (2 assessments x 2 granularities), not 2. Downstream refutes_funnel_figures/refutes_funnel_tables/report_refutes_funnel_html only need a plain pattern = map(refutes_funnel_data, ...) (and similarly for supports_*) since they consume already-4-branched siblings from the same cross(), all in the same order — no re-crossing needed downstream. report_refutes_funnel_html/report_supports_funnel_html deliberately do NOT include assessment in their own pattern (it only has 2 branches, which would mismatch the 4-branched funnel data/figures/tables) — instead they read assessment/granularity back out of the funnel data rds itself (both stored there), which stays correctly aligned with whichever branch is executing.

Each branch depends only on works_citing_parquet, nli_active (to reconstruct the nli_scores_evidence path, same convention as nli_overview_data’s call site, with a granularity=<value>/ segment inserted using the branch’s OWN nli_granularities value, not the single active-config one), and llm_verification_parquet — pure local arrow/dplyr, no network/GPU calls, and no edge into nli_scores_by_claim_evidence’s actual computation (only as an already-built dependency elsewhere in the pipeline). Reuses build_nli_overview_data.R’s doi-lookup join verbatim, since neither source table stores doi. deployment = "main" for the same OOM reason as nli_overview_data (a full per-assessment nli_scores_evidence collect()). nli_overview_data/nli_overview_figures/nli_bm_explorer_html (not otherwise covered by this stale document — see the note at the top) went through the identical cross() treatment for the same reason.

build_label_funnel_figures() renders three PNGs per assessment (overall funnel, per-BM grouped bars, and a normalized per-BM variant where each level is divided by that BM’s own corpus count so it reads 1 — lets BMs be compared on conversion rate regardless of raw corpus size). build_label_funnel_tables() renders one DT::datatable() table (level 3) with a BM dropdown filter — see output/README.md’s Details section for the exact column-curation rationale (bypasses IPBES.R::table_dt(), since its wrapper can’t pass through DT’s top-level filter argument).

Rendered into a standalone report parameterized by both assessment and label, IPBES_Label_Funnel_Report.qmd → report_refutes_funnel_html/ report_supports_funnel_html, via Quarto’s params:/execute_params= mechanism (this project’s first parameterized render — every other multi-output render, td_doc_html, branches over distinct source files instead of one file rendered N times). Target names stay label-specific (report_refutes_funnel_html, not one generic target) since each is its own pattern = map(assessment, ...) branch set; the qmd itself resolves which label’s data to read at render time via targets::tar_read_raw() (the string-based variant of tar_read()) keyed off params$label. Linked from IPBES_Fact_Checker.qmd’s final section and folded into report_output_dir’s deployable copy alongside td_doc_html.

SUPPORTS was added to every llm_verification config’s nli_labels (input/config.yaml) at the same time the SUPPORTS funnel report was added, so Phase 2 now reviews both labels’ certain pairs — see TD_NLI_LLM_two_phase.qmd’s cost table for the measured pair counts.

SPARQL Queries

All three SPARQL queries live in queries/*.sparql and are tracked as format = "file" targets. Changing a query file invalidates only the downstream parquet target.

File Target Traversal
queries/refs.sparql refs_parquet KM → BM → SM → SubChapter ← Reference(doi)
queries/sections.sparql sections_parquet KM → BM → SM → SubChapter(content) → Chapter (projects sm_id)
queries/key_messages.sparql key_messages_parquet KM → BM → SM (text + confidence flags)

All three queries wrap their pattern in GRAPH <%GRAPH_IRI%> { ... }. The placeholder is substituted at query time by read_sparql_query() using assessment_graph_iri() from R/branch_helpers.R — currently http://ontology.ipbes.net/report/<id>. Switching backends is a one-line change in input/config.yaml (sparql_url:).

Orphaned / Deprecated Source Files

Not wired into the current _targets.R; kept for reference:

  • R/build_fulltext.R — legacy Grobid XML / PDF download for citing works
  • R/resolve_citations.R — replace (Author, Year) with OpenAlex W-IDs in section content

Notes

  • All parquet targets are assessment-branched; there is no cached combined object.
  • Each assessment branch rewrites only its own partition directory.
  • Credentials (API_openalex, API_openrouter) are read from the macOS keyring by _targets.R at session start via keyring::key_get().

Required R Packages

Package Role
targets Pipeline engine
yaml Config parsing
keyring API credential retrieval
processx Fuseki subprocess management
httr2 HTTP SPARQL queries, Graph Store Protocol POSTs, NLI server requests
readr CSV response parsing
arrow Parquet I/O
dplyr Data manipulation
tictoc Query timing
jsonlite JSON serialisation for prompts
xml2, stringr Text cleaning (clean_text.R)
ellmer OpenRouter chat client (parked LLM chain)
openalexPro OpenAlex works download
openalexSnowball Snowball search
future, future.apply Parallel workers

System dependency: fuseki-server on PATH (install with brew install fuseki). Only required when sparql_url: fuseki.