---
title: "How the matching thresholds were calibrated"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{How the matching thresholds were calibrated}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

The package asserts a retraction from an exact identifier (DOI or PMID) with high
confidence. For a reference that carries *no* identifier it falls back to title
matching, and two thresholds govern that fallback:

- the **fuzzy threshold** (`getOption("retraction.fuzzy_threshold")`, default
  **0.90**): a title similarity below this is not even surfaced as a possible
  match;
- the **`title_exact` gate** (similarity **0.985** plus an exact publication year
  and a matching first author): only a match clearing this gate is *asserted* as
  retracted; everything else is reported as "possible" for the user to verify.

This article reports a calibration of those two numbers against a labeled corpus,
so they are evidence-based rather than guessed. The labeled corpus ships with the
package in `inst/extdata/calibration_corpus.csv`, and the two scripts that build
and analyze it (`calibration_corpus.R`, `calibration_analysis.R`) live in the
`data-raw/` directory of the
[source repository](https://github.com/choxos/retraction), so the study is
reproducible from a repository checkout.

## The labeled corpus

599 references in three groups:

- **200 exact-title retracted** — records sampled from the Retraction Watch
  corpus, cited with their exact title (as a well-formatted bibliography would).
- **200 perturbed retracted** — the same records with a realistic citation
  variation (about 15% of title words dropped, lower-cased), to probe how the
  thresholds behave on imperfect titles.
- **199 clean** — non-retracted articles sampled from OpenAlex.

Every reference is matched by **title, year, and author only** (the DOI is
withheld), which is exactly the hard case the thresholds govern.

## Result 1: the assertion gate never false-accuses

At the `title_exact` gate (0.985 + year + first author):

| Metric | Value |
|---|---:|
| Precision | **1.000** |
| Recall | 0.532 |
| Clean references false-flagged | **0 / 199 (0.000)** |

Flag rate by group: exact-title retracted **1.000**, perturbed retracted 0.065,
clean **0.000**.

Reading: **no clean reference was ever asserted as retracted** (zero false
accusations), and every exact-title retracted reference was recovered. The
perturbed titles almost never clear the gate (0.065) — by design they fall to
"possible" rather than being asserted, which is the conservative behavior we
want. The overall recall of 0.532 is dominated by the perturbed group; on
citations that reproduce the title faithfully, recall is 1.0.

## Result 2: 0.90 is the empirical sweet spot for surfacing

Sweeping the fuzzy threshold and measuring how often a retracted reference is
*surfaced* (as flagged or possible) versus how often a clean reference is
wrongly surfaced:

| Threshold | Retracted surfaced | Clean surfaced | Precision |
|---:|---:|---:|---:|
| 0.84 | 0.980 | 0.774 | 0.718 |
| 0.86 | 0.958 | 0.437 | 0.815 |
| 0.88 | 0.945 | 0.035 | 0.982 |
| **0.90** | **0.885** | **0.000** | **1.000** |
| 0.92 | 0.790 | 0.000 | 1.000 |
| 0.94 | 0.688 | 0.000 | 1.000 |

**0.90 is the lowest threshold at which no clean reference is surfaced** while
still recovering 88.5% of retracted references. Below 0.88 the clean-surfacing
rate climbs steeply (43.7% at 0.86), which would bury real hits in false
positives. Above 0.90 precision stays perfect but recall falls with no benefit.

## Conclusion

The defaults are validated by this corpus:

- **0.985 `title_exact` gate:** perfect precision, zero false accusations — the
  right posture for a tool that could otherwise mislabel a clean paper.
- **0.90 fuzzy threshold:** the operating point where clean false-positives reach
  zero while retaining high recall.

These are heuristics, not calibrated probabilities, and the corpus is modest
(599 references); the numbers should be revisited as the corpus grows and across
non-English titles. But they show the current thresholds are conservative in the
direction that matters: the package prefers "possible, please verify" over a
false assertion.

## Reproducing

The two scripts below are in the `data-raw/` directory of the
[source repository](https://github.com/choxos/retraction) (they are not part of
the installed package). Run them from a repository checkout:

```{r}
retraction_sync()                       # local corpus
source("data-raw/calibration_corpus.R")   # rebuild the labeled set (online)
source("data-raw/calibration_analysis.R") # recompute the tables above
```