Package {joinery}

joinery: Heuristic Index-Based Record Linkage

Description

Links records that refer to the same entity across sources that share no common key, such as people, firms, or addresses with spelling variation, abbreviations, or reordered words. Linkage is described declaratively as a strategy that normalises, tokenises, phonetically encodes, weights, and blocks each field; candidate pairs are then scored by the rarity-weighted overlap of their tokens and every score is attributed back to individual tokens for explainability. Strategies compose into staged pipelines of exact, fuzzy, and optional embedding-based matching that carry unmatched records forward and resolve entities as connected components. The same strategy runs on an in-memory 'data.table' backend or an out-of-core 'DuckDB' backend, and diagnostic and calibration tools help tune a strategy and filter false positives. The token-retrieval heuristic follows Doherr (2023) doi:10.2139/ssrn.4326848.

Package options

joinery reads a small number of global options. Set them with options(); all have working defaults, so you only touch them to change behaviour.

Embedding strategies embed each record once and reuse the vector on later calls, so a multi-stage run does not pay the (expensive) embedding cost again. Two options control that reuse:

joinery.embedding_reuse

Logical, default TRUE. When TRUE, the data.table and tibble backends keep a per-session cache of embedding vectors keyed by model and record content, and reuse a vector whenever the same text is embedded again. Set to FALSE to embed fresh every time, for example when benchmarking or using a non-deterministic model. (The DuckDB backend reuses through its own persisted embeddings column and ignores this option.) The session cache grows as you embed more distinct records and is only released at the end of the session or by clear_embedding_cache(); call that to reclaim memory in a long-running session.

joinery.embedding_cache_dir

Character path, default unset (NULL). When set, the embedding cache also writes each vector to this directory, so reuse survives across R sessions. When unset, the cache lives only in the current session. The cache is keyed by record content, so a changed record re-embeds on its own; clear stale files with clear_embedding_cache() (disk = TRUE).

Author(s)

Maintainer: Eduard Brüll eduard.bruell@zew.de

See Also

clear_embedding_cache(), embedding_strategy()

Bar chart of candidates-per-record distribution (candidates only)

Description

Bar chart of candidates-per-record distribution (candidates only)

Usage

ambiguity_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (ambiguity_dist).

Apply a fitted filter to match features

Description

Score a Match_Features table with a fitted Filter_Model and return a Calibrated_Matches object. When matches is supplied, the original match table is enriched with tp_prob and predicted_tp columns and stored in the result's ⁠@matches⁠ slot; when matches is NULL, the features table itself is enriched and stored.

Usage

apply_filter(
  features,
  filter_model,
  threshold = NULL,
  threshold_rule = c("youden", "target_recall", "cost_weighted"),
  target_recall = 0.95,
  cost_ratio = 1,
  matches = NULL,
  ...
)

Arguments

Value

A Calibrated_Matches object.

Examples

strat <- search_strategy(
  workshop   ~ normalize_text() + word_tokens(min_nchar = 3),
  proprietor ~ normalize_text() + word_tokens(min_nchar = 2),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.30
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
feats <- match_features(matches, strat,
                        base = workshop_listings, id = "listing_id",
                        target = workshop_register, target_id = "reg_no")
model <- fit_filter(feats, match_labels_example)
# Broadcast the true-positive probability back onto the match rows.
apply_filter(feats, model, matches = matches)

Approximate dates by rounding to coarser time units

Description

approximate_date() rounds dates to the start of broader time periods (month, quarter, half-year, year, or decade). This is useful for fuzzy temporal matching when exact dates may differ slightly but represent the same general time period.

Usage

approximate_date(
  x,
  unit = c("month", "quarter", "half", "year", "decade"),
  format = NULL,
  orders = c("ymd", "dmy", "mdy")
)

Arguments

Details

Rounding always goes to the start of the period:

• "month": 2023-03-15 -> 2023-03-01

• "quarter": 2023-03-15 -> 2023-01-01 (Q1), 2023-05-20 -> 2023-04-01 (Q2)

• "half": 2023-03-15 -> 2023-01-01 (H1), 2023-08-20 -> 2023-07-01 (H2)

• "year": 2023-03-15 -> 2023-01-01

• "decade": 2023-03-15 -> 2020-01-01

Value

A character vector of dates in ISO 8601 format (YYYY-MM-DD), rounded to the start of the specified time unit. Unparseable dates return NA_character_ with a warning.

See Also

normalize_date() for exact dates, date_tokens() to split a date into part tokens.

Other date preparers: date_tokens(), normalize_date()

Examples

approximate_date("2023-03-15", unit = "month")
# "2023-03-01"

approximate_date("2023-03-15", unit = "quarter")
# "2023-01-01"

approximate_date("2023-08-20", unit = "half")
# "2023-07-01"

approximate_date("2023-03-15", unit = "year")
# "2023-01-01"

approximate_date("2023-03-15", unit = "decade")
# "2020-01-01"

approximate_date(c("2023-01-15", "2023-04-20", "2023-09-10"), unit = "quarter")
# c("2023-01-01", "2023-04-01", "2023-07-01")

Encode text phonetically with the Cologne procedure

Description

The Cologne phonetic procedure (Koelner Phonetik) is the German-language counterpart to Soundex. It maps text to a digit string by German pronunciation rules, so variants like "Meier", "Maier", and "Mayer" share one key. Reach for this over as_soundex() or as_metaphone() when the data is German.

Usage

as_cologne(text)

Arguments

Details

Returns text, so it slots ahead of a token generator, or use it directly on a one-word column. Like any phonetic key it favours recall over precision; pair it with a sharper field rather than matching on the key alone.

Value

A character vector of Cologne phonetic keys (digit strings), one per input element.

See Also

Other phonetic encoders: as_metaphone(), as_soundex()

Examples

as_cologne(c("Meier", "Maier", "Mayer"))  # same key

Encode text phonetically with Metaphone

Description

Names that sound alike are often spelled differently: "Smith" and "Smyth", "Meyer" and "Maier". Metaphone encodes text by how it sounds, so those variants share one key and match even though the letters differ. Best on single-word fields such as surnames or company names; it is tuned for English pronunciation (for German, see as_cologne()).

Usage

as_metaphone(text)

Arguments

Details

Runs on either side of a token generator: ahead of one (on a text column), or after one (on a token column, encoding each token in place). Phonetic keys are deliberately coarse, so they trade precision for recall: pair them with a sharper field rather than matching on a phonetic key alone.

Value

A character vector of Metaphone keys, one per input element.

See Also

Other phonetic encoders: as_cologne(), as_soundex()

Examples

as_metaphone("Smith")
as_metaphone(c("Meyer", "Maier"))  # same key

Encode text phonetically with Soundex

Description

Soundex is the classic phonetic code: it keeps the first letter and reduces the rest to a short digit string (for example "Robert" and "Rupert" both become "R163"), so spellings that sound alike share one key. It is coarser and older than Metaphone but widely understood and a good default for English surnames.

Usage

as_soundex(text)

Arguments

Details

Runs on either side of a token generator: ahead of one (on a text column), or after one (on a token column, encoding each token in place). As with any phonetic key it favours recall over precision; pair it with a sharper field rather than matching on the key alone.

Value

A character vector of Soundex keys (letter followed by digits), one per input element.

See Also

Other phonetic encoders: as_cologne(), as_metaphone()

Examples

as_soundex("Robert")
as_soundex(c("Robert", "Rupert"))  # same key

Audit a Search Strategy Against Data

Description

Pre-match diagnostic (Q1). Runs preparation and rarity computation, reports per-column token / rarity statistics and (when block_by is set) block-size distribution and estimated comparison count. Surfaces recommendations linking pre-match symptoms to strategy levers.

Usage

audit_strategy(data, id, strategy, ...)

Arguments

Value

A Strategy_Audit object.

Examples

strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.7
)
audit_strategy(workshop_register, "reg_no", strat)

Base dataset for record linkage example

Description

A dataset containing 3,300 person records with German, Turkish, and Polish names, including addresses across various German cities. Approximately 5% of records are intentional duplicates with small variations to simulate real-world data quality issues.

Usage

base_example

Format

A tibble with 3,300 rows and 7 variables:

id_base

Character identifier for base records (B0001-B3150)

Vorname

First name, weighted by ethnic group prevalence

Nachname

Last name, weighted by ethnic group prevalence

Strasse

Street name, including German street types

Hausnummer

House number, some with letter suffixes

Ort

City or town name

Kreis

Administrative district (Kreis)

Source

Synthetically generated using weighted sampling from common German, Turkish, and Polish names (93%, 4%, and 3% respectively) and realistic German geography.

Apply a function to DuckDB table batches

Description

Streams a DuckDB table through a batch plan and applies a user-defined function to each batch. The function must accept a data.frame and return a data.frame. Results can be collected in memory or written back to DuckDB incrementally.

Usage

batch_map(plan, con, input_table, fn, persist = TRUE, output_table = NULL)

Arguments

Details

Database work is performed batch-by-batch, allowing preprocessing of tables that exceed available RAM. For each batch, a SQL slice or block filter is executed, the function is applied, and (optionally) results are appended to a DuckDB table.

Value

• If persist = TRUE: A tbl_duckdb_connection pointing to the output table.

• If persist = FALSE: A list of data.frames, one per batch.

Block on a Column's Rare Tokens (region-free blocking)

Description

Build a token-blocking key for use inside a strategy's block_by. Where a plain column name blocks two records only when they share a literal value, block_on_tokens() blocks them when they share any of a designated column's (rare) tokens. This is region-free: a record that drifts across a region boundary - a firm that moves to a new postcode, say - still co-blocks with its earlier self through a distinctive name token, and so becomes a candidate where a literal block would never compare them.

Hand it to block_by in place of (or mixed with) a column name:

# fully region-free - share a rare name token, regardless of place
search_strategy(name ~ normalize_text + word_tokens(min_nchar = 3),
                block_by = block_on_tokens("name", max_df = 50))

# region-bounded - share a rare name token AND sit in the same plz2
search_strategy(name ~ normalize_text + word_tokens(min_nchar = 3),
                block_by = list(block_on_tokens("name", max_df = 50), "plz2"))

max_df and min_rarity select which tokens are eligible block keys, using the global (corpus-wide) document frequency: a token appearing in more than max_df records, or whose global rarity falls below min_rarity, is dropped as a key. This is where "block on the distinctive words, not the common ones" lives - a franchise name ("ALDI") is globally common, fails the cap, and never becomes a block key, while a distinctive brand survives. A record with no surviving block key is unreachable via token-blocking in this stage (it contributes no token-block rows).

Token-blocking is the densest operation in the package: every pair sharing a surviving key is materialised. It is safe only behind a real max_df (or min_rarity) plus the always-on fan-out guard. Passing neither cap is a loud warning, not an error, but you almost always want one.

Usage

block_on_tokens(
  column,
  max_df = Inf,
  min_rarity = 0,
  preparer = NULL,
  min_nchar = 3L
)

Arguments

Value

A Block_On_Tokens spec, to be placed in block_by.

See Also

search_strategy()

Examples

# Block on a rare word from the workshop name instead of a region, so a
# workshop still co-blocks with its relocated self. The max_df cap keeps
# common words ("joinery") from becoming block keys.
strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by     = list(block_on_tokens("workshop", max_df = 50, min_nchar = 4),
                      "trade"),
  rarity_scope = "global",
  threshold    = 0.6
)
strat

Bar chart of block sizes (requires block_by on strategy)

Description

Bar chart of block sizes (requires block_by on strategy)

Usage

block_size_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (block_summary$distribution).

Evaluate a fitted filter on labelled pairs

Description

Compute calibration diagnostics for a fitted false-positive filter on a labelled evaluation set. Returns a Filter_Calibration carrying the reliability table, Brier score, log-loss, per-class confusion matrix, and a threshold sweep curve.

Two call shapes:

• calibrate(calibrated_matches, labels) - evaluate on labels held out from the training fit.

• calibrate(calibrated_matches) - evaluate on the training labels stored on the Filter_Model (sanity-check view; do not use for model selection).

Usage

calibrate(x, labels = NULL, bins = 10L, ...)

Arguments

Value

A Filter_Calibration object.

Calibrate matches end-to-end (features -> filter -> apply)

Description

High-level Q5 verb. Builds features via match_features(), fits a Filter_Model via fit_filter(), and applies it via apply_filter() to return a Calibrated_Matches object enriched with tp_prob / predicted_tp. Dispatches on the strategy class.

Usage

calibrate_matches(matches, strategy, ...)

Arguments

Value

A Calibrated_Matches object.

Examples

strat <- search_strategy(
  workshop   ~ normalize_text() + word_tokens(min_nchar = 3),
  proprietor ~ normalize_text() + word_tokens(min_nchar = 2),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.30
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
# One call: build features, fit the filter, apply it. Uses the shipped
# labelled pairs, which line up with this exact search.
calibrate_matches(matches, strat, labels = match_labels_example,
                  base = workshop_listings, id = "listing_id",
                  target = workshop_register, target_id = "reg_no")

Clear the embedding reuse cache

Description

Empties joinery's in-session embedding cache, and optionally the on-disk cache. The cache stores raw embedding vectors so that the data.table and tibble backends reuse them instead of re-embedding on every call. You rarely need to call this by hand; it is mainly useful to force a clean re-embed or to reclaim memory in a long-running session.

Usage

clear_embedding_cache(disk = FALSE)

Arguments

Value

Invisibly NULL.

See Also

compute_embeddings() for how the cache is filled, and joinery (package options) for joinery.embedding_reuse and joinery.embedding_cache_dir.

Bar chart of cluster-size distribution (duplicates only)

Description

Bar chart of cluster-size distribution (duplicates only)

Usage

cluster_size_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (cluster_dist).

Compare Stages of a Multi-Stage Match

Description

Multi-stage diagnostic. Produces per-stage Match_Overview objects, marginal coverage per stage, and overlaid per-stage score distributions. Note that summarise_matches() does not auto-detect a stage column - users explicitly call this verb when they want per-stage analysis (see notes/diagnostics_design.md).

Usage

compare_stages(matches, ...)

Arguments

Value

A Stage_Comparison object.

Examples

exact <- exact_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = c("postcode_area", "trade")
)
fuzzy <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.55
)
g <- multi_stage_search(
  workshop_panel, workshop_panel,
  base_id = "record_id", target_id = "record_id",
  list(exact = exact, fuzzy = fuzzy),
  self = TRUE, source_by = "year", collapse = "rep"
)
# See how much each pass added that earlier passes had not reached.
compare_stages(g, base = workshop_panel, target = workshop_panel)

Compute Embeddings for Records

Description

Compute embedding vectors for records using an Embedding_Strategy. This is a backend-specific generic that handles data retrieval, text assembly, and embedding computation via tidyllm.

Embedding is the expensive part of a vector match, so each record is embedded once and the vector is reused on later calls. The data.table and tibble backends keep a per-session cache keyed by model and record content; the DuckDB backend reuses through its persisted embeddings column. Reuse is controlled by the joinery.embedding_reuse and joinery.embedding_cache_dir options (see joinery package options) and can be cleared with clear_embedding_cache().

Usage

compute_embeddings(data, id, strategy, ...)

Arguments

Value

A backend-specific table with columns: id and embedding (where embedding contains numeric vectors).

See Also

clear_embedding_cache()

Compute Token Rarity for Record Linkage

Description

compute_rarity() assigns a rarity score to each token produced by prepare_search_data(), using the rarity method defined in a Search_Strategy.

Usage

compute_rarity(tokens, strategy, ...)

Arguments

Details

Rarity quantifies how informative a token is when comparing records. In joinery, rarity is always computed:

• using one global rarity metric specified in the strategy,

• per column, because each field has its own token distribution,

• within each block (if the strategy specifies block_by).

The input tokens must be the long-format token table returned by prepare_search_data(), containing at minimum:

• an ID column,

• a column field indicating the source variable,

• a token field,

• a row_id identifying the originating record,

• and any block_by variables required by the strategy.

Backends (e.g., data.frame, data.table, DuckDB relations) may implement their own methods for this generic, but all must return the same logical structure: the original token table with an added numeric rarity column.

Value

The same token table with an added rarity column.

Horizontal bar chart of per-column score contributions

Description

Horizontal bar chart of per-column score contributions

Usage

contribution_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (per_column_contrib).

Bar chart of match coverage (base and/or target)

Description

Bar chart of match coverage (base and/or target)

Usage

coverage_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table.

Extract date components as tokens

Description

date_tokens() parses dates and extracts specified components (year, month, day) as separate tokens. This is useful for flexible date matching where you want to match on specific date parts rather than full dates.

Usage

date_tokens(
  x,
  components = c("year", "month", "day"),
  format = NULL,
  orders = c("ymd", "dmy", "mdy")
)

Arguments

Details

Components are returned as zero-padded strings:

• "year" – 4-digit year (e.g., "2023")

• "month" – 2-digit month (e.g., "01", "12")

• "day" – 2-digit day (e.g., "05", "31")

The order of tokens in the output follows the order of components.

Value

A list of character vectors, one per input element. Each vector contains the requested date components as strings. Unparseable dates return an empty character vector with a warning.

See Also

normalize_date() to match whole dates, approximate_date() to match on coarser periods.

Other date preparers: approximate_date(), normalize_date()

Examples

date_tokens("2023-12-31")
# list(c("2023", "12", "31"))

date_tokens("31.12.2023", components = c("year", "month"))
# list(c("2023", "12"))

date_tokens("12/31/2023", components = "year")
# list("2023")

date_tokens(c("2023-01-15", "15.06.2023"))
# list(c("2023", "01", "15"), c("2023", "06", "15"))

Deduplicate a Table

Description

Generic function that removes or merges duplicate records from a table based on duplicate pairs identified by detect_duplicates().

Usage

deduplicate_table(base_table, duplicates, id, ...)

Arguments

Value

A deduplicated version of base_table.

Examples

ex <- exact_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = c("postcode_area", "trade")
)
dups <- detect_duplicates(workshop_register, id = "reg_no", strategy = ex)
deduped <- deduplicate_table(workshop_register, dups, id = "reg_no")
c(before = nrow(workshop_register), after = nrow(deduped))

Detect Duplicate Records

Description

Find likely duplicate records inside a single table and group them. Records are compared by how much of their rare, informative token content they share (not by character-level edit distance), every pair is scored, and any pair scoring at or above the threshold is linked. Records that link directly or transitively form one duplicate group.

Pass a search_strategy() for fuzzy, scored matching, or an exact_strategy() to group only records whose token sets are identical.

Usage

detect_duplicates(base_table, id, strategy, ...)

Arguments

Value

A table with one row per record that belongs to a duplicate group:

duplicate_group

Group label shared by all records that are duplicates of one another.

id

The record ID.

score

The record's match score within its group.

rank

Rank within the group; rank 1 is the representative kept by deduplicate_table().

⁠<original columns>⁠

Every other column from base_table.

See Also

deduplicate_table() to collapse the groups, search_candidates() for the cross-table version, multi_stage_dedup() for staged passes.

Examples

data(base_example)

strat <- search_strategy(
  Nachname ~ normalize_text() + word_tokens(min_nchar = 3),
  Vorname  ~ normalize_text() + word_tokens(min_nchar = 3),
  Ort      ~ normalize_text(),
  block_by = "Kreis",
  threshold = 0.8
)

dups <- detect_duplicates(base_example, id = "id_base", strategy = strat)
head(dups)

Drop all temporary DuckDB tables created by joinery

Description

The DuckDB backend writes ephemeral tables during batch preprocessing (for example the token tables built by prepare_search_data()). A clean run drops them when it finishes, but a run that is killed partway, or a machine that loses power mid-job, can leave them behind on disk. This sweeps them up.

Usage

drop_joinery_temp_tables(
  con,
  prefixes = c("_joinery_tokens_", "_joinery_tmp_", "_joinery_emb_")
)

Arguments

Details

Each temporary table carries a reserved name prefix such as "_joinery_tokens_" or "_joinery_tmp_". Only tables whose names begin with one of those prefixes are removed, so your own tables are never touched. Pass extra prefixes to cover temporary table types added in future.

Value

A character vector of removed table names, invisibly.

Examples

if (requireNamespace("duckdb", quietly = TRUE) &&
    requireNamespace("DBI", quietly = TRUE)) {
  con <- DBI::dbConnect(duckdb::duckdb(), ":memory:")
  # A stray joinery temp table left behind by an interrupted run:
  DBI::dbWriteTable(con, "_joinery_tmp_demo", data.frame(x = 1))

  drop_joinery_temp_tables(con)  # removes it, returns its name invisibly
  DBI::dbDisconnect(con, shutdown = TRUE)
}

Drop numeric (house-number) tokens from token lists

Description

Symmetric inverse of numeric_tokens(): removes pure-digit tokens (typically house numbers) from a token column. Operates on the list-of-character token vectors produced by earlier steps such as word_tokens(), mirroring filter_stopwords().

Useful in address pipelines where the street name carries the matching signal but the house number is noise (and fans out blocks): tokenize the street, then drop_numeric_tokens() to keep only the name tokens.

Usage

drop_numeric_tokens(tokens, keep_letters = TRUE)

Arguments

Value

A list of character vectors with numeric tokens removed.

See Also

numeric_tokens(), its inverse; filter_stopwords() for the same idea with a named word list.

Other token transformers: drop_short_tokens(), extract_initials(), filter_stopwords(), fuzzy_tokens(), token_shapes(), use_dictionary()

Examples

drop_numeric_tokens(list(c("MAIN", "12", "ST")))
# list(c("MAIN", "ST"))

drop_numeric_tokens(list(c("MAIN", "12A")), keep_letters = FALSE)
# list("MAIN")

Drop short tokens from token lists

Description

Removes tokens shorter than min_nchar characters from a token column. Where word_tokens()'s own min_nchar filters length at tokenisation, this filters length after a token transform - which is where it matters for the phonetic encoders (as_cologne(), as_soundex(), as_metaphone()) and generate_ngrams(): those produce short codes, and a 1-2 character code maps to a very large equivalence class (low distinctiveness), so it behaves as a false-match magnet. Chain drop_short_tokens() after the encoder to keep only the discriminative codes.

Operates on the list-of-character token vectors produced by earlier steps, mirroring filter_stopwords() / drop_numeric_tokens().

Usage

drop_short_tokens(tokens, min_nchar = 2)

Arguments

Value

A list of character vectors with short tokens removed.

See Also

filter_stopwords() and drop_numeric_tokens() for the same list-column idea with other drop rules; word_tokens() for the same length cut applied at tokenisation instead.

Other token transformers: drop_numeric_tokens(), extract_initials(), filter_stopwords(), fuzzy_tokens(), token_shapes(), use_dictionary()

Examples

drop_short_tokens(list(c("BAU", "AG", "X")))
# list(c("BAU", "AG"))  # the 1-char token is dropped at the default min_nchar = 2

# keep only Cologne codes of 4+ digits (drops the collision-prone short class)
drop_short_tokens(as_cologne(list(c("Bülau", "Mertens"))), min_nchar = 4)
# list("67268")

Create a Batch Plan for DuckDB Table Processing

Description

Analyses a DuckDB table and generates a batch plan (data.table) that defines how to split the table into atomic processing units. Each row of the plan represents one batch with row counts, optional row-number windows, and block identifiers (if blocking is used).

Usage

duckdb_batch_plan(
  db_tbl,
  id,
  target_batch_size = NULL,
  min_batch_size = NULL,
  chunk_strategy = "block_consolidated",
  block_by = NULL,
  atomic_blocks = FALSE
)

Arguments

Details

The function supports three chunking strategies:

• "even": Simple row-number chunking, ignores blocks

• "block_first": Each batch = one block (or sub-chunks if block > target_batch_size)

• "block_consolidated": Consolidates small blocks to minimize batch count (default)

Small tables: If total rows < min_batch_size, returns a single batch regardless of strategy. With blocking, still respects blocks.

Row-number windows: For unblocked or large-block sub-chunking, row_start and row_end define window boundaries (1-based, inclusive). For block-based batches (small blocks), these are NA.

Consolidation: "block_consolidated" (default) combines multiple small blocks into single batches up to target_batch_size to reduce overhead. Each batch may contain zero, one, or multiple blocks (depending on sizes and consolidation).

Row ordering: To ensure row_start and row_end windows are consecutive and can be reliably sliced from the DB, the function sorts by the id column before computing row numbers. This ensures reproducible, deterministic batch boundaries.

Value

A data.table with columns:

• batch_id: integer, sequential batch identifier (1, 2, 3, ...)

• row_count: integer, number of rows in this batch

• row_start: integer (or NA), window start for row-number-based batches; NA for block-based

• row_end: integer (or NA), window end for row-number-based batches; NA for block-based

• Additional columns (if block_by specified): one per blocking variable, containing block values

Examples

if (requireNamespace("duckdb", quietly = TRUE) &&
    requireNamespace("DBI", quietly = TRUE) &&
    requireNamespace("dplyr", quietly = TRUE)) {
  con <- DBI::dbConnect(duckdb::duckdb(), ":memory:")
  DBI::dbWriteTable(
    con, "data",
    data.frame(id = 1:1000, region = rep(LETTERS[1:5], length.out = 1000))
  )
  tbl_ref <- dplyr::tbl(con, "data")

  # Unblocked, even row-number chunking
  plan1 <- duckdb_batch_plan(
    tbl_ref, id = "id",
    target_batch_size = 200, chunk_strategy = "even"
  )

  # Blocked, consolidated strategy (default, respects regions)
  plan2 <- duckdb_batch_plan(
    tbl_ref, id = "id",
    target_batch_size = 200, block_by = "region"
  )
  DBI::dbDisconnect(con, shutdown = TRUE)
}

DuckDB Execution Control

Description

Build a Duckdb_Control bundling the DuckDB backend's execution knobs, and pass it as ⁠control =⁠ to prepare_search_data(), detect_duplicates(), or search_candidates() on DuckDB tables. It controls how a match runs (memory, batching, chunking, failure isolation), never what matches - matching semantics stay on the search_strategy().

Two execution stages, two atomicity rules:

• Preprocess batching (tokenization) is per-row, governed by target_batch_size / min_batch_size / chunk_strategy. Any row split is safe.

• Scoring chunking (the overlap join) is block-atomic - a pair only forms within a block, so a block can never be split. chunk_by packs whole blocks under target_batch_size; on_error isolates a pathological block from the rest of the run.

Chunking is a DuckDB (out-of-core) concern; the in-memory data.table backend ignores it.

Usage

duckdb_control(
  target_batch_size = NULL,
  min_batch_size = NULL,
  chunk_strategy = c("block_consolidated", "block_first", "even"),
  chunk_by = NULL,
  on_error = c("skip", "retry", "stop"),
  progress = NULL
)

Arguments

Value

A Duckdb_Control object.

See Also

prepare_search_data(), detect_duplicates(), search_candidates().

Examples

# The control object just bundles execution knobs; it carries no data.
ctrl <- duckdb_control(target_batch_size = 5e5, on_error = "skip")
ctrl


# Pass it to a verb running on a DuckDB table.
if (requireNamespace("duckdb", quietly = TRUE) &&
    requireNamespace("DBI", quietly = TRUE) &&
    requireNamespace("dplyr", quietly = TRUE)) {
  con <- DBI::dbConnect(duckdb::duckdb(), ":memory:")
  DBI::dbWriteTable(con, "reg", as.data.frame(workshop_register))
  strat <- search_strategy(
    workshop ~ normalize_text() + word_tokens(min_nchar = 3),
    block_by  = c("postcode_area", "trade"),
    threshold = 0.7
  )
  detect_duplicates(dplyr::tbl(con, "reg"), "reg_no", strat, control = ctrl)
  DBI::dbDisconnect(con, shutdown = TRUE)
}

Create an Embedding Strategy

Description

Construct an Embedding_Strategy object for semantic matching using embeddings. This is a distinct strategy type from token-based strategies created with search_strategy().

Embedding strategies:

• Represent entire records as embedding vectors

• Use cosine similarity for scoring

• Support blocking variables to restrict comparisons

• Require the tidyllm package for embedding computation

Usage

embedding_strategy(
  columns = NULL,
  embedding_model,
  threshold,
  collapse_sep = " ",
  normalize = TRUE,
  batch_size = 1000,
  block_by = NULL
)

Arguments

Value

An Embedding_Strategy S7 object.

Examples

## Not run: 
library(tidyllm)

# Create an embedding strategy using Ollama
emb_strat <- embedding_strategy(
  columns = c("name", "address"),
  embedding_model = ollama(.model = "mxbai-embed-large"),
  threshold = 0.85
)

# Use in multi-stage workflow
results <- multi_stage_search(
  base_table = customers_a,
  target_table = customers_b,
  base_id = "id_a",
  target_id = "id_b",
  strategies = list(
    token_stage = search_strategy(name ~ normalize_text() + word_tokens()),
    semantic_stage = emb_strat
  )
)

## End(Not run)

Define an Exact Matching Strategy

Description

Creates an Exact_Strategy for exact, score-1.0 token-set matching. Hand it to the same verbs you would hand a search_strategy(): detect_duplicates() to group identical records within a table, search_candidates() to match them across tables. Both return the usual result with score == 1.0.

Two records link only when every column's token set is equal within the same block. This is the same as a fuzzy score of exactly 1.0, reached without any scoring or threshold, and it is robust to empty columns: two records with identical names and both streets blank will link, where a weighted threshold would silently reject them. (A blank column drags a weighted score below its threshold, since its weight stays in the denominator; exact matching has no such ceiling.)

Use it as the cheap first stage of a staged workflow (exact first, then fuzzy on whatever is left): the leftover records come from extract_unmatched(), and multi_stage_dedup() / multi_stage_search() thread them through for you when you pass list(exact_strategy(...), search_strategy(...)).

Usage

exact_strategy(
  ...,
  block_by = NULL,
  rarity = "inverse_freq",
  containment = c("off", "forward", "bidirectional"),
  min_base_rarity = 0,
  min_containment_tokens = 1
)

Arguments

Value

An Exact_Strategy object.

See Also

search_strategy(), detect_duplicates(), search_candidates(), extract_unmatched().

Examples

# Link only workshops whose name tokens are identical within the same area
# and trade. No threshold to tune, and blank columns do not sink a match.
ex <- exact_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = c("postcode_area", "trade")
)
dups <- detect_duplicates(workshop_register, id = "reg_no", strategy = ex)
head(dups)

Explain a Single Match

Description

Attribution diagnostic (Q3). Reconstructs per-column and per-token contributions to a single match score. Dispatches on the second positional argument: a Search_Strategy triggers reconstruction from raw inputs; a tokens-shaped table is used directly.

Usage

explain_match(matches, x, ...)

Arguments

Value

A Match_Explanation object.

Examples

strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.7
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
# Break one pair's score down into its per-token contributions.
first_id <- matches$match_id[matches$source == "target"][1]
explain_match(matches, strat,
              base = workshop_listings, id = "listing_id",
              target = workshop_register, target_id = "reg_no",
              match_id = first_id)

Export a match sample to CSV for manual labelling

Description

Write a sampled set of matches to a CSV pre-filled with an equal column on block-header rows. Users edit the CSV in any spreadsheet, marking only exceptions (e.g. false positives) and leaving the rest as defaults.

Block definition follows the matches schema: for candidate matches (from search_candidates()), the header is the base-side row and candidate rows inherit its default. For duplicate matches (from detect_duplicates()), the header is the rank-1 row and the remaining records in the duplicate group inherit its default.

Usage

export_for_labelling(sample, file, default_label = 1L)

Arguments

Value

Invisibly returns file.

See Also

import_labels()

Extract initials from tokens

Description

Keeps only the first character of each token ("ANNA" becomes "A"). Use it to match on initials when full first names are recorded inconsistently, for example when one source has "Anna Berta Schmidt" and another "A. B. Schmidt".

Usage

extract_initials(tokens)

Arguments

Details

It transforms a token column, so it runs after a token generator such as word_tokens().

Value

A list of character vectors of single-character initials.

See Also

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), filter_stopwords(), fuzzy_tokens(), token_shapes(), use_dictionary()

Examples

extract_initials(list(c("Anna", "Berta")))
# list(c("A", "B"))

Extract Unmatched Records

Description

Identify and extract records from a table that were not matched in a record linkage operation.

Usage

extract_unmatched(data, id, matches, ...)

Arguments

Value

A subset of data containing only records whose IDs do not appear in matches.

Examples

strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.7
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
# The listings that found no register match, ready for a looser next pass.
leftover <- extract_unmatched(workshop_listings, "listing_id", matches)
nrow(leftover)

Filter out stopwords from token lists

Description

Some tokens carry no matching signal but appear everywhere: legal forms like GMBH or LTD, articles, generic words. Because they are common they create many spurious matches and fan out blocks. filter_stopwords() removes named tokens so matching rests on the distinctive ones. The comparison is case-insensitive.

Usage

filter_stopwords(tokens, stopwords)

Arguments

Details

It transforms a token column, so it runs after a token generator such as word_tokens().

Value

A list of character vectors with the stopwords removed.

See Also

drop_numeric_tokens() to remove house numbers the same way.

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), extract_initials(), fuzzy_tokens(), token_shapes(), use_dictionary()

Examples

filter_stopwords(list(c("MUELLER", "GMBH")), stopwords = c("gmbh"))
# list("MUELLER")

Discover candidate stopwords from a prepared token table

Description

Scores every ⁠(src_column, token)⁠ by its document frequency - the share of records in that column whose value contains the token - and returns the tokens common enough to be poor discriminators. These are stopword candidates: feed them to filter_stopwords() in the preparer chain and re-run prepare_search_data().

Usage

find_stopwords(
  tokens,
  max_prop = 0.3,
  top_n = NULL,
  by_block = FALSE,
  block_by = NULL
)

Arguments

Details

Document frequency is computed corpus-wide by default (by_block = FALSE), i.e. across all blocks. This matches the intuition of a stopword as a globally common term. With by_block = TRUE the share is computed within each block and a token is returned if it crosses max_prop in any block, reported at its maximum block-level share - useful when a token is rare overall but saturates a single dense block.

Value

A data.table with one row per flagged ⁠(src_column, token)⁠: src_column, token, df (distinct records containing the token), n_records (records in the column / block), and prop = df / n_records. Sorted by src_column then descending prop. Empty (zero-row) when nothing crosses the threshold.

See Also

filter_stopwords() to apply the result in a preparer chain.

Fit a false-positive filter on labelled match pairs

Description

Fit a baseline classifier to predict whether each scored pair is a true match (equal == 1L) or a false positive (equal == 0L). The baseline path uses stats::glm with the logit link and no external dependencies. The features object is the input from match_features(); labels carry the equal column produced by import_labels().

Usage

fit_filter(
  features,
  labels,
  model = "logistic",
  class_weighted = FALSE,
  na_fill = 0,
  ...
)

Arguments

Value

A Filter_Model object.

Examples

strat <- search_strategy(
  workshop   ~ normalize_text() + word_tokens(min_nchar = 3),
  proprietor ~ normalize_text() + word_tokens(min_nchar = 2),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.30
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
feats <- match_features(matches, strat,
                        base = workshop_listings, id = "listing_id",
                        target = workshop_register, target_id = "reg_no")
# match_labels_example carries the same pairs with a hand-checked `equal` flag.
model <- fit_filter(feats, match_labels_example)
model

Cost/recall frontier scatter for a strategy plan

Description

Plots each candidate block key at (candidate, exact_twin_survival) - the recall axis - with the brute-pair cost in the point labels. The knee is the cheapest candidate whose twin survival stays high.

Usage

frontier_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (frontier).

Collapse near-duplicate tokens to a canonical form

Description

Typos and minor spelling differences split one real token into many ("Neumann", "Neumann" with a slip, "Neuman"). fuzzy_tokens() finds tokens within a string distance of each other, groups them, and rewrites every member of a group to one canonical spelling, so the variants match. Unlike use_dictionary(), which needs a known synonym list, this discovers the groups from the data.

Usage

fuzzy_tokens(x, max_dist = 2, method = "osa", min_nchar = 1)

Arguments

Details

Use it when a field has organic spelling noise and you do not have a dictionary. The canonical form per group is the longest token, breaking ties by the most central token, then alphabetically.

When not to use it:

• High-cardinality columns. It compares every distinct token against every other in one dense distance matrix, so cost and memory grow with the square of the number of distinct tokens. On a large vocabulary (tens of thousands of distinct tokens and up) it is slow and memory-hungry. Normalize aggressively first, and prefer use_dictionary() when the groups are already known.

• When over-merging is costly. Grouping is by connected components, so matches chain transitively: if A is close to B and B to C, all three collapse even when A and C are far apart. A loose max_dist or short tokens can fuse genuinely distinct values. Keep max_dist tight, raise min_nchar to drop noise-prone short tokens, and check the groups on a sample before trusting them.

Value

A list of character vectors, one per input element, with each token replaced by its group's canonical form.

See Also

use_dictionary() when the groups are known in advance.

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), extract_initials(), filter_stopwords(), token_shapes(), use_dictionary()

Examples

fuzzy_tokens(c("Neumann", "Neumaxn", "Neuman"), max_dist = 2)
# every row's token becomes "NEUMANN"

Generate character n-grams from text

Description

An n-gram is a sliding window of n consecutive characters. Matching on character n-grams instead of whole words tolerates typos, truncations, and joined-up spellings, because two strings that differ by a letter still share most of their windows ("meier" and "maier" share "ei", "er", and so on). Reach for it on short, noisy fields where word tokens are too brittle.

Usage

generate_ngrams(text, n)

Arguments

Details

It tokenizes text directly, so it replaces word_tokens() rather than following it. The trade-off is fan-out: every string yields many overlapping tokens, so n-grams cost more to match than words. Larger n is sharper and cheaper, smaller n is fuzzier and denser.

Value

A list of character vectors, one per input element. Strings shorter than n yield an empty vector.

See Also

word_tokens() for whole-word tokens.

Other token generators: numeric_tokens(), word_tokens()

Examples

generate_ngrams("hello", 2)
generate_ngrams("an example", 3)

Import a labelled CSV back into a feature/label table

Description

Read a CSV written by export_for_labelling() (optionally edited by a user), propagate the block-default equal value from each header row onto unmarked rows in that block, validate the schema, and return a data.table ready for fit_filter() / calibrate_matches().

Usage

import_labels(file)

Arguments

Value

A data.table with the same rows as the original sample plus a fully populated equal column (0L / 1L).

See Also

export_for_labelling()

Inspect Tokens for a Specific Column

Description

Extract and examine the tokens generated for a specific column after applying the preprocessing steps defined in a Search_Strategy. Useful for debugging and understanding how text is tokenized.

Usage

inspect_tokens(data, id, strategy, column)

Arguments

Value

A backend-specific table showing the tokens generated for the specified column.

Build a tidymodels recipe for calibration features

Description

Construct a pre-configured recipes::recipe() suitable for fitting a false-positive filter on the output of match_features(). Tags ID columns (searched, found, match_id) with role "id", sets equal as the outcome, and keeps every other numeric column as a predictor. Requires the suggested recipes package.

Usage

joinery_recipe(features, labels, ...)

Arguments

Value

A recipes::recipe() object.

Build a per-pair feature table for calibration

Description

Computes a wide, one-row-per-pair feature data.table from a joinery match result, suitable for downstream calibration / false-positive filtering. The schema is documented in notes/calibration_design.md and treated as the public API. Additions are allowed; reorders or renames are not.

Dispatches on ⁠(matches, strategy)⁠. A Search_Strategy returns the full token schema (core + token-side columns + string similarity). An Embedding_Strategy returns the reduced "embedding" schema (core columns + string similarity + cosine_sim + embedding norms).

Usage

match_features(matches, strategy, ...)

Arguments

Value

A Match_Features object wrapping a wide feature data.table.

Examples

strat <- search_strategy(
  workshop   ~ normalize_text() + word_tokens(min_nchar = 3),
  proprietor ~ normalize_text() + word_tokens(min_nchar = 2),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.30
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
# One row per pair, with the features a filter can learn from.
feats <- match_features(matches, strat,
                        base = workshop_listings, id = "listing_id",
                        target = workshop_register, target_id = "reg_no")
feats

Labelled candidate pairs for calibration examples

Description

A frozen set of candidate match pairs with a ground-truth equal label, for the calibration workflow (fit_filter, apply_filter, calibrate). It is a saved search_candidates() run linking workshop_listings to workshop_register, with equal filled from each listing's true actual_link. The false positives are concentrated in the deliberately hard tiers, common-surname homonyms and planted register duplicates, so a learned filter has real mistakes to catch. The schema matches what import_labels returns, so it feeds the calibration verbs with no manual labelling step.

Usage

match_labels_example

Format

A data frame with 1,862 rows (two rows per pair) and 12 variables:

match_id

Integer pair id. Each id groups one base row and one target row.

score

The candidate score from the frozen search

source

"base" (the searched listing) or "target" (the register candidate). The candidate row carries the label.

id

The record id: a listing_id on base rows, a reg_no on target rows

workshop

Business name of the row

proprietor

Proprietor name of the row

trade

Trade

postcode_area

UK outward-code area

gen_tier

The generation tier of the row, useful for slicing the hard cases

actual_link

The searched listing's true reg_no (present on base rows)

rank

Candidate rank within the pair

equal

Evaluation label. 1 when the target candidate is the listing's true match, 0 otherwise. On base header rows it is the block default.

Source

Synthetically generated by data-raw/generate_match_labels.R from the shipped workshop_listings / workshop_register pair.

See Also

fit_filter, import_labels, workshop_listings

Materialize Records by ID

Description

Rehydrate a set of record IDs back into their full records. The positive (semi-join) complement of extract_unmatched(): where extract_unmatched() produces a residual set of IDs, materialize_records() pulls those IDs back into complete, scorable rows for the next stage.

Usage

materialize_records(data, id, ids, ...)

Arguments

Details

ids is polymorphic. It may be either

• an atomic vector of ID values, or

• a table (data.frame / data.table / backend tbl) carrying the IDs. The lookup order for the ID column is: a column literally named id first (the extract_unmatched() / resolve_entities() output convention), otherwise a column named the same as id.

The return is a semi-join: IDs absent from data are silently dropped (there is nothing to rehydrate), never NULL-filled. IDs are coerced to a common type on both sides, so a BIGINT-corpus / character-id request still matches. Row order is not guaranteed; the caller sorts if needed.

On the DuckDB backend the IDs are always registered as a temp table and joined - never inlined as an ⁠id IN (<literal list>)⁠, which binds in roughly O(n^2) and pins cores for minutes on large residual sets.

Value

The rows of data whose ID is in ids, all columns intact, one row per matching record, in no guaranteed order.

See Also

extract_unmatched(), the negative complement that produces the residual IDs this verb rehydrates.

Staged Duplicate Detection (within one table)

Description

Deduplicate a single table in increasingly tolerant passes. A typical run starts with a cheap exact_strategy() pass that catches the clean duplicates, then applies looser search_strategy() passes (often with wider blocking) to the records still unmatched. All the links found across the passes are grouped into duplicate groups at the end, so a record linked to B in an early pass and B linked to C in a later one all land in the same group.

For linking across two tables or several sources, use multi_stage_search().

Usage

multi_stage_dedup(table, id, strategies, ...)

Arguments

Value

The standard dedup result: duplicate_group | id | score | rank plus the original columns of table, and a stage column recording which pass first linked each record.

See Also

multi_stage_search() for the cross-table version, detect_duplicates() for a single pass, resolve_entities() for the grouping step.

Examples

# Two passes over one table: exact token-set first, then a looser fuzzy pass
# on whatever the exact pass left unmatched.
exact <- exact_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = c("postcode_area", "trade")
)
fuzzy <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.6
)
dups <- multi_stage_dedup(workshop_register, "reg_no",
                          list(exact = exact, fuzzy = fuzzy))
head(dups)

Staged Search Across Tables or Sources

Description

Link the same real-world entity across two tables, or across several datasets or vintages of one dataset, by running an ordered list of strategies as successive search passes. Each pass adds the links it finds to a running record of every match (the ledger), and at the end all the links are grouped into entities, one row per record showing which entity it belongs to.

A typical run starts with a cheap exact_strategy() pass to catch the clean matches, then applies one or more looser search_strategy() passes to the records still unmatched. Use this when the two sides are not interchangeable: for example one record may carry only part of another's information, so it matters which side is searched against which. For finding duplicates within a single table, use multi_stage_dedup() instead.

Usage

multi_stage_search(
  base_table,
  target_table,
  base_id,
  target_id,
  strategies,
  ...
)

Arguments

Value

One row per pooled record describing its entity: entity | id | rep | rank | score | source | covered_sources | n_in_entity | stage. The full list of links found, with the stage and direction of each, is attached as the ledger attribute and read with attr(result, "ledger").

See Also

multi_stage_dedup() for the within-one-table version, resolve_entities() for the grouping step, exact_strategy() for the usual front stage.

Examples

# Follow each workshop across years: pool the panel, search it against itself,
# exact first then fuzzy, collapsing each group found so later passes see less.
exact <- exact_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = c("postcode_area", "trade")
)
fuzzy <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.55
)
g <- multi_stage_search(
  workshop_panel, workshop_panel,
  base_id = "record_id", target_id = "record_id",
  list(exact = exact, fuzzy = fuzzy),
  self = TRUE, source_by = "year", collapse = "rep"
)
head(g)

Bar chart of embedding norm quantiles

Description

Plots p05/p25/p50/p75/p95 of the embedding vector norms. A norm of 1 is annotated; for an L2-normalised strategy all bars should sit on it.

Usage

norm_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (quantile, norm).

Normalize dates to ISO 8601 format (YYYY-MM-DD)

Description

The same day is written "31.12.2023", "12/31/2023", or "2023-12-31" depending on who typed it. normalize_date() parses these mixed formats and rewrites them to one ISO 8601 string (YYYY-MM-DD), so a date column matches on the day it names rather than on how it was formatted. It recognizes European (DD.MM.YYYY), American (MM/DD/YYYY), and ISO-style inputs.

Usage

normalize_date(x, format = NULL, orders = c("ymd", "dmy", "mdy"))

Arguments

Details

Returns text. For matching on individual date parts (year only, year and month) use date_tokens(); to deliberately blur near-dates together use approximate_date().

When format is provided, uses as.Date(x, format) directly. When format = NULL, tries lubridate::parse_date_time() with the specified orders to handle mixed formats flexibly.

Value

A character vector of dates in ISO 8601 format (YYYY-MM-DD). Unparseable dates return NA_character_ with a warning.

See Also

Other date preparers: approximate_date(), date_tokens()

Examples

normalize_date("31.12.2023")
# "2023-12-31"

normalize_date("12/31/2023")
# "2023-12-31"

normalize_date(c("2023-01-15", "15.01.2023", "01/15/2023"))
# c("2023-01-15", "2023-01-15", "2023-01-15")

normalize_date("31-12-2023", format = "%d-%m-%Y")
# "2023-12-31"

Normalize street names across languages

Description

Street names are written many ways for the same place: "Hauptstr.", "Hauptstrasse", "Haupt Strasse". normalize_street() collapses those variants to one canonical spelling so an address column matches on the street name rather than on its abbreviation. It normalizes Unicode, folds to ASCII, upper-cases, and cleans whitespace, then rewrites known street-type tokens from a multilingual dictionary.

Usage

normalize_street(
  x,
  lang = NULL,
  drop_house_numbers = FALSE,
  drop_stopwords = FALSE,
  dict = joinery::street_types,
  stopwords = joinery::street_stopwords
)

Arguments

Details

Returns text, so it sits where normalize_text() would in a pipeline, ahead of a token generator: street ~ normalize_street(lang = "de") + word_tokens().

Exact matches (e.g., "st", "rd.", "via") are always replaced. Suffix matches (e.g., German "strasse" endings or Dutch "straat") are applied only when lang is explicitly specified, which prevents unsafe substitutions such as rewriting the ending of "LINCOLN LANE".

Normalization steps include:

• Unicode -> Latin transliteration and ASCII folding (stri_trans_general)

• Conversion to uppercase

• Removal of non-alphanumeric characters

• Tokenization on spaces and per-token replacement

Exact variants are replaced verbatim with their canonical form. Suffix variants are replaced only when:

• lang is specified, and

• the token ends with a known variant suffix for that language.

Value

A character vector of normalized street names. NA inputs are preserved as NA. Rows reduced to nothing (e.g. a bare house number with drop_house_numbers = TRUE) become "".

See Also

Other text normalizers: normalize_text(), strip_vowels()

Examples

normalize_street("Muellerstrasse", lang = "de")
# "MUELLERSTRASSE"

normalize_street("123 Main St.")
# "123 MAIN STREET"

normalize_street("Calle Mayor 3", lang = "es")
# "CALLE MAYOR 3"

normalize_street("Hauptstr. 123A", lang = "de", drop_house_numbers = TRUE)
# "HAUPTSTRASSE"

normalize_street("An der Alster 5", lang = "de",
                 drop_house_numbers = TRUE, drop_stopwords = TRUE)
# "ALSTER"

Normalize text for matching

Description

The usual first step in a preparer pipeline. Folds text to upper case, transliterates accented and non-Latin characters to ASCII, drops anything that is not a letter, digit, or space, and collapses runs of whitespace. The point is to make superficial differences in case, accents, and punctuation disappear so that "Cafe-Conac" and "cafe conac" reduce to the same text before it is split into tokens.

Usage

normalize_text(text, transliteration = "De-ASCII")

Arguments

Details

Returns text, so it goes ahead of a token generator such as word_tokens() in a strategy: name ~ normalize_text() + word_tokens().

Value

A character vector the same length as text: upper-cased, ASCII, alphanumeric-and-space only, with surrounding and repeated spaces removed.

See Also

word_tokens(), the token generator that usually follows.

Other text normalizers: normalize_street(), strip_vowels()

Examples

normalize_text("Cafe Conac")
normalize_text("Strasse", transliteration = "Latin-ASCII")

Tokenize numeric fields, expanding ranges into individual numbers

Description

Turns numeric/house-number-like text into a list of tokens. Expands ranges such as "12-14" or "7-9" into c("12","13","14"). Uses original spacing/separators to detect ranges, while normalization cleans text for tokenization.

Usage

numeric_tokens(text, keep_letters = TRUE, destructive = FALSE)

Arguments

Value

A list of character vectors, one per input element. Each vector contains numeric tokens, with ranges expanded into sequences.

See Also

drop_numeric_tokens(), its inverse, to discard numbers from a token column instead.

Other token generators: generate_ngrams(), word_tokens()

Examples

numeric_tokens("12-14")
# list(c("12", "13", "14"))

numeric_tokens("7A 9B", keep_letters = TRUE)
# list(c("7A", "9B"))

numeric_tokens("House 5", destructive = TRUE)
# list("5")

Plan a Search Strategy from Raw Inputs

Description

Helps you choose a blocking before you run anything. Where audit_strategy() grades a strategy you have already settled on, and rarity_distribution() reads one column's token distribution, plan_strategy() compares several candidate blockings side by side and shows the trade-off between how many comparisons each one costs and how many true matches it would keep together.

It never builds the pair set, so it is safe to run on a full corpus. For each candidate blocking it reports: how many blocks it makes and how big they are, an estimate of how many record comparisons it implies, and the share of identical-token records that stay in the same block (the recall it would cost you). It also reports how much an exact_strategy() front stage would absorb, the shape of the leftover records, and how discriminative each column is, including a warning when a column that is often empty puts a ceiling on achievable scores.

The strategy you pass supplies only the column preparation steps; its own block_by is ignored, since the blocking is exactly what you are choosing here.

Usage

plan_strategy(
  base,
  strategy,
  target = NULL,
  block_candidates = list(),
  base_id = NULL,
  target_id = NULL,
  n_offenders = 20L,
  min_rarity_grid = NULL,
  containment = FALSE,
  ...
)

Arguments

Value

A Strategy_Plan object.

See Also

audit_strategy() to grade a chosen strategy, rarity_distribution() for one column's distribution, exact_strategy() for the front stage it sizes.

Examples

strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3)
)
# Compare two candidate blockings side by side before committing to one.
plan_strategy(
  workshop_register, strat,
  block_candidates = list(area       = "postcode_area",
                          area_trade = c("postcode_area", "trade")),
  base_id = "reg_no"
)

Prepare Data for Record Linkage Search

Description

Turn a table into the long-format token table the matching verbs work on: it applies each column's preparation steps, splits the text into tokens, and attaches the id and any blocking columns. The other verbs (detect_duplicates(), search_candidates()) call this for you, so you rarely need it directly; reach for it when you want to see or post-process the tokens yourself.

Usage

prepare_search_data(data, id, strategy, ...)

Arguments

Value

A long-format token table with one row per token, carrying the id, the source column, the token, a row_id, and any blocking columns.

See Also

inspect_tokens() for a quick per-column look at the tokens.

Read the Token Rarity Distribution

Description

A pre-match read of how token rarity is distributed in your data. For each column (and block, when the strategy blocks) it reports the spread of token document frequency and rarity, plus an offender list: the most common tokens, the ones that drive a match to balloon. Use it to set min_rarity and max_token_df from what is actually in the data instead of guessing.

It never builds the pair set: it only tokenizes and measures rarity, so it is cheap enough to run on a full corpus before committing to a strategy.

Usage

rarity_distribution(data, id, strategy, ...)

Arguments

Value

A Rarity_Distribution object.

See Also

search_strategy() for the min_rarity / max_token_df levers this verb informs; audit_strategy() for the broader pre-match audit.

Examples

strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = c("postcode_area", "trade")
)
# Read the token distribution and the most common tokens before matching.
rarity_distribution(workshop_register, "reg_no", strat)

Bar chart of median token rarity per column

Description

Bar chart of median token rarity per column

Usage

rarity_histogram(x, ...)

Arguments

Value

Invisibly, the plotted data.table (column_rarity_stats).

Recommendations from a Diagnostic Object

Description

Accessor returning the recommendations strings stored on a diagnostic result object. Returns character(0) when no recommendations fired. The same strings are surfaced inline by the object's print() method.

Methods for individual classes live alongside those classes - diagnostic classes (Match_Overview, Strategy_Audit) in diagnostic_classes.R; calibration classes (Calibrated_Matches, Filter_Calibration) in calibration_classes.R.

Usage

recommendations(x, ...)

Arguments

Value

A character vector.

Group Matched Pairs into Entities

Description

Take a list of matched record pairs (an edge list) and turn it into entities: records that link directly or through a chain of links are grouped together, each group gets an entity number, and one record in each group is marked as its representative.

This is the grouping step detect_duplicates() performs internally, exposed on its own so you can resolve any pair list into entities, for example the output of search_candidates() or a set of links you assembled yourself.

Usage

resolve_entities(
  edges,
  id_a,
  id_b,
  score = NULL,
  vertices = NULL,
  rep_by = NULL,
  block_by = NULL,
  ...
)

Arguments

Details

The result does not depend on the order of rows in edges: the same pairs always produce the same entity, rep, and rank. Entity numbers are assigned by the smallest member id in each group. The representative (the rank-1 member) is chosen by highest best score when a score column is given, then by smallest rep_by when given, then by smallest id.

Value

One row per resolved vertex:

id

The vertex id.

entity

Integer entity (connected-component) label.

rep

The canonical representative id of the entity.

rank

Rank within the entity; rank 1 is the representative.

score

Best incident-edge score per vertex (only when score is supplied).

Examples

# r1-r2 and r2-r3 chain into one entity; r4-r5 form another
edges <- data.table::data.table(
  a = c("r1", "r2", "r4"),
  b = c("r2", "r3", "r5")
)
resolve_entities(edges, id_a = "a", id_b = "b")

Sample Matches for Review

Description

Sampling diagnostic (Q4). Modes: "high", "low", "borderline", "ambiguous", "top_gap", "random".

Usage

sample_matches(matches, ...)

Arguments

Value

A Match_Sample object.

Examples

strat <- search_strategy(
  workshop ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = c("postcode_area", "trade"),
  threshold = 0.7
)
matches <- search_candidates(
  workshop_listings, workshop_register,
  base_id = "listing_id", target_id = "reg_no", strategy = strat
)
# Pull the borderline pairs near the threshold, the ones worth eyeballing.
sample_matches(matches, mode = "borderline", n = 5, threshold = 0.7)

Kernel density of the score distribution

Description

Expands the pre-binned histogram to approximate raw scores before passing to the density estimator.

Usage

score_density(x, threshold = x@score_dist$threshold %||% NA_real_, ...)

Arguments

Value

Invisibly, the data.table of expanded scores.

Score Embedding Pairs Using Cosine Similarity

Description

Compute cosine similarity scores between base and target embeddings. This is a pure scoring function that operates on pre-computed embeddings.

Usage

score_embeddings(base_embeddings, target_embeddings, strategy, ...)

Arguments

Value

A backend-specific table with columns: base_id, target_id, score.

Bar chart of the pre-binned score distribution

Description

Bar chart of the pre-binned score distribution

Usage

score_histogram(x, threshold = x@score_dist$threshold %||% NA_real_, ...)

Arguments

Value

Invisibly, the plotted data.table (histogram with bin_mid column).

Search for Candidate Matches Between Tables

Description

Find candidate matches between two tables: for each record on one side, the records on the other side that share enough rare, informative token content to score at or above the threshold. This is the cross-table counterpart of detect_duplicates().

Pass a search_strategy() for fuzzy, scored matching, or an exact_strategy() to keep only pairs whose token sets are identical.

Usage

search_candidates(base_table, target_table, base_id, target_id, strategy, ...)

Arguments

Value

A table with two rows per matched pair (one for the base record, one for the target record), sharing a match_id:

match_id

Identifier shared by the two rows of a matched pair.

score

The pair's match score.

source

"base" or "target".

id

The record ID.

⁠<original columns>⁠

Every other column from the source table.

rank

Rank of this candidate among a record's matches.

See Also

detect_duplicates() for the within-table version, extract_unmatched() for the residual, multi_stage_search() for staged passes.

Examples

data(base_example)
data(target_example)

strat <- search_strategy(
  Nachname ~ normalize_text() + word_tokens(min_nchar = 3),
  Vorname  ~ normalize_text() + word_tokens(min_nchar = 3),
  Ort      ~ normalize_text(),
  block_by = "Kreis",
  threshold = 0.8
)

matches <- search_candidates(
  base_example, target_example,
  base_id = "id_base", target_id = "id_target",
  strategy = strat
)
head(matches)

Define a Search Strategy for Record Linkage

Description

Creates a Search_Strategy object that specifies how columns should be preprocessed for token index based record linkage, along with optional weights, blocking variables, rarity computation method, rIP smoothing, and similarity threshold.

Usage

search_strategy(
  ...,
  block_by = NULL,
  weights = numeric(),
  rarity = "inverse_freq",
  rarity_scope = c("block", "global"),
  min_rarity = 0,
  max_token_df = Inf,
  threshold = 0.9,
  smoothing = smooth_rip_identity(),
  max_candidates = Inf,
  max_fanout = 5e+07,
  on_fanout = c("cap", "abort", "off"),
  feedback_strength = 0,
  on_missing = c("penalise", "renormalise")
)

Arguments

Value

A Search_Strategy object.

Examples

# Tokenize two name columns, block on region, keep pairs scoring at least 0.8.
strat <- search_strategy(
  Nachname ~ normalize_text() + word_tokens(min_nchar = 3),
  Vorname  ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by  = "Kreis",
  threshold = 0.8
)
strat

Histogram of sampled pairwise cosine similarities

Description

Histogram of sampled pairwise cosine similarities

Usage

similarity_histogram(x, threshold = attr(x, "threshold"), bins = 30L, ...)

Arguments

Value

Invisibly, the histogram data.table with columns bin_lower, bin_upper, bin_mid, count.

Configure rIP smoothing for a search strategy

Description

Helper functions that construct S7 Smoothing objects used by search_strategy() to control how relative identification potential (rIP) is smoothed before scoring.

All helpers are pure configuration; they do not perform any computation by themselves. Backend methods for detect_duplicates() and search_candidates() interpret the resulting Smoothing object.

Usage

smooth_rip_identity()

smooth_rip_log()

smooth_rip_offset(alpha = 0.5)

smooth_rip_softmax(temperature = 1)

Arguments

Details

rIP Smoothing Helpers

Value

An object inheriting from Smoothing that can be passed to the smoothing argument of search_strategy().

Functions

• smooth_rip_identity(): Identity rIP smoothing (no transformation beyond standard per record normalization). This is the default.

• smooth_rip_log(): Logarithmic rIP smoothing. Backends typically apply log1p(rIP) and then renormalize within each record and column.

• smooth_rip_offset(): Offset based rIP smoothing with a constant offset alpha that is added to all rIP values before renormalization.

• smooth_rip_softmax(): Softmax style rIP smoothing with a temperature parameter that controls how sharp or flat the transformed distribution is.

See Also

search_strategy()

Line plot of cumulative base coverage by stage

Description

Uses percentage coverage when base was supplied to compare_stages(), raw record counts otherwise.

Usage

stage_coverage_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (marginal_coverage with stage_idx).

Grouped bar chart of score distributions by stage

Description

Grouped bar chart of score distributions by stage

Usage

stage_score_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (score_dist_by_stage with bin_mid).

Multilingual Street-Name Stopwords

Description

Locative particles and articles that recur inside multi-word street names but carry no discriminative signal for matching - German ⁠AN DER⁠, French ⁠DE LA⁠, Italian DELLA, and so on. Used by normalize_street() when drop_stopwords = TRUE to collapse e.g. "An der Alster" to "ALSTER".

Usage

street_stopwords

Format

An object of class tbl_df (inherits from tbl, data.frame) with 58 rows and 2 columns.

Details

The list is deliberately tight: only true prepositions and articles, never adjectives ("NEUE", "GROSSE") or directionals that can themselves be the distinguishing part of a name. Entries are uppercase ASCII so they join directly against normalize_street()'s already-uppercased, transliterated tokens. When a lang is supplied to normalize_street(), only that language's particles are removed.

Format

A tibble with two columns:

stopword

Character string. The particle in uppercase ASCII (e.g. "AN", "DER", "DE", "DELLA").

lang

ISO 639-1 language code ("de", "en", "fr", "es", "it", "pt", "nl").

Source

Manually curated from common multi-word street-name patterns across languages. Expandable as new particles are encountered.

See Also

normalize_street(), street_types

Multilingual Street-Type Normalization Dictionary

Description

A curated cross-linguistic dictionary of street-type forms used for robust address standardization and record linkage. Each entry maps a variant - including abbreviations, orthographic alternatives, morphological forms, and transliterated spellings - to a canonical street-type label.

Usage

street_types

Format

An object of class tbl_df (inherits from tbl, data.frame) with 143 rows and 4 columns.

Details

Unlike simple suffix lists, this dictionary encodes language-specific normalization rules. Each variant is marked as either:

• "exact" - the variant should only match a token when it appears exactly, e.g. "st." -> "STREET" (English), "pl" -> "PLAZA" (Spanish)

• "suffix" - the variant may safely match a token ending with that sequence, e.g. "gatan" -> "GATA" (Swedish), "strasse" -> "STRASSE" (German)

By separating exact vs. suffix behaviour and tagging each entry with an ISO language code, joinery can normalize addresses without incorrect transformations (e.g. preventing "LINCOLN" -> "LANE", or "VICTOR" -> "RUE"). This structure enables high-precision multilingual address cleaning.

Languages Covered

The dictionary currently includes major street-type systems from:

• German - Straße, Gasse, Weg, Platz, Allee, etc.

• English - Street, Road, Avenue, Boulevard, Lane, etc.

• French - Rue, Avenue, Boulevard, Impasse, Quai, Chemin, etc.

• Spanish - Calle, Avenida, Paseo, Plaza, Camino, etc.

• Italian - Via, Piazza, Corso, Viale, etc.

• Portuguese - Rua, Avenida, Praça, Alameda, Travessa, etc.

• Polish - Ulica, Aleja, Plac, Osiedle, etc.

• Dutch - Straat, Laan, Weg, Plein, etc.

• Turkish - Sokak, Cadde, Bulvar, Meydan, etc.

• Swedish - Gata, Gatan, Vägen, Torg, etc.

• Danish/Norwegian - Gade, Vej, Plads, etc.

• Greek (transliterated) - Odos, Leoforos, Plateia

• Russian (transliterated) - Ulitsa, Prospekt, Pereulok, etc.

Additional languages and street-type systems can be incorporated as needed.

Use in normalize_street()

street_types is used by normalize_street() to:

• standardize street-type tokens to a canonical form,

• optionally apply language-specific suffix rules (lang = "de", "sv", etc.),

• avoid over-normalization by matching only valid variants for the specified language,

• support multilingual cleaning workflows in data preprocessing and record linkage.

Format

A tibble with four columns:

canonical

Character string. The standardized street-type label in uppercase ASCII (e.g. "STRASSE", "AVENUE", "PLAC").

variant

Character string. A lowercase spelling, abbreviation, transliteration, or inflected form seen in raw address data (e.g. "str.", "straße", "avda", "gatan").

type

Either "exact" or "suffix", indicating whether the variant should match only whole tokens or may safely match as a word-final suffix.

lang

ISO 639-1 language code (e.g. "de", "en", "fr", "sv"), used to restrict normalization to the appropriate street-type system.

Source

Manually curated based on postal conventions, open datasets, and commonly observed street-name variations across languages. The dictionary is periodically expanded as new variants are encountered in real-world data.

Strip vowels from text (consonant skeleton)

Description

Reduces text to its consonant skeleton by removing vowels (A, E, I, O, U, including accented variants). Two spellings that differ only in their vowels, such as "MEYER" and "MAYER" or "MUELLER" and "MULLER", collapse to the same skeleton, so they match despite the difference. It is a lighter-weight alternative to the phonetic encoders (as_soundex(), as_metaphone()) when you only want to ignore vowel variation.

Usage

strip_vowels(text)

Arguments

Details

Returns text, so it goes ahead of a token generator in a pipeline.

Value

A character vector with vowels removed, upper-cased and ASCII-folded.

See Also

as_soundex() and as_metaphone() for full phonetic encoding.

Other text normalizers: normalize_street(), normalize_text()

Examples

strip_vowels("Mueller")   # "MLLR"
strip_vowels("Cafe Noir") # "CF NR"
strip_vowels(c("Anna", "Peter"))

Summarise a Match Result

Description

Post-match overview (Q2). Auto-detects whether the input is a duplicate table (presence of duplicate_group column) or a candidate table (presence of match_id and source columns), and reports score distribution, coverage (when base / target are supplied), cluster-size or candidates-per-record distribution, and top-1-vs-top-2 score-gap distribution for candidates. Recommendations link symptoms to strategy levers.

Usage

summarise_matches(matches, ...)

Arguments

Value

A Match_Overview object.

Examples

s <- search_strategy(
  Nachname ~ normalize_text() + word_tokens(min_nchar = 3),
  block_by = "Kreis",
  threshold = 0.8
)
dups <- detect_duplicates(base_example, "id_base", s)
summarise_matches(dups, base = base_example)

Target dataset for record linkage example

Description

A dataset containing 3,000 person records designed to match with base_example. Approximately 80% of records correspond to records in the base dataset but with realistic errors and variations (typos, abbreviated names, title additions, street name variations). The remaining 20% are new records with no match.

Usage

target_example

Format

A tibble with 3,000 rows and 7 variables:

actual_link

The actual base_id in the simulation process

id_target

Character identifier for target records (T0001-T3600)

Vorname

First name, may include titles, initials, or middle names

Nachname

Last name, may contain typos or token swaps

Strasse

Street name with possible abbreviations or typos

Hausnummer

House number with possible letter suffixes

Ort

City or town name, may contain typos

Kreis

Administrative district (Kreis)

Source

Synthetically generated by distorting 80% of base_example records and adding 20% new unmatched records.

Horizontal bar chart of per-token score contributions, coloured by column

Description

Horizontal bar chart of per-token score contributions, coloured by column

Usage

token_contribution_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (shared_tokens with token_label).

Bar chart of average tokens per record per column

Description

Bar chart of average tokens per record per column

Usage

token_frequency_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table (column_token_stats).

Convert tokens to shape signatures

Description

Reduces each token to its letter/digit pattern: every letter becomes "A", every digit "N", anything else "X". The signature ignores the actual characters and keeps only the layout, which is useful for matching on the format of a code or identifier (postal codes, licence plates, product codes) rather than its exact value, or as a coarse blocking key.

Usage

token_shapes(tokens)

Arguments

Details

It transforms a token column, so it runs after a token generator such as word_tokens().

Value

A list of character vectors of shape signatures, one signature per input token.

See Also

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), extract_initials(), filter_stopwords(), fuzzy_tokens(), use_dictionary()

Examples

token_shapes(list(c("MUELLER", "A12B")))
# list(c("AAAAAAA", "ANNA"))

Bar chart of top-1 vs top-2 score gap distribution (candidates only)

Description

Bar chart of top-1 vs top-2 score gap distribution (candidates only)

Usage

top_gap_density(x, ...)

Arguments

Value

Invisibly, the plotted data.table (top_gap_dist with bin_mid).

Map tokens to canonical groups with a lookup table

Description

When you already know which tokens mean the same thing (a curated synonym list, brand-name variants, a code-to-label table), use_dictionary() rewrites each token to its group label so the variants collapse to one token and match. Use it when the mapping is known in advance; when you instead want joinery to discover near-duplicates from the data, use fuzzy_tokens().

Usage

use_dictionary(text, dict)

Arguments

Details

Tokens absent from the dictionary return no group, so chain this after a token generator and keep a sharper field alongside it.

Value

A list of character vectors, one per input element, holding the matched group labels (empty when the token is not in dict).

See Also

fuzzy_tokens() to discover groups from the data instead.

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), extract_initials(), filter_stopwords(), fuzzy_tokens(), token_shapes()

Examples

dict <- data.table::data.table(
  tokens = c("example", "sample"),
  token_group = c("example/sample", "example/sample")
)
use_dictionary("example", dict)
use_dictionary("nonexistent", dict)

Bar chart of vocabulary overlap between base and target per column

Description

Bar chart of vocabulary overlap between base and target per column

Usage

vocab_overlap_plot(x, ...)

Arguments

Value

Invisibly, the plotted data.table.

Split text into word tokens

Description

The workhorse tokenizer. It splits each string on whitespace into a vector of words, the tokens joinery matches on. It almost always follows normalize_text(), which strips punctuation and case first so the split is clean: name ~ normalize_text() + word_tokens().

Usage

word_tokens(text, min_nchar = 0)

Arguments

Details

Set min_nchar to drop very short tokens (single initials, stray letters) that match too easily and add noise.

Value

A list of character vectors, one per input element, each holding that element's word tokens.

See Also

normalize_text(), the usual preceding step; filter_stopwords() to drop common words by name.

Other token generators: generate_ngrams(), numeric_tokens()

Examples

word_tokens("this is an example")
word_tokens("this is an example", min_nchar = 3)  # drops "is", "an"

Workshop external directory (target) for record linkage examples

Description

A messier external directory of the same UK workshops as workshop_register, the target table for cross-source linkage. Listings carry the realistic distortions a scrappy directory introduces, planted as labelled tiers (via gen_tier) so each exercises a specific joinery feature: slogan-stuffed supersets (exact containment), movers that changed postcode area (token blocking with global rarity), phonetic name variants (Cologne/Soundex encoders), shared-venue and bare-category rows that bait over-linking (the containment guards), and common-surname homonyms (ambiguity and calibration). The matchable columns share names with workshop_register so one formula serves both tables.

Usage

workshop_listings

Format

A tibble with 894 rows and 8 variables:

listing_id

Character identifier for listings (e.g. "L00042")

workshop

Directory rendering of the business name, often messy

proprietor

Directory rendering of the proprietor name; may be missing

trade

Trade; half the blocking key

postcode_area

UK outward-code area; half the blocking key

town

Town for the postcode area

actual_link

Evaluation only. The reg_no this listing refers to, or NA for a workshop absent from the register.

gen_tier

Evaluation only. The feature-exercise tier (clean, slogan, variant, mover, phonetic, hub_member, hub_trap, category_trap, homonym_area, homonym_block, homonym_total, new).

Source

Synthetically generated by data-raw/generate_workshop_example.R from workshop_register and a frozen LLM seed of messy renderings.

See Also

workshop_register

Multi-year workshop panel for cross-year linkage examples

Description

A small pooled-long panel of the same UK woodworking workshops as workshop_register, observed across five years (2019 to 2023). It is the runnable example for cross-year entity resolution: each row is one workshop in one year, names drift over time, a minority of workshops relocate to a different postcode area, and workshops enter and exit so trajectories have gaps. The task is to recover the stable identity (true_entity) behind the year-by-year rows. A reappearance window of five years matches the design of the real Yellow Pages panel the package was built against.

Usage

workshop_panel

Format

A tibble with 847 rows and 10 variables:

record_id

Character row id, unique per workshop-year (e.g. "YR-00042")

year

Observation year, 2019 to 2023

workshop

Business name as recorded that year, with light per-year noise

proprietor

Proprietor name, in a directory rendering that varies by year

trade

One of eight woodworking trades

postcode_area

UK outward-code area; changes mid-trajectory for movers

town

Town for the postcode area

established

Year the workshop was established (stable within an entity)

true_entity

Evaluation only. The stable entity key (the workshop_register reg_no); every year-row of one workshop shares it.

change_tier

Evaluation only. The cross-year challenge the entity carries: stable (per-year noise only), name_drift (a structural name change part-way through), mover (a postcode-area relocation), or phonetic (a code-preserving stem twin).

Source

Synthetically generated by data-raw/generate_workshop_panel.R, which draws core workshops from workshop_register and gives each a year span with drift. Seeded and offline; ships no real business.

See Also

workshop_register, workshop_listings

Workshop guild register (base) for record linkage examples

Description

A synthetic register of UK joinery and carpentry workshops, styled like an excerpt from a Guild of Master Craftsmen trade roll. It is the clean base table for the linkage examples: distinctive workshop names paired with boilerplate trade and legal-form terms, so the rarity-versus-boilerplate behaviour of the matcher is visible. Pairs with workshop_listings, the messier external directory. Block on (postcode_area, trade).

Usage

workshop_register

Format

A tibble with 1,052 rows and 15 variables:

reg_no

Character registration number, the base id. Most are "GMC-#####"; planted duplicates, homonyms, and shared-venue rows carry "GMC-D####", "GMC-H####", and "GMC-V####" prefixes respectively.

workshop

Canonical business name (distinctive stem plus trade and legal-form boilerplate)

proprietor

Proprietor name

trade

One of eight woodworking trades; half the blocking key

legal_form

Ltd, LLP, Partnership, or Sole Trader

postcode_area

UK outward-code area (e.g. "LS"); half the blocking key

town

Town for the postcode area

address

Street address

established

Year the workshop was established

employees

Headcount, varying with legal form

apprentices

Number of apprentices

guild_member

Logical, whether a current guild member

sic

UK SIC 2007 industry code for the trade

true_entity

Evaluation only. Same-entity key: planted duplicate rows share it, homonym workshops get distinct keys.

gen_tier

Evaluation only. Which generation tier the row belongs to. Three rows are hub_trap: short-named shared venues ("Trinity Workshops", "The Forge", "Riverside Works") that are themselves guild registered. Their two-token names are a forward-containment subset of every ⁠<workshop>, <venue>⁠ listing, so they bait an exact containment strategy into merging unrelated workshops; the min_containment_tokens guard blocks them.

Source

Synthetically generated. Distinct workshop identities come from a frozen LLM seed (data-raw/llm_workshop_seed.R); all geography, colour columns, planted duplicates, and homonyms are added by the seeded, offline data-raw/generate_workshop_example.R. Ships no real business.

See Also

workshop_listings