Why pre-linkage quality matters

The Fellegi-Sunter EM algorithm scores pairs by comparing field values. Poor data quality in any of those fields degrades the score distribution in ways that no threshold choice can fully recover:

  • High missingness in a comparison variable reduces its discriminating power, flattening the valley between match and non-match clusters.
  • Invalid Medicare numbers that happen to match a wrong AIR record add spurious positive weight to false pairs.
  • Name typos are expected (the algorithm handles them via fuzzy string comparison) but systematic truncation or encoding differences are not.
  • Duplicate IDs in the primary dataset produce multiple output rows per original record, inflating counts and biasing VE estimates.

starling provides three functions specifically for catching these issues before murmuration() is called.

preflight() — structured pre-linkage audit

preflight() runs a battery of checks across both datasets and returns a structured report. It is the recommended first step before any linkage call.

library(starling)
data(cases_notifiable)
data(vax_air)

audit <- preflight(
  data1        = cases_notifiable,
  data2        = vax_air,
  linkage_vars = c("lettername1", "lettername2", "dob", "medicare10"),
  id_col1      = "id_var",
  id_col2      = "id_var",
  date_cols    = c("dob", "onset_date"),
  medicare_col = "medicare10",
  verbose      = TRUE
)
#> 
#> ================================================================ 
#>   starling::preflight() - Pre-Linkage Data Quality Report
#> ================================================================ 
#>   Dataset 1: 300 records x 10 columns
#>   Dataset 2: 400 records x 15 columns
#> 
#> == 1. Completeness of linkage variables ==
#>   Variable             Dataset     Present  Missing  % Missing
#>   ------------------------------------------------------------
#>   lettername1          data1           300        0         0%
#>   lettername2          data1           300        0         0%
#>   dob                  data1            NA       NA        NA%
#>   medicare10           data1           300        0         0%
#>   lettername1          data2           400        0         0%
#>   lettername2          data2           400        0         0%
#>   dob                  data2            NA       NA        NA%
#>   medicare10           data2           400        0         0%
#> 
#> == 2. Duplicate records (by ID column) ==
#>   Dataset       Records   Unique IDs   Duplicates    % Duplicate
#>   ------------------------------------------------------------
#>   data1             300          300            0             0%
#>   data2             400          400            0             0%
#> 
#> == 4. Date plausibility ==
#>   Column: dob
#>     data1 - Missing: 0 | Future: 0 | Pre-1900: 0 | Range: 1905-04-01 to 1989-03-10
#>     data2 - Missing: 0 | Future: 0 | Pre-1900: 0 | Range: 1905-02-09 to 1990-01-01
#>   Column: onset_date
#>     data1 - Missing: 0 | Future: 0 | Pre-1900: 0 | Range: 2024-01-05 to 2024-12-30
#>     data2: column not found
#> 
#> == 5. Medicare number validation ==
#>   data1 - Valid: 0 (0%) | Invalid checksum: 0 (0%) | Missing: 300
#>   data2 - Valid: 0 (0%) | Invalid checksum: 0 (0%) | Missing: 400
#> 
#> == 6. Name field quality ==
#>   Column               Dataset     Too short    Numeric Unusual chars
#>   -----------------------------------------------------------------
#>   lettername1          data1               0          0            0
#>   lettername1          data2               0          0            0
#>   lettername2          data1               0          0            0
#>   lettername2          data2               0          0            0
#> 
#> == 7. Categorical variable consistency (shared columns) ==
#>   'gender': data1 levels = {Female, Male} | data2 levels = {Female, Male}
#> 
#> == Summary of flags ==
#>   2 issue(s) flagged:
#>   [!] [HIGH MISSINGNESS] 'NA' in NA: NA missing
#>   [!] [HIGH MISSINGNESS] 'NA' in NA: NA missing
#> ================================================================

The audit list contains structured results for programmatic access:

# Completeness table
head(audit$completeness)
#>      variable dataset n_records n_present n_missing pct_missing
#> 1 lettername1   data1       300       300         0          0%
#> 2 lettername2   data1       300       300         0          0%
#> 3         dob   data1       300        NA        NA         NA%
#> 4  medicare10   data1       300       300         0          0%
#> 5 lettername1   data2       400       400         0          0%
#> 6 lettername2   data2       400       400         0          0%

# Duplicate ID counts
audit$duplicates
#>   dataset n_records n_unique_ids n_duplicate_ids pct_duplicate
#> 1   data1       300          300               0            0%
#> 2   data2       400          400               0            0%

# Flags raised (empty = all clear)
audit$flags
#> [1] "[HIGH MISSINGNESS] 'NA' in NA: NA missing"
#> [2] "[HIGH MISSINGNESS] 'NA' in NA: NA missing"

check_medicare() — Modulus 10 checksum validation

Australian Medicare numbers contain a check digit at position 9 calculated from positions 1–8 using the weights 1, 3, 7, 9, 1, 3, 7, 9. check_medicare() verifies this and returns a three-state flag:

Value Meaning
1L Checksum passes — number is internally consistent
0L Checksum fails — at least one digit is wrong
NA Missing, blank, or non-numeric — not verifiable 
cases_checked <- check_medicare(
  cases_notifiable,
  medicare_col = "medicare10",
  output_col   = "medicare_valid",
  verbose      = TRUE
)
#> 
#> -- starling::check_medicare() ----------------------------------
#>   Total records    : 300
#>   Missing / blank  : 300
#>   Present (n)      : 0
#>   Valid checksum   : 0  (N/A%)
#>   Invalid checksum : 0  (N/A%)
#>   Output column    : 'medicare_valid'  (1=valid, 0=invalid, NA=missing)
#> ----------------------------------------------------------------

# Distribution of flags
table(cases_checked$medicare_valid, useNA = "always")
#> 
#> <NA> 
#>  300

What to do with invalid Medicare numbers

Do not discard the record — it may still link correctly on name and DOB. Instead, replace the invalid number with NA before linkage so it does not contribute negatively to the score:

cases_checked$medicare10 <- ifelse(
  cases_checked$medicare_valid == 1L,
  cases_checked$medicare10,
  NA_character_
)

flock() — blocking variable construction

flock() creates one to three blocking keys from demographic fields. Blocking restricts murmuration() to comparing only pairs that share a blocking key, making the search tractable for large datasets without sacrificing recall on the key variables.

# Single-field block (gender only — broadest, lowest specificity)
cases_blocked <- flock(
  cases_checked,
  block1_vars    = "gender",         # block1: gender only
  block2_vars    = "gender",         # block2: same here; use composite in production
  block3_vars    = "postcode",       # block3: postcode (finer)
  birth_year_col = "dob"             # derives birth_year column for composite use
)

vax_blocked <- flock(
  vax_air,
  block1_vars    = "gender",
  block3_vars    = "postcode",
  birth_year_col = "dob"
)

# Inspect blocking key distributions
table(cases_blocked$block1)
#> 
#> Female   Male 
#>    141    159
head(sort(table(cases_blocked$block3), decreasing = TRUE))
#> 
#> 4562 4555 4552 4556 4557 4560 
#>   30   29   28   28   28   25

Choosing blocking variables for Australian datasets

Dataset size Recommended block Rationale
< 10 000 records gender Broadest; halves candidate space immediately
10 000–500 000 gender + birth_year composite Balances sensitivity and specificity
> 500 000 postcode + birth_year Fine-grained; requires high postcode completeness

For maximum recall, run murmuration() twice — once with a broad block, once with a fine block — and union the results:

linked_broad <- murmuration(cases_blocked, vax_blocked,
  blocking_var = "block1", ...)
linked_fine  <- murmuration(cases_blocked, vax_blocked,
  blocking_var = "block3", ...)

# Union, keeping the highest-scoring link per case
linked_all <- dplyr::bind_rows(linked_broad, linked_fine) |>
  dplyr::group_by(id_var.x) |>
  dplyr::slice_max(weights, n = 1, with_ties = FALSE) |>
  dplyr::ungroup()

Putting it together

library(starling)
data(cases_notifiable); data(vax_air)

# 1. Audit
preflight(cases_notifiable, vax_air,
          linkage_vars = c("lettername1", "lettername2", "dob", "medicare10"),
          medicare_col = "medicare10")

# 2. Fix Medicare
cases <- check_medicare(cases_notifiable)
cases$medicare10 <- ifelse(cases$medicare_valid == 1L,
                            cases$medicare10, NA_character_)

# 3. Block
cases <- flock(cases, block1_vars = "gender", birth_year_col = "dob")
vax   <- flock(vax_air, block1_vars = "gender", birth_year_col = "dob")

# 4. Link
linked <- murmuration(cases, vax,
  linkage_type    = "v2c",
  event_date      = "onset_date",
  id_var          = "id_var",
  blocking_var    = "block1",
  compare_vars    = c("lettername1", "lettername2", "dob", "medicare10"),
  threshold_value = 17)

Session information

sessionInfo()
#> R version 4.5.2 (2025-10-31 ucrt)
#> Platform: x86_64-w64-mingw32/x64
#> Running under: Windows 11 x64 (build 26100)
#> 
#> Matrix products: default
#>   LAPACK version 3.12.1
#> 
#> locale:
#> [1] LC_COLLATE=C                       LC_CTYPE=English_Australia.utf8   
#> [3] LC_MONETARY=English_Australia.utf8 LC_NUMERIC=C                      
#> [5] LC_TIME=English_Australia.utf8    
#> 
#> time zone: Australia/Brisbane
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] starling_1.1.0
#> 
#> loaded via a namespace (and not attached):
#>  [1] gtable_0.3.6        jsonlite_2.0.0      dplyr_1.2.0        
#>  [4] compiler_4.5.2      Rcpp_1.1.1          tidyselect_1.2.1   
#>  [7] stringr_1.6.0       parallel_4.5.2      jquerylib_0.1.4    
#> [10] scales_1.4.0        yaml_2.3.12         fastmap_1.2.0      
#> [13] ggplot2_4.0.3       R6_2.6.1            generics_0.1.4     
#> [16] knitr_1.51          tibble_3.3.1        lubridate_1.9.5    
#> [19] bslib_0.10.0        pillar_1.11.1       RColorBrewer_1.1-3 
#> [22] rlang_1.2.0         stringi_1.8.7       cachem_1.1.0       
#> [25] reclin2_0.6.0       xfun_0.56           sass_0.4.10        
#> [28] S7_0.2.1            otel_0.2.0          timechange_0.4.0   
#> [31] cli_3.6.5           magrittr_2.0.4      stringdist_0.9.17  
#> [34] digest_0.6.39       grid_4.5.2          rstudioapi_0.18.0  
#> [37] lifecycle_1.0.5     vctrs_0.7.1         lpSolve_5.6.23     
#> [40] data.table_1.18.2.1 evaluate_1.0.5      glue_1.8.0         
#> [43] farver_2.1.2        rmarkdown_2.31      tools_4.5.2        
#> [46] pkgconfig_2.0.3     htmltools_0.5.9