--- title: "How many evaluators, prompts, and runs?" output: rmarkdown::html_vignette: toc: true vignette: > %\VignetteIndexEntry{How many evaluators, prompts, and runs?} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE, purl = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") options(width = 100) ``` A complete **Gtheory4LLM** workflow for the reliability of mean continuous subjective-quality judgments. > **All data below are synthetic.** This tutorial uses no manuscript results, > source-study data, API calls, or real LLM outputs. Its results demonstrate the > software workflow; they do not establish an adequate evaluator or prompt count > for a real task. ```{r library} # This vignette is also installed as a runnable script: # source(system.file("doc", "LLM-workflow.R", package = "Gtheory4LLM")) library(Gtheory4LLM) ``` ## 1. Define the quantity and the sampling design The quantity is an item's mean continuous quality score over evaluators, prompts, and repeated runs. We generate a complete panel of 24 items, four evaluators, three prompts, and two runs: 576 measurements. The model assumes exchangeable random effects for those populations and a Gaussian score model. The scores are simulated continuous numbers, not numeric codes assigned to ordered categories. | Facet | Meaning in this example | |---|---| | Item | The object of measurement. Stable differences between items form the universe-score variance. | | Evaluator and prompt | Both have main effects and direct item interactions. Different items can respond differently to these facets. | | Run | Scoped to an evaluator/prompt pair. A run group labels all items. The run source is a common shift across items; remaining item-specific run variation belongs to the residual in this simulation. | ```{r design} set.seed(20260912) llm_data <- expand.grid(item = factor(seq_len(24)), evaluator = factor(paste0("model_", seq_len(4))), prompt = factor(paste0("prompt_", seq_len(3))), run = factor(seq_len(2))) llm_formula <- quality ~ item * (evaluator + prompt) + evaluator:prompt:run llm_formula ``` This declares six random sources: item, evaluator, prompt, item-by-evaluator, item-by-prompt, and evaluator-by-prompt-by-run. The run group is parent-scoped even though the coded panel is complete. Item-by-run and additional evaluator-by-prompt effects are not silently added. Omitting them is an assumption of this known simulation; a real study needs its own justification. The exact Gaussian engine currently requires a complete coded Cartesian panel with one row per full cell. The formula names the outcome on the left and random sources on the right. We also supply `measured = "item"` so the object of measurement does not depend on term order. Use `gt_design()` when a facet appears in no random-source term, or when declaring within-cell replication or batches; those declarations have no formula syntax. The study-planning vignette illustrates these cases. ```{r simulate} source_sd <- c(item = 1.2, evaluator = 0.4, prompt = 0.3, "item:evaluator" = 0.5, "item:prompt" = 0.4, "evaluator:prompt:run" = 0.25) llm_data$quality <- 0 for (source in names(source_sd)) { members <- strsplit(source, ":", fixed = TRUE)[[1L]] group <- do.call(interaction, c(llm_data[members], list(drop = TRUE))) effects <- rnorm(nlevels(group), sd = source_sd[[source]]) llm_data$quality <- llm_data$quality + effects[as.integer(group)] } llm_data$quality <- llm_data$quality + rnorm(nrow(llm_data), sd = 0.6) ``` ## 2. Inspect feasibility before fitting `gt_preflight()` reports observed counts, source dimensions, free parameters, configured size limits, and the supported reliability scale, without building a model matrix or running an optimizer. ```{r preflight} preflight <- gt_preflight(llm_formula, llm_data, measured = "item") print(preflight) preflight$sources stopifnot(preflight$fitting_feasible) ``` Passing this report means that the listed structural and resource checks pass. It does not prove identification, reliable estimation, numerical acceptance, or approximation adequacy. Read failed checks before changing a resource limit or simplifying a scientific model. The panel audit counts missing cells and cells above or below the declared replication. It retains only a bounded set of examples, using the observed coded levels; it cannot discover a level that never appeared in the data. ```{r panel-audit, fig.width=7, fig.height=4} preflight$panel_audit$summary plot(preflight, type = "cells") incomplete <- gt_preflight(llm_formula, llm_data[-1, ], measured = "item") incomplete$panel_audit$missing_cells ``` These diagnostics do not fill cells or remove records. Investigate unexpected codes or missing measurements before deciding how the study should be analysed. ## 3. Fit and inspect acceptance ```{r fit} fit <- gt_fit(llm_formula, llm_data, measured = "item", control = gt_control(gaussian = list(optimizer = "CSOLNP", extra_tries = 2L, retry_seed = 415L, threads = 1L))) print(summary(fit)) diagnostics <- gt_diagnostics(fit) print(diagnostics[c("numerically_accepted", "selected_attempt", "acceptance_failures", "approximation_adequacy")]) stopifnot(isTRUE(diagnostics$numerically_accepted)) gt_components(fit) ``` The Gaussian default estimator is REML. The trial budget allows the initial fit plus two additional attempts; it is not resampling. The chosen optimizer remains fixed across attempts. Inspect numerical acceptance and source-boundary diagnostics before interpreting the components. A completed optimizer alone is insufficient. If an installation does not support the selected optimizer or a fit is rejected, address that explicitly. The summary table also carries an asymptotic standard error for each source variance, obtained from the numerically differentiated restricted-likelihood Hessian. A component resting on a variance boundary is flagged, because Wald theory does not apply there. The simulation seed and Gaussian retry seed serve different purposes. Keep both, together with the data order, exact package archive, controls, R and dependency versions, numerical libraries and thread count when saving a result. First replay the request in the same environment. Cross-platform replication uses predeclared tolerances on unrounded estimates and also compares numerical acceptance and interval availability; a fixed seed does not promise identical floating-point optimization. A changed decision or a failed tolerance check must remain visible rather than being removed by rounding or a relaxed bound. ```{r variances} summary(fit)$variances ``` ## 4. Compute relative and absolute reliability ```{r reliability} reliability <- gt_reliability(fit) reliability reliability_table <- as.data.frame(reliability) subset(reliability_table, kind == "outcome") reliability$interpretation ``` `Erho2` is the relative coefficient G: common evaluator, prompt, and run shifts do not change relative item comparisons. `Phi` is the absolute coefficient and also includes those shifts as error. Item-dependent interactions contribute to both error definitions. Here both coefficients refer to observed Gaussian-score averages under the declared design. They measure consistency under that model, not accuracy against human reference labels. A coefficient of 0.80 means that 80% of the variance in item means is universe-score variance under this model; it is not 80% labeling accuracy. The interval is a delta-method Wald interval on the logit scale, propagating the fitted parameter covariance matrix through the source covariances. It describes estimation uncertainty in those covariances under the declared model. It is not a guarantee about a future sample of evaluators or prompts. ```{r reliability-figure, fig.width=8, fig.height=4.5} plot(reliability, target = 0.80, main = "Reliability of the declared mean quality score") ``` The dashed reference is an illustrative target. Missing intervals are shown as unavailable, never as a claim of perfect precision. The table records the scale, uncertainty status and any conditional inference, alongside the estimates. ## 5. Compare measurement budgets ```{r dstudy} grid <- expand.grid(evaluator = c(2L, 4L, 6L), prompt = c(2L, 3L, 4L), run = c(1L, 2L)) dstudy <- gt_dstudy(fit, grid) planning <- as.data.frame(dstudy) # Prefixed allocation columns avoid collisions with outcome/estimate names. # These readable aliases are just for this tutorial's known facet names. planning$evaluator <- planning$allocation_evaluator planning$prompt <- planning$allocation_prompt planning$run <- planning$allocation_run planning$measurements_per_item <- planning$measurements_per_object planning <- planning[order(planning$measurements_per_item, -planning$Phi), ] rownames(planning) <- NULL print(planning) ``` Runs are counted within each evaluator/prompt pair, so the budget per item is evaluators x prompts x runs. A row using more levels than observed extrapolates the fitted variance model to the declared exchangeable population. ```{r threshold} screen <- gt_dstudy_target(dstudy, target = 0.80, coefficient = "Phi", outcome = "quality") screen[screen$meets_target %in% TRUE, ] screen[screen$fewest_measurements %in% TRUE, ] ``` A threshold of 0.80 is illustrative; choose and justify your own criterion. Read the interval columns alongside the point projection: several allocations here have overlapping intervals, so the ordering of nearby rows is not established by these data. The screening helper keeps every supplied candidate, including those that miss the target or have an unavailable estimate. It marks all ties for the fewest measurements among qualifying candidates. It performs no search outside this grid and does not guarantee that a future panel will meet the target. ```{r dstudy-figure, fig.width=7, fig.height=5} plot(dstudy, coefficient = "Phi", outcome = "quality", target = 0.80, main = "Candidate measurement allocations") ``` Different allocations can use the same total number of measurements; they are separate points rather than one assumed smooth curve. The figure identifies allocations extending beyond the observed facet counts. Such projections keep the fitted source distributions fixed. Both result tables contain ordinary scalar columns, suitable for CSV output: ```{r export, eval=FALSE, purl=FALSE} write.csv(reliability_table, "reliability.csv", row.names = FALSE) write.csv(as.data.frame(dstudy), "decision-study.csv", row.names = FALSE) pdf("decision-study.pdf", width = 7, height = 5) plot(dstudy, coefficient = "Phi", outcome = "quality", target = 0.80) dev.off() ``` ## 6. Save a portable analysis report A report collects the design, fitted model context, diagnostic flags, coefficient and decision-study tables, and figures. It does not rerun optimization. Supplied coefficient results are checked against projections from the fit; supplied preflight information is checked for compatible aggregate specifications, which does not prove that it came from identical observations. ```{r analysis-report} analysis_report <- gt_report(fit, preflight = preflight, reliability = reliability, dstudy = dstudy) print(analysis_report) ``` ```{r save-analysis-report, eval=FALSE, purl=FALSE} gt_export_report(analysis_report, "analysis-report.html") ``` The HTML file contains its figures and styling and can be opened offline. An existing destination is refused unless replacement is explicitly requested. The report omits observations, row examples and group-level identifiers, while keeping variable and source names and aggregate results. This is not an anonymization guarantee. Inspect study-specific names before sharing it. Retained fitting provenance and the report-generation environment are separate; missing fitting provenance is reported as unavailable. ## 7. Fixed facets A facet is *random* when the study generalizes to a population of its levels, and *fixed* when the universe of generalization is exactly the levels used. LLM evaluation may target exactly a chosen temperature or prompt set, in which case those facets are fixed. Selecting levels deliberately does not by itself decide the universe of generalization. Declare them with `fixed`. Following the mixed model of Brennan (2001), the object-by-fixed-facet variance is averaged over that facet's levels and added to universe-score variance, and a source built only from fixed facets shifts every item equally and leaves the model. ```{r fixed} fixed_prompt <- gt_reliability(fit, fixed = "prompt") fixed_prompt fixed_prompt$source_roles ``` Treating the prompt set as fixed raises both coefficients, because item-by-prompt variation is now part of what the study is trying to measure rather than error. A fixed facet's count cannot be changed, and a decision study may not project over it. `fixed` acts at the reliability stage: it changes how the already-fitted components are aggregated into a coefficient. It does not change the G study, so it cannot repair a source whose variance was misspecified during fitting. Observed seed disagreement can change across temperatures even when latent variance is constant, because category probabilities can change. If diagnostics raise doubts about pooling temperatures, fitting within one temperature is a useful sensitivity analysis for a conditional estimand; declaring a facet fixed is a separate decision about generalization. ## 8. Match other outcomes to their observation model | Outcome | Current scalar reliability support | |---|---| | Gaussian | Observed continuous-score averages. | | Binary or ordinal | Explicit `scale = "latent"` after an accepted fit. This describes latent-response averages, not majority votes, label proportions, or observed ordinal-score averages. | | Unordered categorical | Explicit `scale = "latent"` with `covariance = "exchangeable"` or `"unstructured"`: the mean of the generalized eigenvalues over the category contrasts, with their range, invariant to the reference category. Category-vector and plurality-decision targets come from `gt_categorical_target()`. | Declare category order and references explicitly. Do not recode binary, ordinal, or nominal labels as the continuous quality score used above. Discrete fitting uses a first-order Laplace engine with Gaussian latent random effects, with dense algebra by default and sparse algebra on request through `gt_control(discrete = list(engine = "sparse"))`; `gt_preflight()` reports the selected engine's limits, and changing the link does not remove the random-effects distribution assumption. Discrete fits report point estimates unless `check_hessian = TRUE` requests observed information. A boundary source can be held at its fitted covariance matrix to calculate conditional intervals from the remaining coordinates, if the information checks pass. Read the named conditioning sources in the result: these intervals exclude their estimation and boundary-selection uncertainty and retain the Laplace assumption. Analytic reliability and D studies require complete balanced coded panels even when discrete fitting admits missing whole cells; observed-score, category-vector and decision targets are separate simulated targets from `gt_categorical_target()`. For real native data, a separate installed workflow declares the two specific whole-data requests used in the version 1.0 project records: two ordinal drug-review outcomes (efficacy and safety), and the three binary mental-health groups. Both retain 21,600 rows and ten declared sources. The records report numerical acceptance on macOS and Linux, with sparse analytic-gradient fitting and projected boundary escape. Those runs lacked ordinary information at singular sources. A later macOS analysis kept the accepted parameter estimates unchanged and calculated conditional intervals with the identified source matrices held fixed. These were nonzero singular matrices, not discarded sources. The point fits come from development source 2fc771b, not from a refit of the 1.0.0 archive, and no Linux conditional-information or repeated-sampling coverage result is asserted. The all-four-aspect and six-flag models remain outside this two-model example. These long fits are not run while building this synthetic vignette. For a full run, set `OMP_NUM_THREADS`, `OPENBLAS_NUM_THREADS`, `MKL_NUM_THREADS` and `VECLIB_MAXIMUM_THREADS` to `1` in the environment that launches R, before R starts. Full mode refuses missing or different values. Setting them inside an existing R session does not reliably change an already initialized numerical library. Install the package archive into an isolated library and supply that same archive below. Sourcing the installed script defines a helper without starting the fits: ```{r native-workflow, eval=FALSE, purl=FALSE} source(system.file("examples", "native_1_0_workflow.R", package = "Gtheory4LLM", mustWork = TRUE)) gt_native_workflow(mode = "full", output_dir = "native-1.0-results", archive_path = "Gtheory4LLM_1.0.0.tar.gz") ``` The helper's `mode = "smoke"` uses a small synthetic fixture to check the workflow, not to reproduce a native fit. Its full mode retains the specified requests and records numerical rejection, boundary conditioning and any missing intervals. Its comparison policy separates the historical point-fit references from the saved macOS conditional arithmetic. Passing a prospective interval comparison is not a coverage study. Follow the [native workflow guide](https://github.com/Veronica0206/Gtheory4LLM/blob/main/docs/REAL_DATA_WORKFLOW.md) for the precise source models and result interpretation. At one observation per object-by-facets cell, a discrete fit cannot identify the full-cell source that the default design requests. Declare the same design without it: ```{r discrete-design} gt_design("item", "rater", full_cell = FALSE)$terms_requested ``` A complete panel can still contain little observed outcome variation. This small, deliberately imbalanced example has one positive response. It is a data profile, not an estimation example or a recommended design. ```{r outcome-information, fig.width=7, fig.height=4} information_data <- expand.grid(item = seq_len(6), rater = seq_len(3)) information_data$label <- as.integer(information_data$item == 1 & information_data$rater == 1) information_design <- gt_design("item", "rater", random = ~ item + rater) information <- gt_preflight(information_data, "label", information_design, gt_family("binary"), max_examples = 0) information$outcome_profile$label$categories information$outcome_profile$label$by_variable plot(information, type = "outcomes", outcome = "label") ``` The heatmap describes the fraction of groups in which each category appears. A group with one observation necessarily has no observed variation. These summaries supply no minimum-count threshold, independence claim or guarantee of parameter recovery. Declared categories absent from the entire panel are shown with zero counts and block fitting; the preflight report does not merge them or silently change their order. No finite panel can reveal a category never observed and never declared. ## Where to read next Installed help: `?gt_design`, `?gt_family`, `?gt_preflight`, `?gt_control`, `?gt_fit`, `?gt_diagnostics`, `?gt_reliability`, `?gt_dstudy`, and `?gt_report`. Brennan, R. L. (2001). *Generalizability Theory*. Springer.