Metric Abstract Class

Abstract base class for defining simulation metrics in bayesim. Metrics compute summary statistics or diagnostic values from fitted model results and data bundles. Metric is abstract: it has no direct instances, so calling Metric() errors. Subclasses are created with S7::new_class(..., parent = Metric) and instantiated directly. Because the parent is abstract, S7 honors the defaults a subclass declares for the inherited name/needs/required/summary_type/schema properties when the subclass constructor is called.

Usage

Metric(
  name = character(0),
  needs = character(0),
  required = FALSE,
  summary_type = "mean",
  schema = list()
)

Arguments

name

Character identifier for the metric. Used as a prefix when flattening metric output to column names.

needs

Character vector of required capabilities from the fitter. Common values include "predictions", "log_lik", "loo", "epred". The metric will only receive these values in the context if the fitter provides them. epred is delivered as context$loo_epred, a draws x observations matrix computed on the training set; it is delivered whether or not "loo" is also declared — without "loo" the matrix is built directly instead of through the LOO context. Declaring "epred" also delivers the PSIS weighted-prediction machinery (context$loo_psis, context$loo_psis_ll); declaring "loo" alone delivers only the loo_fit() elpd/p_loo/pareto_k summary (context$loo), so a metric reading the PSIS objects must declare "epred" too.

required

Logical indicating whether metric failure causes task failure. If TRUE, an error in computing this metric will propagate and fail the entire task. If FALSE (default), metric failure results in NA values being recorded.

summary_type

Character; how summarize_simulation() aggregates this metric's flattened columns: "mean" (default, sd/sqrt(n) MCSE), "proportion" (coverage-style sqrt(p(1-p)/n) MCSE), or "none".

schema

Named list of field-level metadata. Each emitted field can declare a role (estimate, binary, count, diagnostic, rank, or artifact), an aggregation (mean, proportion, or none), an MCSE method (sd, binomial, or none), and optional nominal, units, or dimension metadata. summary_type remains supported as a compatibility default for metrics that do not declare a schema.

Value

An S7 class object representing the abstract Metric base class. Construct subclasses directly (e.g. MyMetric()); do not call Metric().

Methods

The compute_metric() S7 generic must be implemented by subclasses.

compute_metric(metric, fit_result, data_bundle, context, task_ctx)

Compute metric values from a fitted model result. This method must be implemented by subclasses.

\itemize{
  \item metric: The Metric S7 object
  \item fit_result: A bayesim_fit_result object containing the fitted
    model output (draws, diagnostics, etc.)
  \item data_bundle: A list containing data-related objects including
    train (training data), test (test data if applicable), response
    (response variable), true_params (true parameter values if known)
  \item context: A list with precomputed values based on the metric's
    `needs` property. May include predictions, log_lik (log-likelihood
    values), loo (leave-one-out cross-validation results), etc.
  \item task_ctx: A list with task identification information including
    task_id, data_idx, fit_idx, rep_idx for tracking and debugging.
}

Returns: A named list with metric values. Names must be non-empty strings.
Values must be one of:
\itemize{
  \item scalar atomic (logical, integer, double, character)
  \item named numeric vector
}

No nested data frames or matrices are allowed in the output.

Metric Output Schema

The compute method must return a named list conforming to the following schema:

• All elements must have non-empty names

• Values must be scalar atomic types or named numeric vectors

• No nested data frames or matrices allowed

• The engine flattens output with prefix <metric_name>__<field>

Examples

# Define a custom RMSE metric
RMSEMetric <- S7::new_class(
  "RMSEMetric",
  parent = Metric,
  properties = list(
    name = S7::new_property(S7::class_character, default = "rmse"),
    needs = S7::new_property(S7::class_character, default = "predictions"),
    required = S7::new_property(S7::class_logical, default = FALSE)
  )
)

S7::method(compute_metric, RMSEMetric) <- function(
  metric, fit_result, data_bundle, context, task_ctx
) {
  preds <- context$predictions$predicted_mean
  actual <- data_bundle$test[[data_bundle$response]]
  list(
    value = sqrt(mean((preds - actual)^2)),
    n_obs = length(actual)
  )
}

# Construct the subclass directly; the declared defaults are honored.
RMSEMetric()@name     # "rmse"
#> [1] "rmse"
RMSEMetric()@needs    # "predictions"
#> [1] "predictions"
RMSEMetric()@required # FALSE
#> [1] FALSE