Hi everyone,
While trying Xcode 27’s Evaluations framework, I reached the point where ModelJudgeEvaluator was producing scores, but I had no way to answer the question of should I trust those scores?
Apple’s Book Tracker calibration example compares a model judge with expert ratings using Cohen’s kappa rather than plain agreement. I wanted that part of the workflow to be reusable instead of another project-local statistics helper.
I built JudgeCalibrationKit for that.
It provides:
- Cohen’s κ, weighted κ, Krippendorff’s α, Spearman correlation, confusion matrices, binary diagnostics, and error measures.
- Deterministic clustered-bootstrap confidence intervals and CI gates.
- Explicit abstention and coverage handling.
- Per-human comparisons and a human-panel baseline rather than one averaged “agreement” score.
- Metadata slice analysis and worst-slice gates, so good aggregate agreement can’t conceal a failing locale or task segment.
- Swift Testing integration and a command-line interface for other CI systems.
Two design choices matter particularly to me.
First, undefined statistics return a typed .unavailable(reason) result rather than silently becoming zero or NaN. Second, a worst-slice confidence interval is calculated for the minimum across all eligible slices, and not only for whichever slice happened to look worst in the observed sample.
The statistics core has no third-party dependencies, imports no Apple framework, and builds on Linux. Optional bridges support Apple Evaluations and Rudrank Riyam’s XCEvalFormat; version 0.3.0 now imports xceval 0.4 report documents and preserves its canonical artifact identity.
Requirements are Swift 6.2, macOS 14, and iOS 16. The Evaluations adapter targets a beta API, although it is compile- and simulator-runtime-verified against Xcode 27 beta 4. The package is Apache-2.0.
I’d especially value feedback from anyone already calibrating LLM judges: does the dataset, pairing, and gate model fit how you store human ratings and make release decisions?