DProvenanceKit: Reasoning observability for AI systems in Swift

Community Showcase
1

Hi everyone,

I've been working on a problem that I think many of us building AI in Swift are facing: how do you debug why an AI agent behaves differently between runs?

Traditional logging tells you what happened. But when a model skips a step, changes its reasoning order, or produces a different output with identical input — you're left staring at walls of logs with no clear answer to why.

I've released DProvenanceKit — a reasoning observability framework for Swift that lets you:

  • Record every reasoning step an agent takes (non-blocking, async-safe)
  • Query for reasoning patterns ("find runs where X happened but Y didn't")
  • Diff two executions to see structural differences
  • Detect regressions automatically with rule-based validation

Think of it as Git for AI logic.

Example:

Swift

// Record an execution
try await DProvenanceKit<MyAIDecision>.run(contextID: "case-123", store: store) {
    DProvenanceKit<MyAIDecision>.record(.documentEvaluated(documentID: "DocA", score: 0.95))
    DProvenanceKit<MyAIDecision>.record(.conflictDetected(reason: "timeline_inconsistency"))
    DProvenanceKit<MyAIDecision>.record(.finalDecisionMade(approved: false))
}

// Query for suspicious patterns
let suspiciousRuns = try await store.queryRuns(
    TraceQueryDSL<MyAIDecision>()
        .requiring(step: "conflictDetected")
        .missing(step: "documentEvaluated")  // Find runs where conflict was reported but no docs evaluated
)

// Diff two runs
let diff = engine.diff(base: runA, comparison: runB)
print(diff.changes)  // See exactly which steps appeared, disappeared, or moved

The design:

  • Built specifically for on-device AI (macOS/iOS) with Apple Foundation Models, MLX, or Core ML
  • Non-blocking recording (touches only in-memory buffer)
  • Durable, crash-safe persistence with SQLite WAL
  • Works with async/await context propagation

Status:

Experimental (core engine complete, actively evolving). Free for development/testing under BSL 1.1.

GitHub: GitHub - Therealdk8890/DProvenanceKit: Reasoning observability and regression testing for AI systems · GitHub

I'm curious if this resonates with anyone here building AI in Swift. What debugging/observability challenges are you facing with AI systems?

4 Likes
2

I don't think you meant to post in the category for discussing development of the Swift compiler. I've moved the post to "Community Showcase."

1 Like
3

Quick update on DProvenanceKit since the original post: I added a small Foundation Models regression demo around a quiet failure mode I keep worrying about. Nothing crashes, the answer still looks plausible, but after a model/prompt/OS change the agent silently stops doing an important step.

The demo:

swift run FoundationModelsRegressionDemo --gate

It compares a baseline trace where a weather agent calls getWeather against a candidate trace where the agent answers directly. The output shape still looks fine, but the reasoning trace lost the tool call and tool output. DProvenanceKit marks that as a high-risk regression and exits non-zero in gate mode.

What I verified locally on 2026-07-09:

  • swift test -> 317 tests, 5 skipped, 0 failures
  • DProvenanceKitCLI evaluate --gate -> 13/13 local corpus cases passed
  • ConformanceHarness -> all Trace Specification v1 vectors reproduced

Repo:

Walkthrough:

I would love feedback from Swift / Foundation Models / MLX builders: are you currently testing only outputs, or are you also tracking reasoning/tool-call path changes between runs?