LocalLM Lab: A Playground for Apple's Foundation Models With Mac Connectors

[svgeek (Ben)]

While experimenting and trying to build an app with Apple's Foundation Models, I kept creating small Xcode projects just to answer basic questions:

• How does this prompt behave?
• Will the model produce the output I expect?
• What happens when the model can use real context from my Mac?

I built LocalLM Lab as a faster way to test those workflows before moving into application code.

The current v0.3.0 release adds connectors, so Apple's on-device model can use selected local context instead of only the prompt text. For example, it can read the system clock, list Calendar or Reminder items, search Contacts, read from a folder you choose, or answer weather questions.

Connectors are off by default and enabled individually. Most are local and read-only. Calendar and Reminders can create new items but cannot edit or delete existing ones. Weather is the exception that leaves the Mac, because it uses an external weather service.

LocalLM Lab also includes:

• Prompt Playground
• System prompts
• A local OpenAI-compatible API to build "big AI" apps without token cost

LocalLM Lab is free to download & use, but not open source.

Signed macOS releases: https://github.com/ancientcomputing/locallm-releases

Product page: https://thisbrain.ai/locallm

Connector details: https://thisbrain.ai/locallm/connectors.html

I'd appreciate feedback from other developers building with Apple Foundation Models. In particular:

• What local connectors would be useful for your workflows?
• How are you currently testing prompts and structured output?
• What tooling still feels missing?

[svgeek (Ben)]

A couple more screenshots here for the Connectors and OpenAI API settings. The inclusion of an API key here is meant to make testing against an OpenAI API endpoint more realistic.

[svgeek (Ben)]

Update on this: v0.4 adds an MCP client, so the on-device model can now connect to real Model Context Protocol servers instead of only the first-party connectors from v0.3.0.

Tested and documented against eight servers so far: DeepWiki and Context7 (no auth), GitHub (personal access token), Notion, Todoist, Linear, and the official MCP reference server (all OAuth, discovered automatically — no setup needed on the service side), and Slack (OAuth, but requires registering a small app first — Slack doesn't support the automatic flow the others do).

The interesting constraint, and the reason I wrote up a whole page on it: the on-device model has a fixed ~4096-token context window, shared across every enabled tool from every connected server. Linear alone exposes 50+ tools, Todoist 45 — enabling even a small, reasonable-looking selection can eat a third of the entire budget. So every newly connected server starts with all tools unchecked, and you enable only the one or two you actually need per task. It's a real limitation compared to what a cloud model can do with the same servers, but there's still a useful slice of MCP you can exercise fully on-device: doc lookups, "what's assigned to me," search, that kind of thing.

Framed differently, the budget constraint is also what makes this a decent way to actually learn MCP hands-on: the model's already sitting there, running locally, so there's no API bill, no sandbox to spin up, and no cloud account needed just to watch a real capability-negotiation/tool-discovery/tool-call exchange happen against a production server.

Full writeup, including exact per-tool token costs where I measured them, tool-by-tool setup steps, and example prompts for each server:

LocalLM Lab MCP Servers

If there's a server you'd want documented next, let me know — happy to add it if it's publicly reachable.

[svgeek (Ben)]

Update on this: v0.6 adds localai-cli, a way to call the on-device model from your own Swift (or Python) code without going through API Lab's HTTP server.

Interface is deliberately minimal: your app spawns localai-cli as a subprocess, writes one JSON object to stdin ({"system_prompt", "user_input", "connectors": [...], "mcp_tools": [...]} for --run, or an OpenAI-style messages array for --chat), and reads one JSON object back from stdout. Every call requires --config <path> explicitly with no implicit lookup that points at the same localai-config.json that LocalLM Lab's own Local AI Settings and MCP Servers panels write.

The part worth calling out here specifically: that config file is the capability ceiling, and localai-cli only ever reads it, never writes it. connectors_enabled in the file is what the user has granted through the app's UI; the connectors/mcp_tools fields in a given request are what that call is asking for. The rule is strictly requested & enabled. Ask for something outside that set and you get {"error": "..."} back before LanguageModelSession is ever touched. There's no partial grant. Omitting the field is a real "nothing," not a default-allow. Same enforcement model connectors/MCP already have inside the app itself, just now reachable from a subprocess call instead of only the app's own UI.

Ships as a matched pair: localai-cli plus localai-playground-run (the actual model-invocation helper it wraps). It needs LocalLM Lab running in the background, since connector/MCP calls relay through its process over a local socket rather than being self-contained in the binary.

Worth being explicit about where this sits: it's still an experimentation tool, not a shipping dependency. The permission/authentication model here works because there's a single trusted app (LocalLM Lab) mediating everything. We aren't where we can say "bundle this in an app you distribute" yet.

There's a fuller worked example now too in "Plate Today", which combines three connectors (clock, calendar, reminders) and one MCP tool call (Todoist's find-tasks-by-date) in a single request, mixing the two capability systems in one call rather than treating them as separate paths. It also does its own preflight: reads localai-config.json directly, checks all four sources are enabled before calling localai-cli at all, and prints exactly which panel/toggle to fix if not. That's client-side validation layered on top of localai-cli's own request-time rejection. This means that a misconfigured run fails immediately with an actionable message instead of a generic {"error": ...} after a wasted model invocation.

Full interface docs + runnable Swift and Python examples (including the --chat shape and config/error-handling edge cases): thisbrain.ai/locallm/cli.html

PS: unrelated to the CLI, but since this crowd will notice -> Prompt Playground's UI has also moved from a browser window to native SwiftUI in the v0.6 release.

[svgeek (Ben)]

Here's another update on LocalLM Lab. This is release for macOS 27 Golden Gate.

1.0.0-beta.1 adds what I've been calling the model layer: one API surface across Apple's on-device FoundationModels model, Claude (via a host-supplied API key) and locally-run open-weight models (Qwen/Deepseek/Gemma/...). Complicated stuff like routing and model residency is owned by the SDK instead of every app rolling its own provider abstraction.

The front door is LocalLMLab, an optional convenience object wiring a model registry, the MCP manager, and the connector/workspace facades together:

```swift

let lab = LocalLMLab(configuration: .init(providers: [

    SystemModelProvider(),

    ClaudeModelProvider(auth: .apiKey(myKey)),

    MLXModelProvider(residentModelLimit: 1),   // from LocalLMLabSDKInference

]))

lab.models.route(.heavy, to: ModelID("mlx:mlx-community/Qwen3-8B-4bit")!)

lab.models.route(.light, to: .system)

let session = try lab.makeSession(route: .heavy, tools: myTools, instructions: sys)

```

ModelProvider is a protocol keyed by scheme : SystemModelProvider (system), ClaudeModelProvider (claude), MLXModelProvider (mlx, the separate LocalLMLabSDKInference binary), and PCCModelProvider (pcc, Apple's Private Cloud Compute). PCCModelProvider ships but **isn't functional in `1.0.0-beta.1`** — a session routed to .pcc fails outright. Check out Private Cloud Compute - Apple Developer for the details.

RouteName is a name your app maps to a ModelID (.heavy, .light, .draft, or any string). The SDK owns residency but never the routing policy: which route (aka model) a task uses is entirely your call as a developer.

The MLXModelProvider lifecycle is the part most worth detailing, since it's new: validate(_:) preflights a repo before pulling any weights. Preflight includes: reachability, MLX format, architecture support and weight size against MLXPreflightLimits.maxWeightFractionOfRAM (default 0.7 of physical RAM). download(_:) streams progress. capabilityProbe(_:) runs a real prompt plus a trivial tool call after download. This is the authoritative signal for whether a given model can reliably tool-call, not something to assume from its name. residencyEventStream emits .warmed/.evicted(reason:)/.loadProgress for a status line. residentModelLimit caps how many models stay resident in RAM at once; switching routes evicts the other.

Weights land in a standard Hugging Face cache ~/.cache/huggingface/hub/ for a bare CLI, redirected automatically to ~/Library/Containers/<bundle-id>/Data/Library/Caches/huggingface/hub/ under App Sandbox. A sandboxed app also needs com.apple.security.network.client for the download itself. This is the same silently-hanging-without-it entitlement gotcha called out in the SDK launch.

makeSession(route:tools:instructions:includeMCPTools:) resolves the route to a model and merges your tools with the enabled MCP session tools automatically (opt out with includeMCPTools: false). session.events carries the side-channel Apple's streaming doesn't give you (.toolCallStarted/.toolCallFinished/.contextCompacted/.modelLoadProgress) and session.contextBudget plus an optional retryOnContextOverflow compact-hook retry are there for sessions long enough to fill a context window.

Two new reference apps show this end to end: code-buddy, a CLI coding agent with .heavy/.light MLX routes, Core's Workspace tools, a live MCP docs server, and streamed output; and repo-qa-local, the minimal version of the same idea. workspace-buddy-local is the sandboxed-MLX filesystem example. There are 8 reference apps in the repo now. All for your reviewing pleasure!

Requires macOS 27 beta, Apple Intelligence enabled, Swift 6 tools, Xcode 27 beta for SDK development.

SDK guide: locallm/docs/sdk-guide.md at main · ancientcomputing/locallm · GitHub

Feature page: thisbrain.ai/locallm/1.0.0-beta

[svgeek (Ben)]

Continuing from the model-layer topic from a couple weeks back, LocalLM Lab 1.0.0-beta.3 extends the model layer to online providers.

A new RemoteModelProvider ships in a fourth binary, LocalLMLabSDKRemote (pure URLSession, no third-party dependencies). It's configured through RemoteProviderConfig: scheme, dialect (.openAIChat / .openAIResponses / .anthropicMessages / .openAICompatible), baseURL, auth, models — with presets for .openAI, .openAIResponses, .anthropic, .openRouter, and .openAICompatible for anything else speaking the OpenAI wire format (LM Studio, vLLM, a self-hosted endpoint).

Routing and session creation are identical to local/on-device routes:

lab.models.route("chat", to: ModelID(scheme: "openai", rest: "gpt-...")!)
let answer = try await lab.makeSession(route: "chat").respond(to: prompt)

Providers can be swapped at runtime without rebuilding LocalLMLab — ModelRegistry.replace(_:) / .removeProvider(scheme:) — useful if you're letting a user reconfigure their own API keys/endpoints.

Online providers also get first-class web search: capabilities.insert(.webSearch) plus defaultOptions.webSearch = true turns it on per-provider, or SessionOptions(webSearch: true) per turn. Results come back through the existing event/citation surface — session.events yields .serverToolCall with the queries, session.citations carries the sources.

One platform-gating detail: LocalLMLabSDKRemote's manifest floor is macOS 26, so linking it doesn't force a macOS 27 deployment target — but RemoteModelProvider itself is @available(macOS 27) and reports .requiresOS("macOS 27") when run on 26. That lets you ship one build that offers online models where the OS supports it and falls back to on-device everywhere else.

If you're building provider-configuration UI, Components doesn't link Remote at all — it calls back through onSave/onRemove/onTest closures with plain RemoteProviderDraft values, so you're not forced to pull in the Remote binary just to let a user type in an API key.

New reference app: model-switch — one chat interface that switches between OpenAI, Anthropic, OpenRouter, a custom OpenAI-compatible server, and Apple's on-device model mid-conversation, with a web search toggle and citations in the transcript. The provider-config glue is small, just 30 lines in ProviderGlue.swift converting a RemoteProviderDraft into a RemoteProviderConfig. It ships with a ModelSwitch.xcodeproj, which is also new for this release: all 11 SwiftUI reference apps now come with their .xcodeproj instead of needing to be built from the command line first (CLI-only examples still ship as Package.swift).

Why bother with online providers at all if you're mainly interested in local/on-device: two reasons beyond raw capability. First, it's a real cost/performance dial where you can route routine turns to a fast local or on-device model and escalate to a frontier model only for the hard cases, without three separate integrations to maintain. Second, it's a useful debugging framework: point a route at a frontier model to confirm your tool definitions and app framework are correct, independent of whether a smaller local or on-device model is just failing to follow your prompt or call tools correctly.

Requires macOS 27 beta on Apple Silicon for online providers, ClaudeForFoundationModels, PCC, and local open-weight models.

SDK guide: locallm/docs/sdk-guide.md at 1.0.0-beta · ancientcomputing/locallm · GitHub
Feature page: thisbrain.ai/locallm/1.0.0-beta

[John_McCall]

You're welcome to provide these updates, but please make them in the original thread for your project. I've gone ahead and moved the last couple back here for you.

[svgeek (Ben)]

Sounds good. I'll plan for a single thread and go from there. Will add today's update to this thread.

Thanks,

Ben

[svgeek (Ben)]

1.0.0-beta.4 replaces the old all-or-nothing tool permission model and brings the MCP client up to the current protocol revision.

Tool authority The old "Full Access" toggle is gone. In its place: ToolCallAuthorizer, a new invocation gate with two levers: graded selection via limited(toMaxImpact:), and per-call authorization. MCP-originated tools can now be tagged (OriginTaggedTool), so a policy can treat them differently from first-party tools.

MCP client to 2025-11-25 The client now negotiates protocol version at initialize and returns structured tool results, MCPToolResult with outputSchema, title, and nextCursor pagination, instead of a bare string. Multi-event responses stream with elicitation handler seams. There's a default elicitation UI (URL-mode, suppressible if you want to own presentation yourself), and OAuth support is now full: CIMD, OIDC discovery, scope step-up, issuer binding. Diagnostics got a real home too: os.Logger throughout, an opt-in MCPDiagnostics buffer with a single logLevel knob for prod vs. dev, tool-call args/isError logging, and the negotiated protocol version surfaced in exportSummary.

VistaNova is a new reference app (open source), and the thing that exercises the MCP client upgrade end to end. It's a web-search tool built on two different models doing two different jobs: Apple's on-device Foundation Model handles the initial search query and the MCP tool call out to Tavily. A downloadable Qwen model (from Hugging Face, via MLX) can summarize what comes back too, if you don't want to dig through the results yourself, though the search results themselves are still there. Wiring two very different models into different steps of one task, through the same session API, is exactly the model-layer story from beta.3. This is what it looks like in a real app.

Also landing: MCP response size limits and transport hardening, Keychain fixes for stored credentials, and applyPatch now rejects no-op hunks instead of silently reporting success.

One more thing: the next protocol revision (`2026-07-28`) is already on our roadmap. If you're building an MCP server based on that revision right now, we'd like to hear from you. Testing against a real compliant server beats coding support blind against a spec draft.

VistaNova is at ancientcomputing/vistanova

[svgeek (Ben)]

Update on the SDK: LocalLM Lab 1.0.0 is out. My last update here was beta.4, so this also covers the release candidate that came in between.

The headline is a source-compatibility promise, now in force. Code that builds against 1.0.0 keeps building unmodified against every 1.x release. New capability arrives as additive surface only: new optional parameters with defaults, new protocol requirements only with a default implementation and new cases on the non-frozen enums (build with @unknown default). A change that would force an edit to working code just to keep building waits for 2.0. This is a promise about source, not binary: you pin an exact version and rebuild. I regenerate the public API surface for every release and diff it, so additions are the only lines allowed to appear.

The betas made no such promise, and if you built against beta.4 a few things will need an edit. They are listed in the CHANGELOG. The main ones: SessionEvent.toolCallStarted and .toolCallFinished carry the call's arguments and a result summary, InstalledModel.init requires resolvedRevision, DownloadableModelProvider gains a required cancelDownload(_:) and RemoteModelProvider.init(_:) now throws.

The other big piece is what happens when your app downloads an AI model at runtime. A Hugging Face repo is not a fixed artifact. The owner can change the files behind the same name, look-alike repos exist and downloads can be corrupted or oversized. A shipped pin (MLXModelProvider(pinnedRevisions:)) makes the model you validated the one users download on first run, and it never falls back to main if that commit can't be fetched. MLXSupplyChainPolicy adds a host-supplied MLXModelTrustPolicy checked before any network call, content-hash verification of every downloaded file (on by default) and an MLXCacheLimits cap. Updates are deliberate and all-or-nothing (checkPinUpdate, updatePin, updatePins), rollback is updatePin(to: oldCommit) and snapshots(for:) and removeSnapshot handle cleanup. Adapters and draft models follow the same policy, and if one is denied the run fails instead of quietly using the plain model.

SessionOptions temperature, max output tokens, topK/topP and seed also take effect on Apple's on-device model now (they were ignored in the betas). One behavior change: topK together with topP, or a seed with neither, throws SessionOptionsError where it used to be ignored. A seed is best-effort on this model.

Two examples ship in the repo. mlx-control-room exposes the MLX tuning knobs, including a speed helper and LoRA adapter pairing, with a gauge beside each one so you can see it change something. VistaNova, previously its own repo, is now an example in the SDK repo: Apple's on-device model runs a search through an MCP server and a downloaded Qwen model summarizes the results, with the summary model shipped pinned.

What comes next is open for suggestions. I am working on MCP client support for protocol revision 2026-07-28 (the SDK currently targets the 2025-11-25 protocol). Another idea on the list is benchmark metrics for people who want to measure how the larger open-weight models perform on the new Mac Studio: tokens per second, time to first token, memory and what a speed helper buys on a big model. If either would be useful to you, or you would rather see something else, reply here.

Building the SDK needs Xcode 27, and open-weight models and hosted providers need macOS 27. Happy to share links if useful.