LocalLM Lab: A Playground for Apple's Foundation Models With Mac Connectors

[svgeek (Ben)]

While experimenting and trying to build an app with Apple's Foundation Models, I kept creating small Xcode projects just to answer basic questions:

• How does this prompt behave?
• Will the model produce the output I expect?
• What happens when the model can use real context from my Mac?

I built LocalLM Lab as a faster way to test those workflows before moving into application code.

The current v0.3.0 release adds connectors, so Apple's on-device model can use selected local context instead of only the prompt text. For example, it can read the system clock, list Calendar or Reminder items, search Contacts, read from a folder you choose, or answer weather questions.

Connectors are off by default and enabled individually. Most are local and read-only. Calendar and Reminders can create new items but cannot edit or delete existing ones. Weather is the exception that leaves the Mac, because it uses an external weather service.

LocalLM Lab also includes:

• Prompt Playground
• System prompts
• A local OpenAI-compatible API to build "big AI" apps without token cost

LocalLM Lab is free to download & use, but not open source.

Signed macOS releases: https://github.com/ancientcomputing/locallm-releases

Product page: https://thisbrain.ai/locallm

Connector details: https://thisbrain.ai/locallm/connectors.html

I'd appreciate feedback from other developers building with Apple Foundation Models. In particular:

• What local connectors would be useful for your workflows?
• How are you currently testing prompts and structured output?
• What tooling still feels missing?

[svgeek (Ben)]

A couple more screenshots here for the Connectors and OpenAI API settings. The inclusion of an API key here is meant to make testing against an OpenAI API endpoint more realistic.

[svgeek (Ben)]

Update on this: v0.4 adds an MCP client, so the on-device model can now connect to real Model Context Protocol servers instead of only the first-party connectors from v0.3.0.

Tested and documented against eight servers so far: DeepWiki and Context7 (no auth), GitHub (personal access token), Notion, Todoist, Linear, and the official MCP reference server (all OAuth, discovered automatically — no setup needed on the service side), and Slack (OAuth, but requires registering a small app first — Slack doesn't support the automatic flow the others do).

The interesting constraint, and the reason I wrote up a whole page on it: the on-device model has a fixed ~4096-token context window, shared across every enabled tool from every connected server. Linear alone exposes 50+ tools, Todoist 45 — enabling even a small, reasonable-looking selection can eat a third of the entire budget. So every newly connected server starts with all tools unchecked, and you enable only the one or two you actually need per task. It's a real limitation compared to what a cloud model can do with the same servers, but there's still a useful slice of MCP you can exercise fully on-device: doc lookups, "what's assigned to me," search, that kind of thing.

Framed differently, the budget constraint is also what makes this a decent way to actually learn MCP hands-on: the model's already sitting there, running locally, so there's no API bill, no sandbox to spin up, and no cloud account needed just to watch a real capability-negotiation/tool-discovery/tool-call exchange happen against a production server.

Full writeup, including exact per-tool token costs where I measured them, tool-by-tool setup steps, and example prompts for each server:

LocalLM Lab MCP Servers

If there's a server you'd want documented next, let me know — happy to add it if it's publicly reachable.

[svgeek (Ben)]

Update on this: v0.6 adds localai-cli, a way to call the on-device model from your own Swift (or Python) code without going through API Lab's HTTP server.

Interface is deliberately minimal: your app spawns localai-cli as a subprocess, writes one JSON object to stdin ({"system_prompt", "user_input", "connectors": [...], "mcp_tools": [...]} for --run, or an OpenAI-style messages array for --chat), and reads one JSON object back from stdout. Every call requires --config <path> explicitly with no implicit lookup that points at the same localai-config.json that LocalLM Lab's own Local AI Settings and MCP Servers panels write.

The part worth calling out here specifically: that config file is the capability ceiling, and localai-cli only ever reads it, never writes it. connectors_enabled in the file is what the user has granted through the app's UI; the connectors/mcp_tools fields in a given request are what that call is asking for. The rule is strictly requested & enabled. Ask for something outside that set and you get {"error": "..."} back before LanguageModelSession is ever touched. There's no partial grant. Omitting the field is a real "nothing," not a default-allow. Same enforcement model connectors/MCP already have inside the app itself, just now reachable from a subprocess call instead of only the app's own UI.

Ships as a matched pair: localai-cli plus localai-playground-run (the actual model-invocation helper it wraps). It needs LocalLM Lab running in the background, since connector/MCP calls relay through its process over a local socket rather than being self-contained in the binary.

Worth being explicit about where this sits: it's still an experimentation tool, not a shipping dependency. The permission/authentication model here works because there's a single trusted app (LocalLM Lab) mediating everything. We aren't where we can say "bundle this in an app you distribute" yet.

There's a fuller worked example now too in "Plate Today", which combines three connectors (clock, calendar, reminders) and one MCP tool call (Todoist's find-tasks-by-date) in a single request, mixing the two capability systems in one call rather than treating them as separate paths. It also does its own preflight: reads localai-config.json directly, checks all four sources are enabled before calling localai-cli at all, and prints exactly which panel/toggle to fix if not. That's client-side validation layered on top of localai-cli's own request-time rejection. This means that a misconfigured run fails immediately with an actionable message instead of a generic {"error": ...} after a wasted model invocation.

Full interface docs + runnable Swift and Python examples (including the --chat shape and config/error-handling edge cases): thisbrain.ai/locallm/cli.html

PS: unrelated to the CLI, but since this crowd will notice -> Prompt Playground's UI has also moved from a browser window to native SwiftUI in the v0.6 release.

[svgeek (Ben)]

Here's another update on LocalLM Lab. This is release for macOS 27 Golden Gate.

1.0.0-beta.1 adds what I've been calling the model layer: one API surface across Apple's on-device FoundationModels model, Claude (via a host-supplied API key) and locally-run open-weight models (Qwen/Deepseek/Gemma/...). Complicated stuff like routing and model residency is owned by the SDK instead of every app rolling its own provider abstraction.

The front door is LocalLMLab, an optional convenience object wiring a model registry, the MCP manager, and the connector/workspace facades together:

```swift

let lab = LocalLMLab(configuration: .init(providers: [

    SystemModelProvider(),

    ClaudeModelProvider(auth: .apiKey(myKey)),

    MLXModelProvider(residentModelLimit: 1),   // from LocalLMLabSDKInference

]))

lab.models.route(.heavy, to: ModelID("mlx:mlx-community/Qwen3-8B-4bit")!)

lab.models.route(.light, to: .system)

let session = try lab.makeSession(route: .heavy, tools: myTools, instructions: sys)

```

ModelProvider is a protocol keyed by scheme : SystemModelProvider (system), ClaudeModelProvider (claude), MLXModelProvider (mlx, the separate LocalLMLabSDKInference binary), and PCCModelProvider (pcc, Apple's Private Cloud Compute). PCCModelProvider ships but **isn't functional in `1.0.0-beta.1`** — a session routed to .pcc fails outright. Check out Private Cloud Compute - Apple Developer for the details.

RouteName is a name your app maps to a ModelID (.heavy, .light, .draft, or any string). The SDK owns residency but never the routing policy: which route (aka model) a task uses is entirely your call as a developer.

The MLXModelProvider lifecycle is the part most worth detailing, since it's new: validate(_:) preflights a repo before pulling any weights. Preflight includes: reachability, MLX format, architecture support and weight size against MLXPreflightLimits.maxWeightFractionOfRAM (default 0.7 of physical RAM). download(_:) streams progress. capabilityProbe(_:) runs a real prompt plus a trivial tool call after download. This is the authoritative signal for whether a given model can reliably tool-call, not something to assume from its name. residencyEventStream emits .warmed/.evicted(reason:)/.loadProgress for a status line. residentModelLimit caps how many models stay resident in RAM at once; switching routes evicts the other.

Weights land in a standard Hugging Face cache ~/.cache/huggingface/hub/ for a bare CLI, redirected automatically to ~/Library/Containers/<bundle-id>/Data/Library/Caches/huggingface/hub/ under App Sandbox. A sandboxed app also needs com.apple.security.network.client for the download itself. This is the same silently-hanging-without-it entitlement gotcha called out in the SDK launch.

makeSession(route:tools:instructions:includeMCPTools:) resolves the route to a model and merges your tools with the enabled MCP session tools automatically (opt out with includeMCPTools: false). session.events carries the side-channel Apple's streaming doesn't give you (.toolCallStarted/.toolCallFinished/.contextCompacted/.modelLoadProgress) and session.contextBudget plus an optional retryOnContextOverflow compact-hook retry are there for sessions long enough to fill a context window.

Two new reference apps show this end to end: code-buddy, a CLI coding agent with .heavy/.light MLX routes, Core's Workspace tools, a live MCP docs server, and streamed output; and repo-qa-local, the minimal version of the same idea. workspace-buddy-local is the sandboxed-MLX filesystem example. There are 8 reference apps in the repo now. All for your reviewing pleasure!

Requires macOS 27 beta, Apple Intelligence enabled, Swift 6 tools, Xcode 27 beta for SDK development.

SDK guide: locallm/docs/sdk-guide.md at main · ancientcomputing/locallm · GitHub

Feature page: thisbrain.ai/locallm/1.0.0-beta

[svgeek (Ben)]

Continuing from the model-layer topic from a couple weeks back, LocalLM Lab 1.0.0-beta.3 extends the model layer to online providers.

A new RemoteModelProvider ships in a fourth binary, LocalLMLabSDKRemote (pure URLSession, no third-party dependencies). It's configured through RemoteProviderConfig: scheme, dialect (.openAIChat / .openAIResponses / .anthropicMessages / .openAICompatible), baseURL, auth, models — with presets for .openAI, .openAIResponses, .anthropic, .openRouter, and .openAICompatible for anything else speaking the OpenAI wire format (LM Studio, vLLM, a self-hosted endpoint).

Routing and session creation are identical to local/on-device routes:

lab.models.route("chat", to: ModelID(scheme: "openai", rest: "gpt-...")!)
let answer = try await lab.makeSession(route: "chat").respond(to: prompt)

Providers can be swapped at runtime without rebuilding LocalLMLab — ModelRegistry.replace(_:) / .removeProvider(scheme:) — useful if you're letting a user reconfigure their own API keys/endpoints.

Online providers also get first-class web search: capabilities.insert(.webSearch) plus defaultOptions.webSearch = true turns it on per-provider, or SessionOptions(webSearch: true) per turn. Results come back through the existing event/citation surface — session.events yields .serverToolCall with the queries, session.citations carries the sources.

One platform-gating detail: LocalLMLabSDKRemote's manifest floor is macOS 26, so linking it doesn't force a macOS 27 deployment target — but RemoteModelProvider itself is @available(macOS 27) and reports .requiresOS("macOS 27") when run on 26. That lets you ship one build that offers online models where the OS supports it and falls back to on-device everywhere else.

If you're building provider-configuration UI, Components doesn't link Remote at all — it calls back through onSave/onRemove/onTest closures with plain RemoteProviderDraft values, so you're not forced to pull in the Remote binary just to let a user type in an API key.

New reference app: model-switch — one chat interface that switches between OpenAI, Anthropic, OpenRouter, a custom OpenAI-compatible server, and Apple's on-device model mid-conversation, with a web search toggle and citations in the transcript. The provider-config glue is small, just 30 lines in ProviderGlue.swift converting a RemoteProviderDraft into a RemoteProviderConfig. It ships with a ModelSwitch.xcodeproj, which is also new for this release: all 11 SwiftUI reference apps now come with their .xcodeproj instead of needing to be built from the command line first (CLI-only examples still ship as Package.swift).

Why bother with online providers at all if you're mainly interested in local/on-device: two reasons beyond raw capability. First, it's a real cost/performance dial where you can route routine turns to a fast local or on-device model and escalate to a frontier model only for the hard cases, without three separate integrations to maintain. Second, it's a useful debugging framework: point a route at a frontier model to confirm your tool definitions and app framework are correct, independent of whether a smaller local or on-device model is just failing to follow your prompt or call tools correctly.

Requires macOS 27 beta on Apple Silicon for online providers, ClaudeForFoundationModels, PCC, and local open-weight models.

SDK guide: locallm/docs/sdk-guide.md at 1.0.0-beta · ancientcomputing/locallm · GitHub
Feature page: thisbrain.ai/locallm/1.0.0-beta

[John_McCall]

You're welcome to provide these updates, but please make them in the original thread for your project. I've gone ahead and moved the last couple back here for you.

[svgeek (Ben)]

Sounds good. I'll plan for a single thread and go from there. Will add today's update to this thread.

Thanks,

Ben

[svgeek (Ben)]

I tried posting today's update to the thread but got blocked.

Thanks,
Ben