Building custom configuration providers

[Honza_Dvorsky]

Hi folks,

since open-sourcing Swift Configuration, I’ve had multiple discussions with community members who are interested in building a custom ConfigProvider for a specific source or format that Swift Configuration doesn’t support out of the box.

This is great to see and I think we’ll benefit by sharing lessons with one another, which will further help refine the ConfigProvider protocol itself before we stabilize it in Swift Configuration 1.0 in the coming months.

If you’re interested in building a custom configuration provider, or are need one that doesn’t exist yet, please reply in this thread with more details: What format or source would the provider integrate with? Would you like to contribute or are you just registering demand?

We also have a #swift-configuration channel on the OSS Swift Slack instance, in case you’d like to join.

cc’ing folks I’ve already discussed this with, let’s get everyone in one place and collect all the providers the community would like to see and build! @sebsto @slashmo @albertodebortoli @adam-fowler @0xTim @FranzBusch @hamzahrmalik @jacubit

[jacubit (Javier)]

Hi everyone,

I'm starting this thread by sharing my learnings so far while implementing a custom ConfigProvider which fetches secrets from a remote Vault. My custom provider falls into the category of ConfigProviders that wrap a REST API client, so its main access pattern is fetch. It wraps a VaultClient from vault-courier and it's meant to provide only secrets, used close to application start and as static configuration.

Here's a quick overview of what a typical Vault URL secret request looks like. In Vault, secrets live in "engines" that can be mounted under different paths; much like a virtual filesystem. Each engine type (e.g., "keyValue", "database", "jwt") has its own parameters, so a typical secret request involves: engine type, mount path and engine-specific parameters.

One takeaway was that it's not immediately clear where these parameters should live in a ConfigProvider implementation. Should they be part of the provider itself, or passed as context through the configuration key? It also raises a broader question about how context is meant to be used within the configuration hierarchy. I suspect anyone building a custom ConfigProvider around a REST API client will encounter similar questions. AFAIK all of the built-in providers ignore the context.

Another question is whether all custom providers should support all ConfigTypes. For Vault, this isn't always practical—only certain engines (like KV) support structured types, while others (e.g., database or JWT secrets) make sense only as strings or bytes.

[jacubit (Javier)]

I explored several approaches:

1. Passing client-specific parameters in context. Simple but makes config keys less meaningful

// Init rest client with base URL
let restClient = VaultClient(...)

let config = ConfigReader(providers: [
	VaultSecretProvider(client: restClient),
	try await JSONProvider(filePath: .init("config.json"))
])

let key = "third_party.service.api_key"
// Ignore by VaultSecretProvider as it doesn't have the context
let apiKey = try await sut.fetchString(forKey: key)

// Pick up by VaultSecretProvider as it has a matching context
let apiKey = try await sut.fetchString(forKey: key,
									   context: ["engine": "keyValue",
												 "mount_path": "path/to/secrets",
												 "url": "https://127.0.0.1:8200/v1/secret/local_test"])

[jacubit (Javier)]

2. Creating fine-grained providers per Vault engine, each handling its own parameters. Still uses the context.

// Init rest client
let restClient = VaultClient(...)
// Provider of Vault key-value secret engine
let keyValueSecretProvider = KeyValueSecretProvider(
	 client: restClient,
	 mount: "path/to/kv_secret/mount"
)

// Provider of Vault database secret engine
let databaseSecretProvider = DatabaseSecretProvider(
	 client: restClient,
	 mount: "path/to/database/mount"
)

let config = ConfigReader(providers: [
	keyValueSecretProvider,
	databaseSecretProvider,
	try await JSONProvider(filePath: .init("config.json")),
	CustomEnvironmentVariablesProvider() // Uses the context, unlike EnvironmentVariablesProvider
])


// - could be read from `keyValueSecretProvider`
// - always fails with `databaseSecretProvider` as the "keyName" and "version" contexts are not defined for that secret engine
// - could be read from the json provider
// - could be read from a custom env var provider THIRD_PARTY_SERVICE_API_KEY_KEY_NAME_DEV_VERSION_2
let apiKey = try await sut.fetchString(
  forKey: "third_party.service.api_key",
  context: ["keyName": "dev", "version": 2]
)

// - always fails with `keyValueSecretProvider` as the "type" and "roleName" contexts are not defined for that secret engine
// - could be read from `databaseSecretProvider`
// - could be read from the json provider
// - could be read from the env var DATABASE_PASSWORD_TYPE_DYNAMIC_ROLENAME_QA
let password = try await sut.fetchRequiredString(
  forKey: "database.password",
  context: ["type": "dynamic", "roleName": "qa"]
)

3. Building a mutable VaultSecretProvider that maps config keys to REST client operations.

// Init rest client
let vaultClient = VaultClient(...)

let absoluteKey1 = AbsoluteConfigKey(["third_party", "service", "api_key"])
let absoluteKey2 = AbsoluteConfigKey(["third_party", "service", "api_key"], context: ["version": 2])
let absoluteKey3 = AbsoluteConfigKey(["server", "database", "credentials"])

let secretProvider = VaultSecretProvider(
	vaultClient: vaultClient,
	evaluationMap: [
		absoluteKey1: try await VaultSecretProvider.keyValueSecret(mount: kvMount, key: secretKeyPath),
		absoluteKey2: try await VaultSecretProvider.keyValueSecret(mount: kvMount, key: secretKeyPath, version: 2),
		absoluteKey3: try await VaultSecretProvider.databaseCredentials(mount: databaseMount, role: .static(name: staticRole))
	]
)

let config = ConfigReader(providers: [
	secretProvider,
	try await JSONProvider(filePath: .init(config2.json))
])

// Secret provider will be used inn the hierarchy
let secret = try await config.fetchRequiredString(
  forKey: "third_party.service.api_key",
  context: ["version": 2],
  as: ServiceSecret.self
)
// Secret provider will be used inn the hierarchy
let staticCredentials = try await sut.fetchRequiredString(
  forKey: "server.database.credentials",
  as: DatabaseCredentials.self
)

// This ConfigKey has not been registered in VaultSecretProvider,
// so the credentials will be read from the JSONProvider
let absoluteKey4 = AbsoluteConfigKey(["job", "database", "credentials"])
let databaseKey: String = absoluteKey4.components.joined(separator: ".")
var credentials = try await sut.fetchRequiredString(
  forKey: databaseKey,
  as: DatabaseCredentials.self
)

// Mutate the secret provider by adding a new mapping key and action
secretProvider.updateEvaluation(
	absoluteKey4,
	with: try await VaultSecretProvider.databaseCredentials(mount: databaseMount, role: .dynamic(name: dynamicRole))
)

// This time, the credential is retrieved from Vault,
// since the key is now registered and takes precedence in the hierarchy.
credentials = try await sut.fetchRequiredString(
  forKey: databaseKey,
  as: DatabaseCredentials.self
)

Would love to hear others' experiences or design thoughts around similar REST-backed ConfigProvider implementations.

[Honza_Dvorsky]

Thanks for sharing your learnings, @jacubit! To me 2 and 3 both fit well into the overall design of Swift Configuration, 1 is trickier as it tightly couples the information that the user of ConfigReader needs to have about the provider, which undermines the purpose of the abstraction.

What are your thoughts on what get and watch calls should do in your providers? A few options:

• prefetch secrets for specific names on launch, so they can be returned by get, not only by fetch
• watch could first make a fetch call to get the first value, and then periodically re-fetch to check if it's changed
• you can also leave it as-is, and have get and watch always return nil values

I hope we'll get to explore this space more as @slashmo expressed some interest in an OpenFeature provider, which also has different access patterns.

[jacubit (Javier)]

@Honza_Dvorsky

What are your thoughts on what get and watch calls should do in your providers? A few options:

• prefetch secrets for specific names on launch, so they can be returned by get, not only by fetch

Yes, this can be done (similarly like the Env var provider on initialization) by passing initial values that get can return. Currently, all options update this cache (via a MutableInMemoryProvider) after a successful fetch.

The watch access is trickier and needs more exploration if should be supported by a VaultSecretProvider. For now, I'm using watchValueFromValue, since actively watching remote secrets isn't something we want to encourage. It's a complex area to handle correctly and can easily lead to bad practices, so I'm leaving it out for the time being.