SwiftPM-Style configuration files

Evolution Pitches
1

Motivation

I like the Design of Swift Package Manager.
More specifically how the configuration/specification is done via the Package.swift file.
This design i think has some advantages over doing configuration with something like json.

  • its strongly typed
  • its more expressive (for instance one can use a enum to express mutually exclusive options)
  • its the same language you are already using
  • could potentially provide autocompletion

While this is possible in current swift its a bit complicated to setup.

I was wondering if we could provide some facilities to make this kind of API Design easier.

Example

To explain the problem i will use a example project of a webserver.

Following the Design of the SwiftPackageManager the project setup would be something like this:

| Package.swift
| WebServer.swift
| Sources
   |__ WebServer
   |    |__ ...
   |__ WebServerDescription
        |__ ...


// Package.swift

let package = Package(
   ...
   targets: [
        .target("WebServerDescription"),
        .target("WebServer",
             dependencies:  [ "WebServerDescription"]
        )
   ]
   ....
)

The goal is to somehow allow the WebServer target to load its Configuration from a .swift file

In this example it could be WebServer.swift

// WebServer.swift

import WebServerDescription

let server = WebServer(
    url: "127.0.0.1",
    port: 8080
)

–––

Ideas

Im sure theres lots of ways this can be done and i dont really know whats feasible.

Heres a couple of ideas that i think could work.

Convert File to Machine Readable Format

This is the approach SwiftPM is using
From what i could gather this is

Running the Package.swift And linking the PackageDescription and passing some arguments that causes the package defined in Package.swift to be encoded as json and written to disk to be then consumed later.

This has the limitation that the code doesn’t run as part of your executable, so you cannot directly interact with the code in the configuration file (like providing a delegate)

For SwiftPM i think is by design, to protect against arbitrary code execution.
But i think "arbitrary code execution" could also be a feature.

Dynamically "link"/ load the configuration file

I don’t exactly know whats possible in that regard.
But maybe something like "Dynamic Method Replacement" from this thread could be utilised to load this "directly" into your code

Any Feedback is welcomed

2

It might not be as clean a solution as what you're imagining, but it should already be possible to do this in the current incarnation of Swift with no modification, and you don't need anything as exotic as dynamic linking.

I believe it could be done in a few steps:

  • WebServerDescription declares configuration types which conform to Codable

  • Some process compiles WebServer.swift and dumps its contents to JSON

  • Whatever program is consuming the configuration also imports WebServerDescription. All it has to do is parse the JSON dumped from the config file, and you now have access to your config.

But honestly I'm not sure when it would be useful to have a config file as a .swift which would need to be loaded dynamically. It seems like in general you could just include WebServer.swift directly in your project..

1 Like
3

I think thats similar if not the same as the approach i tried to explain in " Convert File to Machine Readable Format.

I get that.
I do think there are some usecases where directly loading the code could be useful.
Imagine in the ConfigurationFile you want to modify the behavior of the server by using a closure.

// WebServer.swift

import WebServerDescription

let server = WebServer(
    url: "127.0.0.1",
    port: 8080,
    authenticate: { request in
         let credentials = // some code to get credentials 
         return request.authenticate(using: credentials)
    }
)
4

I am not a security expert, but the example you gave seems to be a reason not to have dynamic config files. Allowing the authentication method to be changed by changing another file on the server seems like a disaster waiting to happen.

But in any case, even this should already be possible in principal in current Swift. It's already possible to load a .dylib at runtime, so it would be in principal possible to build a library from your config file and load it yourself.

I'm still not sure what advantage that would give you over just moving this config file inside your module directly.

5

Usually, reasons for external config include the following:

  • it's more secure to use "pure data" config without any runnable code. it's easy to test code with different injected configs, but you don't usually want to "test" your config or worse, forget to test for something that will then fail in production
  • it might be easier to read config in "non-smart" formats, especially when it's naturally hierarchical
  • you usually want to have different config for different locations, e.g. in development, for tests, in production, ... (e.g. URLs to external APIs)
  • you might want to have an "override" local config that is .gitignored, so you can play around with settings locally or adapt them to something you require personally (e.g. if you're behind a proxy and need special settings or so)

You can use Codable, but I sometimes find Codable slightly awkward and not flexible enough. There's also Configuration from IBM. We used the latter package but had to add some of our own logic in order to:

  • deal with different profiles
  • convert the config into type-safe value objects
  • deal with error handling (e.g. if required values aren't set)
  • use the config values to build the component graph for the application (i.e. all the different "services")

I don't know of any general solution for this in Swift, so this took 1-2 days to set up, but it seems to work well. Somebody could probably build a framework for this, but if it has to be general enough in order to be useful for many people, it probably needs to be quite complex (see Java's Spring and similar frameworks for an example of how complex such frameworks can get) and would also take some time to learn.

6

You are probably right.

It would have just been a nice to have since i am building some tool with a similar setup like SwiftPM.
But maybe using swift for configuration is a bit too impractical, and just providing the tool as a library instead of a compiled executable is the way to go.

thanks anyways

7

I refrained from commenting earlier, because I understood you to already have something working, and you wanted to pitch the idea of including it in Swift. But your last comment sounds more like you want help getting the ability to do it in the first place. So now I’m answering.

The executable nature of the Swift’s package manifest is so wonderfully powerful compared to older solutions like JSON or YAML, which tend to be so exasperatingly repetitive. (That is my opinion anyway.) From the moment I figured out how to implement executable configurations, I have never gone back. Now even if some other tool wants a “pure data” configuration, then I create my own executable configuration that in turn produces whatever inputs that other tool wants. It prevents me from ever having to copy and paste from one project to another (which inevitably gets out of sync). Instead it allows me to import and call parameterized functions that assemble generic, company and then project‐level defaults, and then still override them were necessary.

Anyway, the strategy I use right now is to copy the configuration to a temporary directory, wrap it in an executable Swift package, and execute it. It allows virtually everything to be deferred to SwiftPM—arbitrary package imports, dependency resolution, reliable linking, syntax error messages, platform differences, environment conditionals, etc. And it also inherits other current and future bonus functionality from SwiftPM for free—sandboxing, mirroring, caching, resources, closed‐source (binary) dependencies, distributed caching, etc.

I long ago factored package loading into its own pair of modules in an open source package anyone can use. You can find it here. You can use it as‐is for whatever purpose you have in mind, or you can reverse engineer it, either for your own customized use, or as a starting point towards a proposal for addition to the Swift project.

The main package manager team has also said here on the forums that they would be open to vending a generalized form of their own manifest loading functionality for other tools to use as well. And the formatter team has said that if it were made available, they would consider using it for swift-format. It just isn’t very high on anyone’s priority list.

1 Like
8

Daniel Dunbar did the original implementation for SwiftPM, and I've done it separately since for another project. Essentially:

  1. Build the “manifest format” as a dylib
  2. Make some Codable object that is used by the manifest and in its deinit it dumps itself to stdout as JSON
  3. Build the manifest as an executable that dynamically links to the dylib from 1. and run it

Actually 2. might have to be an at_exit (c function) call.

It's a bit of a hack, would be nice if the Swift driver had some way to do it without the hack, Ruby for instance has a variety of switches that allow it to be used from the command line in interesting ways, one could forsee swift getting some such switches.