Swift Build Ecosystem (Compile Time Code Execution, Build Management)

The motivation for this pitch is to start a discussion about swift compiler features which will eventually lead to a compiler evolution where we can replace build scripts and tools with a centralized pure swift build ecosystem.

The swift package manager has reached a nice feature set of cross-compilation abilities. It is possible to configure the toolchain and compiler flags in nice and lean destination json files. There is a great community interested in cross platform swift development like supporting Linux, Android, Windows so there is a community interest for swift on new targets.

But we have a strong fragmentation when it comes to compilation configurations and build-processes or CI integration. Many people are writing python or ruby scripts others are using make or cmake. I think we could do better and solve this issue in the swift ecosystem. It can become really frustrating to learn writing more complex scripts with hackish bash scripts and also non-human readable code.

You can check the utils directory in the official swift repository to get a clearer picture of the point I'm trying to describe. Just the utilities used to build swift or do some other tasks contains Python, Ruby, Perl, Bash, JSON, Configuration Files...

Think of the Swift interoperability with dynamic languages like Python, we wouldn't even need to rewrite all our scripts in swift but we can bridge them already.

There are really nice command line tools written in swift, which solve problems we are facing in everyday life like code generation, linting, localization, content generation...
So why aren't we using all those features baked right into compile time executed swift code. That's the language we want to write our code in and that is also the language we want to control our ecosystem in. Take the Package.swift file of SwiftPM as an example it is purely written in swift and is used to describe and manage dependencies. It works on all platforms supporting the swift toolchain.
But what happens if you require more complex build management processes, well you could use Xcode or something similar but you would be bound to the macOS operating system. So you stick with make or cmake to manage your build targets.

Some people have started adopting the Ideas of make in swift with projects like beak.
In my opinion, this should be an important part of the swift evolution process.

In the following part are example features described, which could help build up a future build management system.

Please note that the code examples are only for describing the idea and not any API design.

Arbitrary Compile Time Code Execution

We could add a feature to the swift compiler, to run code while being in the compilation process.
By adding a keyword like #run or maybe @compileTime to tell the compiler to execute the code at compile time.

#run lint()
#run downloadAssets()
#run compileAssets()

let compileTimeConstant = #run generateData()

let configuration = #run askForUserInput()

let lookUpTable = #run myReallySlowDataGenerator()

#run bundleFiles(using: configuration)

This could enable new possibilities in swift and would also combine the build process with the project source code. You could even reuse project code to do things at compile time or add features like compile time asserts But we could also stick to other external solutions...

Swift Build Manager

Adopting the ideas of the Swift Package Manager to a Swift Build Manager project which will provide features like for example gradle.
This could require adding a Build.swift file to the root directory and would provide interoperability with the Package.swift file like importing libraries.

Package.swift

// swift-tools-version:4.0

import PackageDescription

let package = Package(
    name: "MySuperCoolFramework",
    products: [
        .library(name: "MySuperCoolFramework", targets: ["MySuperCoolFramework"])
    ],
    dependencies: [
        .package(url: ".../SomeFramework...", .upToNextMajor(from: "1.0.0"))
    ],
    targets: [
        .target(name: "MySuperCoolFramework", dependencies: [])
    ]
)

Build.swift

// This is non realistic pseudo swift code to describe the idea.
// swift-tools-version:4.0

import PackageBuildDescription
import SomeFramework

let build = Build(
	targets: [
		.target(
			name: "MySuperCoolFramework",
			preprocess: [
				.run(lint),
				.run(generateAPI)
			],
			postprocess: [
				.run(makeBundle)
			],
			flags: [...],
			// ...
		)
	]
)

func lint(_ compilerInterface: CompilerInterface) throws {
	Linter.lint(compilerInterface.sourceCode)
}

func generateAPI(_ compilerInferface: CompilerInterface) throws {
	let description = // try download api description files from url or throw
	let sourceCode = APIGen.generate(for: description)

	compilerInferface.compile(sourceCode)
}

func makeBundle() {
	// Bundle executable + resources to a bundle of any kind you like for any platform you use
}

2 Likes

Have you seen the draft proposal for extensible build tools? What you're suggesting seems like it could be solved by a combination of that proposal + maybe some add-on workflow features to help run specific tools (like linters) when you want to that aren't a core part of building your actual product.

Thanks @rballard haven't seen the draft.

I really like the Idea of executing code in compiler time, was hoping to get feedback on that topic.