Introducing Algebraic Effects and Effect Handlers for Swift

Hello, Swift Forum :waving_hand:
I’m excited to announce the preview of swift-effect - an architecture-agnostic effect system that makes side effects (such as I/O, networking, randomness, concurrency) controllable, composable, and testable while keeping application code linear and easy to reason about.

The library implements semantics of Algebraic effects and Effect handlers, and draws inspiration from established effect systems in OCaml 5 and Koka.

Key concepts

  • Effect: a minimal, operation-level abstraction, usually representing a single discrete operation like HTTP.get, Console.print, Int.random, Date.now, etc.
  • Effect Handler: a micro-interpreter for a specific effect type that provides concrete behaviour for an abstract operation. Can be freely composed with other effect handlers to build arbitrarily complex behaviours.
  • with-handler-perform: an effect-handling statement with control-flow semantics similar to do–try–catch. Instead of throwing and catching errors, it performs and handles effects. Handlers can be installed anywhere in the call stack, and may yield ("re-throw") to outer handlers, and therefore compose freely.

Key features

  • Free composition of behaviour via effect handler stacking which doesn't require direct coordination of object graphs or effect transformers.
  • Direct-style. Application code remains linear and can use the full set of control-flow constructs, such as defer, in stark contrast to other effect systems that require restructuring code into non-linear unidirectional data flow or monadic chaining.
  • Mock-less testing of observable behaviour: effectful programs can be intercepted, suspended, and resumed with just-in-time test data, without invasive scaffolding or test-only abstractions in application code.
  • Deterministic testing of Swift Concurrency, with Task, TaskGroup (WIP), and AsyncStream, modelled as controllable effects in their own right.

Effects

Let's take an echo program that prints back user inputs, and exits when nil is inputted.

func echo() {
  while let line = readLine() { // ⚠️ I/O side effect
    print(line) // ⚠️ I/O side effect
  }
}
echo()
> echo()
> Hello
Hello
> Good Bye
Good Bye
>
exit

This deceptively simple program performs two unmanaged I/O side effects,readLine and print, making it practically impossible to test or extend with custom behaviour.

Let's turn readLine and print operations into controllable Effects:

enum Console { // Namespace
  @Effect
  static func readLine() -> String? {
    Swift.readLine() // global handler
  }
  
  @Effect
  static func print(_ line: String) {
    Swift.print(line) // global handler
  }
}

func echo() {
  while let line = Console.readLine() { // ✅ Effect
    Console.print(line) // ✅ Effect
  }
}
func main() {
  echo() // Default behaviour with global handlers ☝️
}

The @Effect macro exposes each operation to the effect system, and generates corresponding effect handlers, which we'll see in action now.

Effect Handlers

Effects are modelled as globally defined, but locally interpreted operations. We can change the behaviour of individual effects by enclosing a program in with-handle-perform effect handling statement, just like we do with do-try-catch error handling:

func main() {
  with { 
    Print { line in // local handler for Console.print
      Swift.print(line.uppercased())
    }
  } perform: {
    echo() // Modified behaviour
  }
}

We've changed the behaviour of echo to respond with uppercased lines without changing its original implementation:

echo() - uppercased
main()
> "Hello"
"HELLO"
> "Good Bye"
"GOOD BYE"
>
exit

When a program performs an effect, such as Console.print, control flow is immediately transferred up to the first corresponding effect handler in the call stack. When the effect is handled, control flow is returned back to the program, allowing it to resume execution.

In this example, effects are handled by:

  • Console.print - local Print uppercased handler.
  • Console.readLine - global handler with Swift.readLine().

Effect composition

Just as do–try–catch can rethrow errors to the next error handler in the call stack, effect handlers can yield effects to the next handler. This design enables natural composition of behaviours, where stacked handlers can transform and augment each other’s behaviour without needing direct coordination.

First, let's generalise our uppercased Print effect handler by replacing the hard-coded Swift.print implementation with an effect:

:warning: Hard-coded:

Print { line in
  let uppercased = line.uppercased()
  Swift.print(uppercased) // ⚠️ hard-coded implementation
}

:white_check_mark: Composable:

Print { line in
  let uppercased = line.uppercased()
  Console.print(uppercased) // ⬆️ yields print effect to the next handler in the call stack
}

Let's compose it with another Print handler that logs printed messages, installed higher in the call stack:

import Module
@main
func main() {
  var log: [String] = []
  with {
    Print { line in
      log.append(line) // 🪵 augments print with logging
      Console.print(line) // ⬆️ yields to the next handler (global)
    }
  } perform: {
    module()
  }
}

// Module.framework
func module() {
   with {
    Print { line in 
	  let uppercased = line.uppercased() // ⚙️ transforms lines
      Console.print(uppercased) // ⬆️ yields to the next handler (main)
    }
  } perform: {
    echo()
  }
}
main() - uppercased + logging
main()
> Hello
HELLO    // log: ["HELLO"]
> Good Bye
GOOD BYE // log: ["HELLO", "GOOD BYE"]
>
exit

We can freely swap the order in which these handlers are installed in the call stack, changing the behaviour - logging unmodified lines before uppercasing them, without modifying either the handlers or the echo program.

main() - logging + uppercased
import Module
@main
func main() {
  var log: [String] = []
  with {
    Print { line in
      let uppercased = line.uppercased() // ⚙️ transforms lines
      Console.print(uppercased) // ⬆️ yields to the next handler (global)
    }
  } perform: {
    module()
  }
}

// Module.framework
func module() {
   with {
    Print { line in 
	  log.append(line) // 🪵 augments print with logging
      Console.print(line) // ⬆️ yields to the next handler (main)
    }
  } perform: {
    echo()
  }
}
main()
> Hello
HELLO    // log: ["Hello"]
> Good Bye
GOOD BYE // log: ["Hello", "Good Bye"]
>
exit

with-handler-perform blocks are result builders, and can install multiple effect handlers at a time, and include control flow statements:

func main() {
  var log: [String] = []
  with {
    if enableLogging {
      Print { line in
        log.append(line)
        Console.print(line)
      }
    }
    ReadLine {
      Console.readLine().uppercased()
    }
  } perform: {
    echo()
  }
}

Testing

Exhaustive just-in-time testing

Let's put echo to test.

The library provides a specialised withTestHandler that enables a powerful testing style in which a program under test is executed step by step, suspended on each performed effect, matched against expectations, and then resumed with just-in-time test data.

@Test
func test() async throws {
  try await withTestHandler {
    echo()                                                    // 1
  } test: { effect in                                         
    try await effect.expect(Console.ReadLine.self) {          // 2
  	  return "Hello"                                          // 3
    }
    try await effect.expect(Console.Print.self) { line in     // 4
	  #expect(line == "Hello")                                // 5
    }
    try await effect.expect(Console.ReadLine.self) {
	  return "Good Bye"
    }
    try await effect.expect(Console.Print.self) { line in
  	  #expect(line == "Good Bye")
    }
    try await effect.expect(Console.ReadLine.self) {
	  return nil // exit                                       // 6
    }
  }
}
A detailed break-down of how this test is executed:
  1. echo() is launched inside withTestHandler context.
  2. The test block is immediately suspended on the first line, awaiting the first effect produced by echo, which is expected to be Console.readLine. When echo performs Console.readLine(), execution of echo is suspended until a value is supplied by an effect handler. This effect is intercepted by the test handler, which attempts to match it against the effect.expect(Console.ReadLine.self) expectation. If an unexpected effect is produces, the test fails.
  3. Since the produced effect matches the expectation, the test resumes into the trailing closure, which acts as a just-in-time ReadLine effect handler. In this case, the handler simply returns the "Hello" line, allowing echo to resume execution and to echo "Hello" back.
  4. The test then proceeds to await effect.expect(Console.Print.self) and is suspended again, waiting for the next effect to be produced by echo.
  5. When echo performs Console.print("Hello"), execution of echo is again suspended while awaiting the print effect to be handled. The test intercepts the effect and attempts to match it against the await effect.expect(Console.Print.self) expectation. Upon a successful match, the test resumes into the just-in-time trailing closure effect handler. In this case, the handler asserts #expect(line == "Hello") and returns, allowing echo to resume execution and proceed to the next readLine loop iteration.
  6. This ping-pong exchange between the program and the test continues until a ReadLine handler returns nil, causing the program to exit and the test to conclude.

Effects almost always represent observable behaviour of a program and can therefore serve as a strong proxy for meaningful, user-facing behaviour, isolated from implementation details.

Non-exhaustive testing

Sometimes we want to focus on testing a specific behaviour, while ignoring others. In other words, perform non-exhaustive and partial testing.

This can be achieved using the already familiar composition of effect handlers:

@Test
func test() async throws {
  try await withTestHandler { // outer test handler
    with {                    // inner "stub" handler
      Print { _ in } // no-op
    } perform: {
      echo()
    }                                                  
  } test: { effect in                                         
    try await effect.expect(Console.ReadLine.self) {
  	  return "Hello"
    }
    try await effect.expect(Console.ReadLine.self) {
  	  return "Good Bye"
    }
    try await effect.expect(Console.ReadLine.self) {
  	  return nil // exit
    }
  }
}

In this example, the innermost no-op Print handler intercepts and "discharges" the print effect by not yielding it, making it invisible to the test and allowing the test to focus solely on Console.readLine and the echo exit logic.

Effect handler composition in tests provides great flexibility in choosing which behaviours to test and which to ignore. It also allows you to replicate traditional ahead-of-time testing approaches, where some or all effects are handled by specialised test doubles such as spies, stubs, and mocks.

Opt-in testing with inner handlers

In this example, the test only receives print effects when "Hello" is printed:

@Test
func test() async throws {
  try await withTestHandler { // outer test handler
    with {                    // inner "filter" handler
      Print { line in
        if line == "Hello" {
          Console.print(line) // ⬆️ yields to the test only for "Hello"
        }
      }
    } perform: {
      echo()
    }                                                  
  } test: { effect in                                         
    try await effect.expect(Console.ReadLine.self) {
  	  return "Hello"
    }
    try await effect.expect(Console.Print.self) { line in
	  #expect(line == "Hello") // visible to test
    }
    try await effect.expect(Console.ReadLine.self) {
	  return "Good Bye"
    }
    try await effect.expect(Console.ReadLine.self) {
	  return nil
    }
  }
}
Opt-out testing with outer handlers

Let’s reverse this composition by enclosing withTestHandler within stub handlers. In this example, the test handler catches all effects but can selectively opt out of testing some of them by yielding to the enclosing handlers:

@Test
func test() async throws {
  with {                        // outer "stub" handlers
    Print { _ in }
    ReadLine { return "foo" }
  } perform: {
    try await withTestHandler { // inner test handler
      echo()                                                  
    } test: { effect in     
    // First echo loop iteration                                 
	effect.yield() // ⬆️ opt-out, yield to "foo" ReadLine
    effect.yield() // ⬆️ opt-out, yield to no-op Print
    
    // Second echo loop iteration                            
    try await effect.expect(Console.ReadLine.self) {
	  return "Good Bye"
    }
    try await effect.expect(Console.Print.self) { line in
	  #expect(line == "Good Bye")
    }
    // Exit
    try await effect.expect(Console.ReadLine.self) {
	  return nil
    }
  }
}

Async Streams and Effects

AsyncStream and AsyncSequence are modelled as controllable effects in their own right, which enables the same powerful testing style with just-in-time test data.

Let's define an Effect that returns an AsyncSequence of random numbers, a number per second:

enum Random {
  @Effect
  static func numbers() -> any AsyncSequence<Int, Never> {
    AsyncThrowingStream {
      try? await Task.sleep(for: .seconds(1))
	  return Int.random(in: 0...100)
    }
  }
}

...and a stateful program that consumes this stream:

@Observable
class Numbers {
  var message: String = ""
  
  func run() async {
    var previousNumber: Int?
    for await number in Random.numbers() {
	  if previousNumber == number {
	    message = "It's the same number again, boooring!!"
	  } else if number.isMultiple(of: 2) {
	    message = "\(number) is a good number"
	  } else {
	    message = "\(number) is a bad number"
	  }
	  previousNumber = number
    }
    message = "Good bye!"
  }
}

Semantically, an Effect that produces an AsyncSequence of elements is equivalent to an AsyncSequence of effects, each producing a single element.

This is exactly how AsyncSequence effects are handled in tests:

@MainActor
@Test
func test() async throws {
	let numbers = Numbers()
	try await withTestHandler {
	  await numbers.run()
	} test: { effect in
	  try await effect.expect(Random.Numbers.self, return: 1)
	  await #expect(numbers.message == "1 is a bad number")
	
	  try await effect.expect(Random.Numbers.self, return: 1)
	  await #expect(numbers.message == "It's the same number again, boooring!!")  
	  
	  try await effect.expect(Random.Numbers.self, return: 2)
	  await #expect(numbers.message == "2 is a good number")
	  
	  try await effect.expect(Random.Numbers.self, return: nil) // EoS
	  await #expect(numbers.message == "Good Bye!")
	}
}

Reliably testing message state updates in response to a next-produced random number would indeed be tricky with traditional ahead-of-time test doubles.

Deterministic testing of Unstructured Concurrency

The library provides a Task.effect API that acts as a drop-in replacement for Task.init. In production, this effect simply returns a normal Task, leaving runtime behaviour unchanged. In tests, however, it allows withTestHandler to control task scheduling in a deterministic way.

Consider this program:

func program() {
  Console.print("Start")
  Task.effect(name: "Task 1") {
    Console.print("Task 1")
  }
  Task.effect(name: "Task 2") {
    Console.print("Task 2")
  }
  Console.print("Exit")
}
program()
> program()
Start
Task 2
Task 1
Exit
> program()
Start
Task 1
Task 2
Exit

The order of "Task 1" and "Task 2" print messages is non-deterministic since unstructured tasks are scheduled and executed concurrently. But in tests we gain control over task scheduling.

In this test, we enqueue Task 1 before Task 2:

@Test
func enqueueInOrder() async throws {
  try await withTestHandler {
    program()                                                               // 1 | Queue: [program]
  } test: { effect in
	try await effect.expect(Console.Print.self) { #expect($0 == "Start") }  // 2
	try await effect.expectTask("Task 1", action: .enqueue)                 // 3 | Queue: [program, Task 1]
	try await effect.expectTask("Task 2", action: .enqueue)                 // 4 | Queue: [program, Task 1, Task 2]
	try await effect.expect(Console.Print.self) { #expect($0 == "Exit") }   // 5 | Queue: [program, Task 1, Task 2]
	try await effect.expect(Console.Print.self) { #expect($0 == "Task 1") } // 6 | Queue: [Task 1, Task 2]
	try await effect.expect(Console.Print.self) { #expect($0 == "Task 2") } // 7 | Queue: [Task 2]
  }
}
A detailed break-down of how this test is executed:
  1. program() is added to the serial test execution queue.
  2. The first print("Start") is handled normally.
  3. The test is suspended, awaiting the Task effect produced by the program via the expectTask expectation. When the program creates Task 1, the effect is intercepted and handled by the test with the .enqueue action, which appends the task to the end of the serial execution queue, after program().
  4. The same happens with Task 2, which is appended after Task 1.
  5. The second print("End") is handled normally, program() exits its scope, and is popped from the execution queue.
  6. Task 1 is executed, performing print("Task 1"), which is expected by the test. Task 1 exits its scope and is popped from the execution queue.
  7. Task 2 is executed, performing print("Task 2"), which is expected by the test. Task 2 exits its scope and is popped from the execution queue.

In this test we suspend each task when it's created and later explicitly enqueue Task 2 before Task 1:

@Test
func enqueueInOrder() async throws {
  try await withTestHandler {
    program()                                                            //   | Queue: [program]
  } test: { effect in
	try await effect.expect(\.Console.print) { #expect($0 == "Start") }  
	let task1 = try await effect.expectTask("Task 1", action: .suspend)  //  | Queue: [program]
	let task2 = try await effect.expectTask("Task 2", action: .suspend)  //  | Queue: [program]                
	task2.enqueue                                                        //  | Queue: [program, Task 2]
	task1.enqueue                                                        //  | Queue: [program, Task 2, Task 1]     
	try await effect.expect(\.Console.print) { #expect($0 == "Exit") }   //  | Queue: [program, Task 2, Task 1]     
	try await effect.expect(\.Console.print) { #expect($0 == "Task 2") } //  | Queue: [Task 2, Task 1]
	try await effect.expect(\.Console.print) { #expect($0 == "Task 1") } //  | Queue: [Task 1]     
  }
}

More examples and features

  • README for more detailed explanations and full list of currently available features.
  • EffectPlayground for additional runnable example.

Get Involved

The library is still in a high-level design phase, so general ideas and feedback on the API, concepts, and mechanics is welcome! Please comment here or join GitHub Discussions. Pull requests and feedback on specific implementation details (especially around Swift Concurrency) are also very welcome, though please expect some of these areas to be revisited in the coming weeks.

Happy effective coding!

13 Likes

It's an interesting topic, need to check the preview! :upside_down_face:

I’ve been thinking a bit about whether we can have a proper effect system after looking at OCaml the other day, and I’ve come to the conclusion that having async and throws already gives you a poor man’s effect system (if managed properly).

This leads to the question of whether one can manage code that way—and if so, why we need to write explicit Effect boilerplate at all. Any thoughts on that?
Yes, you lose understanding of “is it an effect?” and how effects compose, but effects in Swift usually have an imperative nature anyway and can be wrapped + effects adding a bit of complexity.


I'm just discussing our of curiosity, so no pressure :upside_down_face:

I'm assuming that with "Effect boilerplate" and “is it an effect?” you're referring to fully typed effect systems, like one in Koka, where user-defined effects are tracked in function signatures, e.g. func getUsers() http -> User, where http is a first-class user-defined effect type, much like async and throws?

Yes, you lose understanding of “is it an effect?” and how effects compose, but effects in Swift usually have an imperative nature anyway and can be wrapped + effects adding a bit of complexity.

Indeed, and for this reason both OCaml 5 and swift-effect are untyped effect systems, where effects are not tracked by the type system (this would require significant extension of the compiler), and fully embrace the impure and imperative nature of the language. Instead of focusing on the type system part of the problem, swift-effect focuses on implementing the runtime semantics of Algebraic effect and Effect handlers.
By modeling effects as globally defined but locally interpreted functions, we can make effects completely boiler-plate free and transparent to programs, you don't even have to use namespaces like in my examples, e.g. readLine and print can be fully controllable Effects (namespace collision with built-in functions notwithstanding):

func echo() {
  while let line = readLine() { // controlled Effect
    print(line) // controlled Effect
  }
}

func main() {
  with { 
    Print { line in // local handler for print
      Swift.print(line.uppercased())
    }
  } perform: {
    echo() // Modified behaviour with uppercased print
  }
}
1 Like