Correct primitive for awaiting one AsyncSequence, and sending to another?

Using Swift
,
1

I'm working through the Crafting Interpreters book, and for extra fun, I'm playing with using Async Swift to coordinate the various parts together.

One example is the REPL. Fundamentally, it's just a function ot type (AsyncSequence<String>) -> AsyncSequence<String> which takes input lines, and produces the evaluation results to be printed. AsyncSequence.map seems like a good fit for this, but with one short-coming: The REPL should be able to print the prompt (e.g. "> ") before the first line has been entered.

What's the correct way to do this transformation? I was able to make it with by pulling in AsyncAlgorithms for AsyncChannel and combining that with a Task. Is there a simpler way?

import AsyncAlgorithms

struct REPL { // I don't really need this struct, could just be a function I suppose.
	let output = AsyncThrowingChannel<String, Error>()
	
	init<S: AsyncSequence>(lineSource: S) where S: Sendable, S.Element == String {
		Task { [output] in
			
			let prompt = "> "
			await output.send(prompt)
			
			for try await line in lineSource {
				// This is where the interpreter would actually evaluate the line
				let result = "process(\(line))\n"
				
				await output.send(result)
				await output.send(prompt)
			}
			
			output.finish()
		}
	}
}

let repl = REPL(lineSource: FileHandle.standardInput.bytes.lines)
		
for try await output in repl.output {
    print(output, terminator: "")
    try FileHandle.standardOutput.synchronize()
}
1 Like
2

(post deleted by author)

3

A couple of reactions:

  1. Using unstructured concurrency with Task {ā€¦}, without handling cancellation is a bit of an anti-pattern. [I wrote a long and tedious answer explaining how to handle this properly, but in retrospect, that was tangential to the real question here, so I removed that reply.]

  2. Having a sequence that yielded both the output and the next prompt is also a bit of an anti-pattern. If you really did that, Iā€™d yield an enumeration with associated values for either the prompt string or the repl output string, so the consumer could differentiate between them.

    But I would contend that one really does not want a single sequence that intermingles the UI (the prompt) with the output strings. (In your case, your consumer is just printing these strings, so it didnā€™t matter too much, but if it was doing anything else, this process of sending UI prompt and REPL output both as simple strings would make it very difficult to differentiate between the two.)

Personally, this doesnā€™t strike me as a great use-case for the AsyncSequence from another AsyncSequence. I would just iterate through the sequence:

printPrompt()

for try await input in FileHandle.standardInput.bytes.lines {
    let output = try await replOutput(for: input)
    print(output)
    printPrompt()
}

In answer to the question about how to create an AsyncSequence from another AsyncSequence, as you noted, one would generally just map. For example:

let outputSequence = FileHandle.standardInput.bytes.lines.map {
    try await replOutput(for: $0)
}

printPrompt()
for try await output in outputSequence {
    print(output)
    printPrompt()
}

For what itā€™s worth, that document you shared with us shows an example of runPrompt and run, and the simple and natural Swift equivalents might be:

private func runPrompt() async throws {
    var iterator = FileHandle.standardInput.bytes.lines.makeAsyncIterator()
    
    while true {
        print("> ")
        guard let line = try await iterator.next() else { break }
        run(line)
    }
}

private func run(_ source: String) {
    let scanner = Scanner(source)
    let tokens = scanner.scanTokens()
    
    // For now, just print the tokens.
    
    for token in tokens {
        print(token)
    }
}
2 Likes
4

Hey Robert, as always, thanks for chiming in on this topic :)

Heh, I know, but I figured using a Task at all was the wrong path, so I didn't invest much into doing it well

But I would contend that one really does not want a single sequence that intermingles the UI (the prompt) with the output strings.

The way I was thinking about it, the entire REPL constitutes part of the "view layer", as a mediator between the CLI and the underlying interpreter that actually lexes/parses/evaluates the strings. If you wanted a pure-data API to evaluate expressions, you would just call the Interpreter/VM APIs directly, rather than via the REPL.

My primary goal here (apart from just drinking as much async koolaid as I can, for learning's sake) was to be able to test the REPL's responses. My initial design consisted of the REPL taking a TextOutputStream and print()ing to it, but that had two downsides:

  1. The library has already has a _Stdout struct that conforms to TextOutputStream, but to my surprise, it's not public.
  2. To test this, I'd need to make a mock TextOutputStream that spies on the values being passed on it.

Neither of these are a huge deal, but I thought it'd be cool try to express this solely as stream transformations. It worked semi-decently actually. Here's what my test ended up like:

@testable import Lox
import Testing
import AsyncAlgorithms

@Suite
struct REPLTests {
	let inputLines = AsyncChannel<String>()
	let repl: Lox.REPL
	var output: AsyncThrowingChannel<String, Error>.Iterator

	init() {
		self.repl = Lox.REPL(lineSource: inputLines)
		self.output = repl.output.makeAsyncIterator()
	}
	
	@Test mutating func repl() async throws {
        // My interpreter only lexes for now, so that's what's being printed.
		try await expect("1",         prints: "[1.0, EOF]\n")
		try await expect("1 + \u{0}", prints: "Syntax error: uexpected character '\u{0}'\n")
		try await expect("\"abc",     prints: "Syntax error: unterminated string \"abc\"\n")
		try await expect("1 + 2",     prints: "[1.0, +, 2.0, EOF]\n")
		
		try await #expect(output.next() == "lox> "); inputLines.finish()
		try await #expect(output.next() == nil)
	}
	
	mutating private func expect(
		_ input: String, prints expectedOutput: String, sourceLocation: SourceLocation = #_sourceLocation
	) async throws {
		let prompt = "lox> "
		
		try await #expect(output.next() == prompt)
        await inputLines.send(input)
		try await #expect(output.next() == expectedOutput)
	}
}

It's kind neat. I can send in lines via the input channel, then see what "comes out" the other end via the output stream.

What do you think?

5

Can you share the package URL for this import Lox? Iā€™m happy to dig into your example, but would like to make sure Iā€™m using the right package. Or is it just some variation of the REPL from your original question? If so, could you share that with us?

I must confess, just glancing at the code, my reservations persist about the output sequence yielding both prompts and outputs.

And with no disrespect, Iā€™m not yet inclined to join you in the perspective that ā€œthe entire REPL constitutes part of the ā€˜view layerā€™ā€. It seems to me that the capturing of input and outputting prompts/outputs is the view layer, but that the ā€œtake a string and interpret itā€ is not. I still have a lingering suspicion that we are entangling view layer with the processing.

But I will suspend further comments until I can see this package and play around with it. And, regardless of my architectural apprehensions, if it works for you, then thatā€™s great.

In short, if this was just a pedagogic exercise to familiarize yourself with the available APIs, then we should just declare success. But if you are asking the deeper question, whether this is the optimal design for this particular problem, then thatā€™s a different matter.

2 Likes
6

It's in a really messy state right now, but I'll share in the next day or two once the interpreter is actually... interpreting :slight_smile:

2 Likes