Correct primitive for awaiting one AsyncSequence, and sending to another?

Using Swift
,
1

I'm working through the Crafting Interpreters book, and for extra fun, I'm playing with using Async Swift to coordinate the various parts together.

One example is the REPL. Fundamentally, it's just a function ot type (AsyncSequence<String>) -> AsyncSequence<String> which takes input lines, and produces the evaluation results to be printed. AsyncSequence.map seems like a good fit for this, but with one short-coming: The REPL should be able to print the prompt (e.g. "> ") before the first line has been entered.

What's the correct way to do this transformation? I was able to make it with by pulling in AsyncAlgorithms for AsyncChannel and combining that with a Task. Is there a simpler way?

import AsyncAlgorithms

struct REPL { // I don't really need this struct, could just be a function I suppose.
	let output = AsyncThrowingChannel<String, Error>()
	
	init<S: AsyncSequence>(lineSource: S) where S: Sendable, S.Element == String {
		Task { [output] in
			
			let prompt = "> "
			await output.send(prompt)
			
			for try await line in lineSource {
				// This is where the interpreter would actually evaluate the line
				let result = "process(\(line))\n"
				
				await output.send(result)
				await output.send(prompt)
			}
			
			output.finish()
		}
	}
}

let repl = REPL(lineSource: FileHandle.standardInput.bytes.lines)
		
for try await output in repl.output {
    print(output, terminator: "")
    try FileHandle.standardOutput.synchronize()
}
2 Likes
3

A couple of reactions:

  1. Using unstructured concurrency with Task {…}, without handling cancellation is a bit of an anti-pattern. [I wrote a long and tedious answer explaining how to handle this properly, but in retrospect, that was tangential to the real question here, so I removed that reply.]

  2. Having a sequence that yielded both the output and the next prompt is also a bit of an anti-pattern. If you really did that, I’d yield an enumeration with associated values for either the prompt string or the repl output string, so the consumer could differentiate between them.

    But I would contend that one really does not want a single sequence that intermingles the UI (the prompt) with the output strings. (In your case, your consumer is just printing these strings, so it didn’t matter too much, but if it was doing anything else, this process of sending UI prompt and REPL output both as simple strings would make it very difficult to differentiate between the two.)

Personally, this doesn’t strike me as a great use-case for the AsyncSequence from another AsyncSequence. I would just iterate through the sequence:

printPrompt()

for try await input in FileHandle.standardInput.bytes.lines {
    let output = try await replOutput(for: input)
    print(output)
    printPrompt()
}

In answer to the question about how to create an AsyncSequence from another AsyncSequence, as you noted, one would generally just map. For example:

let outputSequence = FileHandle.standardInput.bytes.lines.map {
    try await replOutput(for: $0)
}

printPrompt()
for try await output in outputSequence {
    print(output)
    printPrompt()
}

For what it’s worth, that document you shared with us shows an example of runPrompt and run, and the simple and natural Swift equivalents might be:

private func runPrompt() async throws {
    var iterator = FileHandle.standardInput.bytes.lines.makeAsyncIterator()
    
    while true {
        print("> ")
        guard let line = try await iterator.next() else { break }
        run(line)
    }
}

private func run(_ source: String) {
    let scanner = Scanner(source)
    let tokens = scanner.scanTokens()
    
    // For now, just print the tokens.
    
    for token in tokens {
        print(token)
    }
}
3 Likes
4

Hey Robert, as always, thanks for chiming in on this topic :)

Heh, I know, but I figured using a Task at all was the wrong path, so I didn't invest much into doing it well

But I would contend that one really does not want a single sequence that intermingles the UI (the prompt) with the output strings.

The way I was thinking about it, the entire REPL constitutes part of the "view layer", as a mediator between the CLI and the underlying interpreter that actually lexes/parses/evaluates the strings. If you wanted a pure-data API to evaluate expressions, you would just call the Interpreter/VM APIs directly, rather than via the REPL.

My primary goal here (apart from just drinking as much async koolaid as I can, for learning's sake) was to be able to test the REPL's responses. My initial design consisted of the REPL taking a TextOutputStream and print()ing to it, but that had two downsides:

  1. The library has already has a _Stdout struct that conforms to TextOutputStream, but to my surprise, it's not public.
  2. To test this, I'd need to make a mock TextOutputStream that spies on the values being passed on it.

Neither of these are a huge deal, but I thought it'd be cool try to express this solely as stream transformations. It worked semi-decently actually. Here's what my test ended up like:

@testable import Lox
import Testing
import AsyncAlgorithms

@Suite
struct REPLTests {
	let inputLines = AsyncChannel<String>()
	let repl: Lox.REPL
	var output: AsyncThrowingChannel<String, Error>.Iterator

	init() {
		self.repl = Lox.REPL(lineSource: inputLines)
		self.output = repl.output.makeAsyncIterator()
	}
	
	@Test mutating func repl() async throws {
        // My interpreter only lexes for now, so that's what's being printed.
		try await expect("1",         prints: "[1.0, EOF]\n")
		try await expect("1 + \u{0}", prints: "Syntax error: uexpected character '\u{0}'\n")
		try await expect("\"abc",     prints: "Syntax error: unterminated string \"abc\"\n")
		try await expect("1 + 2",     prints: "[1.0, +, 2.0, EOF]\n")
		
		try await #expect(output.next() == "lox> "); inputLines.finish()
		try await #expect(output.next() == nil)
	}
	
	mutating private func expect(
		_ input: String, prints expectedOutput: String, sourceLocation: SourceLocation = #_sourceLocation
	) async throws {
		let prompt = "lox> "
		
		try await #expect(output.next() == prompt)
        await inputLines.send(input)
		try await #expect(output.next() == expectedOutput)
	}
}

It's kind neat. I can send in lines via the input channel, then see what "comes out" the other end via the output stream.

What do you think?

5

Can you share the package URL for this import Lox? I’m happy to dig into your example, but would like to make sure I’m using the right package. Or is it just some variation of the REPL from your original question? If so, could you share that with us?

I must confess, just glancing at the code, my reservations persist about the output sequence yielding both prompts and outputs.

And with no disrespect, I’m not yet inclined to join you in the perspective that “the entire REPL constitutes part of the ‘view layer’”. It seems to me that the capturing of input and outputting prompts/outputs is the view layer, but that the “take a string and interpret it” is not. I still have a lingering suspicion that we are entangling view layer with the processing.

But I will suspend further comments until I can see this package and play around with it. And, regardless of my architectural apprehensions, if it works for you, then that’s great.

In short, if this was just a pedagogic exercise to familiarize yourself with the available APIs, then we should just declare success. But if you are asking the deeper question, whether this is the optimal design for this particular problem, then that’s a different matter.

2 Likes
6

It's in a really messy state right now, but I'll share in the next day or two once the interpreter is actually... interpreting :slight_smile:

2 Likes
7

OK, for giggles and grins, I’ve gone through chapters 1-4 of that book, and I’ve creating my own Swift mockup. As is the state of Nystrom’s tool by the end of chapter 4, my demonstration only displays the sequence of tokens that were parsed. But this is sufficient for me to deeply understand your user-case and illustrate why the > prompt should not be part of the interpreter output, itself.

I stand by my prior conceptual observations, but my example with two clients, one a CLI and another a GUI, should hopefully drive home the point. As you can see, the > prompts may be useful in the CLI rendition, but they’re wholly unnecessary/undesirable in the SwiftUI client. This view layer logic (the prompt) does not belong in the Slox framework itself, because some clients may want to use a prompt whereas others may not.

Hopefully this illustrates more concretely why the > prompt should not be yielded by the interpreter, itself, but only displayed by the view layer (and only on those user interfaces where it makes any sense).

2 Likes
8

Since the production of values isn't completely tied to the consumption of values map isn't the right fit here. However, we should be able to model this without the need of an unstructured task by merging two async sequences together.

let prompt = ["Welcome to the REPL", "Please enter your prompt"].async
let output = FileHandle.standardInput.bytes.lines
    .map { line in
        return try await process(line)
    }

for try await outputLine in merge(prompt, output) {
    print(outputLine)
}
9

@FranzBusch

First, I agree with you regarding how to build one asynchronous sequence from another: As I had buried in an earlier answer, if you want to create an asynchronous sequence from another asynchronous sequence, map, as you have here, is the right tool for that. And that eliminates the need for unstructured concurrency, too. Absolutely right. And you can marry this with merge, too, as you have in your answer.

But his notion was a CLI app that will present a prompt, >, before accepting a line of input. And after it generates the output for that line of input, it will show the > prompt and wait for the next line to be entered. Etc. His question was how to interleave the prompts with the output.

But, the deeper question, IMHO, is whether this is the right pattern at all for this particular problem, at all. Does the > prompt belong in (or merged with) the output sequence at all? The OP’s pattern of alternating one interpreter output with one prompt has several problems:

  1. As I’ve belabored above, this conflates view logic and non-view logic.

  2. It also makes a fairly simplistic assumption that every input (a program/script to be run) will result in exactly one interpreter output. But as Alexander will likely realize as he progresses through this book, a single input (a program/script to be interpreted) will generally produce a stream of multiple yielded outputs. This assumption of alternating between yielded interpreter outputs with a respective UI prompt is unlikely to hold up. It just isn’t how interpreters work.

2 Likes
10

Hey Robert! It took me a little while, but I got as far as implementing simple expression evaluation, like 1 + 2 = 3. Here's how my REPL looks like now: Sources/REPL.swift · 2eddf541fa8bcae09abe0acc2c7169698729730d · AMomchilov / Public / Lox · GitLab

It creates a new Lox.VM, and repeatedly asks it to vm.run(line), getting a Lox.Value back. This is similar to the way you might embed a Python/Lua/Ruby VM into say, a C project.

The REPL takes that Lox.Value, renders it to a string using its description, and sends that to the output stream, along with the next "lox>" prompt.

Indeed, sending the result and the prompt in two separate steps wasn't great. It made testings much harder than necessary. Instead, for every input line in, I emit one output string with everything in it. Check out the tests

11

I'm really grateful that you jumped into implementing this yourself (sorry to nerd snipe you!). There's lots for me to learn from comparing notes.

Our REPLs are actually very similar (mine, yours). The main difference is that yours calls print directly, whereas mine pushes results to an AsyncChannel.

In my impl, the actual stdout printing is done by a Swift Argument Parser command called REPLCommand (invoked via lox repl, similar to swift repl, as opposed to e.g. lox your/script.lox). It subscribes to the REPL's output channel, and prints each string that it gets.

The point of this channel was decoupling; my test can feed lines into the REPL, and observe the output that it would have produced.

Hmm from a few of these quotes, I get the sense there's a miscommunication:

(FWIW, I agree with all of these statements.)

To clarify, when I said "REPL," I was narrowly referring to one mode of interacting with an "interpreter" via the CLI, as opposed to e.g. executing a whole script file. You can call my interpreter directly, give it an expression, and it'll give you a Lox.Value back with no prompt. The string representation of the result and the lox> prompt, are produced by my REPL ("view layer"), not my interpreter (which purely does the actual lex+parse+eval).

I haven't implemented statements or side-effects yet (that's coming next in chapter 8), so my interpreter is limited to executing single-line (well, single expression) scripts. But I imagine things like (Lox) print statements would work the same way: the VM would have a stdout: AsyncChannel<String>, and the REPL would receive that printed output and pipe it to its own output channel. Once evaluation is done and the interpretter returns the final result, it's business as usual for the REPL: render the result to string, append the prompt, and emit that to the output channel.