SE-0382: Expression Macros

[...]

I did understand that DWARF does its part to support this, and I trust that LLDB will add support for reading these expansion buffers, and that its terminal interface will allow people to step into the exploded form of a macro invocation. It's just that I'm having difficulty seeing how the terminal interface can present this in a way that isn't worse than what it does now (which, as a consequence of the fact that it's a terminal interface, is pretty bad to begin with).

I guess LLDB could emit, to a temporary file, an exploded version of an entire source file that uses macros. Then when it "enters an expansion" from the collapsed file (the one the user wrote), it can set the IP's src location to one inside the explosion file. When you step beyond the explosion (or "step out", which in this context hopefully means "step to just beyond the invocation"), it would switch back.

This may be ok as long as the user doesn't want a view that's only partially exploded (in the case of an expansion that contains nested expansions, to skip over some of the nested expansions, but not all of them, and not the outermost expansion), which at some point they will.

So i keep coming back to this assertion: that a terminal interface is not a user interface. It's not comparable to the experience of using a GUI dedicated to debugging.

Although a user interface could be added to LLDB (at which point it should be easy to add a button or something that makes LLDB toggle between exploded & collapsed views of an invocation of a macro when you click on it with a mouse cursor), it's been something like 15 years since its introduction, and it hasn't happened yet, so I have to assume that it still won't have a UI when people start debugging code that contains uses of macros.

I think we can safely say that any argument with this premise is firmly outside the scope of this proposal review, possibly even the Swift open-source project in its entirety.

3 Likes

Well, no other scope was created to talk about the user's overall experience of trying to read macro-generated code (which will include their attempt to read macro-generated code in a debug session, which, for at least some inputs, will hinge on the debugger's UI or lack thereof), so I thought it might as well be mentioned here.

The partially-expanded approach is the one we're building toward. It retains all of the information needed to see what happened with macro expansion, and one can build a fully-expanded view on top of it. We already have this integrated in the diagnostics infrastructure, and I've shown the path to debugger integration.

A terminal interface is a user interface. It may not be your preferred one, but many folks do prefer to work at the terminal, and often that's all you get because you're ssh'd into some server somewhere. Many folks prefer to work in an IDE. Some folks prefer to use bespoke tools for specific tasks. That's all good. The role of the language is to make it possible to make those tools good, by having a model that admits good tooling. The role of the compiler and language-focused services like SourceKit is to provide the information needed to build that tooling. Then it's up to vendors to actually make that tooling, and users can pick the tooling that works best for them.

Your entire thrust seems to be that, without a bespoke graphical tool for debugging macro expansions, it's impossible to accept the language design. I reject that premise completely. If we can't meet folks where they are, with their preferred tools, we have a much bigger problem. So the LLDBs and IDEs of the world need to have access to the information they need about macro expansions, to build good tooling, and maybe some day someone builds the debugger GUI you want. But there is no way that will ever make sense as a prerequisite to language design.

Doug

13 Likes

Since we need to import SwiftSyntax to implement macro, so there is no easy way to use macro in a single script file?

We might be able to use one from a single script file, but you won't be able to define one.

Doug

2 Likes

I didn't say that and I didn't mean to imply it.

In terms of Conway's law: I do think that, within the overall LLVM system, there ought to be at least one subsystem that presents user interfaces like the ones I'm thinking of (ones that (1) are ported to all supported host platforms that support modern visual displays and (2) are vendor-neutral), but the sub-organization that would do the work of creating such a subsystem does not exist at this time. (If it did, I would have posted to their forum instead of this one.)

(I didn't think of it in these terms until last night though, hence the messy initial post of mine from a few days ago.)

This forum, in contrast, is dedicated to issues that are more obviously related to the design of the subsystem that performs translation from swift source code to machine code. My comments in previous posts are therefore out of scope here. So wrt responses to UI-related comments like this:

...I will not respond on this forum (even though i really want to).

...except for one thing:

Glad to see this! :+1:

Everything else that's in-scope LGTM, and I don't mean to hold you up any further. Please proceed!
(also: good work!)

1 Like

I think you're right.

Macro debugging experience is important. We can read code and send pull requests for Swift, LLDB, and SwiftSyntax, but not for Xcode, and the Swift core team is one of the few people who can talk directly to the Xcode team. I agree that tool-related topics are outside the scope of the language feature review, but I would like to know what kind of macro support is planned for Xcode, as we cannot know anything about Xcode.
For example, it would be nice to be able to see the source code after expanding a macro with a feature like the current Jump to Generated Interface.

6 Likes

Should createUniqueName() have an optional prefix / suffix / template parameter?

context.createUniqueName()            //-> `_unique1`
context.createUniqueName("maxValue")  //-> `_unique2_maxValue`

This would make it easier to understand the output of -dump-macro-expansions, etc.

11 Likes

I'm flip-flopping here again. @Snowy1803 you are totally right, we should stick with -> and allow () to be omitted at the use site. The revisions to the proposal, linked below, include that change. Sorry for the too-quick responses.

This came up a lot, and I've begun to think that this proposal isn't sufficiently complete without the ability to get source location information for a syntax node. For example, the excellent "power asserts" macro depends on having this information. I've added an API for this (on MacroExpansionContext) in the mini re-pitch.

Can't comment on what Xcode. However, I will point out that there is an expand macro refactoring in SourceKit now, which should be usable for any client of SourceKit-LSP, such as the VS Code Extension for Swift.

Yes, this is a good idea. I've pulled it into the mini re-pitch.

Doug

4 Likes

I had a little time to experiment with this. Implemented a custom URL literal.

The good news is that it works - I can implement compile-time checking of URLs to find obvious and non-obvious mistakes. Since I'm writing a package rather than a system library, the result can be a non-optional URL (which I know lots of people will be pleased to hear).

The diagnostics features are very useful. I can warn about URLs which parse but are not completely well-formed as written, and provide a fix-it with the repaired URL string. By ensuring that URLs in source code are well-formed, we can guard against certain kinds of spoofing attacks (e.g. from bidirectional text), and ensure that the URLs that people write are not ambiguous, so they can be reliably audited by others.

I added a special initialiser (for the macro's use only), ensuring that these also behave more like, say, an integer or string literal, and don't need parsing at runtime.

func urlForBand(_ bandName: String) -> WebURL {
  // Does not return an optional.
  // Does not re-parse the string each time.
  var url = #url("https://api.example.com/music/bands/")
  url.pathComponents += bandName
  return url
}

Unfortunately it doesn't get statically allocated, but there's now much, much less of a leap so it's finally a realistic proposition for the optimiser.

Overall, I'm very happy with that.


The less-good news is that, to be honest, building that macro was incredibly frustrating. I have plenty of other ideas I'd like to try, but it was quite draining to duck-tape things together. IMO, we should keep iterating on this implementation, and continue to fill out the rest of the macro story before we put things through review and make it official.

  1. SwiftPM integration is badly needed. Currently, you need to cobble together an Xcode project with your various modules, build settings including paths with substitutions, etc. All of the various pieces need to be very carefully aligned for this to work, otherwise for anybody using a macro, their code just fails to build. I worry that macros are going to get a reputation for fragility, unreliability, and build failures, unless we really nail this part of the design.

    You could say that it's at an early stage - but on the other hand, if it's going through review now, is it really that early? I don't think I'm being unfair by expressing concern that there isn't a single public pitch/sketch/prototype for the build-system side of things at this stage of the process.

    IMO, no macro features should pass review until we have a concrete design for this part of the puzzle. I think it makes a lot of sense to start with the concept of packages declaring targets with compile-time code, and then to build macros on top of that support.

    It's just really difficult to experiment with macros in realistic situations right now, so it's really difficult to critically evaluate.

  2. I'd like to confirm - will the toolchain ship with everything needed to build and run macros? Since this is a language feature, I feel I shouldn't need to download an additional package to get started using it. What if GitHub is down, or blocked in my country?

    IMO, the toolchain should include a full copy of swift-syntax, including any interface files or whatever else is needed for a complete experience (including syntax highlighting and documentation). I should be able to write a Swift project entirely offline and I should be able to define and use macros in it.

  3. I very much agree with what others have said about the swift-syntax library: it is very difficult to use. For simple modifications, it feels overwhelming. I've worked with spiritually similar APIs before - HTML DOM APIs spring to mind - but using this one, from Swift, felt very awkward. Here are a couple of issues that I encountered:

    i) The documentation is very bare when you find any, and because the API makes heavy use of overloads, when attempting to feel your way through the API you'll often encounter situations like this:

    image

    ii) To replace the contents of a string literal, I eventually (after a looong time) stumbled on this incantation. I think this is the right way to do it?:

    stringLiteral.withSegments([.stringSegment(StringSegment(content: url.description))])
    

    iii) Things like SimpleDiagnosticMessage need to be defined manually. Why is it not included? Also, just emitting a relatively simple diagnostic takes a gigantic amount of code with lots of nesting.

    iv) I need to keep wrapping things using Syntax(someNode) -- why does the library not use features such as protocols and generics to express these in terms of a common supertype?

    Overall, it feels like it needs a lot of work before it's ready to go in the hands of every Swift developer.

  4. Foundation APIs didn't work.

    <unknown>:0: warning: compiler plugin not loaded: /Users/karl/<...>/libWebURLMacros.dylib; loader error: dlopen(/Users/karl/<...>/libWebURLMacros.dylib, 0x0005): Symbol not found: (_$sSy10FoundationE37precomposedStringWithCanonicalMappingSSvg)
    Referenced from: '/Users/karl/<...>/libWebURLMacros.dylib'
    Expected in: '/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation'
    
  5. Compiler performance felt really poor and the macro implementation didn't rebuild when I made changes to it. I had to make changes to the user of the macro to prompt a rebuild. I probably messed something up with the configuration, but it's hard to know (see 1).

  6. If we're going with the idea that every macro should depend on its own version of swift-syntax, I worry that will also cause issues for the package ecosystem.

    As more packages use macros, I could easily end up with dozens of copies of swift-syntax, all at different versions, which all need to be downloaded and built independently. How long will it take to clean-build all of those?

  7. I'm not a fan of the @expression public macro syntax:

    @expression
    macro url(_: String) -> WebURL = #externalMacro(module: "WebURLMacros", type: "URLMacro")
    

    Typically, I interpret attributes as describe the thing they are applied to (@inlinable, @frozen, @escaping, @propertyWrapper). This thing is a declaration, not an expression.

    I see that it has now been replaced with @freestanding(expression). I'm not sure that's any better, honestly.

In conclusion, I think macros are an exciting and powerful feature which have the potential to be very popular, but when I consider the lack of design for the critical package manager/build system side of things, and the complexity of the swift-syntax library, I also see a lot of potential for frustration.

I feel this feature is somewhere along the right path, but it isn't quite there yet, and we shouldn't rush it.

27 Likes

SwiftPM-related aspects of macros were separated out of this proposal and will be reviewed separately, as the Evolution process for that topic doesn’t fall under the purview of the language workgroup and (independent of that consideration) because the combined proposal would have severely stretched the limits of a “reviewable unit” for a single proposal. It would also have made little sense to ask the community to review a separated SwiftPM proposal first since there would be no macros to, um, package.

Follow-on pitches and proposals proceed apace and there will be ample opportunity to revisit earlier proposals down the line where later insights come up.

Full decision notes from the language workgroup about this proposal are forthcoming.

Sure. The point I was making is that without even a working prototype of build system support, it's really difficult to extend existing libraries with macros and provide feedback based on that.

If there was some amount of build system support already, perhaps more library developers would be able to participate in the review process, and would have the confidence to try more ambitious things with this early implementation. My experience was that it felt so fragile that I was discouraged from experimenting. I spent a lot of time fighting the build system.

7 Likes

A question born out of a discussion over on the Predicate pitch—do we foresee any paths to allow a macro callsite, which might apply arbitrary transformations to its inputs, to support autocomplete?

Thinking about macros as a system designed to be able to replicate a compiler-integrated feature like result builders, which has rich type-checking needs. Could a macro-based system conceivably support a comparable developer experience for an IDE integration like autocomplete?

(Recognize that developer experience considerations like these may not be strictly topical for a proposal review, but wanted to ask!)

4 Likes

Hi all,

The proposal has been returned for revision focused on a specific number of improvements being re-pitched. The decision notes outline a number of issues which the language workgroup considered accepted in principle and our detailed rationale.

2 Likes

Perhaps they will be sufficiently mitigated in practice. I do think that the syntactic and conceptual complexity of implementing these macros is a good thing to discourage excess usage. In particular, I’m curious to see how well the affordances for creating unique names will play out. The problem of nonobvious identifier conflicts could arise when a developer edits a Swift file that contains a macro that expands to declare some new identifier to add a new, separate identifier that conflicts in an editor environment that doesn’t have sufficient tooling to catch the error (such as a simple command-line editor like nano). Maybe this is a bit too contrived, but I generally think that errors and conflicts in source code should be as obvious as possible just by looking at plain text.

Should createUniqueName(_:) be named makeUniqueName(_:) instead to follow the "factory method" naming guideline?

2 Likes

A second review has been started. Maybe that is a better place to post your question.

1 Like