Swift Lexical Lookup for referenced stuff located outside scope/current file?

I want to lookup variables from a macro expansion in one of my projects (GitHub - RandomHashTags/swift-htmlkit: Write HTML using Swift Macros.). The purpose of this is to replace current interpolation (which is executed at runtime) with the compile-time equivalent of the referenced variable, if possible.

Example:

  • I have a static let test:StaticString declared in file 1 (in struct File1)
  • I use the macro in file 2, referencing the static string in file 1 (as File1.test)

Is there a way to do this? I know the SwiftLexicalLookup is experimental and it definitely has the capabilities to do such a thing, but the MacroExpansionContext's lexicalContext is currently limited to the file it is referenced in (at least in my testing).

I did think of a workaround using a String array as file references where the variables are located and using SwiftParser to find them, but that would require a lot of manual input & file tracking and is kinda hacky.

If it is currently not possible, should I create a PR or Evolution Pitch adding this to swift-syntax?

No, this is not supported and it’s by design. If code in one file could influence the macro expansion in another file, then all files in a module would need to be re-compiled whenever any other file is modified, which would significantly slow down incremental builds. It would also make macros a lot harder to reason about because code anywhere in your codebase might influence your macro, making any kind of local reasoning impossible.

1 Like

I understand the reasoning behind the limitation. The problem I have with it is:

The declaration I want to lookup is already represented syntactically in the macro arguments. I am looking up a relevant declaration known at compile time that does not change based on where the macro is used, not random code in a random file for some random reason. I can currently lookup the relevant declarations using SwiftParser, but that parses the entire file it is declared in.

An endgame solution (imo) would be to have the declaration lexical context for the macro arguments with nothing stripped, when applicable.

You shouldn’t read contents of other files inside a macro (and at least on macOS you shouldn’t be able to because the macro is running inside a sandbox, I’m not sure about other platforms at the moment). @beccadax briefly talks about this in Expand on Swift macros - WWDC23 - Videos - Apple Developer

Maybe an alternative approach is combining a build tool plugin with macros used as annotations?

Can confirm it doesn't work on macOS, but does on Linux. Most servers run on Linux so I don't see a problem using it for now.

I wouldn't need to do this if I had the declared types' syntactic representation (or unstripped declaration lexical context) for the macro arguments, which is known at compile time and to the macro expansion where it is expanded (type-safety is enforced, so we know without a doubt at compile time, in my example below, that Shrek.isLove.rawValue is ExpressibleByStringLiteral.

The problem as a real-world example
  • Macro in question
@freestanding(expression)
public macro html<T: ExpressibleByStringLiteral>(attributes: [HTMLElementAttribute] = [], xmlns: T? = nil, _ innerHTML: T...) -> T
  • HTMLElementAttribute in question
public enum HTMLElementAttribute {
    case title((any ExpressibleByStringLiteral)? = nil)
}
  • Problem (I want StaticString)
extension InterpolationTests {
    enum Shrek : String {
        case isLove, isLife
    }
    @Test func third_party_enum() {
        var string:String = #html(attributes: [.title(Shrek.isLove.rawValue)]) // allowed, but runtime required
        #expect(string == "<!DOCTYPE html><html title=\"isLove\"></html>")

        // my current solution
        var staticString:StaticString = #html(lookupFiles: ["/path/where/shrek/declaration/is.swift", "/path/to/referenced/declaration/in/another/file.swift"], attributes: [.title(Shrek.isLove.rawValue)])
        #expect(staticString.description == "<!DOCTYPE html><html title=\"isLove\"></html>")
    }
}

My solution using SwiftParser (and looking up files) works, albeit only on Linux/non-macOS, in which it replaces Shrek.isLove.rawValue with "isLove". This is perfect for a server environment that compiles it down to a StaticString for better performance. A real world example of this would be for a web server serving static web pages where the Shrek.isLove is accessed throughout the project.

I also have a Localization system written in Swift that returns a StaticString based on the input language for a certain text. Plugging in the localized StaticString in the html macro is currently impossible without converting it to a String at runtime or using my solution.

In my real-world example, and in the project where the problem originates, I use freestanding macros. Having a build tool plugin and attached macros would not fix this problem, at least at first thought.

I'd consider being able to read the contents of other files on any platform from a macro as a bug which is liable to stop working at any time.

Have you measured an actual improvement in performance? In the example you give, I would be surprised if there is.

The performance improvement is a lot. See the benchmark results. In this small real-world example, the static string still performs better than the equivalent string interpolated one. But you would use more than one of these declarations when building a localized web page, which all contribute to less and less performance.

Whether or not my solution gets patched, there is currently no alternative solution.

Do you have this data to share? I mean specifically the effect of replacing Shrek.isLove.rawValue with "isLove".

0 interpolation vs 1, on my 7800x3D cpu:

No interpolation is 1.4x faster than 1 interpolation (on average). Or put another way, 1 interpolation cost 28.6% reduction in performance.

Interesting. Sounds like there is a performance improvement opportunity for the compiler and/or standard library. Sounds like you're basically making an ad-hoc optimization pass to patch around it in the form of a macro (which is not meant to be a supported type of macro).

1 Like

Double checking: this is with optimizations turned on (say, -O -wmo)?

With no optimizations turned on explicitly, using this exact command for the Benchmarks (I have not tested it in Xcode with optimization flags, as I mainly use VS Code):

swift package -c release --allow-writing-to-package-directory benchmark --target Benchmarks --metric throughput --format jmh

That is with optimizations turned on.
Also, package-benchmark actually automatically builds your benchmark in release configuration, even without -c release.

1 Like