RFC: Static build configuration in SwiftIfConfig and build configurations for macro expansion contexts

Hi all,

I have a pull request up to introduce StaticBuildConfiguration. This type is a simple, Codable struct that contains enough data to answer any BuildConfiguration-based queries. The general idea is that we'll have a way to export this configuration as JSON from the Swift compiler, and that the compiler will also provide it to macros via the MacroExpansionContext. Those parts will be coming later: for now, StaticBuildConfiguration is used in testing, and a JSON file containing one can be passed in to swift-parser-cli with the new --build-configuration flag: the build configuration will be used to remove all inactive regions from the parsed source file so that one can see, e.g., the resulting

Here's the full interface to StaticBuildConfiguration, copied out of the PR:

/// A statically-determined build configuration that can be used with any
/// API that requires a build configuration. Static build configurations can
/// be (de-)serialized via Codable.
public struct StaticBuildConfiguration: Codable {
  public init(
    customConditions: Set<String> = [],
    features: Set<String> = [],
    attributes: Set<String> = [],
    importedModules: [String: [VersionedImportModule]] = [:],
    targetOSNames: Set<String> = [],
    targetArchitectures: Set<String> = [],
    targetEnvironments: Set<String> = [],
    targetRuntimes: Set<String> = [],
    targetPointerAuthenticationSchemes: Set<String> = [],
    targetObjectFileFormats: Set<String> = [],
    targetPointerBitWidth: Int = 64,
    targetAtomicBitWidths: [Int] = [],
    endianness: Endianness = .little,
    languageVersion: VersionTuple,
    compilerVersion: VersionTuple
  ) {
    self.customConditions = customConditions
    self.features = features
    self.attributes = attributes
    self.importedModules = importedModules
    self.targetOSNames = targetOSNames
    self.targetArchitectures = targetArchitectures
    self.targetEnvironments = targetEnvironments
    self.targetRuntimes = targetRuntimes
    self.targetPointerAuthenticationSchemes = targetPointerAuthenticationSchemes
    self.targetObjectFileFormats = targetObjectFileFormats
    self.targetPointerBitWidth = targetPointerBitWidth
    self.targetAtomicBitWidths = targetAtomicBitWidths
    self.endianness = endianness
    self.languageVersion = languageVersion
    self.compilerVersion = compilerVersion
  }

  /// The set of custom conditions that are present and can be used with `#if`.
  ///
  /// Custom build conditions can be set by the `-D` command line option to
  /// the Swift compiler. For example, `-DDEBUG` sets the custom condition
  /// named `DEBUG`, which could be checked with, e.g.,
  ///
  /// ```swift
  /// #if DEBUG
  /// // ...
  /// #endif
  /// ```
  public var customConditions: Set<String> = []

  /// The set of language features that are enabled.
  ///
  /// Features are determined by the Swift compiler, language mode, and other
  /// options such as `--enable-upcoming-feature`, and can be checked with
  /// the `hasFeature` syntax, e.g.,
  ///
  /// ```swift
  /// #if hasFeature(VariadicGenerics)
  /// // ...
  /// #endif
  /// ```
  public var features: Set<String> = []

  /// The set of attributes that are available.
  ///
  /// Attributes are determined by the Swift compiler. They can be checked
  /// with `hasAttribute` syntax, e.g.,
  ///
  /// ```swift
  /// #if hasAttribute(available)
  /// // ...
  /// #endif
  /// ```
  public var attributes: Set<String> = []

  /// The set of modules that can be imported, and their version and underlying
  /// versions (if known). These are organized by top-level module name,
  /// with paths (to submodules) handled internally.
  public var importedModules: [String: [VersionedImportModule]] = [:]

  /// The active target OS names, e.g., "Windows", "iOS".
  public var targetOSNames: Set<String> = []

  /// The active target architectures, e.g., "x64_64".
  public var targetArchitectures: Set<String> = []

  /// The active target environments, e.g., "simulator".
  public var targetEnvironments: Set<String> = []

  /// The active target runtimes, e.g., _ObjC.
  public var targetRuntimes: Set<String> = []

  /// The active target's pointer authentication schemes, e.g., "arm64e".
  public var targetPointerAuthenticationSchemes: Set<String> = []

  /// The active target's object file formats, e.g., "COFF"
  public var targetObjectFileFormats: Set<String> = []

  /// The bit width of a data pointer for the target architecture.
  ///
  /// The target's pointer bit width (which also corresponds to the number of
  /// bits in `Int`/`UInt`) can only be queried with the experimental syntax
  /// `_pointerBitWidth(_<bitwidth>)`, e.g.,
  ///
  /// ```swift
  /// #if _pointerBitWidth(_32)
  /// // 32-bit system
  /// #endif
  /// ```
  public var targetPointerBitWidth: Int = 64

  /// The atomic bit widths that are natively supported by the target
  /// architecture.
  ///
  /// This lists all of the bit widths for which the target provides support
  /// for atomic operations. It can be queried with
  /// `_hasAtomicBitWidth(_<bitwidth>)`, e.g.
  ///
  /// ```swift
  /// #if _hasAtomicBitWidth(_64)
  /// // 64-bit atomics are available
  /// #endif
  public var targetAtomicBitWidths: [Int] = []

  /// The endianness of the target architecture.
  ///
  /// The target's endianness can onyl be queried with the experimental syntax
  /// `_endian(<name>)`, where `<name>` can be either "big" or "little", e.g.,
  ///
  /// ```swift
  /// #if _endian(little)
  /// // Swap some bytes around for network byte order
  /// #endif
  /// ```
  public var endianness: Endianness = .little

  /// The effective language version, which can be set by the user (e.g., 5.0).
  ///
  /// The language version can be queried with the `swift` directive that checks
  /// how the supported language version compares, as described by
  /// [SE-0212](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0212-compiler-version-directive.md). For example:
  ///
  /// ```swift
  /// #if swift(>=5.5)
  /// // Hooray, we can use tasks!
  /// ```
  public var languageVersion: VersionTuple

  /// The version of the compiler (e.g., 5.9).
  ///
  /// The compiler version can be queried with the `compiler` directive that
  /// checks the specific version of the compiler being used to process the
  /// code, e.g.,
  ///
  /// ```swift
  /// #if compiler(>=5.7)
  /// // Hoorway, we can implicitly open existentials!
  /// #endif
  public var compilerVersion: VersionTuple
}

extension StaticBuildConfiguration: BuildConfiguration {
  /// Determine whether a given custom build condition has been set.
  ///
  /// Custom build conditions can be set by the `-D` command line option to
  /// the Swift compiler. For example, `-DDEBUG` sets the custom condition
  /// named `DEBUG`, which could be checked with, e.g.,
  ///
  /// ```swift
  /// #if DEBUG
  /// // ...
  /// #endif
  /// ```
  ///
  /// - Parameters:
  ///   - name: The name of the custom build condition being checked (e.g.,
  ///     `DEBUG`.
  /// - Returns: Whether the custom condition is set.
  public func isCustomConditionSet(name: String) -> Bool {
    // "$FeatureName" can be used to check for a language feature.
    if let firstChar = name.first, firstChar == "$" {
      return features.contains(String(name.dropFirst()))
    }

    return customConditions.contains(name)
  }

  /// Determine whether the given feature is enabled.
  ///
  /// Features are determined by the Swift compiler, language mode, and other
  /// options such as `--enable-upcoming-feature`, and can be checked with
  /// the `hasFeature` syntax, e.g.,
  ///
  /// ```swift
  /// #if hasFeature(VariadicGenerics)
  /// // ...
  /// #endif
  /// ```
  ///
  /// - Parameters:
  ///   - name: The name of the feature being checked.
  /// - Returns: Whether the requested feature is available.
  public func hasFeature(name: String) -> Bool {
    features.contains(name)
  }

  /// Determine whether the given attribute is available.
  ///
  /// Attributes are determined by the Swift compiler. They can be checked
  /// with `hasAttribute` syntax, e.g.,
  ///
  /// ```swift
  /// #if hasAttribute(available)
  /// // ...
  /// #endif
  /// ```
  ///
  /// - Parameters:
  ///   - name: The name of the attribute being queried.
  /// - Returns: Whether the requested attribute is supported.
  public func hasAttribute(name: String) -> Bool {
    attributes.contains(name)
  }

  /// Determine whether a module with the given import path can be imported,
  /// with additional version information.
  ///
  /// The availability of a module for import can be checked with `canImport`,
  /// e.g.,
  ///
  /// ```swift
  /// #if canImport(UIKit)
  /// // ...
  /// #endif
  /// ```
  ///
  /// There is an experimental syntax for providing required module version
  /// information, which will translate into the `version` argument.
  ///
  /// - Parameters:
  ///   - importPath: A nonempty sequence of (token, identifier) pairs
  ///     describing the imported module, which was written in source as a
  ///     dotted sequence, e.g., `UIKit.UIViewController` will be passed in as
  ///     the import path array `[(token, "UIKit"), (token, "UIViewController")]`.
  ///   - version: The version restriction on the imported module. For the
  ///     normal `canImport(<import-path>)` syntax, this will always be
  ///     `CanImportVersion.unversioned`.
  /// - Returns: Whether the module can be imported.
  public func canImport(importPath: [(TokenSyntax, String)], version: CanImportVersion) -> Bool {
    // If we don't have any record of the top-level module, we cannot import it.
    guard let topLevelModuleName = importPath.first?.1,
      let versionedImports = importedModules[topLevelModuleName]
    else {
      return false
    }

    // Match on submodule path.
    let submodulePath = Array(importPath.lazy.map(\.1).dropFirst())
    guard let matchingImport = versionedImports.first(where: { $0.submodulePath == submodulePath }) else {
      return false
    }

    switch version {
    case .unversioned:
      return true

    case .version(let expectedVersion):
      guard let actualVersion = matchingImport.version else {
        return false
      }

      return actualVersion >= expectedVersion

    case .underlyingVersion(let expectedVersion):
      guard let actualVersion = matchingImport.underlyingVersion else {
        return false
      }

      return actualVersion >= expectedVersion
    }
  }

  /// Determine whether the given name is the active target OS (e.g., Linux, iOS).
  ///
  /// The target operating system can be queried with `os(<name>)`, e.g.,
  ///
  /// ```swift
  /// #if os(Linux)
  /// // Linux-specific implementation
  /// #endif
  /// ```
  ///
  /// - Parameters:
  ///   - name: The name of the operating system being queried, such as `Linux`,
  ///   `Windows`, `macOS`, etc.
  /// - Returns: Whether the given operating system name is the target operating
  ///   system, i.e., the operating system for which code is being generated.
  public func isActiveTargetOS(name: String) -> Bool {
    targetOSNames.contains(name)
  }

  /// Determine whether the given name is the active target architecture
  /// (e.g., x86_64, arm64).
  ///
  /// The target processor architecture can be queried with `arch(<name>)`, e.g.,
  ///
  /// ```swift
  /// #if arch(x86_64)
  /// // 64-bit x86 Intel-specific code
  /// #endif
  /// ```
  ///
  /// - Parameters:
  ///   - name: The name of the target architecture to check.
  /// - Returns: Whether the given processor architecture is the target
  ///   architecture.
  public func isActiveTargetArchitecture(name: String) -> Bool {
    targetArchitectures.contains(name)
  }

  /// Determine whether the given name is the active target environment (e.g., simulator)
  ///
  /// The target environment can be queried with `targetEnvironment(<name>)`,
  /// e.g.,
  ///
  /// ```swift
  /// #if targetEnvironment(simulator)
  /// // Simulator-specific code
  /// #endif
  /// ```
  ///
  /// - Parameters:
  ///   - name: The name of the target environment to check.
  /// - Returns: Whether the target platform is for a specific environment,
  ///   such as a simulator or emulator.
  public func isActiveTargetEnvironment(name: String) -> Bool {
    targetEnvironments.contains(name)
  }

  /// Determine whether the given name is the active target runtime (e.g., _ObjC vs. _Native)
  ///
  /// The target runtime can only be queried by an experimental syntax
  /// `_runtime(<name>)`, e.g.,
  ///
  /// ```swift
  /// #if _runtime(_ObjC)
  /// // Code that depends on Swift being built for use with the Objective-C
  /// // runtime, e.g., on Apple platforms.
  /// #endif
  /// ```
  ///
  /// The only other runtime is "none", when Swift isn't tying into any other
  /// specific runtime.
  ///
  /// - Parameters:
  ///   - name: The name of the runtime.
  /// - Returns: Whether the target runtime matches the given name.
  public func isActiveTargetRuntime(name: String) -> Bool {
    targetRuntimes.contains(name)
  }

  /// Determine whether the given name is the active target pointer authentication scheme (e.g., arm64e).
  ///
  /// The target pointer authentication scheme describes how pointers are
  /// signed, as a security mitigation. This scheme can only be queried by
  /// an experimental syntax `_ptrath(<name>)`, e.g.,
  ///
  /// ```swift
  /// #if _ptrauth(arm64e)
  /// // Special logic for arm64e pointer signing
  /// #endif
  /// ```
  /// - Parameters:
  ///   - name: The name of the pointer authentication scheme to check.
  /// - Returns: Whether the code generated for the target will use the given
  /// pointer authentication scheme.
  public func isActiveTargetPointerAuthentication(name: String) -> Bool {
    targetPointerAuthenticationSchemes.contains(name)
  }

  /// Determine whether the given name is the active target object file format (e.g., ELF).
  ///
  /// The target object file format can only be queried by an experimental
  /// syntax `_objectFileFormat(<name>)`, e.g.,
  ///
  /// ```swift
  /// #if _objectFileFormat(ELF)
  /// // Special logic for ELF object file formats
  /// #endif
  /// ```
  /// - Parameters:
  ///   - name: The name of the object file format.
  /// - Returns: Whether the target object file format matches the given name.
  @_spi(ExperimentalLanguageFeatures)
  public func isActiveTargetObjectFileFormat(name: String) -> Bool {
    targetObjectFileFormats.contains(name)
  }
}

/// Information about a potentially-versioned import of a given module.
///
/// Each instance of this struct is associated with a top-level module of some
/// form. When the submodule path is empty, it refers to the top-level module
/// itself.
public struct VersionedImportModule: Codable {
  /// The submodule path (which may be empty) from the top-level module to
  /// this specific import.
  public var submodulePath: [String] = []

  /// The version that was imported, if known.
  public var version: VersionTuple? = nil

  /// The version of the underlying Clang module, if there is one and it is
  /// known.
  public var underlyingVersion: VersionTuple? = nil
}

Doug

9 Likes

I'm really curious about the importedModules field here. Is it only those modules that are directly imported by code in the module where the macro is expanded, or does it include any transitive modules that must also be loaded by those direct dependencies? (Or, is it the full set of importable modules available by scanning search paths, since that would be necessary to meet the semantics of canImport?)

If it's not just direct dependencies, then for large builds where a compilation might have hundreds or even a few thousand modules transitively loadable, I'd be concerned about the runtime cost of sending this much data along the wire to the macro and then decoding it.

Yeah, this one is really tricky, because determining the full set of canImport'able modules is hard to do up front. It's also hard to reason about things like submodules and their relationship to the top-level module. I was imagining that the compiler would effectively produce "those modules where we did canImport checks", but that's deeply unsatisfying.

One answer here is that I should remove this field, and have canImport on a StaticBuildConfiguration always throw an error.

Yeah. To be honest, I'm a little concerned about the feature and attribute lists being large enough to require us to do some more intelligent caching protocol with the macro implementations. But the list of imported modules could start larger and grow much faster.

Doug

1 Like

I realize that I should have provided the API changes for the macro expansion context as well, since it's all about this one feature. The MacroExpansionContext would gain the following property:

  /// Returns the build configuration that will be used with the generated
  /// source code. Macro implementations can use this information to determine
  /// the context into which they are generating code.
  ///
  /// When the build configuration is not known, this returns nil.
  var buildConfiguration: (any BuildConfiguration)? { get }

Macros can query this build configuration directly or use other facilities of SwiftIfConfig if they like.

Doug

I wonder if we could add a parameter to the @freestanding/@attached attribute on macros that would describe optional build configuration information that a macro could opt into, so that all requests aren't burdened with data they won't use. That wouldn't solve the potential performance problem for macros that do use it, though.

This is probably too complex to be on the table, but I also wonder if it would be possible to have bidirectional communication between the compiler and macro that's more than just a single request/response cycle. The macro could then actively pull the information it needs instead of the compiler having to send everything up front.

(That also feels like an appealing way to eventually get other complex information into macros, like properties of the type-checked AST...)

We absolutely could make this information opt-in from the macro side. That would mean we don't pay the cost unless a macro needs it. There are theoretical other benefits, such as allowing us to cache macro expansions better (because the build configuration wouldn't be part of the cache key unless specifically requested).

This wouldn't impact the API here; it would mean that we have a language extension proposal that goes along with it (which is fine!).

I was originally thinking along these lines, but more and more and I find that I'm afraid of this direction. It adds a ton of complexity to the implementation, and makes it harder to meaningfully test macros outside of the interaction with the compiler.

Doug

1 Like

I have a mild concern about future extensibility. What if the language introduces additional kinds of platform conditions? Presumably, we'd add new Optional fields to StaticBuildConfiguration , but that would leave us with a mix of required and optional fields in an inconsistent way. Would it make more sense to declare all of them as optional from the start?

(And any #if evaluation using a field of nil would be considered false)

Yes, we would add new optional fields. I don't think we should make everything optional in advance, though: it penalizes users of this type based on a potential for future expansion, when most of those uses won't care about future expansions. I expect many will just take whatever build configuration exists (e.g., because the compiler gives it to them) without looking at specific fields.

One option for nil optional fields is that the corresponding BuildConfiguration function than throw when there is no data, so clients can understand why a particular #if evaluated false. This is what we're doing with canImport now (based on Tony's feedback above).

Doug

1 Like

I'm sure it comes as no shock to you, Doug, to know Swift Testing could use this! :grimacing:

2 Likes

Now that I have this implemented... it's 3.5kb for the JSON-serialized configuration. That's not awful, and because it's consistent across the whole module, it would not be hard to make sure we only transfer it once. At this point, I'm inclined to make this information always be available.

Then you will likely be interested in the compiler integration of this feature: [Macros] Pass a static build configuration to macro implementations by DougGregor · Pull Request #84580 · swiftlang/swift · GitHub

Doug

1 Like

Would you be able to paste an example of the payload somewhere? I'd love to see how it breaks down per module so that I could extrapolate what it would look like for some of our dependency trees that are much bigger.

Let me know when this is far enough along to try out and I'll be happy to guinea-pig it.

Here's the static build configuration for an empty module, as produced by the new frontend option -print-static-build-config but without any -D flags:

{"attributes":["rethrows","differentiable","reasync","IBInspectable","_noMetadata","globalActor","isolated","backDeployed","dynamicMemberLookup","discardableResult","abi","resultBuilder","available","NSManaged","UIApplicationMain","const","_extern","dynamicCallable","preconcurrency","unchecked","inline","_addressable","attached","main","derivative","Sendable","autoclosure","nonobjc","usableFromInline","objc","NSCopying","IBSegueAction","IBOutlet","requires_stored_property_inits","GKInspectable","warn_unqualified_access","escaping","concurrent","transpose","_local","exclusivity","IBAction","objcMembers","propertyWrapper","unsafe","nonexhaustive","noDerivative","nonisolated","convention","inlinable","retroactive","IBDesignable","storageRestrictions","frozen","lifetime","NSApplicationMain","constInitialized","_opaqueReturnTypeOf"],"compilerVersion":{"components":[6,3]},"customConditions":[],"endianness":"little","features":["BuiltinBuildExecutor","ObjCImplementation","AsyncAwait","AddressOfProperty2","InoutLifetimeDependence","BuiltinSelect","ExtensionMacros","Macros","BuiltinCreateAsyncTaskInGroupWithExecutor","FreestandingExpressionMacros","LexicalLifetimes","BuiltinBuildTaskExecutorRef","NonfrozenEnumExhaustivity","BuiltinCreateAsyncTaskOwnedTaskExecutor","BuiltinBuildMainExecutor","RetroactiveAttribute","NoAsyncAvailability","Actors","FreestandingMacros","ExpressionMacroDefaultArguments","ConformanceSuppression","IsolatedConformances","RawIdentifiers","InlineArrayTypeSugar","TypedThrows","OptionalIsolatedParameters","EffectfulProp","MemorySafetyAttributes","ParserValidation","AlwaysInheritActorContext","NoncopyableGenerics","SendableCompletionHandlers","ParserRoundTrip","BuiltinExecutor","NewCxxMethodSafetyHeuristics","NoncopyableGenerics2","IsolatedDeinit","BuiltinAddressOfRawLayout","BuiltinBuildComplexEqualityExecutor","ImplicitSelfCapture","BuiltinStackAlloc","ValueGenericsNameLookup","UnsafeInheritExecutor","AttachedMacros","SpecializeAttributeWithAvailability","LifetimeDependenceMutableAccessors","NonescapableTypes","InheritActorContext","BorrowingSwitch","BuiltinAssumeAlignment","BuiltinCreateTaskGroupWithFlags","BuiltinCreateAsyncDiscardingTaskInGroupWithExecutor","BuiltinContinuation","BitwiseCopyable","ConcurrentFunctions","BuiltinCreateTask","BuiltinEmplaceTypedThrows","BuiltinHopToActor","BuiltinInterleave","IsolatedAny","BuiltinVectorsExternC","BuiltinTaskRunInline","BuiltinJob","BuiltinIntLiteralAccessors","BuiltinCreateAsyncTaskName","BuiltinCreateAsyncTaskInGroup","LayoutPrespecialization","BuiltinCreateAsyncTaskWithExecutor","ExtensionMacroAttr","UnavailableFromAsync","BuiltinTaskGroupWithArgument","MarkerProtocol","AsyncExecutionBehaviorAttributes","BodyMacros","MoveOnly","BitwiseCopyable2","GlobalActors","BuiltinCreateAsyncDiscardingTaskInGroup","MoveOnlyPartialConsumption","BuiltinUnprotectedAddressOf","BuiltinUnprotectedStackAlloc","RethrowsProtocol","SendingArgsAndResults","PrimaryAssociatedTypes2","AssociatedTypeImplements","AssociatedTypeAvailability","NonescapableAccessorOnTrivial","MoveOnlyResilientTypes","BuiltinStoreRaw","NonexhaustiveAttribute","ValueGenerics","IsolatedAny2","ABIAttributeSE0479","ParameterPacks","AsyncSequenceFailure","GeneralizedIsSameMetaTypeBuiltin","Sendable"],"languageMode":{"components":[5,10]},"targetArchitectures":["arm64"],"targetAtomicBitWidths":[128,64,32,16,8],"targetEnvironments":[],"targetOSs":["OSX","macOS"],"targetObjectFileFormats":["MachO"],"targetPointerAuthenticationSchemes":["_none"],"targetPointerBitWidth":64,"targetRuntimes":["_multithreaded","_ObjC"]}

The implementation PR has what I believe is the complete implementation. We could kick off a toolchain build or merge so it can show up in nightlies.

Doug

1 Like

Thanks! I thought based on the post you were replying to that this might also include the importedModules field, which was my sole concern. If there are still no plans to expose that information to macros, then I think what you have here is feasible without scaling issues.

To support #if canImport() in macro expansion contexts, we could include the cached results of canImport() in StaticBuildConfiguration. The compiler already knows all the #if canImport(...) usages within the expanding (and related) source files, along with their correct results, so it should be able to include those in StaticBuildConfiguration.

This wouldn’t allow arbitrary calls to BuildConfiguration.canImport(importPath:version:), but at least it would correctly evaluate the canImport conditions that actually appear in the provided source code.

I had started with this implementation, recording all of the canImports that happened. I didn't like it for a few reasons:

  • The macro would have different behavior from non-macro code in ways that feel very wrong. There could be an explicit "import Foo", but the macro couldn't get a "yes" for canImport(Foo) unless canImport(Foo) was uttered somewhere in the original sources. That seems weird.
  • The macro wouldn't be able to see transitive imports, either, which can be a bit of a surprise.
  • canImport is the only build configuration check that cannot be established just based on the command line. It requires searching for and build modules, which is much heavier weight, but I'd like "give me a static build configuration described by this command line" to be a quick operation so that other clients can use it.

Tony also noted that the transitive import graph can get pretty big, and was concerned about it. On balance, I think we're better off not including support for canImport.

Doug

1 Like

I'd prefer isImported() over canImport() in the context of macros since I can tune my code based on what's imported, but I can't tuna fish add additional import declarations in a macro expansion.

(If that's helpful!)

3 Likes