[Pitch] Package traits

Hey everyone,

@Max_Desiatov and I have drafted up a pitch proposal for SwiftPM to introduce what we call package traits. Traits allow package authors to provide configuration options to tune both the behaviour of the package and the dependencies a package uses. A need for optional dependencies and configurable behaviour has come up in many places over the past years and we think this pitch would address this need.

You can find the whole pitch here

Looking forward to feedback and questions!

Franz

30 Likes

Looks great, thank you for pushing SPM capabilities in this area. The first use that came to mind is having a trait for unit-testing and ui-testing so a target could provide mocks and tests helpers right into the module while making sure those are not included when building for deployment.

I think it would be great to provide a set of traits that are expected to be common to help the package converge toward a smaller set.

1 Like

Overall, I like the direction of giving SwiftPM the ability to define custom conditions that its resolver is aware of and that also map to compiler conditions. I'm a bit confused by this specific part of the motivation, though:

Minimizing build times and binary size

Some packages offer different but adjacent functionality such as the swift-collections package. To reduce build time and binary size impact swift-collections offers multiple different products and users can choose which one they need. This works however, it comes with the downside that if the implementation wants to share code between the different modules it needs to create internal targets. Furthermore, the user has to declare a dependency on different products and import each product module individually.

I think I'm failing to see the connection between this and the proposed traits design or I'm reading the motivation incorrectly. The "downside" described here—a package having to define internal modules to share functionality between more than one of its public modules—seems like it's actually working as intended to me? And doing it that way does reduce binary size and build time, by only having a single compile action and copy of the shared code; the only other alternative that I could see would be to conditionally put the shared code into each of the consuming modules, which would be an anti-goal.

Can you expand on what your thinking was here?

2 Likes

Ah yeah I see where the confusion comes from. So the main motivation that I have with the collection use-case is the amount of modules. Every data structure (or at least grouping of data structure) gets its own module to reduce the binary size impact. However, this comes with the downside of having to declare additional dependencies in the manifest and imports. With traits we could instead define a single Collections module and have each data structure be guarded by a trait.
Similar to swift-collections we have the same problem in swift-nio with NIOs 10s of modules for various purposes. Instead of having NIOCore and NIOPosix we could have a single NIO module and put the posix implementations behind a trait.

Just to expand on this. In general I agree putting shared code into a single module is good; however, currently these shared modules can easily be imported by upstream modules and reach for public code that isn't intended to be used. We had this happen in NIO already. This isn't a strong motivating factor for package traits since the package access level allows us to work around this. It is just a nice added benefit of using traits instead of modules.

Overall, we need both modules and traits in the ecosystem in my opinion. Currently, we are leaning heavily into creating modules for every single small piece when what we need is just a bit of conditional complication or a way to express optional dependencies.

8 Likes

Thanks for clarifying! I believe that this would be extremely problematic for other build systems outside of SwiftPM then. Bazel, for example, intentionally limits and discourages pushing information up from the leaves, unifying it, and pushing it back down to those leaves, because it is harmful to caching and can create a combinatorial explosion of configurations.

The idea that a module's contents could change based on who depends on it feels quite contradictory to the concept of what a "module" is in the first place. I actually consider it to be an anti-goal to paper over this kind of dependency management: users should be much more aware of exactly what they're depending on and should have to be explicit about it, and we should give them the tools to make managing those dependencies easier. I don't believe this is the right tool for that.

For the specific problem of "I have a lot of similar code that I'd like to group together in a single unit", I think the approach currently used by swift-collections is fine: the individual data structures go in their own modules, and an umbrella module re-exports them. Users have the option of choosing specific dependencies or using the umbrella target. We should formalize @_exported import as the solution for this, not traits, because I don't think the cost of the finer granularity is worth these trade-offs.

Unfortunately, I think allowing and encouraging Swift code at the package level to provide knobs that can be controlled by dependents would be actively harmful to the ability of Swift to be well-used by other build systems. If this proposal was accepted and began to be used, Bazel users would likely have no choice but to always define the full set of traits supported by any given package so that the module has everything that any dependent anywhere in the dependency graph might need. This would be worse than the situation today, where users who only need specific data structures import those specific modules. (In our own monorepo, we actually ban users from depending on the Collections umbrella, because we want users to only depend on and import exactly what they need.)

Without the unification aspect, this proposal would otherwise still be quite useful, though, and would be able to be mirrored by other build systems.

11 Likes

To ease this, I think a TraitName type should be defined to be extended with those common traits. As a bonus users could define names to leverage autocomplete.

extension TraitName {
    public static let foobar: Trait = "FooBar"
    public static let foo: Trait = "Foo"
}

Trait(
    name: .foobar,
    isDefault: true,
    enabledTraits: [
        .foo,
        .testing, // this one would be declared in PackageDescription
    ]
)
Example Implementation

I wrote this code in the forum so I surely made a typo somewhere but it gives the idea.

struct TraitName: ExpressibleByStringLiteral {
    private let rawValue: String

    init(stringLiteral value: String) {
        rawValue = value
    }
}

extension TraitName {
    public static let testing: TraitName = "Testing"
}

I understand your concern from a build system perspective but everything is known statically after the dependency resolution has ran. So in fact no module is going to have different content from the view of the build system. If the dependency graph changes then yes we would need to recalculate and reunify the traits across the graph.

Looking at the Rust Bazel rules they support this as well but they do it slightly different. They pass the enabled optional features to the individual library build targets. This seems like a valid approach to me since Bazel is not in the business of dependency resolution and just building. We could do the same for Bazel and CMake based Swift builds. In the end these traits just boil down to defines that have to be passed + paths to the optional dependencies.

Do you think we can take the same approach that the Rust Bazel rules take?

2 Likes

The issue is that more that the view of the module changes at all depending on who uses it. Let's say that two completely independent packages A and B (imagine they're apps, so they're terminal linking targets) depend on a traits-enabled Collections. A's view of Collections could be different than B's depending on which traits each project enables. That means that, unlike today, the compiled artifacts for Collections can't be shared between them; their caches would be independent, as would be the work to build them.

My concern here is that Swift already has too many unsolved problems with scaling to very large builds, and this adds one more scaling issue on top of that before improving the others. For example, before we were able to enable explicit Clang modules in our builds a few years ago, Swift builds were beginning to become simply intractable. SwiftPM builds don't experience these kinds of problems to a significant degree because the package graphs simply don't get big enough for it to matter in most cases, but the fact that the proposal already has to set an arbitrary maximum number of traits is illustrative of the potential scaling problems that could eventually arise longer term.

The Rust rules use the same approach I mentioned above, which was that they have to pass the full set of possible traits that anyone may want to use at the individual library level; there's no unification step that would occur based on who uses it. For example, there would only be a single definition of Collections in the repository and it would have to have all of its traits defined so that all of its APIs are available to anyone who may want to depend on them, and everyone who depends on them (even independent binaries) would get the full set whether they need them or not linked into their binary. That would be a regression from what we have today, where the data structures are in independent modules and each client can choose what they want, while still getting the full benefits of caching and artifact reuse across every project in the repository.

So ultimately I guess the conclusion here is that we could work around it, but it would make things worse because Swift simply doesn't have the tools today to strip out unused code and metadata from binaries at link time well enough to compensate for cutting out the unification step. Overall, I think this is the wrong tool for this job—we should be encouraging library authors to break down their modules into smaller units and make it easier to depend on them, not make modules larger and introduce complexity into the notion of what a module actually contains.

13 Likes

Maybe we are speaking past each other a bit or the unification section in the proposal isn't clear enough.

There is never going to be more than one artefact for a module for one specific binary that's being built. Trait unification happens before building so at the time of building we know exactly what traits to enable for a single module. This way we only build the module once with the union of traits required by all users of a package.

I don't think that's correct. Let's say we are building multiple Swift binaries in a single mono repository and some of these binaries are using swift-collections. We would only need to enable the union of traits required by all binaries not all. This is how the Rust Bazel rules work as well. This would indeed mean that potentially unnecessary code is build for some binaries that don't require all traits.
To work around this one could pass from the binary target down to the Swift libraries the requires set of traits. This seems to be doable in Bazel by passing down the extra args; however, this is totally up to the end user. The Swift Bazel rules would just need to support enabling traits for a library similar to how the Rust Bazel rules do so.

In general, I agree with breaking down code into modules and I have the feeling we are getting a bit hung up on the collections example of the pitch. This is only one motivating example but there are many more where splitting into modules and often separate repositories is seriously regressing usability and discoverability.

Overall, I fully emphasise with the implication of traits in mono repositories where we want to share built modules across different binaries but we still have to keep in mind that the primary way of building Swift applications is using SwiftPM and not having package traits is IMO very lacking and a big advantage that tools like cargo offer. In my opinion, it is okay if users of build systems such as Bazel have to incorporate traits into their way of defining target and their overall build setup if the experience for 95% of the ecosystem with Swift PM is better due to expressibility that traits bring with them.

3 Likes

i am a little worried about how we are going to document packages that are using traits in this way.

are we going to publish documentation for multiple flavors of the same package, each compiled with a slightly different set of traits? this is going to result in a lot of duplicated content and is going to be bad for navigation and searchability.

are we going to only publish one canonical flavor of the package, excluding the trait-guarded features, and say that the documentation is only for users on one particular platform or building with one particular configuration?

don’t get me wrong, i think traits are a sorely needed feature that i too have been wishing for for a long time. but i think that using traits to configure modules to vend significantly different subsets of API based on the trait would be an example of getting a bit carried away with the new feature.

5 Likes

I can't quite tell from the pitch if it stands if it will (and how to arrange it, if it does) cover a few of the use cases that I'm seeing others say "Oh yeah, that'd totally solve my problem!" so I'm tentatively very positive.

Let me lay out my use cases that I'm hoping this helps address, and maybe get a bit of concrete detail on how you'd use traits to solve them - because at the moment, the pitch with foo and bar and how they compose together is just a bit too abstract for me.

The first is that, as a library author, I'd really like to have a testing dependency NOT be transitive to the packages that use my library. I'm guessing that may be a default trait that could be built in and a clear pattern established, but I'm not clear on where and how defines the traits for that. Could you perhaps spell out how that might work, or let clarify that I'm misinterpreting this pitch and it doesn't solve that use case?

The other is somewhat related, again focused on avoiding a transitive dependency - and that's to add in some benchmarking for a library. Today I'm creating these as their own little projects in a subdirectory with a local package reference up a level so that they're self-contained and don't splatter their own dependency down to any consumers. It feels like that may be the identical pattern to the the testing target setup.

For the bits about being able to define specific build configurations when a trait is defined, PLEASE YES. Especially with C++ interop, getting the right build configurations is a significant challenge.

Likewise, I have a project where I'm shimming in an alternative dependency (remote binary vs. local) using an environment variable that this would directly support, and I think very nicely. But what I'm not clear on is the user experience for invoking this alternate. Today it's for "I"ve rebuilt the binary deliverable and I want to work with it" vs. a default binary deliverable that we keep hosted on GitHub, so it's usage is really for developers iterating on code. Do you end up exposing a "I want trait USE-THE-ONE-I-BUILT" as a command line option with swift build or such?

2 Likes

That's only true within a single build though, so maybe I'm not being clear enough. Here's what I'm imagining. First, in a SwiftPM world:

  • Team A has an application with a package that depends on swift-collections.
  • Team B has an application with a package that depends on swift-collections.

Team A and Team B are completely independent, and their apps are independent. From your description, they'll each build Collections differently depending on what traits they have.

The Bazel equivalent (if they're in the same repo) would be that Team A and B live in different directories but they share a dependency on a single canonical version of swift-collections and they're able to share build artifacts from each others' builds. That's where the difficulty arises here; either we have to explode the configurations to let each one choose the exact traits they want for swift-collections (losing the shared build and caching between them) or we have to manually union the traits ahead of time, giving each team more bloat than they might possibly want, and Swift doesn't yet help enough with reducing that.

I don't think that's correct. Let's say we are building multiple Swift binaries in a single mono repository and some of these binaries are using swift-collections . We would only need to enable the union of traits required by all binaries not all.

Sure, it's true that we don't have to define them all up front, but let's say we defined 2 out of 3 possible traits, and then another app needs to use the 3rd trait. They can add it easily enough to the canonical definition of the target, but then that decision affects everyone who uses it.

I'm drilling down on that example because it was listed as a specific motivating example where the traits system would be used. I don't disagree that "the primary way of building Swift applications is using SwiftPM", but I get concerned when what's being proposed sounds like a system by which the dependency resolver and the build system come together to change the way users design modules. It feels like a layering violation between the compiler and the package manager.

I realize I'm arguing against the prevailing winds, especially given that my usage of SwiftPM is fairly low compared to using Bazel for day-to-day Swift development, and I'm trying not to come off too negative or alarmist here. As you've pointed out, Swift isn't the only language going in this direction—we have similar scaling problems with Rust on Bazel for the same reasons. But I feel it's important to raise these issues when they come up based on experience; I'd be less hesitant about this if regular Swift compilation and linking did a better job of eliminating unused code today, without having to resort to single-package-manager-based solutions like this.

2 Likes

This is a great question! The Rust ecosystem handles this very nicely by showing on an API if it is guarded by a trait. For example the join type is only available when the feature io-util is enabled. I think we can do a similar thing for package traits however to get started I would recommend to just build documentation with all traits enabled.

Yes that's totally possible and I would call those development dependencies. You could in fact define a development trait which guards the stage of such dependencies. However, SwiftPM is already doing target based dependency resolution at the moment so if you only depend on something from your tests then your adopters will never pull in this transitive dependency. With package traits you can do the same and make it a lot harder to misconfigure. I think a potential future evolution would be to create development dependencies as a standalone feature and we could implement it with package traits under the hood.

That's the same as your first example. The only wrinkle here is that platform requirement might still force you to create a nested package unless your package's platform requirements align or are higher than what the package-benchmark package requires.

So traits try to tackle some of the env variable usage inside Package.swift however I don't think this particular use-case will work right away because in the current implementation traits are checked post-dependency resolution. So in your example it will clone both dependencies (remote and local) even if only one trait is active. Before building it will then reconcile that and only build the one activated by a trait.
In general though I think for this particular feature we are better off extending the SwiftPM mirroring feature to also work with local paths for mirrors.

I really appreciate your input here and I am a big proponent of Bazel myself. I agree that having better dead code stripping would lessen the impact here.
Since you mentioned that you have experience with Rust's optional features and Bazel, I would be interested to hear if you have a different approach in mind or a suggestion how we can make the integration of traits into Bazel easier while still achieving the same goals.

2 Likes

i know very little about Rust or how it is compiled, but i found it interesting that the linked declaration has no @available-like attributes annotating the trait requirement. how is Rust able to infer the availability with no corresponding annotation?

I'm supportive of the direction overall. Something relatively minor in terms of API spelling stood out when reading the proposal about specifying enabled traits in the manifest.

Citing the example code in the proposal:

 enabledTraits: [
    "SomeTrait",
    EnabledTrait("SomeOtherTrait", condition: .when(traits: ["Foo"])),
]

I was surprised the API was written this way, with a public type initialiser. It doesn't seem congruous with existing Package.swift API. For instance, for swiftSettings, you declare a conditional define like this:

swiftSettings: [
    .define("TEST", .when(configuration: .debug))
]

Is there a reason EnabledTrait isn't aligned here, for instance spelt like .trait("SomeOtherTrait", .when(traits: ["Foo"])?

4 Likes

As a possible future direction, I wonder if this "trait" concept could extend to the standard library in Embedded Swift environments too. This sounds like a great way to categorize removable runtime or standard library features like dynamic allocation, reflection, String, etc., and allow packages downstream to systematically conditionalize on the presence of those standard library traits. And as yet another future direction, while using #if conditions is fine and probably necessary at some level for full flexibility, it'd be interesting if traits could also be checked in an @available-like way, where disabled declarations still exist and can be fully type-checked by the compiler, which could avoid subtle source-breaking behavior such as different overload resolution or inadvertently allowing retroactive conformances over disabled canonical conformances when declarations are entirely removed, and which could also allow for the compiler to check that trait dependencies are correct across modules.

16 Likes

Tokio uses a bunch of macros for their optional features but normally every type that is guarded behind an optional feature in Rust is annotated with a #[cfg(feature = "FEATURE_NAME")]. This is how their documentation system picks this up. The reason why not every single method/type has a cfg macro attached to it is that you can attach it to a whole module definition in Rust and it will just apply down to all nested APIs.

I think that's a great suggestion. I will put it onto my list to update it in the proposal. Thanks!

I would love to see this extend into the stdlib. With the current proposal this already kinda working since it just boils down to compile time conditional checks which we already do for a bunch of these features.

What you are describing is very similar if not exactly how @_spi currently works. When I began the work on package traits this was the first thing I explored. However, both @_spi and what you just described requires that the compiler can see all the code and that it fully types checks and for @_spi even compiles.
This is exactly what package traits don't want. We want to conditionally not compile code. This is incredibly important in the optional dependencies case where certain types might not even be present since the dependency isn't compiled in.
Overall though, I would love to see an integration for traits into the compiler in the future to achieve two things. First, a better syntax than #if and secondly to make the compiler aware of what traits a module has activated on an imported module so the compiler can check that only APIs are used that the module has enabled on the imported module.

2 Likes

I agree it's what you want sometimes, especially when talking about dependencies on platform libraries that simply aren't available in some configurations. It would be useful to also be able to have compiler checking for the traits not used, when possible, especially as the set of configurations in a system grows and it becomes impossible to empirically test them all individually.

2 Likes

Oh yeah I agree on the testability concern. I was thinking about a potential future direction to just run swift build/test --all-trait-combinations that tries to build/test the package with all possible trait combinations one after the other. This would probably be helpful for library authors to make sure everything behaves correctly.

2 Likes

how do we reconcile this with the Swift compiler’s documentation generation model? as i understand it, it is an inherent requirement of lib/SymbolGraphGen to compile (or at least type check) the code to emit symbol information for.

1 Like