Plan for module stability

parseable-interfaces
(Virl) #41

Ok, thanks.

(Helge Heß) #42

Trying to understand that. So a Swift 9 compiler has to support a Swift 5 mode to be able to deal with inlined stuff in Swift 5-built modules, right?
Given that SwiftPM is going to drop support for Swift 3 packages due to Swift 5 dropping Swift 3 language compat, it seems unlikely that the approach is going to work for module stability? (or are you going to guarantee source level compat starting w/ Swift 5?)
Or is there some kind of fallback planned (e.g. the compiler could just disable inlining and link an ABI version embedded in the library).

P.S.: I also think that backwards compat is highly desirably (i.e. linking a Swift 9 built module w/ a Swift 5 built app).

1 Like
(Brent Royal-Gordon) #43

We’ve been decreasing the number of source-breaking changes we make ever since Swift 3, and that in turn has been decreasing the difficulty of maintaining old modes. Swift 3 mode required lots of ugly hacks all over the compiler to maintain awful bugs; Swift 4 mode is already less costly than that.

I’d be interested to hear from the horse’s mouth, however, what we plan to do about that. Are we planning to maintain Swift 5 mode in perpetuity? Or do we figure it’s not a big deal if Swift 9 drops inlining bodies and default parameters in a Swift 5 .swiftinterface file?

1 Like
(Jordan Rose) #44

Yep. The plan here would be to drop the default argument but continue on as best as possible. (We'd record that in the AST too somehow so that the user knows.)

(Jordan Rose) #45

It hasn't exactly been decided yet, but at some point this will be the case. Whether or not it's "Swift 5 mode" that counts as the stable one is still up in the air, but yes, at some point we ought to stop needing distinct language modes, the same way C++11 was 97% backwards-compatible with C++03.

(If Swift 5 is not the stable mode, or even more likely if Swift 4 is not the stable mode, then the guarantee is something closer to "this interface will continue to work up to the next non-patch release of Swift". That's still an improvement over what we have today, and there's other benefits we want out of this format too.)

That's not necessarily true if you don't build the library to be "resilient" (see the original post). That's for another thread, though.

The general problem is still interesting: you normally would want to say "this library provides this interface, but if you're still compiling with Swift 5 you can leave these parts of it out". This first plan doesn't account for that, but I think it's something we could reasonably add to the model. (Remember also that changes to the standard library will no longer be tied to changes in the compiler in precisely the same way on Apple platforms, since new standard library features may not be available on all your deployment targets.)

I'd like to focus on the sort of "minimum viable product" for this thing first, but thanks for bringing it up.

1 Like
(John McCall) #46

We could also try storing functions as some sort of type-annotated syntax tree rather than as actual source. Our current AST is semantically-annotated to the point of being just as much an IR as SIL is, but we could do something more minimal that fully resolves declaration references and erases irrelevant syntactic details. That would be more work in some ways, though.

(Brent Royal-Gordon) #47

Shower thought: should we support #if swift(...)/#if compiler(…) in swiftinterface files? That would allow us to generate backwards-compatible interface files in the future, but of course we have to ship support for it before we know we need it.

Similarly, one specific thing I think we might want it for is non-essential attributes. Imagine that we added @discardableResult in Swift 6; it’d be silly for Swift 5 compilers to reject declarations which used it. On the other hand, if we added mutating in Swift 6, that’s such a fundamental part of the declaration that Swift 5 should probably ignore the whole declaration if it can’t understand it. We could handle this problem by generating an #if swift for the former but not the latter, but (maybe I’m wrong) I don’t believe #if can currently surround a single attribute. Should it?

Or is that a good place to draw the line between attributes and declaration modifiers—attributes are nonessential and unknown ones are ignored in swiftinterface files, while declaration modifiers are essential and unknown ones cause the whole declaration to be ignored?

2 Likes
(Greg Titus) #48

Rather than another format (type-annotated syntax tree) might it be worthwhile to store "type-annotated source"? That is, Swift where all of the type inference is made explicit, which would presumably speed up compilation (although it would require more parsing, it seems like the tradeoff would be positive).

E.g. transforms like:

func f(_ x: Int?, _ y: Double) {}
let n = 0
f(n, 3)

becomes:

func f(_ x: Int?, _ y: Double) {}
let n: Int = 0
f(Optional<Int>(n), 3 as Double)

A typed AST format would be a lot simpler to write, clearly, but on the reading side this would have most of the advantages of having the type annotations for avoiding expensive inference while not requiring another versioned reader. Additionally I can see having a fully-type-annotated Swift writer available as being potentially very useful for explanatory diagnostics.

(John McCall) #49

Unfortunately, as casts aren't enough to eliminate type ambiguities in Swift source; as casts can coerce types.

(Jordan Rose) #50

We thought about including types, but concluded that it could potentially hurt source compatibility as much as help it. That's not true if a depended-upon library stays perfectly source-compatible, of course, but it would make it harder to make small changes that in practice don't break anyone, even though you could contrive a test case where they do.

We can also change all this later, as long as we continue to handle the old format. Greg's suggestion would be particularly easy to adopt since it's just more source, though we'd have to consider the coercion aspect too.

(Fabian) #51

That‘s great to hear! Related post: Improve Incremental Compile Time of Projects that are split up into several Frameworks

(Slava Pestov) #52

Thinking about this some more, there's one point that's not addressed in the plan so far.

We should discuss what to do about implicit or synthesized declarations. Will the textual output include any or all of the following?

  • typealiases for inferred associated types
  • implicit initializers (default, memberwise or inherited)
  • witnesses for derived conformances
  • implicit conformances (the stupid thing where where an enum with no payload cases is always Hashable, anything else?)
  • conformances inherited from a base class
  • implicit destructors on classes
  • the 'lazy' desugaring (stored property named 'foo.storage', original becomes computed getter/setter with opaque body)
  • withSet/didSet desugarging (need a syntax for a stored property but with an opaque setter)
  • any others?

(Edit) Another one -- that thing we just fixed with witness visibility? We'll need some way to declare that a conformance is valid even if a witness cannot be found. Presumably, you don't want to print the private protocol R or its extension in a stable module interface dump:

public struct S : P, R {}
private protocol R {}
public protocol P { func f() }
extension R { func f() { } }

Another interesting point is what if a public type conforms to a public protocol by way of a private protocol. Eg, in the above, change 'private protocol R : P {}', and make S only conform to R and not P. Should a user of the module interface know that the conformance of S : P exists or not?

1 Like
(Slava Pestov) #53

My personal opinion is that since we only need to print and parse source for inlinable function bodies (which I admit is an important use-case, at least for the standard library), some kind of fully type-annotated AST form is overkill. We would need to implement an expression pretty-printer (right now, we can just copy the source code verbatim), as well as a whole new parser and type checker mode that understands this new language. That's a lot of new code that will not be as well-tested as the primary code paths through the parser and type checker.

(👑🦆) #54

I suppose they would have to, otherwise they might try to write their own conformance of S : P and dynamic casts would give unexpected results.

1 Like
(Jordan Rose) #55

Great points. My suggested answers for these:

  • typealiases for inferred associated types - yes, these should be printed explicitly
  • implicit initializers (default, memberwise or inherited) - yes
  • witnesses for derived conformances - yes
  • implicit conformances (the stupid thing where where an enum with no payload cases is always Hashable, anything else?) - yes
  • conformances inherited from a base class - no
  • implicit destructors on classes - yes in a resilient build, no in a non-resilient one
  • the 'lazy' desugaring (stored property named 'foo.storage', original becomes computed getter/setter with opaque body) - hmm, not sure
  • withSet/didSet desugaring (need a syntax for a stored property but with an opaque setter) - I think this is just going to be { get set }, like protocols. Do we care that the getter is trivial?
  • witness thing - no. A failure to find a witness should always just fall back to dynamic dispatch here.
  • protocols conformed to by inheritance from other protocols - yes, honestly I think we should just list all of them even if they're not private

I'm sure more of these will come up as we work on this.

(Slava Pestov) #56

Inherited initializers inherit default arguments from the base class initializer. There's no way to write this in source right now, so we'd have to think of something. Perhaps just an attribute on the ParamDecl.

Why different?

It only really matters for non-resilient builds, where you need to know the stored property layout of the class. In a resilient build, you would just print the lazy property as a computed { get set }.

I think we want to leave open the possibility of the implementation of 'lazy' changing in various ways, so I would suggest printing the desugaring.

If the class is not resilient, you also have to print the stored property representing the underlying storage.

Right, but do you really want to disable all diagnostics while checking conformances in a swiftinterface file? It might hide other errors. I think we might need some way to declare that a witness for a requirement exists but is not visible statically.

(Jordan Rose) #57

Ah, I guess just copying the implementation down as text would be a problem…hm.

A non-resilient build can take advantage of a class having no side effects in its destruction if it knows the types of all the stored properties too. Neither assumption is safe in a resilient situation, but rather than special-case that we can just write a deinitializer in always. (Or never, I guess, but this way is less likely to run across other compiler issues.)

You're right that we don't want to disable diagnostics. I was just thinking that conformance checking would operate in a different mode that allows missing witnesses.

(Slava Pestov) #58

Yeah, because unqualified lookup might produce different results in the subclass (for example, it might have a different set of generic parameters)...

(marc hoffman) #59

That is good to hear!

Is there documentation, a spec, or a good place in code to find out how this metadata is structured? is it stable yet (as of 5.0/5.1?)

(Jordan Rose) #60

Posted an update on this effort at "Update on Module Stability and Module Interface Files". (This thread's a little long to ask people to jump into.)

(Sorry, Marc, not an answer to your question. Which is a good one, but not for this thread.)

2 Likes