[RFC] "Library Evolution Support in Swift ('Resilience')"


(Jordan Rose) #1

Hi, swift-evolution. We've been making references for a while to "resilience" as a cornerstone of the Swift 3.0 work, the collection of features that allows a library to evolve over time while maintaining binary compatibility. Among other things, this is necessary if we want to stop bundling the Swift standard library with any app that uses Swift, a noted complaint from iOS developers. :slight_smile:

If you're wondering what this is all about, take a look at the prologue for the design document:

One of Swift’s primary design goals is to allow efficient execution of code without sacrificing load-time abstraction of implementation.

Abstraction of implementation means that code correctly written against a published interface will correctly function when the underlying implementation changes to anything which still satisfies the original interface. There are many potential reasons to provide this sort of abstraction. Apple’s primary interest is in making it easy and painless for our internal and external developers to improve the ecosystem of Apple products by creating good and secure programs and libraries; subtle deployment problems and/or unnecessary dependencies on the behavior of our implementations would work against these goals.

Our current design in Swift is to provide opt-out load-time abstraction of implementation for all language features. Alone, this would either incur unacceptable cost or force widespread opting-out of abstraction. We intend to mitigate this primarily by designing the language and its implementation to minimize unnecessary and unintended abstraction:

  • Avoiding unnecessary language guarantees and taking advantage of that flexibility to limit load-time costs.
  • Within the domain that defines an entity, all the details of its implementation are available.
  • When entities are not exposed outside their defining module, their implementation is not constrained.
  • By default, entities are not exposed outside their defining modules. This is independently desirable to reduce accidental API surface area, but happens to also interact well with the performance design.

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

RFC stands for "request for comments", and that's what this is: I'd appreciate the eager and discriminating eyes of swift-evolution on this model. It is quite long—nearly ten thousand words—and attempts to be fairly precise in describing what is and isn't allowed, so feel free to focus on the parts that interest you most. This isn't a proposal and won't be going through the Swift Evolution Process, but many existing or planned proposals will affect or support the model described here. (There's a list of them at the end of the document.)

The document is written in ReStructuredText to match the rest of the compiler documentation, but it's using some features from the Sphinx system that GitHub's ReST renderer doesn't support. Consequently, I've put up a rendered form <http://jrose-apple.github.io/swift-library-evolution/>, which I'll update every few days when there are changes. (This is pretty much the same rendering you get from running "make" in the docs/ directory in the Swift repo.) The canonical document is still the one in the Swift repository <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst>.

Looking forward to your feedback!
Jordan


(Jordan Rose) #2

I just sent this to swift-evolution. TLDR: LibraryEvolution.rst is ready for wider review and comments.

(Of course Slava's been hard at work implementing all this for the last few weeks already, but that doesn't make the model any less important!)

Thanks!
Jordan

···

Begin forwarded message:

From: Jordan Rose <jordan_rose@apple.com>
Subject: [RFC] "Library Evolution Support in Swift ('Resilience')"
Date: February 8, 2016 at 18:24:15 PST
To: swift-evolution <swift-evolution@swift.org>

Hi, swift-evolution. We've been making references for a while to "resilience" as a cornerstone of the Swift 3.0 work, the collection of features that allows a library to evolve over time while maintaining binary compatibility. Among other things, this is necessary if we want to stop bundling the Swift standard library with any app that uses Swift, a noted complaint from iOS developers. :slight_smile:

If you're wondering what this is all about, take a look at the prologue for the design document:

One of Swift’s primary design goals is to allow efficient execution of code without sacrificing load-time abstraction of implementation.

Abstraction of implementation means that code correctly written against a published interface will correctly function when the underlying implementation changes to anything which still satisfies the original interface. There are many potential reasons to provide this sort of abstraction. Apple’s primary interest is in making it easy and painless for our internal and external developers to improve the ecosystem of Apple products by creating good and secure programs and libraries; subtle deployment problems and/or unnecessary dependencies on the behavior of our implementations would work against these goals.

Our current design in Swift is to provide opt-out load-time abstraction of implementation for all language features. Alone, this would either incur unacceptable cost or force widespread opting-out of abstraction. We intend to mitigate this primarily by designing the language and its implementation to minimize unnecessary and unintended abstraction:

  • Avoiding unnecessary language guarantees and taking advantage of that flexibility to limit load-time costs.
  • Within the domain that defines an entity, all the details of its implementation are available.
  • When entities are not exposed outside their defining module, their implementation is not constrained.
  • By default, entities are not exposed outside their defining modules. This is independently desirable to reduce accidental API surface area, but happens to also interact well with the performance design.

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

RFC stands for "request for comments", and that's what this is: I'd appreciate the eager and discriminating eyes of swift-evolution on this model. It is quite long—nearly ten thousand words—and attempts to be fairly precise in describing what is and isn't allowed, so feel free to focus on the parts that interest you most. This isn't a proposal and won't be going through the Swift Evolution Process, but many existing or planned proposals will affect or support the model described here. (There's a list of them at the end of the document.)

The document is written in ReStructuredText to match the rest of the compiler documentation, but it's using some features from the Sphinx system that GitHub's ReST renderer doesn't support. Consequently, I've put up a rendered form <http://jrose-apple.github.io/swift-library-evolution/>, which I'll update every few days when there are changes. (This is pretty much the same rendering you get from running "make" in the docs/ directory in the Swift repo.) The canonical document is still the one in the Swift repository <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst>.

Looking forward to your feedback!
Jordan


(Charles Srstka) #3

Hi, swift-evolution. We've been making references for a while to "resilience" as a cornerstone of the Swift 3.0 work, the collection of features that allows a library to evolve over time while maintaining binary compatibility. Among other things, this is necessary if we want to stop bundling the Swift standard library with any app that uses Swift, a noted complaint from iOS developers. :slight_smile:

If you're wondering what this is all about, take a look at the prologue for the design document:

One of Swift’s primary design goals is to allow efficient execution of code without sacrificing load-time abstraction of implementation.

Abstraction of implementation means that code correctly written against a published interface will correctly function when the underlying implementation changes to anything which still satisfies the original interface. There are many potential reasons to provide this sort of abstraction. Apple’s primary interest is in making it easy and painless for our internal and external developers to improve the ecosystem of Apple products by creating good and secure programs and libraries; subtle deployment problems and/or unnecessary dependencies on the behavior of our implementations would work against these goals.

Our current design in Swift is to provide opt-out load-time abstraction of implementation for all language features. Alone, this would either incur unacceptable cost or force widespread opting-out of abstraction. We intend to mitigate this primarily by designing the language and its implementation to minimize unnecessary and unintended abstraction:

  • Avoiding unnecessary language guarantees and taking advantage of that flexibility to limit load-time costs.
  • Within the domain that defines an entity, all the details of its implementation are available.
  • When entities are not exposed outside their defining module, their implementation is not constrained.
  • By default, entities are not exposed outside their defining modules. This is independently desirable to reduce accidental API surface area, but happens to also interact well with the performance design.

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

RFC stands for "request for comments", and that's what this is: I'd appreciate the eager and discriminating eyes of swift-evolution on this model. It is quite long—nearly ten thousand words—and attempts to be fairly precise in describing what is and isn't allowed, so feel free to focus on the parts that interest you most. This isn't a proposal and won't be going through the Swift Evolution Process, but many existing or planned proposals will affect or support the model described here. (There's a list of them at the end of the document.)

The document is written in ReStructuredText to match the rest of the compiler documentation, but it's using some features from the Sphinx system that GitHub's ReST renderer doesn't support. Consequently, I've put up a rendered form, which I'll update every few days when there are changes. (This is pretty much the same rendering you get from running "make" in the docs/ directory in the Swift repo.) The canonical document is still the one in the Swift repository.

Looking forward to your feedback!
Jordan
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I noticed this note in the document:

Note
Swift 2’s implementation of default values puts the evaluation of the default value expression in the library, rather than in the client like C++ or C#. We plan to change this.

What is the reasoning behind this? Keeping the default value in the library seems more resilient; if the default value changes, clients will get the new behavior without requiring a recompile, thus matching the behavior that you’d get if you implemented the same thing via a method with fewer arguments (as you’d have done in Objective-C). This seems far preferable to hard-coding all the default values in the client, IMO.

Charles

···

On Feb 8, 2016, at 8:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:


(David Owens II) #4

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

Regarding this:

Changing or removing a default value is permitted but discouraged; it may break or change the meaning of existing source code.

Maybe I'm being dense, but how is something with a caveat of "discouraged" and "it may break or change" in-line with "the default behavior is safe"? I've got no qualms with putting the default value in the client code; I actually think that is fine.

However, my concern is that you will have different behavior depending on if you simply drop in the updated binary vs recompile against the binary. Even worse if you have multiple components within your app that link against the library. If only one of those components is recompiled on release, you now have a problem of conflicting behavior within your own app because of the client-side calling the API will be using different values.

It would seem that removing the default value could be permitted (though maybe still discouraged) because this will result in a compiler error in the scenario above. It's still possible to have different behavior in your components, but now it's no longer implicitly happening. However, changing the default parameter seems highly problematic.

-David

···

On Feb 8, 2016, at 6:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:


(Drew Crawford) #5

This is an amazing document. I do not understand half of it, but the half I do understand will set software engineering forward ten years.

It will take me a long time to digest all of it, but 2 quick things:

It is legal to change the implementation of an inlineable function in the next release of the library. However, any such change must be made with the understanding that it may or may not affect existing clients.

I think this is wrong. Specifically, let's say I have a security bug in my inlineable function. *Currently*, the industry practice for responding to security issues is "download the new version of the library". But for inlined functions, this is not good enough to correctly apply the patch.

In my view, "somebody" (the linker, the loader, the runtime... it's all greek to me) should say "hold on a minute, you cannot use the new library version, recompile your code". Because as long as it appears the new library version installs/works fine, the security fix is falsely assumed to be applied.

Another thing I think we are missing here is versioning the function bodies themselves. For example (ignore syntax) suppose we have

public (1.0) func removeTheCacheFiles() {
      os.rmdir("/path/to/cache")
}

We may evolve this function in two orthogonal ways:

We may develop other cache files as our program grows
We may discover that we forgot to check if the user is allowed by the security policy to remove the cache files.

Therefore we may evolve this function as follows (again, I use pretend syntax):

public (1.0) func removeTheCacheFiles() {
      precondition(userIsAuthorized()) //this change is backported to 1.0
      os.rmdir("/path/to/cache")
      #if 1.1 { //this code only for 1.1-era callers
        os.rmdir("/path/to/cache2")
      }
}

It is important to support this case because very often in server land, certain clients only want to pick up the security fixes (and not, say, new features). See e.g. this Debian Security FAQ <https://www.debian.org/security/faq#oldversion>, where people spend a huge amount of time backporting security fixes to old versions.

I realize this is not at all the practice in "consumer-grade" applications like iOS/OSX/etc., but it is very entrenched in serverland, and I really think there is value in supporting this at the language level for those people who work in that world.

I think the implementation of this is just to compile all possibilities and just let the client pick the implementation based on the API version. I realize this may result in larger binaries, but only when the feature is used, so it's opt-in.

Apologies for not using fancy compiler words, I am still trying to grasp the full implications of this amazing paper.

···

On Feb 8, 2016, at 8:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

Hi, swift-evolution. We've been making references for a while to "resilience" as a cornerstone of the Swift 3.0 work, the collection of features that allows a library to evolve over time while maintaining binary compatibility. Among other things, this is necessary if we want to stop bundling the Swift standard library with any app that uses Swift, a noted complaint from iOS developers. :slight_smile:

If you're wondering what this is all about, take a look at the prologue for the design document:

One of Swift’s primary design goals is to allow efficient execution of code without sacrificing load-time abstraction of implementation.

Abstraction of implementation means that code correctly written against a published interface will correctly function when the underlying implementation changes to anything which still satisfies the original interface. There are many potential reasons to provide this sort of abstraction. Apple’s primary interest is in making it easy and painless for our internal and external developers to improve the ecosystem of Apple products by creating good and secure programs and libraries; subtle deployment problems and/or unnecessary dependencies on the behavior of our implementations would work against these goals.

Our current design in Swift is to provide opt-out load-time abstraction of implementation for all language features. Alone, this would either incur unacceptable cost or force widespread opting-out of abstraction. We intend to mitigate this primarily by designing the language and its implementation to minimize unnecessary and unintended abstraction:

  • Avoiding unnecessary language guarantees and taking advantage of that flexibility to limit load-time costs.
  • Within the domain that defines an entity, all the details of its implementation are available.
  • When entities are not exposed outside their defining module, their implementation is not constrained.
  • By default, entities are not exposed outside their defining modules. This is independently desirable to reduce accidental API surface area, but happens to also interact well with the performance design.

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

RFC stands for "request for comments", and that's what this is: I'd appreciate the eager and discriminating eyes of swift-evolution on this model. It is quite long—nearly ten thousand words—and attempts to be fairly precise in describing what is and isn't allowed, so feel free to focus on the parts that interest you most. This isn't a proposal and won't be going through the Swift Evolution Process, but many existing or planned proposals will affect or support the model described here. (There's a list of them at the end of the document.)

The document is written in ReStructuredText to match the rest of the compiler documentation, but it's using some features from the Sphinx system that GitHub's ReST renderer doesn't support. Consequently, I've put up a rendered form <http://jrose-apple.github.io/swift-library-evolution/>, which I'll update every few days when there are changes. (This is pretty much the same rendering you get from running "make" in the docs/ directory in the Swift repo.) The canonical document is still the one in the Swift repository <https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst>.

Looking forward to your feedback!
Jordan
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Joe Groff) #6

Regarding @inlineable, I think having the equivalent of __attribute__((always_inline)) is important. As proposed, an inlineable function may factor out helpers into internal-with-availability helpers, but then those helpers are still required to exist for old clients even if a new implementation of the inlineable function is factored differently. Such deep refactorings of inlineable code are not ideal, of course, but it's happened in the past in C++ standard libraries, and seems like freedom we'd want to reserve for the inlineable layer of Swift's own standard library.

-Joe


(Joe Groff) #7

Abstracting default values from the client limits resilience in other, more obnoxious ways. In particular, it means you can't *introduce* new defaulted parameters on published APIs without a deployment target limitation, since the entry point for the default argument evaluator will be missing in older versions. We figured that instantiating the default argument on the client side is more in line with users' expectations in what resilience constraints it imposes. In many cases, you'll have the resilinece you want at a different level; for example, in the common case of an option set where the default is 'all options', evaluating 'func foo(options: Options = .all)' is still resilient if 'Options.all' is.

-Joe

···

On Feb 8, 2016, at 6:47 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 8, 2016, at 8:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

Hi, swift-evolution. We've been making references for a while to "resilience" as a cornerstone of the Swift 3.0 work, the collection of features that allows a library to evolve over time while maintaining binary compatibility. Among other things, this is necessary if we want to stop bundling the Swift standard library with any app that uses Swift, a noted complaint from iOS developers. :slight_smile:

If you're wondering what this is all about, take a look at the prologue for the design document:

One of Swift’s primary design goals is to allow efficient execution of code without sacrificing load-time abstraction of implementation.

Abstraction of implementation means that code correctly written against a published interface will correctly function when the underlying implementation changes to anything which still satisfies the original interface. There are many potential reasons to provide this sort of abstraction. Apple’s primary interest is in making it easy and painless for our internal and external developers to improve the ecosystem of Apple products by creating good and secure programs and libraries; subtle deployment problems and/or unnecessary dependencies on the behavior of our implementations would work against these goals.

Our current design in Swift is to provide opt-out load-time abstraction of implementation for all language features. Alone, this would either incur unacceptable cost or force widespread opting-out of abstraction. We intend to mitigate this primarily by designing the language and its implementation to minimize unnecessary and unintended abstraction:

  • Avoiding unnecessary language guarantees and taking advantage of that flexibility to limit load-time costs.
  • Within the domain that defines an entity, all the details of its implementation are available.
  • When entities are not exposed outside their defining module, their implementation is not constrained.
  • By default, entities are not exposed outside their defining modules. This is independently desirable to reduce accidental API surface area, but happens to also interact well with the performance design.

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

RFC stands for "request for comments", and that's what this is: I'd appreciate the eager and discriminating eyes of swift-evolution on this model. It is quite long—nearly ten thousand words—and attempts to be fairly precise in describing what is and isn't allowed, so feel free to focus on the parts that interest you most. This isn't a proposal and won't be going through the Swift Evolution Process, but many existing or planned proposals will affect or support the model described here. (There's a list of them at the end of the document.)

The document is written in ReStructuredText to match the rest of the compiler documentation, but it's using some features from the Sphinx system that GitHub's ReST renderer doesn't support. Consequently, I've put up a rendered form, which I'll update every few days when there are changes. (This is pretty much the same rendering you get from running "make" in the docs/ directory in the Swift repo.) The canonical document is still the one in the Swift repository.

Looking forward to your feedback!
Jordan
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I noticed this note in the document:

Note
Swift 2’s implementation of default values puts the evaluation of the default value expression in the library, rather than in the client like C++ or C#. We plan to change this.

What is the reasoning behind this? Keeping the default value in the library seems more resilient; if the default value changes, clients will get the new behavior without requiring a recompile, thus matching the behavior that you’d get if you implemented the same thing via a method with fewer arguments (as you’d have done in Objective-C). This seems far preferable to hard-coding all the default values in the client, IMO.


(Jordan Rose) #8

In addition, it's less resilient than you think it is (and than we thought it would be when we introduced it). The library author still needs to document what the default behavior is anyway, so that users know if they want it, and they need to be very careful in changing existing programs, so that they don't break anyone who was depending on the old behavior, and they need to make sure that anybody who supplied the same value explicitly really does not want the new behavior.

After all that, it turns out there aren't really any places where you can safely change the value of a default parameter anyway.

Jordan

P.S. Credit to DaveA for convincing the rest of us. Or me, at least.

···

On Feb 8, 2016, at 19:17 , Joe Groff <jgroff@apple.com> wrote:

On Feb 8, 2016, at 6:47 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 8, 2016, at 8:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

Hi, swift-evolution. We've been making references for a while to "resilience" as a cornerstone of the Swift 3.0 work, the collection of features that allows a library to evolve over time while maintaining binary compatibility. Among other things, this is necessary if we want to stop bundling the Swift standard library with any app that uses Swift, a noted complaint from iOS developers. :slight_smile:

If you're wondering what this is all about, take a look at the prologue for the design document:

One of Swift’s primary design goals is to allow efficient execution of code without sacrificing load-time abstraction of implementation.

Abstraction of implementation means that code correctly written against a published interface will correctly function when the underlying implementation changes to anything which still satisfies the original interface. There are many potential reasons to provide this sort of abstraction. Apple’s primary interest is in making it easy and painless for our internal and external developers to improve the ecosystem of Apple products by creating good and secure programs and libraries; subtle deployment problems and/or unnecessary dependencies on the behavior of our implementations would work against these goals.

Our current design in Swift is to provide opt-out load-time abstraction of implementation for all language features. Alone, this would either incur unacceptable cost or force widespread opting-out of abstraction. We intend to mitigate this primarily by designing the language and its implementation to minimize unnecessary and unintended abstraction:

  • Avoiding unnecessary language guarantees and taking advantage of that flexibility to limit load-time costs.
  • Within the domain that defines an entity, all the details of its implementation are available.
  • When entities are not exposed outside their defining module, their implementation is not constrained.
  • By default, entities are not exposed outside their defining modules. This is independently desirable to reduce accidental API surface area, but happens to also interact well with the performance design.

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

RFC stands for "request for comments", and that's what this is: I'd appreciate the eager and discriminating eyes of swift-evolution on this model. It is quite long—nearly ten thousand words—and attempts to be fairly precise in describing what is and isn't allowed, so feel free to focus on the parts that interest you most. This isn't a proposal and won't be going through the Swift Evolution Process, but many existing or planned proposals will affect or support the model described here. (There's a list of them at the end of the document.)

The document is written in ReStructuredText to match the rest of the compiler documentation, but it's using some features from the Sphinx system that GitHub's ReST renderer doesn't support. Consequently, I've put up a rendered form, which I'll update every few days when there are changes. (This is pretty much the same rendering you get from running "make" in the docs/ directory in the Swift repo.) The canonical document is still the one in the Swift repository.

Looking forward to your feedback!
Jordan
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I noticed this note in the document:

Note
Swift 2’s implementation of default values puts the evaluation of the default value expression in the library, rather than in the client like C++ or C#. We plan to change this.

What is the reasoning behind this? Keeping the default value in the library seems more resilient; if the default value changes, clients will get the new behavior without requiring a recompile, thus matching the behavior that you’d get if you implemented the same thing via a method with fewer arguments (as you’d have done in Objective-C). This seems far preferable to hard-coding all the default values in the client, IMO.

Abstracting default values from the client limits resilience in other, more obnoxious ways. In particular, it means you can't *introduce* new defaulted parameters on published APIs without a deployment target limitation, since the entry point for the default argument evaluator will be missing in older versions. We figured that instantiating the default argument on the client side is more in line with users' expectations in what resilience constraints it imposes. In many cases, you'll have the resilinece you want at a different level; for example, in the common case of an option set where the default is 'all options', evaluating 'func foo(options: Options = .all)' is still resilient if 'Options.all' is.


(Charles Srstka) #9

The library author still needs to document what the default behavior is anyway, so that users know if they want it, and they need to be very careful in changing existing programs, so that they don't break anyone who was depending on the old behavior

It seems to me that using a default parameter means you don’t care about its value. If you explicitly want a certain value, and especially if you’re *depending* on a certain value, it would be best to supply that value explicitly. Using the default is saying “eh, do whatever you think is best."

and they need to make sure that anybody who supplied the same value explicitly really does not want the new behavior.

It seems to me that if you explicitly provide a value to a parameter which has a default, it would appear that you want that value. Else, why go to the trouble of explicitly providing it?

Charles

···

On Feb 8, 2016, at 9:24 PM, Jordan Rose <jordan_rose@apple.com> wrote:


(David Owens II) #10

Actually, there are quite a few places that allow for the same type of behavior: @inlineable, accessors, enums, structs, etc...

As soon as I see these words: "existing clients may use the new implementations, or they may use the implementations from the time they were compiled, or a mix of both" or similar, we've ventured right back into undefined territory.

That concerns me.

-David

···

On Feb 10, 2016, at 12:45 PM, David Owens II via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 8, 2016, at 6:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

Regarding this:

Changing or removing a default value is permitted but discouraged; it may break or change the meaning of existing source code.

Maybe I'm being dense, but how is something with a caveat of "discouraged" and "it may break or change" in-line with "the default behavior is safe"? I've got no qualms with putting the default value in the client code; I actually think that is fine.

However, my concern is that you will have different behavior depending on if you simply drop in the updated binary vs recompile against the binary. Even worse if you have multiple components within your app that link against the library. If only one of those components is recompiled on release, you now have a problem of conflicting behavior within your own app because of the client-side calling the API will be using different values.

It would seem that removing the default value could be permitted (though maybe still discouraged) because this will result in a compiler error in the scenario above. It's still possible to have different behavior in your components, but now it's no longer implicitly happening. However, changing the default parameter seems highly problematic.

-David
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jordan Rose) #11

This is a good point, and I think the "checker" tool described at the end of the document should warn about these kinds of changes as well. I was trying to distinguish between "changing this affects the ABI of your library and therefore breaks memory and type safety" and "changing this merely affects the behavior of your library but will not break memory and type safety", but maybe that's not such an important distinction.

It's important to note that "it may break or change the meaning of existing source code" is something that applies to any behavior change you make in a library; if version 2.0 of an opaque function 'foo' accesses global memory where it didn't before, existing clients may run into concurrency issues if they assumed the function was concurrency-safe. The specific twist for inlineable code (including default argument expressions) is that the change is triggered by recompiling the client, which is why any changes to inlineable code (including default argument expressions) should really preserve the existing contract of the API.

I actually considered leaving out the section about changing a default parameter, but that's no different from removing a default parameter and then immediately doing a second release with a new parameter added back. Calling it out explicitly is intended to serve as a warning more than an endorsement.

Jordan

···

On Feb 10, 2016, at 12:45, David Owens II <david@owensd.io> wrote:

On Feb 8, 2016, at 6:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

Regarding this:

Changing or removing a default value is permitted but discouraged; it may break or change the meaning of existing source code.

Maybe I'm being dense, but how is something with a caveat of "discouraged" and "it may break or change" in-line with "the default behavior is safe"? I've got no qualms with putting the default value in the client code; I actually think that is fine.

However, my concern is that you will have different behavior depending on if you simply drop in the updated binary vs recompile against the binary. Even worse if you have multiple components within your app that link against the library. If only one of those components is recompiled on release, you now have a problem of conflicting behavior within your own app because of the client-side calling the API will be using different values.

It would seem that removing the default value could be permitted (though maybe still discouraged) because this will result in a compiler error in the scenario above. It's still possible to have different behavior in your components, but now it's no longer implicitly happening. However, changing the default parameter seems highly problematic.


(Brent Royal-Gordon) #12

It is legal to change the implementation of an inlineable function in the next release of the library. However, any such change must be made with the understanding that it may or may not affect existing clients.

I think this is wrong.

I don't think there's a reasonable alternative. Breaking someone's entire app because a library changed is not acceptable unless you *know* there's a problem, and checking if some specific piece of inlined code has been replaced so you can fall back to a non-inlined call is likely to be more costly than just emitting a non-inlined call would be.

I think the solution is that you shouldn't mark security-critical code as being eligible for inlining. Anything else is just not going to be workable.

···

--
Brent Royal-Gordon
Architechies


(Jordan Rose) #13

Another thing I think we are missing here is versioning the function bodies themselves. For example (ignore syntax) suppose we have

public (1.0) func removeTheCacheFiles() {
      os.rmdir("/path/to/cache")
}

We may evolve this function in two orthogonal ways:

We may develop other cache files as our program grows
We may discover that we forgot to check if the user is allowed by the security policy to remove the cache files.

Therefore we may evolve this function as follows (again, I use pretend syntax):

public (1.0) func removeTheCacheFiles() {
      precondition(userIsAuthorized()) //this change is backported to 1.0
      os.rmdir("/path/to/cache")
      #if 1.1 { //this code only for 1.1-era callers
        os.rmdir("/path/to/cache2")
      }
}

It is important to support this case because very often in server land, certain clients only want to pick up the security fixes (and not, say, new features). See e.g. this Debian Security FAQ <https://www.debian.org/security/faq#oldversion>, where people spend a huge amount of time backporting security fixes to old versions.

I realize this is not at all the practice in "consumer-grade" applications like iOS/OSX/etc., but it is very entrenched in serverland, and I really think there is value in supporting this at the language level for those people who work in that world.

I think the implementation of this is just to compile all possibilities and just let the client pick the implementation based on the API version. I realize this may result in larger binaries, but only when the feature is used, so it's opt-in.

Hm, this is interesting. Apple does use a form of this, called "linked-on-or-after" checks; if you compile your code against the latest version of the OS, you're opting into new behavior. That said, it's problematic if two of your dependencies disagree on which version of the library they want. (There are a few different possible answers there, all with trade-offs.)

I think we should treat this as an additive feature; the worst that happens is you can't do this kind of multi-compilation right away.

Jordan

P.S. I want to note that most of the "innovative" ideas here are already in the Swift 2 OS availability model; main credit for this goes to Devin Coughlin.

P.P.S. I'll respond to the inlineable thread tomorrow.


(Drew Crawford) #14

I think the solution is that you shouldn't mark security-critical code as being eligible for inlining. Anything else is just not going to be workable.

There is no such thing as "I solemnly swear this function doesn't have a security bug, and if it does, we'll never patch it."

If we have to choose between security and inlining, let's not allow inlining across versioned APIs.


(Brent Royal-Gordon) #15

It seems to me that using a default parameter means you don’t care about its value. If you explicitly want a certain value, and especially if you’re *depending* on a certain value, it would be best to supply that value explicitly. Using the default is saying “eh, do whatever you think is best."

Sometimes it means you don't care about the value; sometimes it means that the parameters are rarely used and the defaults are fine for you. For instance, if you're using the variant of `NSString.compare` with no options, you probably care very much that you're getting the default "minimal frills" behavior and your search covers the entire string. You would be very surprised and probably rather annoyed if a future OS version made your code compare only the first ten characters case-insensitively.

If a library wants to support "just do what you think is best", it should explicitly model that, either by having a value like `.Automatic` or `nil` as the default, or by providing a separate method. `NSString`, for example, takes the latter option in `localizedStandardCompare`.

···

--
Brent Royal-Gordon
Architechies


(Goffredo Marocchi) #16

It seems to me that using a default parameter means you don’t care about its value. If you explicitly want a certain value, and especially if you’re *depending* on a certain value, it would be best to supply that value explicitly. Using the default is saying “eh, do whatever you think is best."

Sometimes it means you don't care about the value; sometimes it means that the parameters are rarely used and the defaults are fine for you. For instance, if you're using the variant of `NSString.compare` with no options, you probably care very much that you're getting the default "minimal frills" behavior and your search covers the entire string. You would be very surprised and probably rather annoyed if a future OS version made your code compare only the first ten characters case-insensitively.

Conscious of infuriating lots of non unit testing fanatics, this does make a good case for writing your own unit tests against the API's you commonly use too ;).

···

Sent from my iPhone
On 9 Feb 2016, at 05:32, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

If a library wants to support "just do what you think is best", it should explicitly model that, either by having a value like `.Automatic` or `nil` as the default, or by providing a separate method. `NSString`, for example, takes the latter option in `localizedStandardCompare`.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Joe Groff) #17

Actually, there are quite a few places that allow for the same type of behavior: @inlineable, accessors, enums, structs, etc...

Of these, inlineable functions definitely have the same problem, but I'm not sure what you're referring to with regard to accessors, enums, or structs. These will all be accessed by opaque interfaces (unless declared fixed-contents), so clients are guaranteed to use the framework implementation.

-Joe

···

On Feb 10, 2016, at 1:00 PM, David Owens II via swift-evolution <swift-evolution@swift.org> wrote:

As soon as I see these words: "existing clients may use the new implementations, or they may use the implementations from the time they were compiled, or a mix of both" or similar, we've ventured right back into undefined territory.

That concerns me.

-David

On Feb 10, 2016, at 12:45 PM, David Owens II via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 8, 2016, at 6:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

Regarding this:

Changing or removing a default value is permitted but discouraged; it may break or change the meaning of existing source code.

Maybe I'm being dense, but how is something with a caveat of "discouraged" and "it may break or change" in-line with "the default behavior is safe"? I've got no qualms with putting the default value in the client code; I actually think that is fine.

However, my concern is that you will have different behavior depending on if you simply drop in the updated binary vs recompile against the binary. Even worse if you have multiple components within your app that link against the library. If only one of those components is recompiled on release, you now have a problem of conflicting behavior within your own app because of the client-side calling the API will be using different values.

It would seem that removing the default value could be permitted (though maybe still discouraged) because this will result in a compiler error in the scenario above. It's still possible to have different behavior in your components, but now it's no longer implicitly happening. However, changing the default parameter seems highly problematic.

-David
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jordan Rose) #18

I do object to the word "undefined", which in the C family of languages means "the compiler can do anything it wants". In this case you will definitely get behavior equivalent to the old code, or equivalent to the new code, and the section on optimizing inlineable functions makes it clear that the inlineable code must not make assumptions about what library it's deployed against. Again, memory and type safety are preserved, while under "undefined behavior" in a C language they definitely would not be.

Jordan

···

On Feb 10, 2016, at 13:00, David Owens II <david@owensd.io> wrote:

Actually, there are quite a few places that allow for the same type of behavior: @inlineable, accessors, enums, structs, etc...

As soon as I see these words: "existing clients may use the new implementations, or they may use the implementations from the time they were compiled, or a mix of both" or similar, we've ventured right back into undefined territory.

That concerns me.

-David

On Feb 10, 2016, at 12:45 PM, David Owens II via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 8, 2016, at 6:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

Regarding this:

Changing or removing a default value is permitted but discouraged; it may break or change the meaning of existing source code.

Maybe I'm being dense, but how is something with a caveat of "discouraged" and "it may break or change" in-line with "the default behavior is safe"? I've got no qualms with putting the default value in the client code; I actually think that is fine.

However, my concern is that you will have different behavior depending on if you simply drop in the updated binary vs recompile against the binary. Even worse if you have multiple components within your app that link against the library. If only one of those components is recompiled on release, you now have a problem of conflicting behavior within your own app because of the client-side calling the API will be using different values.

It would seem that removing the default value could be permitted (though maybe still discouraged) because this will result in a compiler error in the scenario above. It's still possible to have different behavior in your components, but now it's no longer implicitly happening. However, changing the default parameter seems highly problematic.

-David
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(David Owens II) #19

This last point is a specific case of a general tenet of Swift: the default behavior is safe. Where possible, choices made when an entity is first published should not limit its evolution in the future.

Regarding this:

Changing or removing a default value is permitted but discouraged; it may break or change the meaning of existing source code.

Maybe I'm being dense, but how is something with a caveat of "discouraged" and "it may break or change" in-line with "the default behavior is safe"? I've got no qualms with putting the default value in the client code; I actually think that is fine.

However, my concern is that you will have different behavior depending on if you simply drop in the updated binary vs recompile against the binary. Even worse if you have multiple components within your app that link against the library. If only one of those components is recompiled on release, you now have a problem of conflicting behavior within your own app because of the client-side calling the API will be using different values.

It would seem that removing the default value could be permitted (though maybe still discouraged) because this will result in a compiler error in the scenario above. It's still possible to have different behavior in your components, but now it's no longer implicitly happening. However, changing the default parameter seems highly problematic.

This is a good point, and I think the "checker" tool described at the end of the document should warn about these kinds of changes as well. I was trying to distinguish between "changing this affects the ABI of your library and therefore breaks memory and type safety" and "changing this merely affects the behavior of your library but will not break memory and type safety", but maybe that's not such an important distinction.

I think the stated desire to use semantic versioning moves the conversation from strictly ABI and satisfying the compiler to the realm of attempting to create behavioral contracts. So when I see things that will have an impact on behavior too, I get a little concerned.

It's important to note that "it may break or change the meaning of existing source code" is something that applies to any behavior change you make in a library; if version 2.0 of an opaque function 'foo' accesses global memory where it didn't before, existing clients may run into concurrency issues if they assumed the function was concurrency-safe. The specific twist for inlineable code (including default argument expressions) is that the change is triggered by recompiling the client, which is why any changes to inlineable code (including default argument expressions) should really preserve the existing contract of the API.

I agree. Any change to source is a potential breaking change even if the ABI is kept in-tact. This is one of the primary arguments against semantic versioning in the first place in that it's a contract that cannot actually be adhered to.

I think the document would be stronger if it simply stated up-front that you are *only* addressing the ABI compatibility (maybe it does but the semantic version carries a ton of baggage that counters that position). This allows you to have potential behavior breaking changes while not confusing the fact.

It would still be nice to be at least warned on changes that could have a compile-time vs. link-time difference.

-David

···

On Feb 10, 2016, at 1:03 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Feb 10, 2016, at 12:45, David Owens II <david@owensd.io <mailto:david@owensd.io>> wrote:

On Feb 8, 2016, at 6:24 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:


(Dave Abrahams) #20

It can't mean that. When you call a method that has defaults for some of
the arguments, you need to know what it will mean to omit those arguments.
Otherwise, you can't take advantage of the defaults at all, and they
become a danger rather than a convenience.

···

on Mon Feb 08 2016, Charles Srstka <swift-evolution@swift.org> wrote:

On Feb 8, 2016, at 9:24 PM, Jordan Rose <jordan_rose@apple.com> wrote:

The library author still needs to document what the default behavior
is anyway, so that users know if they want it, and they need to be
very careful in changing existing programs, so that they don't break
anyone who was depending on the old behavior

It seems to me that using a default parameter means you don’t care
about its value. If you explicitly want a certain value, and
especially if you’re *depending* on a certain value, it would be best
to supply that value explicitly. Using the default is saying “eh, do
whatever you think is best."

--
-Dave