Dependency consistency within package major version

[tachyonics (Simon Pilkington)]

Hi,

It looks like a module could in theory change dependencies on a minor version or patch change but only to the resolution of a major version. This is probably only going to become an issue when dependency resolution is introduced - where changing dependencies has the potential to suddenly break the consistency of a dependency graph of an unchanged consumer; are modules going to be required to stabilise their dependency graph within a major version?

Expanding on the 'Enforced Semantic Versioning’ section of the proposal doc, a dependency change could be detected and treated the same as a change in the signature of a public method but this section doesn't explicitly mention how or if dependencies will be taken into account for versioning.

-Simon

[Max_Howell]

Hi Simon.

It looks like a module could in theory change dependencies on a minor version or patch change but only to the resolution of a major version.

Specifically, a package could change its dependencies at any version revision. Though in practice an end-user should choose packages from authors they trust to not make such changes without care.

This is probably only going to become an issue when dependency resolution is introduced - where changing dependencies has the potential to suddenly break the consistency of a dependency graph of an unchanged consumer; are modules going to be required to stabilise their dependency graph within a major version?

Indeed, a Package’s dependencies can be changed at any version bump. This is OK because: should a dependency bump its major version it will cause your Package’s major version to bump if that dependency is exposed as part of your Package’s public API. For example:

class Dependency {
  func foo()
}

// Your package

class Consumer {
   func bar() -> Dependency
}

// Now if Dependency changes its major version:

class DependencyNameChanged {
}

// Your package will have its major version changed also:

class Consumer {
   func bar() -> DependencyNameChanged
}

This is hard to detect for users and a source of dependency hell, which is why I strongly desire for the versions of our packaging system to be calculated computationally via tooling we write.

At this time correctly versioning packages is up to the authors; a non-ideal situation because it is *hard*.

Expanding on the 'Enforced Semantic Versioning’ section of the proposal doc, a dependency change could be detected and treated the same as a change in the signature of a public method but this section doesn't explicitly mention how or if dependencies will be taken into account for versioning.

I hope my above example clarified how this would work. We will be examining the AST (or equivalent) and determining version changes that way, so in theory the above would not have triggered a major version bump if the specific class had not changed its public API. Though I think in practice this is rare.

This is probably only going to become an issue when dependency resolution is introduced

We already do dependency resolution.

Max

[tachyonics (Simon Pilkington)]

Max, thanks for your response.

Hi Simon.

It looks like a module could in theory change dependencies on a minor version or patch change but only to the resolution of a major version.

Specifically, a package could change its dependencies at any version revision. Though in practice an end-user should choose packages from authors they trust to not make such changes without care.

This is probably only going to become an issue when dependency resolution is introduced - where changing dependencies has the potential to suddenly break the consistency of a dependency graph of an unchanged consumer; are modules going to be required to stabilise their dependency graph within a major version?

Indeed, a Package’s dependencies can be changed at any version bump. This is OK because: should a dependency bump its major version it will cause your Package’s major version to bump if that dependency is exposed as part of your Package’s public API. For example:

class Dependency {
  func foo()
}

// Your package

class Consumer {
  func bar() -> Dependency
}

// Now if Dependency changes its major version:

class DependencyNameChanged {
}

// Your package will have its major version changed also:

class Consumer {
  func bar() -> DependencyNameChanged
}

This is hard to detect for users and a source of dependency hell, which is why I strongly desire for the versions of our packaging system to be calculated computationally via tooling we write.

At this time correctly versioning packages is up to the authors; a non-ideal situation because it is *hard*.

Expanding on the 'Enforced Semantic Versioning’ section of the proposal doc, a dependency change could be detected and treated the same as a change in the signature of a public method but this section doesn't explicitly mention how or if dependencies will be taken into account for versioning.

I hope my above example clarified how this would work. We will be examining the AST (or equivalent) and determining version changes that way, so in theory the above would not have triggered a major version bump if the specific class had not changed its public API. Though I think in practice this is rare.

I think this is a great approach, particularly for larger projects where even a dev familiar with the project can mistakenly change public API. Plus there is potential for things like automated release notes.

This is probably only going to become an issue when dependency resolution is introduced

We already do dependency resolution.

My question actually related to dependency (sub-dependency) conflict resolution which is listed as future feature.

For example, considering the initial state

Module A -> depends on Module B v.1 (1.0.0) -> depends on Module D v.1
Module A -> depends on Module E v.1 (1.0.0) -> depends on Module D v.1

In this case you would get a consistent dependency graph and everything would correctly compile statically into Module A. But if Module E makes a minor version update so that

Module A -> depends on Module B v.1 (1.0.0) -> depends on Module D v.1
Module A -> depends on Module E v.1 (1.1.0) -> depends on Module D v.2

Without changing any code, Module A suddenly doesn’t have a consistent dependency graph, even if the changes to the public API of Module D don’t happen to impact the public APIs of Module E or Module A. Is this a concern or am I missing something here?

-Simon

···

On 7 Dec 2015, at 12:50 PM, Max Howell <max.howell@apple.com> wrote:

Max

[Max_Howell]

My question actually related to dependency (sub-dependency) conflict resolution which is listed as future feature.

For example, considering the initial state

Module A -> depends on Module B v.1 (1.0.0) -> depends on Module D v.1
Module A -> depends on Module E v.1 (1.0.0) -> depends on Module D v.1

In this case you would get a consistent dependency graph and everything would correctly compile statically into Module A. But if Module E makes a minor version update so that

Module A -> depends on Module B v.1 (1.0.0) -> depends on Module D v.1
Module A -> depends on Module E v.1 (1.1.0) -> depends on Module D v.2

Without changing any code, Module A suddenly doesn’t have a consistent dependency graph, even if the changes to the public API of Module D don’t happen to impact the public APIs of Module E or Module A. Is this a concern or am I missing something here?

Thank you, I understand now.

Indeed, this is yet another possible dependency hell, so I’ll add it to our list [1]

We have at least one strategy that we hope we will be able to implement, i.e.: pure modules. We would like people to make their libraries small, focused and “pure”—in that they do not have any global state (even things like file system access). If they then declare their purity we can load both version 1 and version 2 of the same module into the same process.

This would work even if the modules are exposed publicly via API to another module lower in the graph, though in such cases it may be confusing to consumers of that library. So we may insist this sort of thing can only apply if the modules don’t get exported.

Another avenue we have explored is being more forceful with what package authors can do. That is, prohibiting such actions altogether. This has the benefit that dependency hell is much less likely. Ultimately we decided that sometimes you have to make such changes and it would be against our greater goals of providing flexible tools to have it operate this way.

Instead we plan to add a good deal of warnings to the tool that eventually will lint/publish packages. In the above case we would warn in big colorful letters to the user: “WARNING increasing the dependency version of Foo from v1 to v2 is irresponsible! Be sure you have good rationale before pushing!”

I would be interested to hear any more thoughts you have and if you think our approach will be successful. Thanks,

Max

[1]: https://github.com/apple/swift-package-manager/blob/master/Documentation/DependencyHells.md

[tachyonics (Simon Pilkington)]

As you said, versioning is hard and these are the reasons why.

I am guessing that to be able to load two versions of the same module into the same process the package builder re-namespace one or both to avoid a collision and change the referencing modules. Something like this should work, the only concern is making sure you limit the amount of ‘black magic’ that is going on - make sure its clearly explained what the package builder is doing otherwise there might be side effects - such as the size of a binary increasing because there are now two versions of a sub-module present - that suddenly appear without explanation.

I think the approach of warning on these issues is a good one - at most having them as errors that fail the build but then can be explicitly ignored (as its very easy for devs to take the ‘well it compiled, must be good’ approach and completely ignore warnings). In terms 'make doing the correct/common thing easy and the rest possible’ its rare that you’d actually want to outright prohibit something so your approach seems solid with respect to that.

I’m pretty excited by what I have seen so far with the package builder and it looks like its starting off with a good foundation. Let me know if there is anything I can contribute.

-Simon

···

On 8 Dec 2015, at 11:03 AM, Max Howell <max.howell@apple.com> wrote:

My question actually related to dependency (sub-dependency) conflict resolution which is listed as future feature.

For example, considering the initial state

Module A -> depends on Module B v.1 (1.0.0) -> depends on Module D v.1
Module A -> depends on Module E v.1 (1.0.0) -> depends on Module D v.1

In this case you would get a consistent dependency graph and everything would correctly compile statically into Module A. But if Module E makes a minor version update so that

Module A -> depends on Module B v.1 (1.0.0) -> depends on Module D v.1
Module A -> depends on Module E v.1 (1.1.0) -> depends on Module D v.2

Without changing any code, Module A suddenly doesn’t have a consistent dependency graph, even if the changes to the public API of Module D don’t happen to impact the public APIs of Module E or Module A. Is this a concern or am I missing something here?

Thank you, I understand now.

Indeed, this is yet another possible dependency hell, so I’ll add it to our list [1]

We have at least one strategy that we hope we will be able to implement, i.e.: pure modules. We would like people to make their libraries small, focused and “pure”—in that they do not have any global state (even things like file system access). If they then declare their purity we can load both version 1 and version 2 of the same module into the same process.

This would work even if the modules are exposed publicly via API to another module lower in the graph, though in such cases it may be confusing to consumers of that library. So we may insist this sort of thing can only apply if the modules don’t get exported.

Another avenue we have explored is being more forceful with what package authors can do. That is, prohibiting such actions altogether. This has the benefit that dependency hell is much less likely. Ultimately we decided that sometimes you have to make such changes and it would be against our greater goals of providing flexible tools to have it operate this way.

Instead we plan to add a good deal of warnings to the tool that eventually will lint/publish packages. In the above case we would warn in big colorful letters to the user: “WARNING increasing the dependency version of Foo from v1 to v2 is irresponsible! Be sure you have good rationale before pushing!”

I would be interested to hear any more thoughts you have and if you think our approach will be successful. Thanks,

Max

[1]: https://github.com/apple/swift-package-manager/blob/master/Documentation/DependencyHells.md

[Max_Howell]

As you said, versioning is hard and these are the reasons why.

I am guessing that to be able to load two versions of the same module into the same process the package builder re-namespace one or both to avoid a collision and change the referencing modules.

Yes, it would be nice to get support from the language for this, so I will propose it to swift-core, hopefully something decent in the namespacing area will come of it.

Something like this should work, the only concern is making sure you limit the amount of ‘black magic’ that is going on - make sure its clearly explained what the package builder is doing otherwise there might be side effects - such as the size of a binary increasing because there are now two versions of a sub-module present - that suddenly appear without explanation.

I dislike opaque systems, so I intend to shower warnings when the packages are initially cloned and this situation is discovered.

I think the approach of warning on these issues is a good one - at most having them as errors that fail the build but then can be explicitly ignored (as its very easy for devs to take the ‘well it compiled, must be good’ approach and completely ignore warnings). In terms 'make doing the correct/common thing easy and the rest possible’ its rare that you’d actually want to outright prohibit something so your approach seems solid with respect to that.

We could explore erroring builds unless a —allow-this-stuff type flag is specified. I’m okay with this since IMO the developer should be encouraged to consider these situations carefully. But simultaneously if you make a tool that is tedious, people find alternatives. So we should tread carefully.

I’m pretty excited by what I have seen so far with the package builder and it looks like its starting off with a good foundation. Let me know if there is anything I can contribute.

Great to hear! We’re very exited too and are really glad to get such excellent feedback! Thanks,

Max