[Review] SE-0019 Swift Testing (Package Manager)

Now lint … lint I’m not sure about. It’s problematic as soon as you try to establish a standard, as we’ve already established on this list in the “mandatory self” discussion. The truth is that I don’t care about a third party lib’s weird formatting decisions nearly as much as I care about its tests passing.

Here by “lint” I mean packaging concerns for linting, specifically: is the semantic version the package claims *correct*. Code standards are not part of the publishing lint step as indeed, this seems excessive and not a decision for us to make.

Ah, that makes much more sense! Sorry for being slow on the uptake. Yes, programmatic validation of semantic versioning seems like a fine idea.

Cheers, P

···

On Jan 5, 2016, at 5:11 PM, Max Howell <max.howell@apple.com> wrote:

Now lint … lint I’m not sure about. It’s problematic as soon as you try to establish a standard, as we’ve already established on this list in the “mandatory self” discussion. The truth is that I don’t care about a third party lib’s weird formatting decisions nearly as much as I care about its tests passing.

Here by “lint” I mean packaging concerns for linting, specifically: is the semantic version the package claims *correct*. Code standards are not part of the publishing lint step as indeed, this seems excessive and not a decision for us to make.

We won’t be running tests by default, just building them by default. We have to pick a build-configuration as a default.

I’m sorry I must be missing something, I don’t see how we are only supporting unit tests. At first XCTest is the only framework, but part of the proposal explains how in the future we will support any framework that can conform to a protocol. That protocol is the topic for another proposal.

When there is stuff like this in the proposal:

"We would prefer to go even further by executing the tests each time a module is built as well, but we understand that this would impede debug cycles.”

That implies that there is a desire to run tests by default. That also suggests the design of the proposal is geared for a very specific type of testing category, specifically unit tests. Maybe that’s not what is meant, but that is how I read it.

Then please provide and specify how this can be opt-ed out of clearly. Is it the --without-tests flag? Can we specific default options in the Package.swift so that this always occurs?

As a simple query, go around Apple and ask the compiler team if they want this feature for their code. Ask those developing in the corelibs. Ask any of the app teams. Please also do performance testing on this for various sized projects. It will absolutely have an impact, especially with regards to the current state of Swift compile times.

The proposal specifies there will be a flag, the flag is TBD. Probably exactly what you specified.

What about the defaults so that whenever “swift build” is run, specific flags are used? Is that out of scope - maybe with the build configurations? Or is that intended to be done via a layer of “build scripts” so teams can have consistent workflows?

In the future, we may choose to promote the --test option to be a subcommand of the swift command itself:

$ swift test

However, any such decision would warrant extensive design consideration, so as to avoid polluting or crowding the command-line interface. Should there be sufficient demand and justification for it, though, it would be straightforward to add this functionality.

This doesn’t make sense to me. It’s either straightforward, or it requires extensive design consideration. I personally strongly dislike coupling the notion of building with test execution, so I would much rather see “swift test” used, especially with a look into the future where it’s going to be asked for the ability to run categories of tests and filter to a particular set of tests to be run.

I agree, but my issue with `swift test` is that we are beginning to tightly couple `swift build` with swift itself. I’m not sure we should run-and-gun into this decision. SwiftPM is strictly alpha/beta and any decisions we make on these sorts of issues can change.

The alternatives are:

1. Factor out to a new binary, like spm, or
2. Continue down the path of tightly coupling build code with test execution.

I’d actually like to see the Swift Package Manager factor out to it’s own entity sooner rather than later and instead shell out and use the swift compiler.

This is already how it is.

Huh? Isn’t it tied to the “swift” compiler as a plug-in of sorts? Maybe I don’t understand how it works. I’m talking about removing it completely from the compiler infrastructure. But we can have this discussion a different time.

This would also help with other proposals, such as the “Getting C code compiling” one. But that’s a different proposal...

Another real problem with this type of design, is that is makes more advanced build systems very complicated to make. For example, distributed builds that bring together all of the compiled bits to run tests on get blocked because the build command starts to expect certain intermediate output. This is a real problem my previous team still has to this day with xcodebuild and actively prevents us from doing this exact thing with standard tools from Apple, so we have to basically roll our own. I see this design following in the exact same footsteps.

I’m not sure I understand.

Every time you run “swift build” it’s going to trigger a build. This is should never be required to run tests. Even if you say, “we’ll only re-build on incremental builds”, you are assuming that you are building and running tests within the same environment and that the environment is a build-capable environment. This is not a valid assumption. This is what xcodebuild test does today, it tries to smartly only trigger builds when necessary, but since we build on one set of boxes and run tests on other machines, this simply doesn’t work.

We have the opportunity to build a proper toolset that can be correctly used within many different workflows. However, I already see a lot of the coupling starting to happen today that has happened within xcodebuild already, and that really concerns me. Basically, it means that these tools aren’t going to be usable outside of the smaller-scale apps, and teams like mine are going to have to continue investing in more robust and flexible tools to enable our scenarios.

Well, it is not too late to uncouple these two pieces. Can you provide justification as to why we should decouple them? The engineering is not terrible to decouple them, but it would make things more complicated. Currently a good deal of the knowledge about where built products are and how they relate to targets does not need to be communicated outside of a single process, and simpler systems have less bugs, so I’d prefer not to complicate things without good rationale.

I don’t know what other examples to really provide here… the ability to build is completely orthogonal to the ability to test. Given a set of binaries, I should be able to run the tests. The tests are not always statically linked to the testing target. Often times they are dynamically injected or used to drive the program externally.

Maybe this provides some better context: some of our apps take multiple hours to perform full builds. These apps are tested across multiple OS installations, hardware configurations, and locales. The apps are not built on all of these target machines. All of these machines should be able to run the same tests available on a developer’s machine, and there is no fundamental reason why they cannot be run in the same way (this is historically been the case with Xcode projects because of arbitrary decisions and unnecessary coupling).

With this proposal, how do I run these tests without triggering a build? Is “swift build --test” only going to run the tests without attempting to build first? That is not how I read the proposal.

I spent a good deal of my time when I was on the Office for Mac team at Microsoft working around all of limitations that made it extremely challenging to actually make use of many of the tools coming out of Apple’s Xcode team. I’d like to see the Swift tools not go down the same path and be tightly coupled in each of the phases so that integration because a huge pain, especially when integration needs to happen with non-Swift tools.

-David

···

On Jan 5, 2016, at 4:06 PM, Max Howell <max.howell@apple.com> wrote:

np, I should have been more specific.

···

On Jan 5, 2016, at 3:15 PM, Paul Cantrell <cantrell@pobox.com> wrote:

On Jan 5, 2016, at 5:11 PM, Max Howell <max.howell@apple.com> wrote:

Now lint … lint I’m not sure about. It’s problematic as soon as you try to establish a standard, as we’ve already established on this list in the “mandatory self” discussion. The truth is that I don’t care about a third party lib’s weird formatting decisions nearly as much as I care about its tests passing.

Here by “lint” I mean packaging concerns for linting, specifically: is the semantic version the package claims *correct*. Code standards are not part of the publishing lint step as indeed, this seems excessive and not a decision for us to make.

Ah, that makes much more sense! Sorry for being slow on the uptake. Yes, programmatic validation of semantic versioning seems like a fine idea.

We won’t be running tests by default, just building them by default. We have to pick a build-configuration as a default.

I’m sorry I must be missing something, I don’t see how we are only supporting unit tests. At first XCTest is the only framework, but part of the proposal explains how in the future we will support any framework that can conform to a protocol. That protocol is the topic for another proposal.

When there is stuff like this in the proposal:

"We would prefer to go even further by executing the tests each time a module is built as well, but we understand that this would impede debug cycles.”

That implies that there is a desire to run tests by default. That also suggests the design of the proposal is geared for a very specific type of testing category, specifically unit tests. Maybe that’s not what is meant, but that is how I read it.

I think this is just a misunderstanding. We’d like to run tests all the time, all the tests, all types of tests, this would be great. But practically we cannot.

Then please provide and specify how this can be opt-ed out of clearly. Is it the --without-tests flag? Can we specific default options in the Package.swift so that this always occurs?

As a simple query, go around Apple and ask the compiler team if they want this feature for their code. Ask those developing in the corelibs. Ask any of the app teams. Please also do performance testing on this for various sized projects. It will absolutely have an impact, especially with regards to the current state of Swift compile times.

The proposal specifies there will be a flag, the flag is TBD. Probably exactly what you specified.

What about the defaults so that whenever “swift build” is run, specific flags are used? Is that out of scope - maybe with the build configurations? Or is that intended to be done via a layer of “build scripts” so teams can have consistent workflows?

I would like this feature. We should discuss it in another thread, but both me and Daniel have ideas already in this space, so we’d enjoy the discussion.

In the future, we may choose to promote the --test option to be a subcommand of the swift command itself:

$ swift test

However, any such decision would warrant extensive design consideration, so as to avoid polluting or crowding the command-line interface. Should there be sufficient demand and justification for it, though, it would be straightforward to add this functionality.

This doesn’t make sense to me. It’s either straightforward, or it requires extensive design consideration. I personally strongly dislike coupling the notion of building with test execution, so I would much rather see “swift test” used, especially with a look into the future where it’s going to be asked for the ability to run categories of tests and filter to a particular set of tests to be run.

I agree, but my issue with `swift test` is that we are beginning to tightly couple `swift build` with swift itself. I’m not sure we should run-and-gun into this decision. SwiftPM is strictly alpha/beta and any decisions we make on these sorts of issues can change.

The alternatives are:

1. Factor out to a new binary, like spm, or
2. Continue down the path of tightly coupling build code with test execution.

I’d actually like to see the Swift Package Manager factor out to it’s own entity sooner rather than later and instead shell out and use the swift compiler.

This is already how it is.

Huh? Isn’t it tied to the “swift” compiler as a plug-in of sorts? Maybe I don’t understand how it works. I’m talking about removing it completely from the compiler infrastructure. But we can have this discussion a different time.

swift-build is a separate executable and its sources are in a separate repository, the swift-multitool invokes it.

This would also help with other proposals, such as the “Getting C code compiling” one. But that’s a different proposal...

Another real problem with this type of design, is that is makes more advanced build systems very complicated to make. For example, distributed builds that bring together all of the compiled bits to run tests on get blocked because the build command starts to expect certain intermediate output. This is a real problem my previous team still has to this day with xcodebuild and actively prevents us from doing this exact thing with standard tools from Apple, so we have to basically roll our own. I see this design following in the exact same footsteps.

I’m not sure I understand.

Every time you run “swift build” it’s going to trigger a build. This is should never be required to run tests. Even if you say, “we’ll only re-build on incremental builds”, you are assuming that you are building and running tests within the same environment and that the environment is a build-capable environment. This is not a valid assumption. This is what xcodebuild test does today, it tries to smartly only trigger builds when necessary, but since we build on one set of boxes and run tests on other machines, this simply doesn’t work.

We have the opportunity to build a proper toolset that can be correctly used within many different workflows. However, I already see a lot of the coupling starting to happen today that has happened within xcodebuild already, and that really concerns me. Basically, it means that these tools aren’t going to be usable outside of the smaller-scale apps, and teams like mine are going to have to continue investing in more robust and flexible tools to enable our scenarios.

Well, it is not too late to uncouple these two pieces. Can you provide justification as to why we should decouple them? The engineering is not terrible to decouple them, but it would make things more complicated. Currently a good deal of the knowledge about where built products are and how they relate to targets does not need to be communicated outside of a single process, and simpler systems have less bugs, so I’d prefer not to complicate things without good rationale.

I don’t know what other examples to really provide here… the ability to build is completely orthogonal to the ability to test. Given a set of binaries, I should be able to run the tests. The tests are not always statically linked to the testing target. Often times they are dynamically injected or used to drive the program externally.

Maybe this provides some better context: some of our apps take multiple hours to perform full builds. These apps are tested across multiple OS installations, hardware configurations, and locales. The apps are not built on all of these target machines. All of these machines should be able to run the same tests available on a developer’s machine, and there is no fundamental reason why they cannot be run in the same way (this is historically been the case with Xcode projects because of arbitrary decisions and unnecessary coupling).

With this proposal, how do I run these tests without triggering a build? Is “swift build --test” only going to run the tests without attempting to build first? That is not how I read the proposal.

A fast null-build negates this. But if it turns out our null-builds are not fast enough for large projects we will happily accept a patch to allow you to skip the build and just run the tests.

Notably: currently the tests compile to executables, so once they are built you can run them independently. I intend to keep this design, if possible.

I spent a good deal of my time when I was on the Office for Mac team at Microsoft working around all of limitations that made it extremely challenging to actually make use of many of the tools coming out of Apple’s Xcode team. I’d like to see the Swift tools not go down the same path and be tightly coupled in each of the phases so that integration because a huge pain, especially when integration needs to happen with non-Swift tools.

Thanks for voicing your concerns. I believe we are doing a pretty thorough job already with `swift build` of decoupling it from the rest of the system. As the product evolves and if you feel we are not taking the right approach I’ll gladly receive the criticism and try to respond to it with engineering.

One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?

  - Rick

···

On Jan 6, 2016, at 11:52 AM, Max Howell <max.howell@apple.com> wrote:

Another real problem with this type of design, is that is makes more advanced build systems very complicated to make. For example, distributed builds that bring together all of the compiled bits to run tests on get blocked because the build command starts to expect certain intermediate output. This is a real problem my previous team still has to this day with xcodebuild and actively prevents us from doing this exact thing with standard tools from Apple, so we have to basically roll our own. I see this design following in the exact same footsteps.

I’m not sure I understand.

Every time you run “swift build” it’s going to trigger a build. This is should never be required to run tests. Even if you say, “we’ll only re-build on incremental builds”, you are assuming that you are building and running tests within the same environment and that the environment is a build-capable environment. This is not a valid assumption. This is what xcodebuild test does today, it tries to smartly only trigger builds when necessary, but since we build on one set of boxes and run tests on other machines, this simply doesn’t work.

We have the opportunity to build a proper toolset that can be correctly used within many different workflows. However, I already see a lot of the coupling starting to happen today that has happened within xcodebuild already, and that really concerns me. Basically, it means that these tools aren’t going to be usable outside of the smaller-scale apps, and teams like mine are going to have to continue investing in more robust and flexible tools to enable our scenarios.

Well, it is not too late to uncouple these two pieces. Can you provide justification as to why we should decouple them? The engineering is not terrible to decouple them, but it would make things more complicated. Currently a good deal of the knowledge about where built products are and how they relate to targets does not need to be communicated outside of a single process, and simpler systems have less bugs, so I’d prefer not to complicate things without good rationale.

I don’t know what other examples to really provide here… the ability to build is completely orthogonal to the ability to test. Given a set of binaries, I should be able to run the tests. The tests are not always statically linked to the testing target. Often times they are dynamically injected or used to drive the program externally.

Maybe this provides some better context: some of our apps take multiple hours to perform full builds. These apps are tested across multiple OS installations, hardware configurations, and locales. The apps are not built on all of these target machines. All of these machines should be able to run the same tests available on a developer’s machine, and there is no fundamental reason why they cannot be run in the same way (this is historically been the case with Xcode projects because of arbitrary decisions and unnecessary coupling).

With this proposal, how do I run these tests without triggering a build? Is “swift build --test” only going to run the tests without attempting to build first? That is not how I read the proposal.

A fast null-build negates this. But if it turns out our null-builds are not fast enough for large projects we will happily accept a patch to allow you to skip the build and just run the tests.

Notably: currently the tests compile to executables, so once they are built you can run them independently. I intend to keep this design, if possible.

I spent a good deal of my time when I was on the Office for Mac team at Microsoft working around all of limitations that made it extremely challenging to actually make use of many of the tools coming out of Apple’s Xcode team. I’d like to see the Swift tools not go down the same path and be tightly coupled in each of the phases so that integration because a huge pain, especially when integration needs to happen with non-Swift tools.

Thanks for voicing your concerns. I believe we are doing a pretty thorough job already with `swift build` of decoupling it from the rest of the system. As the product evolves and if you feel we are not taking the right approach I’ll gladly receive the criticism and try to respond to it with engineering.

One more thing:

Notably: currently the tests compile to executables, so once they are built you can run them independently. I intend to keep this design, if possible.

…if I am interpreting this correctly, I’m not sure I like it and I realized another issue isn’t being explicitly-addressed in the proposal:

- If there is test-only utility code, where does it live? How much visibility do the various test modules have into each other?

…and seeing that the tests are currently intended to get compiled to individual executables makes me at least a little concerned that the answer might be “they have no such mutual visibility”.

Can the document at least clarify the intended/philosophical outlook here?

I ask b/c I’ve found that it isn’t that uncommon in certain cases to wind up with data-driven tests; e.g. the actual input-output pairs might live in some JSON/XML/whatever, and the test code just imports-and-parses them, checks your code against them, and complains about mis-matches.

At present it’s easy to create-and-use such test-specific helper code; it’s not clear if that’s the case under the proposal.

Relatedly, I’m aware that resource-packing is currently a “future feature” for the PM itself, so I don’t begrudge it’s not being handled in this proposal; nevertheless it seems important enough to also go into a section touching on “future directions/future goals”.

···

On Jan 6, 2016, at 1:52 PM, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?

  - Rick

No, it doesn’t. While on the Office for Mac team, we repeatedly asked for many of the items in Xcode and xcodebuild to be made less coupled with each of the steps in the process. We were told no in most cases and given hacky workarounds in others that were prone to break with each incremental update to Xcode. Most of these breaks were found out because we picked up the latest beta build and attempted to run our automation tools only to find that something subtle had changed in our workaround or the basis of the workaround is no longer there.

Sure, SPM is “open source”, however, it’s unrealistic for people/teams to maintain forks of the tools to support features they need. Maybe the changes get accepted into a pull request, maybe not.

Honestly, I’m a bit shocked that the separation of building and executing tests is such a controversial thing. There should be three high-level tasks:

1. build
2. run-tests
3. build + run-tests

What I’m saying is that by not supporting #2, you’re creating an adoption blocker for any team that cannot or does not want to build every time it runs their tests. There are many reasons for this. I’ve already given several, however, another reason is instrumentation for things like code-coverage or product code markers, often happen as an intermediate step outside of the normal build operation. This is especially true in the cases of bug investigation. Saying that I cannot trigger the running of these tests without a build step baffling to me.

-David

I am not saying that we won't support this; I expect that we will, for the very good reasons that you've raised. I'm just saying that this isn't some fundamental architectural limitation that needs to be addressed by this proposal; it's just a matter of what options we expose in the (command-line) UI, and is fairly easy to change. (We should make sure that that's the case – that this is implemented as separable build and test steps – but I expect we'd be doing that anyway).

I don't want to derail this thread with discussion of Xcode's behavior, which is outside the scope of this project, but if you want to send me (off-thread) the radar # for the bug(s) you've filed about supporting xcodebuild test-without-build, I'd be happy to take a look at what happened there.

   -Rick

···

On Jan 7, 2016, at 11:18 AM, David Owens II <david@owensd.io> wrote:

One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?

  - Rick

No, it doesn’t. While on the Office for Mac team, we repeatedly asked for many of the items in Xcode and xcodebuild to be made less coupled with each of the steps in the process. We were told no in most cases and given hacky workarounds in others that were prone to break with each incremental update to Xcode. Most of these breaks were found out because we picked up the latest beta build and attempted to run our automation tools only to find that something subtle had changed in our workaround or the basis of the workaround is no longer there.

Sure, SPM is “open source”, however, it’s unrealistic for people/teams to maintain forks of the tools to support features they need. Maybe the changes get accepted into a pull request, maybe not.

Honestly, I’m a bit shocked that the separation of building and executing tests is such a controversial thing. There should be three high-level tasks:

1. build
2. run-tests
3. build + run-tests

What I’m saying is that by not supporting #2, you’re creating an adoption blocker for any team that cannot or does not want to build every time it runs their tests. There are many reasons for this. I’ve already given several, however, another reason is instrumentation for things like code-coverage or product code markers, often happen as an intermediate step outside of the normal build operation. This is especially true in the cases of bug investigation. Saying that I cannot trigger the running of these tests without a build step baffling to me.

If I may make an extended comment that may clear up some disconnects that are troubling other reviewers, that I have been grappling with very recently. I am one of the more aggressive voices in favor of configurability on swift-build-dev–I actually maintain a small fork that has various configurability hacks so the project is usable for me–and I empathize with what David is saying, and what the other folks are saying about "mixed bags" and "first drafts" when a proposal does not entirely fit into my use case.

One thing I have finally learned about SwiftPM proposals is they are subtly different from Language proposals.

Language proposals operate on a language that already exists, is already implemented, and is basically in a useful state, for various values of "basically". Language proposals make small tweaks for small usability wins, and necessarily operate at a minute level of detail with all the problems ironed out to avoid conflicting with current code. Language proposals tend to be largely adversarial: we have a problem, and there is a solution space; many will enter, few will win. C-style for loops should either be in or out; there will only ever be one proposal on that topic; everybody has had plenty of time to think about the full implications of C-style for loops three years into the language. If you think there is some deficiency in SE-0007 your only recourse is to raise objections on this list; the suggestion that you should instead write a different proposal to keep them sounds very much like a brush-off.

SwiftPM, on the other hand, is basically *unusable* (again for various values of "basically"), and there seems to be the prevailing presumption that before anything is done we must have a proposal. As a consequence, there is no testing support in SwiftPM right now at all, and there is the general sense that before any testing can be added, some proposal must be accepted by this list. I think this presumption may be so obvious to some people that nobody bothers to actually say it explicitly, whereas it was not at all obvious to me. This result and the various follow-on consequences of it have only occurred to me recently.

It may be illustrative to think of how Swift might have evolved were it using the swift-evolution process from the beginning. In that situation there would have first been a proposal to *introduce* C-style for loops (because obviously we need something) and then there would have been a proposal for *for-in loops* and finally there would be another proposal to *repeal the C-style for*. We would have in fact had 3 proposals on the "for loop topic" (if not more, because I am probably forgetting key developments) and so a proposal for a very greenfield project is hardly a Final Design for All Time and more a permission to start solving the problem at all.

When that first C-style for-loop proposal was presented somebody very well could have said "I think we should have for-in loops", but it would not have been a brush-off to suggest that should be its own proposal that is separately discussed and implemented *in addition to* the C-style proposal, nor would it have been, in 2013, an objection to C-style-for. It is easy for us in 2016 to smugly suppose we would have always had one for-loop model, but we had 2 models for years, and as the spirited discussion on SE-0007 has shown, it is not obvious to everyone that they should be combined, and the key arguments involve the details of optimizers and so on which are understood only recently.

Now, with the disclaimer that this is my intuition of the situation, rather than any written policy–and things like this should really be written–the present situation is like introducing the C-style for loops, because we need something. I think the standard of evidence for this proposal (and in fact all SwiftPM proposals, and in fact all greenfield project proposals) should be much different than other proposals reviewed on this list. The question we should ask ourselves is: "Is this proposal basically useful? Should someone start implementing it tomorrow?" and not "Is this proposal exhaustively thorough?" We are in month 1 of a project here, things are not going to be thorough. If we are obviously going to box ourselves into a corner, that is one thing, and that class of objection is very important.

But SwiftPM is in a "bad" state (read: it's unimplemented, I don't suggest it's poorly-implemented). You cannot run tests, you cannot link with libdispatch or Foundation, you cannot build a dynamic library, you cannot call an external build system, you cannot cross-compile, you cannot get an optimized build, you cannot define a DEBUG macro... *we have a lot of work to do*. And as I understand it, most, if not all, of these problems are going to come across your desk as a proposal review.

Things like running the tests separately from builds are absolutely important. Things like encouraging a diversity in the type of tests are absolutely important. But I think it is more important to understand the semantics of this review: it is not, I believe, an absolute statement that we have considered and *rejected* all the things *not* included in the proposal. It is, I believe, a statement that we have *permission* to implement all the things *in* the proposal. So the absence of various things should not be worrying, but should instead be redirected into producing proposals that extend this one to support more usecases. Particularly if these things are critical for motivated users (and trust me, I have testing problems not handled here that are critical for me), it seems like writing follow-on proposals is the productive path to accomplishing them, at least as far as I have understood the process so far.

To some extent these extensions are already happening, for example, I have a proposal still in swift-build-dev <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151228/000125.html&gt; that describes in great detail the protocol given merely one sentence in the present proposal, which may be a first step toward diversity in test types. I think somebody should draft a proposal for tests-without-builds. But "we need more features than this" should be an argument for more proposals, not less. Either that or we need to revisit the apparent presumption in the SwiftPM project about proposals-first, because we are never going to get anywhere by waiting for a Final Specification for Tests Forever before implementing any testing at all. And I think there should be the feeling that we may even reverse things written here if they turn out to be wrong with more experience, like we did recently with C-style-for.

I really think this whole idea about proposals-first and evidentiary standards of review for greenfield projects needs to be written up and documented somewhere official, because I was quite confused about how SwiftPM is governed until recently (and maybe I'm still confused! Who knows!) and people looked at me funny when I raised objections very much like David's on other topics in SwiftPM. And those kinds of objections and an adversarial presumption are not unreasonable for people more familiar with the Language proposals as a background. The lack of clarity about the semantics of a review creates misunderstandings that negatively impact both the people writing the proposals and the people trying in good faith to raise objections to them. We can fix that.

What is your evaluation of the proposal?

+1

I desperately need test support, I am currently doing something insane to be able to test my codebases on Linux. The insane thing I am doing needs to die in a fire two weeks ago.

For that matter the insane thing that Foundation and SwiftPM themselves do to run their tests needs to die in a fire. I think we are not quite there yet with Foundation, but this proposal gets us there (I think?) on SwiftPM.

I agree generally with Paul Cantrell's assessment that the actual details are a mixed bag. Specifically:

Paul makes a convincing argument that --without-tests is the right default, and it agrees with my own intuition
I wonder if JUnit XML is really the format we want to encourage (as opposed to something JSON-based, etc.) I certainly don't object to it as an option, and the current language is certainly vague enough to be agreeable to me, but I would want to discourage it as a default format. We should have a native format in something easier to parse.
That build configuration for testing is lacking (e.g. -D TESTING or similar)
That dependencies for test modules are presently unspecified
That we need a test-without-build setting for David's case
That we need some overrideable behavior for "swift build -t" other than "run all test modules"
It probably goes without saying, but we should standardize on a particular return code for "tests failed" (as opposed to combining with the code for build failure or some other kind of error).

To be clear, I don't think any of these are important enough to derail this proposal, but they are things I think should be taken up at some point.

I do think there are a lot of strong points about the proposal:

That there is "one weird trick" (swift build -t) that can run tests for any Swift package, and ideally (e.g. through further extensions) we will extend this to support custom testing frameworks and so on so that there is no longer any reason for an ad-hoc way to run the tests
That tests are a first-class citizen and available anywhere that builds are available. One of the early questions everyone has when they dip their toe into a new project are "how does it build" and "how do I run the test suite" and I really think greasing the wheels here is important to the overall future of the ecosystem. Currently many of the github.com/apple <http://github.com/apple&gt; projects do something "special snowflake" here, and this creates friction. We have an opportunity to avoid this in the future.
I think there is an opportunity here to encourage "bug-reports-as-test-cases", which are the best kind of bug report. Often when I submit bug reports I have a little sample project or something but it is not necessarily obvious how to turn that bug report into a test which would simplify upstream reproducing it and so on. Some standardization here really has the opportunity to make it easy for me to submit bug reports as tests, which simplifies upstream's life considerably.

I think these strong points are a lot more important than some of the details in the first list that may be absent or wrong, and for those reasons I think we just need to do it and see how it goes. The only thing that even makes me hesitate is the --without-tests default, but while I believe in that hill I don't want test support to die on it.

  * Is the problem being addressed significant enough to warrant a change to Swift?

Yes, tests are critical.

  * Does this proposal fit well with the feel and direction of Swift?

Basically. I think we have a lot more work to do for use cases like David's (and also mine... and also others...) but we desperately need some basic way to run "some tests somewhere" and "swift build -t" is a fine way.

Organizing the tests into a "Tests" directory is logical for the common case and extends a current SPMism of using directory layout to infer intent. For legacy interop with Xcode projects there may be other cases we need to take up separately.

  * If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I design and implement tools that wrap "xcodebuild test". This proposal is light on a lot of details I care about, but the fact that there should be some way to build and run tests is not especially controversial.

I also work with Python3's unittest module quite often. I think we can do a lot better than Python, but again this proposal is a first step that doesn't really touch on any issues I care about there.

  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've followed this proposal from its first draft. I've also written a follow-on proposal currently collecting feedback in swift-build-dev.

···

On Jan 7, 2016, at 1:18 PM, David Owens II via swift-evolution <swift-evolution@swift.org> wrote:

One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?

  - Rick

No, it doesn’t. While on the Office for Mac team, we repeatedly asked for many of the items in Xcode and xcodebuild to be made less coupled with each of the steps in the process. We were told no in most cases and given hacky workarounds in others that were prone to break with each incremental update to Xcode. Most of these breaks were found out because we picked up the latest beta build and attempted to run our automation tools only to find that something subtle had changed in our workaround or the basis of the workaround is no longer there.

Sure, SPM is “open source”, however, it’s unrealistic for people/teams to maintain forks of the tools to support features they need. Maybe the changes get accepted into a pull request, maybe not.

Honestly, I’m a bit shocked that the separation of building and executing tests is such a controversial thing. There should be three high-level tasks:

1. build
2. run-tests
3. build + run-tests

What I’m saying is that by not supporting #2, you’re creating an adoption blocker for any team that cannot or does not want to build every time it runs their tests. There are many reasons for this. I’ve already given several, however, another reason is instrumentation for things like code-coverage or product code markers, often happen as an intermediate step outside of the normal build operation. This is especially true in the cases of bug investigation. Saying that I cannot trigger the running of these tests without a build step baffling to me.

-David

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I agree with pretty much all you’ve said. The caveat is have is this: don’t couple the the atomic steps together when incrementally building it up. When the steps aren’t coupled, it’s significantly easier to iteratively make changes to each of the components. It’s also much easier to review and develop additions because you’ve reduced the surface area for the integration points so there is less to consider, and consequently, it’s easier to validate breaking changes.

For example, before any meaningful work can be done to support different types of test runners, the work is going to need to done to break the build & run tests coupling. In this way, I think the proposal does a disservice because it’s not laying a good foundation to build on, it’s laying a foundation that we already know needs to be broken up.

There’s a difference between “get something done quick” and “get the most minimal viable product” (MVP) out the door. My objection to the proposal is that I don’t see it setting up the MVP.

-David

···

On Jan 8, 2016, at 6:08 AM, Drew Crawford <drew@sealedabstract.com> wrote:

<all the good stuff>

The caveat is have is this: don’t couple the the atomic steps together

when incrementally building it up. When the steps aren’t coupled, it’s
significantly easier to iteratively make changes to each of the components.
It’s also much easier to review and develop additions because you’ve
reduced the surface area for the integration points so there is less to
consider, and consequently, it’s easier to validate breaking changes.

For example, before any meaningful work can be done to support different

types of test runners, the work is going to need to done to break the build
& run tests coupling. In this way, I think the proposal does a disservice
because it’s not laying a good foundation to build on, it’s laying a
foundation that we already know needs to be broken up.

There’s a difference between “get something done quick” and “get the most

minimal viable product” (MVP) out the door. My objection to the proposal is
that I don’t see it setting up the MVP.

Can't agree with this more. The point of integration between all these
tools (and atomic steps) should be as minimal as possible, and so far the
proposals that have come through regarding these components of the building
and testing project have either felt very heavy-weight or very coupled (for
example, to XCTest or similar). As an example: A simple function that can
throw an error (and *maybe* returns a simple status of pass/fail?) is
probably enough to facilitate the vast majority of testing frameworks, and
it *doesn't* come with lots of baggage of protocols and implementation
details that we can't assume will be required or even useful for said
framework. We should consider what we can leave out instead.

Swift facilitates all sorts of practical wins in code to promote loose
coupling in an extremely lightweight way, without having to resort to the
relatively heavy OO patterns we see in Java (and, at times, in C#) -- many
of which largely exist because the languages didn't have support for
closures until recently. As someone who has authored a third-party testing
framework (and made it work with the XCode-provided XCTest API, which was a
tremendous headache) I would much prefer to see small, atomic, decoupled
integration points. The way to do that is omitting details and working at a
nice, high level of abstraction.

So thank you, David, for mentioning this, and I'm 100% in agreement with
you.

My biggest concern with this proposal, as it stands, is that it isn't
considering what I think are some very important details that will shape
the way the build-tool works longer-term. If there's a generic testing
protocol provided by the build tool, how do I import my testing framework
within Package.swift *before* Package.swift has been able to download all
my dependencies?

Is there a need for a multi-step, multi-file process here?

I personally would much rather see us focus our discussion on those
questions to create a longer-term proposal than continue to debate a
proposal that itself admits that it amounts to a temporary solution. Maybe
that's worth opening another topic on swift-build-dev for, but for now, I'm
made really uncomfortable by the uncertainty left in this proposal as
someone who wants to use and/or create third-party testing tools -- this
proposal feels very one-size-fits-all, and that really concerns me.

···

On Fri, Jan 8, 2016 at 11:13 AM, David Owens II via swift-build-dev < swift-build-dev@swift.org> wrote:

> On Jan 8, 2016, at 6:08 AM, Drew Crawford <drew@sealedabstract.com> > wrote:
>
> <all the good stuff>

I agree with pretty much all you’ve said. The caveat is have is this:
don’t couple the the atomic steps together when incrementally building it
up. When the steps aren’t coupled, it’s significantly easier to iteratively
make changes to each of the components. It’s also much easier to review and
develop additions because you’ve reduced the surface area for the
integration points so there is less to consider, and consequently, it’s
easier to validate breaking changes.

For example, before any meaningful work can be done to support different
types of test runners, the work is going to need to done to break the build
& run tests coupling. In this way, I think the proposal does a disservice
because it’s not laying a good foundation to build on, it’s laying a
foundation that we already know needs to be broken up.

There’s a difference between “get something done quick” and “get the most
minimal viable product” (MVP) out the door. My objection to the proposal is
that I don’t see it setting up the MVP.

-David

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

A simple function that can throw an error (and *maybe* returns a simple status of pass/fail?) is probably enough to facilitate the vast majority of testing frameworks, and it *doesn't* come with lots of baggage of protocols and implementation details that we can't assume will be required or even useful for said framework. We should consider what we can leave out instead.

This is not the case, for reasons you can find in the third-party test frameworks proposal; but shortly: it is not innovation to have a protocol which merely allows 10 competing implementations of an XCTest-like framework.

Testing is not a one-size-fits-all concept. The role of SwiftPM is IMO:

To have a simple way to run XCTest suites, which is the way most people (including SwiftPM & Foundation & many many codebases) write their tests now, and
To push innovation into third-party test frameworks that are free to explore the problem space in a more unrestricted way that is not bound by the constraints of XCTest, source compatibility, or the swift-evolution proposal process

This proposal is mostly targeted at the first situation. I do believe it is the "minimum viable proposal" to address that first situation. Perhaps the assumption that we are studying (only) the first situation needs to be more clearly stated.

XCTest compatibility is a feature many people require. I understand folks who find it heavyweight for new code. But I think the best way forward is to kick the design problem for other models out to third parties, because this mailing list will not scale to designing a novel testing framework one API at a time. Let's let the ecosystem go to work on this problem.

We need a proposal to do that, and it is currently being drafted on swift-build-dev. Please help.

than continue to debate a proposal that itself admits that it amounts to a temporary solution

I don't think this is the case at all; the proposal admits to being one solution among many, and one that will need further extensions. But people have XCTest test suites, they want to test them, we should support it. There's nothing temporary about that.

···

On Jan 8, 2016, at 12:51 PM, Brian Pratt <brian@pratt.io> wrote:

On Jan 8, 2016, at 11:13 AM, David Owens II <david@owensd.io> wrote:

For example, before any meaningful work can be done to support different types of test runners, the work is going to need to done to break the build & run tests coupling. In this way, I think the proposal does a disservice because it’s not laying a good foundation to build on, it’s laying a foundation that we already know needs to be broken up.

I don't believe this is the case either. Under this proposal, building and running is coupled at the CLI, but introducing new CLI arguments is not any great hardship.

It may help to clearly identify the coupling you mean, and how this proposal very specifically establishes it. Everybody agrees that "coupling is bad, kids" but not everyone understands how this proposal brings it into being.

I wonder if your objection is simply that the XCTest proposal hit review first, rather than the third-party frameworks proposal. That is only because the XCTest folks are simply better motivated to draft it: there is tons of existing code that can benefit, the test framework actually exists, etc. The third-party frameworks are not as well-motivated.

I don’t care that the default and only supported test runner is XCTest for the moment. I can simply write a little wrapper and shell out to some other test runner. I can even get through the fact that tests will be built everytime (they won’t though, because people will simply comment them out in the Package.swift file until they are ready to use them).

What I care about is the coupling of the test phase with the build phase. I cannot work around that.

When I run “swift build -t” or “swift tests” or whatever the CLI will be, I need to be able to do that without re-triggering builds. A no-op build is not sufficient. If you don’t agree with, then I really have no other examples to provide that can convince anyone of that fact.

At some later date, maybe we’ll get “swift build --run-tests” or “swift tests --no-build” that simply runs the tests without attempting to do a build. Ok, maybe we will. If it’s a week later, ok. If it’s a month later, ok. If it’s going to be a year away, that becomes a real issue. However, there is no compelling reason to couple this from the start, especially if there is no architectural considerations to worry about.

-David

···

On Jan 8, 2016, at 5:54 PM, Drew Crawford <drew@sealedabstract.com> wrote:

I don't believe this is the case either. Under this proposal, building and running is coupled at the CLI, but introducing new CLI arguments is not any great hardship.

It may help to clearly identify the coupling you mean, and how this proposal very specifically establishes it. Everybody agrees that "coupling is bad, kids" but not everyone understands how this proposal brings it into being.

I wonder if your objection is simply that the XCTest proposal hit review first, rather than the third-party frameworks proposal. That is only because the XCTest folks are simply better motivated to draft it: there is tons of existing code that can benefit, the test framework actually exists, etc. The third-party frameworks are not as well-motivated.

This is not the case, for reasons you can find in the third-party test

frameworks proposal; but shortly: it is not innovation to have a protocol
which merely allows 10 competing implementations of an XCTest-like
framework.

I don't really think innovation is something worth sacrificing *absolute*
flexibility for. Not in a tool that's primarily meant to enable developers
to distribute and consume shared libraries of code. There are going to be
many ways that developers prefer to write tests and something as generic as
a package manager should, in my mind, allow for all of that. While I again
agree that XCTest is the most sensible default, I think there's value in
expressing the idea (one not called out by either this proposal or the one
for third-party testing frameworks) that the build tool should not have
concrete dependencies on XCTest but rather XCTest should integrate as one
of many potential generic implementations usable with the build tool in the
most minimal interface possible.

That is to say, the build tool shouldn't take an opinion at all how tests
are run, just about the general result. Enforcing any kind of uniformity is
counter-intuitive as people may end up with fragmentation in their build
tools themselves rather than just in their testing frameworks. There are
already more than a handful of Swift testing frameworks that aren't
XCTest-like, even some that are more Quickcheck-inspired, and these
prescriptive testing approaches within an opinionated build tool hinder the
community's ability to innovate on their own. The language and its tools
should give users flexibility to allow for innovation, not try to prescribe
anything about how testing should be done in order to enforce uniformity.
That's a loss.

To have a simple way to run XCTest suites, which is the way most people

(including SwiftPM & Foundation & many many codebases) write their tests
now, and

To push innovation into third-party test frameworks that are free to

explore the problem space in a more unrestricted way that is not bound by
the constraints of XCTest, source compatibility, or the swift-evolution
proposal process.

Right, so, we agree here -- just wanted to make sure I register a concern
that rushing to implement XCTest integration with the build tool feels like
a mistake, especially before the community can meaningfully contribute to
the relatively new third-party frameworks proposal as it stands. Making the
build tool XCTest dependent is probably more difficult than making it
generic, and it seems like we'd backtrack quickly.

Why not just let me tell my Package object that I want a specific
executable target for testing, one that swift-build will build and then run
linking whatever packages I specify from `testDependencies`. Boom, XCTest
works, and so do many other frameworks, and all authors have to do is give
the proper exit status. This mirrors the UNIX philosophy of small,
composable components and I think that's a good approach to take with Swift
and its generic tooling. Further, this makes SwiftPM work for a lot of
generic cases until the third-party frameworks proposal is properly ironed
out. That protocol may be the ultimate long-term win, but I see no reason
to make a short-term sacrifice (XCTest-specific integration) in order to
facilitate it when something generic and simple can achieve basically the
exact same result from a user experience perspective?

Again, it's good for the language to be prescriptive, but it's better to be
flexible and let the community evolve their individual tooling as they wish.

It may help to clearly identify the coupling you mean, and how this

proposal very specifically establishes it. Everybody agrees that "coupling
is bad, kids" but not everyone understands how this proposal brings it into
being.

You're absolutely right -- the nature of the complaints has been very
high-level at this point, so, details: I think, in particular, the
prescription of testing output creates an unnecessarily large overlap
between potential (and existing) third-party frameworks that should be
avoided. It means that SwiftPM would have to implement test reporters that
might make certain assumptions about how the tests are run (synchronously?)
and what output is deemed important. It might make assumptions about how
tests fail and how they report those failures to the user. And, having used
XCTest as distributed by XCode, it would be frustrating to have very little
control over your test output because you have to maintain compliance with
a standard that most certainly -is- bound by the constraints of the
swift-evolution process. If the stated goal is to iterate, it makes sense
to keep the components flexible and light-weight so that iteration can
happen more quickly and outside of the bounds of language review / release
cycle.

I understand that supporting XCTest is the most pragmatic move, short-term,
but I also think the existing proposals convey an overly specific approach
to testing that doesn't feel like it belongs in an otherwise generic build
/ packaging tool.

···

On Fri, Jan 8, 2016 at 7:54 PM, Drew Crawford <drew@sealedabstract.com> wrote:

On Jan 8, 2016, at 12:51 PM, Brian Pratt <brian@pratt.io> wrote:

A simple function that can throw an error (and *maybe* returns a simple
status of pass/fail?) is probably enough to facilitate the vast majority of
testing frameworks, and it *doesn't* come with lots of baggage of protocols
and implementation details that we can't assume will be required or even
useful for said framework. We should consider what we can leave out instead.

This is not the case, for reasons you can find in the third-party test
frameworks proposal; but shortly: it is not innovation to have a protocol
which merely allows 10 competing implementations of an XCTest-like
framework.

Testing is not a one-size-fits-all concept. The role of SwiftPM is IMO:

   - To have a simple way to run XCTest suites, which is the way most
   people (including SwiftPM & Foundation & many many codebases) write their
   tests now, and
   - To push innovation into third-party test frameworks that are free to
   explore the problem space in a more unrestricted way that is not bound by
   the constraints of XCTest, source compatibility, or the swift-evolution
   proposal process

*This proposal is mostly targeted at the first situation.* I do believe
it is the "minimum viable proposal" to address that first situation.
Perhaps the assumption that we are studying (only) the first situation
needs to be more clearly stated.

XCTest compatibility is a feature many people require. I understand folks
who find it heavyweight for new code. But I think the best way forward is
to kick the design problem for other models out to third parties, because
this mailing list will not scale to designing a novel testing framework one
API at a time. Let's let the ecosystem go to work on this problem.

We need a proposal to do that, and it is currently being drafted on
swift-build-dev. Please help.

than continue to debate a proposal that itself admits that it amounts to a
temporary solution

I don't think this is the case at all; the proposal admits to being one
solution among many, and one that will need further extensions. But people
have XCTest test suites, they want to test them, we should support it.
There's nothing temporary about that.

On Jan 8, 2016, at 11:13 AM, David Owens II <david@owensd.io> wrote:

For example, before any meaningful work can be done to support different
types of test runners, the work is going to need to done to break the build
& run tests coupling. In this way, I think the proposal does a disservice
because it’s not laying a good foundation to build on, it’s laying a
foundation that we already know needs to be broken up.

I don't believe this is the case either. Under this proposal, building
and running is coupled at the CLI, but introducing new CLI arguments is not
any great hardship.

It may help to clearly identify the coupling you mean, and how this
proposal very specifically establishes it. Everybody agrees that "coupling
is bad, kids" but not everyone understands how this proposal brings it into
being.

I wonder if your objection is simply that the XCTest proposal hit review
first, rather than the third-party frameworks proposal. That is only
because the XCTest folks are simply better motivated to draft it: there is
tons of existing code that can benefit, the test framework actually exists,
etc. The third-party frameworks are not as well-motivated.

At some later date, maybe we’ll get “swift build --run-tests” or “swift tests --no-build” that simply runs the tests without attempting to do a build. Ok, maybe we will. If it’s a week later, ok. If it’s a month later, ok. If it’s going to be a year away, that becomes a real issue. However, there is no compelling reason to couple this from the start, especially if there is no architectural considerations to worry about.

I think the common case (perhaps it is uncommon for you) is a test-driven implementation loop, where somebody has a failing test, they are working in a text editor or IDE, and they want to know if it the thing they changed caused the tests to pass. This is my workflow when I PR something to Foundation or SwiftPM, and my workflow on other projects.

Certainly that could be accomplished via chaining commands instead (e.g. "swift build && swift test") which better supports your workflow although that is slightly more awkward for the common situation.

But in any case this whole argument feels to me like a bikeshed–we have infinite freedom to support any number of CLI commands, with builds & tests together, separated, in a box, with a fox. This one was elected to support a common workflow, not to the exclusion of all the other things we could do additionally.

But there is language being used on this point (from several people) like:

there is no compelling reason to couple this from the start

My biggest concern with this proposal, as it stands, is that it isn't considering what I think are some very important details that will shape the way the build-tool works longer-term.

For example, before any meaningful work can be done to support different types of test runners, the work is going to need to done to break the build & run tests coupling. In this way, I think the proposal does a disservice because it’s not laying a good foundation to build on, it’s laying a foundation that we already know needs to be broken up.

I fail to understand how the election of one (first) command line argument has any of these follow-on consequences at all. We are discussing a CLI frontend, not an architecture diagram.

If someone believes we are in danger of laying a bad foundation then that is an important objection, but to understand it we need an argument of the form

1. "swift build -t"
2. ???
3. Therefore, [specific irreversible harm]

Until that is laid out in a way I can follow, I can't make sense of the objection.

···

On Jan 8, 2016, at 9:20 PM, David Owens II <david@owensd.io> wrote:

At some later date, maybe we’ll get “swift build --run-tests” or “swift tests --no-build” that simply runs the tests without attempting to do a build. Ok, maybe we will. If it’s a week later, ok. If it’s a month later, ok. If it’s going to be a year away, that becomes a real issue. However, there is no compelling reason to couple this from the start, especially if there is no architectural considerations to worry about.

I think the common case (perhaps it is uncommon for you) is a test-driven implementation loop, where somebody has a failing test, they are working in a text editor or IDE, and they want to know if it the thing they changed caused the tests to pass. This is my workflow when I PR something to Foundation or SwiftPM, and my workflow on other projects.

I don’t run all of my tests when I do this. I re-run the subset that actually apply to what I’m working on. Then before performing a check-in, I re-run all of the tests.

Certainly that could be accomplished via chaining commands instead (e.g. "swift build && swift test") which better supports your workflow although that is slightly more awkward for the common situation.

This is only the “common situation” if what I was talking about in earlier posts is true: the testing that is being support here is primary for unit tests only or integration tests that can run as quickly as unit tests. But yes, I’m saying that the proposal should enable that exact scenario as the building block.

But in any case this whole argument feels to me like a bikeshed–we have infinite freedom to support any number of CLI commands, with builds & tests together, separated, in a box, with a fox. This one was elected to support a common workflow, not to the exclusion of all the other things we could do additionally.

Again, if it so easy, then why not simply come out of the gate with “build”, “test”, and “run-tests”. Surely we’ve already discussed this with more time then it would be to implement if SPM’s architecture is actually going to allow this and that shortcuts are not taken to get the “test” portion working more quickly. Call me jaded by experience with other tools, but whenever I’ve seen shortcuts like these taken, it’s always more work than a simple add of a new command to break what is being treated by the current implementation as an atomic command into two.

But there is language being used on this point (from several people) like:

there is no compelling reason to couple this from the start

My biggest concern with this proposal, as it stands, is that it isn't considering what I think are some very important details that will shape the way the build-tool works longer-term.

For example, before any meaningful work can be done to support different types of test runners, the work is going to need to done to break the build & run tests coupling. In this way, I think the proposal does a disservice because it’s not laying a good foundation to build on, it’s laying a foundation that we already know needs to be broken up.

I fail to understand how the election of one (first) command line argument has any of these follow-on consequences at all. We are discussing a CLI frontend, not an architecture diagram.

If someone believes we are in danger of laying a bad foundation then that is an important objection, but to understand it we need an argument of the form

1. "swift build -t"
2. ???
3. Therefore, [specific irreversible harm]

Until that is laid out in a way I can follow, I can't make sense of the objection.

I don’t know how else to explain my objection; I’ve done so in at least three different ways, including examples of how a similar design affects my previous team with still unsupported behavior in other Apple tools that have the same design as being illustrated here (Rick, I’m reaching out to my old counterparts to see if they have any radars they want followed up with). It’s not irreversible harm, it’s starting the process with unnecessary technical debt.

Fortunately, this time around we can at least fork the tools to get the support we need, if it doesn’t come (and potentially send the fix back or submit proposals), instead of do what Facebook had to do with xctool.

-David

···

On Jan 8, 2016, at 8:28 PM, Drew Crawford <drew@sealedabstract.com> wrote:

On Jan 8, 2016, at 9:20 PM, David Owens II <david@owensd.io <mailto:david@owensd.io>> wrote:

I don't really think innovation is something worth sacrificing *absolute* flexibility for. Not in a tool that's primarily meant to enable developers to distribute and consume shared libraries of code. There are going to be many ways that developers prefer to write tests and something as generic as a package manager should, in my mind, allow for all of that. While I again agree that XCTest is the most sensible default, I think there's value in expressing the idea (one not called out by either this proposal or the one for third-party testing frameworks) that the build tool should not have concrete dependencies on XCTest but rather XCTest should integrate as one of many potential generic implementations usable with the build tool in the most minimal interface possible.

That is to say, the build tool shouldn't take an opinion at all how tests are run, just about the general result. Enforcing any kind of uniformity is counter-intuitive as people may end up with fragmentation in their build tools themselves rather than just in their testing frameworks. There are already more than a handful of Swift testing frameworks that aren't XCTest-like, even some that are more Quickcheck-inspired, and these prescriptive testing approaches within an opinionated build tool hinder the community's ability to innovate on their own. The language and its tools should give users flexibility to allow for innovation, not try to prescribe anything about how testing should be done in order to enforce uniformity. That's a loss.

This is a nice philosophy, but what does any of it mean?

We already have "*absolute* flexibility"–SwiftPM and Foundation tests are simply an ad-hoc script. The script can literally do anything, and report its outputs any way. It can use any testing framework. It can live anywhere in your repository and be named anything, and can require any number of system-installed dependencies (or not), can be written in any language that you may or may not have installed (Python seems popular, some debate if 2 or 3 <Sign in to GitHub · GitHub), it can take any number of arguments, it can build and test together or separately, it can play music in your iTunes library. A proposal that provided "absolute" flexibility would simply be:

# Motivation

We need absolute flexibility in tests.

# Proposed solution

Anyone can write tests however the fuck they want.

Any longer proposal is *necessarily* going to define *some* limits to use of any proposed testing feature. The question is whether some particular limit is sense or nonsense, not whether there should be anything standardized at all. And we need to have a specific conversation around the exact technical details of the interface.

That conversation is currently ongoing on swift-build-dev. If you think a UNIX process model (which does limit flexibility!) is superior to a protocol-based approach, I would love to read an argument for that on the thread. It has some drawbacks we have been talking about (such as no common reporting format for CI / GUI / Xcode to consume) but maybe you have ideas for how we can solve that problem within a UNIX process model interface.

In any case, SE-0019's treatment of the interface is so incidental that I do not see why the interface question (which is in my mind still very debatable at this moment) would be a defining issue in this review.

···

On Jan 8, 2016, at 9:17 PM, Brian Pratt <brian@pratt.io> wrote:

It's not the approach (I'll save my commentary on that for the right
thread) it's the fact that I think the build tool shouldn't be concerned
with things like test output and GUI reporting. That feels like a behavior
that belongs in the testing tools themselves, doesn't it?

···

On Friday, January 8, 2016, Drew Crawford <drew@sealedabstract.com> wrote:

On Jan 8, 2016, at 9:17 PM, Brian Pratt <brian@pratt.io > <javascript:_e(%7B%7D,'cvml','brian@pratt.io');>> wrote:

I don't really think innovation is something worth sacrificing *absolute*
flexibility for. Not in a tool that's primarily meant to enable developers
to distribute and consume shared libraries of code. There are going to be
many ways that developers prefer to write tests and something as generic as
a package manager should, in my mind, allow for all of that. While I again
agree that XCTest is the most sensible default, I think there's value in
expressing the idea (one not called out by either this proposal or the one
for third-party testing frameworks) that the build tool should not have
concrete dependencies on XCTest but rather XCTest should integrate as one
of many potential generic implementations usable with the build tool in the
most minimal interface possible.

That is to say, the build tool shouldn't take an opinion at all how tests
are run, just about the general result. Enforcing any kind of uniformity is
counter-intuitive as people may end up with fragmentation in their build
tools themselves rather than just in their testing frameworks. There are
already more than a handful of Swift testing frameworks that aren't
XCTest-like, even some that are more Quickcheck-inspired, and these
prescriptive testing approaches within an opinionated build tool hinder the
community's ability to innovate on their own. The language and its tools
should give users flexibility to allow for innovation, not try to prescribe
anything about how testing should be done in order to enforce uniformity.
That's a loss.

This is a nice philosophy, but what does any of it mean?

We already have "*absolute* flexibility"–SwiftPM and Foundation tests are
simply an ad-hoc script. The script can literally do anything, and report
its outputs any way. It can use any testing framework. It can live
anywhere in your repository and be named anything, and can require any
number of system-installed dependencies (or not), can be written in any
language that you may or may not have installed (Python seems popular, some
debate if 2 or 3 <Sign in to GitHub · GitHub),
it can take any number of arguments, it can build and test together or
separately, it can play music in your iTunes library. A proposal that
provided "absolute" flexibility would simply be:

# Motivation

We need absolute flexibility in tests.

# Proposed solution

Anyone can write tests however the fuck they want.

Any longer proposal is *necessarily* going to define *some* limits to use
of any proposed testing feature. The question is whether some particular
limit is sense or nonsense, not whether there should be anything
standardized at all. And we need to have a specific conversation around
the *exact* technical details of the interface.

That conversation is currently ongoing on swift-build-dev. If you think a
UNIX process model (which does limit flexibility!) is superior to a
protocol-based approach, I would love to read an argument for that on the
thread. It has some drawbacks we have been talking about (such as no
common reporting format for CI / GUI / Xcode to consume) but maybe you have
ideas for how we can solve that problem within a UNIX process model
interface.

In any case, SE-0019's treatment of the interface is so incidental that I
do not see why the interface question (which is in my mind still very
debatable at this moment) would be a defining issue in this review.

You know... you have a really good point there. I never even stopped to think about that assumption.

I think clearly swift-multitool should frontend, but is there a reason we don't have an independent testing tool from the package manager?

That very much does seem like the kind of architectural thing we should iron out before a proposal is accepted. Maybe I missed an obvious rationale in the earlier discussion.

···

On Jan 9, 2016, at 12:01 AM, Brian Pratt <brian@pratt.io> wrote:

It's not the approach (I'll save my commentary on that for the right thread) it's the fact that I think the build tool shouldn't be concerned with things like test output and GUI reporting. That feels like a behavior that belongs in the testing tools themselves, doesn't it?

On Friday, January 8, 2016, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:

On Jan 8, 2016, at 9:17 PM, Brian Pratt <brian@pratt.io <javascript:_e(%7B%7D,'cvml','brian@pratt.io');>> wrote:

I don't really think innovation is something worth sacrificing *absolute* flexibility for. Not in a tool that's primarily meant to enable developers to distribute and consume shared libraries of code. There are going to be many ways that developers prefer to write tests and something as generic as a package manager should, in my mind, allow for all of that. While I again agree that XCTest is the most sensible default, I think there's value in expressing the idea (one not called out by either this proposal or the one for third-party testing frameworks) that the build tool should not have concrete dependencies on XCTest but rather XCTest should integrate as one of many potential generic implementations usable with the build tool in the most minimal interface possible.

That is to say, the build tool shouldn't take an opinion at all how tests are run, just about the general result. Enforcing any kind of uniformity is counter-intuitive as people may end up with fragmentation in their build tools themselves rather than just in their testing frameworks. There are already more than a handful of Swift testing frameworks that aren't XCTest-like, even some that are more Quickcheck-inspired, and these prescriptive testing approaches within an opinionated build tool hinder the community's ability to innovate on their own. The language and its tools should give users flexibility to allow for innovation, not try to prescribe anything about how testing should be done in order to enforce uniformity. That's a loss.

This is a nice philosophy, but what does any of it mean?

We already have "*absolute* flexibility"–SwiftPM and Foundation tests are simply an ad-hoc script. The script can literally do anything, and report its outputs any way. It can use any testing framework. It can live anywhere in your repository and be named anything, and can require any number of system-installed dependencies (or not), can be written in any language that you may or may not have installed (Python seems popular, some debate if 2 or 3 <Sign in to GitHub · GitHub), it can take any number of arguments, it can build and test together or separately, it can play music in your iTunes library. A proposal that provided "absolute" flexibility would simply be:

# Motivation

We need absolute flexibility in tests.

# Proposed solution

Anyone can write tests however the fuck they want.

Any longer proposal is *necessarily* going to define *some* limits to use of any proposed testing feature. The question is whether some particular limit is sense or nonsense, not whether there should be anything standardized at all. And we need to have a specific conversation around the exact technical details of the interface.

That conversation is currently ongoing on swift-build-dev. If you think a UNIX process model (which does limit flexibility!) is superior to a protocol-based approach, I would love to read an argument for that on the thread. It has some drawbacks we have been talking about (such as no common reporting format for CI / GUI / Xcode to consume) but maybe you have ideas for how we can solve that problem within a UNIX process model interface.

In any case, SE-0019's treatment of the interface is so incidental that I do not see why the interface question (which is in my mind still very debatable at this moment) would be a defining issue in this review.