adding automated-testing of uncompilable features in XCTest


(Benjamin Spratling) #1

Howdy,
  I’d like to see how much interest there is for adding these to the XCTest module. If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal.

There are several features of Swift which cannot be effectively automated-tested.
For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror. There are similar concerns for testability with internal, and public access levels. Tests can be written ensuring these access levels are >= some level, but not == or < some level. In other words the very usefulness of these features cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.

Examples:
Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings. Here are some examples of what I would like to write in my test code:

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.
    + Can be distributed independently from Swift sources, requiring 0 buy-in from Swift community
    + requires a single additional module for the test.
    - Depends on ever-changing binary interface.
    : Intractable, not DRY

2. Use existing sourcekitd.
    + harnesses changes in the compiler’s source code with SourceKit
    - cannot be run in a test suite without extensive work by user to configure bundles explicitly.
      Exceptionally poor user experience
    : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
    + harnesses changes in the compiler’s source code with SourceKit
    + no installation in user’s source code necessary
    : cannot be effectively run by SPM test infrastructure


(Derrick Ho) #2

It bugs me as well that we can not test private components using XCTest.
Objective-c never had this problem since privacy doesn't exist. I would
like to see a version of XCTest that allows us to test every component.

···

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution < swift-evolution@swift.org> wrote:

Howdy,
        I’d like to see how much interest there is for adding these to the
XCTest module. If I have missed some pro or con, or missed a technical
point, your feedback is welcome before I go to the lengths to draw up a
formal proposal.

There are several features of Swift which cannot be effectively
automated-tested.
For instance, when a type or one of its members is private or file
private, an outside test suite cannot access it, and the information that
the type is private is not included in a Mirror. There are similar
concerns for testability with internal, and public access levels. Tests
can be written ensuring these access levels are >= some level, but not ==
or < some level. In other words the very usefulness of these features
cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies
that a function cannot be called on a let struct, or that a property cannot
be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is
not broken, but do not need to be included in a release build, so including
them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the
.swiftmodule files, which are included in test builds, but sometimes
stripped for release.

Examples:
Since these features inherently have to do with testing features which
cannot be stated in compiled code, I recommend specifying names with
Strings. Here are some examples of what I would like to write in my test
code:

XCTAssertEqual(
Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel,
.private)

XCTAssertEqual(
Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable,
false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift
module, which can be included in test builds.
                + Can be distributed independently from Swift sources,
requiring 0 buy-in from Swift community
                + requires a single additional module for the test.
                - Depends on ever-changing binary interface.
                : Intractable, not DRY

2. Use existing sourcekitd.
                + harnesses changes in the compiler’s source code with
SourceKit
                - cannot be run in a test suite without extensive work by
user to configure bundles explicitly.
                        Exceptionally poor user experience
                : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
                + harnesses changes in the compiler’s source code with
SourceKit
                + no installation in user’s source code necessary
                : cannot be effectively run by SPM test infrastructure

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Benjamin Spratling) #3

-Ben

···

Sent from my iPhone.

On Dec 11, 2016, at 11:43 PM, Brian Gesiak <modocache@gmail.com> wrote:

Maybe your goal is to ensure that other programmers don't accidentally change the accessibility levels. Again, I think because they're straightforward, I don't think there's much danger there.

To me, tests for accessibility would be redundant. It would be like adding a test to verify that my struct 'Foo' has a member named 'bar'. If I ever decided to rename 'Foo.bar', I would have to update the test. But I would never "accidentally" rename 'Foo.bar' -- if I changed the name, it's because I modified the code for that purpose. A test adds overhead, but no benefit.

I feel the same about the 'mutating', 'weak', 'unowned', and 'final'.

It's definitely subjective, though. You might feel that the tests I describe above are valuable, and you're entitled to that opinion.

I have been in some situations recently in which encapsulation was untestable, and on a team with >10 developers and no automated test to catch that change, there were issues. In the end we shipped knowing there was a bug, because 15 weeks of work had been based on it, and it would have required a redesign to fix.

If I need a test that there is a property named 'bar', I can write a test which accesses the member, and even specify a required type and explicitly test whether the property does or does not conform to the protocol. Even if it's private, I can use a Mirror to extract its value.

My motivation is: I got tired of hearing "if you did it right, it would have worked", which I now view as an excuse for lack of good design (in most cases). For the past several months, I have been investigating design patterns which prevent developers from making common mistakes, and how to write tests to catch changes to those patterns. I can't even count the number of unfathomable bugs I've fixed merely by improving the design. Swift is notable in that its compiler can enforce many of these patterns, so I prefer it over Obj-C. Proper encapsulation is one such technique, and it turns out it's untestable.

Thanks for your suggestion about researching how other languages do this. I'll see what I can do, though the only experience I have in any of the ones you mentioned is pre-11 C++, so if anyone else knows how such checks are done, I'd greatly appreciate hearing from you.

-Ben


(Brian Gesiak) #4

Hi Benjamin,

I think your proposal is interesting, and you're certainly right: XCTest
has no way to test accessibility levels, or the other attributes you
mention. However, I wonder, what do you hope to achieve by adding an
automated test for accessibility levels?

Maybe your goal is to verify the accessibility level is what you think it
is. However, in my experience, accessibility levels are fairly
straightforward: if I specify a variable is 'public', it's public. If I
specify it to be 'fileprivate', it's private to the file. I could write an
automated test to verify the accessibility level, but the test would
provide me with very little value.

Maybe your goal is to ensure that other programmers don't accidentally
change the accessibility levels. Again, I think because they're
straightforward, I don't think there's much danger there.

To me, tests for accessibility would be redundant. It would be like adding
a test to verify that my struct 'Foo' has a member named 'bar'. If I ever
decided to rename 'Foo.bar', I would have to update the test. But I would
never "accidentally" rename 'Foo.bar' -- if I changed the name, it's
because I modified the code for that purpose. A test adds overhead, but no
benefit.

I feel the same about the 'mutating', 'weak', 'unowned', and 'final'.

It's definitely subjective, though. You might feel that the tests I
describe above are valuable, and you're entitled to that opinion.

If you'd still like to pursue this idea, I think your Swift Evolution
proposal would benefit from research on how other languages and testing
frameworks approach this problem. How do C++ programmers test whether an
attribute is private? How do Java developers test that a class is final?
What about Rust or Go?

- Brian Gesiak

···

On Mon, Dec 12, 2016 at 12:10 AM, Derrick Ho via swift-evolution < swift-evolution@swift.org> wrote:

It bugs me as well that we can not test private components using XCTest.
Objective-c never had this problem since privacy doesn't exist. I would
like to see a version of XCTest that allows us to test every component.

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution < > swift-evolution@swift.org> wrote:

Howdy,
        I’d like to see how much interest there is for adding these to
the XCTest module. If I have missed some pro or con, or missed a technical
point, your feedback is welcome before I go to the lengths to draw up a
formal proposal.

There are several features of Swift which cannot be effectively
automated-tested.
For instance, when a type or one of its members is private or file
private, an outside test suite cannot access it, and the information that
the type is private is not included in a Mirror. There are similar
concerns for testability with internal, and public access levels. Tests
can be written ensuring these access levels are >= some level, but not ==
or < some level. In other words the very usefulness of these features
cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which
verifies that a function cannot be called on a let struct, or that a
property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is
not broken, but do not need to be included in a release build, so including
them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the
.swiftmodule files, which are included in test builds, but sometimes
stripped for release.

Examples:
Since these features inherently have to do with testing features which
cannot be stated in compiled code, I recommend specifying names with
Strings. Here are some examples of what I would like to write in my test
code:

XCTAssertEqual( Module(named:”SingMusicLayout”
)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel,
.private)

XCTAssertEqual( Module(named:”SingMusicLayout”
)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(
for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift
module, which can be included in test builds.
                + Can be distributed independently from Swift sources,
requiring 0 buy-in from Swift community
                + requires a single additional module for the test.
                - Depends on ever-changing binary interface.
                : Intractable, not DRY

2. Use existing sourcekitd.
                + harnesses changes in the compiler’s source code with
SourceKit
                - cannot be run in a test suite without extensive work by
user to configure bundles explicitly.
                        Exceptionally poor user experience
                : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
                + harnesses changes in the compiler’s source code with
SourceKit
                + no installation in user’s source code necessary
                : cannot be effectively run by SPM test infrastructure

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rien) #5

-Ben

Sent from my iPhone.

Maybe your goal is to ensure that other programmers don't accidentally change the accessibility levels. Again, I think because they're straightforward, I don't think there's much danger there.

To me, tests for accessibility would be redundant. It would be like adding a test to verify that my struct 'Foo' has a member named 'bar'. If I ever decided to rename 'Foo.bar', I would have to update the test. But I would never "accidentally" rename 'Foo.bar' -- if I changed the name, it's because I modified the code for that purpose. A test adds overhead, but no benefit.

I feel the same about the 'mutating', 'weak', 'unowned', and 'final'.

It's definitely subjective, though. You might feel that the tests I describe above are valuable, and you're entitled to that opinion.

I have been in some situations recently in which encapsulation was untestable, and on a team with >10 developers and no automated test to catch that change, there were issues. In the end we shipped knowing there was a bug, because 15 weeks of work had been based on it, and it would have required a redesign to fix.

If I need a test that there is a property named 'bar', I can write a test which accesses the member, and even specify a required type and explicitly test whether the property does or does not conform to the protocol. Even if it's private, I can use a Mirror to extract its value.

My motivation is: I got tired of hearing "if you did it right, it would have worked", which I now view as an excuse for lack of good design (in most cases). For the past several months, I have been investigating design patterns which prevent developers from making common mistakes,

That is a very elusive goal.
The best way to do this is through hiring competent people…
One of the quotes I like in this regard: a monkey with a tool is still a monkey…

But I know, we all have to work with the hand we are dealt with. Nevertheless it is easy to sink a lot of work in this subject with little to show for it. People will make mistakes, no matter how good the tool. I have often felt that we need a certain balance, i.e. the tool should hurt you when making a mistake, but then it should not hurt so much that you die. That spurs good practises by the programmer who otherwise might become complacent. I know I get complacent with some Swift/Xcode features… and I hate it when I do, but I still do it…

Btw I have a lot of Ada experience. I love that language. It does way more in enforcing a proper design and catching programmers misttakes. However it never has gotten much traction for general applications. I blame that on the invisibility of avoided error’s…

Just rambling, no real point here… well, maybe one: don’t aim for perfection, but make sure it hurts when perfection is not achieved.

That way we will get paid for experience… :-o)

Rien.

···

On 12 Dec 2016, at 07:23, Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 11, 2016, at 11:43 PM, Brian Gesiak <modocache@gmail.com> wrote:

and how to write tests to catch changes to those patterns. I can't even count the number of unfathomable bugs I've fixed merely by improving the design. Swift is notable in that its compiler can enforce many of these patterns, so I prefer it over Obj-C. Proper encapsulation is one such technique, and it turns out it's untestable.

Thanks for your suggestion about researching how other languages do this. I'll see what I can do, though the only experience I have in any of the ones you mentioned is pre-11 C++, so if anyone else knows how such checks are done, I'd greatly appreciate hearing from you.

-Ben
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jean-Daniel) #6

An interesting reading about testing private members:

https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/

···

Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution <swift-evolution@swift.org> a écrit :

It bugs me as well that we can not test private components using XCTest. Objective-c never had this problem since privacy doesn't exist. I would like to see a version of XCTest that allows us to test every component.

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Howdy,
        I’d like to see how much interest there is for adding these to the XCTest module. If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal.

There are several features of Swift which cannot be effectively automated-tested.
For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror. There are similar concerns for testability with internal, and public access levels. Tests can be written ensuring these access levels are >= some level, but not == or < some level. In other words the very usefulness of these features cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.

Examples:
Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings. Here are some examples of what I would like to write in my test code:

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.
                + Can be distributed independently from Swift sources, requiring 0 buy-in from Swift community
                + requires a single additional module for the test.
                - Depends on ever-changing binary interface.
                : Intractable, not DRY

2. Use existing sourcekitd.
                + harnesses changes in the compiler’s source code with SourceKit
                - cannot be run in a test suite without extensive work by user to configure bundles explicitly.
                        Exceptionally poor user experience
                : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
                + harnesses changes in the compiler’s source code with SourceKit
                + no installation in user’s source code necessary
                : cannot be effectively run by SPM test infrastructure

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Benjamin Spratling) #7

If nothing else, I would hope that we can agree that enabling the testing of all our design choices is a goal.
Towards that, my goal here is falsifiability, the ability to execute a test which could demonstrate that my design is not what it is supposed to be. While exhaustiveness for such tests may be intractable, I believe that for designs with a small number of finite states, it is tractable.
-Ben

···

On Dec 12, 2016, at 2:17 AM, Rien <Rien@Balancingrock.nl> wrote:
That is a very elusive goal.
The best way to do this is through hiring competent people…


(Benjamin Spratling) #8

Howdy,

If my immediate goal were to verify private member values at intermediate steps in some process, which is the concept under consideration in the article you provided, I could access their values with a Mirror.

That's not my immediate goal. My immediate goal is to falsify that the property is private. The logic in the article asserts that a member is private, then asserts that because the member is private we do not need to test it. But it fails to provide an automated test that the member is indeed private and remains so. To accomplish that goal without an automated unit test is painstakingly tedious for a human.

And as to whether the logic in the article is correct, assuming its assumptions are true, it is still only one kind of testing. Depending on the field it might be called "interface" or "black box" testing, and they rightly call that out. But it would not be considered to be the only kind of testing which needs to be done.

-Ben

···

Sent from my iPhone.

On Dec 13, 2016, at 1:35 AM, Jean-Daniel <mailing@xenonium.com> wrote:

An interesting reading about testing private members:

https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/

Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution <swift-evolution@swift.org> a écrit :

It bugs me as well that we can not test private components using XCTest. Objective-c never had this problem since privacy doesn't exist. I would like to see a version of XCTest that allows us to test every component.

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:
Howdy,
        I’d like to see how much interest there is for adding these to the XCTest module. If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal.

There are several features of Swift which cannot be effectively automated-tested.
For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror. There are similar concerns for testability with internal, and public access levels. Tests can be written ensuring these access levels are >= some level, but not == or < some level. In other words the very usefulness of these features cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.

Examples:
Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings. Here are some examples of what I would like to write in my test code:

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.
                + Can be distributed independently from Swift sources, requiring 0 buy-in from Swift community
                + requires a single additional module for the test.
                - Depends on ever-changing binary interface.
                : Intractable, not DRY

2. Use existing sourcekitd.
                + harnesses changes in the compiler’s source code with SourceKit
                - cannot be run in a test suite without extensive work by user to configure bundles explicitly.
                        Exceptionally poor user experience
                : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
                + harnesses changes in the compiler’s source code with SourceKit
                + no installation in user’s source code necessary
                : cannot be effectively run by SPM test infrastructure

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Brian Gesiak) #9

Derrick and Jean-Daniel, it's my understanding that Benjamin is proposing adding a way *to test that members are private/internal/public*, not a way to access private members in tests. If you'd like to discuss what you're describing, it's probably best to start a separate email thread.

- Brian Gesiak

···

On Tue, Dec 13, 2016 at 2:35 AM -0500, "Jean-Daniel via swift-evolution" <swift-evolution@swift.org> wrote:

An interesting reading about testing private members:
https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/

Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution <swift-evolution@swift.org> a écrit :
It bugs me as well that we can not test private components using XCTest. Objective-c never had this problem since privacy doesn't exist. I would like to see a version of XCTest that allows us to test every component.

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:
Howdy,

    I’d like to see how much interest there is for adding these to the XCTest module\.  If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal\.

There are several features of Swift which cannot be effectively automated-tested.

For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror. There are similar concerns for testability with internal, and public access levels. Tests can be written ensuring these access levels are >= some level, but not == or < some level. In other words the very usefulness of these features cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.

Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.

Examples:

Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings. Here are some examples of what I would like to write in my test code:

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:

1. Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.

            \+ Can be distributed independently from Swift sources, requiring 0 buy\-in from Swift community

            \+ requires a single additional module for the test\.

            \- Depends on ever\-changing binary interface\.

            : Intractable, not DRY

2. Use existing sourcekitd.

            \+ harnesses changes in the compiler’s source code with SourceKit

            \- cannot be run in a test suite without extensive work by user to configure bundles explicitly\.

                    Exceptionally poor user experience

            : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests

            \+ harnesses changes in the compiler’s source code with SourceKit

            \+ no installation in user’s source code necessary

            : cannot be effectively run by SPM test infrastructure

_______________________________________________

swift-evolution mailing list

swift-evolution@swift.org

https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Pierre Monod-Broca) #10

Just to share my feelings and ideas.

It seems to me it would be better addressed by code review. Among other things it would start the conversation about what should be made public, what should not, and why.

On the automated part, there could be tools detecting changes to the public APIs (and providing diff) to trigger a review or something.

Pierre

···

Le 13 déc. 2016 à 16:57, Benjamin Spratling via swift-evolution <swift-evolution@swift.org> a écrit :

Howdy,

If my immediate goal were to verify private member values at intermediate steps in some process, which is the concept under consideration in the article you provided, I could access their values with a Mirror.

That's not my immediate goal. My immediate goal is to falsify that the property is private. The logic in the article asserts that a member is private, then asserts that because the member is private we do not need to test it. But it fails to provide an automated test that the member is indeed private and remains so. To accomplish that goal without an automated unit test is painstakingly tedious for a human.

And as to whether the logic in the article is correct, assuming its assumptions are true, it is still only one kind of testing. Depending on the field it might be called "interface" or "black box" testing, and they rightly call that out. But it would not be considered to be the only kind of testing which needs to be done.

-Ben

Sent from my iPhone.

On Dec 13, 2016, at 1:35 AM, Jean-Daniel <mailing@xenonium.com> wrote:

An interesting reading about testing private members:

https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/

Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution <swift-evolution@swift.org> a écrit :

It bugs me as well that we can not test private components using XCTest. Objective-c never had this problem since privacy doesn't exist. I would like to see a version of XCTest that allows us to test every component.

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:
Howdy,
        I’d like to see how much interest there is for adding these to the XCTest module. If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal.

There are several features of Swift which cannot be effectively automated-tested.
For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror. There are similar concerns for testability with internal, and public access levels. Tests can be written ensuring these access levels are >= some level, but not == or < some level. In other words the very usefulness of these features cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.

Examples:
Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings. Here are some examples of what I would like to write in my test code:

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.
                + Can be distributed independently from Swift sources, requiring 0 buy-in from Swift community
                + requires a single additional module for the test.
                - Depends on ever-changing binary interface.
                : Intractable, not DRY

2. Use existing sourcekitd.
                + harnesses changes in the compiler’s source code with SourceKit
                - cannot be run in a test suite without extensive work by user to configure bundles explicitly.
                        Exceptionally poor user experience
                : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
                + harnesses changes in the compiler’s source code with SourceKit
                + no installation in user’s source code necessary
                : cannot be effectively run by SPM test infrastructure

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jeremy Pereira) #11

The Swift compiler can give you the information that you want.

Here is what you can do:

Write a Swift source file that creates an instance of the type you want to test and then tries to access each of the private members.

Write a shell script to compile this source file i a module with the file the type is defined in. Have it capture all the error messages by redirecting stderr and then count them. If it doesn’t have the right number, have the shell script emit a message that looks like a Swift error message.

Install the script in a run script build phase. Now you will get an error every time one of your private properties or methods loses its access modifier.

Personally, I wouldn’t bother. Any test to make sure that private members are private requires a separately maintained list of the private members. If somebody isn’t disciplined enough to add the word “private” to the beginning of a definition, they almost certainly aren’t going to bother updating a separate list. And the consequences of omitting “private” are only that the module has visibility of it. That’s not a huge deal.

···

On 13 Dec 2016, at 15:57, Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:

Howdy,

If my immediate goal were to verify private member values at intermediate steps in some process, which is the concept under consideration in the article you provided, I could access their values with a Mirror.

That's not my immediate goal. My immediate goal is to falsify that the property is private. The logic in the article asserts that a member is private, then asserts that because the member is private we do not need to test it. But it fails to provide an automated test that the member is indeed private and remains so. To accomplish that goal without an automated unit test is painstakingly tedious for a human.

And as to whether the logic in the article is correct, assuming its assumptions are true, it is still only one kind of testing. Depending on the field it might be called "interface" or "black box" testing, and they rightly call that out. But it would not be considered to be the only kind of testing which needs to be done.

-Ben

Sent from my iPhone.

On Dec 13, 2016, at 1:35 AM, Jean-Daniel <mailing@xenonium.com> wrote:

An interesting reading about testing private members:

https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/

Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution <swift-evolution@swift.org> a écrit :

It bugs me as well that we can not test private components using XCTest. Objective-c never had this problem since privacy doesn't exist. I would like to see a version of XCTest that allows us to test every component.

On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:
Howdy,
        I’d like to see how much interest there is for adding these to the XCTest module. If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal.

There are several features of Swift which cannot be effectively automated-tested.
For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror. There are similar concerns for testability with internal, and public access levels. Tests can be written ensuring these access levels are >= some level, but not == or < some level. In other words the very usefulness of these features cannot be tested.

Other attributes to be tested in this way are:

- Mutability of a member. No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.

- That a stored property is weak or unowned.

- That a class or class member is “final”

These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.
Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.

Examples:
Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings. Here are some examples of what I would like to write in my test code:

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)

XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)

Alternatives:
1. Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.
                + Can be distributed independently from Swift sources, requiring 0 buy-in from Swift community
                + requires a single additional module for the test.
                - Depends on ever-changing binary interface.
                : Intractable, not DRY

2. Use existing sourcekitd.
                + harnesses changes in the compiler’s source code with SourceKit
                - cannot be run in a test suite without extensive work by user to configure bundles explicitly.
                        Exceptionally poor user experience
                : sourcekitd XPC architecture only works on macOS

3. Use a standalone tool for tests
                + harnesses changes in the compiler’s source code with SourceKit
                + no installation in user’s source code necessary
                : cannot be effectively run by SPM test infrastructure

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Benjamin Spratling) #12

Howdy,
  Do you see this working with the package manager as well?
-Ben

···

On Dec 14, 2016, at 7:42 AM, Jeremy Pereira <jeremy.j.pereira@googlemail.com> wrote:

The Swift compiler can give you the information that you want.

Here is what you can do:

Write a Swift source file that creates an instance of the type you want to test and then tries to access each of the private members.

Write a shell script to compile this source file i a module with the file the type is defined in. Have it capture all the error messages by redirecting stderr and then count them. If it doesn’t have the right number, have the shell script emit a message that looks like a Swift error message.

Install the script in a run script build phase. Now you will get an error every time one of your private properties or methods loses its access modifier.

Personally, I wouldn’t bother. Any test to make sure that private members are private requires a separately maintained list of the private members. If somebody isn’t disciplined enough to add the word “private” to the beginning of a definition, they almost certainly aren’t going to bother updating a separate list. And the consequences of omitting “private” are only that the module has visibility of it. That’s not a huge deal.


(Derrick Ho) #13

You might be able to ensure access modifiers by using protocols.

If you want to ensure that a class has a property called foo that is
private you can make a private protocol that specifies a private property
called foo.

This isn't a XCTest but it is a compile time check. If a programmer tries
to change a private property to public then the the compiler will complain.

The same can be done with final. Make a protocol with a static method.

···

On Wed, Dec 14, 2016 at 8:42 AM Jeremy Pereira via swift-evolution < swift-evolution@swift.org> wrote:

The Swift compiler can give you the information that you want.

Here is what you can do:

Write a Swift source file that creates an instance of the type you want to
test and then tries to access each of the private members.

Write a shell script to compile this source file i a module with the file
the type is defined in. Have it capture all the error messages by
redirecting stderr and then count them. If it doesn’t have the right
number, have the shell script emit a message that looks like a Swift error
message.

Install the script in a run script build phase. Now you will get an error
every time one of your private properties or methods loses its access
modifier.

Personally, I wouldn’t bother. Any test to make sure that private members
are private requires a separately maintained list of the private members.
If somebody isn’t disciplined enough to add the word “private” to the
beginning of a definition, they almost certainly aren’t going to bother
updating a separate list. And the consequences of omitting “private” are
only that the module has visibility of it. That’s not a huge deal.

> On 13 Dec 2016, at 15:57, Benjamin Spratling via swift-evolution < > swift-evolution@swift.org> wrote:
>
> Howdy,
>
> If my immediate goal were to verify private member values at
intermediate steps in some process, which is the concept under
consideration in the article you provided, I could access their values with
a Mirror.
>
> That's not my immediate goal. My immediate goal is to falsify that the
property is private. The logic in the article asserts that a member is
private, then asserts that because the member is private we do not need to
test it. But it fails to provide an automated test that the member is
indeed private and remains so. To accomplish that goal without an
automated unit test is painstakingly tedious for a human.
>
> And as to whether the logic in the article is correct, assuming its
assumptions are true, it is still only one kind of testing. Depending on
the field it might be called "interface" or "black box" testing, and they
rightly call that out. But it would not be considered to be the only kind
of testing which needs to be done.
>
> -Ben
>
> Sent from my iPhone.
>
> On Dec 13, 2016, at 1:35 AM, Jean-Daniel <mailing@xenonium.com> wrote:
>
>> An interesting reading about testing private members:
>>
>> https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/
>>
>>
>>> Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution < > swift-evolution@swift.org> a écrit :
>>>
>>> It bugs me as well that we can not test private components using
XCTest. Objective-c never had this problem since privacy doesn't exist. I
would like to see a version of XCTest that allows us to test every
component.
>>>
>>>
>>> On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via > swift-evolution <swift-evolution@swift.org> wrote:
>>> Howdy,
>>> I’d like to see how much interest there is for adding these to
the XCTest module. If I have missed some pro or con, or missed a technical
point, your feedback is welcome before I go to the lengths to draw up a
formal proposal.
>>>
>>>
>>> There are several features of Swift which cannot be effectively
automated-tested.
>>> For instance, when a type or one of its members is private or file
private, an outside test suite cannot access it, and the information that
the type is private is not included in a Mirror. There are similar
concerns for testability with internal, and public access levels. Tests
can be written ensuring these access levels are >= some level, but not ==
or < some level. In other words the very usefulness of these features
cannot be tested.
>>>
>>> Other attributes to be tested in this way are:
>>>
>>> - Mutability of a member. No automated test can be written which
verifies that a function cannot be called on a let struct, or that a
property cannot be set at a specific access level.
>>>
>>> - That a stored property is weak or unowned.
>>>
>>> - That a class or class member is “final”
>>>
>>> These are concepts which need to be unit-tested to ensure good design
is not broken, but do not need to be included in a release build, so
including them in the XCTest framework seems like an appropriate
destination.
>>> Moreover, the information for all of these features exists in the
.swiftmodule files, which are included in test builds, but sometimes
stripped for release.
>>>
>>> Examples:
>>> Since these features inherently have to do with testing features which
cannot be stated in compiled code, I recommend specifying names with
Strings. Here are some examples of what I would like to write in my test
code:
>>>
>>> XCTAssertEqual(
Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel,
.private)
>>>
>>> XCTAssertEqual(
Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable,
false)
>>>
>>>
>>> Alternatives:
>>> 1. Building an independent .swiftmodule parser in a single Swift
module, which can be included in test builds.
>>> + Can be distributed independently from Swift sources,
requiring 0 buy-in from Swift community
>>> + requires a single additional module for the test.
>>> - Depends on ever-changing binary interface.
>>> : Intractable, not DRY
>>>
>>> 2. Use existing sourcekitd.
>>> + harnesses changes in the compiler’s source code with
SourceKit
>>> - cannot be run in a test suite without extensive work
by user to configure bundles explicitly.
>>> Exceptionally poor user experience
>>> : sourcekitd XPC architecture only works on macOS
>>>
>>> 3. Use a standalone tool for tests
>>> + harnesses changes in the compiler’s source code with
SourceKit
>>> + no installation in user’s source code necessary
>>> : cannot be effectively run by SPM test infrastructure
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution@swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution@swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Benjamin Spratling) #14

Howdy,
  Thanks, I’ll consider this as an alternative.
  I don’t see that this solves the issues of whether a property on a class is let, however.
-Ben

···

On Dec 14, 2016, at 11:41 PM, Derrick Ho <wh1pch81n@gmail.com> wrote:

You might be able to ensure access modifiers by using protocols.

If you want to ensure that a class has a property called foo that is private you can make a private protocol that specifies a private property called foo.

This isn't a XCTest but it is a compile time check. If a programmer tries to change a private property to public then the the compiler will complain.

The same can be done with final. Make a protocol with a static method.


(Derrick Ho) #15

Use protocols

{get}
Can be used for let

{get set}
Can be used for var

···

On Thu, Dec 15, 2016 at 1:16 AM Benjamin Spratling via swift-evolution < swift-evolution@swift.org> wrote:

Howdy,
        Thanks, I’ll consider this as an alternative.
        I don’t see that this solves the issues of whether a property on a
class is let, however.
-Ben

> On Dec 14, 2016, at 11:41 PM, Derrick Ho <wh1pch81n@gmail.com> wrote:
>
> You might be able to ensure access modifiers by using protocols.
>
> If you want to ensure that a class has a property called foo that is
private you can make a private protocol that specifies a private property
called foo.
>
> This isn't a XCTest but it is a compile time check. If a programmer
tries to change a private property to public then the the compiler will
complain.
>
> The same can be done with final. Make a protocol with a static method.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Brian King) #16

Another relatively new option is using Sourcery
(https://github.com/krzysztofzablocki/Sourcery). It sits on top of
SourceKit and provides a Stencil template to generate code. You could
generate code that reflects the visibility of your functions and write
tests that validate the generated code.

I think Mirror types may be able to accommodate this type of testing
in the future, but this may fill the gap for now.

Brian

···

On Thu, Dec 15, 2016 at 1:37 AM, Benjamin Spratling via swift-evolution <swift-evolution@swift.org> wrote:

Howdy,
        Do you see this working with the package manager as well?
-Ben

On Dec 14, 2016, at 7:42 AM, Jeremy Pereira <jeremy.j.pereira@googlemail.com> wrote:

The Swift compiler can give you the information that you want.

Here is what you can do:

Write a Swift source file that creates an instance of the type you want to test and then tries to access each of the private members.

Write a shell script to compile this source file i a module with the file the type is defined in. Have it capture all the error messages by redirecting stderr and then count them. If it doesn’t have the right number, have the shell script emit a message that looks like a Swift error message.

Install the script in a run script build phase. Now you will get an error every time one of your private properties or methods loses its access modifier.

Personally, I wouldn’t bother. Any test to make sure that private members are private requires a separately maintained list of the private members. If somebody isn’t disciplined enough to add the word “private” to the beginning of a definition, they almost certainly aren’t going to bother updating a separate list. And the consequences of omitting “private” are only that the module has visibility of it. That’s not a huge deal.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution