builtin command for invoking a subset of the tests


(Alexis) #1

When fixing tests, it’s often useful to be able to run some subset of them, usually based on some pattern. From my searching, the recommended way to do this seems to be to directly invoke `lit.py`. Doing this by hand is tedious, so I use the following script:

    #!/bin/bash
    
    ../llvm/utils/lit/lit.py -sv --param swift_site_config=../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/lit.site.cfg "$@"

But this has the unfortunate downside of hard-coding the compiler to use. It’s also unfortunate for newcomers to the build system, because they need to hunt down this magical invocation, or suffer through running all the tests on every change.

It seems to me that build-script should support this kind of invocation, so that we can say something like:

    utils/build-script -r —test-only test/stdlib/Dictionary*

Thoughts?


(Tony Parker) #2

Hi Alexis,

In fact there is already a script which is closer to what you want, in swift/utils/run-test. It could probably use some additional love and attention to be a bit more usable (for example, printing out the help if you invoke it with no arguments), but I use it all the time.

- Tony

···

On Oct 14, 2016, at 8:24 AM, Alexis via swift-dev <swift-dev@swift.org> wrote:

When fixing tests, it’s often useful to be able to run some subset of them, usually based on some pattern. From my searching, the recommended way to do this seems to be to directly invoke `lit.py`. Doing this by hand is tedious, so I use the following script:

   #!/bin/bash

   ../llvm/utils/lit/lit.py -sv --param swift_site_config=../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/lit.site.cfg "$@"

But this has the unfortunate downside of hard-coding the compiler to use. It’s also unfortunate for newcomers to the build system, because they need to hunt down this magical invocation, or suffer through running all the tests on every change.

It seems to me that build-script should support this kind of invocation, so that we can say something like:

   utils/build-script -r —test-only test/stdlib/Dictionary*

Thoughts?
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Alexis) #3

Oh hey, great! Then perhaps the bigger issue is that this program should be better highlighted in the developer docs, and possibly even called out in the build-script docs?

I agree that it needs a bit of TLC, though. Having to specify the build-dir is a bit disappointing. Any reason it can’t support “use the one that results from these build-script flags” as the default workflow? (I can understand wanting to keep build-dir around for special cases)

···

On Oct 14, 2016, at 11:29 AM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Alexis,

In fact there is already a script which is closer to what you want, in swift/utils/run-test. It could probably use some additional love and attention to be a bit more usable (for example, printing out the help if you invoke it with no arguments), but I use it all the time.

- Tony

On Oct 14, 2016, at 8:24 AM, Alexis via swift-dev <swift-dev@swift.org> wrote:

When fixing tests, it’s often useful to be able to run some subset of them, usually based on some pattern. From my searching, the recommended way to do this seems to be to directly invoke `lit.py`. Doing this by hand is tedious, so I use the following script:

  #!/bin/bash

  ../llvm/utils/lit/lit.py -sv --param swift_site_config=../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/lit.site.cfg "$@"

But this has the unfortunate downside of hard-coding the compiler to use. It’s also unfortunate for newcomers to the build system, because they need to hunt down this magical invocation, or suffer through running all the tests on every change.

It seems to me that build-script should support this kind of invocation, so that we can say something like:

  utils/build-script -r —test-only test/stdlib/Dictionary*

Thoughts?
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Tony Parker) #4

Oh hey, great! Then perhaps the bigger issue is that this program should be better highlighted in the developer docs, and possibly even called out in the build-script docs?

I agree that it needs a bit of TLC, though. Having to specify the build-dir is a bit disappointing. Any reason it can’t support “use the one that results from these build-script flags” as the default workflow? (I can understand wanting to keep build-dir around for special cases)

Agreed on both points. Probably worth tracking as a starter bug in JIRA. =)

- Tony

···

On Oct 14, 2016, at 9:54 AM, Alexis <abeingessner@apple.com> wrote:

On Oct 14, 2016, at 11:29 AM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Alexis,

In fact there is already a script which is closer to what you want, in swift/utils/run-test. It could probably use some additional love and attention to be a bit more usable (for example, printing out the help if you invoke it with no arguments), but I use it all the time.

- Tony

On Oct 14, 2016, at 8:24 AM, Alexis via swift-dev <swift-dev@swift.org> wrote:

When fixing tests, it’s often useful to be able to run some subset of them, usually based on some pattern. From my searching, the recommended way to do this seems to be to directly invoke `lit.py`. Doing this by hand is tedious, so I use the following script:

#!/bin/bash

../llvm/utils/lit/lit.py -sv --param swift_site_config=../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/lit.site.cfg "$@"

But this has the unfortunate downside of hard-coding the compiler to use. It’s also unfortunate for newcomers to the build system, because they need to hunt down this magical invocation, or suffer through running all the tests on every change.

It seems to me that build-script should support this kind of invocation, so that we can say something like:

utils/build-script -r —test-only test/stdlib/Dictionary*

Thoughts?
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Alexis) #5

I've filed

https://bugs.swift.org/browse/SR-2960
https://bugs.swift.org/browse/SR-2961

(I have no idea of I've done that right)

···

Sent from my iPhone

On Oct 14, 2016, at 1:01 PM, Tony Parker <anthony.parker@apple.com> wrote:

On Oct 14, 2016, at 9:54 AM, Alexis <abeingessner@apple.com> wrote:

Oh hey, great! Then perhaps the bigger issue is that this program should be better highlighted in the developer docs, and possibly even called out in the build-script docs?

I agree that it needs a bit of TLC, though. Having to specify the build-dir is a bit disappointing. Any reason it can’t support “use the one that results from these build-script flags” as the default workflow? (I can understand wanting to keep build-dir around for special cases)

Agreed on both points. Probably worth tracking as a starter bug in JIRA. =)

- Tony

On Oct 14, 2016, at 11:29 AM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Alexis,

In fact there is already a script which is closer to what you want, in swift/utils/run-test. It could probably use some additional love and attention to be a bit more usable (for example, printing out the help if you invoke it with no arguments), but I use it all the time.

- Tony

On Oct 14, 2016, at 8:24 AM, Alexis via swift-dev <swift-dev@swift.org> wrote:

When fixing tests, it’s often useful to be able to run some subset of them, usually based on some pattern. From my searching, the recommended way to do this seems to be to directly invoke `lit.py`. Doing this by hand is tedious, so I use the following script:

#!/bin/bash

../llvm/utils/lit/lit.py -sv --param swift_site_config=../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/lit.site.cfg "$@"

But this has the unfortunate downside of hard-coding the compiler to use. It’s also unfortunate for newcomers to the build system, because they need to hunt down this magical invocation, or suffer through running all the tests on every change.

It seems to me that build-script should support this kind of invocation, so that we can say something like:

utils/build-script -r —test-only test/stdlib/Dictionary*

Thoughts?
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev