How to run the tests out-of-tree?


(Karl) #1

Hi

So I’ve been working on cross-compiling for ARM, and it seems pretty good - only a bit of refactoring of the build script really needed. It’s a popular request, the lack of which is hindering lots of people and businesses who would like to experiment with swift on their ARM-based devices.

The thing that’s really missing before I can propose merging these changes is running the in-tree tests (swift/tests/ folder) to validate the products. I basically want to copy that folder in to a package and run it on the target device, but I’m not really sure how to do that with lit. I’ve built the unit test binaries (the ones which statically link against the runtime), and those obviously are easy enough to package and script for out-of-tree use.

Can anybody help with this?

Thanks

Karl


(Jordan Rose) #2

Hi, Karl. I don’t think it would be too hard to handle most of them. The tests are actually pretty standalone: they require the LLVM tool “lit <http://llvm.org/docs/CommandGuide/lit.html>” (in llvm/utils/lit/) and some helper tools from the build directory like “FileCheck” and “not”. To run them, the build system generates a valid lit.site.cfg file from test/lit.site.cfg.in and then points lit.py at that.

That’s really all you need. I regularly run lit by hand just pointing to the directory containing the lit.site.cfg file.

% /Volumes/Data/swift-public/llvm/utils/lit/lit.py -sv /Volumes/Data/swift-public/build/ninja/swift-macosx-x86_64/test-macosx-x86_64/

So if you can come up with a valid lit.site.cfg file and set PATH appropriately, you’ll be most of the way there, and then you can iron out remaining issues later.

Jordan

···

On May 12, 2016, at 07:45, Karl via swift-dev <swift-dev@swift.org> wrote:

Hi

So I’ve been working on cross-compiling for ARM, and it seems pretty good - only a bit of refactoring of the build script really needed. It’s a popular request, the lack of which is hindering lots of people and businesses who would like to experiment with swift on their ARM-based devices.

The thing that’s really missing before I can propose merging these changes is running the in-tree tests (swift/tests/ folder) to validate the products. I basically want to copy that folder in to a package and run it on the target device, but I’m not really sure how to do that with lit. I’ve built the unit test binaries (the ones which statically link against the runtime), and those obviously are easy enough to package and script for out-of-tree use.

Can anybody help with this?

Thanks

Karl
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Greg Parker) #3

You should look at the iOS device and iOS simulator machinery in swift/test/lit.cfg for one solution. Both cases use lit substitutions like %target-run to execute a helper tool that can run the test in the appropriate environment and collect the output. Our iOS device testing tool is not open-source, but it uploads a test's source and build directories to the device, runs the test executable on the device, and collects stdout and stderr and any output files. Any other tools used by the test script such as FileCheck still run on the test hosting machine, not on the target device being tested.

Some of the test scripts are complicated, interleaving compile commands and execution commands, so they can't be cleanly separated into a cross-compilation step and an on-device execution step. The %target-run mechanism allows on-device execution without forcing test authors to separate the device-side content.

···

On May 12, 2016, at 7:45 AM, Karl via swift-dev <swift-dev@swift.org> wrote:

So I’ve been working on cross-compiling for ARM, and it seems pretty good - only a bit of refactoring of the build script really needed. It’s a popular request, the lack of which is hindering lots of people and businesses who would like to experiment with swift on their ARM-based devices.

The thing that’s really missing before I can propose merging these changes is running the in-tree tests (swift/tests/ folder) to validate the products. I basically want to copy that folder in to a package and run it on the target device, but I’m not really sure how to do that with lit. I’ve built the unit test binaries (the ones which statically link against the runtime), and those obviously are easy enough to package and script for out-of-tree use.

Can anybody help with this?

--
Greg Parker gparker@apple.com <mailto:gparker@apple.com> Runtime Wrangler


(Karl) #4

My original idea was a lot simpler - create a script grabbing the specific executables we need (FileCheck, llvm-link, etc) and the tests, package it up, ship it out via ssh and run it on the intended target, piping the output back. I just couldn’t really figure out what the specific dependencies where - the site config seemed to want the entire source and build tree of swift and LLVM, and I don’t want to package all of that stuff up.

Karl

···

On 13 May 2016, at 02:36, Greg Parker <gparker@apple.com> wrote:

On May 12, 2016, at 7:45 AM, Karl via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

So I’ve been working on cross-compiling for ARM, and it seems pretty good - only a bit of refactoring of the build script really needed. It’s a popular request, the lack of which is hindering lots of people and businesses who would like to experiment with swift on their ARM-based devices.

The thing that’s really missing before I can propose merging these changes is running the in-tree tests (swift/tests/ folder) to validate the products. I basically want to copy that folder in to a package and run it on the target device, but I’m not really sure how to do that with lit. I’ve built the unit test binaries (the ones which statically link against the runtime), and those obviously are easy enough to package and script for out-of-tree use.

Can anybody help with this?

You should look at the iOS device and iOS simulator machinery in swift/test/lit.cfg for one solution. Both cases use lit substitutions like %target-run to execute a helper tool that can run the test in the appropriate environment and collect the output. Our iOS device testing tool is not open-source, but it uploads a test's source and build directories to the device, runs the test executable on the device, and collects stdout and stderr and any output files. Any other tools used by the test script such as FileCheck still run on the test hosting machine, not on the target device being tested.

Some of the test scripts are complicated, interleaving compile commands and execution commands, so they can't be cleanly separated into a cross-compilation step and an on-device execution step. The %target-run mechanism allows on-device execution without forcing test authors to separate the device-side content.

--
Greg Parker gparker@apple.com <mailto:gparker@apple.com> Runtime Wrangler


(Dmitri Gribenko) #5

Are you building a native compiler or a cross-compiler?

Dmitri

···

On Thu, May 12, 2016 at 6:33 PM, Karl Wagner via swift-dev <swift-dev@swift.org> wrote:

My original idea was a lot simpler - create a script grabbing the specific
executables we need (FileCheck, llvm-link, etc) and the tests, package it
up, ship it out via ssh and run it on the intended target, piping the output
back. I just couldn’t really figure out what the specific dependencies where
- the site config seemed to want the entire source and build tree of swift
and LLVM, and I don’t want to package all of that stuff up.

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Karl) #6

I’m cross-compiling a native compiler. That is to say, I’m not building a cross-compiler in the way we build for Android or iOS: a native compiler with foreign standard libraries. Both the compiler and stdlib are foreign to the build machine.

I’d like to produce an installable package (that much is good), and a standalone package of tests. I’ll have a go at it in the next few days, then maybe we can add an installable component to produce a test package. I was pretty disappointed that the “testsuite-tools” component we have basically links back to the in-tree tests; It’s not really “installable".

Karl

···

On 13 May 2016, at 03:40, Dmitri Gribenko <gribozavr@gmail.com> wrote:

On Thu, May 12, 2016 at 6:33 PM, Karl Wagner via swift-dev > <swift-dev@swift.org> wrote:

My original idea was a lot simpler - create a script grabbing the specific
executables we need (FileCheck, llvm-link, etc) and the tests, package it
up, ship it out via ssh and run it on the intended target, piping the output
back. I just couldn’t really figure out what the specific dependencies where
- the site config seemed to want the entire source and build tree of swift
and LLVM, and I don’t want to package all of that stuff up.

Are you building a native compiler or a cross-compiler?

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Dmitri Gribenko) #7

A standalone test package is something we really want, but I suspect
it might turn into a non-trivial project.

Dmitri

···

On Thu, May 12, 2016 at 7:24 PM, Karl Wagner <razielim@gmail.com> wrote:

I’m cross-compiling a native compiler. That is to say, I’m not building a cross-compiler in the way we build for Android or iOS: a native compiler with foreign standard libraries. Both the compiler and stdlib are foreign to the build machine.

I’d like to produce an installable package (that much is good), and a standalone package of tests. I’ll have a go at it in the next few days, then maybe we can add an installable component to produce a test package. I was pretty disappointed that the “testsuite-tools” component we have basically links back to the in-tree tests; It’s not really “installable".

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/