Running Swift Standard Library Tests

I'm trying to add Result to the standard library. While I can successfully build Swift (using utils/build-script --release-debuginfo --xcode) and have added Result.swift in both stdlib/public/core and test/stdlib/, I can't seem to run the tests. Trying to run all tests fails right way in the CommonMark tests, and trying to run lit tests always results in unresolved tests and warnings like warning: couldn't find 'sil-opt' program, try setting SIL_OPT in your environment. Am I really supposed to set 11 env vars to the binaries that were built? Or am I missing something?

All tests are still unresolved after setting all recommended vars.


The generated Xcode project does not integrate with the test runner, but the tests can be run with the 'check-swift' target.

I'm not trying to run the tests through Xcode, but through lit. Are you saying that if I built with Xcode support, I can't run the tests with lit?

I believe if you're using Xcode lit does not know how to run tests. I don't develop with Xcode though so I could be wrong here. What does your lit configuration look like?


# - Local configuration for the 'lit' test runner -*- python -*-
# This source file is part of the open source project
# Copyright (c) 2014 - 2017 Apple Inc. and the Swift project authors
# Licensed under Apache License v2.0 with Runtime Library Exception
# See for license information
# See for the list of Swift project authors
# -----------------------------------------------------------------------------

import os
import platform
import sys

config.llvm_src_root = "/Users/jshier/Desktop/Code/swift-sources/llvm"
config.llvm_obj_root = "/Users/jshier/Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/llvm-macosx-x86_64"
config.llvm_tools_dir = "/Users/jshier/Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/llvm-macosx-x86_64/%(build_mode)s/bin"
config.llvm_libs_dir = "/Users/jshier/Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/llvm-macosx-x86_64/%(build_mode)s/lib"
config.llvm_code_generators = "X86;ARM;AArch64;PowerPC;SystemZ;Mips".split(";")
config.lit_tools_dir = ""
config.swift_src_root = lit_config.params.get("swift_src_root", "/Users/jshier/Desktop/Code/swift-sources/swift")
config.swift_obj_root = "/Users/jshier/Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64"
config.target_triple = "x86_64-apple-macosx10.9"
config.variant_triple = "x86_64-apple-macosx10.9"
config.variant_sdk = "/Applications/"
config.variant_suffix = "-macosx-x86_64"
config.swiftlib_dir = "/Users/jshier/Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/Debug/lib/swift"
config.swift_test_results_dir = \
    lit_config.params.get("swift_test_results_dir", "/Users/jshier/Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME)/swift-test-results/x86_64-apple-macosx10.9")
config.darwin_xcrun_toolchain = "default"
config.android_ndk_path = ""
config.android_ndk_gcc_version = ""

config.coverage_mode = "FALSE"
config.lldb_build_root = ""

# Please remember to handle empty strings and/or unset variables correctly.

if "FALSE" == "TRUE":

if "FALSE" == "TRUE":

if ''.lower() in ['full', 'thin']:

if "TRUE" == "TRUE":

if "TRUE" == "TRUE":

if "TRUE" == "TRUE":

if "TRUE" == "TRUE":

if "FALSE" == "TRUE":

if "TRUE" == "FALSE":

if "TRUE" == "TRUE":

if "Xcode" == "Xcode":
    xcode_bin_dir = os.path.join(config.llvm_obj_root, "RelWithDebInfo",
    lit_config.note('Adding to path: ' + xcode_bin_dir)
    config.environment['PATH'] = \
      os.path.pathsep.join((xcode_bin_dir, config.environment['PATH']))


if "TRUE" == "TRUE":

# Let the main config do the real work.
if config.test_exec_root is None:
    config.test_exec_root = os.path.dirname(os.path.realpath(__file__))
    config, os.path.join(config.swift_src_root, "test", "lit.cfg"))

I've manually set the environment vars for the built tools as well.

My apologies, how are you invocating lit? Using your config it looks like you should use something like this:

~: Desktop/Code/swift-sources/llvm/utils/lit/ -sv Desktop/Code/swift-sources/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/

If the above works for you, don't forget you can add the --filter= to only run a single test

When I run that, it just prints UNRESOLVED for everything. e.g. UNRESOLVED: Swift(macosx-x86_64) :: remote-run/upload-and-download.test-sh (3987 of 4253) I can't find what that means.

Same if I try to run against just my Result.swift test file manually:
lit -s --param swift_site_config=${SWIFT_BUILD_DIR}/test-macosx-x86_64/ ${SWIFT_SOURCES_DIR}/test/stdlib/Result.swift

UNRESOLVED: Swift(macosx-x86_64) :: stdlib/Result.swift (1 of 1)
Testing Time: 0.11s
Unresolved Tests (1):
    Swift(macosx-x86_64) :: stdlib/Result.swift

  Unresolved Tests   : 1

Ah yeah, so using the Xcode configuration the build script doesn't know how to build the test suite, thus you can't run lit. Although the README states that you can at least test using the check-swift target.

Guess I'll switch over to the non-Xcode build, as check-swift current fails to build for me due to failures in the CommonMarkTests.

Rebuilding without Xcode support seems to let tests work automatically.

There is a way to run the lit tests with an Xcode build, IIRC swift/utils/run-test --build-dir path-to-build-dir/Xcode-DebugAssert, but I haven't tried. It simply doesn't support building all necessary targets for those tests, and judging by myself and what I know about how Apple people work on the compiler and toolchain, it's significantly more convenient to have both a ninja build for compiling and testing and an --xcode build for editing.

1 Like

Yes, this is also what I do when I'm working on the compiler.