[build-script] Automatic inferences of dependencies using new option: "--infer"

Hey everyone! Just wanted to bring to your attention a new build-script flag I added that should hopefully make everyone's lives easier. Specifically, as of [build-script] Add option --infer to infer dependencies. by gottesmm · Pull Request #32256 · apple/swift · GitHub one can optionally tell build-script to automatically infer dependencies. This means for instance to build swiftpm, one can now just do:

./swift/utils/build-script --infer --swiftpm

instead of:

./swift/utils/build-script --build-ninja --libcxx --llbuild --swiftpm
--install-swift --install-llbuild --install-swiftpm --install-libcxx
'--llvm-install-components=llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld'
--libdispatch --libicu --foundation --xctest --install-libicu
--install-foundation --install-libdispatch --install-xctest --release
--assertions

Thus extending/finishing the work I talked about in: [build-script] New option: "--install-all" - #2 by johannesweiss (one can use install-all as well, but infer yields significantly more brevity). Keep in mind this is not enabled by default on purpose since I do not have time to response very quickly to problems that come up. Instead please everyone take this out for a spin and tell me what you think.

If people have good experiences with this over the next few weeks, we should consider making this the default mode when we are not using presets (presets still will use the old mode).

I hope this saves people fingers some strain!

10 Likes

Hi @Michael_Gottesman, many thanks for this, this will greatly simplify invocations of build-script, especially for non-Apple platforms!

I see that these changes involved adding the dependencies logic to the Python codebase. Is there a plan to specify the dependencies in CMake files in the future, so that those are resolved automatically by CMake/ninja?

The plan longer term is to move away from build-script and having the dependencies in build-script itself. In fact, my favored outcome is to just have build-script effectively generate cmake caches. The original purpose of build-script was not to make a new build-system, but rather to standardize how we build by using presets. Supporting simple command line presets does not entail building a whole new build system.

That being said, this work is just an interim step until we reach that point to help people's fingers. It is important to realize that this is part of a longer term transition. We can not just break the world over night.

2 Likes

I tried using the --infer option — without enabling any additional projects — on the default branches:

swift/utils/update-checkout --clone

swift/utils/build-script --infer --reconfigure --release-debuginfo --test

There was a failure when running the tests:

lit.py: /Users/benrimmington/Public/apple/llvm-project/llvm/utils/lit/lit/TestingConfig.py:102: fatal: unable to parse config file '/Users/benrimmington/Public/apple/swift/test/lit.cfg', traceback: Traceback (most recent call last):
  File "/Users/benrimmington/Public/apple/llvm-project/llvm/utils/lit/lit/TestingConfig.py", line 89, in load_from_path
    exec(compile(data, path, 'exec'), cfg_globals, None)
  File "/Users/benrimmington/Public/apple/swift/test/lit.cfg", line 1921, in <module>
    lldb_python_path = get_lldb_python_path(config.lldb_build_root)
  File "/Users/benrimmington/Public/apple/swift/test/lit.cfg", line 103, in get_lldb_python_path
    return subprocess.check_output([lldb_path, "-P"]).rstrip()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

FAILED: test/CMakeFiles/check-swift-macosx-x86_64

The bin/lldb -P subprocess raised a "no such file or directory" error, because the lldb project wasn't built. When I removed the --infer option, building and testing succeeded (without involving the lldb project).

Environment:

  • CMake 3.17.0
  • macOS 10.15.5
  • Xcode-beta 12.0

I have seen this before. I am not 100% sure why it happens, but I think it is b/c of how lit is checking that lldb is built which is clearly insufficient. File a bug on bugs.swift.org and +CC me?

I filed SR-13097, which has been automatically assigned to Mishal Shah.

1 Like

@benrimmington just to loop back around, I fixed this here: [lit] Detect lldb by checking both for lldb_build_root and that that root contains an lldb exec. by gottesmm · Pull Request #32643 · apple/swift · GitHub.

1 Like

@Michael_Gottesman thanks, it works now. The tests finished (with your new warning about lldb_build_root).