Progress on Opaque SIL Values

Development Compiler
1

Hi everyone,

SIL opaque values (-enable-sil-opaque-values) is a compiler infrastructure project that defers the lowering of resilient and generic types to addresses, until the AddressLowering pass, so they are treated like loadable SSA values for better optimization. It's an idea proposed nearly 10 years ago (!) in Proposal: Opaque SIL values . I'll defer to the original proposal for an overview and additional motivation, as it is still true today.

The first stage of this project has been to make SILGen fully compatible with SIL opaque values and run AddressLowering soon after SILGen. This would allow us to have -enable-sil-opaque-values on by default and set us up to unlock the first of several benefits of this project: simplify SILGen.

Many people over the years have put in work to get that first stage done. Unfortunately, every time we have reached a milestone in terms of full functionality, such as building the standard library cleanly, we've regressed.

Progress is lost primarily from new contributions to SILGen or SIL that are not tested under SIL Opaque Values. So, I'd like to ask everyone to please test your changes under -enable-sil-opaque-values too:

  • For a non-executable test doing -emit-sil , -emit-ir or later: Add a RUN line with -enable-sil-opaque-values . That covers both SILGen and AddressLowering compatibility.
    • If you have separate SILGen and SIL test files, you can test just SILGen under opaque values with -emit-silgen-ossa -enable-sil-opaque-values , which goes a bit further than -emit-silgen to reach verified SIL, but without running any diagnostic or mandatory passes.
  • For a test using %target-run-simple-swift, we have a build target check-swift-optimize_none_with_opaque_values that will compile and run your test under -enable-sil-opaque-values.
  • Tests that only go up to -typecheck or an earlier stage of the compiler don't need additional testing.

In an effort to prevent further backsliding as work continues on opaque values, I plan to increase the amount of testing and usage of it:

  • Specific sub-targets of the standard library will start building with the flag enabled. So far, we have swift-backtrace using it and I'm eyeing Volatile, RegexParser, RegexBuilder, and Distributed next.
  • I've started auditing and automatically updating tests to have an additional RUN line exercising opaque values.
  • CI will start running the execution tests under opaque values. This will mean you won't need a second RUN line if your test is using %target-run-simple-swift

Here are some guidelines while working on new code in SILGen:

  • Rely on SILFunctionConventions to determine whether an argument or result of a function is transferred indirectly (i.e., as an address) or directly as an SSA value.
  • Avoid using TypeLowering::isAddressOnly() , as it will give only the fixed, post-AddressLowering answer, which is wrong in SILGen under opaque values. I'm working to make that API unavailable within SILGen with this PR and introducing isLoadableOrOpaque.
  • Much of what is backing opaque values today in the compiler relies upon the SILAddressConventions type and the AddressLowering pass, so they're worth a read.
15 Likes
2

I'd like to highlight a point I made in this initial post. If you're adding a new executable_test, we run the test both with and without -enable-sil-opaque-values in CI as of this change. So it's important to look in the CI logs for the run configuration under which a failure has occurred. For example, in the current macOS x86_64 smoke test bot's logs, you can look for the following strings right after the testing summary:

  • check-swift-validation-macosx-x86_64 finished => Regular Run
  • check-swift-validation-optimize_none_with_opaque_values-macosx-x86_64 =>Opaque Values Run

The intention behind the additional run is to remind people who are adding new features/changes to the compiler that their work should strive to be compatible with opaque values.

If you run into a failure only under the opaque values run, please check to see whether it is related to your compiler changes. You can run the build target check-swift-optimize_none_with_opaque_values locally to reproduce the CI failure. For example, if say AddressLowering or SILGen is crashing on your new SIL instruction, then you should add opaque values support for it in your PR.

If after investigating, you see it is a pre-existing issue or unrelated to your changes, please keep your test as-is and simply add // XFAIL: swift_test_mode_optimize_none_with_opaque_values . This is the preferred action, marking it as "to be fixed" under that run. An XFAIL keeps it running in CI under opaque values, helping those working on it know the impact of their bug fix if they cause your test to unexpectedly pass. When that does happen, it signals that there is already some test coverage available, helping avoid redundant test additions. Plus, a fix can sometimes untangle issues for an unrelated language feature, and it's good to have free test coverage for that turned on by simply removing the XFAIL.

2 Likes