Are there notes (or docs) on how to use a nightly development snapshot with projects (swift-distributed-actors)?

Joseph_Heck · October 30, 2021, 12:50am

I wanted to compile and generally explore the new swift-distributed-actors code, on my Mac laptop with Xcode (13.1 - but not the latest beta). I downloaded the nightly toolchain and set it as the toolchain to use within Xcode, but when I open Package.swift, I'm getting a failure reporting that it's still thinking it is using Swift 5.5:

package at '/Users/heckj/src/swift-distributed-actors' is using Swift tools version 5.6.0 but the installed version is 5.5.0

I'm guessing there's something obvious that I'm missing here, and I scanned around, but haven't spotted it. What's the trick to using Xcode with the nightly snapshot when opening up the Package.swift within Xcode? I seem to recall some previous mechanics that involved setting an environment variable. Is this noted somewhere that I should be able to find?

cc /@ktoso

joe

ktoso · October 30, 2021, 4:56am

Hi Joe, thanks for digging into it!

I just realized there isn’t really an Xcode which will be able to open up any 5.6-dev project just yet. Sadly there’s no real workaround for that per se.

I’ll look into workarounds to be able to edit the project in Xcode though — similar to the _Concurrency library in its early days there is some limitations with how Xcode and custom toolchains interact. As we currently also need SwiftPM plugins, whose APIs changed between 5.5 and 5.6 (since they were not a stable feature), this project indeed has a painful build story right now… we’ll be able to remove the plugins though, as soon as we gain replacement features (to “make a message”) in the language.

——

Workaround ideas until then… (sorry I’m away from a computer now)

I think it should be possible to remove all mentions of plugins in the Package.swift, and then change it to require swift 5.5. You may need to add skipping availability checks to the unsafe options (where we enable the distributed feature). And have Xcode at least import then.

You could try using a different editor like CLion or just VSCode. CLion should be able to import the project if the .plugin bits are removed (I tried that), though it won’t know about the new keywords and freak out a little bit (formatting might break).

Sadly we’re depending on a lot of bleeding edge features here and the IDE support is lagging still…

—-

You should be able to build the project using swiftPM though and that’s what the team has been using successfully. There is some guidance about this in the README — let me know if something doesn’t work, and we’ll figure it out;

Sorry for the rough editing experience; sadly depending on so many in-flight unreleased features makes the development experience a bit painful (for us as well, we’re in the same boat).

I’ll dig more into it and update the readme as soon as I find more ideas

slashmo · November 1, 2021, 10:44am

Hi folks! For anyone else looking into building the swift-distributed-actors library, I found the following to work for me, but only using CLion:

Set the Swift toolchain in CLion to the latest nightly one (for me that was 10-28)
Remove the .plugin from Package.swift in some other editor (NOT CLion)
Open the project in CLion, resolve the Swift package dependencies
Close CLion
Re-add the .plugin to Package.swift
Re-open the project in CLion, but DO NOT resolve Swift packages within CLion
Add DYLD_LIBRARY_PATH to the run configuration
Build/Run tests

I suspect the reason this works is that CLion stores that it already resolved the dependencies inside the .idea folder. Additionally, CLion invokes the build command with the --skip-update flag.

Joseph_Heck · November 1, 2021, 10:51pm

I've been having a terrible time getting the DYLD_LIBRARY_PATH to take effect to link in the correct _Distributed module while on macOS. I have my toolchain set to the nightly available as of the 28th:

export TOOLCHAIN=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-10-28-a.xctoolchain

But running the tests is failing to link up the right version.
DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" $TOOLCHAIN/usr/bin/swift test results in:

[0/0] Build complete!
2021-11-01 15:44:17.321 xctest[47710:352127] The bundle “swift-distributed-actorsPackageTests.xctest” couldn’t be loaded. Try reinstalling the bundle.
2021-11-01 15:44:17.321 xctest[47710:352127] (dlopen(/Users/heckj/src/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests, 0x0109): Library not loaded: /usr/lib/swift/libswift_Distributed.dylib
  Referenced from: /Users/heckj/src/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests
  Reason: tried: '/usr/lib/swift/libswift_Distributed.dylib' (no such file), '/usr/local/lib/libswift_Distributed.dylib' (no such file), '/usr/lib/libswift_Distributed.dylib' (no such file))

I tried some variations (DYLD_FALLBACK_FRAMEWORK_PATH) that I spotted in man dyld, but with the same result. I haven't dropped and punted to invoking via Linux and Docker (yet) - and I double checked the linkage of the resulting bundle with otool -L and that the relevant dylib (libswift_Distributed.dylib) was in the toolchain that I downloaded.

Any suggestions on how to further debug what's happening (or not happening, as the case may be) with the dynamic library loading stuff?

The README calls out the toolchain from the 26th - do I need to hunt down and download that specific nightly version in order to use this? (EDIT: apparently not. I grabbed the toolchain from the 26th and followed the examples from the README verbatim, and I'm getting the same errors.)

ktoso · November 2, 2021, 12:28am

Hi Joseph, I'm trying to get some more hints from people who might know...

In the meantime I am wondering if this isn't similar to a different issue we saw, and the solution was to disable System Integrity Protection. I think Moritz has that disabled if I remember correctly, and if it works for him...

Joseph_Heck · November 2, 2021, 12:31am

Thanks - I'll give it a shot and report back. I'm also going to spin up Docker and get that working as an alternative to at least be able to see the various moving parts in action.

ktoso · November 2, 2021, 12:31am

Yeah docker definitely works as a fallback — our CI working fine after all.

ktoso · November 2, 2021, 2:16am

I wiggled around this a bit more -- can you share and confirm the exact invocations and errors you are getting?

A plain swift test invocation won't work; this it explained in the readme:

-> % DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" $TOOLCHAIN/usr/bin/swift test
[0/0] Build complete!
2021-11-02 11:13:08.352 xctest[20156:4340969] The bundle "swift-distributed-actorsPackageTests.xctest" couldn't be loaded. Try reinstalling the bundle.
2021-11-02 11:13:08.352 xctest[20156:4340969] (dlopen(/Users/ktoso/code/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests, 0x0109): Library not loaded: /usr/lib/swift/libswift_Distributed.dylib
  Referenced from: /Users/ktoso/code/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests
  Reason: tried: '/usr/lib/swift/libswift_Distributed.dylib' (no such file), '/usr/local/lib/libswift_Distributed.dylib' (no such file), '/usr/lib/libswift_Distributed.dylib' (no such file))

Instead, you have to do this dance: $TOOLCHAIN/usr/bin/swift build --build-tests && DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" xctest -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest - it boils down to security policies and the DYLD_ prefixed env variables being stripepd out when creating sub-processes, which is the case for swiftpm launching xctest.

ktoso@mikazukplusplus [11:13:08] [~/code/swift-distributed-actors] [main]
-> % clear; echo "TOOLCHAIN=$TOOLCHAIN"; echo; $TOOLCHAIN/usr/bin/swift build --build-tests && DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" xctest -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest
TOOLCHAIN=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-10-27-a.xctoolchain

[0/0] Build complete!
Test Suite 'Selected tests' started at 2021-11-02 11:13:21.698
Test Suite 'swift-distributed-actorsPackageTests.xctest' started at 2021-11-02 11:13:21.700
Test Suite 'DistributedReceptionistTests' started at 2021-11-02 11:13:21.700
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_mustHaveWellKnownAddress]' started.
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_mustHaveWellKnownAddress]' passed (0.092 seconds).
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithEmptyRefForUnknownKey]' started.
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithEmptyRefForUnknownKey]' passed (0.009 seconds).
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithRegisteredRefsForKey]' started.
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithRegisteredRefsForKey]' passed (0.006 seconds).
Test Suite 'DistributedReceptionistTests' passed at 2021-11-02 11:13:21.808.
	 Executed 3 tests, with 0 failures (0 unexpected) in 0.106 (0.107) seconds
Test Suite 'swift-distributed-actorsPackageTests.xctest' passed at 2021-11-02 11:13:21.808.
	 Executed 3 tests, with 0 failures (0 unexpected) in 0.106 (0.108) seconds
Test Suite 'Selected tests' passed at 2021-11-02 11:13:21.808.
	 Executed 3 tests, with 0 failures (0 unexpected) in 0.106 (0.110) seconds

Joseph_Heck · November 2, 2021, 2:56am

Ah - so the DYLD_LIBRARY_PATH dance is needed for running any tests, not just targeting an individual test to run?

I was following along the README - the earliest section above that (swift-distributed-actors/README.md at main · apple/swift-distributed-actors · GitHub), effectively verbatim:

export TOOLCHAIN=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-10-28-a.xctoolchain
$TOOLCHAIN/usr/bin/swift build --build-tests

(warnings and such, but the end result was [1210/1210] Build complete!. Then tried running all the tests:

$TOOLCHAIN/usr/bin/swift test

Which resulted in the errors, identical to what you showed above. When I tried to replicate your xctest command that prefixed things with the DYLD_LIBRARY_PATH addition, I get a path issue - unable to find xctest:

$TOOLCHAIN/usr/bin/swift build --build-tests && DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" xctest -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest
[0/0] Build complete!
zsh: command not found: xctest

If I prefix things with xcrun xctest, it invokes an instance of xctest from within Xcode for me locally. I did some searches (find), and it doesn't look like xctest is included within the standard macOS toolchain, and defaults using the xctest from Xcode has the same underlying dylib issues.

I tried using swift-test directly as well - and not surprisingly, it resulted in the same issue I saw previously. I'd hoped that I could somehow pass in the DYLD_LIBRARY_PATH in such a way that the swift test commands would utilize it while running whatever it does under the covers.

If this isn't a path that works, we might want to change up some of the README a smidge so that's a touch more clear. In particular swift-distributed-actors/README.md at main · apple/swift-distributed-actors · GitHub

Normal `swift build` and `swift test` invocations should work fine without the environment variable.

If there's a way to pass in DYLD_LIBRARY_PATH to run all the swift tests together, I'm more than happy to use it. Or likewise, if there's something I should have done to get and compile xctest locally for direct usage, I'm totally game. Closest I found in the toolchain was $TOOLCHAIN/usr/libexec/swift/pm/swiftpm-xctest-helper

ktoso · November 2, 2021, 4:32am

Thanks for bearing with me as we figure out this process to build @Joseph_Heck.

I'll make sure to improve the readme based on our findings here. Sorry that you have to suffer through this, but at least we'll make it easier for others.

Right yes, you need to pass this when executing programs that use the distributed things.

The readme shows running samples:

echo "TOOLCHAIN=$TOOLCHAIN"

DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" \
  $TOOLCHAIN/usr/bin/swift run \
  --package-path Samples \
  SampleDiningPhilosophers dist

and I think I should rename this "running filtered tests" to just running tests, the readme does show the latter...

Especially thanks for

It seems you're right, and just my specific setup seems to be lucky / configured such that the DYLD env var works in that setup...

It seems we should be able to do the following though, would you be able to confirm on your setup? (I'm starting to doubt my own environment, and don't really have a clean setup I could verify this on).

// 1 find xctest
-> % find /Applications/Xcode-XXX.app  | grep xctest
/Applications/Xcode-XXX.app/Contents/Developer/usr/bin/xctest

// 2. run the xctest bundle
$TOOLCHAIN/usr/bin/swift build --build-tests && \
DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" \ 
  /Applications/Xcode\ Customer.app/Contents/Developer/usr/bin/xctest \
 .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

or

$TOOLCHAIN/usr/bin/swift build --build-tests && \
DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" \
  /Applications/Xcode\ Customer.app/Contents/Developer/usr/bin/xctest \
  -XCTest "DistributedActorsTests.DistributedReceptionistTests" \
 .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

to run a specific test class.

I believe this should work... could you verify @Joseph_Heck @slashmo ?

Joseph_Heck · November 2, 2021, 4:53am

No problem!

Here's the detail:

find /Applications/Xcode.app | grep xctest resulted in:

/Applications/Xcode.app/Contents/Developer/usr/bin/xctest

export TOOLCHAIN=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-10-28-a.xctoolchain
export XCTEST=/Applications/Xcode.app/Contents/Developer/usr/bin/xctest

➜  swift-distributed-actors git:(main) echo $XCTEST
/Applications/Xcode.app/Contents/Developer/usr/bin/xctest
➜  swift-distributed-actors git:(main) echo $TOOLCHAIN
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-10-28-a.xctoolchain

Then I ran:

$TOOLCHAIN/usr/bin/swift build --build-tests && $XCTEST .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

The result of which was a failure linking to the dispatch library:

[0/0] Build complete!
2021-11-01 21:42:01.713 xctest[4792:35302] The bundle “swift-distributed-actorsPackageTests.xctest” couldn’t be loaded. Try reinstalling the bundle.
2021-11-01 21:42:01.713 xctest[4792:35302] (dlopen(/Users/heckj/src/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests, 0x0109): Library not loaded: /usr/lib/swift/libswift_Distributed.dylib
  Referenced from: /Users/heckj/src/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests
  Reason: tried: '/usr/lib/swift/libswift_Distributed.dylib' (no such file), '/usr/local/lib/libswift_Distributed.dylib' (no such file), '/usr/lib/libswift_Distributed.dylib' (no such file))

I tried the specific version as well:

$TOOLCHAIN/usr/bin/swift build --build-tests && $XCTEST -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

but with the same results:

[0/0] Build complete!
2021-11-01 21:45:51.518 xctest[5002:38562] The bundle “swift-distributed-actorsPackageTests.xctest” couldn’t be loaded. Try reinstalling the bundle.
2021-11-01 21:45:51.518 xctest[5002:38562] (dlopen(/Users/heckj/src/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests, 0x0109): Library not loaded: /usr/lib/swift/libswift_Distributed.dylib
  Referenced from: /Users/heckj/src/swift-distributed-actors/.build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest/Contents/MacOS/swift-distributed-actorsPackageTests
  Reason: tried: '/usr/lib/swift/libswift_Distributed.dylib' (no such file), '/usr/local/lib/libswift_Distributed.dylib' (no such file), '/usr/lib/libswift_Distributed.dylib' (no such file))

My supposition is that the xctest that's included with Xcode can't or won't allow me to influence the DYLD path - possibly just while SIP is still enabled. I haven't disabled it, per your earlier suggestion - with the intel laptop that I'm working on, I'm "not supposed to", so I'll have to pass on that part of the validation effort for now.

I'll be super curious to see what @slashmo might report, if he has SIP disabled locally.

(and I'm happy to run/try further experiments with my "stock" macOS/Xcode + nightly toolchain combination)

ktoso · November 2, 2021, 5:01am

Sorry my bad, I missed the DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" in my snippets now, see above corrected.

The DYLD_LIBRARY_PATH=... must be right before the XCTEST binary invocation. That should work.

not enough: $TOOLCHAIN/usr/bin/swift build --build-tests && $XCTEST -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

->

good: $TOOLCHAIN/usr/bin/swift build --build-tests && DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" $XCTEST_BINARY -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

I'll be adjusting the readme as we go: Update readme with more info on build setup by ktoso · Pull Request #853 · apple/swift-distributed-actors · GitHub

Joseph_Heck · November 2, 2021, 5:07am

Doh! I should have caught that up front, sorry - with the addition the DYLD_LIBRARY_PATH in the middle there, right before the xctest invocation, it's functional:

➜  swift-distributed-actors git:(main) $TOOLCHAIN/usr/bin/swift build --build-tests && DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" $XCTEST_BINARY -XCTest "DistributedActorsTests.DistributedReceptionistTests" .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest
[0/0] Build complete!
Test Suite 'Selected tests' started at 2021-11-01 22:04:00.497
Test Suite 'swift-distributed-actorsPackageTests.xctest' started at 2021-11-01 22:04:00.498
Test Suite 'DistributedReceptionistTests' started at 2021-11-01 22:04:00.498
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_mustHaveWellKnownAddress]' started.
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_mustHaveWellKnownAddress]' passed (0.019 seconds).
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithEmptyRefForUnknownKey]' started.
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithEmptyRefForUnknownKey]' passed (0.008 seconds).
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithRegisteredRefsForKey]' started.
Test Case '-[DistributedActorsTests.DistributedReceptionistTests test_receptionist_shouldRespondWithRegisteredRefsForKey]' passed (0.008 seconds).
Test Suite 'DistributedReceptionistTests' passed at 2021-11-01 22:04:00.535.
	 Executed 3 tests, with 0 failures (0 unexpected) in 0.036 (0.037) seconds
Test Suite 'swift-distributed-actorsPackageTests.xctest' passed at 2021-11-01 22:04:00.535.
	 Executed 3 tests, with 0 failures (0 unexpected) in 0.036 (0.037) seconds
Test Suite 'Selected tests' passed at 2021-11-01 22:04:00.535.
	 Executed 3 tests, with 0 failures (0 unexpected) in 0.036 (0.038) seconds

And the "run 'em all!" variation:

$TOOLCHAIN/usr/bin/swift build --build-tests && DYLD_LIBRARY_PATH="$TOOLCHAIN/usr/lib/swift/macosx/" $XCTEST_BINARY .build/x86_64-apple-macosx/debug/swift-distributed-actorsPackageTests.xctest

works as well! (I'll spare you the output, it was lengthy):

Test Suite 'All tests' passed at 2021-11-01 22:08:37.266.
	 Executed 548 tests, with 3 tests skipped and 0 failures (0 unexpected) in 164.192 (164.229) seconds

Thanks - this has it all rolling for me!

ktoso · November 2, 2021, 5:09am

Wohoo, fantastic. Sorry that this was so much of a hassle… sadly we’re hitting a number of limitations and/or security things with this build…

Soon it’ll be easier though :-) thanks and I look forward to hearing from you! Ask anything you want really.

Shilpee · November 25, 2021, 6:44am

After resolving dependancies, I am getting 2 build errors. I tried searching and found that I need to add run configurations.(New to Clion)

Q)I read the package.swift file and there are many targets i.e. DistributedActors, executable targets and test targets as well. Which one do I need to add as target in the run configuration. What will be the executable for adding configuration. What other changes do I need to do in the run/debug dialog box.

I have installed the swift plugin and set the toolchains to development snapshot.
As far as the second issue is concerned, I didn't get anything on the internet. It would be helpful if you can provide some guidance.

ktoso · November 26, 2021, 7:16am

The error is about plugins which @slashmo has explained in his step 2):

Remove the .plugin from Package.swift in some other editor (NOT CLion)

You have to remove .plugin declarations from the file, since CLion and all other IDEs are not yet aware of their existence and thus can't parse the build file.

You should be able to get it going by following the steps as Moritz explained in hist post there.

Shilpee · November 26, 2021, 6:43pm

It was written NOT CLion. So I didn't tried that step. Thanks for pointing that out.

However, I tried building and each time I received error claiming " zsh: no such file ....usr/bin/swift ". I am not able to figure out which step I am missing.

I think one possible reason of getting this error is that I am exporting the older toolchain ..28 one. So I tried to export again with the development snapshot-2021-11-20. But was getting the same error