For builds with remotely built artifacts, there may be another short term solution. If the linking is forced to be local, one option may be to generate an otherwise empty .swiftmodule, but have it built with both -serialize-debugging-options and the necessary search paths and flags needed for debugging.
I believe target.swift-extra-clang-flags -I/my/local/includes should work if you include the modulemap directories. Are you asking for a solution before Swift 5.1?
Yes I'd like to find a solution before 5.1. Propagating flags to lldb is also something I'd hope to avoid.
Won't this special swiftmodule also need to be built locally for this to work?
Yes, it would need to be built locally. Are there issues you're seeing with that?
I hope my understanding of this isn't incorrect, but one thing we do is pass -Wl,-add_ast_path to the linker for the full set of swiftmodules compiled into the binary so that lldb can load them from there instead of relying on them being present on the filesystem. We don't do that for Clang modules because we rely on the implicit module cache, and we don't have a way of getting that cleanly from our remote build machines (and we'd have multiple copies of the same dependencies with different paths, etc...)
I'm trying to fix that by precompiling our C/Obj-C dependencies as explicit modules and propagating those in our build graph, which I hope improves our remote build performance. If I'm able to get that working, could we also pass those files to the linker via -Wl,-add_ast_path, and if so, would lldb need to be passed any additional flags at all? Wouldn't it then have all the ASTs it needs from both Swift and Clang?
If that's true, how would this extend to non-Apple platforms like Linux? With ELF, there's an extra swift -modulewrap action that produces an extra .o file that contains the swiftmodule, equivalent to what -Wl,-add_ast_path would do. I'm not sure if something similar exists for Clang PCMs.
There is no such thing as -add_ast_path for Clang PCMs. Like the Swift serialized module format, the Clang PCM format isn't stable and thus not a great match for debug info as it could only be loaded by an LLDB that has the exact same Clang compiler embedded as the one used to precompile the PCM. LLDB's reliance on serialized Swift modules is a shortcut that we need to eventually replace with something more stable or forward-compatible. The proper way to load a Clang module is to import it from the module map and rely on the Clang module cache to avoid recompiling it. Using explicit clang modules with Swift is not supported (though I suppose it could work to some degree).
Note that the PCMs generate by the Swift compiler are compiled using -gmodules, which produces a PCM file that is a Mach-O container with a .clang_ast section and several debug info sections containing DWARF type information for a types defined in the module. The Swift compiler records breadcrumbs pointing to the imported PCMs such that dsymutil and LLDB can collect the DWARF type definitions from the PCMs. This allows LLDB to synthesize Clang type definitions from DWARF when the Clang module failed to import from sources (Comment the complex interaction between Swift and DWARFimporterDelegate. by adrian-prantl · Pull Request #1923 · apple/swift-lldb · GitHub). This is meant to be a fallback mechanism and I'm not saying you should rely on this, but you also should try not break it. Dsymutil will print warnings about missing PCMs if you accidentally break it.
I'm curious to learn more about the problems you are running into with building Clang modules on the remote machines. Is it just the include path mismatch between local and remote file system? You may need to either copy back the modules in the remote module cache or run dsymutil remotely to not break the flow I described above.
1 Like
Yes, that's exactly the problem we have if we try to use -gmodules today; since the paths in the breadcrumbs don't match because each machine has a module cache in a different location, we end up with a bunch of warnings about the modules not being present at link time, IIRC.
This is precisely the problem I'm attempting to solve. Since it turns out that the version of clang linked into swiftc isn't always the same version as the clang binary distributed in the same Xcode toolchain (there's usually a slight revision number drift), and matching up the exact invocation with ClangImporter internally would be difficult anyway, my goal is to implement a new swift driver mode that I can invoke and have it emit a PCM using the same clang invocation that it would later use to import it. I posted about it in some more detail a while back here—I need to update that thread since I've made slightly more progress recently.
So, my hope was that if I have a dependency graph of mixed Swift and Objective-C code, and my PCMs are explicit dependencies, then those breadcrumbs are now paths that are guaranteed to be valid (and remapped via -debug-prefix-map?) and LLDB would have enough information to reconstruct types without having to rely on knowing the full set of include paths and other flags that were present during the build. Since many of our projects are broken up into hundreds of modules, that set of flags becomes rather large.
As a follow up, I've been able to get lldb to evaluate expressions with a build that uses:
-no-serialize-debugging-options- Xcode 11
- lldb 5.1 target settings
However, the surprise was I didn't need target.swift-extra-clang-flags.
In our case we have some precompiled static frameworks. The only lldb settings I have to set are:
target.sdk-pathtarget.swift-framework-search-paths
In swift-framework-search-paths, I included precompiled ObjC frameworks. Those frameworks have module maps, but no swiftmodules. This is somehow working, without setting swift-extra-clang-flags. This doesn't work with Xcode 10.3 and lldb 5.0.
Follow up: It turns out this isn't enough. lldb produces no errors with this config, but silently fails to resolve some types. This means some variables it can print (some expressions it can eval), but some variables are missing the type. I've posted a question with details on another thread that seems more related: Xcode could not resolve type from dSYM if build folder is different - #19 by Dave_Lee
When you are building with Bazel remotely, does the link step also happen remotely?
I didn't see this comment earlier, but with Bazel we link locally, specifically so that the -add_ast_path values are valid local paths.
Even if LLDB was immune to all these failures due to absolute paths to another computer that built the binary you're using, you're forgetting about somethin'.
Playgrounds get wrecked by absolute paths to other machines.
Hi all. We have similar requirements for getting debuggable Buck builds working with remotely built static libraries, and I would be interested in coming to a solution for this that did not involve custom lldbinit settings.
Previously mentioned was the idea of applying flag remapping logic to the serialized debugging info in the swift compiler so that debug prefixes were applied there too. I have a working proof of concept for this that more or less takes the code from https://github.com/apple/llvm-project/blob/swift/main/lldb/source/Plugins/TypeSystem/Swift/SwiftASTContext.cpp#L1388 and applies it to the clang flags and search paths when serializing swiftmodule files. Before I get around to cleaning this up for review, it would be good to know if there is general opposition to this approach, or if there is a better path to getting this working now?
Or better yet, how about just include all-product-headers.yaml in the DSYM instead of having it referenced externally? Like... why do we need to reference some external file? Why cant a DSYM just contain everything it needs? I don't get it.
Also, I've found that locally building all the dependency frameworks from source, clearing DerivedData and all other conceivable caches, quitting Xcode and force-quitting all simulators while simultaneously deleting all their caches, etc. do not get rid of the foreign paths. No matter what I do, LLDB still fails due to some foreign path that doesn't exist anywhere on my computer that I can find. It's as if LLDB lives somewhere in RAM and has secretly cached this path in some encrypted cache somewhere, which it will always use as the source of truth no matter what is in DerivedData.
I tried deleting all the following paths to no avail:
/Users/REDACTED/Library/Caches/BuildService
/Users/REDACTED/Library/Caches/prebuild
/Users/REDACTED/Library/Caches/com.apple.dt.Xcode
/Users/REDACTED/Library/Caches/org.carthage.CarthageKit
/Users/REDACTED/Library/Developer
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/clang
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.CoreSimulator.CoreSimulatorService
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.CoreSimulator.simctl
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.CoreSimulator.SimRenderingServices.SimMetalHost
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.CoreSimulator.SimRenderingServices.SimRenderServer
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.CoreSimulator.SimStreamProcessorServices.SimStreamProcessorService
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.DeveloperTools
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.dt.CommandLineTools.installondemand
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.dt.IDECacheDeleteAppExtension
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.dt.Instruments
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.dt.Xcode
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.dt.xcodebuild
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/com.apple.iphonesimulator
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/org.llvm.clang
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/C/org.llvm.clang.502
/private/var/folders/w7/wtfbbq_omg1337x_whythisnotworkx/T/lldb
/private/tmp/Developer