PLUGIN_SEARCH_OPTION in swiftmodule make it not cacheable

wizzardchao · March 28, 2025, 8:16am

As swift 6.x starts to use different swiftmodule for TypeSystemSwiftTypeRefForExpression when constructing swift ast context, we are required to provide all our swiftmodules to linker with -add-ast-path . with --serialize-debugging-options on.

In this case, the product swiftmodule file will have many local absolute paths in the option_blocks & input_blocks

Currently we are using bazel as our build system, which means we cannot cache these swiftmodule.

I found there is already one swift-frontend command line option -prefix-serialized-debugging-options handling this case.

github.com/swiftlang/swift

lib/Serialization/Serialization.cpp

2bb098c0b


      
          
          if (M->supportsExtensibleEnums()) {
            options_block::ExtensibleEnumsLayout ExtensibleEnums(Out);
            ExtensibleEnums.emit(ScratchRecord);
          }
          
          if (Options.SerializeOptionsForDebugging) {
            options_block::SDKPathLayout SDKPath(Out);
            options_block::XCCLayout XCC(Out);
          
            const auto &PathRemapper = Options.DebuggingOptionsPrefixMap;
            const auto &PathObfuscator = Options.PathObfuscator;
            auto sdkPath = M->getASTContext().SearchPathOpts.getSDKPath();
            SDKPath.emit(
                ScratchRecord,
                PathObfuscator.obfuscate(PathRemapper.remapPath(sdkPath)));
            auto &Opts = Options.ExtraClangOptions;
            for (auto Arg = Opts.begin(), E = Opts.end(); Arg != E; ++Arg) {
              StringRef arg(*Arg);
              if (!Options.ExplicitModuleBuild && arg.starts_with("-ivfsoverlay")) {
                // FIXME: This is a hack and calls for a better design.

github.com/swiftlang/swift

lib/Frontend/Frontend.cpp

2bb098c0b


      
          // the public.
          serializationOpts.SerializeOptionsForDebugging =
              opts.SerializeOptionsForDebugging.value_or(
                  !module->isExternallyConsumed());
          
          serializationOpts.PathObfuscator = opts.serializedPathObfuscator;
          if (serializationOpts.SerializeOptionsForDebugging &&
              opts.DebugPrefixSerializedDebuggingOptions) {
            serializationOpts.DebuggingOptionsPrefixMap =
                getIRGenOptions().DebugPrefixMap;
            auto &remapper = serializationOpts.DebuggingOptionsPrefixMap;
            auto remapClangPaths = [&remapper](StringRef path) {
              return remapper.remapPath(path);
            };
            serializationOpts.ExtraClangOptions =
                getClangImporterOptions().getRemappedExtraArgs(remapClangPaths);
          } else {
            serializationOpts.ExtraClangOptions = getClangImporterOptions().ExtraArgs;
          }
          if (LangOpts.ClangTarget &&
              !getClangImporterOptions().DirectClangCC1ModuleBuild) {

I think with this option we can map all frontend parameters related to bazel root andbox. We can recover them if needed from lldb side.

Unfortunately, it only handles SDK_NAME, XCC, SEARCH_PATH. The PLUGIN_SEARCH_OPTION related ones are untouched, which makes it not possible to make the swiftmodule cacheable.
And I just checked lldb source code, lldb only use the information in swiftmodule for ClangImporter, these PLUGIN_SEARCH_OPTIONs will not even be used.

Can we make the PLUGIN_SEARCH_OPTION also be mapped using the same way?

cachemeifyoucan · April 2, 2025, 6:29pm

Thanks for bringing up the concern for swift binary module's cacheability. I think it makes sense to provide an option to prefix mapped the plugin search that goes into binary swift module, and extending -prefix-serialized-debugging-options to handle that seems reasonable.

On the other hand, LLDB does read the plugin search options from the swift binary module to set up its ASTContext so the expression evaluator can handle swift macros. We do need to teach LLDB to map the path back in order for debugging to work. There are various layers of path remapping in LLDB (e.g. target.source-map), and we need to teach LLDB to remap plugin search options when it creates swift ASTContext.

After all the changes, user can use the swift-frontend flag together with lldb options to achieve cacheable swift module and also not sacrificing debugging capability. Can you file an issue to keep track of the problem?

wizzardchao · April 3, 2025, 6:42am

I saw you have been already working on it here. https://github.com/swiftlang/swift/pull/80474
Is it all the changes from module generating side? Just wondering why we only do the extra mapping for only PluginSearchOption::Kind::ResolvedPluginConfig but not for other PluginSearchOption enum.

Also I checked the lldb code when reconstructing swift ast context:

github.com/swiftlang/llvm-project

lldb/source/Plugins/TypeSystem/Swift/SwiftASTContext.cpp

d71e0a757


      
          if (exists(plugin))

For ResolvedPluginConfig,LoadPluginExecutable and the 2nd part of ExternalPluginPath (an absolute path to a swift-plugin-server inside xcode set as xcode default), the paths are read directly while other option kinds ( LoadPluginLibrary/PluginPath) are mapped with swift-plugin-server-for-path lldb setting option.

Will it work as expected after making plugin paths in swiftmodule relative from compiler change you made?
I am not sure if I missed something, do you know if any following changes in lldb should be made ?

cachemeifyoucan · April 3, 2025, 4:39pm

If I were you, I will not rely on my PR to handle Bazel. My patch is about the CAS based swift caching using explicit module build, and it cannot be used in Bazel without someone who can do a proper integration. Bazel, as it is right now, needs a different path remapping option in compiler.

However, I think both can utilize the same path remapping function in LLDB. I think you found the correct place for reading the option from module and there is a remap function in that scope and we can just use that to achieve what we need. @Adrian_Prantl is the expert here.