Lldb is slow to resolve local vars

beefon · March 13, 2020, 11:30pm

Thanks Adrian again. I've used a bad terminology or incorrectly described what I did.

Why are you removing module map files?

I'm removing modulemap files generated by CocoaPods for me. Turns out if I remove them and let Xcode generate default ones, lldb works faster. For now I do this for frameworks that only contain Swift code though.

CocoaPods modulemap:

framework module Gallery {
  umbrella header "Gallery-umbrella.h"

  export *
  module * { export * }
}

module Gallery.Swift {
    header "Gallery-Swift.h"
    requires objc
}

I delete this ^^ and let Xcode generate a default modulemap which looks like:

framework module Gallery {
    header "Gallery-Swift.h"
    requires objc
}

Again, I do this for 171 Swift frameworks. This speeds up lldb quite a bit.

should strive to just make everything a Clang module

Again thanks for this, I've eliminated bridging header entirely. Now there are only clang modules. This gave ~11 seconds improvement (61 ses -> ~50).

Now, back to CocoaPods' modulemaps. After I've deleted them, lldb perf has improved: 50 sec -> 36 sec.

Since the only difference I spotted is this part:

framework module Gallery {
  umbrella header "Gallery-umbrella.h"

  export *
  module * { export * }
}

, and because -umbrella.h headers contain @import UIKit; wrapped with some #if-s, I decided that this is what causes slowdowns of lldb. Is this a correct theory?

I do not understand what you mean with "unwrap" here.

I thought that lldb would preprocess #imports like clang does (expand them recursivaly to a megabytes of text). My tests showed this is not a case, or at least this is not a bottleneck at all, and you confirmed that I shouldn't be doing this.

Adrian_Prantl · March 13, 2020, 11:53pm

I do not understand what you mean with "unwrap" here.

I thought that lldb would preprocess #import s like clang does (expand them recursivaly to a megabytes of text). My tests showed this is not a case, or at least this is not a bottleneck at all, and you confirmed that I shouldn't be doing this.

To be clear, LLDB doesn't process header files directly at all. LLDB embeds Clang and asks it to import modules and Clang does the same thing it does during normal compilation. Unless Clang is somehow in PCH mode, there should not be a difference between @import and #import. So there might be something I'm missing here and I'd like to understand that. Do you also see compile-time improvements with your changes, too, or does it get faster only during debug time?

Finally, are you using 10.3 or the latest 10.4 beta? Does that make a difference?

beefon · March 14, 2020, 9:20am

Do you also see compile-time improvements with your changes, too, or does it get faster only during debug time?

No difference in clean build time at all. Only debugger speed improvement.

Finally, are you using 10.3 or the latest 10.4 beta?

This is on Xcode 11.3.1. Can't confirm now on latest beta, project is not buildable in Xcode 11.4. I will try to make it work and report back.

beefon · March 15, 2020, 2:19pm

Xcode 11.4b3 is always shows "Internal error" banner when lldb pauses on a breakpoint. After a while lldb-rpc-server crashes. So I'm unable to check for lldb speed.

Adrian_Prantl · March 16, 2020, 3:31pm

Can you please send me a crash log for lldb-rpc-server? Either attach it here or send a feedback ticket through feedback.apple.com?

beefon · March 16, 2020, 3:36pm

Both SourceKitService and lldb-rpc-server crashed in 11.4b3:

Adrian_Prantl · March 16, 2020, 3:58pm

Thanks you! I have not seen this kind of crash before. It's happening while deserializing a Clang module. If things are working correctly this shouldn't help, but I wonder if this also reproduces after you completely erase the DerivedData directory and rebuild.

beefon · March 16, 2020, 3:59pm

Definitely reproducible after erasing derived data.

Adrian_Prantl · March 17, 2020, 3:46pm

Is there any chance you could share a project with me that reproduces the crash?

beefon · March 17, 2020, 5:38pm

I can't share my project unfortunately, and I wasn't able generate a sample project to reproduce this crash.

beefon · March 24, 2020, 8:01am

Our CI measurements show there IS some build time improvements after I iteratively removed modulemap files generated by CocoaPods:

All three measurements do build main app and its dependencies.

Legend: unit_tests are multiple unit test bundles (per module/framework), functional_tests - single UI test bundle, component_tests - single application test bundle.

Jon_Shier · March 24, 2020, 4:05pm

Can't seem to load that image. Could you summarize with a table?

beefon · March 24, 2020, 4:07pm

Jon_Shier · March 24, 2020, 4:09pm

Odd, seems to load now.

In any event, were you able to remove such models automatically using CocoaPods, or did you do it manually in some way that prevented CocoaPods from regenerating them?

beefon · March 24, 2020, 4:11pm

We've added a hook using CocoaPods Podfile APIs, so this patching happens as part of pod install and takes ~1 second.

Jon_Shier · March 24, 2020, 4:50pm

Mind sharing your hook?

zack2012 · March 25, 2020, 2:24pm

Hi @Dave_Lee, How can I disable CarbonBlack?

beefon · March 25, 2020, 2:48pm

I can't share sources, but the idea is the following:

Subscribe to post_install in Podfile
Patch build settings for both top level and dependency targets, something like that:

def patch_build_flags(installer)
  installer.pbx_pod_targets.each do |pbx_pod_target|
    pbx_pod_target.xcbuild_configurations.each do |xcbuild_configuration|
      optimize_support_files(xcbuild_configuration.build_settings)
    end
  end

  installer.aggregate_targets.each do |aggregate_target|
    aggregate_target.each_xcconfig do |xcconfig, build_configuration|
      optimize_support_files(xcconfig.attributes)
    end
  end
end

def optimize_support_files(settings)
    settings['EXCLUDED_SOURCE_FILE_NAMES'] = '*-dummy.m'

    modulemap_file_relative_path = settings['MODULEMAP_FILE']
    if modulemap_file_relative_path != nil
        modulemap_file_relative_path = "#{repo_root}/<YourProjectName>/Pods/#{modulemap_file_relative_path}"
        target_support_files_path = File.expand_path("..", modulemap_file_relative_path)
        basename = File.basename(modulemap_file_relative_path, File.extname(modulemap_file_relative_path))

        umbrella_header_path = "#{target_support_files_path}/#{basename}-umbrella.h"
        update_uikit_import(umbrella_header_path) if File.file?(umbrella_header_path)

        @pods_with_disabled_modulemap = @pods_with_disabled_modulemap || []
        if @pods_with_disabled_modulemap.include?(basename)
            log "[build_settings_modification]: Removing modulemap for #{basename}"
            erase_file_contents(modulemap_file_relative_path)
            settings.delete('MODULEMAP_FILE')
        end
        
        @pods_modulemap_files = @pods_modulemap_files || {}
        if @pods_modulemap_files[basename]
            modulemap_path = "../#{@pods_modulemap_files[basename]}"
            settings['MODULEMAP_FILE'] = modulemap_path
            log "[build_settings_modification]: Using #{modulemap_path} for #{basename}"
        end
    end
end

We've also developed some syntax sugar around pods in Podfile for explicitness:

configure 'DeepLinks', { :path => '../', :disable_modulemap => true } # removes modulemap file entirely
configure 'Toolkit', { :path => '../', :testspecs => ['UnitTests'], :modulemap_file => '../Toolkit/Toolkit.modulemap' } # use provided modulemap file

We store modulemap settings per pod in global variable (thanks Ruby) and then modify file system state in post install.

ffittschen · March 30, 2020, 2:23pm

We also faced the same issue recently and the measures described in this thread really helped us!

A bit of context: We are using Carthage to build our dependencies and as described by @Adrian_Prantl in this comment, there were a lot of .swiftinterface files in the dSYMs. After deleting all of those, resolving local variables in a breakpoint is almost instant again.

I could also verify that the .swiftinterface files are not part of the dSYMs anymore with the Xcode 11.4 release version.

But to clarify again: Am I right with the assumption that only .swiftinterface files within dSYMs should be deleted, since the .swiftinterface within frameworks are required for module stability, correct?

Adrian_Prantl · March 30, 2020, 5:20pm

Dsymutil is supposed to copy swiftinterface files into the .dSYM bundle to make the types in those interfaces available to the debugger. In dsymutil from Xcode 11.0 — 11.3 there was a bug that also copied swiftinterface files for modules in the SDK into the dSYM bundles. LLDB prefers the info from the .dSYM over the SDK, so it always loaded these. Importing modules from textual interfaces is slow, because they need to be recompiled. The SDK has a prebuilt module cache for the modules it provides, loading prebuilt binary modules is a lot faster. Dsymutil in Xcode 11.4 avoids copying system interfaces into the .dSYM bundles, so LLDB now always loads the prebuilt modules from the SDK. You still want textual interfaces for any non-SDK modules in the .dSYM.

If you are using Xcode 11.4+ no workaround regarding .swiftinterface files is necessary whatsoever.

I'm hope that "deleting your .swiftinterface" files will not become part of some cargo-cult instructions circulating on the internet

let me know if you have more questions!

-- adrian