1

Problem:
The problem is that the debugger does not see the types from the context.
Debug panel does not display types, only names.
At the same time, breakpoint work, dSYM loaded (did it by hand - add-dsym ) and symbols exist.

(lldb) po self
error: <EXPR>:3:1: error: use of unresolved identifier 'self'
self
^~~~
(lldb) fr variable
self = <could not resolve type>
(lldb) image lookup -r -n DevPodViewController*.
10 matches found in /Users/user/DD/DebugProblemApp-guvizewcykoobqfyyzhzzcggtkeq/Build/Products/Debug-iphonesimulator/DebugProblemApp.app/Frameworks/DevPod.framework/DevPod:
        Address: DevPod[0x0000000000001820] (DevPod.__TEXT.__text + 0)
        Summary: DevPod`DevPod.DevPodViewController.viewDidLoad() -> () at DevPodViewController.swift:5        Address: DevPod[0x00000000000019a0] (DevPod.__TEXT.__text + 384)
        Summary: DevPod`type metadata accessor for DevPod.DevPodViewController at <compiler-generated>        Address: DevPod[0x00000000000019f0] (DevPod.__TEXT.__text + 464)
        Summary: DevPod`@objc DevPod.DevPodViewController.viewDidLoad() -> () at <compiler-generated>        Address: DevPod[0x0000000000001a30] (DevPod.__TEXT.__text + 528)
        Summary: DevPod`DevPod.DevPodViewController.__allocating_init(nibName: Swift.Optional<Swift.String>, bundle: Swift.Optional<__C.NSBundle>) -> DevPod.DevPodViewController at DevPodViewController.swift:3        Address: DevPod[0x0000000000001a80] (DevPod.__TEXT.__text + 608)
        Summary: DevPod`DevPod.DevPodViewController.init(nibName: Swift.Optional<Swift.String>, bundle: Swift.Optional<__C.NSBundle>) -> DevPod.DevPodViewController at DevPodViewController.swift:3        Address: DevPod[0x0000000000001c10] (DevPod.__TEXT.__text + 1008)
        Summary: DevPod`@objc DevPod.DevPodViewController.init(nibName: Swift.Optional<Swift.String>, bundle: Swift.Optional<__C.NSBundle>) -> DevPod.DevPodViewController at <compiler-generated>        Address: DevPod[0x0000000000001cc0] (DevPod.__TEXT.__text + 1184)
        Summary: DevPod`DevPod.DevPodViewController.__allocating_init(coder: __C.NSCoder) -> Swift.Optional<DevPod.DevPodViewController> at DevPodViewController.swift:3        Address: DevPod[0x0000000000001d00] (DevPod.__TEXT.__text + 1248)
        Summary: DevPod`DevPod.DevPodViewController.init(coder: __C.NSCoder) -> Swift.Optional<DevPod.DevPodViewController> at DevPodViewController.swift:3        Address: DevPod[0x0000000000001dd0] (DevPod.__TEXT.__text + 1456)
        Summary: DevPod`@objc DevPod.DevPodViewController.init(coder: __C.NSCoder) -> Swift.Optional<DevPod.DevPodViewController> at <compiler-generated>        Address: DevPod[0x0000000000001e10] (DevPod.__TEXT.__text + 1520)
        Summary: DevPod`DevPod.DevPodViewController.__deallocating_deinit at DevPodViewController.swift:3

And I can even create this type

(lldb) po DevPodViewController()
<DevPod.DevPodViewController: 0x7fbdb5d1b690>
(lldb) po DevPodViewController.self
DevPod.DevPodViewController

Screen:

How came across a problem:
I am trying to implement solutions to reuse prebuilt dynamic frameworks.
I came across a problem with debug and learned how to consistently reproduce it using a simpler example.

More about the demo project:
I have a project with iOS application target.
The target has a dependency on the dynamic framework in which the controller is located.
This is a framework I want to debug. There is a configuration file - Config.xcconfig.
This file says that you need to link with the framework and where to look for this framework.

FRAMEWORK_SEARCH_PATHS = $(inherited) "${CONFIGURATION_BUILD_DIR}/DevPod"
OTHER_LDFLAGS = $(inherited) -framework "DevPod"
OTHER_CFLAGS = $(inherited) -iquote "${CONFIGURATION_BUILD_DIR}/DevPod/DevPod.framework/Headers"
CONFIGURATION_BUILD_DIR = ${BUILD_DIR}/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME)

There is also a shell build step inside project file that copies the framework and dSYM from the path I specified to the folder where it should be, according to the configuration file. (For linking)
And one more build step that move framework from the build folder inside the application. ( inside .app for load )
I also have another target in the same project. It is needed to build the framework.
The main target does not depend on the framework in the project settings only through the configuration file. ( It is important )

Steps to reproduce:

  1. Open framework project - DebugProblemApp.xcodeproj
  2. Choose a scheme to build the framework - DevPod
  3. Built it for iOS simulator
  4. Then copy the results framework and dSYM from Dereived Data in the DevPodFramework folder.
  5. Switch to main scheme - DebugProblemApp
  6. Add breakpoint to DevPodViewController file - DevPod/DevPodViewController.swift line 7
  7. Run
  8. Check that everything works - the variable (self) is displayed // or it may not work already
  9. Delete everything from Derived Data folder
  10. Run again
  11. Check that variable information is no longer displayed

Result:
Debug panel does not display types, only names.

Additional Information:
In this case, the path to the source files (the files which the framework originated from) was not changed (equal).
If the framework was from an Objective C files, debug would work. (Verified)

Off-topic: In the final version of the tool ( when the source path does not match ), I was going to use dSYM patching through a plist with the key DBGBuildSourcePath. This method works, except for the problem described above.

Link to demo project.

3 Likes
2

@Adrian_Prantl knows the dSYM code best.

3

I tried to follow your steps, but I couldn't reproduce the issue you are seeing. Debugging with dSYMs and a completely erased DerivedData folder I can still debug the program fine.

Can you post a types log from your second debug session?
(Put "log enable lldb types -f /tmp/types.log" into ~/.lldbinit-Xcode.)

4

For some reason, build with Xcode 10.2 starts the build of the framework. Before that I checked on Xcode 10.1.

I made several edits to the demo project to exclude the above reason. You need to be sure that the repository is cloned - not downloaded. The repository should be a git repository, there is a dependency in the scripts.

New steps:

  1. Open framework project - DevPod/DevPod.xcodeproj
  2. Run it. It should automatically transfer files to the DevPodFramework folder
  3. Delete everything from Derived Data folder
  4. Open main project - DebugProblemApp.xcodeproj
  5. Add breakpoint to DevPodViewController file - DevPod/DevPodViewController.swift line 7 (Added files are just references/links they don’t participate in the main target build)
  6. Run it
  7. Variable information is no longer displayed

Thanks for attention. Please try to reproduce again with a new demo project and new steps.

The first time Xcode can complain about the lack of a framework, but this is a glitch, after clearing the Derived Data folder, the error disappears.

5

Thanks for the updated instructions, but I think it would be most effective to looks at your types log instead; it will contain the reason for why debugging doesn't work.

6

Sure - types.log
log enable lldb all

7

The most notable thing is that no serialized Swift modules are found.

You mentioned that you manually did an add-dsym. Why was that necessary? Did you put the executable into a directory that is not indexed by Spotlight? (i.e., does a parent directory en in .noindex?)

Do you still have to manually load the .dSYMs if you put application and .dSYM into a directory where Spotlight can find the .dSYM bundles? Does debugging work if LLDB automatically finds the .dSYMs? If the answer is yes, then the reason is the SwiftASTContext doesn't get notified about later available additional .dSYM bundles. That would be a bug.

8

You mentioned that you manually did an add-dsym . Why was that necessary? Did you put the executable into a directory that is not indexed by Spotlight? (i.e., does a parent directory en in .noindex ?)

No, the folder is indexed. But for some reason dSYM does not load. I do it manually, but only for the first time.

Update 1:
Now I noticed that dSYM is sometimes automatically loaded, but the breakpoint did not work.

(lldb) image list
[  3] 7281248A-4B44-3DDF-A4F7-BC17791B5E80 0x00000001038c7000 /Users/vadim/DD/DebugProblemApp-dclefjfocetjauckpfeekldmfwmo/Build/Products/Debug-iphonesimulator/DebugProblemApp.app/Frameworks/DevPod.framework/DevPod 
      /Users/vadim/Desktop/DebugProblemApp3/DevPodFramework/DevPod.framework.dSYM/Contents/Resources/DWARF/DevPod

For some reason, the wrong dSYM was found. The content is the same (same dSYM), but I would like it to be the dSYM from Derived Data folder.

Does debugging work if LLDB automatically finds the .dSYMs
Perhaps lldb never automatically finds the .dSYMs. I'll try to check it out.

Update 2:
It loads automatically, and everything works (types also), only if I do not clear the contents of the Derived Data folder after the second step. (after building the project with framework)
If I delete everything after everything has worked ( the situation mentioned above ) and run the new project again, then dim will be loaded (perhaps because it was loaded earlier), but there will be no types.

SwiftASTContext doesn't get notified about later available additional dSYM bundles
Can I somehow do this manually ? Or check that it is?

9

LLDB will only find the .dSYM in the DerivedData folder when it's directly next to the executable/dylib otherwise it will go through Spotlight, but Spotlight never indexes DerivedData because there is a .noindex directory inside.

SwiftASTContext doesn't get notified about later available additional dSYM bundles

Can I somehow do this manually ? Or check that it is?

I am actually pretty sure that this would not work, because there exists no code in LLDB to handle that situation. We should definitely file a bug for that.

May I ask what it is that you are actually trying to achieve? From your steps it sounds like you might be testing whether your app is debuggable after moving it to a different machine, but there are lots of other reasons (Clang modules mostly) why this won't necessarily work at this point in time.

10

but Spotlight never indexes DerivedData because there is a .noindex directory inside

The only non-index (.noindex) folder inside the DerivedData folder is the folder with intermediate files - Intermediates.noindex.
My final path to the application is - /Users/username/DD/DebugProblemApp-fucfslzxpotphfeaohzvxfbkkvkh/Build/Products/Debug-iphonesimulator/DebugProblemApp.app ( Custom DerivedData path )

LLDB will only find the .dSYM in the DerivedData folder when it's directly next to the executable/dylib

Yes, I know about these two methods. Just thought that in my example dSYM is located next to the executable file (next to the framework). But apparently it is not.
Where should I put .dSYM so that it will be automatically found and loaded?
I tried to put it inside the application in the Framework folder (next to the framework), but that didn't work either.

May I ask what it is that you are actually trying to achieve?
From your steps it sounds like you might be testing whether your app is debuggable after moving it to a different machine, but there are lots of other reasons (Clang modules mostly) why this won't necessarily work at this point in time.

Sure.

Almost so. I'm trying to achieve reuse and debugging of the framework builded on another machine.
The difference is that only dynamic frameworks are reused, the application itself is always builded only locally.
In the end, the idea is similar to this article - Attaching sources to iOS/macOS binaries compiled on another machine. I am developing a solution that will be built into the project from the developers and, during the build process, download the dynamic frameworks builded on the remote machine and link them to the application and to another local builded frameworks.

As far as I know this is possible. I did some research and the only problem is the problem from this topic.

I also based on several articles and documents:
Symbols on macOS
Attaching sources to iOS/macOS binaries compiled on another machine
Support -fdebug-compilation-dir - alternative way
Improving swift-lldb support for Path Remappings

I also know that a similar problem has already been solved in other build systems: facebook/buck and bazelbuild/bazel. But as far as I know, completely different approaches are used there (including patching the paths to the sources in the binary)

@Adrian_Prantl I would be very grateful if I could suggest how to bring the current solution to a working state or an alternative solution that will work.

11

I made a few corrections so that there were no other dSYMs, besides the one that needs to be loaded.
Now the correct dSYM is always loaded, but the problem remains. So the problem is clearly not with the fact that dSYM does not load automatically.

12

Can you post another types log with those changes applied? If the dSYM gets loaded automatically, I'd expect a the swift modules from the dylib to show up in the output as "Found AST file data entries" for that module. It might also help to do this experiment with a current Swift 5.1 toolchain as we've greatly improved the types log output since then.

13

I think I found the root of the problem.
In dSYM there is an absolute path to the source framework.

It can be detected by executing this command or using MachOView:

dwarfdump -a DevPod.framework.dSYM/Contents/Resources/DWARF/DevPod | grep DevPod.framework

Example output:

0x0000227d: "/Users/username/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework"

At the same time, the build of the main application occurs in a different folder:

/Users/username/DD/DebugProblemApp-fucfslzxpotphfeaohzvxfbkkvkh/Build/Products/Debug-iphonesimulator/DebugProblemApp.app
/Users/username/DD/DebugProblemApp-fucfslzxpotphfeaohzvxfbkkvkh/Build/Products/Debug-iphonesimulator/DevPod/DevPod.framework // The framework that is linked to the application.
/Users/username/DD/DebugProblemApp-fucfslzxpotphfeaohzvxfbkkvkh/Build/Products/Debug-iphonesimulator/DevPod/DevPod.framework.dSYM // dSYM that I check ( dSYM that is loading )

Important: Before each launch, you need to kill the lldb-rpc-server process so that the information from dSYM is not cached.

If the framework also exist in path from dSYM at the time of loading the dSYM, then everything works (except the problem below). At the same time, all other files (artifacts of the framework build) can be deleted (except the problem below).

After that I tried to find out which files from the framework are needed.
After removing files from the framework, I realized that the following files are needed:

/Users/username/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework/Headers/DevPod.h
/Users/username/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework/Headers/DevPod-Swift.h
/Users/username/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework/Modules/module.modulemap

If one of these files is missing, the types will not be displayed. All other files are not important (except the problem below).
This is true for Swift 4.2 and Xcode 10.1 - Xcode 10.2.

About DevPod.framework/Headers/DevPod-Swift.h. I can disable the generation of this file using the flag SWIFT_INSTALL_OBJC_HEADER == NO. I'm not sure that this is a good idea, and the problem is not fixed with the other files. After this check, I returned the flag to its former state.

Then I realized that these files are important too. (All other files can also be deleted.)

/Users/username/DD/DevPod-aujdjllyhepdvcbapusozbgvtvlj/Build/Intermediates.noindex/DevPod.build/Debug-iphonesimulator/DevPod.build/all-product-headers.yaml
/Users/username/DD/DevPod-aujdjllyhepdvcbapusozbgvtvlj/Build/Intermediates.noindex/DevPod.build/Debug-iphonesimulator/DevPod.build/module.modulemap

If these files do not exist, the command (lldb) po self will display an error: error: Couldn't IRGen expression, no additional error.
But at the same time, the debug panel works and the other command works:

(lldb) frame variable
(DevPod.DevPodViewController) self = 0x00007fefa5f033d0 {
  UIKit.UIViewController = {
  ...

Found out the dependencies between these files.
If I delete or rename only all-product-headers.yaml file, then the behavior described above is reproduced ( about Couldn't IRGen expression ).
If I delete or rename another file - Build/Intermediates.noindex/DevPod.build/Debug-iphonesimulator/DevPod.build/module.modulemap and all-product-headers.yaml, then the behavior will be the same as in the initial problem - debug panel does not display types and (lldb) po self return error: <EXPR>:3:1: error: use of unresolved identifier 'self'.

What is the reason for this behavior, I do not understand.
Just in case, I note that the framework pointed to by the dSYM, the framework with which the applications are linked and the framework that moves into the application are absolutely identical (copies). All necessary headers and modulemap files exist in all copies of the framework.

I saved all the logs of various situations - link

I hope there is a solution.

P.S.: I also slightly updated the scripts in the demo project, if it caused problems, maybe it was fixed.

2 Likes
14

That's the kinds of problems I alluded to with earlier my statement about Clang modules possibly causing problems for you. This doesn't really make a difference for the end-user, but for completeness: the search paths pointing to the header files most probably come from the serializes .swiftmodule files inside the .dSYM bundle and not from DWARF itself.

As long as you ship the Clang dependencies with your library, you can use a combination of -fdebug-prefix-map and the remapping dictionary in the .dSYM bundle to point to the actual location on the machine you are debugging. LLDB must initialize a Swift compiler to import the swiftmodules from your framework, and for that to work it must know the search paths to all Clang dependencies. You can see the search paths that LLDB uses in the types log as "Extra clang arguments". This is printed once per dylib, and once for the global expression context, which is the sum of all dylib options. If you can (by playing with -fdebug-prefix-map and the .dSYM dictionary) get these search paths used by LLDB to match the user's machine then debugging will work.

1 Like
15

For reference, this testcase from the LLDB testsuite manually generates a remapping dictionary inside a .dSYM bundle:

1 Like
16

Can the -fdebug-prefix-map flag change the path not only to the sources, but also to the headers and .modulemap ?

It would help me a lot if I were told exactly how these flags should be used, maybe an example could be shown. I am familiar with using the DBGSourcePathRemapping flag with the flag, but I don’t understand how it will help me to solve problems with headers, modulemap and all-product-headers.yaml.

How to work around problems with this file - all-product-headers.yaml ? Why is it needed?
It is located in intermediate files. Should I save it or move it somewhere? For example, put inside the framework? Maybe I can turn it off?

I also found out that its content of this file does not affect the debug. I can generate empty content for this file.

{
  'version': 0,
  'case-sensitive': 'false',
  'roots': []
}

And everything will work. But the file must be and must be valid otherwise the problem with the output is reproduced - error: Couldn't IRGen expression, no additional error.
If this file is in such an empty format, then no other intermediate files are needed ( Before that, I also mentioned that this file is also needed - Intermediates.noindex/DevPod.build/Debug-iphonesimulator/DevPod.build/module.modulemap ). And if the file is in its original state, then another file is needed.

Can this solution lead to problems ( generating all-product-headers.yaml content ) ? If so, what problems can there be? Maybe there is a better solution?

17

Can the -fdebug-prefix-map flag change the path not only to the sources, but also to the headers and .modulemap ?

Yes!

It would help me a lot if I were told exactly how these flags should be used

The basic problem you want to solve is that the absolute include paths on the build machine won't exist on the user's machine. One way to do this is by using -fdebug-prefix-map to remap the build and source directories to a relative path, e.g., $BUILD_ROOT=., or to another well-known path and then potentially use the dictionary in the dSYM to remap the remapped paths to their actual location on the user's machine.

How to work around problems with this file - all-product-headers.yaml

I don't have a good answer for this. This VFS overlay exists because you need a different set of headers visibly while building a mixed-language framework and while importing it. This is why does not affect debug time.

And everything will work. But the file must be and must be valid otherwise the problem with the output is reproduced - error: Couldn't IRGen expression, no additional error .

The real error will be in the types log. (We should probably do a better job at surfacing it).

Can this solution lead to problems ( generating all-product-headers.yaml content ) ?

Which files are being remapped by the all-product-headers overlay in you example?

2 Likes
18

Which files are being remapped by the all-product-headers overlay in you example?
All files. Was:

{
  'version': 0,
  'case-sensitive': 'false',
  'roots': [
    {
      'type': 'directory',
      'name': "/Users/vvsmal/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework/Headers",
      'contents': [
        {
          'type': 'file',
          'name': "DevPod-Swift.h",
          'external-contents': "/Users/vvsmal/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework/Headers/DevPod-Swift.h"
        },
        {
          'type': 'file',
          'name': "DevPod.h",
          'external-contents': "/Users/vvsmal/Desktop/DebugProblemApp/DevPod/DevPod/DevPod.h"
        }
      ]
    },
    {
      'type': 'directory',
      'name': "/Users/vvsmal/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Products/Debug-iphonesimulator/DevPod.framework/Modules",
      'contents': [
        {
          'type': 'file',
          'name': "module.modulemap",
          'external-contents': "/Users/vvsmal/DD/DevPod-gebvdrztdkfjfphimqjdufcsdetc/Build/Intermediates.noindex/DevPod.build/Debug-iphonesimulator/DevPod.build/module.modulemap"
        }
      ]
    }
  ]
}

After:

{
  'version': 0,
  'case-sensitive': 'false',
  'roots': []
}

One way to do this is by using -fdebug-prefix-map to remap the build and source directories to a relative path, e.g.
If I understood correctly, then in the new Xcode 11, this flag will work automatically if I setted DBGBuildSourcePath flag in the plist file inside dSym.

19

hi @Adrian_Prantl, I have maybe a similar issue.

lldb knows of the variables, but doesn't know of some types, for example Swift.String. In the types log, I see:

SwiftASTContext("SomeModuleTests")::GetTypeFromMangledTypename("$sSSD")
SwiftASTContext("SomeModuleTests")::GetTypeFromMangledTypename("$sSSD") -- found in the negative cache
SwiftASTContext("SomeModuleTests")::GetTypeFromMangledTypename("$sSSD") -- not cached, searching

I'm not sure what the "negative cache" is, and despite saying "found", it appears that it's not actually finding it. In fact, it seems to be missing all of the Swift module.

When using -serialize-debugging-options, those lines aren't present, and instead there's this:

SwiftASTContextForExpressions::GetModule("Swift") -- found Swift

This line is not present when using -no-serialize-debugging-options. So it seems the swiftmodules contain references to Swift stdlib? Is there another way to create that reference without using serialized debugging options?

Here's a gist with the two type logs: no-serialize-debugging-options.txt · GitHub

Any direction would be much appreciated. We want to use -no-serialize-debugging-options to make swiftmodule files are cacheable and usable from any machine, which means ensuring it has no hard coded paths.

20

Generally, if you use -no-serialize-debugging-options, then LLDB obviously won't have any search paths to any of your Swift modules' Clang module dependencies, nor will it have any record of other (e.g., -D) options that you used.
I am surprised that this would extend to the Swift stdlib, though, since LLDB uses special code to hand-load the standard library. Can you package up a testcase that reproduces your issue with the stdlib and send it my way via JIRA or radar?