Making a SourceKit-LSP Client: Find References Fails (Solved ✅)

flowtoolz · May 18, 2022, 3:48pm

So I'm back at building this codebase visualizer:

It relies on LSP, and I use sourcekit-lsp as my example language server.

As you can see, extracting document symbols works fine. Next, I wanna add dependency inspection using the "find references" request.

SourceKit-LSP reacts to the request like so: It "successfully" returns an empty array of references (standard out). Then it outputs two strings (standard error):

[2022-05-18 13:08:35.292] error response (Request Failed): Loading the stdlib failed
[2022-05-18 13:08:35.293] cursor info failed DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift):5:1: ResponseError(code: LanguageServerProtocol.ErrorCode(rawValue: -32001), message: "sourcekitd request failed: error response (Request Failed): Loading the stdlib failed")

The interesting part might be: sourcekitd request failed: error response (Request Failed): Loading the stdlib failed

Here is what I found out: To process the references request, sourcekit-lsp also invokes the symbol info request which then tries to retrieve a CursorInfo from sourcekitd which fails. This would be in CursorInfo.swift around line 107:

let handle = self.sourcekitd.send(skreq, self.queue) { ... }

I'm aware that finding references crosses document boundaries and involves the index db, so I built the analyzed package from the command line in hopes to update its index but to no avail.

I'm pretty stuck and hope someone can point me in the right direction. What does a sourcekit-lsp client have to be aware of to request symbol references? Any hint means the world to me

ahoppen · May 18, 2022, 4:15pm

It appears that for some reason sourcekitd (which is backing the semantic functionality of SourceKit-LSP for Swift) isn’t able to find the standard library. How did you install SourceKit-LSP or are you using the copy from Xcode? Would it be possible for you to share information how to reproduce this issue (e.g. the project that you open in SourceKit-LSP and the requests that you send)?

flowtoolz · May 18, 2022, 4:42pm

Thanks for the lightning fast response

It runs /usr/bin/xcrun sourcekit-lsp and my Xcode is at 13.3.1.

I also did the export TOOLCHAINS=swift thing to be sure.

The flow of LSP messages from the client perspective:

Request initialization
- the only client capability: "hierarchicalDocumentSymbolSupport": true
- server capabilities include "referencesProvider" : true
Notify that I did open document X
Request symbols of document X

Request references for a symbol of document X

The request params look like this:

{
  "textDocument" : {
    "uri" : "file:\/\/\/Users\/seb\/Library\/Containers\/com.flowtoolz.codeface\/Data\/file:\/Users\/seb\/Desktop\/GitHub%2520Repos\/SwiftObserver\/Package.swift"
  },
  "position" : {
    "line" : 4,
    "character" : 0
  },
  "context" : {
    "includeDeclaration" : false
  }
}

An example project that the visualizer opens with sourcekit-lsp is SwiftObserver.

ahoppen · May 19, 2022, 10:31am

Hmm, interesting. Could you try running SourceKit with the SOURCEKIT_LOGGING=3 environment variable and see if you get any debug output to stderr that might give us more information as to what’s going wrong?

Also, do you have some kind of script that easily reproduces sending these messages to sourcekit-lsp? I wasn’t able to build something easily. If you don’t, that’s fine, the SourceKit logging mentioned above might be sufficient to figure out what’s going wrong.

flowtoolz · May 19, 2022, 4:26pm

Here is the stderr output of sourcekit-lsp (sk log level 3) up until the error. I shortened the array data in 4 occurrences of key.syntaxmap: [ ... ]:

[2022-05-19 14:08:14.073] SourceKitServer: Request<initialize>(
  id: "629E0592-BDA1-4FED-A1D9-097B9952B885",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: InitializeRequest(processId: Optional(47991), rootPath: nil, rootURI: Optional(LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/)), initializationOptions: nil, capabilities: LanguageServerProtocol.ClientCapabilities(workspace: nil, textDocument: Optional(LanguageServerProtocol.TextDocumentClientCapabilities(synchronization: nil, completion: nil, hover: nil, signatureHelp: nil, references: nil, documentHighlight: nil, documentSymbol: Optional(LanguageServerProtocol.TextDocumentClientCapabilities.DocumentSymbol(dynamicRegistration: nil, symbolKind: nil, hierarchicalDocumentSymbolSupport: Optional(true))), formatting: nil, rangeFormatting: nil, onTypeFormatting: nil, declaration: nil, definition: nil, typeDefinition: nil, implementation: nil, codeAction: nil, codeLens: nil, documentLink: nil, colorProvider: nil, rename: nil, publishDiagnostics: nil, foldingRange: nil, callHierarchy: nil, semanticTokens: nil))), trace: nil, workspaceFolders: nil)
)
[2022-05-19 14:08:14.076] SourceKitServer: Response<initialize("629E0592-BDA1-4FED-A1D9-097B9952B885")>(
  success(LanguageServerProtocol.InitializeResult(capabilities: LanguageServerProtocol.ServerCapabilities(textDocumentSync: Optional(LanguageServerProtocol.TextDocumentSyncOptions(openClose: Optional(true), change: Optional(LanguageServerProtocol.TextDocumentSyncKind.incremental), willSave: Optional(true), willSaveWaitUntil: Optional(false), save: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentSyncOptions.SaveOptions>.value(LanguageServerProtocol.TextDocumentSyncOptions.SaveOptions(includeText: Optional(false)))))), hoverProvider: Optional(true), completionProvider: Optional(LanguageServerProtocol.CompletionOptions(resolveProvider: Optional(false), triggerCharacters: Optional(["."]), allCommitCharacters: nil)), signatureHelpProvider: nil, definitionProvider: Optional(true), typeDefinitionProvider: nil, implementationProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentAndStaticRegistrationOptions>.bool(true)), referencesProvider: Optional(true), documentHighlightProvider: Optional(true), documentSymbolProvider: Optional(true), workspaceSymbolProvider: Optional(true), codeActionProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.CodeActionServerCapabilities>.value(LanguageServerProtocol.CodeActionServerCapabilities.supportsCodeActionRequests(true))), codeLensProvider: nil, documentFormattingProvider: nil, documentRangeFormattingProvider: nil, documentOnTypeFormattingProvider: nil, renameProvider: nil, documentLinkProvider: nil, colorProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentAndStaticRegistrationOptions>.bool(true)), foldingRangeProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentAndStaticRegistrationOptions>.bool(true)), declarationProvider: nil, executeCommandProvider: Optional(LanguageServerProtocol.ExecuteCommandOptions(commands: ["semantic.refactor.command"])), workspace: nil, callHierarchyProvider: nil, semanticTokensProvider: nil, experimental: nil)))
)
[2022-05-19 14:08:14.260] [debug]: loading manifest for 'swiftytoolz' v. unknown from cache
[2022-05-19 14:08:14.291] SourceKitServer: Notification<initialized>(
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: InitializedNotification()
)
[2022-05-19 14:08:14.292] SourceKitServer: Notification<textDocument/didOpen>(
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: DidOpenTextDocumentNotification(textDocument: LanguageServerProtocol.TextDocumentItem(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift), language: swift, version: 1, text: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n"))
)
sourcekit: [2:sourcekitd_initialize:4611: 0.0000] initializing
[2022-05-19 14:08:14.295] Using toolchain XcodeDefault (com.apple.dt.toolchain.XcodeDefault) for DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift)
[2022-05-19 14:08:14.295] SourceKitServer: Request<textDocument/documentSymbol>(
  id: "3377EC38-8C8D-40C5-80F0-C739EF1D5DFE",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: DocumentSymbolRequest(textDocument: LanguageServerProtocol.TextDocumentIdentifier(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift)))
)
[2022-05-19 14:08:14.295] registerForChangeNotifications(/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift)
[2022-05-19 14:08:14.295] Initial build settings received for opened file DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift)
sourcekit: [2:sourcekitd_send_request_sync-before:7687: 0.0035] {
  key.request: source.request.editor.open,
  key.name: "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift",
  key.compilerargs: [
    "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift"
  ],
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n"
}
[2022-05-19 14:08:14.295] {
  key.request: source.request.editor.open,
  key.name: "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift",
  key.compilerargs: [
    "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift"
  ],
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n"
}
sourcekit: [4:handleInternalUIDRequest:8967: 0.0151] service queried UID for: source.statistic.num-requests
sourcekit: [4:handleInternalUIDRequest:8967: 0.0152] service queried UID for: source.statistic.num-semantic-requests
sourcekit: [4:handleInternalUIDRequest:8967: 0.0152] service queried UID for: source.request.configuration.global
sourcekit: [4:handleInternalUIDRequest:8967: 0.0152] service queried UID for: source.request.protocol_version
sourcekit: [4:handleInternalUIDRequest:8967: 0.0153] service queried UID for: source.request.compiler_version
sourcekit: [4:handleInternalUIDRequest:8967: 0.0153] service queried UID for: source.request.crash_exit
sourcekit: [4:handleInternalUIDRequest:8967: 0.0154] service queried UID for: source.request.test_notification
sourcekit: [4:handleInternalUIDRequest:8967: 0.0155] service queried UID for: source.request.demangle
sourcekit: [4:handleInternalUIDRequest:8967: 0.0156] service queried UID for: source.request.mangle_simple_class
sourcekit: [4:handleInternalUIDRequest:8967: 0.0157] service queried UID for: source.request.enable-compile-notifications
sourcekit: [4:handleInternalUIDRequest:8967: 0.0157] service queried UID for: source.request.buildsettings.register
sourcekit: [4:handleInternalUIDRequest:8967: 0.0158] service queried UID for: source.request.dependency_updated
sourcekit: [4:handleInternalUIDRequest:8967: 0.0158] service queried UID for: source.request.docinfo
sourcekit: [4:handleInternalUIDRequest:8967: 0.0159] service queried UID for: source.request.editor.open
sourcekit: [4:handleInternalUIDRequest:8967: 0.0165] service queried UID for: source.diagnostic.stage.swift.parse
sourcekit: [4:handleInternalUIDRequest:8967: 0.0165] service queried UID for: source.lang.swift.decl.var.global
sourcekit: [4:handleInternalUIDRequest:8967: 0.0166] service queried UID for: source.lang.swift.accessibility.internal
sourcekit: [4:handleInternalUIDRequest:8967: 0.0166] service queried UID for: source.lang.swift.expr.argument
sourcekit: [4:handleInternalUIDRequest:8967: 0.0166] service queried UID for: source.lang.swift.structure.elem.expr
sourcekit: [4:handleInternalUIDRequest:8967: 0.0167] service queried UID for: source.lang.swift.expr.array
sourcekit: [4:handleInternalUIDRequest:8967: 0.0168] service queried UID for: source.lang.swift.expr.call
sourcekit: [4:handleInternalUIDRequest:8967: 0.0169] service queried UID for: source.lang.swift.syntaxtype.comment
sourcekit: [4:handleInternalUIDRequest:8967: 0.0169] service queried UID for: source.lang.swift.syntaxtype.keyword
sourcekit: [4:handleInternalUIDRequest:8967: 0.0170] service queried UID for: source.lang.swift.syntaxtype.identifier
sourcekit: [4:handleInternalUIDRequest:8967: 0.0171] service queried UID for: source.lang.swift.syntaxtype.string
sourcekit: [2:sourcekitd_send_request_sync-after:7687: 0.0172] {
  key.offset: 0,
  key.length: 336,
  key.diagnostic_stage: source.diagnostic.stage.swift.parse,
  key.syntaxmap: [
    {
      key.kind: source.lang.swift.syntaxtype.comment,
      key.offset: 0,
      key.length: 29
    },
    {
      key.kind: source.lang.swift.syntaxtype.keyword,
      key.offset: 30,
      key.length: 6
    },
   
    ...
  ]
}
[2022-05-19 14:08:14.309] {
  key.offset: 0,
  key.length: 336,
  key.diagnostic_stage: source.diagnostic.stage.swift.parse,
  key.syntaxmap: [
    {
      key.kind: source.lang.swift.syntaxtype.comment,
      key.offset: 0,
      key.length: 29
    },
    {
      key.kind: source.lang.swift.syntaxtype.keyword,
      key.offset: 30,
      key.length: 6
    },
    
    ...
  ]
}
sourcekit: [2:sourcekitd_send_request-before:7687: 0.0175] {
  key.request: source.request.editor.open,
  key.name: "DocumentSymbols:/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift",
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n",
  key.syntactic_only: 1
}
[2022-05-19 14:08:14.309] SourceKitServer: Notification<textDocument/publishDiagnostics>(
  clientID: ObjectIdentifier(0x000060000170c140),
  params: PublishDiagnosticsNotification(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift), version: Optional(1), diagnostics: [])
)
[2022-05-19 14:08:14.309] updatedLexicalTokens(response:for:) took 0.18 ms to execute
[2022-05-19 14:08:14.309] {
  key.request: source.request.editor.open,
  key.name: "DocumentSymbols:/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift",
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n",
  key.syntactic_only: 1
}
sourcekit: [2:sourcekitd_send_request-after:7687: 0.0179] {
  key.offset: 0,
  key.length: 336,
  key.diagnostic_stage: source.diagnostic.stage.swift.parse,
  key.syntaxmap: [
    {
      key.kind: source.lang.swift.syntaxtype.comment,
      key.offset: 0,
      key.length: 29
    },
    {
      key.kind: source.lang.swift.syntaxtype.keyword,
      key.offset: 30,
      key.length: 6
    },
   
    ...
  ]
}
[2022-05-19 14:08:14.310] {
  key.offset: 0,
  key.length: 336,
  key.diagnostic_stage: source.diagnostic.stage.swift.parse,
  key.syntaxmap: [
    {
      key.kind: source.lang.swift.syntaxtype.comment,
      key.offset: 0,
      key.length: 29
    },
    {
      key.kind: source.lang.swift.syntaxtype.keyword,
      key.offset: 30,
      key.length: 6
    },
    
    ...
  ]
}
[2022-05-19 14:08:14.311] SourceKitServer: Response<textDocument/documentSymbol("3377EC38-8C8D-40C5-80F0-C739EF1D5DFE")>(
  success(Optional(LanguageServerProtocol.DocumentSymbolResponse.documentSymbols([LanguageServerProtocol.DocumentSymbol(name: "package", detail: nil, kind: LanguageServerProtocol.SymbolKind(rawValue: 13), deprecated: nil, _range: LanguageServerProtocol.CustomCodable<LanguageServerProtocol.PositionRange>(wrappedValue: Range(Position(line: 4, utf16index: 0)..<Position(line: 18, utf16index: 1))), _selectionRange: LanguageServerProtocol.CustomCodable<LanguageServerProtocol.PositionRange>(wrappedValue: Range(Position(line: 4, utf16index: 4)..<Position(line: 4, utf16index: 11))), children: Optional([]))])))
)
sourcekit: [2:sourcekitd_send_request-before:4611: 0.0226] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift"
  ],
  key.offset: 57,
  key.sourcefile: "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift"
}
sourcekit: [4:handleInternalUIDRequest:4611: 0.0227] service queried UID for: source.request.editor.close
[2022-05-19 14:08:14.314] SourceKitServer: Request<textDocument/references>(
  id: "9BEBB389-84FE-4A76-9C5D-47350C6F04DD",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: ReferencesRequest(textDocument: LanguageServerProtocol.TextDocumentIdentifier(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift)), position: Position(line: 4, utf16index: 0), context: LanguageServerProtocol.ReferencesContext(includeDeclaration: true))
)
[2022-05-19 14:08:14.314] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift"
  ],
  key.offset: 57,
  key.sourcefile: "/Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift"
}
sourcekit: [4:handleInternalUIDRequest:4867: 0.0228] service queried UID for: source.request.editor.replacetext
sourcekit: [4:handleInternalUIDRequest:4867: 0.0229] service queried UID for: source.request.editor.formattext
sourcekit: [4:handleInternalUIDRequest:4867: 0.0230] service queried UID for: source.request.editor.expand_placeholder
sourcekit: [4:handleInternalUIDRequest:4867: 0.0230] service queried UID for: source.request.editor.open.interface
sourcekit: [4:handleInternalUIDRequest:4867: 0.0231] service queried UID for: source.request.editor.open.interface.header
sourcekit: [4:handleInternalUIDRequest:4867: 0.0231] service queried UID for: source.request.editor.open.interface.swiftsource
sourcekit: [4:handleInternalUIDRequest:4867: 0.0232] service queried UID for: source.request.editor.open.interface.swifttype
sourcekit: [4:handleInternalUIDRequest:4867: 0.0233] service queried UID for: source.request.editor.extract.comment
sourcekit: [4:handleInternalUIDRequest:4867: 0.0234] service queried UID for: source.request.convert.markup.xml
sourcekit: [4:handleInternalUIDRequest:4867: 0.0234] service queried UID for: source.request.editor.find_usr
sourcekit: [4:handleInternalUIDRequest:4867: 0.0234] service queried UID for: source.request.editor.find_interface_doc
sourcekit: [4:handleInternalUIDRequest:4867: 0.0235] service queried UID for: source.request.module.groups
sourcekit: [4:handleInternalUIDRequest:4867: 0.0235] service queried UID for: source.request.syntacticrename
sourcekit: [4:handleInternalUIDRequest:4867: 0.0236] service queried UID for: source.request.find-syntactic-rename-ranges
sourcekit: [4:handleInternalUIDRequest:4867: 0.0236] service queried UID for: source.request.codecomplete.close
sourcekit: [4:handleInternalUIDRequest:4867: 0.0236] service queried UID for: source.request.codecomplete.cache.ondisk
sourcekit: [4:handleInternalUIDRequest:4867: 0.0237] service queried UID for: source.request.codecomplete.setpopularapi
sourcekit: [4:handleInternalUIDRequest:4867: 0.0237] service queried UID for: source.request.codecomplete.setcustom
sourcekit: [4:handleInternalUIDRequest:4867: 0.0238] service queried UID for: source.request.statistics
sourcekit: [4:handleInternalUIDRequest:4867: 0.0238] service queried UID for: source.request.codecomplete
sourcekit: [4:handleInternalUIDRequest:4867: 0.0239] service queried UID for: source.request.codecomplete.open
sourcekit: [4:handleInternalUIDRequest:4867: 0.0239] service queried UID for: source.request.codecomplete.update
sourcekit: [4:handleInternalUIDRequest:4867: 0.0240] service queried UID for: source.request.typecontextinfo
sourcekit: [4:handleInternalUIDRequest:4867: 0.0240] service queried UID for: source.request.conformingmethods
sourcekit: [4:handleInternalUIDRequest:4867: 0.0241] service queried UID for: source.request.indexsource
sourcekit: [4:handleInternalUIDRequest:4867: 0.0241] service queried UID for: source.request.cursorinfo
sourcekit: [2:sourcekitd_send_request-after:4867: 0.0244] error response (Request Failed): Loading the stdlib failed
[2022-05-19 14:08:14.316] error response (Request Failed): Loading the stdlib failed
[2022-05-19 14:08:14.316] cursor info failed DocumentURI(storage: file:///Users/seb/Library/Containers/com.flowtoolz.codeface/Data/file:/Users/seb/Desktop/GitHub%2520Repos/SwiftyToolz/Package.swift):5:1: ResponseError(code: LanguageServerProtocol.ErrorCode(rawValue: -32001), message: "sourcekitd request failed: error response (Request Failed): Loading the stdlib failed")
[2022-05-19 14:08:14.317] SourceKitServer: Response<textDocument/references("9BEBB389-84FE-4A76-9C5D-47350C6F04DD")>(
  success([])
)

Between the document symbol- and the references request, the log says: sourcekit: [4:handleInternalUIDRequest:4611: 0.0227] service queried UID for: source.request.editor.close. Editor close? Does that sound odd?

It is theoretically possible (everything's pushed) for others to reproduce this, since the project is still open source, but I don't expect anyone to go to that extent :

Build and run LSPService (local Vapor service providing access to sourcekit-lsp, required by Codeface)
Build and run Codeface (macOS 12+ app)
In Codeface: Press Cmd + L to load a Swift package (opens a dialog).
- (Choose a simple package, it doesn't yet work with sourcekit-lsp itself for instance)
Codeface logs (empty) arrays of references with prefix ✅ References:.

If there's anything else I can provide to unveil the issue, I'm obviously eager to do so

flowtoolz · May 22, 2022, 2:30pm

Here's a better log painting a more complete picture and also omitting the syntax trees:

[2022-05-22 15:25:23.318] SourceKitServer: Request<initialize>(
  id: "E8CF2EC5-EC50-4951-A8F9-4EE9A2FD3D7A",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: InitializeRequest(processId: Optional(91552), rootPath: nil, rootURI: Optional(LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/)), initializationOptions: nil, capabilities: LanguageServerProtocol.ClientCapabilities(workspace: nil, textDocument: Optional(LanguageServerProtocol.TextDocumentClientCapabilities(synchronization: nil, completion: nil, hover: nil, signatureHelp: nil, references: nil, documentHighlight: nil, documentSymbol: Optional(LanguageServerProtocol.TextDocumentClientCapabilities.DocumentSymbol(dynamicRegistration: nil, symbolKind: nil, hierarchicalDocumentSymbolSupport: Optional(true))), formatting: nil, rangeFormatting: nil, onTypeFormatting: nil, declaration: nil, definition: nil, typeDefinition: nil, implementation: nil, codeAction: nil, codeLens: nil, documentLink: nil, colorProvider: nil, rename: nil, publishDiagnostics: nil, foldingRange: nil, callHierarchy: nil, semanticTokens: nil))), trace: nil, workspaceFolders: nil)
)
[2022-05-22 15:25:23.322] SourceKitServer: Response<initialize("E8CF2EC5-EC50-4951-A8F9-4EE9A2FD3D7A")>(
  success(LanguageServerProtocol.InitializeResult(capabilities: LanguageServerProtocol.ServerCapabilities(textDocumentSync: Optional(LanguageServerProtocol.TextDocumentSyncOptions(openClose: Optional(true), change: Optional(LanguageServerProtocol.TextDocumentSyncKind.incremental), willSave: Optional(true), willSaveWaitUntil: Optional(false), save: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentSyncOptions.SaveOptions>.value(LanguageServerProtocol.TextDocumentSyncOptions.SaveOptions(includeText: Optional(false)))))), hoverProvider: Optional(true), completionProvider: Optional(LanguageServerProtocol.CompletionOptions(resolveProvider: Optional(false), triggerCharacters: Optional(["."]), allCommitCharacters: nil)), signatureHelpProvider: nil, definitionProvider: Optional(true), typeDefinitionProvider: nil, implementationProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentAndStaticRegistrationOptions>.bool(true)), referencesProvider: Optional(true), documentHighlightProvider: Optional(true), documentSymbolProvider: Optional(true), workspaceSymbolProvider: Optional(true), codeActionProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.CodeActionServerCapabilities>.value(LanguageServerProtocol.CodeActionServerCapabilities.supportsCodeActionRequests(true))), codeLensProvider: nil, documentFormattingProvider: nil, documentRangeFormattingProvider: nil, documentOnTypeFormattingProvider: nil, renameProvider: nil, documentLinkProvider: nil, colorProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentAndStaticRegistrationOptions>.bool(true)), foldingRangeProvider: Optional(LanguageServerProtocol.ValueOrBool<LanguageServerProtocol.TextDocumentAndStaticRegistrationOptions>.bool(true)), declarationProvider: nil, executeCommandProvider: Optional(LanguageServerProtocol.ExecuteCommandOptions(commands: ["semantic.refactor.command"])), workspace: nil, callHierarchyProvider: nil, semanticTokensProvider: nil, experimental: nil)))
)
[2022-05-22 15:25:24.919] [debug]: loading manifest for 'swiftytoolz' v. unknown from cache
[2022-05-22 15:25:24.948] SourceKitServer: Notification<initialized>(
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: InitializedNotification()
)
[2022-05-22 15:25:24.949] SourceKitServer: Notification<textDocument/didOpen>(
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: DidOpenTextDocumentNotification(textDocument: LanguageServerProtocol.TextDocumentItem(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift), language: swift, version: 1, text: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n"))
)
sourcekit: [2:sourcekitd_initialize:5639: 0.0000] initializing
[2022-05-22 15:25:24.951] Using toolchain XcodeDefault (com.apple.dt.toolchain.XcodeDefault) for DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift)
[2022-05-22 15:25:24.952] SourceKitServer: Request<textDocument/documentSymbol>(
  id: "60425F3E-5717-440C-AAD9-CFEFD4D1CBF4",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: DocumentSymbolRequest(textDocument: LanguageServerProtocol.TextDocumentIdentifier(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift)))
)
[2022-05-22 15:25:24.952] registerForChangeNotifications(/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift)
[2022-05-22 15:25:24.952] Initial build settings received for opened file DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift)
[2022-05-22 15:25:24.952] {
  key.request: source.request.editor.open,
  key.name: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift",
  key.compilerargs: [
    "-swift-version",
    "5",
    "-I",
    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
    "-package-description-version",
    "5.6.0",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
  ],
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n"
}
sourcekit: [2:sourcekitd_send_request_sync-before:5639: 0.0033] {
  key.request: source.request.editor.open,
  key.name: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift",
  key.compilerargs: [
    "-swift-version",
    "5",
    "-I",
    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
    "-package-description-version",
    "5.6.0",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
  ],
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n"
}
sourcekit: [4:handleInternalUIDRequest:7943: 0.0118] service queried UID for: source.statistic.num-requests
sourcekit: [4:handleInternalUIDRequest:7943: 0.0118] service queried UID for: source.statistic.num-semantic-requests
sourcekit: [4:handleInternalUIDRequest:7943: 0.0119] service queried UID for: source.request.configuration.global
sourcekit: [4:handleInternalUIDRequest:7943: 0.0119] service queried UID for: source.request.protocol_version
sourcekit: [4:handleInternalUIDRequest:7943: 0.0120] service queried UID for: source.request.compiler_version
sourcekit: [4:handleInternalUIDRequest:7943: 0.0120] service queried UID for: source.request.crash_exit
sourcekit: [4:handleInternalUIDRequest:7943: 0.0120] service queried UID for: source.request.test_notification
sourcekit: [4:handleInternalUIDRequest:7943: 0.0121] service queried UID for: source.request.demangle
sourcekit: [4:handleInternalUIDRequest:7943: 0.0122] service queried UID for: source.request.mangle_simple_class
sourcekit: [4:handleInternalUIDRequest:7943: 0.0122] service queried UID for: source.request.enable-compile-notifications
sourcekit: [4:handleInternalUIDRequest:7943: 0.0123] service queried UID for: source.request.buildsettings.register
sourcekit: [4:handleInternalUIDRequest:7943: 0.0123] service queried UID for: source.request.dependency_updated
sourcekit: [4:handleInternalUIDRequest:7943: 0.0124] service queried UID for: source.request.docinfo
sourcekit: [4:handleInternalUIDRequest:7943: 0.0124] service queried UID for: source.request.editor.open
sourcekit: [4:handleInternalUIDRequest:7943: 0.0131] service queried UID for: source.diagnostic.stage.swift.parse
sourcekit: [4:handleInternalUIDRequest:7943: 0.0132] service queried UID for: source.lang.swift.decl.var.global
sourcekit: [4:handleInternalUIDRequest:7943: 0.0132] service queried UID for: source.lang.swift.accessibility.internal
sourcekit: [4:handleInternalUIDRequest:7943: 0.0133] service queried UID for: source.lang.swift.expr.argument
sourcekit: [4:handleInternalUIDRequest:7943: 0.0134] service queried UID for: source.lang.swift.structure.elem.expr
sourcekit: [4:handleInternalUIDRequest:7943: 0.0135] service queried UID for: source.lang.swift.expr.array
sourcekit: [4:handleInternalUIDRequest:7943: 0.0135] service queried UID for: source.lang.swift.expr.call
sourcekit: [4:handleInternalUIDRequest:7943: 0.0136] service queried UID for: source.lang.swift.syntaxtype.comment
sourcekit: [4:handleInternalUIDRequest:7943: 0.0137] service queried UID for: source.lang.swift.syntaxtype.keyword
sourcekit: [4:handleInternalUIDRequest:7943: 0.0137] service queried UID for: source.lang.swift.syntaxtype.identifier
sourcekit: [4:handleInternalUIDRequest:7943: 0.0138] service queried UID for: source.lang.swift.syntaxtype.string
[2022-05-22 15:25:24.963] SourceKitServer: Notification<textDocument/publishDiagnostics>(
  clientID: ObjectIdentifier(0x0000600001704e80),
  params: PublishDiagnosticsNotification(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift), version: Optional(1), diagnostics: [])
)
[2022-05-22 15:25:24.963] updatedLexicalTokens(response:for:) took 0.18 ms to execute
sourcekit: [2:sourcekitd_send_request-before:5639: 0.0142] {
  key.request: source.request.editor.open,
  key.name: "DocumentSymbols:/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift",
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n",
  key.syntactic_only: 1
}
[2022-05-22 15:25:24.963] {
  key.request: source.request.editor.open,
  key.name: "DocumentSymbols:/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift",
  key.sourcetext: "// swift-tools-version:5.6.0\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"SwiftyToolz\",\n    products: [\n        .library(\n            name: \"SwiftyToolz\",\n            targets: [\"SwiftyToolz\"]\n        ),\n    ],\n    targets: [\n        .target(\n            name: \"SwiftyToolz\",\n            path: \"Code\"\n        ),\n    ]\n)\n",
  key.syntactic_only: 1
}
[2022-05-22 15:25:24.964] SourceKitServer: Response<textDocument/documentSymbol("60425F3E-5717-440C-AAD9-CFEFD4D1CBF4")>(
  success(Optional(LanguageServerProtocol.DocumentSymbolResponse.documentSymbols([LanguageServerProtocol.DocumentSymbol(name: "package", detail: nil, kind: LanguageServerProtocol.SymbolKind(rawValue: 13), deprecated: nil, _range: LanguageServerProtocol.CustomCodable<LanguageServerProtocol.PositionRange>(wrappedValue: Range(Position(line: 4, utf16index: 0)..<Position(line: 18, utf16index: 1))), _selectionRange: LanguageServerProtocol.CustomCodable<LanguageServerProtocol.PositionRange>(wrappedValue: Range(Position(line: 4, utf16index: 4)..<Position(line: 4, utf16index: 11))), children: Optional([]))])))
)
sourcekit: [2:sourcekitd_send_request-before:4867: 0.0205] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "-swift-version",
    "5",
    "-I",
    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
    "-package-description-version",
    "5.6.0",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
  ],
  key.offset: 57,
  key.sourcefile: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
}
sourcekit: [4:handleInternalUIDRequest:4867: 0.0206] service queried UID for: source.request.editor.close
[2022-05-22 15:25:24.970] SourceKitServer: Request<textDocument/references>(
  id: "E3F7350E-5F76-4CB8-A1BC-2E66E67D17EA",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: ReferencesRequest(textDocument: LanguageServerProtocol.TextDocumentIdentifier(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift)), position: Position(line: 4, utf16index: 0), context: LanguageServerProtocol.ReferencesContext(includeDeclaration: false))
)
sourcekit: [4:handleInternalUIDRequest:4867: 0.0207] service queried UID for: source.request.editor.replacetext
[2022-05-22 15:25:24.970] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "-swift-version",
    "5",
    "-I",
    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
    "-package-description-version",
    "5.6.0",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
  ],
  key.offset: 57,
  key.sourcefile: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
}
sourcekit: [4:handleInternalUIDRequest:4867: 0.0208] service queried UID for: source.request.editor.formattext
sourcekit: [4:handleInternalUIDRequest:4867: 0.0208] service queried UID for: source.request.editor.expand_placeholder
sourcekit: [4:handleInternalUIDRequest:4867: 0.0209] service queried UID for: source.request.editor.open.interface
sourcekit: [4:handleInternalUIDRequest:4867: 0.0210] service queried UID for: source.request.editor.open.interface.header
sourcekit: [4:handleInternalUIDRequest:4867: 0.0211] service queried UID for: source.request.editor.open.interface.swiftsource
sourcekit: [4:handleInternalUIDRequest:4867: 0.0212] service queried UID for: source.request.editor.open.interface.swifttype
sourcekit: [4:handleInternalUIDRequest:4867: 0.0213] service queried UID for: source.request.editor.extract.comment
sourcekit: [4:handleInternalUIDRequest:4867: 0.0213] service queried UID for: source.request.convert.markup.xml
sourcekit: [4:handleInternalUIDRequest:4867: 0.0215] service queried UID for: source.request.editor.find_usr
sourcekit: [4:handleInternalUIDRequest:4867: 0.0215] service queried UID for: source.request.editor.find_interface_doc
sourcekit: [4:handleInternalUIDRequest:4867: 0.0216] service queried UID for: source.request.module.groups
sourcekit: [4:handleInternalUIDRequest:4867: 0.0216] service queried UID for: source.request.syntacticrename
sourcekit: [4:handleInternalUIDRequest:4867: 0.0217] service queried UID for: source.request.find-syntactic-rename-ranges
sourcekit: [4:handleInternalUIDRequest:4867: 0.0218] service queried UID for: source.request.codecomplete.close
sourcekit: [4:handleInternalUIDRequest:4867: 0.0219] service queried UID for: source.request.codecomplete.cache.ondisk
sourcekit: [4:handleInternalUIDRequest:4867: 0.0219] service queried UID for: source.request.codecomplete.setpopularapi
sourcekit: [4:handleInternalUIDRequest:4867: 0.0220] service queried UID for: source.request.codecomplete.setcustom
sourcekit: [4:handleInternalUIDRequest:4867: 0.0220] service queried UID for: source.request.statistics
sourcekit: [4:handleInternalUIDRequest:4867: 0.0221] service queried UID for: source.request.codecomplete
sourcekit: [4:handleInternalUIDRequest:4867: 0.0222] service queried UID for: source.request.codecomplete.open
sourcekit: [4:handleInternalUIDRequest:4867: 0.0223] service queried UID for: source.request.codecomplete.update
sourcekit: [4:handleInternalUIDRequest:4867: 0.0223] service queried UID for: source.request.typecontextinfo
sourcekit: [4:handleInternalUIDRequest:4867: 0.0224] service queried UID for: source.request.conformingmethods
sourcekit: [4:handleInternalUIDRequest:4867: 0.0224] service queried UID for: source.request.indexsource
sourcekit: [4:handleInternalUIDRequest:4867: 0.0225] service queried UID for: source.request.cursorinfo
sourcekit: [2:sourcekitd_send_request-after:4867: 0.0528] {
  key.internal_diagnostic: "Unable to resolve cursor info."
}
sourcekit: [4:handleInternalUIDRequest:2567: 0.0528] service queried UID for: source.notification.editor.documentupdate
[2022-05-22 15:25:25.002] {
  key.internal_diagnostic: "Unable to resolve cursor info."
}
[2022-05-22 15:25:25.002] {
  key.notification: source.notification.editor.documentupdate,
  key.name: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift"
}
sourcekit: [2:sourcekitd_send_request_sync-before:2567: 0.0530] {
  key.request: source.request.editor.replacetext,
  key.name: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift",
  key.offset: 0,
  key.length: 0,
  key.sourcetext: ""
}
sourcekit: [4:handleInternalUIDRequest:5639: 0.0531] service queried UID for: source.lang.swift.ref.class
[2022-05-22 15:25:25.002] SourceKitServer: Response<textDocument/references("E3F7350E-5F76-4CB8-A1BC-2E66E67D17EA")>(
  success([])
)
sourcekit: [4:handleInternalUIDRequest:5639: 0.0532] service queried UID for: source.lang.swift.ref.function.method.class
[2022-05-22 15:25:25.002] {
  key.request: source.request.editor.replacetext,
  key.name: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Package.swift",
  key.offset: 0,
  key.length: 0,
  key.sourcetext: ""
}
sourcekit: [4:handleInternalUIDRequest:4867: 0.0532] service queried UID for: source.diagnostic.stage.swift.sema
[2022-05-22 15:25:25.002] SourceKitServer: Notification<textDocument/publishDiagnostics>(
  clientID: ObjectIdentifier(0x0000600001704e80),
  params: PublishDiagnosticsNotification(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Package.swift), version: Optional(1), diagnostics: [])
)
[2022-05-22 15:25:25.002] updatedSemanticTokens(response:for:) took 0.02 ms to execute

ahoppen · May 23, 2022, 8:17am

The issue with the first log appears to be that we don’t get compiler arguments when opening the documents. In particular we don’t have an -sdk and -target setting and thus we don’t know where to look for the Swift standard library, which results in the “Loading the stdlib failed” error.

In the second log we seem to be getting compiler arguments (did you change anything between the two captures?) and so there also is no longer an error about stdlib loading. I suspect the reason why you don’t get any results here is that there’s some issue on your side around which offset you pass. I tried running Codeface locally and it sent a cursor info request for Package.swift in the line after import PackageDescription. The offset for cursor info should be zero-based and point to the start of the token you want to retrieve cursor info for. Could you check that the offsets you pass are indeed correct?

// swift-tools-version: 5.6
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription
#^HERE^#
let package = Package(
<snip>

flowtoolz · May 23, 2022, 12:06pm

Thank you for the many pointers

Is there anything the client can do to provide these arguments? Are they generally available (since other requests need and get them) or only strictly required for symbol references?

The second is a) from the actual client (LSPService) which directly launches sourcekit-lsp (so there might be more output visible from the sk-lsp executable), and b) from opening an even simpler project (SwiftyToolz).

Oh great, I indeed missed that Loading this simpler project, the only error is Unable to resolve cursor info. So then I'll debug with SwiftyToolz for now and tackle this error first.

I pass it the range start position contained in the symbol I got from the symbols request. But let me check whether the offsets sent actually make sense, then I'll report back ...

ahoppen · May 23, 2022, 1:12pm

SourceKit-LSP should determine compiler arguments automatically. If it can’t do that, that could either be a bug in SourceKit-LSP or something is broken in your setup. If you can still reproduce it, it would be worth trying to figure out what’s going wrong.

I think you should use the start of the selectionRange instead of range. At least for a Package.swift, range contains the entire let package = Package(…) while selectionRange contains package, which is probably what you want. Also, I was incorrect in my previous post, the location is actually in front of let instead of on the empty line. I think I made some off-by-one error.

flowtoolz · May 24, 2022, 2:07pm

Opening that very simple package and using selectionRange now avoids all errors However I haven't received references yet.

I guess somethin in my setup is wrong. I reinstalled Xcode and its command line tools, reset xcode-select and rebuilt the analyzed package. I've got no idea what to look for.

Here's a shorter log, just showing what happens during one references request, for a symbol (class Log) which is definitely referenced, even from the same file. Also, Position(line: 122, utf16index: 13) is correct:

sourcekit: [2:sourcekitd_send_request-before:8967:33.2505] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "-module-name",
    "SwiftyToolz",
    "-incremental",
    "-emit-dependencies",
    "-emit-module",
    "-emit-module-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/SwiftyToolz.swiftmodule",
    "-parse-as-library",
    "-c",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/BuildEnvironment.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Clipboard.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Color.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Copyable.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Font.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/JSON.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Log.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/ReadableError.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Array+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Dictionary+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Int+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Optional+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/String+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/SwiftHelpers.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/WeakArray.swift",
    "-I",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug",
    "-target",
    "arm64-apple-macosx10.10",
    "-swift-version",
    "5",
    "-enable-batch-mode",
    "-index-store-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/index/store",
    "-Onone",
    "-enable-testing",
    "-g",
    "-j10",
    "-DSWIFT_PACKAGE",
    "-DDEBUG",
    "-module-cache-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/ModuleCache",
    "-parseable-output",
    "-parse-as-library",
    "-emit-objc-header",
    "-emit-objc-header-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/SwiftyToolz.build/SwiftyToolz-Swift.h",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
    "-F",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks",
    "-I",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib",
    "-L",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib",
    "-working-directory",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz"
  ],
  key.offset: 3373,
  key.sourcefile: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Log.swift"
}
sourcekit: [4:handleInternalUIDRequest:8967:33.2507] service queried UID for: source.request.editor.formattext
sourcekit: [4:handleInternalUIDRequest:8967:33.2508] service queried UID for: source.request.editor.expand_placeholder
[2022-05-24 14:50:39.594] SourceKitServer: Request<textDocument/references>(
  id: "35D7EF8E-34EB-4CBB-8A29-0A0E45268BC7",
  clientID: ObjectIdentifier(0x0000600002c01600),
  params: ReferencesRequest(textDocument: LanguageServerProtocol.TextDocumentIdentifier(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/seb/Desktop/GitHub%20Repos/SwiftyToolz/Code/Log.swift)), position: Position(line: 122, utf16index: 13), context: LanguageServerProtocol.ReferencesContext(includeDeclaration: false))
)
[2022-05-24 14:50:39.594] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "-module-name",
    "SwiftyToolz",
    "-incremental",
    "-emit-dependencies",
    "-emit-module",
    "-emit-module-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/SwiftyToolz.swiftmodule",
    "-parse-as-library",
    "-c",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/BuildEnvironment.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Clipboard.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Color.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Copyable.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Font.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/JSON.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Log.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/ReadableError.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Array+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Dictionary+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Int+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/Optional+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/String+SwiftyToolz.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Swift+SwiftyToolz/SwiftHelpers.swift",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/WeakArray.swift",
    "-I",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug",
    "-target",
    "arm64-apple-macosx10.10",
    "-swift-version",
    "5",
    "-enable-batch-mode",
    "-index-store-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/index/store",
    "-Onone",
    "-enable-testing",
    "-g",
    "-j10",
    "-DSWIFT_PACKAGE",
    "-DDEBUG",
    "-module-cache-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/ModuleCache",
    "-parseable-output",
    "-parse-as-library",
    "-emit-objc-header",
    "-emit-objc-header-path",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/.build/arm64-apple-macosx/debug/SwiftyToolz.build/SwiftyToolz-Swift.h",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
    "-F",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks",
    "-I",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib",
    "-L",
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib",
    "-working-directory",
    "/Users/seb/Desktop/GitHub Repos/SwiftyToolz"
  ],
  key.offset: 3373,
  key.sourcefile: "/Users/seb/Desktop/GitHub Repos/SwiftyToolz/Code/Log.swift"
}
sourcekit: [4:handleInternalUIDRequest:8967:33.2509] service queried UID for: source.request.editor.open.interface
sourcekit: [4:handleInternalUIDRequest:8967:33.2510] service queried UID for: source.request.editor.open.interface.header
sourcekit: [4:handleInternalUIDRequest:8967:33.2511] service queried UID for: source.request.editor.open.interface.swiftsource
sourcekit: [4:handleInternalUIDRequest:8967:33.2511] service queried UID for: source.request.editor.open.interface.swifttype
sourcekit: [4:handleInternalUIDRequest:8967:33.2511] service queried UID for: source.request.editor.extract.comment
sourcekit: [4:handleInternalUIDRequest:8967:33.2512] service queried UID for: source.request.convert.markup.xml
sourcekit: [4:handleInternalUIDRequest:8967:33.2513] service queried UID for: source.request.editor.find_usr
sourcekit: [4:handleInternalUIDRequest:8967:33.2513] service queried UID for: source.request.editor.find_interface_doc
sourcekit: [4:handleInternalUIDRequest:8967:33.2514] service queried UID for: source.request.module.groups
sourcekit: [4:handleInternalUIDRequest:8967:33.2515] service queried UID for: source.request.syntacticrename
sourcekit: [4:handleInternalUIDRequest:8967:33.2516] service queried UID for: source.request.find-syntactic-rename-ranges
sourcekit: [4:handleInternalUIDRequest:8967:33.2516] service queried UID for: source.request.codecomplete.close
sourcekit: [4:handleInternalUIDRequest:8967:33.2517] service queried UID for: source.request.codecomplete.cache.ondisk
sourcekit: [4:handleInternalUIDRequest:8967:33.2517] service queried UID for: source.request.codecomplete.setpopularapi
sourcekit: [4:handleInternalUIDRequest:8967:33.2518] service queried UID for: source.request.codecomplete.setcustom
sourcekit: [4:handleInternalUIDRequest:8967:33.2519] service queried UID for: source.request.statistics
sourcekit: [4:handleInternalUIDRequest:8967:33.2519] service queried UID for: source.request.codecomplete
sourcekit: [4:handleInternalUIDRequest:8967:33.2520] service queried UID for: source.request.codecomplete.open
sourcekit: [4:handleInternalUIDRequest:8967:33.2520] service queried UID for: source.request.codecomplete.update
sourcekit: [4:handleInternalUIDRequest:8967:33.2521] service queried UID for: source.request.typecontextinfo
sourcekit: [4:handleInternalUIDRequest:8967:33.2521] service queried UID for: source.request.conformingmethods
sourcekit: [4:handleInternalUIDRequest:8967:33.2522] service queried UID for: source.request.indexsource
sourcekit: [4:handleInternalUIDRequest:8967:33.2522] service queried UID for: source.request.cursorinfo
sourcekit: [4:handleInternalUIDRequest:8967:33.2534] service queried UID for: source.lang.swift
[2022-05-24 14:50:39.597] SourceKitServer: Response<textDocument/references("35D7EF8E-34EB-4CBB-8A29-0A0E45268BC7")>(
  success([])
)

ahoppen · May 24, 2022, 4:28pm

Curious. I wonder why we don’t see a response to the cursorinfo request. Could you push your latest state of LSPService and Codeface to a branch so I can try reproducing it locally?

flowtoolz · May 24, 2022, 4:48pm

For brevity, I had omitted the logs which contain the extensive syntax trees ...

Anyway, both repos are pushed. I can't thank you enough for your effort! I'll think of a task in the sk-lsp repo that I can do to give somethin back, at least documenting all my insights, which I wanted to do anyway when Ben had helped me a while back. Maybe I could add a trouble shooting FAQ for client developers or somethin ...

tikimcfee · May 24, 2022, 9:05pm

Hey there! Chiming in on this as I've also been trying to find the right sauce to make this fly. Seeing the details posted so far is great, as I've actually run in to not only the stdlib error, but a great many hard to parse errors around workspaces, compilation directives, and malformed requests. I admit to spending a lot of time grepping around the core libraries for error messages to understand some of these error flows!

As a part of debugging this, I'd like to offer my support as well to add test harnesses, documented samples, or anything else that would help elucidate use cases and desired API surfaces for users of sourcekit-lsp. Since it reaches into so many other parts of the development toolkit, running through this process might also ferret out other underlying warts that could use tweaking or validation. Grabbing and validating compiler arguments, understanding limitations around Xcode vs. SPM projects, stuff like that.

I'm an eager early-adopter of the service layer being developing with LSPService, and will gladly offer requested logs or trials using any code paths identified that may be incorrect or unstable.

Thanks for your time, and a great many more thanks for the assistance in understanding the complexities of this (quite powerful and useful) language tool.

ahoppen · May 25, 2022, 7:25am

I just ran Codeface with the current state and I think there’s still an issue with your offsets. When I look at the log, I don’t see a request for Position(line: 122, utf16index: 13) for Log.swift, instead, I see the following

Request<textDocument/references>(
  id: "27DAEA89-F559-4278-A056-F0A55FC5B6DD",
  clientID: ObjectIdentifier(0x000060000185c080),
  params: ReferencesRequest(textDocument: LanguageServerProtocol.TextDocumentIdentifier(uri: LanguageServerProtocol.DocumentURI(storage: file:///Users/alex/Downloads/SwiftyToolz-master/Code/Log.swift)), position: Position(line: 122, utf16index: 7), context: LanguageServerProtocol.ReferencesContext(includeDeclaration: false))
)

that points to 122:7, which AFAICT still originates from range and not selectionRange.

ahoppen · May 25, 2022, 7:30am

Support to improve sourcekit-lsp (or any other tool in the open source Swift project) is always very welcome, no matter if it’s through pull requests, more tests, documentation or reproducible bug reports. If you find something that can be improved or that you think might be worth sharing with a wider audience, I would be very happy if you open up a pull request or bug report.

flowtoolz · May 25, 2022, 8:06am

You probably need to update the SwiftLSP package, wich is a dependency of Codeface.

ahoppen · May 25, 2022, 10:05am

Found the issue Turned out to be very simple.

When you are launching sourcekit-lsp, you are only passing the environment variables set in the Config along. Because of this sourcekit-lsp didn’t have your PATH environment variable and thus couldn’t find uname. Hence, it assumed that it was running under Linux and thus looked for the indexer library as libIndexStore.so instead of libIndexStore.dylib, which obviously it can’t find in the Xcode toolchain. Thus it couldn’t perform any index lookups and wasn’t returning any references.

The fix is to update line 199 to something like

process.environment = ProcessInfo.processInfo.environment.merging(config.environmentVariables ?? [:], uniquingKeysWith: { (env, overridden) in overridden })

which inherits the current process’s environment variables if they weren’t overridden in the config.

If you aren’t doing it already, also note that you need to build the package you want to open using swift build from the command line first in order to generate the index, as described in GitHub - apple/sourcekit-lsp: Language Server Protocol implementation for Swift and C-based languages.

flowtoolz · May 25, 2022, 11:11am

I knew I'd feel stupid at the end of this

Before setting the sk log level to 3 for debugging, I had set the process environment to nil, somehow being under the impression this would not amount to "using this method", which, according to documentation, prevents environment from being inherited. But of course, setting it either way deprives sk-lsp of the current environment variables

At least this reinforces, that somethin like LSPService (whether via WebSockets or not) could save client developers quite a few struggles.

Now, it also works on SwiftObserver Applying it to the sourcekit-lsp repo cuts the WebSocket connection at some point, but that's another problem for another time

Thanks again for enduring my ignorance. I'll be back with a visualization of sourcekit-lsp and with a compilation of advice for its client developers.

flowtoolz · May 25, 2022, 11:23am

I guess the only outcome for sourcekit-lsp itself is that it should somehow notify clients when it can't find that indexer library. Isn't that case kind of a big deal? At least for client debugging, it should output a message to stdErr.

ahoppen · May 25, 2022, 1:10pm

Yes, I think that should be the takeaway. We should provide some log output if we can’t determine the host OS or find one of the tools. I’ll create a PR for it in the next couple of days.