New DocC supported symbol structure in Swift 5.9 - warnings and a question of how to deal with them

[Joseph_Heck]

I've been iterating through some older projects and rebuilding them with the latest SDK (Xcode 15 beta 2 in this case), and I've noticed that my docc build processes are sprouting a huge number of warnings without anything changing in the code from earlier versions of swift (some 5.8, some back to 5.6).

The (admittedly hacky) process I was using to pull together all the relevant symbols was to generate the documentation archive as normal, including the --emit-digest option, and then doing some data parsing of the generated docs/linkable-entities.json to get all the identifiers. The identifiers themselves have changed up in pattern - where previously symbols such as -, <, and > were transformed to a placeholder _ and disambiguation extensions (such as -6f26z) added, those symbols appear to be now legit.

Hacky process example:

cat docs/linkable-entities.json | jq '.[].referenceURL' -r > all_identifiers.txt
sort all_identifiers.txt \
    | sed -e 's/doc:\/\/com\.github\.heckj\.MeshGenerator\/documentation\///g' \
    | sed -e 's/^/- ``/g' \
    | sed -e 's/$/``/g' > all_symbols.txt

There's some fixits in Xcode that do a nice job of identifying a few mismatches, and prompting for the corrected values, but it doesn't seem to hit all of them. And I'm still seeing some of the _ replacement pattern in place there - I'm assuming for some patterns that still aren't quite legit for use in a URL structure, but I'm not sure where the line is drawn.

So the question drops down to:

Is there a better way to get a list of all the unique symbols so that I can go through and curate them appropriately?

The current process I'm using seems to work reasonably well, but a bug with Xcode where it's not always clearing errors on a clean & rebuild of docs is making this a painful thing to update.

In one particular example, I've updated a symbol to:
- ``MeshGenerator/LineSegment/>=(_:_:)-sn39``

But at the top of the DocC file, I'm still getting a warning from the older symbol structure that it's not legit:
'_(_:_:)-3x8q' doesn't exist at '/MeshGenerator/LineSegment'

example from Xcode:

Is this an erroneous error, or am I mistaking how to set up those symbols?

(and this isn't Xcode specific - same warnings appear in CLI output. this was just easier to present)

[ronnqvist (David Rönnqvist)]

Regarding those specific warnings

@Joseph_Heck

Is this an erroneous error, or am I mistaking how to set up those symbols?

It's hard to say. Based on the screenshot those warnings appear to be reported on line 1 column 1 which some parts of the code use as a fallback when they don't have a source location for a diagnostic but know what file it's in. If the same diagnostics happen when calling DocC on the command line, would you mind opening an issue on the DocC repo so that we can investigate what's wrong? If this only happens in Xcode, please report a feedback instead (since Xcode specific issues are out-of-scope for the DocC open source project).

Regarding links syntax in general in DocC

In 5.9 we switched DocC to a new link resolution implementation which decoupled the link syntax from the restrictions of "topic references" (not a user-facing term). In practice this meant that symbol links and documentation links can support any unicode character—so that links can be spelled the same way as the symbol name is spelled in code—and that links can disambiguate by capitalization.

Previously, symbols like <(_:_:) and >(_:_:) would need to be written as _(_:_:) because the link needed to match the topic reference and the reference couldn't contain "<" or ">". Since these links collide, they needed to be disambiguated and since both symbols are operators, the only other disambiguation was the folded hash of their unique identifiers.

In 5.9, these restrictions still apply to the topic reference (and to the web URL of these pages) but they no longer apply to the links, which can be written as <(_:_:) and >(_:_:) without any disambiguation. Since the referenceURL in the "linkable entities" file is the topic reference, that information can't be used to find the shortest recommended spelling of these links.

The new link resolver implementation doesn't support the links written as _(_:_:) but tries to determine what symbol it's referring to and present suggestions. If the unsupported link contains disambiguation, DocC should be able to suggest the name for that exact symbol. If there are cases where this doesn't work we'd appreciate an issue on the DocC open source repo.

@Joseph_Heck

Is there a better way to get a list of all the unique symbols so that I can go through and curate them appropriately?

Regarding workflows for determining all unique symbols; my general recommendation would be IDE integrations that can offer code completion or similar features to suggest links. This however, isn't a workflow that's available when using DocC directly on the command line. For that we don't have a solution today.

I've been working on an experimental command for DocC to generate markdown representations of the automatic curation—where each symbol is written using its shortest and least disambiguated form—to bring this functionality to command line workflows. However, that command needs some discussion regarding what its command line interface should be and how it should behave. I'm planning on posting about it in the forums at some point to gather feedback, discuss the interface, and talk about how to integrate it as an experimental feature but I'm still catching up on a few other things. Until then, you can build from that branch if you want to see the curation it generates for your project. The PR includes a few examples of the command and it's inputs and behaviors.

[Joseph_Heck]

Thanks David - great detail. I'll do some "grep" work to make sure I haven't missed a symbol reference in the code comments or markdown, and if the error appears erroneous after that, I'll set up an issue for the erroneous error reporting.

(I'll also give your new/experimental command a try - appreciate it!)

UPDATE: I did some further grep checking to make sure I didn't have a symbol stashed anywhere and was still seeing the errors - didn't spot anything - so I opened Incorrect error reporting on older topic-style symbol reference that isn't used in code or documentation markdown · Issue #640 · apple/swift-docc · GitHub with the details to reproduce.

[ronnqvist (David Rönnqvist)]

Summarizing some information from that issue since I don't expect people in the forums to follow along in discussions in the repo;

Those specific warnings are a bug. The warnings are false positives—the links and curation in the built documentation will work as expected. I've fixed the bug on main and will cherry pick the fix for the 5.9 release.

[Joseph_Heck]

Thanks for the fix!