Human readable alternative for DocC symbol link disambiguation

Development Swift-DocC
,
1

Hi all,

I'd like to pitch an enhancement to symbol links in Swift-DocC to add a more human readable alternative for link disambiguation based on function signatures.

Introduction

To link to a symbol in DocC, you write the path to the symbol wrapped in a set of double backticks (``), for example ``SlothCreator/Sloth/eat(_:quantity:)``. If two symbols have the same name at a certain scope, the link is ambiguous and need to be disambiguated somehow.

Today, Swift-DocC supports two types of link disambiguation; symbol kind disambiguation (for example ``Color/red-property``) and symbol hash disambiguation (for example ``Sloth/update(_:)-4ko57``). See Formatting Your Documentation Content for more information about the current link disambiguation alternatives.

A link with a symbol kind disambiguation can be read and understood by a developer but a link with a symbol hash disambiguation is incomprehensible to people. For example, it's not clear which of these two functions that update(_:)-4ko57 refers to.

/// Updates the sloth's power.
///
/// - Parameter power: The sloth's new power.
mutating public func update(_ power: Power) {
    self.power = power
}

/// Updates the sloth's energy level.
///
/// - Parameter energyLevel: The sloth's new energy level.
mutating public func update(_ energyLevel: Int) {
    self.energyLevel = energyLevel
}

Symbol kind disambiguation works well when the ambiguous symbols are types, properties, or enum cases but doesn't help to distinguish between functions, operators, subscripts, or initializers where the symbol name is overloaded to accept different parameters or have different return values. Today, these need to use symbol hash disambiguation instead.

Symbol hash disambiguation work in IDE environments where the tooling can generate the hash disambiguation but they're very difficult to work with outside of IDE integrations and the resulting link text—no matter how it was created—is incomprehensible to a person. This, for example, makes it hard to know what symbol a link refers to when reviewing a documentation pull request.

Proposed Solution

As a human readable and human writable alternative to symbol hash disambiguation, I propose that we add support for disambiguating symbol links with "some amount of function signature information". This information is already available in symbol graph files as the "functionSignature" mix-in.

The syntax I propose for this is based how function signatures are written in Swift and many other languages; (ParameterTypes) -> ReturnTypes. Parameter types or return types that are the same across multiple symbols can be written as "_" to avoid needing to write the full function signature. If only the return type information or only the parameter type information is sufficient to disambiguate the link, the other part of the type signature may be omitted.

With this syntax, the two update(_:) functions above could be written as Sloth/update(_:)-(Power) and Sloth/update(_:)-(Int) indicating that they each take one parameter and that it is a Power value in the first link and an Int value in the second link. Like other link disambiguation, the symbol name and the disambiguation would be separated by a dash ("-").

Detailed Design

To get a better sense for this syntax, let's look at an example with more parameters and more return values. Consider this Swift function:

func something(one: Int, two: String, three: Bool) -> (Double, Int) { ... }

If there's another function with the same name but a single return value -> Double, then the two functions can be disambiguated with ->(_,_) and ->_ to indicate that the link returns either a two-element tuple or a single value. The developer could also fill in one or more of the types in the disambiguation, writing it like ->(Double,_), ->(_,Int), or ->(Double,Int) to be more explicit. Similarly, the second link could also be written like ->Double, ->(_) or ->(Double).

If instead the other function has the same return types but its second argument was a Substring instead of a String, then those two functions can be can be disambiguated with -(_,String,_) and -(_,Substring,_) to indicate the type of the 2nd parameter. Like with tuple return values, the developer could chose to fill in the two unspecific types if they prefer to be more explicit.

Parameters would always be written surrounded by parenthesis, even if there's only one parameter. Parenthesis around the return types is optional when there's only one return type.

This new syntax would be supported in symbol links (and documentation links referring to symbols) but DocC wouldn't use type signature disambiguation in file names or web URLs.

In diagnostics for ambiguous links, DocC would suggest the minimal necessary disambiguation, preferring to disambiguate links in this order;

  • kind
  • return type(s)
  • parameter types
  • hash

Return types are suggested before parameter types because functions often return fewer values than they have parameters.

It would be supported to specify both parameters and return values, but the parameters must be specified before the return values. This restriction is to help with readability.

An early implementation of this experimental support can be found in this PR. The actual implementation—once merged—would exist behind a feature flag until we're confident in its quality and in the overall direction.

In that implementation, the link can use either kind and hash disambiguation or type signature disambiguation. This restriction is arbitrary, and could be lifted, but there shouldn't be necessary for link disambiguation to mix the two. If a piece of information doesn't disambiguate the link it can simply be omitted, leaving only the pieces that do disambiguate the link.

This isn't a replacement for symbol hash disambiguation. DocC will continue do support that as the fallback when this syntax can't disambiguate a link or when the developer prefers it for its exactness. The goal is to not need to use symbol hash disambiguation in many common cases but it's not a goal for this syntax would be able to fully describe and disambiguate every function. There's a trade-off between ease-of-use in the common case and richness and wide support.

Open Questions:

What's the right trade-off between brief disambiguation and preciseness of type information?

The implementation so far doesn't have syntax for parameter types or return types that are arrays, optionals, dictionaries, or generics. For example, a Range<Int> parameter is currently spelled as Range.

These could be added over time as non-breaking future syntax enhancements to allow developers to use the function type disambiguation to be used on more cases. We could also reverse this and start by always specifying the full generic type name and add syntax simplifications over time.

This question also applies to other languages, for example C or Objective-C, regarding how pointer parameters are spelled. Would developers expect to refer to a string parameter/return type in Objective-C as NSString or NSString*

How should closure types be spelled?

The implementation so far doesn't have syntax for parameters or return types that are closures. It's not clear to me if a closure type parameter disambiguation would be too complex looking to be readable and writable. For example, a hypothetical disambiguated link for reduce(_:_:) could be written as
reduce(_:_:)-(_,(Result,Element)->Result). Would such a syntax be discoverable enough that a developer could write it without IDE support?

Alternatives Considered

We considered a few different syntax alternatives. The main alternative spelled the parameter types inline with the function name, for example; Sloth/update(_:Power). However—ignoring the difficulties of paring this syntax—we found that it wasn't always easy to know if a word referred to the type of the previous parameter or the name of the next parameter. Adding the parameters inline could also lead to an inconsistent syntax for return values—which comes last in a Swift declaration but first in a C or Objective-C declaration.

Instead of simplifying the spelling of generic parameter or return types to only the wrapping type (for example Range), we considered always requiring the full spelling (for example Range<Int>). This would allow the current implementation to work in more cases at the cost of brevity.

Another alternative syntax we considered uses both the argument label and parameter name, for example Sloth/update(_ power:). Both this syntax and the inline-parameter-type-syntax raised questions about how the syntax would look if it wasn't consistently to all parameters and how it would apply to other languages. We considered how IDEs could improve the legibility of these syntax variations but the goal of the syntax is to be readable without IDE features.

We also considered using another characters than "-" do separate the type signature disambiguation from the symbol name but didn't find another good separator character. Using : would read well when the full type signature is written but could look a bit odd when only the parameters or only the return value was specified.

Specifying only the minimal amount of return types or parameters types has a risk of an unambiguous link becoming ambiguous in the future when another symbol is added. This can be avoided by always spelling out all the parameters or even the entire signature, but that makes the disambiguation much longer. Ultimately, I feel that a developer who wants a future proof disambiguation can chose to use symbol hash disambiguation instead.

I'm looking forward to your feedback on the proposed syntax.

– David

13 Likes
2

I had to skim through the proposal because I’m in a rush this morning, but it looks great to me! As someone who doesn’t use Xcode, it’d be great to be able to write disambiguations by hand (and even for people who do use Xcode, this is a game changer for readability). Big +1 from me; I like the idea of providing partial signatures with just enough info to disambiguate.

1 Like
3

This is a fantastic idea—I support anything that lessens the need for hash-based disambiguation and this seems like a pragmatic approach that optimizes for simplicity/the user experience while covering most cases of disambiguation.

I like that you can incrementally specify the disambiguation, but I wonder if ya'll considered an approach where you could support copy-pasting the entire declaration from code into the link in some way, e.g. Sloth/func update(_ power: Power) -> Foo. This could also help with disambiguating different symbol types e.g. enums and enum cases. I'm trying to think through how I'd write or update one of these links outside of an IDE with autocomplete—I'd probably have the function declaration up next to the docs to refer to as I write the link, so it seems like extra work to have to reformat the decl as a name followed by a type signature.

Also, regardless of the above, can/should we support arbitrary whitespace inside the link? Now that in-source links don't need to match web urls, we can break a lot of rules :slight_smile:

3 Likes
4

how would we parse these? would there be multiple equivalent spellings for the same declaration, such as

Sloth/func update(_ power:some Power) -> Foo
Sloth/func update<P:Power>(_ power:P) -> Foo
Sloth/func update<P>(_ power:P) -> Foo where P:Power

?

i’ll also add that - since i’m picking up a lot of criticism of FNV-1 hashes in this thread - that hashes do have the notable benefit of being short and fixed-length. some of my API signatures can get very long, and having to do linebreaking in a doccomment to disambiguate is not fun.

2 Likes
5

We haven't, but syntactically it would be possible. The biggest issue I foresee with using the entire declaration is that it's difficult to apply incrementally and any incremental behavior may need to be implemented per language. This would likely result in links becoming significantly longer when they need to be disambiguated.

There are also parts of the declaration that developers wouldn't necessarily know without looking at the declaration, for example the declaration for sort(by:) is

mutating func sort(by areInIncreasingOrder: (Self.Element, Self.Element) throws -> Bool) rethrows

A developer may be able to deduce that it's mutating and that it rethrows but they wouldn't be able to find that the parameter name is "areInIncreasingOrder" without looking at the original declaration.

Yes, we can use whitespace in the link. One of the alternatives considered is using whitespace as separator between the symbol name and the disambiguation.

I'm mostly not sure what we'd want to use whitespace for yet. We could allow it arbitrarily and ignore it but that could prevent us from using it for something else in the future (for example as a disambiguation separator).

Yes, the hashes are nice and short, stable, and sufficiently unique (for what they're used for). They definitely have their place and will continue to be supported.

This pitch is about providing the developer with more alternatives for how they want to disambiguate the link; defaulting to the more human readable disambiguation where possible.

2 Likes
6

Since the proposed syntax seems to be well received I've continued to work on this implementation. There's still a bit more work to do but I'll update this thread when this is available in a build—behind a feature flag—and people can try it out in their own projects.

5 Likes
7

It would be really nice to avoid the hashes, but...

(You're likely far into this implementation, so I apologize for the lateness of this brainstorming from the peanut gallery.)

I'm uncomfortable with the idea of minimizing based on known alternatives. Doesn't this means that adding a function (esp. by extension) can render existing docc links ambiguous? And it also means that there are multiple disambiguating forms one might choose? Can one use a typealias to shorten the form? (esp. for generics, parameter packs, complete tuples..) (As defined where?)

I would prefer an algorithm that could generate the link just by looking at the method, i.e., by using its full signature (i.e., as used for lookup). Ideally it would be a syntactical rendering provided by swift-syntax, reduced to URL form (even available in macros?).

For writing, when computers generate the links, the length is not a problem. When people generate them, having fixed rules based only on the signature is a nice property, and obviously tooling would help.

For reading, the full signature would be a benefit to both humans and computers.

Then the current rules for a minimized signature are really an exception permitted in limited cases.

Or: why not declare id's for ambiguous targets, or just for clarity's sake?

A completely separate option is to use an attribute to define the anchor-id of an ambiguous/overridden target. Moving the onus to the link referent/target rather than the link writer tracks change-responsibility for naming targets, and could enable some name refactorings to proceed without having to update docc comments. When a developer adds a new ambiguous target, she has the onus of defining the doc-id attribute, and the existing links safely point to their original referents. (i.e., one can refer to that target only by using the id)

Indeed, the id might be preferred as a more readable form whenever the target has a complex signature. Because the id is both read and written by humans, developers would work out some style guidelines and conventions for these names (perhaps with most defaulting to generated id's based on the minimization rules being proposed). Id's would have the benefit of being easily searched, since the form is the same for declaration and use, while the current links require tooling support and something like demangling. Hand-written id's might also be tailored for clarity or content-assist. And of course they should I'd prefer they be URL-compliant identifiers from the get-go.

Id's could likely be supported immediately and forever, with accessory tooling for generating minimized forms evolving over time. A big benefit of declare-side id's is that they permit refactoring and provide a fast index over link targets. Indeed, links could be implemented by having the compiler generate default doc-id attributes, if none are declared explicitly. Tooling then only consumes these doc-id attributes, whether implicit or explicit.

There are other benefits to declaring an id. E.g., in complex types, some features act as a team for the type to play a role. The id's for all the features could reflect the common role, providing some semantic grouping unconstrained by the field/function names which are typically otherwise constrained by protocol requirements or naming conventions.

Using an attribute also means you can use macros to generate them. That means the doc-id solution zoo has three forms: implicit (by rule, from the docc/compiler team), explicit (arbitrary, from the declaring developer), and tailored (by custom macro rules at the team or package level). Having the docc team responsible only for the implicit rules would enable them to cover the main cases (and probably 95% of the links) without having to disambiguate every conceivable declaration.

1 Like
8

No worries. Better now than when these changes are merged and available in a build (but even then we can still change things).

Yes, it does. I briefly discuss this issue at the end of the "Alternatives Considered" section:

Even today, adding a new symbol carries a risk of making existing links ambiguous. For example, if only

mutating public func update(_ power: Power)

exist then links to that function doesn't require any disambiguation. If I later add

mutating public func update(_ energyLevel: Int)

then any existing ``Sloth/update(_:)`` links would become ambiguous. There's a tradeoff between making links easy to read and write and making links more isolated from external changes. We could solve this by always requiring the hash disambiguation on every link but it would make links harder to write and harder to read.

So far, we've prioritized writability/readability and don't require any disambiguation until the link becomes ambiguous. Still, the link syntax allows for developers to add disambiguation, even when it's not needed, so a developer who wants to make a different tradeoff could do so and add hash disambiguation to all their links. We don't have any tools to help them do so, but the link syntax supports it.

This is already the case today. The link syntax supports both "kind" disambiguation and "hash" disambiguation and developers can chose to over-specify disambiguation as combinations of these two (including specifying the language). For example, consider the following declaration:

public struct Color {
    public var red, green, blue: Double

    public static let red = Color(red: 1.0, green: 0.0, blue: 0.0)
}

The most readable way to disambiguate a link to the red color component (the property) would be Color/red-property but these alternatives are also supported:

  • Color/red-swift.property
  • Color/red-b7hgx
  • Color/red-property-b7hgx
  • Color/red-swift.property-b7hgx

DocC prefers the "kind" disambiguation because its the most readable and writable but the "hash" disambiguation is also supported and would be a more robust.

No, we don't have any such capabilities for symbol links. It'd be happy to discuss specific examples (even hypothetical ones) if there are cases where you feel like the disambiguation would be unfeasibly long.

In practice I believe that the disambiguation should be "somewhat short" in most cases since it only needs to include the one parameter type or return type that's unique for that symbol. If that happens to be a large tuple or complex closure type that's barely readable/writable then the hash disambiguation remains a short alternative (that's also not readable/writable).

I'm not sure I have a good sense of what such a link would look like. Could you give an example?

If it's allowing the developer to specify part of the type signature then it sounds similar to what the syntax that I'm proposing. If it brings in other parts of the symbol declaration then that brings other ambiguity problems, but I'd be happy to go into more detail over some examples.

I think this is a really cool idea with great potential and something that we should discuss more. They each address the problem from different angles and I see them as different independent solutions that could possibly coexist to give the developer even more choices in the future.

1 Like
9

as a general principle, i think that designing identity systems is a very difficult and costly process, and that you should avoid having multiple identity systems for the same domain whenever possible.

for a swift declaration, the truest identity is its mangled name - it encodes everything intrinsic about the declaration, even if it is not possible to read back this information unless you are the swift runtime. but the problem with the mangled name is twofold:

  1. mangled names can get very (thousands of characters) long
  2. it is not possible to discover the mangled name of a declaration without performing some kind of disassembly, which is unacceptable when you are trying to write documentation quickly

which then brings us to the FNV-1 hashes, which solve problem #1, although there is still no way to discover the FNV-1 hash of a symbol you want to link to on linux. (XCode users are quite fortunate to have a tool to perform the hash lookup.)

in my opinion, the problem is that it is too difficult to look up the hashes of the declarations you want to link to. i think that designing and teaching a bespoke disambiguation syntax to avoid having to deal with hashes is not the right approach. we need to make the hashes easier to work with instead of trying to replace them with a more complex and less comprehensive alternative.

2 Likes
10

Discoverability is one of the problems but readability is another problem.

Even if a developer could easily look up the FNV-1 hash for a certain symbol and write the link

``update(_:)-4ko57``

That developer (or someone else on their team) can't easily tell, after the fact, if that link refers to update(_ power: Power) or update(_ energyLevel: Int). This problem could for example show up in a code review or when someone is reading the code/markup outside of an IDE.

The developer could mitigate this by adding a comment in the markup that clarity what symbol the link is referring to, for example

``update(_:)-4ko57`` <!-- Power -->

but it would the developers responsibility to ensure that such a comment is correct and maintained.

3 Likes
12

between now and last July when i posted this, i’ve discovered a few things about how the hashes behave that have led me to reach a different conclusion.

we’ve known for a while that DocC hashes basically force ABI stability on packages that might not actually have any ABI stability at all. for example, going from

func foo(x:__owned [Int])

to

func foo(x:consuming [Int])

is an API compatible change, but it is ABI breaking and therefore, also documentation-breaking.

i used to think this was only a hassle when making changes to packages, but as it turns out, this also prevents collaboration between developers contributing to the same version of a package on different platforms. because ABI is different on different platforms (duh.)

for a real-world example, consider Alamofire, which supports both linux and macOS. Alamofire contains many APIs that use symbols from Foundation, such as HTTPURLResponse, in DataStreamRequest.onHTTPResponse(on:perform:). the mangled name of HTTPURLResponse is different depending on what platform you are on, which means the DocC hash needed to reference that method is also different depending on what platform you are on.

practically, this means a project’s documentation must standardize on a “canonical platform”, (likely macOS) otherwise contributors will not be able to agree on a set of hashes to use. this makes it impossible for contributors on a different platform to write documentation for the package, as they have no way of knowing the right hashes to use.

(conversely, because i write packages for linux, this means that nobody on macOS can contribute to my packages, if they must update documentation when making changes.)

tooling improvements cannot realistically help in this situation, since the tooling would need to be able to generate the interfaces for a different platform and at that point you are close to just cross-compiling the package itself. which is challenging if you don’t have access to the necessary SDK.

TLDR; using FNV-1 hashes is the same as requiring all developers to contribute from the same platform, which is incompatible with a project’s goal of supporting multiple platforms.

7 Likes
13

As I was going through creating the documentation for Time (in particular this page, source here), I found myself wishing I could manually specify the disambiguation hashes for easier reading, maybe via an - Anchor: ... tag in the docs that would get parsed and used in the URL.

@taylorswift would being able to specify the anchor/hash address your concerns? (I would expect that colliding hashes would produce an error while compiling the documentation)

14

yes, i think that would work. i think that best practice should be to always use a random string, as i have had negative experiences with naming/numbering overloads.

generating a random string has to be done ad hoc today but it feels like a simple thing that could be added to existing tooling.

15

I'm not intending to revive this threat but just wanted to post an update for anyone who's interested that this enhancement is available on the release/6.1 branch—which was cut from main on Nov 13—and will be released alongside Swift 6.1.

Updated user-facing documentation about this enhancement is in progress with the plan to publish the updated documentation sometime during the 6.1 release.

3 Likes
16

would it be possible to use an even simpler disambiguation syntax that just matches on an identifier token, without any additional structure? something like

- ``forecast(for:at:)-(DateInterval)``
- ``forecast(for:at:)-(MinuteByMinuteForecast)``
- ``forecast(for:at:)-(Date, HourByHourForecast)``

instead of

- ``forecast(for:at:)-(DateInterval,_)``
- ``forecast(for:at:)->MinuteByMinuteForecast``
- ``forecast(for:at:)->(Date,_)->HourByHourForecast``
17

Not really because positional information is important for uniqueness.

18

it’s already the case that the proposed syntax does not uniquely disambiguate all possible signatures. so the question here is whether the blank parameters (_) and return marker (->) add enough disambiguation power to justify the increased syntactical complexity. it’s easy to contrive scenarios where the pattern matching syntax can disambiguate something a simpler identifier-token based rule cannot, but i’m not convinced this is representative of real world APIs.

19

Positional information is very useful for disambiguation. I counted about ~100 symbols across a handful of Apple's large frameworks where the number of parameters or the number of return values is sufficient to disambiguate overloads without needing to specify a type.

One case where this pattern comes up is when one overload has a return value and the other overload doesn't. For example, these two overloads from Dispatch/DispatchQueue:

func sync(
    execute workItem: DispatchWorkItem
)
func sync<T>(
    execute work: () throws -> T
) rethrows -> T

or these two overloads from Foundation/Data:

func copyBytes(
    to pointer: UnsafeMutablePointer<UInt8>,
    from range: Range<Data.Index>
)
func copyBytes<DestinationType>(
    to buffer: UnsafeMutableBufferPointer<DestinationType>,
    from range: Range<Data.Index>? = nil
) -> Int

can both be disambiguated with ->() and ->_ respectively.

Another place where this patterns comes up is in C and C++ frameworks when each overload has a different number of parameters. For example, these two initializers from AudioDriverKit/IOUserAudioObject:

bool init();
bool init(IOUserAudioDriver *in_audio_driver);

can be disambiguated with -() and -(_). This pattern (where at least one initializer has no argument) happens fairly frequently for C++ code.

This patterns also occurs in C and C++ code when some overloads skip some parameters. For example, these two overloads from NetworkingDriverKit/IOUserNetworkEthernet:

virtual IOReturn registerEthernetInterface(
    IOUserNetworkPacketQueue **queues, 
    uint32_t numQueues, 
    IOUserNetworkPacketBufferPool *txPool, 
    IOUserNetworkPacketBufferPool *rxPool
);
virtual IOReturn registerEthernetInterface(
    ether_addr_t macAddress, 
    IOUserNetworkPacketQueue **queues, 
    uint32_t numQueues, 
    IOUserNetworkPacketBufferPool *txPool, 
    IOUserNetworkPacketBufferPool *rxPool
);

can be disambiguated with -(_,_,_,_) and -(_,_,_,_,_).

That's just a few examples where the type names aren't even involved in the disambiguation.

In other cases, the positional information is what allows a single type name to disambiguate overloads. For example, these four overloads from the Swift API for vDSP:

static func divide<U, V>(
    _ scalar: Float,
    _ vector: U,
    result: inout V
) where U : AccelerateBuffer, V : AccelerateMutableBuffer, U.Element == Float, V.Element == Float

static func divide<U, V>(
    _ scalar: Double,
    _ vector: U,
    result: inout V
) where U : AccelerateBuffer, V : AccelerateMutableBuffer, U.Element == Double, V.Element == Double

static func divide<U, V>(
    _ vector: U,
    _ scalar: Float,
    result: inout V
) where U : AccelerateBuffer, V : AccelerateMutableBuffer, U.Element == Float, V.Element == Float

static func divide<U, V>(
    _ vector: U,
    _ scalar: Double,
    result: inout V
) where U : AccelerateBuffer, V : AccelerateMutableBuffer, U.Element == Double, V.Element == Double

can be disambiguated with -(Float,_,_), -(Double,_,_), -(_,Float,_), and -(_,Double,_) respectively. Without the positional information, it wouldn't be possible to disambiguate overloads like these with only a single parameter type name.

This pattern also comes up in types that implement equality operators with another type, resulting in disambiguated links like:

==(_:_:)-(Int,_)
==(_:_:)-(_,Int)

Real world examples aside, I also personally find that the distinction between parameter types and return types in the syntax makes it easier to understand the disambiguation when reading a link in a documentation comment. If I read ``something(with:and:)->String`` or ``something(with:and:)-(_,String)`` I don't have to guess what the String refers to because the syntax is very predictable.

I expect that developers who encounter this syntax in a fixit suggestion to a DocC warning or in the documentation will be able to figure it out pretty quickly and that once they understand it they'll be able to read and write links like this without needing to refer back to the documentation again exactly because the syntax is so predicable.

2 Likes
20

Both the documentation and this post are inconsistent about whether - or -> separates the return type, as well as if parentheses are required for simple return types. What are the actual rules?

21

I've looked though the in progress documentation again and I can't spot the inconsistency that you're referring to. Would you mind helping me by pointing it out so that I can update the documentation for everyone's benefit?