# Symbol Graph Adaptions for Documenting Extensions to External Types in DocC

**URL:** <https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684>\
**Category:** Swift-DocC\
**Tags:** documentation, docc\
**Created:** [April 11, 2022, 12:41pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684 "2022-04-11T12:41:22Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [April 11, 2022, 12:41pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/1 "2022-04-11T12:41:22Z")

</div>

Hi everyone! In [my recent post](https://forums.swift.org/t/document-extensions-to-external-types-using-docc/56060) about DocC we worked out how we want to present extensions to external types in DocC. The proposed solution can be found [here](https://forums.swift.org/t/document-extensions-to-external-types-using-docc/56060/26). Unfortunately, we also found out that the symbol graph files currently do not contain all the information this solution requires and further, that the current structure might not be ideal to ensure easy processing in Swift-DocC.

I am very pleased with the result from the last thread, so I hope we can have a similarly productive discussion regarding the implementation details here!

## Current State

I took the last days to check out the [symbol graph generation](https://github.com/apple/swift/tree/main/lib/SymbolGraphGen) in the Swift compiler and the output it generates.

### AST

In the AST we have easy access to all the information we need. This includes specifically

- comments on extension blocks e.g.:

```swift
/// THIS COMMENT HERE
public extension SomeExternalType: SomeProtocol {
    /* ... */
}

```

- information on the extended type (e.g. its `kind` (`swift.struct` vs `swift.protocol` vs ...))

Those were two areas where we weren't sure if that information would be easily accessible in the UX design phase.

### Symbol Graph Files

Extensions to external types are currently excluded from the module's main symbol graph. I.e. where the main module is `SwiftDocC`, public extensions to the standard library declared in `SwiftDocC` are located in `SwiftDocC@Swift.symbols.json`. This includes the `symbol` of `kind` e.g. `swift.property`, `swift.func`, or `swift.struct` as well as `relationships` of `kind` `memberOf` and `conformsTo`.

The `symbol` has a `swiftExtension.extendedModule` field containing the extended module's name (e.g. `"Swift"`). This also applies to extensions to local types.

Both relationships refer to the actual type declaration in the external module, e.g. `String` with identifier `"s:SS"`. For extensions to external types, the `symbol` for this original type declaration is never present in the same file. It might be emitted as part of another symbol graph file, but e.g. for the Swift standard library it isn't emitted at all (locally).

## Proposed Solution

My general idea is to adapt the symbol graph's structure in a way that best fits the structure in the UX design so that meddling with the graph in Swift-DocC is avoided. Meanwhile, the symbol graph should continue to transport _raw_ data only. Specifically, I wouldn't want to bring any of the heuristics mentioned in the UX pitch discussion into the Swift compiler.

### General Structure

The basis for my proposed solution is @ethankuster 's [reply on the UX thread](https://forums.swift.org/t/document-extensions-to-external-types-using-docc/56060/21). You can find a diagram of my proposed graph structure below:

 ![Symbol Graph Schema](https://global.discourse-cdn.com/swift/original/3X/1/a/1adae76f23054a03e209b6577e09f946cb924f80.png)

We introduce an alternative "extension" version for all type symbol kinds (`struct`, `class`, `enum`, `protocol`), e.g. next to `swift.struct` we also introduce `swift.struct.extension`. There exists **one** such symbol (of the respective kind) in the graph **per extended external type**.

The existing `memberOf` and `conformsTo` relationships (for extended external types) also end/start in these new symbols instead of referring to the original type-declaration's symbol.

Furthermore a new symbol `swift.module.extension` is introduced. This symbol represents the external module that is publicly extended. There exists **one** such symbol in the graph **per imported module** _where at least one type is publicly extended_.

The `swift.TYPE_KIND.extension` symbols are connected to the `swift.module.extension` symbols via a new `memberOf` relationship. (We could also use a different name here if we want to reserve the term "member" for type-members.)

Both these types of symbols carry little information (e.g. no comments) as they have no direct representation in the source code. The first type of symbols basically represents **all** usages of `extension SomeExternalType` whereas the second type of symbol represents **all** usages of the respective `import SomeModule`. I emphasize **all** , because there can be multiple occurrences of both in the source code! Nevertheless, we need these aggregate-symbols in order to achieve the structure that feels natural in DocC as pointed out in the UX discussion.

### Adding Extension Block Comments

If we decide that we want to utilize comments above extension blocks to automatically generate more meaningful documentation on the extended type's documentation pages in Swift-DocC later on, I propose the following solution:

A new symbol of kind `swift.extension` is introduced. There exists **one** such symbol **per extension block with at least one public member**. This symbol carries information such as the comments above the respective extension block.

Additional "memberOf" and "conformsTo" relationships are introduced in accordance with the members/conformance declared in the respective block. The relationships to the `swift.TYPE_KIND.extension` symbols remain intact!

Finally, new `contributesTo` (naming totally up for discussion) relationships are added between the `swift.extension` symbols and the `swift.TYPE_KIND.extension` symbol that correspond to the same extended external type.

These symbols and relationships would be inspected and transformed into synthesized additions to other symbols based on various heuristics in Swift-DocC, and finally removed from the internal symbol graph representation. The prime example for this would be aggregating comments from above extension blocks and adding them to the respective `swift.TYPE_KIND.extension` symbol.

### Unifying the Graphs

The existence of the `swift.module.extension` symbols would also allow us to move all these symbols into the main symbol graph, e.g. they are emitted in the main `SwiftDocC.symbols.json` file, not `SwiftDocC@Swift.symbols.json`. This should also simplify processing the data in Swift-DocC.

Note that all these changes would be hidden behind a flag in the Swift compiler, so that this changed behavior is opt-in for other projects depending on symbol graphs!

## Alternatives Considered

### Synthesizing Artificial Symbols in Swift-DocC

One option would be synthesizing the `swift.TYPE_KIND.extension` and `swift.module.extension` symbols in Swift-DocC. This would make sense as they do not really exist in code and do not really transport any information/metadata beyond the structure they provide.

However, this structure is required in DocC in order to achieve the desired outcome from a UX perspective and therefore the symbols would have to be synthesized after importing the symbol files in Swift-DocC.

While I cannot fully grasp the architectural consequences this would have inside Swift-DocC yet, I think the biggest problem with this approach is the following:

At some point we have imported the symbol graph file in Swift-DocC. At this point the symbol graph is described using [SymbolKit](https://github.com/apple/swift-docc-symbolkit) and it would be relatively easy to synthesize the missing nodes and relationships. However, SymbolKit is _The specification and reference model for the Symbol Graph **File Format**._ Therefore it would be odd to add symbols to SymbolKit that do not exist in the symbol graph files emitted by the Swift compiler. Furthermore, the most important type, [`Symbol`](https://apple.github.io/swift-docc-symbolkit/documentation/symbolkit/symbolgraph/symbol), is a `struct`, so extending the library (in terms of altering the graph's structure) from within Swift-DocC should prove to be difficult.

Synthesizing the pages later on in Swift-DocC is quite difficult I think. This would involve re-writing large parts of the conversion logic as there would be many new rules to follow and tricky situations that just couldn't happen before, e.g. relationships where one end doesn't exist.

### Do Not Unify Symbol Graph Files

An option to keep the changes to the symbol graph files rather low would be to keep extensions to external types in separate files. That would mean we'd have no `swift.module.extension` symbols as this information can be deduced from what file contains a symbol.

Just synthesizing the top level documentation pages for the separate symbol graphs could be manageable inside Swift-DocC, however I'm not sure if the effort is justified. After all, the symbol graph files would still contain new symbols and relationships, which I guess is considered a breaking change. And if we already break it, we can also break it in a way that really fits our needs.

## Discussion Points

Please keep in mind that I'm still rather new to this entire code-base, so I'm really dependent on your expert opinions for these large architectural decisions!

Of course, any feedback is welcome! I also want to bring up three questions to you all right at the beginning:

Firstly, what would the `precise` identifier for all the `X.extension` symbols be? How does the Swift compiler currently generate them? Is there some documentation I could read?

Secondly, should we add a "extensionOf" relationship between `swift.TYPE_KIND.extension` and the respective original declaration symbols (even though this original declaration is not part of the same symbol graph)? I think this could cause problems in the conversion in Swift-DocC again, however it also seems right to have a reference to the original type declaration when considering the discussion about [[SR-15431] Support DocC references to symbols defined in another module · Issue #208 · swiftlang/swift-docc · GitHub](https://bugs.swift.org/browse/SR-15431) in the UX thread. If I have to mess with the symbol graph file format once I guess it would be best to plan ahead a bit.

Finally, for my broader understanding: What role does Swift-DocC play in the universe of Swift symbol graph files? Is it the main reason they exist or just one of many consumers of this interface? What would us introducing a new symbol graph file format mean for the old format? Would both versions have to be maintained next to each other indefinitely, or would the old one phase out after a short(ish) period of time?

---

<div class="post-metadata">

**Author:** ![daniel-grumberg](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/daniel-grumberg/32/22029_2.png) [@daniel-grumberg](https://forums.swift.org/u/daniel-grumberg)\
**Post date:** [April 12, 2022, 11:02am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/2 "2022-04-12T11:02:26Z")

</div>

Love the proposal and thank you for doing this work!

> [@theMomax](#):
>
> Nevertheless, we need these aggregate-symbols in order to achieve the structure that feels natural in DocC as pointed out in the UX discussion.

I am little confused by the point above, why would we need the aggregate symbol for each extended imported module? Could the source module information not be encoded in the extension symbol directly via, a source module field? Unless we need to know more than the name of the extended module, I don't see the need to have a brand new symbol for it. I also think that the alternative where we do one symbol per extension block might be easier to implement in the compiler, merging these should be straight forward in DocC and make the symbol graph a closer match with the original source code. This would be useful, as you pointed out, for extension block documentation comments, and if Swift adds extension block level features. One example I can think of is extension block level default access control specifier, but off the top of my head I think it is reflected in the underlying symbols. I don't think there is anything else like that in the pipeline as far as I know (I also don't know if it would make sense from a language design perspective for Swift to introduce these kinds of features).

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [April 12, 2022, 12:23pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/3 "2022-04-12T12:23:53Z")

</div>

Hi Daniel, thanks for your feedback!

> [@daniel-grumberg](#):
>
> I am little confused by the point above, why would we need the aggregate symbol for each extended imported module? Could the source module information not be encoded in the extension symbol directly via, a source module field? Unless we need to know more than the name of the extended module, I don't see the need to have a brand new symbol for it.

You are totally right, the information carried by my proposed `swift.TYPE_KIND.extension` and `swift.module.extension` symbols already exists in the current symbol graph file format, i.e. it could be reconstructed inside Swift-DocC from e.g. `pathComponents` or the `swiftExtension.extendedModule` fields. I already discussed the problem I see with this approach in the alternatives considered section:

> [@theMomax](#):
>
> At some point we have imported the symbol graph file in Swift-DocC. At this point the symbol graph is described using [SymbolKit](https://github.com/apple/swift-docc-symbolkit) and it would be relatively easy to synthesize the missing nodes and relationships. However, SymbolKit is _The specification and reference model for the Symbol Graph **File Format**._ Therefore it would be odd to add symbols to SymbolKit that do not exist in the symbol graph files emitted by the Swift compiler. Furthermore, the most important type, [`Symbol`](https://apple.github.io/swift-docc-symbolkit/documentation/symbolkit/symbolgraph/symbol), is a `struct` , so extending the library (in terms of altering the graph's structure) from within Swift-DocC should prove to be difficult.
> 
> Synthesizing the pages later on in Swift-DocC is quite difficult I think. This would involve re-writing large parts of the conversion logic as there would be many new rules to follow and tricky situations that just couldn't happen before, e.g. relationships where one end doesn't exist.

At least based on the repository's URL SymbolKit is kind of owned by Swift-DocC, so maybe redefining SymbolKit as a model that is a **superset** of the model in the symbol graph files would be an option, though!

> [@daniel-grumberg](#):
>
> I also think that the alternative where we do one symbol per extension block might be easier to implement in the compiler, merging these should be straight forward in DocC and make the symbol graph a closer match with the original source code. This would be useful, as you pointed out, for extension block documentation comments

I agree that independently of whether or not we add the `swift.TYPE_KIND.extension` and `swift.module.extension` symbols, we should definitely add a symbol for each extension block ( `swift.extension` in my original post) so we can actually transport all the information we need. In case we don't add the `swift.TYPE_KIND.extension` symbol, this symbol would also have to carry the type kind information!

---

<div class="post-metadata">

**Author:** ![daniel-grumberg](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/daniel-grumberg/32/22029_2.png) [@daniel-grumberg](https://forums.swift.org/u/daniel-grumberg)\
**Post date:** [April 12, 2022, 1:44pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/4 "2022-04-12T13:44:14Z")

</div>

CC @Franklin as I am new myself to DocC and he knows more than I do.

> [@theMomax](#):
>
> At least based on the repository's URL SymbolKit is kind of owned by Swift-DocC, so maybe redefining SymbolKit as a model that is a **superset** of the model in the symbol graph files would be an option, though!

I am opposed to this as SymbolKit is the reference for SGF processing. If this was really needed we could create the types and extensions in DocC itself though. However, I really don't think we would need to synthesize special nodes, we could merge them when generating the `DocumentationNode`. However, upon second thought, it might make for less friction to do this at the SGF layer.

---

<div class="post-metadata">

**Author:** ![franklin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franklin/32/17924_2.png) [@franklin](https://forums.swift.org/u/franklin)\
**Post date:** [April 13, 2022, 9:05am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/5 "2022-04-13T09:05:35Z")

</div>

Hi @theMomax! This is looking really great.

I'm hugely in favor of modeling the extended types in the main symbol graph as opt-in behavior. The existing model of emitting extended module APIs in separate files is suited for displaying extension information in their extended module's documentation, but not in the defining module's documentation. For example, you'd include `SwiftDocC@Swift.symbols.json` when building documentation for _Swift_. I find your [proposal](https://forums.swift.org/t/document-extensions-to-external-types-using-docc/56060) much better UX-wise, so it makes sense to me to modify the symbol graph format to cater for that. Modeling the extended types in the main symbol graph allows documentation compilers to continue just representing whatever API hierarchy the symbol graph describes. With this architecture, I believe that the work on the Swift-DocC side would be quite minimal (just a matter of understanding these new symbol kinds and relationships, I believe).

The new symbol kinds and relationships make sense to me. Just a few notes:

- An extended type (`swift.TYPE_KIND.extension`) can conform to multiple protocols I believe, so this should be a many-to-many relationship, if I understand the diagram correctly.
- I'm not quite sure about reusing the `memberOf` relationship to model extended type -\> extended module relationships. I don't have a better naming suggestion off-hand though, so I would like to hear others' thoughts as well.
- Regarding the `swift.extension` symbol type: having the module being modeled as both a member of the `swift.TYPE_KIND.extension` and `swift.extension` is a bit odd to me. As you mentioned, this would require special processing on the Swift-DocC side to aggregate the doc comments, and we'd lose the 1 symbol = 1 documentation page that the DocC model currently has. I'm wondering if emitting `swift.extension` symbols should be done in separate "mode" entirely (`--emit-extension-symbols` or alike), in which the symbol -\> `swift.TYPE_KIND.extension` relationship wouldn't exist. If the goal here is to aggregate documentation comments, we could also take your original model without `swift.extension`, and add a new array field in `swift.TYPE_KIND.extension` that contains all the (unordered) doc comments and source locations associated with all the extensions of this type in the module. However, from an authoring UX perspective, I'm not sure quite sure how that would work; you'd want to control the ordering of these. Any ideas? I'm also not opposed to scoping this out of this proposal.

Could you please attach an example symbol graph with your proposed additions, to make sure everyone is one the same page?

> [@theMomax](#):
>
> Firstly, what would the `precise` identifier for all the `X.extension` symbols be? How does the Swift compiler currently generate them? Is there some documentation I could read?

This is a great question. I don't think we should use same precise identifier (aka USR) as the extended type (i.e., `s:SS`) because consumers would see that type itself as being defined in the module. For example, Swift-DocC does USR-based link resolution as part of its compilation process, and as it stands, would consider `s:SS` to be symbol _defined_ in the module, since its symbol graph contains a symbol with that USR. Swift-DocC would need to special-case symbols with kind `swift.*.extension`, but I think a better approach would be to have a separate USR for them entirely. It would be useful to record the relationship between the symbol and its extended symbol's as well though, as you mention further down—this will help with inter-module linking in in the future. I'm not quite sure whether the Swift compiler synthesizes a different USR for extensions (maybe @QuietMisdreavus would know). You could place a breakpoint in `SymbolGraphGen` and inspect what the declarations that are being processed.

> [@theMomax](#):
>
> Secondly, should we add a "extensionOf" relationship between `swift.TYPE_KIND.extension` and the respective original declaration symbols (even though this original declaration is not part of the same symbol graph)? I think this could cause problems in the conversion in Swift-DocC again, however it also seems right to have a reference to the original type declaration when considering the discussion about [[SR-15431] Support DocC references to symbols defined in another module - Swift](https://bugs.swift.org/browse/SR-15431) in the UX thread. If I have to mess with the symbol graph file format once I guess it would be best to plan ahead a bit.

Yes, I think we should do that. It's not required in the symbol graph model for the `target` of a relationship to be defined in the same symbol graph (i.e., it doesn't have to be in the `symbols` array). These relationships get a `targetFallback` property that textually describe the target. For example:

```swift
{
    "kind": "conformsTo",
    "source": "c:objc(cs)Bar",
    "target": "s:SH",
    "targetFallback": "Swift.Hashable"
}

```

I think tracking this kind of relationship makes sense for symbol extension -\> symbol makes sense as well and this will be useful when performing inter-module link resolution, because DocC would be able to look up `s:SH` in another DocC archive. It might not make sense to display this relationship on the documentation page of the extension though—the info seems a bit redundant, at least until we can make the target symbol an actual link.

> [@theMomax](#):
>
> Finally, for my broader understanding: What role does Swift-DocC play in the universe of Swift symbol graph files? Is it the main reason they exist or just one of many consumers of this interface? What would us introducing a new symbol graph file format mean for the old format? Would both versions have to be maintained next to each other indefinitely, or would the old one phase out after a short(ish) period of time?

The symbol graph format is maintained independently from Swift-DocC as it is used by other documentation compiler as well. It's important to minimize breakages as much as possible. That being said, the symbol graph format has not reached stability yet (Swift symbol's graph is currently at 0.5.3 [https://github.com/apple/swift/blob/main/lib/SymbolGraphGen/FormatVersion.h](https://github.com/apple/swift/blob/main/lib/SymbolGraphGen/FormatVersion.h)), but we should still aim to reduce breakages. It's also worth noting that the model aims to be generic enough for compilers of other languages to also be able to emit symbol graphs (e.g., there is an effort underway for clang to emit symbol graph files for Objective-C), so there needs to be some alignment so that documentation compilers can consume the same format.

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [April 15, 2022, 2:08pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/6 "2022-04-15T14:08:56Z")

</div>

Thank you for your valuable feedback @franklin!

> [@franklin](#):
>
> An extended type ( `swift.TYPE_KIND.extension` ) can conform to multiple protocols I believe, so this should be a many-to-many relationship, if I understand the diagram correctly.

You are absolutely right, I think I messed that up while reorganizing the drawing.

> [@franklin](#):
>
> Regarding the `swift.extension` symbol type: having the module being modeled as both a member of the `swift.TYPE_KIND.extension` and `swift.extension` is a bit odd to me. As you mentioned, this would require special processing on the Swift-DocC side to aggregate the doc comments, and we'd lose the 1 symbol = 1 documentation page that the DocC model currently has.

I agree that this solution doesn't feel perfect with the redundant `memberOf` relationship between `<<Member Symbol>>` and ``\<\<swift.TYPE\_KIND.extension\>\>`. Yes, it would require special processing on Swift-DocC's side, but I think this processing would be rather easy to implement as we are only **removing** information. Nevertheless, I'm definitely open to other suggestions.

> [@franklin](#):
>
> I'm wondering if emitting `swift.extension` symbols should be done in separate "mode" entirely ( `--emit-extension-symbols` or alike), in which the symbol -\> `swift.TYPE_KIND.extension` relationship wouldn't exist.

I don't think that would be useful for what we are trying to achieve right now. After all, we don't want to require two compiler runs to build a single documentation catalogue.

> [@franklin](#):
>
> If the goal here is to aggregate documentation comments, we could also take your original model without `swift.extension` , and add a new array field in `swift.TYPE_KIND.extension` that contains all the (unordered) doc comments and source locations associated with all the extensions of this type in the module.

Depending on what capabilities we are aiming for, this could be a very good solution! I think with this approach we would commit to the following two restrictions:

- The documentation does not know which extension block declares a specific member or if two members belong to the same documentation block.
- As you already indicated, extension block comments cannot be referenced for curation. This would basically result in a "take it or leave it" situation. If no comment for the `swift.TYPE_KIND.extension` page is provided via manual curation, we try to synthesize one from the extension block comments. Otherwise, we just use the provided comment as is. There is no syntax to manually include extension block comments.

If we are willing to accept these (I am), this solution works perfectly fine and should be really straight forward to process in Swift-DocC.

> [@franklin](#):
>
> Could you please attach an example symbol graph with your proposed additions, to make sure everyone is one the same page?

As a JSON file? Yes, sure I'll try to provide one as soon as I find time.

> [@franklin](#):
>
> I think a better approach would be to have a separate USR for them entirely.

That's also my intuition. I hope I can research the current structure of USRs more extensively in the next weeks.

> [@franklin](#):
>
> Yes, I think we should do that. It's not required in the symbol graph model for the `target` of a relationship to be defined in the same symbol graph (i.e., it doesn't have to be in the `symbols` array). These relationships get a `targetFallback` property that textually describe the target.

Thanks for the info, I wasn't aware of that. In that case I think we should definitely include it in the symbol graph files, even though we won't show it in the UI until we can link to other modules.

> [@franklin](#):
>
> The symbol graph format is maintained independently from Swift-DocC as it is used by other documentation compiler as well. It's important to minimize breakages as much as possible. That being said, the symbol graph format has not reached stability yet (Swift symbol's graph is currently at 0.5.3 [swift/FormatVersion.h at main · apple/swift · GitHub](https://github.com/apple/swift/blob/main/lib/SymbolGraphGen/FormatVersion.h)), but we should still aim to reduce breakages. It's also worth noting that the model aims to be generic enough for compilers of other languages to also be able to emit symbol graphs (e.g., there is an effort underway for clang to emit symbol graph files for Objective-C), so there needs to be some alignment so that documentation compilers can consume the same format.

That's good to know, thank you! In that case I guess we'll have to maintain both symbol graph formats for quite some time...

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [April 22, 2022, 9:25am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/7 "2022-04-22T09:25:14Z")

</div>

I finally got around to investigate the USRs for `extension` blocks and `import` statements.

### Extension Blocks

Let's start with `extension` blocks. We need this USR if we want to include `swift.extension` symbols, but a common prefix of this USR should also be used for the `swift.TYPE_KIND.extension` symbols.

You can find the relevant USR generation function here: [swift/lib/AST/USRGeneration.cpp at ee7446f24341b42da86cdf66d60e1d49b931a678 · swiftlang/swift · GitHub](https://github.com/apple/swift/blob/ee7446f24341b42da86cdf66d60e1d49b931a678/lib/AST/USRGeneration.cpp#L330)

The extension block USR follows the following pattern:

If the extension block defines at least one member, we just prefix this member's USR with an `e:`:

```swift
s:e:<<USR_FIRST_EXTENSION_BLOCK_MEMBER>>

```

If not, and the extension block defines at least one conformance, we have the `e:` followed by the extended type's USR and the USR of the protocol we conform to:

```swift
s:e:<<USR_EXTENDED_TYPE>><<USR_PROTOCOL_EXTENDED_TYPE_CONFORMS_TO>>

```

A few examples:

```swift
// USR: s:e:s:SS9SwiftDocCE15myFuncExtensionyyF
public extension String {
    // USR: s:SS9SwiftDocCE15myFuncExtensionyyF
    func myFuncExtension() {
        
    }
    // USR: s:SS9SwiftDocCE19myPropertyExtensionSbvp
    var myPropertyExtension: Bool {
        true
    }
}

// USR: s:9SwiftDocC17MyAwesomeProtocolP
public protocol MyAwesomeProtocol { }

// USR: s:e:s:s5Int64Vs:9SwiftDocC17MyAwesomeProtocolP
extension Int64: MyAwesomeProtocol {

}

// USR: s:9SwiftDocC17MyBwesomeProtocolP
public protocol MyBwesomeProtocol { }

// USR: s:e:s:s5Int64V9SwiftDocCE02myA4PropSbvp
extension Int64: MyBwesomeProtocol {
    // USR: s:s5Int64V9SwiftDocCE02myA4PropSbvp
    public var myInt64Prop: Bool {
        false
    }
}

// USR: s:e:s:SS9SwiftDocCE4BLUBV
public extension String {
    // USR: s:SS9SwiftDocCE4BLUBV
    struct BLUB { }
}

```

Therefore I'd suggest using the common prefix for all these USRs as the precise identifier for symbols of type `swift.TYPE_KIND.extension`:

```swift
s:e:<<USR_EXTENDED_TYPE>>

```

Note that the `<<USR_FIRST_EXTENSION_BLOCK_MEMBER>>` always begins with `<<USR_EXTENDED_TYPE>>`, as the member is defined on this extended type.

The respective identifiers for the `swift.struct.extension` symbols for `Int64` and `String` would be the following:

```swift
s:e:s:s5Int64V
s:e:s:SS

```

This should be unique within the defining module.

### Import Statements

I think the closes thing we have to the `swift.module.extension` symbol in the source code are `import` statements. Unfortunately, they do not have an USR.

That said, I think we can just use an even shorter prefix by cutting the USR after the module identifier.

Examples would be:

Foundation: `s:e:s:10Foundation`  
Markdown: `s:e:s:8Markdown`  
Swift Standard Library (skips the module identifier): `s:e:s:`

I could also imagine to drop the `s:e:` prefix or replace it by `s:m:` (for module) for the `swift.module.extension` symbol's identifier.

Which option would you prefer, or does anyone have any entirely different suggestions?

---

<div class="post-metadata">

**Author:** ![franklin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franklin/32/17924_2.png) [@franklin](https://forums.swift.org/u/franklin)\
**Post date:** [April 26, 2022, 10:53am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/8 "2022-04-26T10:53:42Z")

</div>

Ah, that's interesting. I did not expect each extension block to have a separate USR. Thanks for doing that research!

I don't think we should be synthesizing USRs in `SymbolGraphGen`; we should be using whatever USR the Swift compiler emits. Otherwise the symbol graph would contain references to symbols that effectively don't exist from the Swift compiler's perspective. Preserving the notion that the symbol graph is a representation of whatever APIs the module contains is important design-wise. If we generate identifiers, they shouldn't be treated as the USR of a symbol.

@Xi_Ge it looks like you're the original author of `printExtensionUSR`; do you know if there is a USR that is unique across all extensions to the same symbol? E.g., a common extension USR for `Swift.String` in:

```swift
public extension String { func foo() {} }
public extension String { func bar() {} }

```

If so, we could then just relate `foo()` and `bar()` to that USR in the symbol graph.

If there is no such USR, then I guess this would mean including each extension block as a separate declaration in the symbol graph with the USR that the Swift compiler assigns it. We'd also need a way of linking these back together (e.g., via a common identifier) on the Swift-DocC side (maybe via a SymbolKit API) so that they are indeed seen as a single symbol. I'd also be interested in hearing @QuietMisdreavus's thoughts on this.

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [May 11, 2022, 9:53am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/9 "2022-05-11T09:53:41Z")

</div>

> [@franklin](#):
>
> If there is no such USR, then I guess this would mean including each extension block as a separate declaration in the symbol graph with the USR that the Swift compiler assigns it.

Since there wasn't any input on the other options, I went ahead exploring this strategy.  
I successfully altered the `SymbolGraphGen` module to emit `"swift.extension"` symbols for each extension block. These carry all the information we need. There's still some work to be done, but I don't see any blockers there.

> [@franklin](#):
>
> We'd also need a way of linking these back together (e.g., via a common identifier) on the Swift-DocC side (maybe via a SymbolKit API) so that they are indeed seen as a single symbol.

I agree. Essentially, we want SymbolKit to remain the _reference model for the Symbol Graph File Format_, while also being extensible enough so we can use it in Swift-DocC to handle our slightly modified symbol graph format. (We don't want to introduce an extra graph library for our new format for performance and maintenance reasons.)

Unfortunately, that might require some breaking changes in SymbolKit. Currently, [`Symbol.KindIdentifier`](https://apple.github.io/swift-docc-symbolkit/documentation/symbolkit/symbolgraph/symbol/kindidentifier) is an `enum`, which means we cannot simply extend it in Swift-DocC to add new cases (`swift.module.extension`, `swift.struct.extension`, ...).

I think the prettiest (but also most intrusive) solution would be to adopt the more extensible struct-with-static-constants pattern in `Symbol.KindIdentifier` as a replacement for the `enum`, just as it was done for [`Relationship.Kind`](https://apple.github.io/swift-docc-symbolkit/documentation/symbolkit/symbolgraph/relationship/kind-swift.struct). We could then easily add our new "cases" in Swift-DocC, but also loose some of the safety features enums have in combination with switch statements, etc.

An alternative (less intrusive) solution is based on the [`Symbol.KindIdentifier.unknown`](https://apple.github.io/swift-docc-symbolkit/documentation/symbolkit/symbolgraph/symbol/kindidentifier/unknown) case. We would use that case when dealing with our custom symbols in Swift-DocC and carry the rest of the information either in [`Symbol.Kind.displayName`](https://apple.github.io/swift-docc-symbolkit/documentation/symbolkit/symbolgraph/symbol/kind-swift.struct/displayname) or a new optional property.

What do you think?

---

<div class="post-metadata">

**Author:** ![franklin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franklin/32/17924_2.png) [@franklin](https://forums.swift.org/u/franklin)\
**Post date:** [May 12, 2022, 10:09am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/10 "2022-05-12T10:09:59Z")

</div>

Making `Symbol.KindIdentifier` a struct with static properties sounds great to me, it makes the API much more flexible. The SymbolKit API is still evolving, so even though we should prefer not breaking public API, I'd argue that this is an acceptable change because kind identifiers shouldn't be a closed set anyway. This also provides the flexibility for clients to add their own kind identifiers. We'll just want to make sure to leave the PR up for a bit so that the community has time to add a default case to switch statements and provide feedback. There might be some changes to do in Swift-DocC as well before landing the SymbolKit changes to ensure that the Swift CI toolchain builds still succeed.

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [May 24, 2022, 1:47pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/11 "2022-05-24T13:47:05Z")

</div>

> [@franklin](#):
>
> If there is no such USR, then I guess this would mean including each extension block as a separate declaration in the symbol graph with the USR that the Swift compiler assigns it.

I just opened a draft PR based on that strategy: [https://github.com/apple/swift/pull/59047](https://github.com/apple/swift/pull/59047)

I described the changes there. Please let me know what you think!

There are three points I'd be especially happy to hear your opinions on:

- I currently only changed the behavior for extensions to **external** types, as we didn't really need any more information for the local case. However, this also implies that we still cannot access documentation comments on top of extensions to internal types. During the pitch discussion, I had the feeling that the community neither wanted to embrace this spot for documentation, nor entirely prohibit its use. My thought process was the following: If we allow this spot to contain documentation, people will use it and we have to somehow process it and find a suitable spot in the documentation catalog or append it to some other documentation string. The result will probably always be worse than if we just forced them to write one comprehensive piece of documentation above the original type declaration.
- I did not unify the Symbol Graph Files. We have to transform the graph later anyway, so merging them before doesn't really make a difference. Furthermore, this way programs that only look at the main Symbol Graph Files (without `@`) wouldn't even break if we were to make the new behavior the default.
- This is also the last point: should we make the new behavior the default and basically invert the flag to say `-omit-extension-block-symbols` ( **o** mit)?

Note that all three points should be easy to adapt, so changing them would be of little effort.

* * *

Here's also some examplary symbol graph files I generated from the SwiftDocC module: [download zip archive](https://gist.github.com/theMomax/a3d6654b84bf2222dce3b9f1c6ee9d40/raw/6a3d0cf409b3d191b50c05a76a7a7891d77abc08/symbol-graphs.zip)

I added the following code to the project which should make it easy to observe the changes I made:

```swift
/// USR: s:e:s:SS9SwiftDocCE4BLUBV
public extension String {
    /// USR: s:SS9SwiftDocCE4BLUBV
    struct BLUB { }
}

/// USR: s:e:s:SS9SwiftDocCE15myFuncExtensionyyF
public extension String {
    /// USR: s:SS9SwiftDocCE15myFuncExtensionyyF
    func myFuncExtension() {
        
    }
    /// USR: s:SS9SwiftDocCE19myPropertyExtensionSbvp
    var myPropertyExtension: Bool {
        true
    }
}

/// USR: s:9SwiftDocC17MyAwesomeProtocolP
public protocol MyAwesomeProtocol { }

/// USR: s:e:s:s5Int64Vs:9SwiftDocC17MyAwesomeProtocolP
extension Int64: MyAwesomeProtocol {

}

/// USR: s:9SwiftDocC17MyBwesomeProtocolP
public protocol MyBwesomeProtocol { }

/// USR: s:e:s:s5Int64V9SwiftDocCE02myA4PropSbvp
extension Int64: MyBwesomeProtocol {
    /// USR: s:s5Int64V9SwiftDocCE02myA4PropSbvp
    public var myInt64Prop: Bool {
        false
    }
}

/// USR: s:e:s:Sa9SwiftDocCSQRzlE11myArrayPropSbvp
extension Array: MyAwesomeProtocol where Element: Equatable {
    /// USR: s:Sa9SwiftDocCSQRzlE11myArrayPropSbvp
    public var myArrayProp: Bool {
        false
    }
}

```

I can also generate the symbol graph files for another project if you prefer.

---

<div class="post-metadata">

**Author:** ![franklin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franklin/32/17924_2.png) [@franklin](https://forums.swift.org/u/franklin)\
**Post date:** [June 14, 2022, 10:25am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/12 "2022-06-14T10:25:34Z")

</div>

Thank you, Max! Sorry for the late response here.

> [@theMomax](#):
>
> - I currently only changed the behavior for extensions to **external** types, as we didn't really need any more information for the local case. However, this also implies that we still cannot access documentation comments on top of extensions to internal types. During the pitch discussion, I had the feeling that the community neither wanted to embrace this spot for documentation, nor entirely prohibit its use. My thought process was the following: If we allow this spot to contain documentation, people will use it and we have to somehow process it and find a suitable spot in the documentation catalog or append it to some other documentation string. The result will probably always be worse than if we just forced them to write one comprehensive piece of documentation above the original type declaration.

This sounds good to me. If we ever wanted to include extensions to local types in the future, the design you're proposing would allow for that quite nicely.

> [@theMomax](#):
>
> - This is also the last point: should we make the new behavior the default and basically invert the flag to say `-omit-extension-block-symbols` ( **o** mit)?

I'm in favour of making this the default behaviour, as long as clients like Swift-DocC won't break when the Swift compiler changes get merged. Otherwise, we should not make it the default for now to leave some time for clients to add support, then we can make it the default.

Thanks for attaching the symbol graph files, it's greatly appreciated to understand the changes! I'm very excited by your progress here.

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [June 15, 2022, 9:20am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/13 "2022-06-15T09:20:59Z")

</div>

Hi Franklin, no worries at all...I found plenty of other work up the stack I could do in the meantime!

I'm glad you agree with my overall design decisions!

In regards to what should be the default behavior:

> [@franklin](#):
>
> I'm in favour of making this the default behaviour, as long as clients like Swift-DocC won't break when the Swift compiler changes get merged.

"Like Swift-DocC" is difficult to define. Tools that - _like Swift-DocC_ - only parse the main SGFs (i.e. the ones without `@`) won't break as the only change there is an additional property on the `swiftExtension` mixin. However, an additional unknown property in JSON should always be ignored.

Tools that do parse extension SGFs (i.e. the ones with `@`) will probably break! They will encounter symbols and relationships of unknown kind (`swift.extension` and `extensionTo`), as well as relationships the `source` of which is a symbol of unknown kind (`conformsTo` and `extensionTo`).

I tried it with DocC (current main + earliest open source commit), and both versions ran successfully. Jazzy, which is the only other consumer of symbol graph files I know of, will break, though!

```swift
jazzy/symbol_graph/symbol.rb:119:in `init_kind': Unknown symbol kind 'swift.extension' (RuntimeError)

```

However, they also flag the feature as somewhat experimental in their [readme](https://github.com/realm/jazzy#docs-from-swiftmodules-or-frameworks):

> ### Docs from .swiftmodules or frameworks
> 
> _This feature is new and relies on a new Swift feature: there may be crashes and mistakes: reports welcome._
> 
> Swift 5.3 adds support for symbol graph generation from .swiftmodule files. This looks to be part of Apple's toolchain for generating their online docs.

If Jazzy really is the only other tool using SGFs, another option could also be to make the new behavior default and give their maintainers a friendly heads up once the PR has been merged. I reckon there'll be still some time left until the next Swift release then.

---

<div class="post-metadata">

**Author:** ![franklin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franklin/32/17924_2.png) [@franklin](https://forums.swift.org/u/franklin)\
**Post date:** [June 15, 2022, 9:30am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/14 "2022-06-15T09:30:27Z")

</div>

Swift-DocC _does_ parse `@` extension files. If you compile documentation with `A.symbols.json`, `A@B.symbols.json`, and `B.symbols.json`, DocC will pick up the `@` symbol graph file. However, you're right that this behavior isn't possible when building via Xcode or SwiftPM. We should still make sure that the workflow continues to work for other integrations of `docc`, though. Given that, I think we should make the behavior opt-in initially, and after some time (a few months, say), make it the default, as you're proposing. And yes, since these changes aren't for Swift 5.7 but the next release, we have some time :)

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [June 15, 2022, 9:34am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/15 "2022-06-15T09:34:05Z")

</div>

> [@franklin](#):
>
> Swift-DocC _does_ parse `@` extension files. If you compile documentation with `A.symbols.json`, `A@B.symbols.json`, and `B.symbols.json`, DocC will pick up the `@` symbol graph file. However, you're right that this behavior isn't possible when building via Xcode or SwiftPM.

I didn't know that; thank you for clarifying!

> [@franklin](#):
>
> Given that, I think we should make the behavior opt-in initially, and after some time (a few months, say), make it the default, as you're proposing.

It's decided then 👍

---

<div class="post-metadata">

**Author:** ![johnfairh](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/johnfairh/32/760_2.png) [@johnfairh](https://forums.swift.org/u/johnfairh)\
**Post date:** [June 15, 2022, 9:46am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/16 "2022-06-15T09:46:10Z")

</div>

Don't worry about any Jazzy breakage, I'll sort it out as you land your stuff.

We should probably delete that warning, it really does date from Swift 5.3 time...

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [June 15, 2022, 10:00am UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/17 "2022-06-15T10:00:14Z")

</div>

Wow @johnfairh, that was quick! 🙂

Thanks for your offer, but I think we'll still make it opt-in first, just to potentially avoid some disruption in the development of docc. I'll then change the default behavior in the Swift compiler once I have made docc compatible with the new format.

---

<div class="post-metadata">

**Author:** ![taylorswift](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/taylorswift/32/43413_2.png) [@taylorswift](https://forums.swift.org/u/taylorswift)\
**Post date:** [June 15, 2022, 5:25pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/18 "2022-06-15T17:25:13Z")

</div>

> [@theMomax](#):
>
> Tools that do parse extension SGFs (i.e. the ones with `@`) will probably break! They will encounter symbols and relationships of unknown kind (`swift.extension` and `extensionTo`), as well as relationships the `source` of which is a symbol of unknown kind (`conformsTo` and `extensionTo`).

> [@theMomax](#):
>
> I tried it with DocC (current main + earliest open source commit), and both versions ran successfully. Jazzy, which is the only other consumer of symbol graph files I know of, will break, though!

[`swift-biome`](https://github.com/kelvin13/swift-biome/tree/unstable) relies on the `@` filenames to assign namespaces to symbols, please make this opt-in!

---

<div class="post-metadata">

**Author:** ![theMomax](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/themomax/32/22719_2.png) [@theMomax](https://forums.swift.org/u/theMomax)\
**Post date:** [June 15, 2022, 5:34pm UTC](https://forums.swift.org/t/symbol-graph-adaptions-for-documenting-extensions-to-external-types-in-docc/56684/19 "2022-06-15T17:34:49Z")

</div>

Thanks for the heads up. As already stated, it’ll be opt in first. I’ll make sure to notify everyone once the new behavior is fixed and you can start adopting it.