Enablement of DocC Extension Support as an Opt-In Feature

Hi everyone!

Back in March I pitched Documenting Extensions to External Types using DocC. Today, over nine months, and twelve PRs to six repositories later, development is completed and I'm happy to announce it's finally time for you to try it out!

Before I get into the details of using it, I want to thank everyone who's helped me along the way, by participating in discussions, reporting bugs, reviewing PRs, or otherwise. Special thanks to @daniel-grumberg, @ronnqvist, @ethankusters, @franklin, and @QuietMisdreavus, this would not have been possible without you!

Finally, please note that this discussion is about enabling extension support as an opt-in feature, not as a default. I'll go into more detail about this at the end of this post.

How to try it

  1. Use a toolchain generated from main or release/5.8 branches after January 7th. You can download it here.
  2. Use the add-extended-types-flag branch of my fork of the DocC plugin when adding the swift-docc-plugin to your Package.swift:
dependencies: [
    .package(url: "https://github.com/themomax/swift-docc-plugin", branch: "add-extended-types-flag")
]
  1. Add the --include-extended-types flag when running your commands. E.g. you can run the following command to preview your documentation in the browser:
swift package --disable-sandbox preview-documentation --target YOUR_TARGET_NAME --include-extended-types

For further instructions on how to use the DocC Package Plugin in general, run swift package plugin generate-documentation -h, or find the documentation online.

You should now see any (public) extensions you've made in the target you're documenting to types from other targets in your documentation archive.

You should now see a new section "Extended Modules" in your module's top-level documentation page, as well as in the sidebar. This section lists all the targets you've added public extensions to. You can start exploring the "Extended Types" and the symbols you extended them with by digging into the page-hierarchy.

You can refer to these new symbols using the following formats:

  • relative: ExtendedModuleName/ExtendedType/extensionSymbol
  • absolute: /ExtendingModuleName/ExtendedModuleName/ExtendedType/extensionSymbol

Here's a quick example with a small extension to the standard library:

/// This comment will be used in the documentation
/// archive to document the "Extended Struct" `Int`.
extension Int {
	/// This comment documents the added symbol as usual.
	public var isEven: Bool { self % 2 == 0 }
}

/// We can refer to the extension on `Int`
/// as ``Swift/Int/isEven``.
public struct MyStruct { }

We can override or append to the documentation for the "Extended Struct" Int, its property isEven and the "Extended Module" Swift using a Markdown file in your Documentation Catalog as usual:

Assuming the target we're documenting is called SampleTarget, the Markdown file containing the documentation for the "Extended Module" Swift should start with the following line:

# ``SampleTarget/Swift``

Curation, i.e. altering the hierarchy of documentation pages in your Documentation Archive also works as usual using absolute or relative references.

Please try it out and post your feedback here! I'd also love if you could share links to some documentation archives you updated with this feature!

Enablement Plans

The Documentation WG plans to make this feature available via the non-experimental feature-flag --include-extended-types in Swift 5.8. In case this discussion reveals that there is a lot to be fixed, we might opt for adding an experimental prefix to the flag, or postponing the release of this feature to the next minor version of Swift.

Depending on the experience we make with the feature in Swift 5.8 and its popularity among documentation authors, we might make this feature the default behavior in a future version of Swift. However, this is to be decided on another day.

Looking forward to your feedback!

Cheers

~ Max

23 Likes

Hey @theMomax

Great instructions - thank you. I gave it a whirl this morning, but ran into an issue - more with the toolchain than I think this code. I grabbed the main branch nightly form the 16th (https://download.swift.org/development/xcode/swift-DEVELOPMENT-SNAPSHOT-2023-01-16-a/swift-DEVELOPMENT-SNAPSHOT-2023-01-16-a-osx.pkg) to give this a while, updated my PATH, and had a run.

Even with putting your branch into the Package.swift, the dependencies are reflecting Apple's main branch for the docc-plugin, so with the command:
swift package --disable-sandbox preview-documentation --target MyPackage --include-extended-types, I'm getting the error:

Error: Unknown option '--include-extended-types'
Usage: docc convert preview [<options>] <source-bundle-path>
  See 'docc convert preview --help' for more information.

I tried swift package update, but I'm having a devil of time getting it to resolve to your version for testing. swift package show-dependencies keeps reporting:

.
├── swift-collections<https://github.com/apple/swift-collections.git@1.0.4>
└── swift-docc-plugin<https://github.com/apple/swift-docc-plugin@1.0.0>

Any chance you've run into this, and have any hints on how to force the cache to let go and refresh?

(I tried blowing away the cache by getting into ~/Library/Caches/org.swift.swiftpm and nuke everything in there, but it all resolved again to swift-docc-plugin<https://github.com/apple/swift-docc-plugin@1.0.0>)

I also did a quick check on docc and noticed that it's still the x86 only version. I'm on an M1. Is that an issue for concern that'll bite me once I get past the package resolution bits?

lipo -info /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin/docc

Non-fat file: /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin/docc is architecture: x86_64

Any experimenting advice?

I believe @TimTr has run into this lately but I've been unable to reproduce so far unfortunately. @abertelrud or @NeoNacho are there any known issues relating to SwiftPM's caching and plugin dependencies?

Argh - I did this to myself. I had an updated Package.swift for different versions, and I was looking at the wrong one to update the dependencies.

I guess I had to ask the question to step back and look at it to see my mistake. My apologies. The x86 version of docc didn't mess with me at all, and I'm previewing the extensions that I made in a library without issue.

It looks fantastic (other than my lack of docs for this) ;-)

2 Likes

This is a real treat to use :smiley:, thanks for all your hard work! I finally have proper documentation for my Swift extensions package - link to the fresh new documentation: Documentation

Thanks! :heart:

4 Likes

Thank you everyone for all the positive feedback so far and thanks for posting your live archive @adamwulf, it's great to see the feature in action!

Thanks to the Swift Package Index and its automated documentation hosting we were able to additionally verify the soundness of my implementation by generating docs for a larger batch of packages. Huge shoutout to @finestructure for setting up the build pipeline with the custom toolchain configuration and going through the build results with me!

I'm happy to report that our findings are very positive!

All 220 packages that
a) opted into automated documentation hosting,
b) compiled successfully with the main trunk toolchain we used, and
c) build documentation successfully with the regular toolchain the Swift Package Index uses in production environments
also build documentation successfully with the custom toolchain configuration using the --include-extended-types flag.

In other words: no failures were caused by the Extension Support Feature :tada:

10 Likes

This is really great news! Did you also have a chance to inspect some of the produced documentation and see if there were any issues in extension pages? If not, unless there is more feedback, I'm supportive of enabling this feature as opt-in in 5.8!

1 Like

Amazing news! :partying_face:

1 Like

We didn’t inspect the documentation archives as that would have been a significant manual effort. Nevertheless, I’ve done quite some manual and automated testing during development, so I also don’t think it‘s really necessary.

1 Like

This is a very important feature for DocC. Thank you for working on this!

Implementation bugs notwithstanding, I can't think of a reason off the top of my head why this should not become the default, and I'm a little surprised that the Documentation workgroup seems to see this as an open question. Do you or the Documentation workgroup envision any downsides/problems if this feature is enabled by default?

1 Like

The goal is to make it the default in the next release of Swift, after 5.8. Landing the feature under a flag gives more time for library maintainers to move to a model where extensions are included in documentation, and gives us an opportunity to make enhancements / resolve potential bugs before it's enabled by default. Since this change will increase the number of pages emitted in documentation, it's beneficial to give people some time to adapt (e.g., for curation).

4 Likes

Makes sense. Thanks!

Thanks @ole for your kind words and thanks @Franklin for clarifying!


On a general note:

Since there's only been positive feedback so far, we've decided to merge the respective PR. That is, the feature is now on main and will be tagged in a new minor version before the first official beta release of Swift 5.8.

Users who want to use the --include-extended-types flag will have to:

  • use a Swift 5.8 toolchain
  • run swift package update swift-docc-plugin in every package where they want to use the feature

The updated version of the plugin will be backwards compatible for people using a Swift 5.7 toolchain or lower. If the --include-extended-types flag is used with such toolchain, a warning will be emitted prompting the user to update to Swift 5.8.

3 Likes

Really nice work!

As I have a lot of frameworks in swift packages that make heavy use of extensions I would like to use it now. But as an ordinary Xcode user I need some help in this topic.

I normally use this commands in terminal under macOS to create my documentation:

cd ~/Documents/Apps/RemoteSwiftPackages/BindingHelper
swift package --allow-writing-to-directory ~/Documents/Apps/RemoteSwiftPackages/BindingHelper/docs  generate-documentation --target BindingHelper --disable-indexing --transform-for-static-hosting --hosting-base-path BindingHelper --output-path ~/Documents/Apps/RemoteSwiftPackages/BindingHelper/docs

Package dependencies are:

    dependencies: [
        .package(url: "https://github.com/apple/swift-docc-plugin", branch: "main"),
    ],

I downloaded and installed a Swift 5.8 development snapshot from https://www.swift.org/download/.

I just added update, toolchain and the extended types flag:

cd ~/Documents/Apps/RemoteSwiftPackages/BindingHelper
swift package --toolchain org.swift.58202302231a update swift-docc-plugin
swift package --toolchain org.swift.58202302231a --allow-writing-to-directory ~/Documents/Apps/RemoteSwiftPackages/BindingHelper/docs generate-documentation --target BindingHelper --disable-indexing --transform-for-static-hosting --hosting-base-path BindingHelper --output-path ~/Documents/Apps/RemoteSwiftPackages/BindingHelper/docs --include-extended-types

I does not work, it complains about a too low toolchain version for extended types. I would be very thankful for a hint: what do I wrong or what I have to do to make it work.

This is great! I’ve been trying it out with Xcode 14.3 beta 2, and I’m really excited to use this feature for our packages. Is there a way to direct Xcode to use the --include-extended-types flag on a pure Swift package, or do I need to resort to the CLI?

@Wolfi, @prachi I'm really sorry for the late response.

@Wolfi, I think the problem is that you are still using the default version of SwiftPM when using the --toolchain option. I guess that only the compiler invocations are directed to the toolchain you specify with the flag. I usually just override my PATH locally to point to whatever toolchain I need:

export PATH=/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin:"${PATH}”

@prachi I haven't checked with the exact version you are using, but you should be able to get it to work using the instructions here.