Extending Swift-DocC to support Objective-C documentation

Hey everyone! I’m here to propose an extension to Swift-DocC to enable documentation for symbols that have representations in multiple languages, e.g., between Swift and Objective-C. I’m proposing for this support to land in main under the --enable-experimental-objective-c-support feature flag.

Motivation

DocC is currently architected to render symbol documentation in a single language (Swift). However, there are cross-language projects that would benefit from collecting multiple “language variants” together into the same set of documentation, for example for Objective-C APIs that can be called from Swift or vice-versa.

Proposed Solution

We will add support for generating documentation for Objective-C–only APIs and APIs that are available both in Swift and Objective-C. The base user experience will be very similar to what DocC supports for Swift documentation today, with the addition of a new language toggle in the sidebar for APIs that are available in both languages:

Screen Shot 2021-10-29 at 16.16.33

Documentation for APIs that are available in multiple languages will be published in a single render JSON (see Extending Swift-DocC Render JSON to support multi-language symbols) and will be accessible at the same URL path. We also introduce support for a new language query parameter (e.g., /documentation/square?language=objc) in Swift-DocC-Render to keep track of the language you’re currently viewing documentation for. For APIs that are available in Swift and Objective-C, Swift-DocC will construct the web URL path based on the Swift name of the API.

When writing links to APIs that are available in multiple language, you can use either the Swift or Objective-C name of the API. For example, Square/init(size:) and Square/initWithSize: are both valid.
New Markdown syntax to write language-specific documentation (e.g., to show a Swift code listing on the Swift version of the documentation page and an Objective-C code listing on the Objective-C one) will be discussed at a later stage.

Detailed Design

Regarding the implementation of the base support, the main changes will be to:

  1. Teaching Clang to emit symbol graphs that Swift-DocC can load, specified here: [cfe-dev] [RFC] clang support for API information generation in JSON
  2. Teaching SymbolKit to “unify” symbol graphs together into a representation that accumulates them together (specified below)
  3. Teaching Swift-DocC to use these “unified symbol graphs” to load symbol information into its “topic graph” (also specified below)
  4. Hook this all into the newly-added DocC support for specifying language variants data in Render JSON, specified here: Extending Swift-DocC Render JSON to support multi-language symbols

I propose extending SymbolKit to introduce the concept of a “unified symbol graph” and “unified symbols”, which collect symbol graphs together that describe the same module or symbol from different programming languages or platforms. From there, DocC will have a single representation of a framework’s APIs, even for APIs that available on multiple languages.

The key idea behind the unified symbol and graph types is that symbols and symbol graphs that “describe the same symbols” can be unified on the “precise identifier”, like a USR or mangled name (for individual symbols) or the module name (for whole symbol graphs). All of the other fields in the symbol or graph can be collected in a map that indexes them based on the language/platform that it relates to (for individual symbols) or the graph that contributed the information (for whole symbol graphs).

For example, this is what a unified symbol graph could look like:

/// A combined representation of multiple ``SymbolGraph``s that describe the same module.
public class UnifiedSymbolGraph {

    /// The module described by this unified symbol graph.
    public var moduleName: String
    
    /// The module metadata objects from the symbol graphs, indexed by the symbol graph filename.
    public var moduleData: [URL: SymbolGraph.Module]
    
    /// The symbol graph metadata objects, indexed by the symbol graph filename.
    public var metadata: [URL: SymbolGraph.Metadata]
    
    /// The combined symbols in this module, indexed by precise identifier.
    public var symbols: [String: UnifiedSymbolGraph.Symbol]
    
    /// The list of symbol relationships in this module.
    public var relationships: [SymbolGraph.Relationship]
}

And a unified symbol:

extension UnifiedSymbolGraph {
    /// A combined representation of multiple ``SymbolGraph/Symbol``s that describe the same symbol.
    public class Symbol {
        /// The precise identifier for this symbol, e.g. a USR or mangled name.
        public var uniqueIdentifier: String
        
        /// If this symbol has a type (e.g. a property or static variable),
        /// the precise identifier referencing the type.
        public var type: String?
        
        // the Selector type is defined below
        
        /// The module metadata objects for the symbol graphs that contained this symbol.
        public var modules: [Selector: SymbolGraph.Module]
        
        /// The kinds of this symbol.
        public var kind: [Selector: SymbolGraph.Symbol.Kind]
        
        /// The path components of this symbol.
        public var pathComponents: [Selector: [String]]
        
        /// The title and name information for this symbol.
        public var names: [Selector: SymbolGraph.Symbol.Names]
        
        /// The doc comments of this symbol.
        public var docComment: [Selector: SymbolGraph.LineList]
        
        /// The visibility or access level of this symbol.
        public var accessLevel: [Selector: SymbolGraph.Symbol.AccessControl]
        
        /// Additional information about this symbol that is not necessarily common to all symbols.
        public var mixins: [Selector: [String: Mixin]]
    }
}

The Selector type represents the language and/or platform that a piece of field data came from:

extension UnifiedSymbolGraph {
    /// The specific language and/or platform that a symbol's data came from.
    public struct Selector: Equatable, Hashable {
        /// The interface language that a symbol is available in.
        public let interfaceLanguage: String
        
        /// A platform that the symbol has specific data for.
        public let platform: String?
    }
}

To aid in using these unified symbol graphs, utilities can be written to merge parsed SymbolGraphs together based on their filenames and contents. For example, here’s a potential GraphCollector type that could be used to accumulate symbol graphs as they are loaded:

public class GraphCollector {
    /// Used to differentiate whether a file used to load symbols described
    /// symbols defined in that module or extensions defined in a different module.
    public enum GraphKind {
        /// The symbol graph at this filename is a "primary" one.
        case primary(URL)
        
        /// The symbol graph at this filename contains extensions from a different module.
        case `extension`(URL)
    }
    
    /// Load the given symbol graph into this ``GraphCollector``'s state.
    public func mergeSymbolGraph(_ inputGraph: SymbolGraph, at url: URL) { /* ... */ }
    
    /// Finish loading any extension symbol graphs that were set aside, and return the merged data.
    public func finishLoading() -> (unifiedGraphs: [String: UnifiedSymbolGraph], graphSources: [String: [GraphKind]]) { /* ... */ }
}

This GraphCollector can then be used in Swift-DocC’s SymbolGraphLoader to merge symbol graphs together while they are being loaded.

14 Likes

PRs with the initial implementation are now open!

7 Likes

Though this is already merged, I have a question for this PR. @QuietMisdreavus

Why UnifiedSymbolGraph is designed to be a class not a struct like SymbolGraph?

I think it is worth that we mark some API as mutating and do some more stuff to make it a value type.

A problem with this is that Objective-C is Apple-specific, but Swift is cross-platform. If you supported it, you'd also need support C++ equally for Windows, and Python for ML tasks. I would greatly appreciate C++ support - especially since I can't add syntax coloring to Metal Shading Language (similar to C++) in ARHeadsetKit tutorials.

Side note: I don't particularly like Objective-C, which means my comment may be biased. If Objective-C was exclusively supported in documentation pages and not in tutorial code listings, I would be fine with that. I have had enough frustration with Apple making their Metal sample code in Objective-C (even code released in 2021), and if Apple made new Metal tutorials using DocC, I'd like them to stick to exclusively using Swift.

Metal was introduced in 2014 the same year with Swift. Since Swift was a baby at that time. Metal was of course built upon with ObjC. Even we can use swift to get its API, but its core is still ObjC oriented.

Yes, but we could say the same about every other API - including CoreGraphics and CoreImage, which were primarily Objective-C at the time. Most of Apple's legacy documentation is in Objective-C, and a lot of Apple's sample code is from around 2017 when Swift wasn't particularly stable. Apple made a massive shift of content to the archive website in 2018, but had to keep their Metal tutorials on the main one for future reference.

Even Apple has started to move on to Swift + Metal. In the A15 Bionic Enhancements video, there was not a single line of Objective-C (the exception being MTLGPUFamilyApple8). Metal requires the Objective-C runtime, not the Objective-C language.

1 Like

I don’t see how this logically follows. It’s entirely possible to support Objective-C without adding support for the other languages as well.

2 Likes

I'm saying only adding Objective-C would steer Swift more in the direction of being Apple-centric. That's the opposite of what we're trying to do with Swift 6 - adding support for Windows, Android, BSD - neither of which have Objective-C.

The description in the original post is pretty clear that it's proposing a generalization to allow the tool to support multi-language views of an interface and that there's no specific hard-coding of Objective-C. Given that, I'm not sure what you're objecting to.

6 Likes

It's also worth noting that the original post mentions this clang RFC which isn't specific to Objective-C either. As @John_McCall pointed out, the proposed architectural changes in DocC put us in a good spot for C++ support, since DocC will be able to process C++ symbols in symbol graph files like any other symbol.

6 Likes

Swift-DocC was already architected to support a generic format for describing APIs: the symbol graph. It doesn't connect directly to the Swift compiler - it just reads in these files. Symbol graphs, as a format, were also intended to be fairly language-agnostic to begin with; part of the trick is getting them in the first place. There are currently some Swift-specific assumptions in how to lay out and structure docs, so getting Objective-C into the system will work to break those up and support anything that can generate a symbol graph.

7 Likes

I would also like to see DocC allowing us to document extensions to types declared in other frameworks. I did that a lot with Metal in ARHeadsetKit, and I couldn't document it.

3 Likes

Hi!

Just to check with you, we can try it out right? :slight_smile:

Thanks for working on this. It would be great to have support for Objective-C documentation as well.

1 Like

Quick progress update on this: I just merged a PR that enables Swift-DocC to process Objective-C symbol graph files. The clang portion of the integration that will produce these symbol graph files is still in active development (for example, see PR) and a further update with instructions will be shared once Objective-C doc generation can be used end-to-end. Also, one next step here will be to integrate this in the new DocC SwiftPM plugin.

10 Likes