The state of Swift documentation

Closure capture lists.

4 Likes

Thanks @Dante-Broggi. I clearly need this functionality. :stuck_out_tongue_winking_eye:

+1, As a streach goal => WWDC Videos Archive

3 Likes

Thanks for starting this discussion @ole. I agree with pretty much everything you've said.

With regards to the first part, I believe this is already the case. As best I can tell, the documentation on developer.apple.com seems to be rendered directly from—or kept in very close sync with—the inline documentation for the standard library, which swift.org does actually note on their page for the stdlib:

The Swift standard library, along with its tests and inline documentation, are a part of the main Swift repository.

(emphasis mine).

One way to accomplish many of these goals, it seems, would be to bring swift.org itself under the auspices of the Swift open source project, so that community members could submit PRs to build out the documentation that you're pushing for here. Is this something that would be open for discussion?

I think it's reasonable for Apple to want to keep a mirror of the Swift standard library documentation in a place where it can appear alongside all the Apple-specific APIs, and it probably makes sense for APIs to list Xcode compatibility versions on the Apple-specific documentation page. However, that shouldn't keep the Swift project from keeping it's own canonical documentation (separate but presumably sourced from the inline documentation) which Apple's mirror can hopefully use as the source-of-truth.

5 Likes

I think I would add to this: We should separate out the specification section of The Swift Programming Language into its own document, containing:

  • lexical rules
  • parsing rules
  • type-checking rules

in separate sections. The rules should form the specification for the language, so if you can find a variance from one of the rules, then it's a defect. This could be TSPL : Part 2. Specification or something like that, with Part 1 : Tutorial and explanation as the first part.

We should also document the form of the rules in this section. Lexical rules could be sets of contiguous characters, or regular expressions, or whatever, but we should make that clear. Similarly parsing rules would be EBNF or some other form of grammar, but defined first. For type-checking we need some clear notation but I'm not up-to-date on what are good choices.

Once you have the rules, you should be able to go back and forth to the source of the compiler with them. Right now you can do that up to a point with the lexical rules--they're in comments in the parser (maybe not completely). This does not need to be automated, but you should be able to look at a bunch of type checking code and find a reference to the type-checking rules in the spec that you're implementing, and be able to search in Sema code for a heading to orient yourself to what implements what.

This helps compiler implementors. It also clarifies TSPL by separating the tutorial/explanatory part of the book from the structural part of the book more sharply.

The big gain is more documentation of type-checking rules though. That might be independent of @ole's original aim though, but there is a gap right now I think.

2 Likes

I would like to join in support of this post. I am eager to learn about Swift and it’s inner workings to be able to develop stable library wrappers. Wide number such wrappers is, in my opinion, essential for usability of Swift on non-mac platforms. Such task however requires in depth knowledege of Swift. I have posted about this in the past, for example: Swift ABI documentation .

Materials about Swift’s inner workings are scattered, old and outdated, in “TODO” state for years or in a form of hypertext link to the compiler’s source code https://swift.org/swift-compiler/#compiler-architecture.

Lack of knowledge of C++ and LLVM is a me problem, however, complete and updated documentation of things such as memory layout, type metadata, the Swift Runtime and SIL passes and optimizations would help me a lot.

4 Likes

Thanks for pointing this out. I have no insight into how the standard library documentation at developer.apple.com is generated. I know from (painful) experience that the docs for Apple's SDK libraries often don't mirror the documentation in header comments (especially for new APIs the header comments are often the better source), but the process could well be different from Swift.

And I should note that I have no (big) problems with the contents of the standard library documentation itself, aside from the mentioned Apple-centric hosting and availability annotations.

It's certainly true that all or most of the documentation is also available directly in the standard library source files, but I don't consider the doc comments a suitable replacement for a documentation website:

  • Header comments are hard to find and read except when you're actively writing code and your IDE has a good "Jump to definition" feature. I often want to look up documentation when I'm not in a code editor.

  • Header docs are hard to link to. I'd generally have to link to a specific line/range on Github, which means I need to link to a specific commit for stability. If someone else clicks on that link two years later, the documentation they find may no longer be up to date (sometimes this might be exactly what you want, but I'd argue creating a permalink to the "current version" is the more frequent use case).

6 Likes

Right, I'm in total agreement that the doc comments are not a suitable resource for the vast majority of Swift developers—I just wanted to draw the distinction (that you made more explicit in your post, thank you) between control over the content of the documentation and control of the presentation of the documentation.

There's an existing note about translations of The Swift Programming Language on swift.org, and I frequently see community members encouraged to report issues with TSPL as bugs on bugs.swift.org, so it does seem that TSPL is part of the Swift project in some sense. I'd love to see it opened up to allow the community to propose edits and correct inaccuracies, though.

1 Like

I'd really love to see progress in all of these areas, thank you for such a cogent exploration of these topics @ole! I think that one of the major questions will come down to "who will be willing to do the work?"

For example, there is nothing stoping an community driven language manual, but such a project would be a large amount of work, and it need an architect/driver.

-Chris

16 Likes

Technically there's nothing stopping the community from rewriting the Swift compiler in Swift either :laughing: . But for many of the same reasons as a documentation rewrite (though a magnitude or two greater), it's not likely to happen.

The biggest of these reasons is that there's no guarantee that Apple will integrate or even link to such documentation. Without it being officially available and an easy to use resource by the majority of the community, why would the community's most skilled documentation writers contribute to such a project rather than, say, write their own books?

A close second is that it's unlikely Apple will allow many or even any of their documentation writers to contribute to such a project, which would mean Swift's documentation would be split between the community project and Apple's. I'm not sure whether or not that would be an improvement over the status quo, but I certainly don't think it would be a good idea. I would guess that any Apple involvement in such a project would be contingent on their ownership of the copyright, even if the result is ultimately released under a Creative Commons license of some kind. And without Apple's involvement, for both this and the previous reason, I don't see how such a project could succeed.

Of course, if Apple shifts its stance towards the community and starts integrating more community tools or packages, or allows devs or writers to work on community projects, this approach may become more viable. But I don't think it is right now.

4 Likes

Huge +1 to this. Something like a GitHub repo of markdown files at apple/swift-docs that auto deploys to Swift.org would be great. If it were that easy to contribute I definitely would.

11 Likes

If I may, I'd also point out at another area where documentation is not only lacking for non-Apple platforms — it's generally lacking at all.

It's currently almost impossible to learn about Swift's runtime and ABI without reading the compiler code. Although Swift is a very high-level language, I love the fact that it still provides APIs for performance-critical code and allows to opt out of reference counting and overflow checking, for instance. That being said, if a feature is not exposed as API, there's virtually no information on it — or, if there is some, it's not very apparent if the documentation is being kept up to date.

Here are some examples that come to mind:

  • Data structure layout. It is a very important topic when dealing with modern hardware (cache lines and all that), but the only relevant document hasn't been updated for 2+ years, so I'm not sure if users could rely on it. If the document is outdated, then the existence of functions like, say, unsafeBitCast is rather pointless — how would one be sure if two types have the same layout, in case they're not both imported from a C header?
  • Inlining behaviour. It's not possible (hopefully only yet) in Swift to explicitly create inlined arrays, but at least knowing how the compiler could be forced into doing that could help writing more performant code that reduces pointer chase — especially in case of multidimensional arrays.

I'm not sure if it's really possible to reliably document these at all since either could be implementation details that can change, but leaving users without such information also seems bad to me.

TL;DR: Documentation that would clarify low-level stuff is either missing or feels outdated, and it's sad!

16 Likes

Solid, Ole. This would have reduced many frustrations for me (an Apple developer even.)

1 Like

@ole I love these ideas and would like to suggest that in addition swift.org should be expanded to be a site which embraces and highlights projects and contributions for Swift both in and outside of Apple. In addition to the language itself, someone coming to swift.org doesn't find pointers to resources and active areas. The challenge, of course, is that these initiatives are expensive (in time, infrastructure, people, ...) and (based on our experience at java.net - a Sun financed project run by O'Reilly) require commitment from some people employed to focus on the tasks.

4 Likes

Big +1 to this. Thanks for bringing these issues to everybody's attention.

This is a huge issue. Even if you take advantage of everything the toolchain provides (e.g. sourcekit-lsp), the experience on Linux is still really barebones.

Not all software is entirely cross-platform, and you often need to browse the available API as it looks on your system. Xcode has CMD+Click to show generated interfaces, but on Linux there's just nothing. sourcekit-lsp can only handle very basic workflows, and code completion is really not a substitute to having a complete interface in front of you. It just feels unloved - like a hack more than a supported platform.

Even if we have some better web documentation (which is really important), we still need good local tools. I'm aware of the swift-doc project and I'm really hopeful that it'll get us closer to where we need to be.

This has come up before, and there are a couple of things to note.

The first thing is that Foundation's documentation is just low quality in general. Take a look at the documentation for Date.timeIntervalSince(_:), for example:

  1. The parameter is referred to as both another and anotherDate. Both are wrong: the parameter's actual name is just date.
  2. The documentation warns us that the result is undefined if the object is nil -- but the parameter isn't an optional type, so it can't be nil.
  3. "the receiver" is a bit of an Obj-C anachronism, because of the whole message-based dispatch. It's a minor nit, but I don't think we tend to talk about senders/receivers in Swift.

This isn't limited to Linux. It's just as bad on Darwin platforms:

image

Interestingly there is apparently another, separate set of documentation (although honestly, it appears to be just as poor in places):

Honestly I think the Foundation team need to seriously rethink their documentation strategy overall. One of the goals should be to improve access and consistency, but they also need to improve quality. Since they don't take community contributions, all we can do is grumble at the poor experience we're forced to tolerate.

Nobody knows what the evolution process for XCTest is.

17 Likes

@Karl Thanks for making these points and providing an example. The Objective-C roots of Foundation definitely show through in its documentation.

One of these is for Date and the other one is for NSDate, that's why there are two slightly different pages:

I just wanted to mention that I maintain a version of the Swift reference manuals in the excellent Org-Mode format at https://github.com/danielmartin/swift-helpful/tree/master/swift-info

All Swift keywords are indexed, and there's also a topic index as well (ie, you don't need to search for things according to how Apple decided to name them in the documentation sections). From Org-Mode, a script exports it to GNU Info format (so you have Swift documentation while programming in Emacs), but, as you can see, there's an HTML version as well. All is automatically generated from a single documentation file. Code snippets are executable samples via Org Babel, so my documentation is actually a Swift Playground. People using GNU/Linux or Windows can interactively play with the documentation samples.

Maybe we can implement a new Swift driver that would be part of the Xcode toolchain that could access my documentation (and/or more sources). We can name this new binary swift-doc, for example, and Xcode would use its functionality to show info about Swift keywords. If we implement the shared logic as a library, SourceKit could use it as another source of documentation that would complement extracted documentation from code comments. People using Xcode/Xcode Playground would click on a Swift keyword and get info about a Swift concept.

If we want to go ahead with this new swift-doc toolchain binary, we would ideally need some coordination between the team at Apple that generates the documentation at swift.org and the open source community, to avoid needing to convert and manually index the documentation whenever a new version of Swift is released. Probably, the Swift docs should be maintained in the open as a prerequisite.

I don't know if there's interest in this new swift-doc toolchain binary that can make my Swift documentation style accessible to the masses (right now, only Emacs users are enjoying it). Here's a similar toolchain binary used by Rust: https://doc.rust-lang.org/rustdoc/what-is-rustdoc.html

11 Likes

I'm not really convinced we need to coordinate with the closed source docs to make progress in this area. Most of what's been described should be possible using swift symbolgraph-extract, which is available in toolchains from 5.3 onwards and seems designed for this use case. Its output includes doc comments, type relationships, and guidance on how to format headings/subheadings/etc. of documentation pages without needing to make many individual SourceKit requests.

For example, this command can be used to generate a symbol graph for the macOS standard library, and includes all the information needed to generate documentation pages or an index:
swift symbolgraph-extract --module-name=Swift --pretty-print --output-dir /some/output/dir --target=x86_64-apple-macosx --sdk=/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.16.sdk. I don't think there's anything stopping us from integrating this into the build process and installing symbol graphs for the standard library alongside the other documentation in /usr/share/doc/swift for use by toolchain or external tools.

Partial symbolgraph output (it's very long)
{
  "metadata": {
    "formatVersion": {
      "major": 0,
      "minor": 5,
      "patch": 0
    },
    "generator": "Apple Swift version 5.3 (swiftlang-1200.0.16.9 clang-1200.0.22.5)"
  },
  "module": {
    "name": "Swift",
    "platform": {
      "architecture": "x86_64",
      "vendor": "apple",
      "operatingSystem": {
        "name": "macosx",
        "minimumVersion": {
          "major": 10,
          "minor": 15,
          "patch": 0
        }
      }
    }
  },
  "symbols": [
    {
      "kind": {
        "identifier": "swift.method",
        "displayName": "Instance Method"
      },
      "identifier": {
        "precise": "s:STsE10compactMapySayqd__Gqd__Sg7ElementQzKXEKlF::SYNTHESIZED::s:s5Int16V5WordsV",
        "interfaceLanguage": "swift"
      },
      "pathComponents": [
        "Int16",
        "Words",
        "compactMap(_:)"
      ],
      "names": {
        "title": "compactMap(_:)",
        "navigator": [
          {
            "kind": "keyword",
            "spelling": "func"
          },
          {
            "kind": "text",
            "spelling": " "
          },
          {
            "kind": "identifier",
            "spelling": "compactMap"
          },
          {
            "kind": "text",
            "spelling": "<ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": ">(("
          },
          {
            "kind": "typeIdentifier",
            "spelling": "UInt",
            "preciseIdentifier": "s:Su"
          },
          {
            "kind": "text",
            "spelling": ") "
          },
          {
            "kind": "keyword",
            "spelling": "throws"
          },
          {
            "kind": "text",
            "spelling": " -> ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": "?) "
          },
          {
            "kind": "keyword",
            "spelling": "rethrows"
          },
          {
            "kind": "text",
            "spelling": " -> [ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": "]"
          }
        ],
        "subHeading": [
          {
            "kind": "keyword",
            "spelling": "func"
          },
          {
            "kind": "text",
            "spelling": " "
          },
          {
            "kind": "identifier",
            "spelling": "compactMap"
          },
          {
            "kind": "text",
            "spelling": "<ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": ">(("
          },
          {
            "kind": "typeIdentifier",
            "spelling": "UInt",
            "preciseIdentifier": "s:Su"
          },
          {
            "kind": "text",
            "spelling": ") "
          },
          {
            "kind": "keyword",
            "spelling": "throws"
          },
          {
            "kind": "text",
            "spelling": " -> ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": "?) "
          },
          {
            "kind": "keyword",
            "spelling": "rethrows"
          },
          {
            "kind": "text",
            "spelling": " -> [ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": "]"
          }
        ]
      },
      "docComment": {
        "lines": [
          {
            "text": "Returns an array containing the non-`nil` results of calling the given"
          },
          {
            "text": "transformation with each element of this sequence."
          },
          {
            "text": ""
          },
          {
            "text": "Use this method to receive an array of non-optional values when your"
          },
          {
            "text": "transformation produces an optional value."
          },
          {
            "text": ""
          },
          {
            "text": "In this example, note the difference in the result of using `map` and"
          },
          {
            "text": "`compactMap` with a transformation that returns an optional `Int` value."
          },
          {
            "text": ""
          },
          {
            "text": "    let possibleNumbers = [\"1\", \"2\", \"three\", \"///4///\", \"5\"]"
          },
          {
            "text": ""
          },
          {
            "text": "    let mapped: [Int?] = possibleNumbers.map { str in Int(str) }"
          },
          {
            "text": "    // [1, 2, nil, nil, 5]"
          },
          {
            "text": ""
          },
          {
            "text": "    let compactMapped: [Int] = possibleNumbers.compactMap { str in Int(str) }"
          },
          {
            "text": "    // [1, 2, 5]"
          },
          {
            "text": ""
          },
          {
            "text": "- Parameter transform: A closure that accepts an element of this"
          },
          {
            "text": "  sequence as its argument and returns an optional value."
          },
          {
            "text": "- Returns: An array of the non-`nil` results of calling `transform`"
          },
          {
            "text": "  with each element of the sequence."
          },
          {
            "text": ""
          },
          {
            "text": "- Complexity: O(*m* + *n*), where *n* is the length of this sequence"
          },
          {
            "text": "  and *m* is the length of the result."
          }
        ]
      },
      "functionSignature": {
        "parameters": [
          {
            "name": "transform",
            "declarationFragments": [
              {
                "kind": "identifier",
                "spelling": "transform"
              },
              {
                "kind": "text",
                "spelling": ": ("
              },
              {
                "kind": "typeIdentifier",
                "spelling": "Self",
                "preciseIdentifier": "s:STsE4Selfxmfp"
              },
              {
                "kind": "text",
                "spelling": ".Element"
              },
              {
                "kind": "text",
                "spelling": ") "
              },
              {
                "kind": "keyword",
                "spelling": "throws"
              },
              {
                "kind": "text",
                "spelling": " -> ElementOfResult"
              },
              {
                "kind": "text",
                "spelling": "?"
              }
            ]
          }
        ],
        "returns": [
          {
            "kind": "text",
            "spelling": "[ElementOfResult"
          },
          {
            "kind": "text",
            "spelling": "]"
          }
        ]
      },
      "swiftGenerics": {
        "parameters": [
          {
            "name": "ElementOfResult",
            "index": 0,
            "depth": 1
          }
        ]
      },
      "swiftExtension": {
        "extendedModule": "Swift",
        "constraints": [
          {
            "kind": "conformance",
            "lhs": "Self",
            "rhs": "Sequence"
          }
        ]
      },
      "declarationFragments": [
        {
          "kind": "keyword",
          "spelling": "func"
        },
        {
          "kind": "text",
          "spelling": " "
        },
        {
          "kind": "identifier",
          "spelling": "compactMap"
        },
        {
          "kind": "text",
          "spelling": "<ElementOfResult"
        },
        {
          "kind": "text",
          "spelling": ">("
        },
        {
          "kind": "externalParam",
          "spelling": "_"
        },
        {
          "kind": "text",
          "spelling": " "
        },
        {
          "kind": "internalParam",
          "spelling": "transform"
        },
        {
          "kind": "text",
          "spelling": ": ("
        },
        {
          "kind": "typeIdentifier",
          "spelling": "UInt",
          "preciseIdentifier": "s:Su"
        },
        {
          "kind": "text",
          "spelling": ") "
        },
        {
          "kind": "keyword",
          "spelling": "throws"
        },
        {
          "kind": "text",
          "spelling": " -> ElementOfResult"
        },
        {
          "kind": "text",
          "spelling": "?) "
        },
        {
          "kind": "keyword",
          "spelling": "rethrows"
        },
        {
          "kind": "text",
          "spelling": " -> [ElementOfResult"
        },
        {
          "kind": "text",
          "spelling": "]"
        }
      ],
      "accessLevel": "public"
    },
8 Likes

Is swift symbolgraph-extract itself documented anywhere? It's the first time I'm hearing of this tool, maybe it's just me.

7 Likes

@Daniel_Martin This looks awesome. Thanks for making this and sharing it here.

I'm not an Emacs user, but I'd love to have keyword/attribute documentation integrated into SourceKit-LSP. I also love your idea of linking directly into the standard library source code.

Terms of Service

Privacy Policy

Cookie Policy