Documentation Workgroup meeting: September 9th, 2024

Swift Documentation Workgroup Updates
1

The Documentation Workgroup will be holding its next meeting on Monday, September 9th, 2024 from 8:00am to 9:00am PT (or see the time in your own timezone).

This meeting will be open to anyone who wishes to contribute. If you wish to participate, please reach out to @swift-documentation-workgroup in forums via DM for a link to the WebEx meeting.

Meeting notes for this meeting will posted in this thread shortly after.

Please propose agenda items in this thread.

1 Like
2

Documentation Workgroup Meeting 9 September, 2024

Requested Topics

  • (Sven) The recently announced multi target feature and if and when and how we could start testing that in SPI.
    • What toolchain would we need? 6.0? 6.1?
      • 6.0 has basic multi-target support
      • 6.1 adds the navigation element + overview page
    • Are there any special flag required to pass in via .spi.yml?
      • No, but we'll want to leverage the fact that now all targets are built by default
      • We'll need to update to use the new --target t1 --target t2 syntax
  • (Joe) The “Mentioned in” new/experimental feature that was recently added - generally what’s outstanding with it and taking it forward to “non-experimental”?
  • (Joe) Where should “documentation” updates go? In the case of Snippets, there isnt anything outside of the forums, and when I re-looked into the documentation for Docc it was heavily fractured between Docc itself and the docc-plugin.
    • Most specifically, where is most appropriate for snippets documentation to get created?
  • (Sofia) how we can improve DocC to better convey cross-platform availability.

Attendees

David Ronnqvist
Sofia Rodriquez Morales
Vera Mitchell
Joe Heck
Franklin Schrans
Daniel Grumberg
Sven
Dave Verwer
Kyle

Discussion

Franklin: Welcome Sofia! to the workgroup - official part of the workgroup now.

Multi-target docs

  • Sven: Is is ready to try out? SPI has a drop-down at the top to reference multiple docs, this seems to add on the side? What are the requirements to run this? Req toolchain? Flags needed?

  • David: most function with 6.0 toolchain. Combined navigation is in the 6.1 toolchain. 6.0 definitely has the start, uncertain how far you'll get. There is a new flag (--target flag), that takes a vector (list) of targets as options. When omitted, it builds every target in the package and merges it together.

  • Sven: when SPI triggers, --target is an option that SPI requires to opt-in to generating any docs. We're considering taking trigger and wrap it into this option

  • David: If that's getting used a lot, it may make sense to move to defaulting to product. (this default may not make sense for building internal documentation) If you merge mult targets w/ 6.0 toolchain, each will have seperate navigation. Navigation switches to local target. Combined nav is on 6.1 toolchain.

  • Franklin: regarding 6.1 combined navigation, what's the delta in experience?

  • David: In 6.0, You can intentionally write a link between targets, those become symbol links automatically. Clicking jumps to the other target, and the navigation "jumps" to the other target. Navigator focuses on one target at a time in 6.0. A general landing page doesn't exist with 6.0

  • Sven: SPI currently broken up across separate targets.
    Dave V: sounds like aiming to ship with 6.1 may have a better experience.

  • David: generally agreed - also navigation/quick-jump would search between all the relevant targets

  • David: there's another benefit to doing everyting in one plugin call, all base path, config/etc get passed to all relevant targets. Those wouldn't need to be updated across multiple configurations

Mentioned In

  • Joe: Is there something experimental about Mentioned In that we need to iron out?
  • David: I think it's marked experimental to give people a chance to try it out.

Documentation for documentation tooling

  • Joe: Where would be most appropriate to add documentation for Mentioned In or Snippets?
  • Vera: Probably makes sense in the DocC tool documentation (Documentation). We could have a page that outlines the experimental features, how to use them and what they do. That way, when they become official, we have a good basis for documentation.
  • SofĂ­a: We also have other experimental features that are only discoverable through docc help.
  • Vera: I'd like to add docs for the overloads feature in that document.
  • David: Package plugin documentation should be more about package-specific concerns, rather than every flag that the plugin forwards to DocC.
  • David: Some aspects of snippets are worth discussing in both the DocC tool and plugin documentation.
  • Vera: The --skip-protocol-implementations is also somewhere in the middle, since it's a plugin flag that gets passed to the Swift compiler.
  • swift-docc/Sources/docc/DocCDocumentation.docc at main · swiftlang/swift-docc · GitHub <-- this one for the documentation repo

Cross platform availability

  • Sofia: how can we better convey cross-platform availability aspects within the documentation? I've drafted out a document with a list action items we can take we could take (DocC Availability Information for cross-paltform libraries - HackMD)

  • Joe: compiler availability indicates only minimum required swift/SDK platform version, doesn't include any sort of detail for "this is epxected - or NOT expected - to work" for each platform. SPI, for example, determines compatibility by building on each of the platforms (linux, Apple's devices through Xcodebuild)

  • David: The Info.plist that Sofia is talking about allows for specifying "unavailable" for a platform (for documentation-only purposes)

  • Sven: Difficult to test/isn't always a reliable indicator. Generally working well, but it's not tested/perfect. Certainly not eco-system wide.

  • Joe: windows build isn't yet supported anywhere for this sort of effort

  • Franklin: what actions should we take to promote this?

  • Sven: would like to have a "I expect this to work everywhere"

  • Vera: minimum version of swift a possibility, maybe scrape/look at swift-tools version, potentially using the same for swift-embedded.

  • Sven: tool version isn't always accurate

  • Sofia: Trying to guess right version is quite tricky - if they know/expect, best to assert it.
    Sofia: Next step (AI)- will open a forum post to discuss how to add to documentation, start to work out the details.

  • Joe: Embedded harder to make it work, so it'll potentially be important to folks

  • Franklin: I don't think there's an availability marker for "this works for embedded". Does SPI test for embedded?
    Sven: not today - we do build for linux and/or Apple platforms (xcode docbuild). SPI allows opt-in for specific means to build. Some symbols are only available on specific platforms. Is there any vision of how we should handle a multi-platform doc archive?

  • Sofia: if the symbol doesn't show in the availability for the platform?

  • Kyle: Can we have a multi OS chooser just like the version chooser on SPI?
    for example:
    • documentations
    • macos
    • linux
    • ios

  • David: It really boils down to compiler symbol generation - can't get the symbols for other platforms than the one you're building for. Example - iOS & Linux may have different flags. Would require first build for all the platforms, then merge them together... Docc can do that for macOS, tvOS, iOS, linux - but there's no built in way to wrangle that, since it requires builds for each platform.

  • Sven: is cross-compilation an option?

  • Joe: need the SDKs/toolchain for the platform

  • Franklin: if we build on different platforms, then it sounds like we can combine with Docc

  • Sofia: as long as symbols have same USR, it'll smoothly merge.

  • Franklin: some USR may be different - argument in one platform different from another.

  • Vera: may show duplicate symbols, which will result in multiple pages.

  • Franklin: workflow of combining symbolgraphs is something we do regularly at Apple, but it's not productized.

  • Sofia: will post to workgroup first - please contribute any feedback into draft hackMD or in forum post.

1 Like
3

(Off-topic, so feel free to not reply or redirect...)

I'm curious about the limitations/objectives in your case, and the gap between what you do and what's available. Is it expected to be available with 6.1?

SymbolKit.GraphCollector produces UnifiedSymbolGraph for each module. But is that restricted to a given build, and thus unusable for combining output of different builds?

It accommodates some per-language differences in symbols for the same API. If we assume whatever library evolution flags are consistent across builds for a given library, what other build factors/flags could lead to incommensurable symbols?

I'm asking because I'm interested in combining symbols from different build for analysis, but haven't scaled beyond a small set of packages. Knowing what dragons are lurking could help :)

As an aside to an aside, does any CI-the-world operation like swiftinit or swiftpackageindex make available their swiftinterface or symbolgraph files? It would be nice not to replicate their builds just to do the analysis.

4

There was more discussion in the meeting than I could capture in notes. The DocC pipeline relies on symbols generated by the compiler, and there’s no export of compiler generated symbol graphs for other platforms - to get the symbols for a platform, you need to compile for that platform (and with relevant SDKs available). So even today, in some libraries you might see different symbols for iOS vs macOS vs Linux of the code has ifdef for platform compatibility.

The start of that discussion was around how to better promote platform availability in documentation, indicating a library supports windows, linux, or embedded in addition to Apple’s platforms. The discussion slid sideways into what’s possible (and what are the challenges) in supporting documentation across the matrix of both platforms and versions. Internally, Apple is merging together different platforms for their docs, but coordinating compilation and collection of the resulting is limited to a single platform with the swift-docc plugin today.

Swift Package Index supports choosing what platform is used to build your docs (linux vs xcode using a scheme to target iOS or macOS), but they also don’t merge - it’s just a generation choice. (I don’t know about UniDoc - Dianna wasn’t at todays meeting to provide any background details for their service).

Cross compilation might help out here, but there’s also the challenge of needing relevant SDKs available for some of the other platforms. How to wrangle any of this and combine going forward was sort of a tangential (but interesting) question to the topic of how to support library authors providing information about support of various platforms for their library.

1 Like
5

i don’t think there are any hard technical reasons why symbols cannot be merged, it is just a lot of hard work to get it right as there are many quirks/bugs in lib/SymbolGraphGen so you need to spend a lot of time doing data sanitization to get useful results from it.

Swiftinit does not persist the symbol JSON files, those are extremely bloated, waste a lot of storage, and expensive to parse as JSON is not really well-optimized to store this kind of data.

Unidoc does compile binary symbol graphs that contain most of the information from the original JSON inputs; it uses these to produce the rendered documentation, and Swiftinit persists these artifacts in the cloud. these are the “Symbol Graph” files listed under the tags page for each package on Swiftinit.

right now we don’t make these publicly downloadable, but in theory we could make them available for third party use. the main challenge would be keeping up with the Symbol Graph ABI, which is quite complex and changes over time.

6

Thanks to you and @Joseph_Heck for your quick replies and work!

Thanks for the direct link/hint. Sorry for asking before checking out unidoc (particularly your recently released preview and local commands) and for my delay in responding.

After some futzing, I'm up working now with unidoc's compile command to dump the bson to investigate possible analyses. (Unidoc balks on most of my target packages, and I'm still sorting through how much I can do without standing up mongodb. (Feedback elsewhere)) Consuming the data seems manageable, and the code is pleasantly readable.

For the analyses, please let me know if there's anything I can do to help or motivate the extra work of publishing the binaries. A single snapshot-set for some proof-of-concept prototyping would validate possibilities without tangling too much in versioning (and the snapshot needn't be all-one-version). It would help even to stage partial sets by importance/traffic (e.g., apple+swift/swiftlang, pointfreeco+alamofire+rxswift, swiftyjson...).

2 Likes
7

hi Wes!

it’s quite impressive that you got this far, the compile command doesn’t really have a tutorial yet and the Symbol Graph format is even less well-documented.

do you mind elaborating what packages Unidoc is failing to compile, and what errors you are getting?

i think this is a relatively straightforward addition, we just need to change some security settings for the S3 bucket, add a CloudFront distribution for it, and add download links to the website for logged-in users.

can you open an issue in the Unidoc repo?

8

Here's a hiccup while reporting missing documentation:

Error interrupts diagnostic message from compile command · Issue #332 · tayloraswift/swift-unidoc · GitHub>

For the

extra work of publishing the binaries

Please see

Publishing symbol graph bson · Issue #333 · tayloraswift/swift-unidoc · GitHub >

Thanks!

1 Like