Swift Evolution Metadata Proposed Changes

[James_Dempsey]

Thank you for your feedback and questions.

Historically, the metadata file has been 'semi-private', available at a public URL for use by the Swift Evolution Dashboard. A number of community members have discovered the URL and have built additional clients.

@nikolai.ruhe

is there documentation of the new (or old) schema somewhere?

At present the best documentation is the proposal in this thread's original post.

@nikolai.ruhe

For example, I have a question about the semantics of the "link" field in the proposals. It seems to be a relative link based on https://github.com/apple/swift-evolution/blob/main/proposals/. The link's base is not mentioned in the file. Maybe there could be a top-level "extractionRoot" field or similar?

Yes, that is the base URL.

I agree that this should be documented, so clients can construct the full URL if desired. Since the value has remained constant over a period of years with no known plans for it to change for years to come, I don't know that it needs to be included in the metadata itself.

@nikolai.ruhe

I'd love if there was a basic Swift package to parse the file into proper types. This could also be a great place to find documentation.

I completely agree and have been thinking along the same lines.

[nikolai.ruhe]

@James_Dempsey

Changed top-level field names

• From sha to commit

There's still a "sha" field in the individual proposals. Is this intentional?

[James_Dempsey]

@nikolai.ruhe

@James_Dempsey

Changed top-level field names

• From sha to commit

There's still a "sha" field in the individual proposals. Is this intentional?

Yes, the top-level commit field is the commit hash of the repository.

The sha field is the sha of the proposal file itself and mirrors the naming of the value returned by the GitHub API.

[nikolai.ruhe]

@James_Dempsey

[...] a basic Swift package to parse the file into proper types. [...]

I completely agree and have been thinking along the same lines.

If you'd appreciate any help with that, please let me/us know.

Also: Thanks for all the work you already put into this!

[James_Dempsey]

@Victor_Martins

I propose we add the following fields to each proposal:

{
  "id": String,
  "title": String,
  ...
  "previousProposalIDs": [String]?,
  "discussions": [ // A list of the related discussions in chronological order
    {
      "title": String, // "Pitch", "Review", "Acceptance", ...
      "link": String
    }
  ]
}

I think these would be useful additions. For example, I have been thinking it might be a good addition to the dashboard to link directly to the review thread for proposals in active review.

I also think these are good as originally pitched in terms of naming / structure / types.

Additional feedback from others is welcome.

Some thoughts:

previousProposalIDs

• This is not used often, so I agree it should be an optional value, similar to implementation and trackingBugs.

• In the original header field, the value contains relative links to the previous proposals. I agree that only the proposal ID is necessary, since the full information about the previous proposal is present in that proposal's metadata. (Side note: The proposal template says the link text should include more info about the relationship such as "Extracted from SE-XXXX", but none of the existing proposals with this header field follow this guideline.)

• Although the original header field is named "Previous Proposal", I agree that since the metadata only includes IDs that previousProposalIDs is a better name.

• I think including only the previous proposals is preferable to 'related proposals' which could include both previous and follow-on proposals. This makes the relationship clear and it isn't too onerous to derive the inverse relationship for clients who wish to do so.

discussions

• All proposals should have links to discussion threads so a non-optional array makes sense.

• The original header field is labeled "Review". I think it is a good rule of thumb to prefer the naming used in the proposal, barring a good reason.

  In this case, I think discussions is much more clear in describing what the values are. Even the proposal template for the "Review" field defines it as: "a history of all discussion threads about this proposal".

• Although the header field is supposed to list these in chronological order, some proposals currently list them in reverse chronological order. I believe the metadata field should be in the same order as in the proposal and that the proposals should be updated via PR.

  (In general, I don't think the metadata extraction should be performing additional processing to fix errors in proposals. The proposals should be updated.)

• I could see title or name being used for each discussion, but I am fine with title.

[James_Dempsey]

@nikolai.ruhe

If you'd appreciate any help with that, please let me/us know.

Also: Thanks for all the work you already put into this!

Thank you. I will keep the forum up to date with any further developments.

And you are very welcome!

[Victor_Martins]

@nikolai.ruhe

I'd love if there was a basic Swift package to parse the file into proper types. This could also be a great place to find documentation.

I'll certainly have to implement it for my app. If you aren't asking for an official package, I could probably move it to a package and make it public when I have some time!

[Victor_Martins]

@James_Dempsey

I think these would be useful additions.

@James_Dempsey

I also think these are good as originally pitched in terms of naming / structure / types.

Cool!

@James_Dempsey

(In general, I don't think the metadata extraction should be performing additional processing to fix errors in proposals. The proposals should be updated.)

Got it. And it makes a lot of sense to me.

[Victor_Martins]

Something I've found it a bit hard to grasp is the difference between upcoming feature flags and experimental feature flags. There's a full explanation on swift.org about Upcoming Feature Flags, but I don't think there's a clear explanation of the experimental ones.

At the moment, I believe Upcoming Feature Flags are compiler flags that allow us to make use of source-breaking changes that were proposed, accepted and implemented. While Experimental Feature Flags are compiler flags to be used during the review of the proposal to allow a better understanding and improve the discussions.

Could you confirm that my understanding is correct? If that's the case, I believe the new schema has a field for upcoming feature flags but is still missing support for experimental feature flags. Was this considered? I tried reading the previous posts but this is still not clear to me.

[James_Dempsey]

@Victor_Martins

Something I've found it a bit hard to grasp is the difference between upcoming feature flags and experimental feature flags. There's a full explanation on swift.org about Upcoming Feature Flags, but I don't think there's a clear explanation of the experimental ones.

At the moment, I believe Upcoming Feature Flags are compiler flags that allow us to make use of source-breaking changes that were proposed, accepted and implemented. While Experimental Feature Flags are compiler flags to be used during the review of the proposal to allow a better understanding and improve the discussions.

Could you confirm that my understanding is correct?

The original proposal SE-0362 describes both, and yes, your understanding is correct.
(At least, it matches my understanding, which I believe is correct.)

I believe the new schema has a field for upcoming feature flags but is still missing support for experimental feature flags. Was this considered?

Yes. The main reason is that upcoming feature flags already have a standard representation in the proposal files, so they can already be reliably extracted. Experimental feature flags are included in ad hoc ways which are not currently able to be reliably extracted. To be reliably extracted, a standard would need to be defined and made part of the evolution proposal templates, which would entail proposing something to the Language Steering Group and then adding metadata fields based on what the LSG decided.

Going through that process seemed like it would be better deferred for a few reasons:

• People who are interested in reviewing a proposal and further downloading nightly toolchains to enable experimental features are going to be reading the original proposal. So, it doesn't seem particularly urgent to present the information on the dashboard. (As opposed to upcoming feature flags, where the dashboard can filter to answer questions like "what are all the upcoming feature flags I can use to prepare for Swift 6")

• Adding experimental feature flags in whatever form they may take in metadata is additive and wouldn't be a breaking change to the schema.

• I wanted to be respectful of the LSG's time since they are in the midst of working through a large number of proposals and reviews as they're driving towards Swift 6. (And this effort already took up some of their time determining how to annotate UFFs with which language mode the flag is no longer needed because the feature is always enabled).

• The breaking change metadata proposal and transition is already a fair amount of effort so deferring limits the complexity.

[James_Dempsey]

FYI, I added discussions and previousProposalIDs to the original proposal post.

As I was writing up the description for discussions, it occurred to me the title of the discussion is really the title of the discussion thread in the forums, so I used 'name' instead.

[Victor_Martins]

@James_Dempsey

To be reliably extracted, a standard would need to be defined and made part of the evolution proposal templates, which would entail proposing something to the Language Steering Group and then adding metadata fields based on what the LSG decided.

Ohh got it! Yeah, taking this into consideration it makes a lot of sense to defer this discussion.

I'll probably keep diving into the full proposal content to extract this information for my use case.

[MahdiBM (Mahdi Bahrami)]

+1 for the fact that @James_Dempsey increased the transition period from 1 month to 3 months Thanks for that. (reference)

[James_Dempsey]

The review period was 30 days, but the transition itself as described in the proposal was always planned to be 90 days.

But I'm glad that timeframe will work for you.

[MahdiBM (Mahdi Bahrami)]

@James_Dempsey

The review period was 30 days, but the transition itself as described in the proposal was always planned to be 90 days.

Ah, makes sense. Thanks again!