Evolution process discussion

[Karl]

@jawbroken

It's interesting that most of the suggestions here are about adding more stages to the process and making review threads run longer and receive more posts, given that this discussion was spawned by a review process that was described as “particularly drawn-out” and “tested the patience of some of the participants”. It's always easy to argue for more process, more scrutiny and more input from more people, but I don't know how that helps here.

I guess not all of us feel that it was so drawn-out. Perhaps by swift's standards.

Personally, I've felt for a while that swift moves too quickly. Things go almost immediately from "good idea" to "let's ship this in the next release of the standard library". That will hopefully be addressed by the new preview library:

Adding these packages serves the goal of allowing for rapid adoption of new standard library features, enabling sooner real-world feedback, and allowing for an initial period of time where that feedback can lead to source- and ABI-breaking changes if needed.

At the same time, you can make a reasonable argument that Swift also moves too slowly (without it being a contradiction). We still have a lot of work to do in order to provide a pleasant interface to basic functionality.

Personally, I think the major job of the standard library is to provide interface types. Whatever other types or algorithms your module uses internally is its own business, but when it comes to interfacing with other modules, everybody needs to agree on what an Array is, for instance. However, it's (more-or-less) OS neutral, so it doesn't help with lots of the code we actually write. With that in mind, if you think Swift moves too slowly, I think you need to look beyond the standard library:

• Foundation needs a radical refresh. It has an important role in the ecosystem, as it provides the interface types for OS-level functionality (like the filesystem), like the standard library does for generic types and algorithms. You definitely want all modules to have the same understanding of FS primitives. However, the current interface doesn't fit well with modern Swift, it's not very pleasant to use, and there's nothing we (the community) can do about it. I think Apple's arguments against an open evolution process have become weaker, since the Swift standard library is now also a Darwin system library. There's no reason for the new Foundation to be less open than the standard library. Maybe this time we'll even get an OrderedSet

• We also need a new library (let's call it Basic), for important non-interface types and functionality. This would include useful collections like BTree, and modern interfaces for things like command-line parsing. The reason this is not part of Foundation is so that its version can float, and so that it can be back-deployed to older Darwin OSes.

In other words:

	Stable	Floating
Generic	stdlib	stdlib-preview
OS-level	Foundation	"Basic"

The major difference between a "Basic" package and stdlib-preview, is that things in the preview package are expected to become stable and migrate to the stdlib eventually. Things in Basic wouldn't necessarily have that expectation.

[Matt_McLaughlin]

Sorry all - this is a bit long...

think the central issue is that we want to front load as much as possible all the OhCrapIDidn’tThinkOfThat problems that require the proposer (and community) to work through thorny issues. And we need the proposer and community at large to be able to correctly identify those issues that the core team thinks need more work earlier. What draws reviews out is when legitimately important to deal with issues trickle in little by little over longer time periods or the proposer and community don’t realize that something is going to be an issue to the core team.

So the operative task is getting the people capable of identifying the OhCrapIDidntThinkOfThat issues into the thread as early as possible, and making sure the core team gives feedback early to shape the direction of the conversation.
———
A couple of spitball thoughts:

What about a formal review cadence? Like reviews always drop on the first week of the month? That way infrequent but insightful posters can know the time when their presence could add the most value to the community.

And perhaps the review manager could be more proactive about soliciting feedback. Perhaps spend a few minutes looking up who has ever commented on a similar proposal and mentioning them so they get a notification.

To that end, since it is the job of the proposer to do their due diligence before proposing perhaps we could make it required that the proposer list the threads that they’ve read through for background. Then the review manager can just pull the posters from those threads for notification come review time.

Finally, while I get the core team not wanting to poison the process by weighing in too early, I think it would be useful to at the very least get a Top 5 Issues list before the review begins so that posters know which part of the proposal it’s most important to kick the tires on.

[dabrahams (Dave Abrahams)]

Moderator note: this post was originally the first in a separate thread; with Dave's permission, I've merged it and my response into this thread.

When SE-270 was accepted, the review manager invited a discussion about the review process:

@John_McCall

This review was particularly drawn-out… the Core Team would be very interested in suggestions for how review managers can draw attention to these debates and counter-proposals and solicit community feedback on them without excessively dragging out reviews or requiring lots of additional work from proposal authors.

From the full text of the acceptance announcement, it's clear that the core team is dissatisfied with something about how the review went. I interpret what I read there as follows—which may be inaccurate; please feel free to correct me:

• The core team thinks the author's response to counter-proposals could have been more complete.
• The core team is concerned that when reviews get very involved, potential reviewers tune out.
• The core team is concerned about the effort it takes from authors to get a proposal accepted.

I'd like to point out first that this was not actually a drawn-out review. There were just 15 posts over the nine days between the announcement of the review starting and its resolution. If the process of getting the proposal accepted was exhausting it's because it actually got three back-to-back reviews, in quick succession. Given that review feedback resulted in substantial revisions each time, it seems clear to me that the proposal should have gone back to the pitch phase for more collaborative design work, but instead we returned to find yet another review had started, with a new revision of a large proposal.

Having worked with most of members of the core team for many years, I know and respect them, and draw no conclusion about why the proposal was handled this way. That said, to someone less connected it could easily appear that core team had decided the proposal needed to pass, and if there were objections, it was just going to keep running reviews until everyone—including core team members—got so sick of the arguing that they decided to “just accept it already.” Therefore, I think handling feedback this way is bad PR, discouraging to reviewers, and so tiring for everyone that it can't help leading to worse results for the language.

In the end, I didn't say anything about counter-proposals because I don't think they're at the core of a problem—I just put that in the title because it's what we were invited to discuss—and therefore I don't know if any of what I've written actually addresses the core team's concerns. But that's my take on what could have gone better, as a reviewer.

[John_McCall]

First off, thank you for your response.

I do feel that counter-proposals are at the core of the question. This thread was not meant to be about SE-0270 specifically, but it's an illuminating example. Over the course of SE-0270, we received a lot of feedback from the community. Most of that feedback was over what we would normally consider minor aspects of the proposal: method names and whether to include a few secondary APIs. Your feedback was the major outlier, because while you did make some comments of that kind, you also made a number of much deeper objections and suggested an alternative design that was quite different in nature.

The Core Team is satisfied with how the minor feedback was handled. The author responded by making (in our view) minor revisions to the proposal, and the community seemed generally satisfied with those changes. In the end, there were some disagreements about how to apply the naming guidelines, and the Core Team simply made a decision on those issues. While the overall review did drag out a bit, that was largely due to the US holiday schedule — well, that and the review-manager switch, which we should've foreseen the need for.

Our concern really is about our handling of counter-proposals like yours. In this case, while the proposal author and the Core Team did spend some time discussing it, it wasn't something the community ended up providing much feedback about. In general, the Core Team wants to prevent the evolution process from being overtly biased towards the first proposal to make it into review, and ensuring that counter-proposals are adequately "briefed" and discussed during review is an important part of that. That's why we're interested in ideas for how to better call attention to major new ideas that come up during review. That may just be as simple as the review manager explicitly directing people's attention to them.

[dabrahams (Dave Abrahams)]

@John_McCall

In the end, there were some disagreements about how to apply the naming guidelines, and the Core Team simply made a decision on those issues.

Characterizing the disagreements present in the end as being about how to apply naming guidelines trivializes something that is IMO rather serious, besides being inaccurate—there was no disagreement about applying guidelines in the last review. . The first version of the proposal demonstrated (in my view) that it lacked clarity about the abstraction being proposed, and the disagreements at the end demonstrated (to me) that the lack of clarity had persisted through two subsequent revisions. It seems to me that the most likely reason that the the issue is being characterized in this trivial way is that the core team was just tired of dealing with it.

Our concern really is about our handling of counter-proposals like yours. In this case, while the proposal author and the Core Team did spend some time discussing it, it wasn't something the community ended up providing much feedback about.

That is easily explained, in my opinion. Most peoples' tolerance for disagreement is extremely limited. Because the proposal was sent immediately back into review without confirming that the concerns that spurred counterproposals had actually been addressed, there was a high risk that the disagreements that caused me to make counter-proposals would persist, which in fact they did. By the time we got to the second round of review and disagreement was still evident, most people tuned out.

I should also add, I made counter-proposals in an effort to be constructive rather than simply critical, but even if I had just made substantive criticisms with no proposals, it would have been far better for the core team to have waited for a consensus position to form in a second pitch phase. In other words, I still don't think the presence of counter-proposals are the issue.

In general, the Core Team wants to prevent the evolution process from being overtly biased towards the first proposal to make it into review, and ensuring that counter-proposals are adequately "briefed" and discussed during review is an important part of that. That's why we're interested in ideas for how to better call attention to major new ideas that come up during review. That may just be as simple as the review manager explicitly directing people's attention to them.

IMO it's unrealistic to expect to observe broad interest in one person's deeply considered objections, especially if those objections become a repeated source of disagreement over three closely-spaced reviews. The better course would have been to encourage consensus building among those who were interested, outside the pressured context of a formal review.

[typesanitizer (Varun Gandhi)]

The Rust community also has discussed this issue before, so their observations might be helpful. (blog, discussion thread).

I like the idea of breaking up the proposal into several steps each with clearly defined goals. I want to draw particular attention to one of the points mentioned there:

Steady state: At some point, the discussion reaches a “steady state”. This implies a kind of consensus – not necessarily a consensus about what to do, but a consensus on the pros and cons of the feature and the various alternatives. Note that reaching a steady state does not imply that no new comments are being posted. It just implies that the content of those comments is not new

I feel like evolution proposals sometimes spend too little time talking about the rest of the solution space (alternatives) and the downsides. There are lots of axes to consider, such as tooling support (formatters, linters, syntax highlighting, debugger etc), quality of diagnostics, potential compiler performance cliffs, backwards deployment and so on. Often, these get brought up in the discussion but the resulting conclusion ("yes, we will take a hit on X but we accept that tradeoff" or "no, we don't want to give up X which is why we do this") doesn't necessarily make its way back into the proposal document.

Taking time out to discuss this also means that a proposal already incorporates its counter-proposals in a way, because it talks about the different points in the design space and why it prefers one over the other.

---

At the same time, I feel like the linear nature of the forum is not really well-suited to having multiple overlapping discussions. It is fine if there are 2-3 discussions but with a large proposal, it can quickly balloon into many slightly different conversations and it isn't really clear who is in agreement with whom. Using GitHub issues might be one possible solution. Having "shepherds" (either the review manager or someone else) actively summarize the state of the discussion in between might also be helpful. The different stages kinda' force that to happen as there needs to be some summarizing at the end of each stage.

---

At the same time, this process might be a bit too much for certain proposals which are small in scope (certainly, what exactly falls under small is subjective... for example, I think SE-0276: Multi-pattern catch clauses qualifies as small whereas a type system feature like variadic generics would not qualify as small). For those, we can continue to have the more lightweight process today.

[xwu (Xiaodi Wu)]

@typesanitizer

The Rust community also has discussed this issue before, so their observations might be helpful. (blog , discussion thread).

I like the idea of breaking up the proposal into several steps each with clearly defined goals.

This is a very interesting document to read. There are as many steps proposed after implementation as there are before, and even end-user explainers and docs are drafted before several stages of evaluation. Moreover, the dynamic between the community and the team is imagined as one where each has a role to play in an ongoing dialogue; commitment to bringing a feature into the language is a staged process, not one where the community simply talks and talks until the core team brings down a verdict. Bringing such a structured process to Swift Evolution could foster a more vibrant community.

People who evaluate proposals come from many different perspectives; for instance, some are speaking to the design of the feature in question, while others are speaking to how the feature will affect the design of their own work. We ask all-comers to evaluate the proposed design of certain features, and lately it’s become clear that some people misunderstand what feature is being proposed or even why—which is hardly surprising when the relevant docs and explainers appropriate for a diverse audience don’t necessarily even exist at review time.

A multi-step process would allow people better to choose what aspects of the process they’d like to be involved in. This will differ from topic to topic even for one person. For instance, I am certainly interested in SwiftPM’s maturation, but I am much more capable of speaking to the design of new numerics APIs than of SwiftPM features. Therefore, I would want to be involved more heavily in the earlier design stages of numerics-related proposals but only the refinement stages of SwiftPM-related proposals. I would be more likely to help write explainers for the former, and more likely to want to read explainers for the latter in order to be able give useful feedback on the user experience.

I feel like fundamental issues have arisen when participants are trying to help design a given feature or have fundamental things to say about the design, and the proposal itself is at the final stages of evaluation or has even gone into extra innings. In the workflow sketched out in the linked RFC, designing is something like step 2, and final approval based on refining the user experience is something like step 10. It is little wonder that the process becomes frustrating for all involved when some are on step 2 and others are trying to complete step 10.

Finally, I feel like the core team needs to think seriously about what stages of the process they actually want to elicit community involvement for, which has clearly differed from proposal to proposal. These expectations need to be set out explicitly.

[Lantua]

@xwu

Finally, I feel like the core team needs to think seriously about what stages of the process they actually want to elicit community involvement for, which has clearly differed from proposal to proposal. These expectations need to be set out explicitly.

I start to agree much more strongly about this. Especially in the on-going SE-0279: Multiple Trailing Closures - #319 by karim, a noticeable amount of people start to say something like this will probably be pushed through, because SwiftUI. It’s rather frustrating when we don’t know what’s on the table, and what’s not. It makes for the feeling that opinions haven’t been heard.

It’s fine, great even, to say up front that a proposal will definitely accepted (possibly with some minor tweak), or that this particular problem will be a priority and so a solution of some form will need to be created. Or this is a potentially good solution, but we can wait for a better one. A clear message is probably important here.

Granted, these kind of things may not be known initially, but should there be an in-thread announcement, it should at least be highlight, pinned, or edited into the original post, so that everyone is on the same page.

[DevAndArtist (Adrian Meister)]

I second what you said. Even though I support the direction the mentioned proposal is going, I don't like the rushed and forced design. We should invest more time to polish it since we kinda already know that it will be accepted anyway, so we should at least get a chance to tweak the design so it suits not only SwiftUI but at least some major part of the Swift community and does not hurt the other part.

Just remember the property wrappers threads. So many ideas and tweaks had to be shared to go from a super restricted and inflexible design to something a lot of us are using on their daily bases now.

[Jumhyn (Frederick Kellison-Linn)]

I think it's somewhat insulting to call the design "rushed and forced." Even though I prefer a different direction, it's clear that the proposal authors have well reasoned arguments supporting the design. They're not arguments that I find personally compelling when weighed against the alternatives raised in the thread, but it's certainly not obvious that those counter-proposals are better than the proposal under review, given the number of community members who are actively arguing in favor of accepting the proposal as-is.

[DevAndArtist (Adrian Meister)]

Not meant to offend anyone with my wording, but it's also very strange that the pitch thread didn't find that much traction compared to the proposal thread which contains "-1" in around 99% of the replies.

I view the design as "rushed and forced" because the proposal only explored a few possible syntactical designs to solve one issue, but has not considered the bigger picture or any possible future extensibility. I'm again going to compare this to property wrappers. The proposal went through several pitch threads and review threads and it went from a highly restricted design to something that can be extended in the future with even more power (we just need some other language features first). However this is not even close for the trailing closure proposal. It would add one new sugar syntax and we're stuck with it forever, even if it might block something in the future. Sure I can't tell you now what it can potentially block, all I'm saying is that I'm highly worried that we haven't explored that area for this particular proposal.

[Jumhyn (Frederick Kellison-Linn)]

The feedback was not as overwhelmingly negative as you claim and it's dismissive of the authors and proponents of the proposal to assert otherwise. But beyond that, I think your point that "the pitch thread didn't find that much traction" (though I'm not sure I agree—it had a decent amount of participation from many highly active community members) is indicative of part of the issue. It's clear from the review thread that many community members have strong feelings about this proposal—why wasn't the pitch thread as contentious as the review thread?

The answer, I think, is related to some of the problems that @xwu notes above. It does seem like there's a step missing when a proposed change can go from a dormant thread without much participation to "one week until decision" without much warning. It would be unreasonable to expect full participation from the community on every pitch (especially since many of them never end up manifesting as formal proposals), and so the start of a review constitutes the first signal from the Core Team to the community (absent participation in pitch threads) that a particular problem seems at least potentially worth addressing.

Maybe that signal is worth formalizing into an actual step of "Exploring the solution space" or something of the sort. This would essentially be the Core Team indicating that the "Motivation" section outlines a hole that they might be interested in filling in the language, if the appropriate solution can be found, and opening a discussion around that problem without an inherently "blessed" solution, as there is in review threads.

[Jon_Shier]

@Jumhyn

why wasn't the pitch thread as contentious as the review thread?

Really? It shouldn't be shocking that pitch thread generally has less participation than review threads, especially for contentious issues which most people believe are bad ideas (i.e. "I don't need to participate in this thread, there's no way this makes it to review!"). This is doubly so for pitches with no obvious evolutionary paths. Triply so for a pitch that went into review so quickly, since there was no time to incubate alternate ideas like those proposed in the review thread. I think your suggestion about more communication is good but I don't think anyone should be surprised when pitch and review threads have differing levels of participation.

[Ben_Cohen]

@Jumhyn

@DevAndArtist

it's also very strange that the pitch thread didn't find that much traction compared to the proposal thread which contains "-1" in around 99% of the replies.

The feedback was not as overwhelmingly negative as you claim and it's dismissive of the authors and proponents of the proposal to assert otherwise.

As the review manager of the proposal, and without weighing in on the other aspects of this conversation, I would like to reinforce @Jumhyn's point. The review is by no means as negative as described here, and to say so is a clear misrepresentation.

[Jon_Shier]

Just to provide some data, my search using Discourse shows:

• 18 +1s
• 2 -0.5s
• 37 -1s

Not as lopsided as I thought but still 68% negative. How much that matters who knows.

[xwu (Xiaodi Wu)]

@Ben_Cohen

The review is by no means as negative as described here, and to say so is a clear misrepresentation.

@Jon_Shier

Not as lopsided as I thought but still 68% negative. How much that matters who knows.

I went through the first 18 unique replies that were overall appraisals of the proposal, skipping replies that were questions, clarifications, replies to other reviewers, etc. It has been said, of course, that review threads are not polls, and neither do "likes" necessarily reflect agreement in totality, but I think the overall tenor of the conversation is quite clear:

(Why 18? Because both the length of replies and number of "likes" diminished dramatically by that point, suggesting that many participants might have felt the comments to be repetitive or tuned out by then. And because my time is finite and the trend was pretty evident.)

Overall appraisal / likes

1. "-1" / 9
2. "I don't like it" / 26
3. "-1" / 41
4. "-1" / 5
5. "-1" / 3
6. "-1" / 1
7. "+1" / 2
8. "+1" / 2
9. "I think rushing out this sugar would be a mistake" / 18
10. "+1" (there was no other text in this reply) / zero
11. "I have doubts" / 31
12. "I'm a big fan" / zero
13. "+1 to the proposal as written, with the -0.5 caveat" / 1
14. "-1" / 3
15. "-1" / 13
16. "-1" / zero
17. "-1" / 4
18. "solid -1" / 2

This represents 156 s on 13 replies as compared to 5 s on 5 replies. Again, yes, there are all sorts of problems inherent in taking these numbers at face value, but frankly, if this trend in feedback is not considered to be greatly lopsided, then I'm not sure what sort of feedback from the community would be.

[Ben_Cohen]

I don't plan on engaging further on this topic but I didn't say it was not skewed in one direction – I said that 99% was clear misrepresentation. It is unhelpful hyperbole. And while I don't think hearts are particularly relevant indicator (because reviews are not votes, and as the same people hearted multiple negative posts), I note that you ignored the hearts on the proposal itself.

[Letan (Letanyan Arumugam)]

But we're in the weeds here. Did you only count unique people who liked posts? Just looking at posts 1 and 2 I found a few people who overlapped in liking both.

I think the only way to gain an understanding of the overall emotion of a thread would be to use some machine learning

[xwu (Xiaodi Wu)]

@Ben_Cohen

I don't plan on engaging further on this topic but I didn't say it was not skewed in one direction – I said that 99% was clear misrepresentation. It is unhelpful hyperbole. And while I don't think hearts are particularly relevant indicator (because reviews are not votes, and as the same people hearted multiple negative posts), I note that you ignored the hearts on the proposal itself.

I'm not sure how to interpret the 20 s on an announcement that a proposal is to be reviewed. The text of what's proposed isn't even on the post. Are users thrilled that the problem is being tackled? Do they like the solution proposed? But I agree we don't need to engage in this sort of exegesis here.

For what it's worth, though, 156 / 161 is 97%; it may well be an imperfect metric, and you may disagree with it for the reasons you outline (which I'm sympathetic to), but I wouldn't agree that 99% is "clear misrepresentation" and "unhelpful hyperbole." I totally understand how someone looking at that thread would come away with the impression that @DevAndArtist and @Jon_Shier share. At a minimum, I think it would be fair to say that the community's response was somewhat more emphatic than merely being "skewed in one direction."

[Jon_Shier]

For comparison, the announcement of the review of SE-0280 that started at the same time currently has 25 likes on a thread of 49 replies, 3.4k views, and 175 total likes. SE-0279's has 20 on a thread of 341 replies (and growing), 6.5k views, and 1.1k likes.