Modify Accessors

[michelf (Michel Fortin)]

@dabrahams

Those changes to defer might improve defer , but I don't think we can just change defer because we'll still have the problem that there's implicit control flow (exiting the function whenever an error propagates through yield ).

Since that control flow is so problematic, could we just get rid of the need to bother with it? Say you have this generator:

func generate() {
   yield a
   yield b
   yield c
}

and the first yield fails, we could just continue running the function as if nothing happened. The following yields would just become no-ops. As far as the generator is concerned, yield can't fail, it'll just do nothing if the yielding context disappears.

We should also have a way for the generator to avoid doing useless work, but as an optimisation. Just check for the context being broken:

func generate() {
   yield &a
   if brokenYield { return }
   yield &b
   if brokenYield { return }
   yield &c
}

So if you don't bother handling the error path, your generator will continue to behave correctly. The only downside is it'll do more work than necessary.

There's also no need to check after every yield: only check when it actually avoids work:

func generate() {
   var (a, b) = fetchAB()
   yield &a
   yield &b
   commitAB(a, b)

   if brokenYield { return } // this is a good point to stop

   var (c, d) = fetchCD()
   yield &c
   yield &d
   commitCD(c, d)
}

[dabrahams (Dave Abrahams)]

@michelf

we could just continue running the function as if nothing happened. The following yield s would just become no-ops.

That's still implicit control flow, only it's much subtler. Just one example of subtlety: if there are side-effects as part of the yield expression, do they execute? Doesn't matter what your answer is; the point is you have to think about it.

[sveinhal (Svein Halvor Halvorsen)]

@dabrahams

It's still a lie; we haven't yielded yet.

Not that it matters much for the discussion at hand, but that was my point: if you invert the flag, it’s not a lie: it haven’t yet yielded successfully.

But I digress. ¯_(ツ)_/¯

[michelf (Michel Fortin)]

@dabrahams

Just one example of subtlety: if there are side-effects as part of the yield expression, do they execute? Doesn't matter what your answer is; the point is you have to think about it.

There's not question in my mind that it should behave the same as throwing function calls with inout parameters: setters are run when an exception is thrown in such a case; so setters must be run when yield fails too. I think it'd be problematic if yield behaved any differently.

To be clear, all the code you see in the generator would run as written until the end of the function. If you have yield &a it'll run the getter and setter of &a (or _modify) even if there's no one to yield to anymore, just as if you were passing it inout to a function with an empty body. There is really no error path in the generator (unless you write one explicitly), it just runs it course.

While maybe not optimal, the generator should behave correctly even if you don't give a single though about errors, and that's the point.

---

The last example of my last post illustrates how good a non-failing yield is at expressing the control flow. Here's the equivalent rewrite using defer assuming a yield that exits the generator on failure:

func generate() {
   do {
      var (a, b) = fetchAB()
      defer { commitAB(a, b) }
      yield &a
      yield &b
   }
   do {
      var (c, d) = fetchCD()
      defer { commitCD(c, d) }
      yield &c
      yield &d
   }
}

The problem is it's likely you'll forget to put the commit calls in a defer with a do block for correctly scoping those defer. I said "likely you'll forget", but I should have said you'll definitely forget it you aren't mindful of the error path while writing the code.

Using finally makes the flow a bit more linear, but doesn't really help to remind you you need to write all this control flow in the first place:

func generate() {
   do {
      var (a, b) = fetchAB()
      yield &a
      yield &b
   } finally {
      commitAB(a, b)
   }
   do {
      var (c, d) = fetchCD()
      yield &c
      yield &d
   } finally {
      commitCD(c, d)
   }
}

I didn't want to write an example using yield … else because it's just awful having to duplicate those commit calls. That said, the obligatory else would be a good reminder of the error path. People generally want to focus on what they want the code to do when writing code, not error handling and they might see the compiler complaining as an annoyance you need to silence by writing an empty else. If you do that you end up with the same problem as above, but at least you were warned I guess.

I think it's far better if we can just get rid of the error paths when they're not needed.

[lorentey (Karoy Lorentey)]

Drive-by comment: we should allow nonmutating modify accessors, for use in cases where the type implements reference semantics.

One example of such a case is ManagedBufferPointer.header, which unfortunately has a mutating _modify in Swift 5.3.

[Jason_Gregori]

@John_McCall

The implicit unwind edge out of yield is a problem, and I don't think it's reasonable to expect programmers to proactively remember it, and so something like unconditionally requiring try yield is probably the best choice. If we come up with a reasonable way to allow try to be omitted on ordinary expressions, well, it should apply equally here.

I agree with John. Maybe if yield is the last line of modify, try can be omitted. But in other cases swift could require try and that would be the "tell" that the code after this may not run and to be careful.

[tera]

i wonder if swift could have used the same approach as C++ - their std::vector / std::map are also COW-able value types and have to deal with this issue.

[Karl]

As this thread approaches its 2nd birthday , a quick summary of where thing stand:

• (ABI stable) The standard library makes extensive use of _modify
• (ABI stable) Foundation's new AttributedString makes use of _modify
• (ABI stable) swift-system's FilePath.ComponentView makes use of _modify
• swift-collections makes use of _modify (just one example, there are many)
• swift-nio tries to do the right thing and has a modify method instead . Bless 'em.

In my own projects, I of course avoid underscored functions or attributes, because they technically are not stable and could disappear or change semantics without notice. The only exception is _modify. Unlike most underscored features, using _modify accessors in third-party code is well within the realm of "acceptable risk" IMHO.

It's critical to many API designs, and used so extensively (by ABI stable libraries, no less), that the chance of it being removed or suddenly becoming incompatible with current behaviour is basically zero.

[Lantua]

@Karl

the chance of it being removed or suddenly becoming incompatible with current behaviour is basically zero.

It saddens me that the modify accessor as is written has become de facto standard long before it has entered a wider-audience discussion. More so now that people start using it extensively. From the design perspective, that only makes it much harder to iron out the wrinkle around the feature, and this accessor sure has a few.

Don't get me wrong, from the practical standpoint, the modify accessor is an important and a much-needed feature for essentially any performance-sensitive code, especially in a CoW-dominant language like Swift. It's just that it could have been much more. It could have felt like a deeply-integrated feature rather than a tacked-on one, could have.

[Karl]

Some of the ideas in this thread, like requiring a try yield with the ability to catch errors, would (I'm guessing) probably not be ABI-compatible with libraries that have already shipped. And some libraries, like swift-system and swift-collections, are also shipped in source form, so any projects with a fixed-version dependency would fail to build if the compiler stopped accepting _modify with the exact behaviour it currently has.

So yeah, (underscored) _modify is de facto stable and the compiler will have to support it for a long time. But that doesn't mean we can't change things for the separate (non-underscored) modify feature.

[Jumhyn (Frederick Kellison-Linn)]

@Karl

So yeah, (underscored) _modify is de facto stable and the compiler will have to support it for a long time.

I don't think that there's anything from a policy perspective that would prevent special handling being added to allow _modify to be updated in ways that wouldn't break Apple frameworks but may be incompatible with arbitrary libraries which depend on it. It is, after all, an underscored feature which has not been accepted via the evolution process.

It's also possible that there are non-ABI implementation details of _modify that are not relied on by Apple frameworks, but are relied on by third-party code. As a policy matter, again, we should not consider the behavior of _modify constrained by these clients.

Now, it may be that, pragmatically, such special handling wouldn't be worth the effort or wouldn't be possible to achieve in practice. Just want to make sure we don't overstate the stability of _modify. Any changes could be made without an evolution proposal, only taking Apple's migration story into account.

[ahti (Lukas Stabe 🙃)]

@Karl

It's critical to many API designs, and used so extensively (by ABI stable libraries, no less), that the chance of it being removed or suddenly becoming incompatible with current behaviour is basically zero.

I don't think that is as sure as you think it is, and I certainly would discourage anyone from taking this as advice that one can just use _modify and expect no consequences.

The only thing that is ensured by its use in ABI-stable libraries is there will have to be some (probably private) feature to let the libraries that use it right now continue to expose the same ABI. The ABI and spelling of any modify that is eventually accepted need have nothing in common, and _modify itself could disappear or completely change behaviour at any point.

[Karl]

Try to imagine what would happen if _modify actually did change its behaviour - it would break the builds of everybody using source libraries such as swift-system and swift-collections. At the very least, there would need to be a lengthy transition period where both implementations coexisted, until it was certain that nobody was using an older version of the package.

The ABI stability issue is one thing. The source stability requirements for these packages is harder to get away from.

[Jumhyn (Frederick Kellison-Linn)]

Only if it changes behavior it a way that breaks those specific packages. I don't know that it would break the source stability promise to update the behavior for, say, all modules not named Swift, SystemPackage, Collections, etc., or to update behavior that Apple packages happen not to depend on but third-party libraries might. The compatibility promise for an underscored, unofficial feature is quite narrow, absent any official guidance from the core team/Apple themselves.

ETA: source stability could be achieved even by adding code which specifically checked for an exact source match (down to the character) of each existing use of _modify in Apple frameworks and conditioning its new behavior (or removal) based on that. I don't think that's a particularly likely future, but it wouldn't violate existing ABI or source compatibility promises to update the language in such a way.

[Karl]

@Jumhyn

I don't know that it would break the source stability promise to update the behavior for, say, all modules not named Swift , SystemPackage , Collections , etc.

Even that wouldn't be sufficient; module names are not universally unique. So this hack would need to permeate to the package manager... and even that wouldn't be sufficient; you might be using a mirror or fork.

@Jumhyn

source stability could be achieved even by adding code which specifically checked for an exact source match (down to the character) of each existing use of _modify in Apple frameworks and conditioning its new behavior (or removal) based on that.

What happens if they need to fix a bug? It would break _modify on older compilers which don't recognise the new source.

But yeah, of course it is not an official language feature and generally I don't support using those either; still, in this specific case, I think it stands that:

@Karl

the chance of it being removed or suddenly becoming incompatible with current behaviour is basically zero.

Not completely zero, but unlikely enough that I wouldn't worry about it. I'm more worried about random, accidental regressions than _modify suddenly and intentionally changing.

[Jumhyn (Frederick Kellison-Linn)]

@Karl

Not completely zero, but unlikely enough that I wouldn't worry about it. I'm more worried about random, accidental regressions than _modify suddenly and intentionally changing.

Yeah, I think we basically agree. I just want to make sure that we're precise in this thread about what "behavior" and "regression" encompasses. To the extent that Apple's libraries don't depend on a certain behavior or feature, I don't think it really makes sense to even call a change a "regression," even if it breaks third-party code.

[ben-cohen (BenC)]

@Karl

Try to imagine what would happen if _modify actually did change its behaviour - it would break the builds of everybody using source libraries such as swift-system and swift-collections.
[...]
The ABI stability issue is one thing. The source stability requirements for these packages is harder to get away from.

There are no source stability commitments when it comes to underscored keywords or attributes and no-one should have any expectations in this regard, either at compile time or runtime.

However, if _modify were to become modify, this would also provide an opportunity to mitigate source breakage by only applying the new syntax or semantics to modify and not _modify, if that proved a worthwhile expenditure of effort.

[Karl]

@ben-cohen

There are no source stability commitments when it comes to underscored keywords or attributes and no-one should have any expectations in this regard, either at compile time or runtime.

I agree that this is usually how it works.

The point is that the authors of swift-system and swift-collections obviously had these expectations when they made use of _modify in their libraries, both of which have released 1.0 (stable) versions. The proverbial horse has bolted.

It seems that that was an error - it happens. In future they (and others) will need to be more careful about publishing releases using unstable language features. But as things stand, there appears to be very little (if any) room for _modify to change, without breaking clients who did the right thing and never used _modify directly in their code.

[xwu (Xiaodi Wu)]

@Karl

The point is that the authors of swift-system and swift-collections obviously had these expectations when they made use of _modify in their libraries, both of which have released 1.0 (stable) versions. The proverbial horse has bolted.

I’m not sure why that would be. The source stability guarantees are for the end users who adopt the libraries, not the libraries themselves, which by using an unreleased feature have committed to updating/rewriting as necessary in tandem with the feature’s evolution.

[Karl]

Adopters may be using an older version of the library (perhaps a dependency on a specific version, or a mirror or fork).

If current releases of these packages are not guaranteed to compile with Swift 5.5.2 or 5.6 or 6.0, they are not stable releases. Issuing a patch later is not sufficient.