1

Previous thread: String interpolation revamp: design decisions

Fix ExpressibleByStringInterpolation

String interpolation is a simple and powerful feature for expressing complex, runtime-created strings, but the current version of the ExpressibleByStringInterpolation protocol has been deprecated since Swift 3. We propose a new design which improves its performance, clarity, and efficiency.

Work on this has progressed to a stage where we can now discuss the specific design. Please read the draft proposal and let us know what you think!

(Meta note: I could not title this thread "Fix ExpressibleByStringInterpolation" because of a vague "Title unclear" error. Something might need tweaking in Discourse.)

@Michael_Ilseman

27 Likes
2

I really like this design! It's straightforward and efficient, subsuming all of the use cases of the existing string-interpolation protocol while providing far more flexibility.

I think the section describing appendInterpolation could be more descriptive of what this accomplishes. As I understand it, we're effectively treating \(...) directly as an argument list for the call to appendInterpolation, labels and all. I think it would help to describe it that way, and give a concrete example, e.g.,

  func appendInterpolation<T: FloatingPoint>(_ value: T, precision: Int) { ... }

would permit one to write:

print("Pi to five digits: \(pi, precision: 5)")

Doug

6 Likes
3

I like the draft! So far, on a quick read, my only nit (and it's a small one) is with the argument label "totalLiteralSize":

  • For string-related work, the usual quantum is the extended grapheme cluster (aka Swift "character"), so a size measured in code units should call out the fact that it's counting code units (especially since no end user of interpolation will need to write this out, meaning brevity is unnecessary).
  • It's a count of code units just like the following parameter is a count of interpolation segments, and I see no reason to use the non-standard term "size" when "count" would do.

So, I'd suggest something like "literalCodeUnitCount".

3 Likes
4

I like that name too. We considered a few alternative names, including literalCapacity which pairs with intuition from reserveCapacity. The particular names pitched are the weakest of opinions held :slight_smile:

1 Like
5

Like your pet string interpolation use-case being LocalizableString, my pet string interpolation use case is (os_)logging.

I'm pretty happy that it looks like it can still mostly avoid building up the interpolation if it's not going to be consumed. To do so you have to transform code (appendInterpolation calls) into data, so there's still that caveat that a complex interpolation might have to build up an array, but I'm not sure that's an optimization worth designing around.

Some minor questions:

  • Are there any rules in the design about sending in empty literal segments like it does now? i.e., in the degenerate case "\(foo)". I can imagine a StringInterpolationProtocol type preferring to guarantee that it is non-empty.
  • How does literalCodeUnits interact if, for instance, your StringLiteralType is StaticString? The UTF-8 code units are being calculated in the compiler for that, will literalCodeUnits simply sum those up or will the total code unit count be the number of grapheme clusters?

Overall, it's a great redesign. Easy to follow, unlocks way more expressive API. Great stuff. I concur about the codeUnitCount naming but will decline to participate in that particular bike shedding.

1 Like
6

In the proposal, isn't the sentence:

“Strange” interpolations like (x, y) or (foo: x) which are currently accepted by the Swift compiler will be errors in Swift 5 mode. In Swift 4.2 mode, we will preserve the existing behavior with a warning; this means that Swift 4.2 code will only be able to use appendInterpolation overloads with a single unlabeled parameter, unless all other parameters have default values.

against the proposed future standard library improvements like:

Logging, e.g. (secret: userString)

7

Currently, these interpolations implicitly form a tuple and interpolate it. The proposal changes things so that they will now be argument lists, so interpolations which previously worked (and did something weird) will now fail because there’s no appendInterpolation method with that signature.

2 Likes
8

I like this a lot. I gave it a test run using my Postgres library. Here's what it looked like: PG.swift example of https://gist.github.com/brentdax/fe23d4d9b0fdf8cd1949094f6ef1a936 · GitHub

It was very easy to implement and adds quit a bit of safety to Query (currently because it is ExpressibleByStringLiteral string interpolation works, but doesn't use SQL bindings). This definitely has my vote as is.

2 Likes
9

Emptiness is not something distinguished in the type system, so you won't be able to overload based on that. The literal count given would be zero.

We're not baking in any particular guarantees about the pattern of appendLiteral and appendInterpolation calls generated, but I don't see why we would want or need to emit appendLiteral on empty literals. Do you think there's value in nailing down the exact call sequence generated as part of the compiler-library contract?

The purpose of this is similar to reserveCapacity, giving a hint to allow conformers to skip ahead to a point on the exponential-growth curve closer to what they will end up. A count of graphemes is likely to not be as useful, especially the closer a conformer's backing storage representation resembles something string-like.

1 Like
10

No, I don't, really - it's just a change from the current rather ad-hoc rules and I want that nailed down. :slight_smile:

11

Another like for the design - there are lots of tantalising future directions (e.g. for localised strings).

I'm a bit concerned about appendInterpolation. We have a few cases now of "requirements" which can have flexible signatures. Since we have no way to express that in the language, it gets lost from the protocol's formal members. @dynamicCallable suffers a similar problem (although it also has a bigger one - we have no way to document attribute requirements at all).

By all means, go for it with this one, but we should really think about generalising some of these ad-hoc, private language features at some point.

7 Likes
12

Everything Michael says here is correct, but I’ll add that we can’t actually know the exact number of graphemes, because the compiler may be using an older or newer Unicode version than the one used at runtime.

13

One small update: We may be able to make it so that, if you want default interpolation behavior and your init(stringLiteral:) takes a String, all you have to do is change ExpressibleByStringLiteral to ExpressibleByStringInterpolation. Swift would then use String's StringInterpolation and call your init(stringLiteral:) with the result. This wasn't part of the prototype so I need to make sure it doesn't hurt performance somehow, but I don't think it will.

Since everyone seems mostly happy with this design, I think we can bikeshed a little:

I'm not in love with this label either. We're in sort of an awkward spot, though. What I want to say is basically "pass this to String.reserveCapacity(_:)", but that method is maddeningly vague about what its parameter means:

Reserves enough space in the string's underlying storage to store the specified number of ASCII characters.

Because each character in a string can require more than a single ASCII character's worth of storage, additional allocation may be necessary when adding characters to a string after a call to reserveCapacity(_:).

Moreover, I don't think we actually have a Character-counting function in the compiler, and if we did, it would only be correct if the compiler was using the same version of Unicode as the end user. And, of course, at the end of this we might still need to grow the buffer even if all the interpolations are empty, since reserveCapacity(_:) assumes ASCII-ness.

So that makes the whole size thing kind of awkward. We don't want to say "characters" (because we can't tell you that), we don't want to say "minimum" (because we overestimate a little bit), etc. literalCapacity or something like it might actually be a good way to go—it's vague enough that we can refine it later while clearly suggesting it's suitable to be passed to a reserveCapacity(_:) method.

(We could perhaps sidestep this whole problem by passing a string literal instead of a count. I think that, even though it's backed by a literal, calling removeAll(keepingCapacity: true) on it would give you a fresh buffer with the same capacity and ASCII-ness as the old one—@Michael_Ilseman would know for sure. But doing that without bloating the string literal tables would probably require changes to string literals in both the AST and SIL to make sure literals could share an entry. And I don't think it's something I ever benchmarked.)

1 Like
14

This sounds like a round-about way of faking a move into the initializer, but there's really no gain. Conformers can ignore the parameter if they like, String will pass it on to reserveCapacity.

15

What I'm actually trying to do is smuggle in the builtin string literal protocols' size, isASCII flag, and (implicit) encoding knowledge so we can pre-allocate something with the same properties, but without actually exposing that information to end users. Maybe we can design an underscored initializer for that, though.

Yeah, that's fair.

1 Like
16

The interpolationCount works well for reserving capacity because it corresponds to exactly how many times appendIntepolation() will be called.

I could see it being equally useful to know the literalCount ahead of time, in cases where the literal segments will also be stored in an array.

On the other hand, totalLiteralSize is much more of a hint. It's unlikely that this value will allow anyone to pre-allocate exactly the right resources, but might help them get close enough to run more efficiently.

Given that, maybe it makes sense for there to be three arguments:

public protocol StringInterpolationProtocol {
  init(literalCount: Int, interpolationCount: Int, totalSizeHint: Int)
}

This might be getting a little far out, but that might even create some interesting opportunities for the compiler. Consider this interpolation:

var string = "Welcome, \(hasName ? name : "friend")"

Since the length of "friend" can be determined statically, the totalSizeHint in this case could be 15.

Alternatively, a simple design could be to just pass the length of the entire expression as the hint (say, 39 for the example above).

Ideally, the algorithm for determining the totalSizeHint could evolve over time (without breaking compatibility).

1 Like
17

Currently, the compiler still always alternates literal and interpolated segments, so there are always interpolationCount + 1 literal segments. Some of those segments will be empty and we might optimize them away in the future, but I suspect that interpolationCount + 1 will still be a good enough estimate for the number of literal segments.

We did something like that previously. We stepped back from it because we decided that the conforming type may have a better idea than the compiler.

For example: Suppose your type generates a format string and keeps an array of arguments to format it with. In your example, this type does not want 6 added to its totalSizeHint; it wants 2 (for %s). It always wants 2, every single time. There’s no way for the compiler to know this, and no way for the conforming type to compensate for the bad estimate.

Now, we control the compiler and we also control the DefaultStringInterpolation type used by String, so we can always make little tweaks for those two specifically; for instance, in the example above, we could add 6 to the totalLiteralSize and subtract 1 from the interpolationCount, safe in the knowledge that this will do the right thing for that particular instance, and without any source compatibility issues. But for the general case of custom StringInterpolationProtocol conformers, I don’t think we can afford that. We know that these types have something custom about them; that thing is relatively likely to require a custom hinting computation. It’s best to just give the type the information we can and let it decide what to do.

18

My concern is that logging and localization are rather important concerns, and it may not be worth bike shedding string interpolation now if those are not taken into account.

  • For localization, the order, formatting, and presence/absence of parameters may differ based on localized form. Several localization systems (such as NSLocalizedString) work based on a text lookup, which could be the literal "My name is (name)!", but that doesn't seem like it would work with a "localized interpolation" since the interpolation is still piecemeal.
  • For logging, some external systems have similar requirements to localization above. They treat the logging text separate from parameters, both in an effort to save space and to provide for localization within the logging system itself.
19

Under this design, it should be possible to use string interpolation syntax to build up a format string and arguments. Something like this:

struct InterpolatedFormatString: StringInterpolationProtocol {
  var format: String = ""
  var arguments: [Any] = []

  mutating func appendLiteral(_ literal: String) {
    format += escapePercents(literal)
  }
  mutating func appendInterpolation(_ number: Int) {
    format += "%zd"
    arguments.append(number)
  }
  mutating func appendInterpolation(_ number: Double) {
    format += "%g"
    arguments.append(number)
  }
  /*etc.*/
}

You could then feed the built format string and argument list to a logger, or use it as a key into a localization table.

6 Likes
20

Let me prefix this reply by admitting that I'm not a power user of interpolation, so this is mostly speculation/imagination...

I was thinking that one of the motivations for this design was to decouple the conformance implementations from the actual compiler-generated code.

One example of where the literalCount could be useful might be:

let combined = "\(prefix)\(middle)\(suffix)"

In this case, there's no need to generate any calls to appendLiteral(). In a hypothetical conformance that stores the literal segments in an array, it might be helpful to know that there will not be any.

Another interesting case would be allowing the compiler to split literal segments (that is, call appendLiteral() more than once in a row). For example:

let mixed = "👍 Nice work, \(name)"

Here, it might be more efficient to split the literal into an emoji-containing segment ("👍") and a pure-ASCII segment (" Nice work, ").

A related example would be a literal that just barely crosses an exponential growth boundary, like:

let longString = "Pretend this segment contains 257 characters. \(name)"

Here, maybe it could be more efficient to split the literal into a 256-character string, and a 1-character string. (Although I would guess that statically-declared strings might allocate space more exactly.)

Ah, yeah, this is a good example.

It still suffers from the problem that reserveCapacity(totalLiteralSize + interpolationCount * 2) isn't correct for literals that contain non-ASCII content, though.

Maybe the answer is ... more context! :smile:

protocol StringInterpolationProtocol {
    init(context: StringInterpolationContext)
}

struct StringInterpolationContext {
    var literalCount: Int
    var interpolationCount: Int
    var isASCIILiteral: Bool
    var totalLiteralUTF8Count: Int
    var totalLiteralUTF16Count: Int
    var totalLiteralUnicodeScalarCount: Int
}

Of course, there's no way to know whether the interpolated values will contain non-ASCII characters, so maybe that's not so useful.