Allow more characters (like whitespaces and punctuations) for escaped identifiers

[allevato (Tony Allevato)]

@gwendal.roue

With this proposal, all identifiers have two syntactic forms, only one of them being always parseable.

But isn't that already the case today? You can escape identifiers today that don't actually need to be escaped:

let `x` = 5
print(x)    // 5
print(`x`)  // also 5

@gwendal.roue

This function is currently simple (it only has to check for a known list of keywords).

This function will become complex.

"More" complex, but it's difficult to imagine it being significantly more complex, based on the implementation already provided by the PR author. Now instead of checking a list of keywords, it also checks the token to determine if it contains any non-identifier-safe code points (with a special case for property wrapper dollar signs).

Special cases are unfortunate, of course, but that brings us to the next point:

It is likely that this function will not be easily available to third-party tooling. Those may just give up and escape everything. Or ship with a poor-man buggy implementation.

Why do you think it's unlikely for that function to be available to third-party tooling? The Swift syntax parser is already factored out into a dylib that ships with the toolchain for use by SwiftSyntax. I think it would be entirely reasonable and within the realm of possibility to add the relevant C binding to that library and Swift API to SwiftSyntax to answer the question "does this identifier need to be escaped?"

Moreover, since this list of keywords that the compiler checks is already not available to third-party tooling, those tools already have to duplicate some logic and may get it wrong. That list of keywords may also differ subtly between Swift versions (although hopefully must not anymore, for source compatibility reasons). The list of keywords where this is necessary is also somewhat non-obvious, because certain reserved words are context-sensitive. This recent thread is a great example, where the word set can be used as an identifier outside of a computed property, but in one specific location in an accessor block, it must be escaped or it's interpreted as a keyword:

struct S {
  var set = [0]  // OK, "set" is unambiguous here

  var first: Int? {
    set.first  // error: "set" treated as start of accessor
  }
}

So the conditions under which code generators have to mangle or escape identifiers are already somewhat fraught with edge cases, and I don't think expanding the space of valid escaped identifiers exacerbates that significantly—especially if we take the opportunity to provide clients with an API that matches the one used by the compiler, which would be an improvement over the state of things today.