It seems that final as in
final class SomeClass { ... }
is parsed intro a TokenKind.identifier(value), where value is final. Is this a bug or by design? I'd expect a specific TokenKind instead. Are there any other tokens parsed as TokenKind.identifiers?
Certain keywords in Swift are "contextual keywords", meaning that they're only treated as keywords in the context where they're used as keywords, otherwise they're treated allowed to be used as identifiers. The contexts where they can be used as keywords vs. used as identifiers can usually be unambiguously parsed so there's no need to distinguish them.
That's why you can write this:
let final = 5 // final: Int = 5
without requiring backticks around final.
This most often happens with keywords added to the language later (since people with existing code might be already using that term as an identifier, and unconditionally changing it to a keyword would break parsing their code), or for keywords that are only used in very limited contexts.
Some examples that immediately jump to mind are final, open, get, set, async, await... but there are many others as well AFAIK.
Wouldn't that mean SwiftSyntax/libSyntax should be able to distinguish them based on context? I mean, right now the consumer has to check if .identifier(value) is a regular identifier or a contextual keyword, which is clearly not ideal. I'm just trying to see if this is something that might change in the future or I should always deal with this myself in any code depending on SwiftSyntax/libSyntax.
I suppose it could, but the tokens/nodes are auto-generated from the same definitions that the compiler uses internally, and it also keeps them as identifiers there. I imagine it's easier to just maintain one representation than it is to conditionally change the token that gets generated in the SwiftSyntax representation based on the rest of the parse.
You mean the ones used by libSyntax? Alright, makes sense. It's a bit less convenient, but not a SwiftSyntax problem as far as I see. Thank you for your detailed answer!
We should classify final as a ContextualKeyword here, not as an Identifier. Would you mind opening a bug report on Issues · apple/swift-syntax · GitHub so we can track fixing this?