Demystifying SyntaxParsingContext [Compound variable names]

Development Compiler
1

I've been investigating adding support for argument labels in variable names over the past couple days. The high-level approach to the parsing work, as I see it, is to identify all* the locations that currently expect an identifier for a name, and modify them to expect an unqualified-decl-name instead.

* Certain places are excluded, e.g., argument labels and enum case names.

Part of this effort includes allowing patterns to bind compound names, so that you could write the following:

enum E {
    case c(foo: (Int) -> Void)
}

let e: E = .c(foo: { _ in })

switch e {
case .c(let bar(x:)):
    bar(x: 0)
}

At the point where the name actually gets parsed, we have the following code:

DeferringContextRAII Deferring(*SyntaxContext);

// ... consume an identifier, create a NamedPattern, etc.

if (SyntaxContext->isEnabled()) {
  ParsedPatternSyntax PatternNode =
      ParsedSyntaxRecorder::makeIdentifierPattern(SyntaxContext->popToken(), 
                                                  *SyntaxContext);
  ParsedExprSyntax ExprNode =
      ParsedSyntaxRecorder::makeUnresolvedPatternExpr(std::move(PatternNode),
                                                      *SyntaxContext);
  SyntaxContext->addSyntax(std::move(ExprNode));
}

The normal parsing code is straightforward to me, but I'm having a bit of trouble understanding where SyntaxContext fits into all of this. Big picture question is, what role does it play in the parsing process as opposed to the "normal" AST?

But beyond the big picture question, I'm also unsure what the correct way is to extend this portion of the code to handle compound names. Currently, we create the ParsedPatternSyntax by simply calling popToken() on SyntaxContext, which no longer works in a potentially-compound-name world (since the name may consist of arbitrarily many tokens). It's not clear to me from investigating other use sites how I should go about recovering the proper Parsed*Syntax bits needed to create the ParsedPatternSyntax. At the very least, I think I'll need to create a ParsedDeclNamePatternSyntax that is built up from a ParsedDeclNameSyntax, but I'm still not sure how to get my hands on that ParsedDeclNameSyntax once we've gone through the normal parsing route for a DeclName.

If anyone familiar with this aspect of the parser could help me understand SyntaxParsingContext (and also how it interacts with DeferringContextRAII), I would be very appreciative!

1 Like
2

SyntaxParsingContext is used to help build the libSyntax/SwiftSyntax tree at the same time as the parser is constructing the semantic AST. Integrating libSyntax into the compiler pipeline has a helpful overview of the current model (and how it might change in the future, the proposed changes from that post aren't on master yet).

I'm not sure what's the best way to handle this, but @rintaro might have some ideas.

1 Like
3

Thanks @owenv, that thread was just the sort of overview I was looking for. I'll see if digging into libSyntax a little more helps unlock anything, and wait to see whether @rintaro has any thoughts!

4

SyntaxParsingContext is currently a framework of building a libSyntax tree as a by-product of AST. In normal compilations, SyntaxParsingContext is disabled, but some SourceKit requests uses libSyntax tree. Also, SwiftSyntax parses Swift code using this.

But beyond the big picture question, I'm also unsure what the correct way is to extend this portion of the code to handle compound names.

First of all, you have to decide how bar(x:) pattern should be represented in the Syntax tree. They are defined in https://github.com/apple/swift/tree/master/utils/gyb_syntax_support/ . Since, we already have a similar syntax in expressions (e.g. name(x:)), we can adopt the same structure to the pattern as well. Specifically, just like IdentifierExpressionSyntax, you can add DeclNameArguments child to IdentifierPatternSyntax.

As for parsing the argument part of the pattern, you should still be able to use parseDeclNameRef(). Probably, the code would be something like:

  // These RAII context automatically wrap the parsed name and the optional arguments.
  // <UnresolvedPatternExpr><IdentifierPattern>{name} {argument (opt.)}</IdentifierPattern></UnresolvedPatternExpr>
  // and propagate it to the parent context.

  SyntaxParsingContext PatternExprCtx(SyntaxContext,
                                      SyntaxKind::UnresolvedPatternExpr);
  SyntaxParsingContext IdentPatternCtx(SyntaxContext,
                                       SyntaxKind::IdentifierPattern);

  // Parse the unqualified-decl-name.
  DeclNameLoc loc;
  DeclNameRef name = parseDeclNameRef(loc, diag::expected_pattern,
                                      DeclNameFlag::AllowCompoundNames);

  Pattern *pattern = /* construct AST from `loc`/`name` */;
  return makeParserResult(new (Context) UnresolvedPatternExpr(pattern));
5

Thanks @rintaro, that's all very helpful.

Yeah, that seems like the right way forward. Thanks!

Yep, I'm using parseDeclNameRef to parse the compound names in various places. With regards to the SyntaxParsingContext aspect of your snippet: does that approach still apply within the scope of the DeferringContextRAII object? I balked at the snippet in my post since it appears we're creating the ParsedPatternSyntax and ParsedExprSyntax objects directly rather than relying on the usual RAII approach, and it's not obvious to me how to translate that to the compound name case. Would you be able to shed a little light on DeferringContextRAII?

6

SyntaxParsingContext has callback mechanism to create actual nodes from the parsed information. Using that, libSyntax creates C++ nodes whereas SwiftSyntax creates Swift nodes. After creating the node, they are opaque object from the parser so the parser cannot get any information from them. DeferringContextRAII just defers the callbacks so the parser keep the parsed information around and extract them later. In this case, the pattern is extracted from UnresolvedPatternExpr here https://github.com/apple/swift/blob/fadfb6c945bc24acba746e9a949e7c82eade9073/lib/Parse/ParsePattern.cpp#L1202-L1206

does that approach still apply within the scope of the DeferringContextRAII object?

Yes, it should work inside DeferringContextRAII.

I balked at the snippet in my post since it appears we're creating the ParsedPatternSyntax and ParsedExprSyntax objects directly rather than relying on the usual RAII approach,

There're several way to create a node. RAII, builder, and factory. I don't believe there was specific reason to use the current approach. RAII should be totally OK.

1 Like
7

Thank you for the detailed responses @rintaro!