SE-0354 (Second Review): Regex Literals

Regarding the new syntax: how does Swift diagnose incorrect regular expression syntax? What do you get here, for instance?

let foo = (/hello|(world))/;
2 Likes

@Ben_Cohen, do we have a toolchain with the currently proposed behavior to check such things ourselves?

2 Likes

The approach taken with the regex proposals (as with Swift Concurrency) is that the work is getting integrated under a compiler flag (-enable-bare-slash-regex in this case) while under review. This means you can use the nightly toolchains from swift.org (either main or release/5.7) to try out the feature. But it looks like recent nightly toolchain builds haven't been posted yet – I'm checking on this and the latest 5.7 branch should be available shortly.

That said, looking specifically at the diagnostics currently output by the compiler when code is invalid should not be considered something that is covered by this review.

The primary reason for this is that the bar for evolution proposals is a prototype implementation that demonstrates how the feature is used. The expectation is not that this prototype is yet "shippable" or even mergeable into the main branch without additional work. Part of the work to get it to that point, which happens after proposal acceptance, is often quality-of-implementation work such as good quality diagnostics when the compiler hits invalid code.

Of course, sometimes having this kind of QoI is highly desirable at the proposal stage. Without it, reviewers need to reason about the results of using a fully productized implementation, not just the prototype provided for review. A similar example is runtime performance optimization – with some proposals, performance is a key driver and so not having the final fully optimized implementation may present challenges to reviewers who might be considering whether, say, such a proposal is a worthwhile tradeoff versus the complexity it might add to the language.

Nevertheless, having a full production-worth implementation is felt to be too high a bar for proposal to make it to the review stage. So we ask reviewers to bear with the proposal and try and work through these things on paper instead.

Feedback on whether that bar should be raised is welcome, but would be more appropriate on a dedicated thread, probably one in the Evolution/Discussion category. Feedback on diagnostic implementation is also welcome, but probably belongs in the Development/Compiler category.


So to bring it back to the immediate question, I guess it really needs to come back as another question: as a human looking at that code, on paper what would the ideal diagnostic be for this code?

let foo = (/hello|(world))/;

Once there's consensus amongst us humans for what the "right" diagnostic is to give for this code (bearing in mind you can have the compiler more than one diagnostic for two different interpretations) then we can discuss whether it's possible given the parsing rules to have the compiler emit them. If the answer might be "no", then that's very relevant to the proposal review. Such feedback might lead to re-considering deprecating the prefix / operator, for example.

It's worth noting that diagnostics on invalid code are able to use more information than is available when parsing valid code. For example, in the f(/,/) case, the diagnostic can make use of knowledge from the type checker that there isn't a unary function that would accept a Regex but there is a binary function that takes two binary functions.

9 Likes

Thanks for the detailed response. I completely understand that we can't expect much from the diagnostics at this stage.

On the other hand, I think playing with a rough implementation of the rules and trying to see how compiler reacts to various situations can give more insight into whether the current rules are going to be enough for a good developer experience or not.

For example, what is going to happen in a place like playground when compiler is continuously trying to parse and diagnose as you type, and being in the middle of a regex literal is a totally new and weird place to be for the compiler.

For other literal types, there are good distinct indicators at least for their beginning, but / can be harder to detect at the start of a regex literal. For example, editor can confidently insert a closing delimiter as we type the opening delimiter, (which helps compiler with partially typed code) but this is only possible with / if compiler already expects a regex literal in that position. I want to get a better feeling of how many times that context is available to the compiler to see how the experience of typing a regex literal is going to be compared to, say, a string.

3 Likes

What's the rationale for extended literal (#/.../#) to enable free-spacing mode (?x) by default, compared to others, e.g., case-insensitive mode? I read the doc a few times but don't see it. Furthermore, is there a way to disable it?

Update: the Swift 5.7 toolchain snapshot as of last night is now available on swift.org.

4 Likes

To avoid any misunderstanding: #/ followed by a newline (and with a matching newline preceding the /#) enables extended-syntax (non-semantic whitespace + # comments) mode. #/.../# alone does not do it.

You might still ask why the multi-line literal is not also case-insensitive as well as whitespace-insensitive, of course.

It looks like no:

➜  ~ cat multiline.swift
if #available(macOS 9999, *) {
    let r = #/
        (?-x hello world)
    /#
}
➜  ~ xcrun --toolchain "Swift 5.7 Development Snapshot 2022-05-15 (a)" swiftc -enable-bare-slash-regex multiline.swift
multiline.swift:3:9: error: cannot parse regular expression: extended syntax may not be disabled in multi-line mode
        (?-x hello world)
        ^
➜  ~

This probably needs clarification/justification in the proposal.

3 Likes

Huh, the proposal doesn't mention case insensitivity. Is that a part of the proposed regex ecosystem at all? Seems like it belongs in here somewhere. (Apologies if I missed it.)

It's part of the regex syntax proposal:

let r = /(?i:h)ello (?i:w)orld/
let m = try! r.firstMatch(in: "Hello World")
print(m!.output) // prints Hello World
3 Likes

It occurs to me that another line of argument is that Swift simply should not support extended mode at all. Once again, I am musing, not necessarily advocating. The argument is that the concise literal syntax is best for short regexes, any regex that does not fit on a single line should use the builder DSL to break it into multiple lines.

Wondering how this plays out, I tried translating @hamishknight’s example from above:

…into a builder DSL expression with a similar spirit of formatting:

let kind = Reference(Substring.self)
let date = Reference(Substring.self)
let account = Reference(Substring.self)
let amount = Reference(Substring.self)

let regex = Regex {
  // Match a line of the format e.g "DEBIT  03/03/2022  Totally Legit Shell Corp  $2,000,000.00"
  let fieldBreak = /\s\s+/
  Capture(/\w+/,               as: kind);    fieldBreak
  Capture(/\S+/,               as: date);    fieldBreak
  Capture(/(?: (?!\s\s) . )+/, as: account); fieldBreak  // Note that account names may contain spaces.
  Capture(/.*/,                as: amount)
}

Is that compelling enough to dispense with extended mode altogether? I’m not sure.

The repetition of Reference(Substring.self) is certainly unsatisfying, and makes me wish again for the DSL to support named capture groups as tuple labels to parallel the behavior of literals. (One day, hopefully!)

If we’re willing to dispense with the clarity and safety of named capture groups, the DSL builder version isn't such a bad alternative to extended mode:

let regex = Regex {
  // Match a line of the format e.g "DEBIT  03/03/2022  Totally Legit Shell Corp  $2,000,000.00"
  let fieldBreak = /\s\s+/
  Capture(/\w+/); fieldBreak             // kind
  Capture(/\S+/); fieldBreak             // date
  Capture(/(?:(?!\s\s).)+/); fieldBreak  // account (Note that account names may contain spaces.)
  Capture(/.*/)                          // amount
}

I’d say that the builder is an improvement for my own multiline example from above, although it's probably less representative of common usage than Hamish’s example:

 #/
     (
         hello        # morning
         |
         good night   # evening  (this and only this space character is preserved)
     )
     (
         ,\s+
         every
         (body|one)
     )?
/#
Regex {
	ChoiceOf {
		"hello"       # morning
		"good night"  # evening  (no special handling of space character necessary)
	}
	Optionally {
		/,\s+/
		"every"
		/body|one/
	}
}

Perhaps multiline / extended mode won’t pull its weight as a feature in Swift? I’m not sure I’ve convinced myself here, but it’s worth considering the question.

12 Likes

Is there a reason we can't specify matching options as flags following the closing / (or /#) like in other languages?

let firstPart  = /abc | d /xi
let secondPart = /ef  | gh/xi

I don't see it mentioned in the proposal. I suppose this omission could be for disambiguating with the / operator. It seems to me this will be impacting how easy regexes can be copy-pasted from other places, so it should be worth a note.

It can be rewritten like this of course:

let firstPart  = /(?xi)abc | d /
let secondPart = /(?xi)ef  | gh/

so functionality isn't left out, only familiarity.

1 Like

Could the parser go even further? If there is a valid interpretation without regex literals, use it. I think this would remove all ambiguities and all source breakage.

func foo(_ x: (_: Int, _: Int) -> Int) -> [Int] { [] }
func foo(_ x: (_: Int, _: Int) -> Int, _ y: (_: Int, _: Int) -> Int) -> [Int] { [] }
func foo(_ x: Regex) {}

// Not regex:          vs.  Regex:
foo(/).reduce(4, /)         foo(#/).reduce(4, /#)
foo(/, /)                   foo(#/, /#)

// Must be regex -  '/' is not a postfix unary operator
foo(/, 4/)

Treating /…/ as syntactic sugar over #/…/# that can only be used when unambiguous.

Or would that result in new/bigger problems?

1 Like

I’d say: Just leading/trailing whitespace.
The hello world example is convincing for me.
(I assume (?xx) would make all whitespace non-semantic)

Maybe:

  • /…/, #/…/# -> all whitespace is semantic
  • Multi line #/…/#, Single line ##/…/## -> leading/trailing (after comment removal) whitespace non-semantic
  • Multi line ##/…/##, Single line ###/…/### -> all whitespace non-semantic

Parsing happens before other parts of the compilation pipeline, so does not have access to semantic information like the types of function arguments (and other similar things – a notable example being whether or not the expression is inside a result builder).

Factoring in that kind of thing would have far-reaching implications for things like compilation time, and the ability for non-Swift compilers to parse Swift (including potentially factoring out a componentized Swift parser from the Swift compiler).

Diagnostics for failed parses can be produced with the help of that information, though, so fixits can benefit from it.

6 Likes

This is what is proposed and it's unlikely that you'd commonly want more than one #. It's similar to "raw" strings, though perhaps even more rare.

Note that case insensitivity is a semantic option and not a syntactic one, which is why it's primary expression is via API. E.g. /abc/.ignoresCase(). We do support the regex syntax for enabling and disabling it. It's possible to argue that any semantic option could/should be set or unset by different literal syntax, but it is a little odd and I'm not aware of much precedent.

Regarding multi-line regexes, traditionally, a newline sequence encoded into a regex would be treated verbatim and match that exact sequence. This is rarely what is actually desired; and if you're splitting a regex across multiple lines for organization or clarity purposes, you nearly always want non-semantic whitespace as well.

The area in the Venn diagram where you want to split a regex across lines, ignore the newlines and surrounding spaces, but keep semantic whitespace within a line for long runs of verbatim content is very small. I'm not aware of any precedent (which doesn't argue we shouldn't do it, but does question how high that demand is).

Is the .ignoresCase() an API that's actually being proposed somewhere? I couldn't find it from a quick search but also might have missed one of the proposals...

1 Like

You'd need to remove the whitespace inside those regexes, so you'd have:

let kind = Reference(Substring.self)
let date = Reference(Substring.self)
let account = Reference(Substring.self)
let amount = Reference(Substring.self)

let regex = Regex {
  // Match a line of the format e.g "DEBIT  03/03/2022  Totally Legit Shell Corp  $2,000,000.00"
  let fieldBreak = /\s\s+/
  Capture(/\w+/,               as: kind);    fieldBreak
  Capture(/\S+/,               as: date);    fieldBreak
  Capture(/(?:(?!\s\s).)+/,    as: account); fieldBreak  // Note that account names may contain spaces.
  Capture(/.*/,                as: amount)
}

Are you envisioning the scenario where a multi-line regex treats contained newlines as verbatim content, or would they be outright forbidden?

Similarly, what does a newline in a literal with semantic whitespace entail? Verbatim treatment or error? What about spaces around the next line?

Syntactic options are a little different in practice than semantic options, even though they use the same mechanism in traditional regex syntax. (Regex syntax conflates things that the builders treat orthogonally or via API).

The i would preferably be spelled as regex.ignoresCase(), which extends well to structured builders. E.g., string literals are verbatim by default, but you could add that to ignore case for just that component. (Assuming we want the API directly on String, otherwise it might be spelled "literal content".regex.ignoresCase())

Ignoring whitespace could be a modifier, but that implies a semantic change. E.g., it seems like /abc/.ignoringWhitespace() intends to match the input "a b\r\nc".

This is in [Pitch] Unicode for String Processing

@nnnnnnnn any update or thoughts here?

1 Like

Oh, thanks, missed that. Fixed the OP.

Is there a verbatim context in this proposal? I thought that #/…/# as proposed either (1) ignores whitespace or (2) has to be on a single line. Apologies if I missed that….

I mean what should the compiler behavior be for a #/.../# literal that has a newline inside, in the context of your scenario where there is no multi-line/extended mode? Would the compiler reject it or would the newline be treated as verbatim content of the regex?

In my scenario, the compiler would reject it. (Again, musing, not necessarily advocating. I'm on the fence myself.)