Pitch: New Leaf Body Syntax

pitch

(Tanner) #1

Pitch: New Leaf Body Syntax

Leaf is a templating language built for Swift. This pitch proposes introducing a new syntax for declaring tag bodies that would simplify the parser and make using Leaf easier.

Current Syntax

Leaf has supported the following syntax for declaring tag bodies since version 1.0.

<h1>Users</html
<ul>
#for(user in users) {
    <li>#(user.name)</li>
}
</ul>

This syntax, which we'll refer to as curly-brace bodies, was inspired by Swift's closure syntax. Much of Leaf's original design was copied directly from Swift in an effort to make the new templating language easy to learn.

While this syntax is easy to learn and great for example code, it suffers from some issues in real world use cases--especially when Leaf is used alongside JavaScript.

Problems with Curly-Brace Bodies

The main problem with curly-brace bodies is the prevelance of } in normal template code. Because } denotes the end of a body in this syntax, any appearances of } in the template's text must be escaped.

let xs = [-5, 3, 42, -3.14]
#if(count(xs) != 0) {
	|x| = { x, if x >= 0; -x, otherwise; }

	Solve equations:
	#for(x in xs) {
		|#(x)| = ?
	}
}

In the above example, Leaf would (perhaps unexpectedly) assume the } in the definition of |x| signals the end of the #if tag's body.

The solution to this problem is to escape any } that do not actually intend to close a tag body using \.

#if(count(xs) != 0) {
	|x| = { x, if x >= 0; -x, otherwise; \}

	Solve equations:
	#for(x in xs) {
		|#(x)| = ?
	}
}

While escaping is fairly trivial for this example, one can imagine that it might get out of hand when dealing with large amounts of embedded JavaScript.

Proposed Solution: Colon Syntax

This pitch proposes introducing the following syntax to make parsing tag bodies more predictable and resilient:

  • Tags intending to include a body are immediately trailed by a :
  • #:<tagname> token is used to close a body.

Take a look at the following example:

#if(count(xs) != 0):
	|x| = { x, if x >= 0; -x, otherwise; }

	Solve equations:
	#for(x in xs):
		|#(x)| = ?
	#:for
#:if

This new syntax has a number of key benefits over the existing syntax.

  • Tags that do not declare a body are easier to parse (no need to check for unescaped {)
  • Body closing syntax is less likely to appear in template text (#:for is less common than })
  • Body closing tag matches opening tag (#:for ends #for)
    • This makes it easier for humans to read templates
    • It also allows the Leaf compiler to better catch copy/paste errors

Edge Cases

Escaping Trailing Colon

It's likely that a colon could appear directly after a tag in cases when the developer is not declaring a body.

#for(user in users):
    - #(user.name): #(user.email)
#:for

In these cases, the trailing colon can be escaped or added via string concatenation.

#(user.name)\: #(user.email)
#(user.name + ":") #(user.email)
#(user.name)#(":") #(user.email)

Escaping Closing Tag

Although unlikely, it's possible that a closing body tag such as #:if could appear in the template text. In this case, the closing tag could be escaped like any normal tag using \ before #.

#if(showLink):
    The link is: http://foo.co/index.html\#:ifasdf
#else:
    No link.
#:if

Single Line Bodies

One of the nice advantages of the curly-brace body syntax is that declaring single line bodies is concise. Take the following example of setting a page title.

#set("title") { Home }

This is considerably less concise with the proposed colon syntax.

#set("title"): Home #:set

One solution that is already available for some Leaf tags is passing a string as the second parameter. This could be expanded to all tags that accept bodies as a general pattern in Leaf.

#set("title", "Home")

This has the added advantage of not introducing extraneous white space around the content.

Deprecated Support for Curly-Brace Syntax

It should be possible to continue supporting the existing curly-brace syntax for declaring tag bodies alongside the new colon syntax. This usage would be considered deprecated and would emit a warning in debug build modes. Support for the syntax would likely be removed in the next version of Leaf.

It's not possible to 100% guarantee support for both syntaxes will be reasonable to implement in the parser, but if it is, it would likely be greatly appreciated by developers with large, existing codebases using Leaf.

Alternatives Considered

End-Syntax

If Leaf had a way to statically determine whether or not a given tag usage required a body, it could be possible to omit requirement of the trailing :. This could allow for the following syntax:

#for(user in users)
   Hello #(user.name)!
#endfor

This syntax does not suffer from the Escaping Trailing Colon issue. However, statically determining whether a tag requires a body could prove difficult to implement and likely restrict custom Leaf tags.

One difficult case is the #set tag.

#set("title", "Home")
#set("content")
    <sidebar>...</sidebar>
#endset

This use-case needs support for dynamically requiring a tag body based on the number of arguments supplied.

Here's an extend example using #if

#if(a)
    <p>#(a)</p>
#elseif(b)
    <h1>#(b)</h1>
#else
    <strong>Nothing</strong>
#endif

Curly-Brace Counting

vapor/leaf#119 proposed counting opening curly braces to more intelligently detect closing braces. While this works well for detecting curly-braces in typical JavaScript usage, it fails to account for some seemingly feasible cases, such as using this smiley face.

#for(user in users) {
	Hello #(user.name)! :}
}

#2

Would a stack-based algorithm work to solve this problem, or would that suffer the same problems as the counting solution?


(Caleb Kleveter) #3

What do you mean by a 'stack-based algorithm'?


(Tanner) #4

I think that would have the same issue as the counting solution. i.e., templates like the following would continue to cause problems:

#if(foo) {
   Hello :}
}
#if(foo) {
   This is an example of broken javascript:
   var x = { }}
}

#5

@calebkleveter it's a way to parse things like HTML

http://hz2.org/blog/parsing_html.html

Every time you see an opening tag, you push that tag onto a stack and start tracking the contents of a body. Every time you see a closing tag, you pop the uppermost tag off the stack and can accept that the last body you were tracking is now complete assuming the closing tag matches the uppermost (opening) tag on the stack.

There are some quirks in HTML that this doesn't work perfectly for, but it does work for simple snippets.

@tanner0101 I see what you're going for. How using common characters as tags causes problems. Thanks!


(Steven Van Impe) #6

Have you given any thought to just deprecating Leaf and going with Stencil instead?

I've never found or heard any arguments as to why Vapor needed a new template language in the first place:

  • Are there fundamental differences between Leaf and Stencil?
  • Are they trying to solve a different problem?
  • Are they trying to solve a problem in a fundamentally different way?

As far as I can tell, the answer to these all seems to be "no".

Our community is way too small to be wasting development efforts. It's great that we have multiple frameworks exploring different avenues, but templating languages aren't exactly a new problem that needs exploration. Swift already has a templating language (Stencil) that works fine, has a familiar syntax, isn't tied to a web framework but is used by other projects, such as Sourcery.

As a user, I'd rather see development time spent on better tooling support, improved code quality and documentation, ... than multiple languages that differ only in syntax.


(Tim) #7

Leafs handling of futures is pretty unique (and a very nice feature!). So I for one would be against loosing that.

Unless that’s all handled by TemplateKit of course and Stencil can sit on top of that.


(Tanner) #8

If this pitch were proposing a new templating language, I would definitely agree with your reasoning. But Leaf is over 2 years old now and a lot of people that use it and like it. I think it would be unfair to those people to deprecate Leaf.

I also disagree that Leaf does not have any differences to Stencil. Leaf is more similar to Stencil than something like Mustache, sure, but they are not identical. Leaf's syntax is arguably more concise. Leaf's extensible tag mechanism allows for more than one argument, default arguments, etc. Leaf is also built directly on and for NIO and Codable, which makes it a performant choice given Vapor 3's architecture.

That said, I think Vapor should have good implementations of existing standards. One of my goals for Vapor 4 is to introduce a pure NIO / Codable based implementation of Mustache. Given Mustache's dead-simple syntax this should prove to be an even more performant alternative to Leaf and could possibly end up being Vapor's recommended package for API development (simple views, email views, etc).

In regards to Stencil, I think given its popularity in the Swift community, it won't be long until there is a NIO friendly implementation with good performance characteristics. When that becomes available (hopefully through the SSWG process) I would be more than happy to support an official Vapor integration.


(Steven Van Impe) #9

Perhaps proposing a standard through the SSWG (whether that's an existing language or a new one) is the right way forward for the long term? Multiple languages don't just lead to duplicated development efforts, there's also the issue of keeping documentation and education materials up to date. For us in education, choice is actually a bad thing. It makes it that much harder for students to find the right information.

Anyway, apologies for going off-topic. This is just one of those issues that I probably care more about than others.

I'll keep an eye on the SSWG. If a standard were to emerge, I'll adopt it right away. Sure, migration always hurts, but it will hurt even more when our community (and codebases) grows :slight_smile:


(Robert Muckle-Jones) #10

Have you taken a thorough look through SE-0200: “Raw” mode string literals? I think that might give you some ideas for solving this problem - ways that would allow you to keep the existing brace block delimiters.


(Gábor Sebestyén) #11

What about simply closing tag heads and the terminal part with the same # character?

#if(condition)#
...
#/if#

(Henry Allsuch) #12

I don't think this has been mentioned already, but its my opinion a template language should be very light and thin. Leaf shouldn't try to solve too many problems and become bloated. Application logic shouldn't be in the view layer, so discouraging the use of } in templates is a good thing. If it's a different language i.e. javascript then it shouldn't typically be inline and be linted, minified and tested.

I am a fan of Twig for PHP in that respect, https://twig.symfony.com/.


(Tanner) #13

I tend to agree actually. Especially considering how powerful Swift is, I think you should try to do as much logic there as possible.

However, I believe this discussion is tangential to the issue at hand. Leaf is already widely used and loved and it's not included with Vapor by default, meaning it's up to the end user which templating language they use.


#14

@tanner0101 Here are some thoughts for consideration…

Bracket Use Case: Math Presentation

In some of my use cases, the single brackets { and } are the weakest part of Leaf. This is not a case of trying to "put application logic in the view layer", rather this is the case of content which displays math. Not math logic execution or computation. Just math presentation.

Here is an HTML math display expression where the {…} bracket pair indicate where a square root symbol should be presented for display:

<p>Two equations: \(A^T_S = B\) and \(\sqrt{ab}\).</p>
    <p>\[A^T_S = B\]  </p>
<p>\[\sqrt{ab}\]  </p>

The correct resulting display looks like:

CorrectOutput

Such LaTeX math sections exist in html and can be rendered by libraries such as MathJax (more comprehensive) and KaTeX (faster).

As a some-what related FYI, Apple has added LaTeX mathematical equation support to Pages, Numbers, Keynote, and iBooks Author. Apple: HT207569, HT202501

It would be nice if the server side Swift community kept some math presentation examples in their test cases, where applicable.

Double Bracket Alternative

Another alternative to consider instead just changing the single bracket {...} to a double bracket {{...}}. A double bracket approach

An opening double bracket {{ following a Leaf tag has the property of being "no-splittable" i.e. remains a double bracket.

Example 1: Visually think of {{ as a bold bracket. Notice how the double bracket enhances readability of Leaf tag bodies.

#if(count(xs) != 0) {{
	|x| = { x, if x >= 0; -x, otherwise; }

	Solve equations:
	#for(x in xs) {{
		|#(x)| = ?
	}}
}}

Example 2: Double brackets would avoid many ambiguities.

#for(user in users) {{
	Hello #(user.name)! :}
}}

Example 3. Works with HTML LaTeX math expressions.

#set("body") {{
<p>Two equations: \(A^T_S = B\) and \(\sqrt{ab}\).</p>
    <p>\[A^T_S = B\]  </p>
<p>\[\sqrt{ab}\]  </p>
}}

Example 4. This one warrants some consideration.

First, maybe Leaf should not need to handle other broken code? Yet, consider this as content which is discussing bad code (not actual broken javascript).

#if(foo) {{
   This is an example of broken javascript:
   var x = { }}
}}
approach (in the one case) result
forward double bracket counting fails
reverse double bracket counting OK
outer-to-inner double bracket pairing OK
push-pop single/double context stack OK (†)

† The push/pop stack is rule dependent. Here is a basic "stack wins" rules sequence which works in this case, but would likely need rework for the larger set of cases. Note that the first double bracket, which is anchored to a Leaf tag, is treated to be a non-splittable double bracket.

  1. opening double bracket: push double-non-splittable
  2. opening single bracket: push single
  3. closing }} is mismatch to stack. Stack wins. Do match to single opening bracket on stack: pop single.
  4. one opening } still exists. Is mismatch with stack. Non-splittable bracket on stack. Stack wins: Ignore and maybe print message.
  5. closing }} which does match non-splittable double bracket on the stack: pop double-non-splittable. done.

StackAlgorithm

Summary: Double brackets (splittable and non-splittable) provide more context information than single brackets. Double brackets would reduce the overall range of possible conflicts. Some logic may be possible to handle the desired cases, but would need a more definitive set of cases to know for sure. And, the continuation of a bracket-type syntax might feel familiar to existing users.

Colon Alternative

Comment: The alternative of adding a colon ':' to the syntax mix would seem to...

  • increase complexity (adds another character to the syntax mix),
  • be less readable, and
  • give some Leaf syntax a sad ): ending ... in emoji speak.

End-Syntax Alternative

Comment: The End-Syntax mentioned in the original post would also work OK for the use cases that I deal with. This approach appears to work from a syntax viewpoint and would avoid the bracket issues altogether. This approach also appears to be a good candidate.


(Tanner) #15

Thanks for the detailed response.

I agree with this. I've created an issue for us to add these to the next version.

Current development work on Leaf 4 is happening at: https://github.com/vapor/leaf-kit/. We seem to have landed on the following syntax:

<h1>#(title)</h1>

#if(user):
    <p>Hello #(user.name)</p>
#endif

We call the colon a "tag body indicator" and it tells the parser to look for a corresponding #end... tag. We've found this approach dramatically simplifies lexing and parsing over Leaf 3 while also solving the issue described in this post. Simplified lexing not only helps make Leaf more reliable and performant, but will make writing syntax highlighters easier, too.


#16

Great! Thanks for the link to the current development.

Yep, the #end... tag is a solid choice.


As a detail, I would lean towards a character pair like )# instead of ): for the "tag body indicator".

<h1>#(title)</h1>

#if(user)#
    <p>Hello #(user.name)</p>
#endif

Why?

First, consider whether #if(user): or #if(user)# pops more visually for someone writing and reviewing code.

Second, consider whether a # "tag body indicator" could also provide a thematic branding that helps the Leaf tag syntax be quickly recognized in a consistent way.

Third, well … this may be just me, but consider whether you can kind-of see a sad face ):.

That said, I look forward to an #end... tag Leaf version with either : or # for the "tag body indicator". :-)


#17

@tanner0101 I ended up with some some additional thoughts on the #end... syntax that seemed worth sharing for consideration.

I realized that while #if(...): ... #endif would work for the html LaTeX math use case, it brakes another use case in which content discusses programming in Swift/C/C++.

Use Case: Markdown to HTML Workflow

This use case is where markdown is translated to html using some generally available command line utility (or linkable library) such as pandoc, multimarkdown, cmark, hoedown, discount, SwiftMark, etc.

The goal is to not require any additional content processing between the generated html snippet and what gets delivered to the web browser. (e.g. content escaping due to templating syntax.)

More precisely, the templating language challenge test criteria for this use case is "If a given html snippet generated from markdown renders correctly when placed AS-IS the inside the <body>...</body> tags of an html page, does that same snippet render correctly when embedded AS-IS inside the templating language syntax?"

Where #end... Could Have Issues

Below are some blog post examples which can be successfully embedded and rendered in Leaf 3 without additional processing. However, the #end... syntax could have issues with the these same (previously working) snippets.

Example 1: Swift Blog Content

<p>Here's a Swift snippet which illustrates 
selective platform compilation.</p>

<pre><code class="swift">
#if os(Linux)
  //
#elseif os(macOS)
  //  
#else
  // 
#endif
</code></pre>

Example 2: C/C++ Blog Content

<p>Hey robotics club members, the
<a href="https://forum.arduino.cc/index.php?topic=98155.0">Arduino forum</a>
    had the answer we were looking for!</p>

<pre><code class="cpp">
#if !(defined(ARDUINO) && ARDUINO >= 100)
  #include <EthernetDNS.h>   
#endif
</code></pre>

Similar syntax conflicts could occur with other programming languages like various BASIC dialects, C# and Verilog. However, the LLVM languages Swift and C/C++ seemed closer to home for this forum. ;-)

Detail: #endif vs. #endif#

Here is a ): example where an #endif syntax has ambiguity issues.

<h1>#(title)</h1>

#if(showswiftexample):
<pre><code class="swift">
#if(DEBUG)
   loglevel = LogLevel.verbose 
#endif
</code></pre>
#endif

In contrast, the #if(...)# ... #endif# syntax is unambiguous.

<h1>#(title)</h1>

#if(showswiftexample)#
<pre><code class="swift">
#if(DEBUG)
   loglevel = LogLevel.verbose 
#endif
</code></pre>
#endif#

Messaging To Users

Changing from #if(...) {...} to #if(...): ... #endif will clearly be a breaking change when released. So, this could be a good time to instead nudge the Leaf(4) syntax to the stronger #if(...)# ... #endif# syntax.

Not only can #if(...)# ... #endif# add robustness to the templating language processing, the communication to existing and potential new users could be strengthened as well.

Consider the following...

  • #branding# Messaging. The html templating category is already somewhat crowded so strong, clear messaging is essential to promote acceptance. Own the #...# syntax pattern. Visualize #leaf(4)# clearly. Message #leaf(4)# consistently. Present #leaf(4)# boldly.

  • #visibility# Readable. The #leaf(4)# tags are clearly, visually differentiated in the code.

  • #workflow# Efficient. Reduces content (re-)processing. For example, render html generated from markdown as-is without an added speed bump.

  • #robust# Tolerant. Cleanly renders everything one would typically expect plus mixed content like programming language discussion snippets, and complex content like html LaTeX mathematical presentation.

  • #performant# Fast. Built directly on Swift NIO and Codable. Dramatically simplified lexing and parsing over Leaf 3.

  • #customizable# YourWay. The place to be for #yourcustomizedtag()#. More than one argument. Default arguments.

Leaf4

All good reasons for both existing users and new users to be #happy(∞)#. ;-}

Observation: communications is likely as much a critical factor for long term success of Leaf as finding a technically implementable syntax.


Well, ok, maybe some the comments were a bit much. However, hopefully the comments provide some understanding regarding why #if(...)# ... #endif# is worth consideration. Cheers.


#18

@tanner0101 There is an interesting edge case for the "Markdown to HTML Workflow" challenge test criteria. What happens to markdown blog content which discussed the templating language itself?

In the case of a hypothetical #Leaf(4)#, the #raw# ... #endraw# pattern would be interesting to consider.

For the example below, two ground rules are applied:

  1. Since the markdown blogger is writing about #Leaf(4)#, it follows that the blogger also understands and uses correct primary #Leaf(4)# syntax. e.g. #raw# ... #endraw#

  2. When nested #raw# ,,, #raw#...#endraw# ,,, #endraw#, the outer #raw# ... #endraw# is processed by #Leaf(4)# and the inner content remains as-is.

    • Before: #raw# ,,, #raw#...#endraw# ,,, #endraw#
    • After: ,,, #raw#...#endraw# ,,,

Example: How to write about using #Leaf(4)# in a Markdown source document that will be later process by #Leaf(4)#.

example.md (Markdown source.)

MarkdownLeaf01_

example-before-leaf.html (Snippet after markdown to html generation.)

MarkdownLeaf02_

example-after-leaf.html (What gets delivered to the browser.)

MarkdownLeaf03_

In this example the outer #raw# ... #endraw# inside the markdown fenced code syntax preserves the inner #Leaf(4)# example for final rendering in the browser.


(Tanner) #19

Hahaha, yes. I love this. :)

#if(DEBUG) would be ambiguous regardless of which symbol is used for the tag body indicator. All versions of Leaf have used the # symbol as the tag start indicator. So, Leaf would parse #if(DEBUG) as a tag named if with a single parameter DEBUG. To avoid Leaf parsing this, you would need to escape the tag start indicator:

<h1>#(title)</h1>

#if(showswiftexample):
<pre><code class="swift">
\#if(DEBUG)
   loglevel = LogLevel.verbose 
\#endif
</code></pre>
#endif

I still think using # as the tag body indicator is an interesting discussion, I just think this particular issue is unrelated. Fixing ambiguity of the tag start indicator is also an interesting discussion. We've considered allowing people to configure a different character for the tag start symbol. For example, someone working with Markdown might prefer using $

# Section: $(section.title)

Hello $(user.name)

We've also considered allowing for people to specify more than one byte as the tag start indicator, so that you could do an emoji or a string of characters.

<h1>@@(title)</h1>

For the sake of the tag body indicator though, the parser will have already been triggered to look at the tag in question by the tag start indicator. So really the choice becomes an aesthetic one. Which symbol do we think looks better?

#if(user):
    Hello #(user.name)
#endif
#if(user)#
    Hello #(user.name)
#endif
#if(user)🤠
    Hello #(user.name)
#endif

For me, at least, it's not immediately clear why other tags like #("echo") shouldn't also be suffixed with a #. The :, given its related usage in English grammar, makes it clear that something of note should follow.

To summarize, the discussion about Leaf's tag start indicator is interesting, but beside the point of the tag body indicator spelling. I do think we should discuss this, just possibly as a separate pitch.


Regarding the #raw tag, I would love to see something like that in Leaf. But that is worthy of its own pitch. I think it would be great if it worked like Swift 5's raw strings in that you could dynamically configure the delimiters.

In Swift:

56%20PM

In Leaf (maybe):

#raw
Now we can use #(foo) without escaping
#endraw

#raw#
Now we can use #endraw without escaping!
#endraw#