[discussion] compile-time execution and metaprogramming (was: "constexpr" for Swift 5)

As with many other threads, the "constexpr" one changed into a fixed-size array thread :laughing:, so instead of fighting it, let's branch off into another topic that also almost took over.

The bottom line of this discussion is to weight pros and cons of a compile-time execution feature and a metaprogramming feature built on top of it.

The way I imagine it to work is to have a new keyword `compiletime` (or something similar) that can be applied to any executable block (property accessor, property observer, function, closure), which would restrict the use of runtime-only features of swift (the exact list of which is yet to be defined) and make that block of code eligible for guaranteed compiletime execution. Being eligible for compile-time execution essentially means having the return value (and values of mutated state) pass as a literal in the eyes of the compiler. Obviously, restrictions of the subset of Swift (which would most definitely include access to source code) in the compile-time context are transitive: the entire call hierarchy has to adhere to the restrictions with no exceptions. Also, obviously, in order to eligible code blocks to execute, all their captured context and parameters have to also be compiletime. Compiletime variables are mutable state during the compilation, which are demoted to compile-time constants after the compilation.

According to John McCall, the beginnings of this already exist in the form of a simple expression folding that could be extended to encompass more types of code to be executed at compile-time.

The metaprogramming part is all about generating language constructs programmatically. Currently, that niche is occupied by text-based boilerplate generators like Gyb, Sourcery and the like.
With the compile-time execution available, boilerplate could be remove from the language entirely due to an unprecedented ability to modify the fully parsed, structurized binary representation of the code form inside the code.
A set of built-in types and functions could be used to generate instances of TypeAliasDeclaration, StructureDefinition, VariableDeclaration and such. These would then be fed into a compile-time in-language representation of the compiler to cause it to insert them into the current module for use by other code. This would work very similarly to LLVM's instruction builder class.

According to David Sweeris, getting rid of compiler magic is one of Swift's goals and you can see how this will be single most powerful way of reducing the compiler to its bare minimum and moving most of the runtime into the standard library.

My main point of why I'd like to start an in-depth discussion about this now is because this is a huge endeavor with tremendous consequences, which need time to think through and plan accordingly. Swift 4 is about to release in a few months and Swift 5's stages and priorities will be on the table. I'd like to see where this can go *before* we miss the window of opportunity on such a proposal and are stuck waiting for another release cycle.

I'd like to hear as many opinions as possible (especially from the core team, which is ultimately going to have to decide whether or not this is ever going to happen).

1 Like

This is a very huge topic, but I can't fight the feeling that the final goal of the concept has mostly been ignored so far — maybe it's to exotic compared to macros:
I don't know a single language that has adopted such features (except, of course, Lisp :wink: — but I guess it didn't adopt anything, it's just possible to do lots of crazy things with it ;-).
None the less, I think the concept is fascinating, and I hope it gains traction.

If it does, I predict the ML isn't structured enough to bring the topic forward, and there's no such thing as "sub-mailinglists" yet. I don't know if Discourse would help here, but I also don't know if the switch will ever happen, so maybe we could create a wiki-page* to collect thoughts?

- Tino

* it's rather easy, but to be honest, my experience is that it doesn't work out :frowning:
Maybe big topics like this (multithreading would be another one) aren't suited to be shaped by the whole "community" — but let's wait, one or two statements from Core may be enough to fuel attention ;-).

This is a very huge topic, but I can't fight the feeling that the final goal of the concept has mostly been ignored so far — maybe it's to exotic compared to macros:
I don't know a single language that has adopted such features (except, of course, Lisp :wink: — but I guess it didn't adopt anything, it's just possible to do lots of crazy things with it ;-).
None the less, I think the concept is fascinating, and I hope it gains traction.

If it does, I predict the ML isn't structured enough to bring the topic forward, and there's no such thing as "sub-mailinglists" yet. I don't know if Discourse would help here, but I also don't know if the switch will ever happen, so maybe we could create a wiki-page* to collect thoughts?

- Tino

* it's rather easy, but to be honest, my experience is that it doesn't work out :frowning:
Maybe big topics like this (multithreading would be another one) aren't suited to be shaped by the whole "community" — but let's wait, one or two statements from Core may be enough to fuel attention ;-).

While we're considering compile-time execution, we should take a look at the `#run` directive in Jon Blow's Jai programming language. It allows you to run arbitrary code in your program at compile time and then utilize the output in your program.

This page has a good summary and an example of the feature: https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md#arbitrary-compile-time-code-execution

I'm no expert, so I'm not sure if there's a reason something like this wouldn't work in the world of Swift, but it seems extremely elegant and flexible.

Jarod

···

On Jul 31, 2017, 23:58 -0700, Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org>, wrote:

As with many other threads, the "constexpr" one changed into a fixed-size array thread :laughing:, so instead of fighting it, let's branch off into another topic that also almost took over.

The bottom line of this discussion is to weight pros and cons of a compile-time execution feature and a metaprogramming feature built on top of it.

The way I imagine it to work is to have a new keyword `compiletime` (or something similar) that can be applied to any executable block (property accessor, property observer, function, closure), which would restrict the use of runtime-only features of swift (the exact list of which is yet to be defined) and make that block of code eligible for guaranteed compiletime execution. Being eligible for compile-time execution essentially means having the return value (and values of mutated state) pass as a literal in the eyes of the compiler. Obviously, restrictions of the subset of Swift (which would most definitely include access to source code) in the compile-time context are transitive: the entire call hierarchy has to adhere to the restrictions with no exceptions. Also, obviously, in order to eligible code blocks to execute, all their captured context and parameters have to also be compiletime. Compiletime variables are mutable state during the compilation, which are demoted to compile-time constants after the compilation.

According to John McCall, the beginnings of this already exist in the form of a simple expression folding that could be extended to encompass more types of code to be executed at compile-time.

The metaprogramming part is all about generating language constructs programmatically. Currently, that niche is occupied by text-based boilerplate generators like Gyb, Sourcery and the like.
With the compile-time execution available, boilerplate could be remove from the language entirely due to an unprecedented ability to modify the fully parsed, structurized binary representation of the code form inside the code.
A set of built-in types and functions could be used to generate instances of TypeAliasDeclaration, StructureDefinition, VariableDeclaration and such. These would then be fed into a compile-time in-language representation of the compiler to cause it to insert them into the current module for use by other code. This would work very similarly to LLVM's instruction builder class.

According to David Sweeris, getting rid of compiler magic is one of Swift's goals and you can see how this will be single most powerful way of reducing the compiler to its bare minimum and moving most of the runtime into the standard library.

My main point of why I'd like to start an in-depth discussion about this now is because this is a huge endeavor with tremendous consequences, which need time to think through and plan accordingly. Swift 4 is about to release in a few months and Swift 5's stages and priorities will be on the table. I'd like to see where this can go *before* we miss the window of opportunity on such a proposal and are stuck waiting for another release cycle.

I'd like to hear as many opinions as possible (especially from the core team, which is ultimately going to have to decide whether or not this is ever going to happen).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

While we're considering compile-time execution, we should take a look at the `#run` directive in Jon Blow's Jai programming language. It allows you to run arbitrary code in your program at compile time and then utilize the output in your program.

This page has a good summary and an example of the feature: https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md#arbitrary-compile-time-code-execution

I read through the page and the way compile-time mechanism in Jai is very close to what I had in mind. I'm curious about how it turned out for them.

···

On Aug 1, 2017, at 8:50 PM, swift@lng.la wrote:

I'm no expert, so I'm not sure if there's a reason something like this wouldn't work in the world of Swift, but it seems extremely elegant and flexible.

Jarod

On Jul 31, 2017, 23:58 -0700, Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org>, wrote:

As with many other threads, the "constexpr" one changed into a fixed-size array thread :laughing:, so instead of fighting it, let's branch off into another topic that also almost took over.

The bottom line of this discussion is to weight pros and cons of a compile-time execution feature and a metaprogramming feature built on top of it.

The way I imagine it to work is to have a new keyword `compiletime` (or something similar) that can be applied to any executable block (property accessor, property observer, function, closure), which would restrict the use of runtime-only features of swift (the exact list of which is yet to be defined) and make that block of code eligible for guaranteed compiletime execution. Being eligible for compile-time execution essentially means having the return value (and values of mutated state) pass as a literal in the eyes of the compiler. Obviously, restrictions of the subset of Swift (which would most definitely include access to source code) in the compile-time context are transitive: the entire call hierarchy has to adhere to the restrictions with no exceptions. Also, obviously, in order to eligible code blocks to execute, all their captured context and parameters have to also be compiletime. Compiletime variables are mutable state during the compilation, which are demoted to compile-time constants after the compilation.

According to John McCall, the beginnings of this already exist in the form of a simple expression folding that could be extended to encompass more types of code to be executed at compile-time.

The metaprogramming part is all about generating language constructs programmatically. Currently, that niche is occupied by text-based boilerplate generators like Gyb, Sourcery and the like.
With the compile-time execution available, boilerplate could be remove from the language entirely due to an unprecedented ability to modify the fully parsed, structurized binary representation of the code form inside the code.
A set of built-in types and functions could be used to generate instances of TypeAliasDeclaration, StructureDefinition, VariableDeclaration and such. These would then be fed into a compile-time in-language representation of the compiler to cause it to insert them into the current module for use by other code. This would work very similarly to LLVM's instruction builder class.

According to David Sweeris, getting rid of compiler magic is one of Swift's goals and you can see how this will be single most powerful way of reducing the compiler to its bare minimum and moving most of the runtime into the standard library.

My main point of why I'd like to start an in-depth discussion about this now is because this is a huge endeavor with tremendous consequences, which need time to think through and plan accordingly. Swift 4 is about to release in a few months and Swift 5's stages and priorities will be on the table. I'd like to see where this can go *before* we miss the window of opportunity on such a proposal and are stuck waiting for another release cycle.

I'd like to hear as many opinions as possible (especially from the core team, which is ultimately going to have to decide whether or not this is ever going to happen).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

It's hard to say at this point since Jai is in an early state and isn't publicly available yet, but it seems to be working well on a small scale at least from the demos that have been shown.

If anyone's curious, this is the video where the feature was originally demoed. It's from the very beginning of the language's development, so a lot of it is probably out of date, but it should still be a good introduction to the feature: https://www.youtube.com/watch?v=UTqZNujQOlA

Jarod

···

On Aug 1, 2017, 11:20 -0700, Gor Gyolchanyan <gor.f.gyolchanyan@icloud.com>, wrote:

> On Aug 1, 2017, at 8:50 PM, swift@lng.la wrote:
>
> While we're considering compile-time execution, we should take a look at the `#run` directive in Jon Blow's Jai programming language. It allows you to run arbitrary code in your program at compile time and then utilize the output in your program.
>
> This page has a good summary and an example of the feature: https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md#arbitrary-compile-time-code-execution

I read through the page and the way compile-time mechanism in Jai is very close to what I had in mind. I'm curious about how it turned out for them.

> I'm no expert, so I'm not sure if there's a reason something like this wouldn't work in the world of Swift, but it seems extremely elegant and flexible.
>
> Jarod
>
> On Jul 31, 2017, 23:58 -0700, Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org>, wrote:
> > As with many other threads, the "constexpr" one changed into a fixed-size array thread :laughing:, so instead of fighting it, let's branch off into another topic that also almost took over.
> >
> > The bottom line of this discussion is to weight pros and cons of a compile-time execution feature and a metaprogramming feature built on top of it.
> >
> > The way I imagine it to work is to have a new keyword `compiletime` (or something similar) that can be applied to any executable block (property accessor, property observer, function, closure), which would restrict the use of runtime-only features of swift (the exact list of which is yet to be defined) and make that block of code eligible for guaranteed compiletime execution. Being eligible for compile-time execution essentially means having the return value (and values of mutated state) pass as a literal in the eyes of the compiler. Obviously, restrictions of the subset of Swift (which would most definitely include access to source code) in the compile-time context are transitive: the entire call hierarchy has to adhere to the restrictions with no exceptions. Also, obviously, in order to eligible code blocks to execute, all their captured context and parameters have to also be compiletime. Compiletime variables are mutable state during the compilation, which are demoted to compile-time constants after the compilation.
> >
> > According to John McCall, the beginnings of this already exist in the form of a simple expression folding that could be extended to encompass more types of code to be executed at compile-time.
> >
> > The metaprogramming part is all about generating language constructs programmatically. Currently, that niche is occupied by text-based boilerplate generators like Gyb, Sourcery and the like.
> > With the compile-time execution available, boilerplate could be remove from the language entirely due to an unprecedented ability to modify the fully parsed, structurized binary representation of the code form inside the code.
> > A set of built-in types and functions could be used to generate instances of TypeAliasDeclaration, StructureDefinition, VariableDeclaration and such. These would then be fed into a compile-time in-language representation of the compiler to cause it to insert them into the current module for use by other code. This would work very similarly to LLVM's instruction builder class.
> >
> > According to David Sweeris, getting rid of compiler magic is one of Swift's goals and you can see how this will be single most powerful way of reducing the compiler to its bare minimum and moving most of the runtime into the standard library.
> >
> > My main point of why I'd like to start an in-depth discussion about this now is because this is a huge endeavor with tremendous consequences, which need time to think through and plan accordingly. Swift 4 is about to release in a few months and Swift 5's stages and priorities will be on the table. I'd like to see where this can go *before* we miss the window of opportunity on such a proposal and are stuck waiting for another release cycle.
> >
> > I'd like to hear as many opinions as possible (especially from the core team, which is ultimately going to have to decide whether or not this is ever going to happen).
> >
> > _______________________________________________
> > swift-evolution mailing list
> > swift-evolution@swift.org
> > https://lists.swift.org/mailman/listinfo/swift-evolution

Terms of Service

Privacy Policy

Cookie Policy