[Planning][Request] "constexpr" for Swift 5

[Gor_Gyolchanyan]

I am not sure that macros and annotations would be good, either. It may be impossible to avoid them, given Swift’s goal of “world domination”. Both features are very useful in certain applications.

I’d argue that exposing swift’s front-end as a built-in module with APIs for creating instances of language constructs and building them programmatically and then allowing one to run that code at compile time would make it much easier to achieve “world domination” because generating code through a well-defined and well-documented API written in the same language as the one being generated would be much easier than using the equivalent of what Xcode uses to represent the code for semantical highlighting, navigation and debugging purposes (which, I imagine, is just text with some metadata).

We could continue to use custom tools like Cog and Sourcery when we need to, or standardize and better integrate their functionality. I honestly am not sure which is a better solution.

Well, gyb is the tool written specifically for swift and a large portion of the standard library and the `Builtin` module is generated by gyb. That doesn’t look like a design choice to me. It looks more like the lesser of two evils (the other being manual, error-prone boilerplate).

"Hygienic Macros” and “Annotations” have both been discussed. Both of these features would bring power and complexity to the language, to be sure. I just wanted to voice my opinion that these features, if and when they arrive, should be transparent, i.e., the generated code should be visible and sensibly formatted so that it can be examined and debugged. This would be a hard problem, and I am not sure if it is even possible for complex macro & annotation combinations…

If the metaprogramming facilities are implemented as a builtin module that exposes the swift frontend as a compile-time API, it would be much easier to adapt the debugging tools to encompass it, than trying to reverse-engineer the code, generated by semantically-unaware macro system, since the generated code was never in textual form in the first place. It would even allow things like Xcode generating a textural representation of the generated source for debugging purposes (where your mentioned formatters would come in handy).

C macros are not hygienic and not “transparent”. C preprocessor output, for example, is largely unusable.

The uses of macros and annotation overlap. Introducing macros and annotations at the same time would provide a chance to sensibly balance the features of each and provide a coherent set of guidelines for developers.

I’m not sure what you mean by “annotations”, but if you’re referring to arbitrary user-defined metadata attached to language constructs, then the API-based codeine would make this exceedingly easy, because LLVM libraries natively support metadata on pretty much everything, so it would be a simple matter of not forgetting to expose those to Swift as well.

···

On Jul 30, 2017, at 9:29 PM, Christopher Kornher <ckornher@me.com> wrote:

- Chris K

On Jul 30, 2017, at 12:03 PM, Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I don’t think a Cog-like or Gyb-like macro system would be good. That’s essentially a preprocessor and we all know how that worked out for C. The only a code-generation feature can exist in a safe language like Swift is by making it a first-class citizen as described previously. I’m not sure whether or not the core team would consider such a thing, bit I am very certain that a text-based macro system is something they’d never consider.

On Jul 30, 2017, at 8:54 PM, Christopher Kornher <ckornher@me.com <mailto:ckornher@me.com>> wrote:

I assume that the planned “hygienic macro” facility, if it has access to metadata, or “annotations", could be used for these sorts of meta tasks, .

This is slightly off-topic, but I hope that there is a way for macros to be fully expanded and debuggable (and even formatted). I use Cog with Swift as a kind of "macro-system on steroids” that provides these features. I would love to see a Cog-like system included as a standard feature of the language in the future. It could be a component of annotations, macros, or both.

I would like to see a source code generation pre-pass added for annotations, and hopefully macros when they arrive, so that developers can see, debug, and code against the expanded code. I realize that may be difficult or even impossible for some macros.

GOG uses Python as its macro language, which is certainly compatible with Apple’s toolchain. Using a real programming language for annotations, is extremely powerful.

The danger of adding a macro system too early, is that it can be used as a “cheat” to implement functionality that should be part of the base language and discourage language development. I trust that the core team will not let this happen.

See Cog:
Cog | Ned Batchelder

A fully integrated Cog-like facility this should have language support to cleanup the (pretty ugly) delimiters and eventually tool support to selectively toggle expansion, as we can today with code blocks, for example, in many tools.

I don’t mean to derail this discussion, but it seems that an annotation or macro system would be appropriate for this kind of feature.

- Chris

On Jul 30, 2017, at 11:03 AM, Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Tino Heth:
If you read my reply to Daniel Vollmer, you’ll find that we’re thinking about the exact same thing. Your code snippers show my vision of compiletime beautifully .
Now what I really want at this point is to here the opinions of the core team on this topic.

Swift Core Team:
Have you guys thought of this? Do you think this is a good idea to put on the table or do you have different plans?

On Jul 30, 2017, at 7:56 PM, Tino Heth <2th@gmx.de <mailto:2th@gmx.de>> wrote:

more elaborate compile-time facilities, which would also provide extremely powerful meta programming features

That's an interesting twist — but whenever you put a "meta" somewhere, things tend to get complicated, and people end up with different associations to the topic… ;-)
I took a brief look at the C++ document, but it seemed still to much macro-like to me.

My take on the topic would be he ability to express common programming tasks (declaring a class, overriding a method…) in the language itself.
Imagine
public class ProxyViewController: UIView {}
Could be written as
let subclass = createClass(classname: "ProxyViewController", superclass: UIViewController, accessLevel: .public)

Quite stupid at first sight, and basically the exact opposite of syntactic sugar ("syntactic salt" already has a meaning… so I'd call it "syntactic pepper" ;-).
But now imagine that:

for (method, implementation) in UIViewController.methods where method.accessLevel == .open {
    subclass.methods[method] = { parameters in
        print("Subclass method \(method) called with \(parameters)")
        return implementation(parameters)
    }
}

Not that stupid anymore, isn't it?
I think this would be way cooler than poking around with variants of search & replace…

- Tino

(to get syntax colouring, I wrote ~30 lines of Swift that turn the straw man example into valid code… it's fun, maybe I play with it a little bit more ;-)

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[Felix_Cloutier1 (Félix Cloutier)]

>
>> The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as well use a regular parameter if there's no compile-time evaluation involved. In that case, fixed-sized arrays will be useless, because they'll be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why *shouldn't* it get all the benefits of a fixed-sized array without having to use a special syntax? Put another way, why shouldn't fixed-size be one of those optional attributes for arrays, like ownership will be for variables, that users can opt into for more performance but is otherwise automatically worked out by the compiler?

The optimization is defeated as soon as the storage buffer might escape, just like for closures. Consider this:

func foo(arr: [Int])

var array = [1, 2, 3, 4]
foo(arr: array)
array[1] = 2

If `array` is "unrolled" to the stack, and `foo` keeps a reference to the storage, then `array[1] = 2` has to pessimistically create a new storage buffer to maintain COW semantics. It could also leave a dangling reference to the buffer if the function returns before all of the other references are gone. I'd rather not get into @escaping arrays.

I don't think that you claimed that this should be a solution to the fixed-size array problem, but just in case, also note that it's not. We can't make any [Int] layout-compatible with C fixed-size arrays because the length has to be encoded in the type (as it cannot be encoded in the data itself).

Félix

···

Le 31 juil. 2017 à 00:40, Xiaodi Wu via swift-evolution <swift-evolution@swift.org> a écrit :
On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com <mailto:darylew@mac.com>> wrote:

[xwu (Xiaodi Wu)]

>
>> The parameters for a fixed-size array type determine the type's
size/stride, so how could the bounds not be needed during compile-time? The
compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run
time. It already has to do that for generic types and types with resilient
members. That does, of course, have performance consequences, and those
performance consequences might be unacceptable to you; but the fact that we
can handle it means that we don't ultimately require a semantic concept of
a constant expression, except inasmuch as we want to allow users to
explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as
well use a regular parameter if there's no compile-time evaluation
involved. In that case, fixed-sized arrays will be useless, because they'll
be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why
*shouldn't* it get all the benefits of a fixed-sized array without having
to use a special syntax? Put another way, why shouldn't fixed-size be one
of those optional attributes for arrays, like ownership will be for
variables, that users can opt into for more performance but is otherwise
automatically worked out by the compiler?

The optimization is defeated as soon as the storage buffer might escape,
just like for closures. Consider this:

func foo(arr: [Int])

var array = [1, 2, 3, 4]
foo(arr: array)
array[1] = 2

If `array` is "unrolled" to the stack, and `foo` keeps a reference to the
storage, then `array[1] = 2` has to pessimistically create a new storage
buffer to maintain COW semantics. It could also leave a dangling reference
to the buffer if the function returns before all of the other references
are gone. I'd rather not get into @escaping arrays.

Sure, and hence my point: suppose now `foo` is a function in the stdlib,
and the stdlib authors have annotated the function so that it is `func
foo(arr: fixed [Int])`. Then, any user who writes `var array = ...` could
benefit from a performance boost because the compiler will not longer have
to be pessimistic about copying in order to maintain COW semantics. This is
why I made an explicit analogy to the design proposed for ownership in
Swift, where end users don't have to understand it in order to benefit from
the feature because the functions they call can give sufficiently important
hints to help the compiler avoid unnecessary copying.

I don't think that you claimed that this should be a solution to the

fixed-size array problem, but just in case, also note that it's not. We
can't make any [Int] layout-compatible with C fixed-size arrays because the
length has to be encoded in the type (as it cannot be encoded in the data
itself).

I don't understand this point. Can you elaborate on what you mean here? Why
does it have to be layout-compatible?

···

On Mon, Jul 31, 2017 at 3:39 AM, Félix Cloutier <felixcloutier@icloud.com> wrote:

Le 31 juil. 2017 à 00:40, Xiaodi Wu via swift-evolution < > swift-evolution@swift.org> a écrit :
On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution < > swift-evolution@swift.org> wrote:

> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution < >> swift-evolution@swift.org> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:

Félix

[xwu (Xiaodi Wu)]

>
>> The parameters for a fixed-size array type determine the type's
size/stride, so how could the bounds not be needed during compile-time? The
compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run
time. It already has to do that for generic types and types with resilient
members. That does, of course, have performance consequences, and those
performance consequences might be unacceptable to you; but the fact that we
can handle it means that we don't ultimately require a semantic concept of
a constant expression, except inasmuch as we want to allow users to
explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as
well use a regular parameter if there's no compile-time evaluation
involved. In that case, fixed-sized arrays will be useless, because they'll
be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why
*shouldn't* it get all the benefits of a fixed-sized array without having
to use a special syntax? Put another way, why shouldn't fixed-size be one
of those optional attributes for arrays, like ownership will be for
variables, that users can opt into for more performance but is otherwise
automatically worked out by the compiler?

As far as I know, the pinnacle of uses for fixed-size arrays is having a

···

On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution < swift-evolution@swift.org> wrote:

> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution < > swift-evolution@swift.org> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:
compile-time pre-allocated space of the necessary size (either literally at
compile-time if that's a static variable, or added to the pre-computed
offset of the stack pointer in case of a local variable).

> Value equality would still affect the type-checker, but I think we could
pretty easily just say that all bound expressions are assumed to
potentially resolve unequally unless they are literals or references to the
same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test
for equality? Moreover, as far as I know, Equatable is not recognized by
the compiler in any way, so it's just a regular protocol. What would make
it special? Some types would implement operator == to compare themselves to
other types, that's beyond the scope of Equatable. What about those? And
how are custom operator implementations going to serve this purpose at
compile-time? Or will it just ignore the semantics of the type and reduce
it to a sequence of bits? Or maybe only a few hand-picked types will be
supported?

The seemingly simple generic value parameter concept gets vastly
complicated and/or poorly designed without an elaborate compile-time
execution system... Unless I'm missing an obvious way out.

> The only hard constraint is that types need to be consistent, but that
just means that we need to have a model in which bound expressions are
evaluated exactly once at runtime (and of course typically folded at
compile time).

What exactly would it take to be able to execute select piece of code at
compile-time? Taking the AST, converting it to LLVM IR and feeding it to
the MCJIT engine seems to be easy enough. But I'm pretty sure it's more
tricky than that. Is there a special assumption or two made about the code
that prevents this from happening?

> John.
>
>> Or do you mean that the bounds are integer literals? (That's what I
have in the design document now.)
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:
>>
>>>> On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution < > swift-evolution@swift.org> wrote:
>>>> The “constexpr” facility from C++ allows users to define constants
and functions that are determined and usable at compile-time, for
compile-time constructs but still usable at run-time. The facility is a key
step for value-based generic parameters (and fixed-size arrays if you don’t
want to be stuck with integer literals for bounds). Can figuring out
Swift’s story here be part of Swift 5?
>>>
>>> Note that there's no particular reason that value-based generic
parameters, including fixed-size arrays, actually need to be constant
expressions in Swift.
>>>
>>> John.
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[xwu (Xiaodi Wu)]

Sent from my iPhone

>
>> The parameters for a fixed-size array type determine the type's
size/stride, so how could the bounds not be needed during compile-time? The
compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run
time. It already has to do that for generic types and types with resilient
members. That does, of course, have performance consequences, and those
performance consequences might be unacceptable to you; but the fact that we
can handle it means that we don't ultimately require a semantic concept of
a constant expression, except inasmuch as we want to allow users to
explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as
well use a regular parameter if there's no compile-time evaluation
involved. In that case, fixed-sized arrays will be useless, because they'll
be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why
*shouldn't* it get all the benefits of a fixed-sized array without having
to use a special syntax? Put another way, why shouldn't fixed-size be one
of those optional attributes for arrays, like ownership will be for
variables, that users can opt into for more performance but is otherwise
automatically worked out by the compiler?

Because in some cases humans may make an easier judgement call than the
compiler?

Hence, why I'm asking why it shouldn't be an option for the human to help
the compiler with an annotation.

Because the compiler complexity is already growing and compile time is not

exactly showing massive reductions and people are worried that as Swift
gains more features and complexity with genetics and reflection and other
features that compilation time will grow even more?

Do you have any evidence that this would be a performance-limiting task for
the compiler? Even if so, what's wrong with limiting the compiler to
inferring only the easiest cases, and allowing for manual annotation to
help the compiler in more complicated ones as an optional performance
enhancement? In other words, my question is, what is wrong with fixed-sized
arrays being--syntactically--simply arrays with resizing disabled? The flip
side of the coin is, why shouldn't normal arrays simply
be--syntactically--fixed-sized arrays with resizing enabled?

As far as I know, the pinnacle of uses for fixed-size arrays is having a

···

On Mon, Jul 31, 2017 at 02:49 Goffredo Marocchi <panajev@gmail.com> wrote:

On 31 Jul 2017, at 08:40, Xiaodi Wu via swift-evolution < > swift-evolution@swift.org> wrote:
On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution < > swift-evolution@swift.org> wrote:

> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution < >> swift-evolution@swift.org> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:
compile-time pre-allocated space of the necessary size (either literally at
compile-time if that's a static variable, or added to the pre-computed
offset of the stack pointer in case of a local variable).

> Value equality would still affect the type-checker, but I think we
could pretty easily just say that all bound expressions are assumed to
potentially resolve unequally unless they are literals or references to the
same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test
for equality? Moreover, as far as I know, Equatable is not recognized by
the compiler in any way, so it's just a regular protocol. What would make
it special? Some types would implement operator == to compare themselves to
other types, that's beyond the scope of Equatable. What about those? And
how are custom operator implementations going to serve this purpose at
compile-time? Or will it just ignore the semantics of the type and reduce
it to a sequence of bits? Or maybe only a few hand-picked types will be
supported?

The seemingly simple generic value parameter concept gets vastly
complicated and/or poorly designed without an elaborate compile-time
execution system... Unless I'm missing an obvious way out.

> The only hard constraint is that types need to be consistent, but that
just means that we need to have a model in which bound expressions are
evaluated exactly once at runtime (and of course typically folded at
compile time).

What exactly would it take to be able to execute select piece of code at
compile-time? Taking the AST, converting it to LLVM IR and feeding it to
the MCJIT engine seems to be easy enough. But I'm pretty sure it's more
tricky than that. Is there a special assumption or two made about the code
that prevents this from happening?

> John.
>
>> Or do you mean that the bounds are integer literals? (That's what I
have in the design document now.)
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:
>>
>>>> On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution < >> swift-evolution@swift.org> wrote:
>>>> The “constexpr” facility from C++ allows users to define constants
and functions that are determined and usable at compile-time, for
compile-time constructs but still usable at run-time. The facility is a key
step for value-based generic parameters (and fixed-size arrays if you don’t
want to be stuck with integer literals for bounds). Can figuring out
Swift’s story here be part of Swift 5?
>>>
>>> Note that there's no particular reason that value-based generic
parameters, including fixed-size arrays, actually need to be constant
expressions in Swift.
>>>
>>> John.
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Panajev (Goffredo Marocchi)]

>
>> The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as well use a regular parameter if there's no compile-time evaluation involved. In that case, fixed-sized arrays will be useless, because they'll be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why *shouldn't* it get all the benefits of a fixed-sized array without having to use a special syntax? Put another way, why shouldn't fixed-size be one of those optional attributes for arrays, like ownership will be for variables, that users can opt into for more performance but is otherwise automatically worked out by the compiler?

Because in some cases humans may make an easier judgement call than the compiler? Because the compiler complexity is already growing and compile time is not exactly showing massive reductions and people are worried that as Swift gains more features and complexity with genetics and reflection and other features that compilation time will grow even more?

···

Sent from my iPhone

On 31 Jul 2017, at 08:40, Xiaodi Wu via swift-evolution <swift-evolution@swift.org> wrote:

On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org> wrote:
> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:

As far as I know, the pinnacle of uses for fixed-size arrays is having a compile-time pre-allocated space of the necessary size (either literally at compile-time if that's a static variable, or added to the pre-computed offset of the stack pointer in case of a local variable).

> Value equality would still affect the type-checker, but I think we could pretty easily just say that all bound expressions are assumed to potentially resolve unequally unless they are literals or references to the same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test for equality? Moreover, as far as I know, Equatable is not recognized by the compiler in any way, so it's just a regular protocol. What would make it special? Some types would implement operator == to compare themselves to other types, that's beyond the scope of Equatable. What about those? And how are custom operator implementations going to serve this purpose at compile-time? Or will it just ignore the semantics of the type and reduce it to a sequence of bits? Or maybe only a few hand-picked types will be supported?

The seemingly simple generic value parameter concept gets vastly complicated and/or poorly designed without an elaborate compile-time execution system... Unless I'm missing an obvious way out.

> The only hard constraint is that types need to be consistent, but that just means that we need to have a model in which bound expressions are evaluated exactly once at runtime (and of course typically folded at compile time).

What exactly would it take to be able to execute select piece of code at compile-time? Taking the AST, converting it to LLVM IR and feeding it to the MCJIT engine seems to be easy enough. But I'm pretty sure it's more tricky than that. Is there a special assumption or two made about the code that prevents this from happening?

> John.
>
>> Or do you mean that the bounds are integer literals? (That's what I have in the design document now.)
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:
>>
>>>> On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution <swift-evolution@swift.org> wrote:
>>>> The “constexpr” facility from C++ allows users to define constants and functions that are determined and usable at compile-time, for compile-time constructs but still usable at run-time. The facility is a key step for value-based generic parameters (and fixed-size arrays if you don’t want to be stuck with integer literals for bounds). Can figuring out Swift’s story here be part of Swift 5?
>>>
>>> Note that there's no particular reason that value-based generic parameters, including fixed-size arrays, actually need to be constant expressions in Swift.
>>>
>>> John.
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Felix_Cloutier1 (Félix Cloutier)]

Sure, and hence my point: suppose now `foo` is a function in the stdlib, and the stdlib authors have annotated the function so that it is `func foo(arr: fixed [Int])`. Then, any user who writes `var array = ...` could benefit from a performance boost because the compiler will not longer have to be pessimistic about copying in order to maintain COW semantics. This is why I made an explicit analogy to the design proposed for ownership in Swift, where end users don't have to understand it in order to benefit from the feature because the functions they call can give sufficiently important hints to help the compiler avoid unnecessary copying.

I don't think that you claimed that this should be a solution to the fixed-size array problem, but just in case, also note that it's not. We can't make any [Int] layout-compatible with C fixed-size arrays because the length has to be encoded in the type (as it cannot be encoded in the data itself).

I don't understand this point. Can you elaborate on what you mean here? Why does it have to be layout-compatible?

Then it seems that I have stricter requirements for fixed-size arrays than you do, and I'd be curious to hear what you want to get out of fixed-size arrays. If we compare a hypothetical `fixed [int]` to a hypothetical `FixedSizeArray<T, N>`, these are some of the things that matter to me which `fixed` can't offer:

It doesn't allow you to specify the size of the array, it's just a promise that the array is some immutable size. For instance, a `fixed [CGFloat]` would be a terrible type to represent a vector, but a FixedSizeArray<CGFloat, 4> would be appropriate, at least for the backing storage. A raw tuple would be a poor choice because dynamically indexing into them is painful.
`fixed` is only useful when the compiler can determine the size of the array statically. This makes it mostly useless as a storage qualifier if you received the array as a parameter (*even* if you received a `fixed` array), because you know that it has a constant size but you don't know what that size is.
Therefore, using a fixed-size array as a generic parameter (crucially, such as `fixed [fixed [Int]]`) is unlikely to work.
Even if that semantic hurdle is overcome, we'd still have no idea how much memory to allocate for the outer array's buffer to make it work.
Even if `fixed [fixed [Int]]` could work, then each inner array could still have a different size, which is almost certainly not what you intend by nesting two fixed-size arrays.
Layout compatibility is important if you want to use fixed-size arrays to replace the clunky tuples that currently represent fixed-size arrays in structs exported from C (which is probably my one single biggest motivation for fixed-size arrays). You can't achieve layout compatibility if the size is part of the data instead of part of the type.

Besides, attaching fixed-size array semantics to an inherently variable-size Array is awkward. For `fixed` to be effective, it needs to disable methods that change the size of the array, or warn that you're using them. I don't like the cross-concern impact: now a keyword needs to know about method implementations to restrict them. It also has to work with extension methods on the Array type, and it shouldn't apply to just mutating functions because mutations that don't change the length of the array are fine.

What would you use `fixed [Int]` for? Only as an optimization tool?

Félix

[Gor_Gyolchanyan]

>
>> The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as well use a regular parameter if there's no compile-time evaluation involved. In that case, fixed-sized arrays will be useless, because they'll be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why *shouldn't* it get all the benefits of a fixed-sized array without having to use a special syntax? Put another way, why shouldn't fixed-size be one of those optional attributes for arrays, like ownership will be for variables, that users can opt into for more performance but is otherwise automatically worked out by the compiler?

Because the compiler shouldn't know what an array is (that is, what a Swift array is, not the C array that LLVM has a primitive type for). And if the compiler doesn't know what an array is and what resizing means, then the array has to define that optimization hint itself, which will either involve new syntax for attaching metadata for declarative special-case solutions. Besides, it won't be getting the full benefits of fixed-sized arrays. The biggest benefit of fixed-size arrays is allocation cost (which is exactly zero for static fixed-sized arrays and a single clock cycle for unescaped local fixed-sized arrays), which is not going to improve if an array is simply forbidden from resizing.

For really fast storage, Clang has __builtin_alloca, which could be wrapped in an __attribute__((__always_inline__)) function and exposed to Swift.
For semantically non-resizable arrays, we can simply implement a FixedSizeArray with no changes to the language.
Considering these options, I don't think solving the use case of fixed-size arrays is worth complicating the language.
This is why I'm pushing for elaborate compile-time execution feature. It won't solve a specific use-case, it will solve a very wide domain of use cases.

···

On Jul 31, 2017, at 10:40 AM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:
On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com <mailto:darylew@mac.com>> wrote:

As far as I know, the pinnacle of uses for fixed-size arrays is having a compile-time pre-allocated space of the necessary size (either literally at compile-time if that's a static variable, or added to the pre-computed offset of the stack pointer in case of a local variable).

> Value equality would still affect the type-checker, but I think we could pretty easily just say that all bound expressions are assumed to potentially resolve unequally unless they are literals or references to the same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test for equality? Moreover, as far as I know, Equatable is not recognized by the compiler in any way, so it's just a regular protocol. What would make it special? Some types would implement operator == to compare themselves to other types, that's beyond the scope of Equatable. What about those? And how are custom operator implementations going to serve this purpose at compile-time? Or will it just ignore the semantics of the type and reduce it to a sequence of bits? Or maybe only a few hand-picked types will be supported?

The seemingly simple generic value parameter concept gets vastly complicated and/or poorly designed without an elaborate compile-time execution system... Unless I'm missing an obvious way out.

> The only hard constraint is that types need to be consistent, but that just means that we need to have a model in which bound expressions are evaluated exactly once at runtime (and of course typically folded at compile time).

What exactly would it take to be able to execute select piece of code at compile-time? Taking the AST, converting it to LLVM IR and feeding it to the MCJIT engine seems to be easy enough. But I'm pretty sure it's more tricky than that. Is there a special assumption or two made about the code that prevents this from happening?

> John.
>
>> Or do you mean that the bounds are integer literals? (That's what I have in the design document now.)
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:
>>
>>>> On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>>>> The “constexpr” facility from C++ allows users to define constants and functions that are determined and usable at compile-time, for compile-time constructs but still usable at run-time. The facility is a key step for value-based generic parameters (and fixed-size arrays if you don’t want to be stuck with integer literals for bounds). Can figuring out Swift’s story here be part of Swift 5?
>>>
>>> Note that there's no particular reason that value-based generic parameters, including fixed-size arrays, actually need to be constant expressions in Swift.
>>>
>>> John.
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org <mailto:swift-evolution@swift.org>
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[Tino]

OTOH, if the compiler can prove that a local array is never resized, why *shouldn't* it get all the benefits of a fixed-sized array without having to use a special syntax? Put another way, why shouldn't fixed-size be one of those optional attributes for arrays, like ownership will be for variables, that users can opt into for more performance but is otherwise automatically worked out by the compiler?

I think it's a good idea in general, but I don't think I'd notice if this feature would exist, so just to save some work for the compiler (and Core), I'd prefer to first get the feature itself, and improve it later.
So from my high-level perspective, I think the order should be
0) Add support for value literals as generic parameters
1) Add Array<T, size: UInt>
2) Add support for generic parameters that are evaluated at compile time
3) Make everything fast ;-)
n) Add useful additions like detecting arrays that aren't declared as fixed length, but could

I think the alternative to macros (don't care much how hygienic they are ;-) could be even more beneficial, but it's a topic that stands on its own (not sure wether this thread is more for fixed size arrays or metaprogramming… ;-)

[John_McCall]

It seems to me that this applies just the same to type generic parameters. Are there any reason that you can't instantiate a generic type at runtime, or is it just because the need is not as evident as it would/could be with non-type generic parameters?

Are you under the impression that Swift does not, today, instantiate generic types at runtime?

A lot of this entire thread seems to be premised on really weird, incorrect assumptions about the Swift compilation model.

John.

···

On Jul 31, 2017, at 3:58 AM, Félix Cloutier <felixcca@yahoo.ca> wrote:

Le 30 juil. 2017 à 21:10, John McCall via swift-evolution <swift-evolution@swift.org> a écrit :

On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:
The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.

Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Value equality would still affect the type-checker, but I think we could pretty easily just say that all bound expressions are assumed to potentially resolve unequally unless they are literals or references to the same 'let' constant.

The only hard constraint is that types need to be consistent, but that just means that we need to have a model in which bound expressions are evaluated exactly once at runtime (and of course typically folded at compile time).

John.

Or do you mean that the bounds are integer literals? (That's what I have in the design document now.)

Sent from my iPhone

On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:

On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution <swift-evolution@swift.org> wrote:
The “constexpr” facility from C++ allows users to define constants and functions that are determined and usable at compile-time, for compile-time constructs but still usable at run-time. The facility is a key step for value-based generic parameters (and fixed-size arrays if you don’t want to be stuck with integer literals for bounds). Can figuring out Swift’s story here be part of Swift 5?

Note that there's no particular reason that value-based generic parameters, including fixed-size arrays, actually need to be constant expressions in Swift.

John.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[David_Sweeris]

Not at all... it'd let us use non-type parameters to affect a value's type... The classic example (or at least the one that I keep typing out whenever the topic comes up) is vector matrix math:
func * <T: Numeric, M: Integer, N: Integer, P: Integer> (lhs: Matrix<T, M, N>, rhs: Matrix<T, N, P>) -> Matrix<T, M, P> {
    // no need to check if the dimensions at runtime because the type system turned dimension mismatches into a compile-time error
    ...
}

Sticking with the math theme, if you can make the variable's name (that is, your package's variable type, not a Swift var or let) part of its type, I suspect there's a trick you could do in a symbolic manipulation library involving simplifying equations (but I haven't thought it through enough to say for sure).

- Dave Sweeris

···

On Jul 31, 2017, at 12:15 AM, Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org> wrote:

On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:
The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.

Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as well use a regular parameter if there's no compile-time evaluation involved. In that case, fixed-sized arrays will be useless, because they'll be normal arrays with resizing disabled. As far as I know, the pinnacle of uses for fixed-size arrays is having a compile-time pre-allocated space of the necessary size (either literally at compile-time if that's a static variable, or added to the pre-computed offset of the stack pointer in case of a local variable).

[John_McCall]

The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.

Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as well use a regular parameter if there's no compile-time evaluation involved. In that case, fixed-sized arrays will be useless, because they'll be normal arrays with resizing disabled.

You're making huge leaps here. The primary purpose of a fixed-size array feature is to allow the array to be allocated "inline" in its context instead of "out-of-line" using heap-allocated copy-on-write buffers. There is no reason that that representation would not be supportable just because the array's bound is not statically known; the only thing that matters is whether the bound is consistent for all instances of the container.

That is, it would not be okay to have a type like:
  struct Widget {
    let length: Int
    var array: [length x Int]
  }
because the value of the bound cannot be computed independently of a specific value.

But it is absolutely okay to have a type like:
  struct Widget {
    var array: [(isRunningOnIOS15() ? 20 : 10) x Int]
  }
It just means that the bound would get computed at runtime and, presumably, cached. The fact that this type's size isn't known statically does mean that the compiler has to be more pessimistic, but its values would still get allocated inline into their containers and even on the stack, using pretty much the same techniques as C99 VLAs.

As far as I know, the pinnacle of uses for fixed-size arrays is having a compile-time pre-allocated space of the necessary size (either literally at compile-time if that's a static variable, or added to the pre-computed offset of the stack pointer in case of a local variable).

The difference between having to use dynamic offsets + alloca() and static offsets + a normal stack slot is noticeable but not nearly as extreme as you're imagining. And again, in most common cases we would absolutely be able to fold a bound statically and fall into the optimal path you're talking about. The critical guarantee, that the array does not get heap-allocated, is still absolutely intact.

Value equality would still affect the type-checker, but I think we could pretty easily just say that all bound expressions are assumed to potentially resolve unequally unless they are literals or references to the same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test for equality?

The Equatable protocol does guarantee reflexivity.

Moreover, as far as I know, Equatable is not recognized by the compiler in any way, so it's just a regular protocol.

That's not quite true: we synthesize Equatable instances in several places.

What would make it special? Some types would implement operator == to compare themselves to other types, that's beyond the scope of Equatable. What about those? And how are custom operator implementations going to serve this purpose at compile-time? Or will it just ignore the semantics of the type and reduce it to a sequence of bits? Or maybe only a few hand-picked types will be supported?

The seemingly simple generic value parameter concept gets vastly complicated and/or poorly designed without an elaborate compile-time execution system... Unless I'm missing an obvious way out.

The only thing the compiler really *needs* to know is whether two types are known to be the same, i.e. whether two values are known to be the same. An elaborate compile-time execution system would not be sufficient here, because again, Swift is not C or C++: we need to be able to answer that question even in generic code rather than relying on the ability to fold all computations statically. We do not want to add an algebraic solver to the type-checker. The obvious alternative is to simply be conservatively correct by treating independent complex expressions as always yielding different values.

The only hard constraint is that types need to be consistent, but that just means that we need to have a model in which bound expressions are evaluated exactly once at runtime (and of course typically folded at compile time).

What exactly would it take to be able to execute select piece of code at compile-time? Taking the AST, converting it to LLVM IR and feeding it to the MCJIT engine seems to be easy enough. But I'm pretty sure it's more tricky than that. Is there a special assumption or two made about the code that prevents this from happening?

We already have the ability to fold simple expressions in SIL; we would just make sure that could handle anything that we considered really important and allow everything else to be handled dynamically.

John.

···

On Jul 31, 2017, at 3:15 AM, Gor Gyolchanyan <gor.f.gyolchanyan@icloud.com> wrote:

On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:

John.

Or do you mean that the bounds are integer literals? (That's what I have in the design document now.)

Sent from my iPhone

On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:

On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution <swift-evolution@swift.org> wrote:
The “constexpr” facility from C++ allows users to define constants and functions that are determined and usable at compile-time, for compile-time constructs but still usable at run-time. The facility is a key step for value-based generic parameters (and fixed-size arrays if you don’t want to be stuck with integer literals for bounds). Can figuring out Swift’s story here be part of Swift 5?

Note that there's no particular reason that value-based generic parameters, including fixed-size arrays, actually need to be constant expressions in Swift.

John.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[xwu (Xiaodi Wu)]

Sure, and hence my point: suppose now `foo` is a function in the stdlib,
and the stdlib authors have annotated the function so that it is `func
foo(arr: fixed [Int])`. Then, any user who writes `var array = ...` could
benefit from a performance boost because the compiler will not longer have
to be pessimistic about copying in order to maintain COW semantics. This is
why I made an explicit analogy to the design proposed for ownership in
Swift, where end users don't have to understand it in order to benefit from
the feature because the functions they call can give sufficiently important
hints to help the compiler avoid unnecessary copying.

I don't think that you claimed that this should be a solution to the

fixed-size array problem, but just in case, also note that it's not. We
can't make any [Int] layout-compatible with C fixed-size arrays because the
length has to be encoded in the type (as it cannot be encoded in the data
itself).

I don't understand this point. Can you elaborate on what you mean here?
Why does it have to be layout-compatible?

Then it seems that I have stricter requirements for fixed-size arrays than
you do, and I'd be curious to hear what you want to get out of fixed-size
arrays. If we compare a hypothetical `fixed [int]` to a hypothetical
`FixedSizeArray<T, N>`, these are some of the things that matter to me
which `fixed` can't offer:

   - It doesn't allow you to specify the size of the array, it's just a
   promise that the array is some immutable size. For instance, a `fixed
   [CGFloat]` would be a terrible type to represent a vector, but a
   FixedSizeArray<CGFloat, 4> would be appropriate, at least for the backing
   storage. A raw tuple would be a poor choice because dynamically indexing
   into them is painful.

Shouldn't this be solved by improvements to tuples? It's a highly desired
feature (by me and at least a few others) to allow tuples to conform to
protocols, whether fixed-size arrays are added or not. I expect that
conforming homogeneous tuples to Collection and enabling subscripting is
simply a matter of time.

   - `fixed` is only useful when the compiler can determine the size of
   the array statically. This makes it mostly useless as a storage qualifier
   if you received the array as a parameter (*even* if you received a `fixed`
   array), because you know that it has a constant size but you don't know
   what that size is.
   - Therefore, using a fixed-size array as a generic parameter
   (crucially, such as `fixed [fixed [Int]]`) is unlikely to work.
   - Even if that semantic hurdle is overcome, we'd still have no idea
   how much memory to allocate for the outer array's buffer to make it work.

As John McCall has replied, the array's bounds don't need to be statically
known for fixed-size arrays to have benefits.

   - Even if `fixed [fixed [Int]]` could work, then each inner array
   could still have a different size, which is almost certainly not what you
   intend by nesting two fixed-size arrays.

That's fair, but why at that point wouldn't you make your own Matrix type
of fixed size, which uses an Array of fixed size as the underlying storage?

   - Layout compatibility is important if you want to use fixed-size
   arrays to replace the clunky tuples that currently represent fixed-size
   arrays in structs exported from C (which is probably my one single biggest
   motivation for fixed-size arrays). You can't achieve layout compatibility
   if the size is part of the data instead of part of the type.

For me, that's an anti-goal, as IMO tuples are the most sensible way of
bridging a lot of these fixed-size arrays from C. Quite simply, I'd argue
that the most idiomatic way to represent four CGFloat instances is
`(CGFloat, CGFloat, CGFloat, CGFloat)`. The solution to certain operations
being clunky with tuples is to improve the ergonomics of tuples. For
instance, if we need a shorthand to avoid typing all those `CGFloat`s, then
add one: `(4 * CGFloat)`. If we need subscripting, then add it.

Besides, attaching fixed-size array semantics to an inherently

variable-size Array is awkward. For `fixed` to be effective, it needs to
disable methods that change the size of the array, or warn that you're
using them. I don't like the cross-concern impact: now a keyword needs to
know about method implementations to restrict them. It also has to work
with extension methods on the Array type, and it shouldn't apply to just
mutating functions because mutations that don't change the length of the
array are fine.

The idea is that all facilities which would benefit from knowing that an
array is of a fixed count would opt into that benefit by indicating as
such. That is, all stdlib array methods that are guaranteed to preserve the
size of the array would be annotated as such. Again, by analogy to the
ownership manifesto's design where functions that take shared arguments
could be optimized on the basis of such annotation. The rest would fall out
naturally.

What would you use `fixed [Int]` for? Only as an optimization tool?

Yes.

···

On Mon, Jul 31, 2017 at 11:45 AM, Félix Cloutier <felixcloutier@icloud.com> wrote:

[Chris_Lattner]

If you’re claiming that there is no way to express the natural pointer size of the machine, then that’s factually incorrect. This will compute the target-specific size of %T (even if it is a pointer):
  getelementptr %T* null, i64 1

The problems major problems are with C though: sizeof(long) is itself a huge can of worms, consider LP64 vs LLP64 architectures, as one simple example. C has a preprocessor that exposes machine parameters too early, and many other problems exist. AFAIK, these issues aren’t exposed by Swift, except via the Builtin module and imported C APIs & types.

-Chris

···

On Aug 1, 2017, at 3:36 PM, Jean-Daniel via swift-evolution <swift-evolution@swift.org> wrote:

I disagree. Most of the standard library is in protocols and small generic wrapper structures, which are prime candidates for compile-time evaluation. Even if Foundation has to be dropped (which is actually not that big of a deal), I think adapting the standard library to be less architecturally dependent to make it accessible in compiletime wouldn’t be such a big problem considering that it is already largely driven by protocols.
I think the architectural differences are negligible largely due to LLVM IR, which is portable (even byte ordering is taken into account).

Why this long due myth subsist, LLVM IR is not portable. You can produce portable IR by limiting the scope of the input language, but it does not even provides a way to express sizeof(long) in a architecture independent way. And I don’t even mention the type layouts.

[Tino]

Imho this topic was much better than that other one ;-) — and I just realised that of metaprogramming build on top of reflection wasn't discussed in its own thread yet…
I fear "constexpr" is already burned, because people associate it with things like calculating Fibonacci numbers at compile time (which is kind of cool, but doesn't have much merit for most of us).

Right now, there is SE-185, which allows to synthesise Equatable and Hashable.
To do so, nearly 1500 lines of C++ are needed, and even if we assume that two thirds are comments and whitespace, it's still a big piece of code that could only be written by someone with deep knowledge about C++ and the Swift compiler.
Compile time metaprogramming could do the same, but in probably less than twenty lines of Swift that could be written rather easily by anyone who knows the language…

So to update my list of things that might be added, there are also some points that are already there and whose implementation could have been simplified drastically:

- Forwarding of protocol conformance (Kotlin, for example, has this: When a member conforms to a protocol, you don't have to write a bunch of methods that just say "let my member do this")
- init with reduced boilerplate
- Subtyping for non-class types, including a "newtype" option
- Property behaviours

- Equatable, Hashabable
- Encoding/Decoding

I still wonder that virtually no one else seems to be thrilled about the power of the idea… @gor.f.gyolchanyan would you like join an attempt to raise the attention?

- Tino

[xwu (Xiaodi Wu)]

>
>> The parameters for a fixed-size array type determine the type's
size/stride, so how could the bounds not be needed during compile-time? The
compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run
time. It already has to do that for generic types and types with resilient
members. That does, of course, have performance consequences, and those
performance consequences might be unacceptable to you; but the fact that we
can handle it means that we don't ultimately require a semantic concept of
a constant expression, except inasmuch as we want to allow users to
explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as
well use a regular parameter if there's no compile-time evaluation
involved. In that case, fixed-sized arrays will be useless, because they'll
be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why
*shouldn't* it get all the benefits of a fixed-sized array without having
to use a special syntax? Put another way, why shouldn't fixed-size be one
of those optional attributes for arrays, like ownership will be for
variables, that users can opt into for more performance but is otherwise
automatically worked out by the compiler?

Because the compiler shouldn't know what an array is (that is, what a
Swift array is, not the C array that LLVM has a primitive type for).

Wait, why not? It already does, afaict, just as it knows about integers,
strings, etc. In fact, as I understand it, one dividing line between the
standard library and Foundation is whether or not a type needs compiler
knowledge of it.

And if the compiler doesn't know what an array is and what resizing means,

then the array has to define that optimization hint itself, which will
either involve new syntax for attaching metadata for declarative
special-case solutions. Besides, it won't be getting the full benefits of
fixed-sized arrays. The biggest benefit of fixed-size arrays is allocation
cost (which is exactly zero for static fixed-sized arrays and a single
clock cycle for unescaped local fixed-sized arrays), which is not going to
improve if an array is simply forbidden from resizing.

I'm not speaking about under the hood; I'm speaking about spelling:

let foo: [Int] = [1, 2, 3] // this clearly needs no resizing
var bar: [Int] = [1, 2, 3] // not clear if this needs resizing
var baz: fixed [Int] = [1, 2, 3] // allow the user to tell the compiler

Under the hood, Swift optimizes as best it can.

For really fast storage, Clang has __builtin_alloca, which could be wrapped

···

On Mon, Jul 31, 2017 at 02:51 Gor Gyolchanyan <gor.f.gyolchanyan@icloud.com> wrote:

On Jul 31, 2017, at 10:40 AM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:
On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution < > swift-evolution@swift.org> wrote:

> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution < >> swift-evolution@swift.org> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:

in an __attribute__((__always_inline__)) function and exposed to Swift.
For semantically non-resizable arrays, we can simply implement a
FixedSizeArray with no changes to the language.
Considering these options, I don't think solving the use case of
fixed-size arrays is worth complicating the language.
This is why I'm pushing for elaborate compile-time execution feature. It
won't solve a specific use-case, it will solve a very wide domain of use
cases.

As far as I know, the pinnacle of uses for fixed-size arrays is having a

compile-time pre-allocated space of the necessary size (either literally at
compile-time if that's a static variable, or added to the pre-computed
offset of the stack pointer in case of a local variable).

> Value equality would still affect the type-checker, but I think we
could pretty easily just say that all bound expressions are assumed to
potentially resolve unequally unless they are literals or references to the
same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test
for equality? Moreover, as far as I know, Equatable is not recognized by
the compiler in any way, so it's just a regular protocol. What would make
it special? Some types would implement operator == to compare themselves to
other types, that's beyond the scope of Equatable. What about those? And
how are custom operator implementations going to serve this purpose at
compile-time? Or will it just ignore the semantics of the type and reduce
it to a sequence of bits? Or maybe only a few hand-picked types will be
supported?

The seemingly simple generic value parameter concept gets vastly
complicated and/or poorly designed without an elaborate compile-time
execution system... Unless I'm missing an obvious way out.

> The only hard constraint is that types need to be consistent, but that
just means that we need to have a model in which bound expressions are
evaluated exactly once at runtime (and of course typically folded at
compile time).

What exactly would it take to be able to execute select piece of code at
compile-time? Taking the AST, converting it to LLVM IR and feeding it to
the MCJIT engine seems to be easy enough. But I'm pretty sure it's more
tricky than that. Is there a special assumption or two made about the code
that prevents this from happening?

> John.
>
>> Or do you mean that the bounds are integer literals? (That's what I
have in the design document now.)
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:
>>
>>>> On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution < >> swift-evolution@swift.org> wrote:
>>>> The “constexpr” facility from C++ allows users to define constants
and functions that are determined and usable at compile-time, for
compile-time constructs but still usable at run-time. The facility is a key
step for value-based generic parameters (and fixed-size arrays if you don’t
want to be stuck with integer literals for bounds). Can figuring out
Swift’s story here be part of Swift 5?
>>>
>>> Note that there's no particular reason that value-based generic
parameters, including fixed-size arrays, actually need to be constant
expressions in Swift.
>>>
>>> John.
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Gor_Gyolchanyan]

>
>> The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.
>
> Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Doesn't this defeat the purpose of generic value parameters? We might as well use a regular parameter if there's no compile-time evaluation involved. In that case, fixed-sized arrays will be useless, because they'll be normal arrays with resizing disabled.

OTOH, if the compiler can prove that a local array is never resized, why *shouldn't* it get all the benefits of a fixed-sized array without having to use a special syntax? Put another way, why shouldn't fixed-size be one of those optional attributes for arrays, like ownership will be for variables, that users can opt into for more performance but is otherwise automatically worked out by the compiler?

Because the compiler shouldn't know what an array is (that is, what a Swift array is, not the C array that LLVM has a primitive type for).

Wait, why not? It already does, afaict, just as it knows about integers, strings, etc. In fact, as I understand it, one dividing line between the standard library and Foundation is whether or not a type needs compiler knowledge of it.

And if the compiler doesn't know what an array is and what resizing means, then the array has to define that optimization hint itself, which will either involve new syntax for attaching metadata for declarative special-case solutions. Besides, it won't be getting the full benefits of fixed-sized arrays. The biggest benefit of fixed-size arrays is allocation cost (which is exactly zero for static fixed-sized arrays and a single clock cycle for unescaped local fixed-sized arrays), which is not going to improve if an array is simply forbidden from resizing.

I'm not speaking about under the hood; I'm speaking about spelling:

let foo: [Int] = [1, 2, 3] // this clearly needs no resizing
var bar: [Int] = [1, 2, 3] // not clear if this needs resizing
var baz: fixed [Int] = [1, 2, 3] // allow the user to tell the compiler

If a simple compiler hint is what you mean, then I'd go for something less array-centric and more along the lines of "arbitrary compiler hint", much like how C preprocessor has pragmas.
I'd go for this syntax instead:

var baz: @hint(fixed) Int = [1, 2, 3]

Which has the added benefit of prepping the general syntax for any other part of the language where the compiler could use a little hand-holding from the human.
I didn't want it to be simply @fixed, because the singular attributes so far follow the pattern of being mandatory, rather than optional hints, so that we don't confuse people by having at attribute that may or may not actually do anything and reserve that to a single very specific attribute.

But this is all outside of the topic.

Within the topic there were two major points made regarding compile-time evaluation:
* It's not warranted to consider this unless actual use cases are brought forth.
* Many of those use cases can be solved at run-time (e.g. generic value parameters).

My counter-point to those are:
* It's always possible to do things at run-time. I think it isn't worth spending time implementing a feature and complicating the language if it won't provide enough performance and convenience gains.
* Many of the features discussed here (including generic value parameters and fixed-size arrays) would be much simpler, more convenient, more performant and more powerful if the underlying deal-breaking lack of compile-time facilities was remedied.

···

On Jul 31, 2017, at 11:09 AM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:
On Mon, Jul 31, 2017 at 02:51 Gor Gyolchanyan <gor.f.gyolchanyan@icloud.com <mailto:gor.f.gyolchanyan@icloud.com>> wrote:

On Jul 31, 2017, at 10:40 AM, Xiaodi Wu <xiaodi.wu@gmail.com <mailto:xiaodi.wu@gmail.com>> wrote:
On Mon, Jul 31, 2017 at 02:15 Gor Gyolchanyan via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> On Jul 31, 2017, at 7:10 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>> On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com <mailto:darylew@mac.com>> wrote:

Under the hood, Swift optimizes as best it can.

For really fast storage, Clang has __builtin_alloca, which could be wrapped in an __attribute__((__always_inline__)) function and exposed to Swift.
For semantically non-resizable arrays, we can simply implement a FixedSizeArray with no changes to the language.
Considering these options, I don't think solving the use case of fixed-size arrays is worth complicating the language.
This is why I'm pushing for elaborate compile-time execution feature. It won't solve a specific use-case, it will solve a very wide domain of use cases.

As far as I know, the pinnacle of uses for fixed-size arrays is having a compile-time pre-allocated space of the necessary size (either literally at compile-time if that's a static variable, or added to the pre-computed offset of the stack pointer in case of a local variable).

> Value equality would still affect the type-checker, but I think we could pretty easily just say that all bound expressions are assumed to potentially resolve unequally unless they are literals or references to the same 'let' constant.

Shouldn't the type-checker use the Equatable protocol conformance to test for equality? Moreover, as far as I know, Equatable is not recognized by the compiler in any way, so it's just a regular protocol. What would make it special? Some types would implement operator == to compare themselves to other types, that's beyond the scope of Equatable. What about those? And how are custom operator implementations going to serve this purpose at compile-time? Or will it just ignore the semantics of the type and reduce it to a sequence of bits? Or maybe only a few hand-picked types will be supported?

The seemingly simple generic value parameter concept gets vastly complicated and/or poorly designed without an elaborate compile-time execution system... Unless I'm missing an obvious way out.

> The only hard constraint is that types need to be consistent, but that just means that we need to have a model in which bound expressions are evaluated exactly once at runtime (and of course typically folded at compile time).

What exactly would it take to be able to execute select piece of code at compile-time? Taking the AST, converting it to LLVM IR and feeding it to the MCJIT engine seems to be easy enough. But I'm pretty sure it's more tricky than that. Is there a special assumption or two made about the code that prevents this from happening?

> John.
>
>> Or do you mean that the bounds are integer literals? (That's what I have in the design document now.)
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:
>>
>>>> On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>>>> The “constexpr” facility from C++ allows users to define constants and functions that are determined and usable at compile-time, for compile-time constructs but still usable at run-time. The facility is a key step for value-based generic parameters (and fixed-size arrays if you don’t want to be stuck with integer literals for bounds). Can figuring out Swift’s story here be part of Swift 5?
>>>
>>> Note that there's no particular reason that value-based generic parameters, including fixed-size arrays, actually need to be constant expressions in Swift.
>>>
>>> John.
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org <mailto:swift-evolution@swift.org>
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[Gor_Gyolchanyan]

OTOH, if the compiler can prove that a local array is never resized, why *shouldn't* it get all the benefits of a fixed-sized array without having to use a special syntax? Put another way, why shouldn't fixed-size be one of those optional attributes for arrays, like ownership will be for variables, that users can opt into for more performance but is otherwise automatically worked out by the compiler?

I think it's a good idea in general, but I don't think I'd notice if this feature would exist, so just to save some work for the compiler (and Core), I'd prefer to first get the feature itself, and improve it later.
So from my high-level perspective, I think the order should be
0) Add support for value literals as generic parameters

Value literals would be limited to the ones accepted by ExpressibleBy*Literal, which would be very limiting for anything that isn't an Int or a Float. This might solve the specific case of fixed-size arrays, but otherwise would be nearly useless.

···

On Jul 31, 2017, at 12:16 PM, Tino Heth <2th@gmx.de> wrote:

1) Add Array<T, size: UInt>
2) Add support for generic parameters that are evaluated at compile time
3) Make everything fast ;-)
n) Add useful additions like detecting arrays that aren't declared as fixed length, but could

I think the alternative to macros (don't care much how hygienic they are ;-) could be even more beneficial, but it's a topic that stands on its own (not sure wether this thread is more for fixed size arrays or metaprogramming… ;-)

[Felix_Cloutier1 (Félix Cloutier)]

No, this is a misunderstanding: I mean "you" as yourself, not "you" as the compiler. As far as I know, given a type variable, as in `let foo: NSObject.Type = NSArray.self`, you (the developer) can't use `foo` to instantiate a generic type.

The question stands again: are there any specific reasons that you can't do that right now, or is it just because the need for this is not particularly evident?

Félix

···

Le 31 juil. 2017 à 11:29, John McCall <rjmccall@apple.com> a écrit :

On Jul 31, 2017, at 3:58 AM, Félix Cloutier <felixcca@yahoo.ca> wrote:
It seems to me that this applies just the same to type generic parameters. Are there any reason that you can't instantiate a generic type at runtime, or is it just because the need is not as evident as it would/could be with non-type generic parameters?

Are you under the impression that Swift does not, today, instantiate generic types at runtime?

A lot of this entire thread seems to be premised on really weird, incorrect assumptions about the Swift compilation model.

John.

Le 30 juil. 2017 à 21:10, John McCall via swift-evolution <swift-evolution@swift.org> a écrit :

On Jul 30, 2017, at 11:43 PM, Daryle Walker <darylew@mac.com> wrote:
The parameters for a fixed-size array type determine the type's size/stride, so how could the bounds not be needed during compile-time? The compiler can't layout objects otherwise.

Swift is not C; it is perfectly capable of laying out objects at run time. It already has to do that for generic types and types with resilient members. That does, of course, have performance consequences, and those performance consequences might be unacceptable to you; but the fact that we can handle it means that we don't ultimately require a semantic concept of a constant expression, except inasmuch as we want to allow users to explicitly request guarantees about static layout.

Value equality would still affect the type-checker, but I think we could pretty easily just say that all bound expressions are assumed to potentially resolve unequally unless they are literals or references to the same 'let' constant.

The only hard constraint is that types need to be consistent, but that just means that we need to have a model in which bound expressions are evaluated exactly once at runtime (and of course typically folded at compile time).

John.

Or do you mean that the bounds are integer literals? (That's what I have in the design document now.)

Sent from my iPhone

On Jul 30, 2017, at 8:51 PM, John McCall <rjmccall@apple.com> wrote:

On Jul 29, 2017, at 7:01 PM, Daryle Walker via swift-evolution <swift-evolution@swift.org> wrote:
The “constexpr” facility from C++ allows users to define constants and functions that are determined and usable at compile-time, for compile-time constructs but still usable at run-time. The facility is a key step for value-based generic parameters (and fixed-size arrays if you don’t want to be stuck with integer literals for bounds). Can figuring out Swift’s story here be part of Swift 5?

Note that there's no particular reason that value-based generic parameters, including fixed-size arrays, actually need to be constant expressions in Swift.

John.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[fclout]

It doesn't allow you to specify the size of the array, it's just a promise that the array is some immutable size. For instance, a `fixed [CGFloat]` would be a terrible type to represent a vector, but a FixedSizeArray<CGFloat, 4> would be appropriate, at least for the backing storage. A raw tuple would be a poor choice because dynamically indexing into them is painful.

Shouldn't this be solved by improvements to tuples? It's a highly desired feature (by me and at least a few others) to allow tuples to conform to protocols, whether fixed-size arrays are added or not. I expect that conforming homogeneous tuples to Collection and enabling subscripting is simply a matter of time.

Of course, if this happens, the point is moot and we have the type of fixed-size arrays that I've been asking for. However, crystal-balling Chris's last comment on the matter, it looks like it might not be happening too soon <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170724/038225.html&gt;\.

`fixed` is only useful when the compiler can determine the size of the array statically. This makes it mostly useless as a storage qualifier if you received the array as a parameter (*even* if you received a `fixed` array), because you know that it has a constant size but you don't know what that size is.
Therefore, using a fixed-size array as a generic parameter (crucially, such as `fixed [fixed [Int]]`) is unlikely to work.
Even if that semantic hurdle is overcome, we'd still have no idea how much memory to allocate for the outer array's buffer to make it work.

As John McCall has replied, the array's bounds don't need to be statically known for fixed-size arrays to have benefits.

This (partially) applies to the first point, but it leaves a lot of holes. The size does not need to be statically known, but you need to know how much memory you're going to need ahead of allocating it. How much memory do you need for this?

var foo = fixed [Int]()
for i in 0..<param {
  foo.append(i)
}

Arrays can't work if elements don't have a fixed size. How big is an element in this example?

var foo = fixed [fixed [Int]]()
foo.append([1])
foo.append([2, 3])

Where do you allocate this array's storage, and how big is it?

struct Foo {
  var array: fixed [Int]
}

var foo = Foo()
foo.array.append(4)

In the general case, this makes `fixed` meaningful as a type annotation for parameters, but instantiating such an object in the first place would require you to give up a lot of flexibility in how it's populated. The unconstrained problem of finding how many elements you'll have in an array is uncomputable.

You could say that a fixed-size array stays as big as it was when it was instantiated, but this still causes problems with fixed-size arrays in structures (storage has already been allocated when the initializer runs), and with generic collections (you couldn't initialize a variable-sized array of fixed-size arrays, for instance).

Even if `fixed [fixed [Int]]` could work, then each inner array could still have a different size, which is almost certainly not what you intend by nesting two fixed-size arrays.

That's fair, but why at that point wouldn't you make your own Matrix type of fixed size, which uses an Array of fixed size as the underlying storage?

I'd do it if I needed to, but there are plenty of 2D arrays that are just that and don't need their own wrapper type to function, and I'd be happier to not do it.

Layout compatibility is important if you want to use fixed-size arrays to replace the clunky tuples that currently represent fixed-size arrays in structs exported from C (which is probably my one single biggest motivation for fixed-size arrays). You can't achieve layout compatibility if the size is part of the data instead of part of the type.

For me, that's an anti-goal, as IMO tuples are the most sensible way of bridging a lot of these fixed-size arrays from C. Quite simply, I'd argue that the most idiomatic way to represent four CGFloat instances is `(CGFloat, CGFloat, CGFloat, CGFloat)`. The solution to certain operations being clunky with tuples is to improve the ergonomics of tuples. For instance, if we need a shorthand to avoid typing all those `CGFloat`s, then add one: `(4 * CGFloat)`. If we need subscripting, then add it.

I think that any template-based fixed-size array structure would still need that kind of (4 * CGFloat) syntax to be implemented, or some other facility to allocate a parameterized amount of automatic storage. With that said, if we can implement fixed-size arrays with just that without having to wait for the restrictions on anonymous types to be lifted, to me, it's a fully acceptable solution.

Besides, attaching fixed-size array semantics to an inherently variable-size Array is awkward. For `fixed` to be effective, it needs to disable methods that change the size of the array, or warn that you're using them. I don't like the cross-concern impact: now a keyword needs to know about method implementations to restrict them. It also has to work with extension methods on the Array type, and it shouldn't apply to just mutating functions because mutations that don't change the length of the array are fine.

The idea is that all facilities which would benefit from knowing that an array is of a fixed count would opt into that benefit by indicating as such. That is, all stdlib array methods that are guaranteed to preserve the size of the array would be annotated as such. Again, by analogy to the ownership manifesto's design where functions that take shared arguments could be optimized on the basis of such annotation. The rest would fall out naturally.

There are more problems to this that you may be able to iron out but that make the story more complex still. How do you annotate MutableCollection-based algorithms, or any other protocol-based extension that doesn't directly accept an array? What's the syntax of an Array extension method, which doesn't have an explicit self parameter, to mean that the function is mutating but doesn't change the array's bounds?

···

Le 31 juil. 2017 à 18:54, Xiaodi Wu <xiaodi.wu@gmail.com> a écrit :