associated objects


(Ted van Gaalen) #1

What it does is allow developers to extend the language to do things that it doesn't support.

Hitting boundaries... or so it seems... This is very interesting.
In spite of its very modern approach, wether we like it or not, Swift is
still a conventional Hard Coded Statical Programming Language,
(here further in this text referred to as HCSPL)
which means that, like in C++ or C# etc. its language
elements are essentially predefined, "fixed", and that
it is therefore a daunting task to make the language as
versatile as possible and to satisfy everyone's requirements,
We can see that every day in swift-evolution and for that matter
all other HCSPL (C#, C++, Python, ObjC etc) related forums.

HCSPL are in a sense a traditional compromise to "The Machine”
and enables us -within the limitations of our current state of main
stream technology- to make highly optimised fast running compiled applications.
That we know is the virtue of hard coded static languages.
"The Machine" however gets more advanced and faster day by day
thereby diminishing the need for HCSPL in its course.

However, (as far as I know) , due to the nature of HCSPL, like Swift,
there is no (direct and supported and convenient) way supporting
meta-programming facilities, e.g. to generate define objects (classes),
and its instances on the fly, that is, at run time.

In most cases, one can live with this HCSPL "compromise", especially with Swift,
which is very flexible and offers many convenient ways to solve a wide spectrum
of programming challenges. However, one could think of many applications like
in AI where applications can improve and extend themselves, learning to adjust
and adapt to their environment, thereby altering and tuning its performance,
like our brains do...naturally.
In contrast, a HCSPL forces one to match one's problem solving to the language
elements that are available in the HCSPL. In a sense it's like having a big box
with Lego bricks, that is predefined discrete elements, which allow one to build things
with it, however it still remains limited as Lego.
This is not the case with a dynamic language like Smalltalk.

Since ca 1980 I have a high interest in Smalltalk (and other OOP dynamic
programming languages too). Some would argue that Smalltalk is a
programming language, but take a closer look, (e.g. with Pharo).
It is just a relatively small set of mainly syntactic rules defining the
environment wherein the Smalltalk system lives (almost literally) as a
dynamic object hierarchy, were everything is an object, which as we know
is a completely different approach compared to HCSPLs.

Before Swift had arrived, my hope was that the successor of Objective C
would be Smalltalk, or at least Smalltalk as a breeding environment for
developing macOS, iOS, tvOS and watchOS applications. In this role it would then
be desirable that the apps thus produced are "closed apps" so to speak, that is that
the development environment is not available to the end-user.
The thought of a e.g. "new Xcode/Smalltalk" being the next Apple development
environment seemed reasonable, also because ObjectiveC had features
inspired by Smalltalk. (as all OOP languages have)
Alas, it didn't happen..but I keep dreaming...

Perhaps out of context, one then might ask how to interface that
Xcode/Smalltalk to all those existing libraries (Cocoa UIKit etc.)
This can be done by letting Smalltalk crawl through all API definitions,
generating its own interface classes dynamically. This sounds ambitious
but it can be done. Something a HCSPL like Swift could never do as it is.

And now we don't have Smalltalk but instead Swift, not bad at all, currently
I find it the most advanced HCSPL now available.

However, my verbose text here could be food for thoughts:

Is it possible to have best of (these completely different) both worlds?
Would it be possible in Swift to have facilities to generate objects
dynamically at runtime? and, if desirable, how can such be implemented?

Kind Regards
TedvG

Robert,

What it does is allow developers to extend the language to do things that it doesn't support. Associated Objects is just a flexible way to allow developers to do that, and that's how I used it in Objective-C, so that's what I thought of in Swift when I found that "I want to do <x> but the language doesn't support it". Maybe there's a completely different way to achieve the same thing - but I just went with what I know.

"I want to do <x> but the language doesn't support it” is the whole reason for this list! I want to know what “<x>” is in order to wrap my head around your proposal better. I’m not here to invalidate your work with a slew of criticisms, I’m just trying to understand your proposal by probing you about it.

So it's difficult to come up with concrete examples of "things a developer might want to do but the language doesn't support" because almost by definition they are unforeseen. I can only enumerate the things I have wanted to do in Obj-C and Swift and how I got around it.

- Dynamically add properties to a class (obc-c) / implement stored properties (swift).

- Add per-instance methods at run-time.

- Perform a function when some other object is deallocated (haven't solved this in Swift yet, but in obj-c associated object deallocation is well-defined so I used that).

- Other unforeseen things…

Again, these are features and I want motivations behind them. Why should you be able to dynamically add stored properties and methods to any Swift class? Why should you be able to observe deallocation if deallocation calls are not guaranteed (remember, Objective-C and Swift do not guarantee -dealloc/deinit calls in any order at any time - please don’t assume an RAII-like model because it will lead to memory leaks and malformed code).

I will say: Perhaps if you’re having trouble motivating the inclusion of this feature, you may want to step back and reevaluate its place in the language and why you wanted to write this library in the first place. Often times, I find that really helps get the creative juices flowing. Or, if I can’t come up with anything, it means I go back to step 1 and start over. Proposals can be as much a learning process for us as it is for you.

So maybe Associated Objects isn't the answer and I should have stated the problem better, instead of jumping to what I thought the answer might be... the problem I want to solve is this:

As a developer I want to do <x> but the language doesn't support it... what helpful thing *can* I use right now that allows me to achieve this? I accept the disclaimer by ticking this box:

[ ] Yes, I understand that I'm not using Swift anymore, but some custom run-time thing that I'm building myself on top of Swift, so if I want type safety I have to implement it, if I want copy-on-write or other optimisations, I have to implement it, and I understand that performance might not be the best too.

So from what I’ve gathered you’ve written a library to do this yourself. You have a vision for how you want to use this library. You think that vision is compatible with a language-level change. From my perspective, your job now is to articulate that vision and motivate it to this list. I don’t think associated objects and the other dynamic features you mention are fundamentally incompatible with Swift, I just think the design that I’ve seen so far may not be the best way of going about it and I’m voicing my concerns as much.

If Swift can provide something to help developers go beyond the abilities the current version, isn't that a good idea?

No one disputes this. We just want to see your rationale.

···

On Sep 30, 2016, at 9:40 AM, Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:


(Chris Lattner) #2

Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a false dichotomy. Swift can definitely have both awesome dynamic features while still having performance, predictability and safety.

Would it be possible in Swift to have facilities to generate objects
dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:

I’m sure it isn’t ideal, but it proves that it can be done. When we have bandwidth to reevaluate this area from first principles, I’m sure we can make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity, but for that simplicity it makes many tradeoffs that we’re not willing to make. We are willing to make the internal implementation of Swift complex if that means that we get a beautiful model for programmers - one that preserves the virtues of safety-by-default, predictability, performance, and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is tired, and also wildly inaccurate.

-Chris

···

On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution <swift-evolution@swift.org> wrote:


(Jay) #3

Are stored properties in extensions already being discussed elsewhere? Is
this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can only
think of two ways:

1. An extra pointer per instance (with resulting ABI compatability
implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift
implementation, so there may be other/better ways - I'd like to know if
there are?

If not, and option 2 was employed, a little foresight might enable the
mechanism to be overloaded in the future for other dynamic features too. A
bit flag (I'm hoping there's a spare bit in an existing flags field
somewhere?) could indicate whether any feature had caused the object to be
added to this lookup and deinit could check this bit and make sure the
object is removed, thus any stored properties are nilled. The lookup value
could be a struct with one member (extensionStoredProperties), and
additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps
there are much better, more optimal, more ingenious, more unknown-by-me
ways of doing such things, so apologies if this whole idea is way-off the
mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to
implement their own dynamic features in Swift.. so such desires could be
satisfied in the short term until they could be done properly in the
language.

···

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:

On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution < > swift-evolution@swift.org> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a
false dichotomy. Swift can definitely have both awesome dynamic features
while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have
bandwidth to reevaluate this area from first principles, I’m sure we can
make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity,
but for that simplicity it makes many tradeoffs that we’re not willing to
make. We are willing to make the internal implementation of Swift complex
if that means that we get a beautiful model for programmers - one that
preserves the virtues of safety-by-default, predictability, performance,
and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is
tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Ted van Gaalen) #4

Hi Chris
I am a bit slow to respond, please read inline, thanks.

Is it possible to have best of (these completely different) both worlds?

Yes, of course it is.

Your email spends a lot of words

Does it?

trying to form a false dichotomy.

Well, sorry, thank you. but No, I don’t think so:
I am merely describing the advantages and
disadvantages of static programming languages.
It doesn’t mean it should be one or the other, just observing.
Just thinking about the future in years, not in weeks.
Even so, I still like Swift.

Swift can definitely have both awesome dynamic features while still having performance, predictability and safety.

Yes, I take it that it is possible to implement these.
Currently it is not possible to do meta programming in Swift, that is, create
arbitrary functions, properties and new classes
during run time. hypothetical along the lines as if it were so:

    // Warning! hypothetical as yet not compilable coding follows:

    let Monolith: Class?
    
    var classdefstring = "parentclass: SpaceObject;" +
            "var l = 9.0; var h = 4.0; var w = 1.0; " +
            "var color = " + defaultMonolithColorStr +
            ";func volume() -> Double { return l * w * h} “ +
            aStringWithAdditionalComponentDefinitions

    func defMonolithClass()
    {
        do // try to define a class, NOT an instance.
        {
            try Monolith = Meta.defineClass(classdefstring)
        }
        catch Meta.classDefinitionError
        {
            // handle a possible class definition error
            // e.g. syntax errors. symbols not found etc.
        }
        catch Meta.creationConflict
        {
            // class cannot be defined or redefined when instances of Monolith exist.
            // necessary?
        }

        do // dynamically add components to the Monolith
        {
            try Meta.addPropertyTo( Monolith,"mass", 3.4)
            try Meta.addPropertyTo( Monolith,"physics", PhysicsBody)
            try Meta.addFunctionTo( Monolith, someFunctionDefinitionString)
        }
        catch Meta.ClassExtenderError
        {
            // handle the error
        }
        catch Meta.creationConflict
        {
            // class definition cannot be altered when instances of Monolith exist.
            // necessary?
        }
    }
    
    defMonolithClass()
    
    // Make an instance of the dynamically generated class
    public let mn = Monolith()
    
    let vol = mn.volume
    
    // One can remove after usage, normally ARC should take care of that.
    // unless one would wish to reinstate/redefine Monolith again.
    mn = nil // remove instance

    Monolith = nil // should throw an error when instances of Monolith are still around.

    defMonolithClass() // again, possibly with other definition strings

    // Of course ARC should remove dynamic class defs automagically when no longer
    // needed.
    
There could be overloads of defining classes etc. using other methods than string wise declaration..
Of course, Meta… for defining structs and other components could be made in a similar fashion.
I’d guess that it might be very hard and complicated to implement meta creation similar to the above?
Probably needed the app to interact many times with a compiler present on the target OS/device? -
thereby making small pluggable “bricks”
Just brainstorming here: I know nearly nothing of compiler construction.

Would it be possible in Swift to have facilities to generate objects
dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

Very complex code (to me, hard to read) . As far as I can see, it’s a clever hack but vulnerable because
depending completely on the underlying memory occupation and layout of an existing class definition?
Totally new classes/structs etc. cannot be defined with it?
(like in the above hypothetical class def example)

I’m sure it isn’t ideal, but it proves that it can be done. When we have bandwidth to reevaluate this area from first principles, I’m sure we can make improvements on it.

Yeah, I guess it can be done, that would be great!

I will grant you that Smalltalk is a beautiful language in its simplicity, but for that simplicity it makes many tradeoffs that we’re not willing to make.

The only tradeoffs I can think of so far are:

  - Run time speed. in spite of precompiled methods etc Smalltalk is not so fast as compiled statical
programming language executables. However, the VMs are getting much better/ faster
these days. Dedicated hardware (processors) would really help too.

- Total freedom, but to me personally this is not a tradeoff at all…

- what else?

We are willing to make the internal implementation of Swift complex if that means that we get a beautiful model for programmers -
one that preserves the virtues of safety-by-default,

Safety by default? to some extend, yes. However, my experience with many programming
languages is, that it almost doesn’t matter how much effort is given
to make a PL save. I am still traumatised by many [some words removed here]
source code through all the years I had to work with
(of course only others are writing baaaaaaad code, naturally, I myself am perfect :o)

predictability, performance, and joy-to-develop-in.

Yes, I enjoy working with Swift very much as well, thank you (and others) for creating this language.

The meme of “Swift can never (or will never) support dynamic features” is tired, and also wildly inaccurate.

That was a bit overcharged from me. I had that impression, being not too broadly informed in this meta stuff.
Met vriendelijke groeten, kind regards, one of your colleagues/friends from another time :o)
TedvG

···

On 1 Oct 2016, at 01:47, Chris Lattner <clattner@apple.com> wrote:
On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution <swift-evolution@swift.org> wrote:

-Chris


(Jay) #5

I have been thinking further on this and in addition to my previous two
thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is
currently always zero), this could be used to signify the presence of an
additional struct that immediately follows after the existing object data.
I *think* that this method would allow binary compatibility with older
modules. Instances that have this bit set would allow stored properties in
extensions. The struct at the end would have one member, a pointer to a
table of additional objects/values, stored properties defined in extensions
could be stored in here, using a hash derived from the
module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with
additional features, for example a list of deinit hooks, dynamically added
methods, etc. The struct itself may also be dynamic in size/layout if such
complexity is warranted by size or extensibility concerns. Perhaps it
should start with some flags and its size (size would be fixed and only
increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than
my previous two ideas. It doesn't require global lookups or additional
levels of indirection beyond accessing the dynamic data/feature itself.

···

On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk> wrote:

Are stored properties in extensions already being discussed elsewhere? Is
this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can
only think of two ways:

1. An extra pointer per instance (with resulting ABI compatability
implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift
implementation, so there may be other/better ways - I'd like to know if
there are?

If not, and option 2 was employed, a little foresight might enable the
mechanism to be overloaded in the future for other dynamic features too. A
bit flag (I'm hoping there's a spare bit in an existing flags field
somewhere?) could indicate whether any feature had caused the object to be
added to this lookup and deinit could check this bit and make sure the
object is removed, thus any stored properties are nilled. The lookup value
could be a struct with one member (extensionStoredProperties), and
additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps
there are much better, more optimal, more ingenious, more unknown-by-me
ways of doing such things, so apologies if this whole idea is way-off the
mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to
implement their own dynamic features in Swift.. so such desires could be
satisfied in the short term until they could be done properly in the
language.

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution < > swift-evolution@swift.org> wrote:

On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution < > swift-evolution@swift.org> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a
false dichotomy. Swift can definitely have both awesome dynamic features
while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have
bandwidth to reevaluate this area from first principles, I’m sure we can
make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity,
but for that simplicity it makes many tradeoffs that we’re not willing to
make. We are willing to make the internal implementation of Swift complex
if that means that we get a beautiful model for programmers - one that
preserves the virtues of safety-by-default, predictability, performance,
and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is
tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charlie Monroe) #6

There is a 4th way.

Introduce an internal protocol Associatable, which would tell the compiler to add an additional (hidden) field to the object which would include the "dictionary" of key -> value associated values. (It would be off-limits to extensions, of course).

This way:

- it won't be a single dictionary containing all the associated values
- classes can opt-in to this
- the dictionary will be per-instance

This is a midway between the current implementation of ObjC associated objects and of what someone has suggested to have an extra space for each object for the AO...

···

On Oct 9, 2016, at 9:47 PM, Jay Abbott via swift-evolution <swift-evolution@swift.org> wrote:

I have been thinking further on this and in addition to my previous two thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is currently always zero), this could be used to signify the presence of an additional struct that immediately follows after the existing object data. I *think* that this method would allow binary compatibility with older modules. Instances that have this bit set would allow stored properties in extensions. The struct at the end would have one member, a pointer to a table of additional objects/values, stored properties defined in extensions could be stored in here, using a hash derived from the module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with additional features, for example a list of deinit hooks, dynamically added methods, etc. The struct itself may also be dynamic in size/layout if such complexity is warranted by size or extensibility concerns. Perhaps it should start with some flags and its size (size would be fixed and only increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than my previous two ideas. It doesn't require global lookups or additional levels of indirection beyond accessing the dynamic data/feature itself.

On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:
Are stored properties in extensions already being discussed elsewhere? Is this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can only think of two ways:

1. An extra pointer per instance (with resulting ABI compatability implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift implementation, so there may be other/better ways - I'd like to know if there are?

If not, and option 2 was employed, a little foresight might enable the mechanism to be overloaded in the future for other dynamic features too. A bit flag (I'm hoping there's a spare bit in an existing flags field somewhere?) could indicate whether any feature had caused the object to be added to this lookup and deinit could check this bit and make sure the object is removed, thus any stored properties are nilled. The lookup value could be a struct with one member (extensionStoredProperties), and additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps there are much better, more optimal, more ingenious, more unknown-by-me ways of doing such things, so apologies if this whole idea is way-off the mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to implement their own dynamic features in Swift.. so such desires could be satisfied in the short term until they could be done properly in the language.

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a false dichotomy. Swift can definitely have both awesome dynamic features while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have bandwidth to reevaluate this area from first principles, I’m sure we can make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity, but for that simplicity it makes many tradeoffs that we’re not willing to make. We are willing to make the internal implementation of Swift complex if that means that we get a beautiful model for programmers - one that preserves the virtues of safety-by-default, predictability, performance, and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charles Srstka) #7

How about a 5th way?

5. Let extensions introduce stored properties, but only in the same module as the type’s definition. Then, the compiler can just take any extensions into consideration when it’s determining the size of the type, just as if the properties had been declared in the type. Declaring stored properties on an extension outside of the type’s module results in a compiler error, exactly as today. This would, without any performance drawbacks, solve one of the big problems that people are hoping to solve via stored properties in extensions—the ability to organize members by protocol conformance. So, instead of this:

protocol P {
  var foo: String { get }
  func bar()
}

protocol Q {
  var baz: Int { get }
  func qux()
}

class C: P {
  var foo: String // <- what is this doing here?
  var baz: Int // <- ditto
}

extension C: P {
  func bar() {}
}

extension C: Q {
  func qux() {}
}

we could simply:

protocol P {
  var foo: String { get }
  func bar()
}

class C: P {}

extension C: P {
  var foo: String
  func bar() {}
}

extension C: Q {
  var baz: Int
  func qux() {}
}

So much cleaner, and since the compiler will simply turn the latter into the former at compile time, works without sacrificing anything in terms of performance.

Also, if we really decide that we need to have stored properties on *all* extensions, this idea still leaves the door open for using one of the methods discussed in this thread, in the case where the extension is in a different module from the original type.

Charles

···

On Oct 9, 2016, at 2:53 PM, Charlie Monroe via swift-evolution <swift-evolution@swift.org> wrote:

There is a 4th way.

Introduce an internal protocol Associatable, which would tell the compiler to add an additional (hidden) field to the object which would include the "dictionary" of key -> value associated values. (It would be off-limits to extensions, of course).

This way:

- it won't be a single dictionary containing all the associated values
- classes can opt-in to this
- the dictionary will be per-instance

This is a midway between the current implementation of ObjC associated objects and of what someone has suggested to have an extra space for each object for the AO...

On Oct 9, 2016, at 9:47 PM, Jay Abbott via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I have been thinking further on this and in addition to my previous two thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is currently always zero), this could be used to signify the presence of an additional struct that immediately follows after the existing object data. I *think* that this method would allow binary compatibility with older modules. Instances that have this bit set would allow stored properties in extensions. The struct at the end would have one member, a pointer to a table of additional objects/values, stored properties defined in extensions could be stored in here, using a hash derived from the module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with additional features, for example a list of deinit hooks, dynamically added methods, etc. The struct itself may also be dynamic in size/layout if such complexity is warranted by size or extensibility concerns. Perhaps it should start with some flags and its size (size would be fixed and only increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than my previous two ideas. It doesn't require global lookups or additional levels of indirection beyond accessing the dynamic data/feature itself.

On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:
Are stored properties in extensions already being discussed elsewhere? Is this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can only think of two ways:

1. An extra pointer per instance (with resulting ABI compatability implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift implementation, so there may be other/better ways - I'd like to know if there are?

If not, and option 2 was employed, a little foresight might enable the mechanism to be overloaded in the future for other dynamic features too. A bit flag (I'm hoping there's a spare bit in an existing flags field somewhere?) could indicate whether any feature had caused the object to be added to this lookup and deinit could check this bit and make sure the object is removed, thus any stored properties are nilled. The lookup value could be a struct with one member (extensionStoredProperties), and additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps there are much better, more optimal, more ingenious, more unknown-by-me ways of doing such things, so apologies if this whole idea is way-off the mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to implement their own dynamic features in Swift.. so such desires could be satisfied in the short term until they could be done properly in the language.

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a false dichotomy. Swift can definitely have both awesome dynamic features while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have bandwidth to reevaluate this area from first principles, I’m sure we can make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity, but for that simplicity it makes many tradeoffs that we’re not willing to make. We are willing to make the internal implementation of Swift complex if that means that we get a beautiful model for programmers - one that preserves the virtues of safety-by-default, predictability, performance, and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charles Srstka) #8

*Replace both instances of “class C: P” with just “class C” since the conformance comes in the extension. That’s what I get for writing this quickly.

Charles

···

On Oct 9, 2016, at 3:43 PM, Charles Srstka <cocoadev@charlessoft.com> wrote:

protocol P {
  var foo: String { get }
  func bar()
}

protocol Q {
  var baz: Int { get }
  func qux()
}

class C: P {
  var foo: String // <- what is this doing here?
  var baz: Int // <- ditto
}

extension C: P {
  func bar() {}
}

extension C: Q {
  func qux() {}
}

we could simply:

protocol P {
  var foo: String { get }
  func bar()
}

class C: P {}

extension C: P {
  var foo: String
  func bar() {}
}

extension C: Q {
  var baz: Int
  func qux() {}
}


(Jay) #9

Charlie,

What you suggest defines how you would use it from your code, not how it
would be implemented in the language. If you look at my AO implementation
it does what you say:
https://github.com/j-h-a/AssociatedObjects/blob/develop/AssociatedObjects/AssociatedObjects.swift
i.e. has a protocol called 'Associable' and you opt classes into it to get
the behaviour. This works and is usable, but the implementation leaves a
lot to be desired (it's not optimal and while the interface is clean the
implementation is not). Anyway - I was trying to steer the conversation
AWAY from AOs towards stored properties in extensions, since Robert Widmann
helped me to understand that AO was just a *means*, whereas stored
properties in extensions is the *end*.

In fact we don't need a solution to the problem of "how to define/use
stored properties in extensions" because the existing syntax for extensions
is perfectly fine. Currently you get an error if you try to define a stored
property in an extension, so no new syntax is needed, we just remove that
error and make it work.

Of course a runtime-check may be needed if there is doubt about whether a
dynamically linked module supported this feature - so this might invalidate
what I just said above, or it might still be possible if the runtime does
the check automatically when an extension is linked and puts a different
implementation in place for older modules.

I'm just airing some thoughts at the moment to see what people think and
try to get some technical feedback on viability. So it's not all fully
thought through :smiley:

There is a 4th way.

Introduce an internal protocol Associatable, which would tell the compiler
to add an additional (hidden) field to the object which would include the
"dictionary" of key -> value associated values. (It would be off-limits to
extensions, of course).

This way:

- it won't be a single dictionary containing all the associated values
- classes can opt-in to this
- the dictionary will be per-instance

This is a midway between the current implementation of ObjC associated
objects and of what someone has suggested to have an extra space for each
object for the AO...

I have been thinking further on this and in addition to my previous two
thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is
currently always zero), this could be used to signify the presence of an
additional struct that immediately follows after the existing object data.
I *think* that this method would allow binary compatibility with older
modules. Instances that have this bit set would allow stored properties in
extensions. The struct at the end would have one member, a pointer to a
table of additional objects/values, stored properties defined in extensions
could be stored in here, using a hash derived from the
module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with
additional features, for example a list of deinit hooks, dynamically added
methods, etc. The struct itself may also be dynamic in size/layout if such
complexity is warranted by size or extensibility concerns. Perhaps it
should start with some flags and its size (size would be fixed and only
increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than
my previous two ideas. It doesn't require global lookups or additional
levels of indirection beyond accessing the dynamic data/feature itself.

Are stored properties in extensions already being discussed elsewhere? Is
this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can only
think of two ways:

1. An extra pointer per instance (with resulting ABI compatability
implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift
implementation, so there may be other/better ways - I'd like to know if
there are?

If not, and option 2 was employed, a little foresight might enable the
mechanism to be overloaded in the future for other dynamic features too. A
bit flag (I'm hoping there's a spare bit in an existing flags field
somewhere?) could indicate whether any feature had caused the object to be
added to this lookup and deinit could check this bit and make sure the
object is removed, thus any stored properties are nilled. The lookup value
could be a struct with one member (extensionStoredProperties), and
additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps
there are much better, more optimal, more ingenious, more unknown-by-me
ways of doing such things, so apologies if this whole idea is way-off the
mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to
implement their own dynamic features in Swift.. so such desires could be
satisfied in the short term until they could be done properly in the
language.

Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a
false dichotomy. Swift can definitely have both awesome dynamic features
while still having performance, predictability and safety.

Would it be possible in Swift to have facilities to generate objects
dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have
bandwidth to reevaluate this area from first principles, I’m sure we can
make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity,
but for that simplicity it makes many tradeoffs that we’re not willing to
make. We are willing to make the internal implementation of Swift complex
if that means that we get a beautiful model for programmers - one that
preserves the virtues of safety-by-default, predictability, performance,
and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is
tired, and also wildly inaccurate.

-Chris

···

On Sun, 9 Oct 2016 at 20:54 Charlie Monroe <charlie@charliemonroe.net> wrote:
On Oct 9, 2016, at 9:47 PM, Jay Abbott via swift-evolution < swift-evolution@swift.org> wrote:
On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk> wrote:
On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:
On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution < swift-evolution@swift.org> wrote:

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jay) #10

Charles,

That would be good. It is a nicer way to write what is already possible to
achieve, but it's not a way to 'implement' stored properties in extensions.

···

On Sun, 9 Oct 2016 at 21:45 Charles Srstka <cocoadev@charlessoft.com> wrote:

*Replace both instances of “class C: P” with just “class C” since the
conformance comes in the extension. That’s what I get for writing this
quickly.

Charles

On Oct 9, 2016, at 3:43 PM, Charles Srstka <cocoadev@charlessoft.com> > wrote:

protocol P {
var foo: String { get }
func bar()
}

protocol Q {
var baz: Int { get }
func qux()
}

class C: P {
var foo: String // <- what is this doing here?
var baz: Int // <- ditto
}

extension C: P {
func bar() {}
}

extension C: Q {
func qux() {}
}

we could simply:

protocol P {
var foo: String { get }
func bar()
}

class C: P {}

extension C: P {
var foo: String
func bar() {}
}

extension C: Q {
var baz: Int
func qux() {}
}


(Charlie Monroe) #11

No, I've also suggested how it would be implemented. It would, as I've described, require support from the compiler and runtime - the protocol conformance would tell the compiler to include an extra space in the instance layout for [AnyHashable : Any], which would get to be used by the runtime to store AO.

Note that the implementation noted below does not use any locks at all - unlike ObjC AO, it's not thread-safe.

With AO, the main bottleneck always was that everything was stored in one place - this way, each object would have its own AOs stored within itself. This way, instead of a single spin (!) lock (https://opensource.apple.com/source/objc4/objc4-680/runtime/objc-references.mm), you can use a lock pool - e.g. you have a dozen locks, depending on the hash of the object itself, you decide which lock to use - this lowers the contention a lot.

Try to run a few threads, read, write AO using the ObjC runtime - you'll see how painfully slow it is - this is not something that should be in Swift.

···

On Oct 9, 2016, at 10:15 PM, Jay Abbott <jay@abbott.me.uk> wrote:

Charlie,

What you suggest defines how you would use it from your code, not how it would be implemented in the language. If you look at my AO implementation it does what you say:
https://github.com/j-h-a/AssociatedObjects/blob/develop/AssociatedObjects/AssociatedObjects.swift
i.e. has a protocol called 'Associable' and you opt classes into it to get the behaviour. This works and is usable, but the implementation leaves a lot to be desired (it's not optimal and while the interface is clean the implementation is not). Anyway - I was trying to steer the conversation AWAY from AOs towards stored properties in extensions, since Robert Widmann helped me to understand that AO was just a *means*, whereas stored properties in extensions is the *end*.

In fact we don't need a solution to the problem of "how to define/use stored properties in extensions" because the existing syntax for extensions is perfectly fine. Currently you get an error if you try to define a stored property in an extension, so no new syntax is needed, we just remove that error and make it work.

Of course a runtime-check may be needed if there is doubt about whether a dynamically linked module supported this feature - so this might invalidate what I just said above, or it might still be possible if the runtime does the check automatically when an extension is linked and puts a different implementation in place for older modules.

I'm just airing some thoughts at the moment to see what people think and try to get some technical feedback on viability. So it's not all fully thought through :smiley:

On Sun, 9 Oct 2016 at 20:54 Charlie Monroe <charlie@charliemonroe.net <mailto:charlie@charliemonroe.net>> wrote:
There is a 4th way.

Introduce an internal protocol Associatable, which would tell the compiler to add an additional (hidden) field to the object which would include the "dictionary" of key -> value associated values. (It would be off-limits to extensions, of course).

This way:

- it won't be a single dictionary containing all the associated values
- classes can opt-in to this
- the dictionary will be per-instance

This is a midway between the current implementation of ObjC associated objects and of what someone has suggested to have an extra space for each object for the AO...

On Oct 9, 2016, at 9:47 PM, Jay Abbott via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I have been thinking further on this and in addition to my previous two thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is currently always zero), this could be used to signify the presence of an additional struct that immediately follows after the existing object data. I *think* that this method would allow binary compatibility with older modules. Instances that have this bit set would allow stored properties in extensions. The struct at the end would have one member, a pointer to a table of additional objects/values, stored properties defined in extensions could be stored in here, using a hash derived from the module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with additional features, for example a list of deinit hooks, dynamically added methods, etc. The struct itself may also be dynamic in size/layout if such complexity is warranted by size or extensibility concerns. Perhaps it should start with some flags and its size (size would be fixed and only increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than my previous two ideas. It doesn't require global lookups or additional levels of indirection beyond accessing the dynamic data/feature itself.

On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:
Are stored properties in extensions already being discussed elsewhere? Is this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can only think of two ways:

1. An extra pointer per instance (with resulting ABI compatability implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift implementation, so there may be other/better ways - I'd like to know if there are?

If not, and option 2 was employed, a little foresight might enable the mechanism to be overloaded in the future for other dynamic features too. A bit flag (I'm hoping there's a spare bit in an existing flags field somewhere?) could indicate whether any feature had caused the object to be added to this lookup and deinit could check this bit and make sure the object is removed, thus any stored properties are nilled. The lookup value could be a struct with one member (extensionStoredProperties), and additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps there are much better, more optimal, more ingenious, more unknown-by-me ways of doing such things, so apologies if this whole idea is way-off the mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to implement their own dynamic features in Swift.. so such desires could be satisfied in the short term until they could be done properly in the language.

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a false dichotomy. Swift can definitely have both awesome dynamic features while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have bandwidth to reevaluate this area from first principles, I’m sure we can make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity, but for that simplicity it makes many tradeoffs that we’re not willing to make. We are willing to make the internal implementation of Swift complex if that means that we get a beautiful model for programmers - one that preserves the virtues of safety-by-default, predictability, performance, and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charles Srstka) #12

Right. The question is whether we *need* to add stored properties out-of-module, and what the use case for that is. To me it seems that adding them in-module is by far the more common use case, for the purposes of implementing protocols.

At any rate, the rewrite option would be a great addition to Swift regardless of what our answer to the first question is.

Charles

···

On Oct 9, 2016, at 4:32 PM, Jay Abbott <jay@abbott.me.uk> wrote:

Charles,

That would be good. It is a nicer way to write what is already possible to achieve, but it's not a way to 'implement' stored properties in extensions.

On Sun, 9 Oct 2016 at 21:45 Charles Srstka <cocoadev@charlessoft.com <mailto:cocoadev@charlessoft.com>> wrote:
*Replace both instances of “class C: P” with just “class C” since the conformance comes in the extension. That’s what I get for writing this quickly.

Charles

On Oct 9, 2016, at 3:43 PM, Charles Srstka <cocoadev@charlessoft.com <mailto:cocoadev@charlessoft.com>> wrote:

protocol P {
  var foo: String { get }
  func bar()
}

protocol Q {
  var baz: Int { get }
  func qux()
}

class C: P {
  var foo: String // <- what is this doing here?
  var baz: Int // <- ditto
}

extension C: P {
  func bar() {}
}

extension C: Q {
  func qux() {}
}

we could simply:

protocol P {
  var foo: String { get }
  func bar()
}

class C: P {}

extension C: P {
  var foo: String
  func bar() {}
}

extension C: Q {
  var baz: Int
  func qux() {}
}


(Paul Cantrell) #13

Yes please! A big strong +1 to this from me. I can think of several specific chunks of problem code that this would clean up immensely.

Contra Karl in another message, it’s _in-module_ stored property extensions that I want most frequently. By far.

It seems to me that Charles’s idea could be introduced as its own proposal. If out-of-module stored property extensions do eventually become feasible, then Charles’s proposal is a good stepping stone. If they never do, then his proposal has done no harm.

I realize this probably falls into the post-ABI stability bucket, but I’d love to help write/support the proposal when its time comes.

Cheers,

Paul

···

On Oct 9, 2016, at 3:43 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

Let extensions introduce stored properties, but only in the same module as the type’s definition. Then, the compiler can just take any extensions into consideration when it’s determining the size of the type, just as if the properties had been declared in the type. Declaring stored properties on an extension outside of the type’s module results in a compiler error, exactly as today. This would, without any performance drawbacks, solve one of the big problems that people are hoping to solve via stored properties in extensions—the ability to organize members by protocol conformance.


(Jay) #14

I understand Charlie. A few points though:
a) I'm talking about pure Swift, this is absolutely nothing to do with
Objective-C;
b) If you read back on the AO thread you'll see that initially I was
thinking "I want AO because it will give me stored properties in
extensions" when really I should have been thinking "I want stored
properties in extensions". So this discussion is no longer about AO at all,
hence the new subject.
c) Protocol conformance in an extension cannot tell the compiler to add
members to classes defined in other precompiled modules, as it is not
compiling them. When you do have the code and you are compiling it
yourself, you actually don't *need* stored properties in extensions because
you can work around it (although it would be nice), so the case where you
actually need it is where you want to extend someone else's precompiled
class.

···

On Mon, 10 Oct 2016 at 06:04 Charlie Monroe <charlie@charliemonroe.net> wrote:

No, I've also suggested how it would be implemented. It would, as I've
described, require support from the compiler and runtime - the protocol
conformance would tell the compiler to include an extra space in the
instance layout for [AnyHashable : Any], which would get to be used by the
runtime to store AO.

Note that the implementation noted below does not use any locks at all -
unlike ObjC AO, it's not thread-safe.

With AO, the main bottleneck always was that everything was stored in one
place - this way, each object would have its own AOs stored within itself.
This way, instead of a single spin (!) lock (
https://opensource.apple.com/source/objc4/objc4-680/runtime/objc-references.mm),
you can use a lock pool - e.g. you have a dozen locks, depending on the
hash of the object itself, you decide which lock to use - this lowers the
contention a lot.

Try to run a few threads, read, write AO using the ObjC runtime - you'll
see how painfully slow it is - this is not something that should be in
Swift.

On Oct 9, 2016, at 10:15 PM, Jay Abbott <jay@abbott.me.uk> wrote:

Charlie,

What you suggest defines how you would use it from your code, not how it
would be implemented in the language. If you look at my AO implementation
it does what you say:

https://github.com/j-h-a/AssociatedObjects/blob/develop/AssociatedObjects/AssociatedObjects.swift
i.e. has a protocol called 'Associable' and you opt classes into it to get
the behaviour. This works and is usable, but the implementation leaves a
lot to be desired (it's not optimal and while the interface is clean the
implementation is not). Anyway - I was trying to steer the conversation
AWAY from AOs towards stored properties in extensions, since Robert Widmann
helped me to understand that AO was just a *means*, whereas stored
properties in extensions is the *end*.

In fact we don't need a solution to the problem of "how to define/use
stored properties in extensions" because the existing syntax for extensions
is perfectly fine. Currently you get an error if you try to define a stored
property in an extension, so no new syntax is needed, we just remove that
error and make it work.

Of course a runtime-check may be needed if there is doubt about whether a
dynamically linked module supported this feature - so this might invalidate
what I just said above, or it might still be possible if the runtime does
the check automatically when an extension is linked and puts a different
implementation in place for older modules.

I'm just airing some thoughts at the moment to see what people think and
try to get some technical feedback on viability. So it's not all fully
thought through :smiley:

On Sun, 9 Oct 2016 at 20:54 Charlie Monroe <charlie@charliemonroe.net> > wrote:

There is a 4th way.

Introduce an internal protocol Associatable, which would tell the compiler
to add an additional (hidden) field to the object which would include the
"dictionary" of key -> value associated values. (It would be off-limits to
extensions, of course).

This way:

- it won't be a single dictionary containing all the associated values
- classes can opt-in to this
- the dictionary will be per-instance

This is a midway between the current implementation of ObjC associated
objects and of what someone has suggested to have an extra space for each
object for the AO...

On Oct 9, 2016, at 9:47 PM, Jay Abbott via swift-evolution < > swift-evolution@swift.org> wrote:

I have been thinking further on this and in addition to my previous two
thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is
currently always zero), this could be used to signify the presence of an
additional struct that immediately follows after the existing object data.
I *think* that this method would allow binary compatibility with older
modules. Instances that have this bit set would allow stored properties in
extensions. The struct at the end would have one member, a pointer to a
table of additional objects/values, stored properties defined in extensions
could be stored in here, using a hash derived from the
module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with
additional features, for example a list of deinit hooks, dynamically added
methods, etc. The struct itself may also be dynamic in size/layout if such
complexity is warranted by size or extensibility concerns. Perhaps it
should start with some flags and its size (size would be fixed and only
increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than
my previous two ideas. It doesn't require global lookups or additional
levels of indirection beyond accessing the dynamic data/feature itself.

On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk> wrote:

Are stored properties in extensions already being discussed elsewhere? Is
this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can
only think of two ways:

1. An extra pointer per instance (with resulting ABI compatability
implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift
implementation, so there may be other/better ways - I'd like to know if
there are?

If not, and option 2 was employed, a little foresight might enable the
mechanism to be overloaded in the future for other dynamic features too. A
bit flag (I'm hoping there's a spare bit in an existing flags field
somewhere?) could indicate whether any feature had caused the object to be
added to this lookup and deinit could check this bit and make sure the
object is removed, thus any stored properties are nilled. The lookup value
could be a struct with one member (extensionStoredProperties), and
additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps
there are much better, more optimal, more ingenious, more unknown-by-me
ways of doing such things, so apologies if this whole idea is way-off the
mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to
implement their own dynamic features in Swift.. so such desires could be
satisfied in the short term until they could be done properly in the
language.

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution < > swift-evolution@swift.org> wrote:

On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution < > swift-evolution@swift.org> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a
false dichotomy. Swift can definitely have both awesome dynamic features
while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have
bandwidth to reevaluate this area from first principles, I’m sure we can
make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity,
but for that simplicity it makes many tradeoffs that we’re not willing to
make. We are willing to make the internal implementation of Swift complex
if that means that we get a beautiful model for programmers - one that
preserves the virtues of safety-by-default, predictability, performance,
and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is
tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charlie Monroe) #15

Yes, there are valid use cases.

For example, in case of many UI elements, you do not have control over their instantiation and need to associate a "help description" with them as an example. Which can be done by an additional property helpDescription added to NSView/UIView.

Another example that I've used is with CoreData, where you can store additional information on the MOC - NSManagedObjectContext is strongly discouraged from being subclassed. Example of the additional information is a synchronization manager assigned to the MOC.

And since in Swift, classes can be final, there may not be a way to subclass the class to add your properties instead.

IMHO, it is a valid request to be able to add additional stored properties via extensions.

···

On Oct 10, 2016, at 9:15 PM, Charles Srstka <cocoadev@charlessoft.com> wrote:

Right. The question is whether we *need* to add stored properties out-of-module, and what the use case for that is. To me it seems that adding them in-module is by far the more common use case, for the purposes of implementing protocols.

At any rate, the rewrite option would be a great addition to Swift regardless of what our answer to the first question is.

Charles

On Oct 9, 2016, at 4:32 PM, Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:

Charles,

That would be good. It is a nicer way to write what is already possible to achieve, but it's not a way to 'implement' stored properties in extensions.

On Sun, 9 Oct 2016 at 21:45 Charles Srstka <cocoadev@charlessoft.com <mailto:cocoadev@charlessoft.com>> wrote:
*Replace both instances of “class C: P” with just “class C” since the conformance comes in the extension. That’s what I get for writing this quickly.

Charles

On Oct 9, 2016, at 3:43 PM, Charles Srstka <cocoadev@charlessoft.com <mailto:cocoadev@charlessoft.com>> wrote:

protocol P {
  var foo: String { get }
  func bar()
}

protocol Q {
  var baz: Int { get }
  func qux()
}

class C: P {
  var foo: String // <- what is this doing here?
  var baz: Int // <- ditto
}

extension C: P {
  func bar() {}
}

extension C: Q {
  func qux() {}
}

we could simply:

protocol P {
  var foo: String { get }
  func bar()
}

class C: P {}

extension C: P {
  var foo: String
  func bar() {}
}

extension C: Q {
  var baz: Int
  func qux() {}
}


(Karl) #16

I actually think out-of-module is the more common and important case. If you’re using a types from an external framework, it’s strange that you can retroactively add functionality but not additional data. That limitation, AFAICT, is purely due to implementation. Some well-constructed frameworks will allow you to add stored data to the types being used by subclassing. Not all are so well constructed, and as we have more and more value-types subclassing is not going to be a solution.

You can add stored properties to objects in Python, which is one of the really great things about it. If you need to add a stored property to track some additional state you layer on top of a framework’s components, it’s staggering how easy it is - you don’t even need to define any new types.

I personally think it’s important for Swift as a scripting language to have this ability - for value-types as well as classes, by default. We could optimise it away completely for types which aren’t public, and you should be able to explicitly declare a type non-extendable (similar to declaring it non-subclassable with ‘final’) to opt-out. I’m sure it will never happen, but there you go.

···

On 10 Oct 2016, at 21:15, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

Right. The question is whether we *need* to add stored properties out-of-module, and what the use case for that is. To me it seems that adding them in-module is by far the more common use case, for the purposes of implementing protocols.

At any rate, the rewrite option would be a great addition to Swift regardless of what our answer to the first question is.

Charles


(Charles Constant) #17

+1 for me for "in-module" as a stop-gap, since I imagine it would be the
quickest, and least disruptive, way to make this happen.

I would add the caveat that if we do so, I really hope we commit to making
stored properties available *everywhere* later.

Even though it's more often than not "in-module" where I need this, the
fewer "exceptions to the rule" we have in Swift the better. It gets tricky
trying to plan things out when one has to juggle too many features that
work in one context, but not another.

So +1 with the hope that it doesn't stay "in-module" till the end of time.

···

On Fri, Oct 14, 2016 at 6:01 PM, Paul Cantrell via swift-evolution < swift-evolution@swift.org> wrote:

> On Oct 9, 2016, at 3:43 PM, Charles Srstka via swift-evolution < > swift-evolution@swift.org> wrote:
>
> Let extensions introduce stored properties, but only in the same module
as the type’s definition. Then, the compiler can just take any extensions
into consideration when it’s determining the size of the type, just as if
the properties had been declared in the type. Declaring stored properties
on an extension outside of the type’s module results in a compiler error,
exactly as today. This would, without any performance drawbacks, solve one
of the big problems that people are hoping to solve via stored properties
in extensions—the ability to organize members by protocol conformance.

Yes please! A big strong +1 to this from me. I can think of several
specific chunks of problem code that this would clean up immensely.

Contra Karl in another message, it’s _in-module_ stored property
extensions that I want most frequently. By far.

It seems to me that Charles’s idea could be introduced as its own
proposal. If out-of-module stored property extensions do eventually become
feasible, then Charles’s proposal is a good stepping stone. If they never
do, then his proposal has done no harm.

I realize this probably falls into the post-ABI stability bucket, but I’d
love to help write/support the proposal when its time comes.

Cheers,

Paul

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Haravikk) #18

+1 from me too for internal extensions; I think this is the logical place to start for now so we can see how many use cases this actually covers. Personally I don't try to add functionality to types from other libraries, and when I have to I try as much as possible to do it via wrapping; obviously this doesn't cover the async type cases and others where it'd be hard to re-wrap the values, but I think I'd like to know more about how common such cases actually are.

However, allowing stored properties in local extensions is an absolute must for a first step, and would be very useful as a first step.

The problem I have with doing it for external types is that I just don't see how it can be done efficiently; associated objects means looking up the object to find what its associated values are, which isn't a negligible cost if you're doing it frequently, and it makes me very uncomfortable to think of hiding what is actually happening, as developers may think they're just using a regular property without really appreciating the actual costs involved. At the very least they need to handled through a .associated property or whatever to make it much clearer that these aren't native properties in the normal sense.

···

On 15 Oct 2016, at 02:01, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 9, 2016, at 3:43 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

Let extensions introduce stored properties, but only in the same module as the type’s definition. Then, the compiler can just take any extensions into consideration when it’s determining the size of the type, just as if the properties had been declared in the type. Declaring stored properties on an extension outside of the type’s module results in a compiler error, exactly as today. This would, without any performance drawbacks, solve one of the big problems that people are hoping to solve via stored properties in extensions—the ability to organize members by protocol conformance.

Yes please! A big strong +1 to this from me. I can think of several specific chunks of problem code that this would clean up immensely.


(Thorsten Seitz) #19

Has anybody thought about the semantic issues of out-of-module extensions with stored properties apart from the implementation issues?

Such properties could potentially wreak havoc with the semantics of the type being extended. How would these properties play nice with an existing definition of equality, for example? How can it be guaranteed that their value is consistent with the remaining state? And kept that way in case of mutability?

-Thorsten

···

Am 15.10.2016 um 03:01 schrieb Paul Cantrell via swift-evolution <swift-evolution@swift.org>:

On Oct 9, 2016, at 3:43 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

Let extensions introduce stored properties, but only in the same module as the type’s definition. Then, the compiler can just take any extensions into consideration when it’s determining the size of the type, just as if the properties had been declared in the type. Declaring stored properties on an extension outside of the type’s module results in a compiler error, exactly as today. This would, without any performance drawbacks, solve one of the big problems that people are hoping to solve via stored properties in extensions—the ability to organize members by protocol conformance.

Yes please! A big strong +1 to this from me. I can think of several specific chunks of problem code that this would clean up immensely.

Contra Karl in another message, it’s _in-module_ stored property extensions that I want most frequently. By far.

It seems to me that Charles’s idea could be introduced as its own proposal. If out-of-module stored property extensions do eventually become feasible, then Charles’s proposal is a good stepping stone. If they never do, then his proposal has done no harm.

I realize this probably falls into the post-ABI stability bucket, but I’d love to help write/support the proposal when its time comes.

Cheers,

Paul

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charlie Monroe) #20

I understand Charlie. A few points though:
a) I'm talking about pure Swift, this is absolutely nothing to do with Objective-C;

Sure, but in either case you will need to somehow solve locking of the structure holding the additional values.

b) If you read back on the AO thread you'll see that initially I was thinking "I want AO because it will give me stored properties in extensions" when really I should have been thinking "I want stored properties in extensions". So this discussion is no longer about AO at all, hence the new subject.

Agreed, though the actual backing of the values won't be a lot different from the AO - it can be called differently, but the fact that it is likely to be stored in a dictionary brings resemblence to AO - and what I've mentioned are a few downsides to how AO are implemented at this point.

c) Protocol conformance in an extension cannot tell the compiler to add members to classes defined in other precompiled modules, as it is not compiling them. When you do have the code and you are compiling it yourself, you actually don't *need* stored properties in extensions because you can work around it (although it would be nice), so the case where you actually need it is where you want to extend someone else's precompiled class.

This is why I mentioned that this "magic protocol" would be off limits to extensions. Perhaps the choice of a protocol to this was incorrect - better way would be to use an annotation like @allows_extension_properties or similar.

I bet you that in any case, there will be a lot of people on this list that will want explicit opt-in, or at least a way to opt-out.

···

On Oct 10, 2016, at 1:13 PM, Jay Abbott <jay@abbott.me.uk> wrote:

On Mon, 10 Oct 2016 at 06:04 Charlie Monroe <charlie@charliemonroe.net <mailto:charlie@charliemonroe.net>> wrote:
No, I've also suggested how it would be implemented. It would, as I've described, require support from the compiler and runtime - the protocol conformance would tell the compiler to include an extra space in the instance layout for [AnyHashable : Any], which would get to be used by the runtime to store AO.

Note that the implementation noted below does not use any locks at all - unlike ObjC AO, it's not thread-safe.

With AO, the main bottleneck always was that everything was stored in one place - this way, each object would have its own AOs stored within itself. This way, instead of a single spin (!) lock (https://opensource.apple.com/source/objc4/objc4-680/runtime/objc-references.mm), you can use a lock pool - e.g. you have a dozen locks, depending on the hash of the object itself, you decide which lock to use - this lowers the contention a lot.

Try to run a few threads, read, write AO using the ObjC runtime - you'll see how painfully slow it is - this is not something that should be in Swift.

On Oct 9, 2016, at 10:15 PM, Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:

Charlie,

What you suggest defines how you would use it from your code, not how it would be implemented in the language. If you look at my AO implementation it does what you say:
https://github.com/j-h-a/AssociatedObjects/blob/develop/AssociatedObjects/AssociatedObjects.swift
i.e. has a protocol called 'Associable' and you opt classes into it to get the behaviour. This works and is usable, but the implementation leaves a lot to be desired (it's not optimal and while the interface is clean the implementation is not). Anyway - I was trying to steer the conversation AWAY from AOs towards stored properties in extensions, since Robert Widmann helped me to understand that AO was just a *means*, whereas stored properties in extensions is the *end*.

In fact we don't need a solution to the problem of "how to define/use stored properties in extensions" because the existing syntax for extensions is perfectly fine. Currently you get an error if you try to define a stored property in an extension, so no new syntax is needed, we just remove that error and make it work.

Of course a runtime-check may be needed if there is doubt about whether a dynamically linked module supported this feature - so this might invalidate what I just said above, or it might still be possible if the runtime does the check automatically when an extension is linked and puts a different implementation in place for older modules.

I'm just airing some thoughts at the moment to see what people think and try to get some technical feedback on viability. So it's not all fully thought through :smiley:

On Sun, 9 Oct 2016 at 20:54 Charlie Monroe <charlie@charliemonroe.net <mailto:charlie@charliemonroe.net>> wrote:
There is a 4th way.

Introduce an internal protocol Associatable, which would tell the compiler to add an additional (hidden) field to the object which would include the "dictionary" of key -> value associated values. (It would be off-limits to extensions, of course).

This way:

- it won't be a single dictionary containing all the associated values
- classes can opt-in to this
- the dictionary will be per-instance

This is a midway between the current implementation of ObjC associated objects and of what someone has suggested to have an extra space for each object for the AO...

On Oct 9, 2016, at 9:47 PM, Jay Abbott via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I have been thinking further on this and in addition to my previous two thoughts about implementation, I have had another idea...

3. If there is a bit spare in the object header somewhere (that is currently always zero), this could be used to signify the presence of an additional struct that immediately follows after the existing object data. I *think* that this method would allow binary compatibility with older modules. Instances that have this bit set would allow stored properties in extensions. The struct at the end would have one member, a pointer to a table of additional objects/values, stored properties defined in extensions could be stored in here, using a hash derived from the module/protocol/extension/property name (or something better if it exists).

The struct could be very simple as described above or more complex, with additional features, for example a list of deinit hooks, dynamically added methods, etc. The struct itself may also be dynamic in size/layout if such complexity is warranted by size or extensibility concerns. Perhaps it should start with some flags and its size (size would be fixed and only increase with revisions so this would double as a 'version' number).

If viable - this would be a much better way to implement this feature than my previous two ideas. It doesn't require global lookups or additional levels of indirection beyond accessing the dynamic data/feature itself.

On Mon, 3 Oct 2016 at 04:13 Jay Abbott <jay@abbott.me.uk <mailto:jay@abbott.me.uk>> wrote:
Are stored properties in extensions already being discussed elsewhere? Is this one of those deferred-but-not-indexed-anywhere subjects?

I wonder how stored properties could potentially be implemented, I can only think of two ways:

1. An extra pointer per instance (with resulting ABI compatability implications) to hold a collection of the stored items.

2. A global lookup for any instance where stored properties have been set.

I'm not a language implementation expert, or familiar with the swift implementation, so there may be other/better ways - I'd like to know if there are?

If not, and option 2 was employed, a little foresight might enable the mechanism to be overloaded in the future for other dynamic features too. A bit flag (I'm hoping there's a spare bit in an existing flags field somewhere?) could indicate whether any feature had caused the object to be added to this lookup and deinit could check this bit and make sure the object is removed, thus any stored properties are nilled. The lookup value could be a struct with one member (extensionStoredProperties), and additional members can be added in future for new features.

I get the impression from the associated objects discussion that perhaps there are much better, more optimal, more ingenious, more unknown-by-me ways of doing such things, so apologies if this whole idea is way-off the mark :smiley:

Jay

P.S. Note that stored properties in extensions could enable developers to implement their own dynamic features in Swift.. so such desires could be satisfied in the short term until they could be done properly in the language.

On Sat, 1 Oct 2016 at 00:49 Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
On Sep 30, 2016, at 2:51 PM, Ted F.A. van Gaalen via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> Is it possible to have best of (these completely different) both worlds?

Yes, of course it is. Your email spends a lot of words trying to form a false dichotomy. Swift can definitely have both awesome dynamic features while still having performance, predictability and safety.

> Would it be possible in Swift to have facilities to generate objects
> dynamically at runtime? and, if desirable, how can such be implemented?

Here’s an extant implementation that you can use today:
https://github.com/Zewo/Reflection

I’m sure it isn’t ideal, but it proves that it can be done. When we have bandwidth to reevaluate this area from first principles, I’m sure we can make improvements on it.

I will grant you that Smalltalk is a beautiful language in its simplicity, but for that simplicity it makes many tradeoffs that we’re not willing to make. We are willing to make the internal implementation of Swift complex if that means that we get a beautiful model for programmers - one that preserves the virtues of safety-by-default, predictability, performance, and joy-to-develop-in.

The meme of “Swift can never (or will never) support dynamic features” is tired, and also wildly inaccurate.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution