[Pitch] Introduce User-defined "Dynamic Member Lookup" Types

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduction>Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

    a = someValue.someMember
    someValue.someMember = a
and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"]
  someValue[dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties <https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html> and underlying messaging infrastructure <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html>). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html>, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#motivation-and-context>Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities <https://pandas.pydata.org/> and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> the code would have explicit member lookups all over the place:

  // import pickle
  let pickle = Python.get(member: "import")("pickle")

  // file = open(filename)
  let file = Python.get(member: "open")(filename)

  // blob = file.read()
  let blob = file.get(member: "read")()

  // result = pickle.loads(blob)
  let result = pickle.get(member: "loads")(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}
It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#example-usage>Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternatives-considered>Alternatives considered

A few alternatives were considered:

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#add-ability-to-provide-read-only-members>Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol {
  associatedtype DynamicMemberLookupValue

  // gettable only
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get }
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol {
  // gettable and settable.
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
  }
This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#naming>Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

+1 to both proposals. I don't know enough to comment further, but I know this would help in the adoption of Swift on my team(s) (we call a lot of Python, although currently via exec-ing some bash scripts; it's a mess, but works well in the CLI world the stuff was written for).

···

On Nov 14, 2017, at 23:29 , Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

  • Proposal: SE-NNNN
  • Author: Chris Lattner
  • Review Manager: TBD
  • Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

    a = someValue.someMember

    someValue.
someMember = a
and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"
]
  someValue[
dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties and underlying messaging infrastructure). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal

Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog
:
    
def __init__(self, name
):
        
self.name =
name
        
self.tricks = [] # creates a new empty list for each dog

def add_trick(self, trick
):
        
self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle"
)

// file = open(filename)
  let file = Python.open
(filename)

// blob = file.read()
  let blob = file.read
()

// result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal the code would have explicit member lookups all over the place:

  // import pickle
  let pickle = Python.get(member: "import")("pickle"
)

// file = open(filename)
  let file = Python.get(member: "open"
)(filename)

// blob = file.read()
  let blob = file.get(member: "read"
)()

// result = pickle.loads(blob)
  let result = pickle.get(member: "loads"
)(blob)

// dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol
{
  
associatedtype DynamicMemberLookupValue

subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
}
}

It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal
{
  
subscript(dynamicMember member: String) ->
PyVal {
    
get
{
      
let result = PyObject_GetAttrString(borrowedPyObject, member)!

return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    
set
{
      
PyObject_SetAttrString
(borrowedPyObject, member,
                             newValue.
toPython().borrowedPyObject
)
    }
  }
}

Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol
{
  
associatedtype DynamicMemberLookupValue

// gettable only
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get
}
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol
{
  
// gettable and settable.
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
}
  }

This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

How would this work for a property access that caused a python exception to
be thrown? Are we assuming that only "good" non-throwing properties will
be bridged in this way?

>>> x = "hello"
>>> x.foobar

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'foobar'
···

On Tue, Nov 14, 2017 at 11:29 PM, Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/
lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback
on making member lookup dynamically extensible. My primary motivation is
to improve interoperability with dynamic languages like Python, Perl, Ruby,
Javascript, etc, but there are other use cases (e.g. when working with
untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an
implementation can be done in a way with changes that are localized, and
thus not have significant impact on the maintainability of the compiler as
a whole. Once the pitch phase of this proposal helps refine the details,
I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype
Python bindings. I’ll share them in the next day or two in case they are
useful to provide context. It is amazingly simple: less than 500 lines of
Swift code (plus some small additional C header glue to work around clang
importer limitations) enables impressive interoperability. The only
problems are the verbosity addressed by this proposal and the peer
DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks
in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

   - Proposal: SE-NNNN
   <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
   - Author: Chris Lattner <https://github.com/lattner>
   - Review Manager: TBD
   - Status: Awaiting implementation

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduction>
Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the
standard library. Types that conform to it provide "dot" syntax for
arbitrary names which are resolved at runtime. It is simple syntactic sugar
which allows the user to write:

    a = someValue.someMember
    someValue.someMember = a

and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"]
  someValue[dynamicMember: "someMember"] = a

Many other languages have analogous features (e.g. the composition of
Objective-C's explicit properties
<https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html> and
underlying messaging infrastructure
<https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html>).
This sort of functionality is great for implementing dynamic language
interoperability, dynamic proxy APIs
<https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html>,
and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal
<https://lists.swift.org/pipermail/swift-evolution/>

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#motivation-and-context>Motivation
and Context

Swift is well known for being exceptional at interworking with existing C
and Objective-C APIs, but its support for calling APIs written in scripting
languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount
of effort into integrating Clang ASTs, remapping existing APIs in an
attempt to feel "Swifty", and by providing a large number of attributes and
customization points for changing the behavior of this integration when
writing an Objective-C header. The end result of this massive investment of
effort is that Swift provides a *better* experience when programming
against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear:
1) there are several different languages of interest, and they each have
significant interest in different quarters: for example, Python is big in
data science and machine learning, Ruby is popular for building server side
apps, and even Perl is in still widely used. 2) These languages have
decades of library building behind them, sometimes with significant
communities <https://pandas.pydata.org/> and 3) there are one or two
orders of magnitude more users of these libraries than there are people
currently using Swift.

While it is theoretically possible to expend the same level of effort on
each of these languages and communities as has been spent on Objective-C,
it is quite clear that this would both ineffective as well as bad for
Swift: It would be ineffective, because the Swift community has not
leverage over these communities to force auditing and annotation of their
APIs. It would be bad for Swift because it would require a ton of
language-specific support (and a number of third-party dependencies) onto
the compiler and runtime, each of which makes the implementation
significantly more complex, difficult to reason about, difficult to
maintain, and difficult to test the supported permutations. In short, we'd
end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic
programming model where almost everything is discovered at runtime, and
many of them are explicitly designed to be embedded into other languages
and applications. This aspect allows us to embed APIs from these languages
directly into Swift with no language support at all - without not the level
of effort, integration, and invasiveness that Objective-C has benefited
from. Instead of invasive importer work, we can write some
language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can
"embrace" these dynamic language APIs (making them directly available in
Swift) which reduces the pain of someone moving from one of those languages
into Swift. It is true that the APIs thus provided will not feel "Swifty",
but if that becomes a significant problem for any one API, then the
community behind it can evaluate the problem and come up with a solution
(either a Swift wrapper for the dynamic language, or a from-scratch Swift
reimplementation of the desired API). In any case, if/when we face this
challenge, it will be a good thing: we'll know that we've won a significant
new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language
APIs into Swift today, the resultant API is unusable for two major reasons:
member lookup is too verbose to be acceptable, and calling behavior is
similarly too verbose to be acceptable. As such, we seek to provide two
"syntactic sugar" features that solve this problem. These sugars are
specifically designed to be dynamic language independent and, indeed,
independent of dynamic languages at all: we can imagine other usage for the
same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> protocol
and a related DynamicMemberLookupProtocol proposal (this proposal). With
these two extensions, we think we can eliminate the need for invasive
importer magic by making interoperability with dynamic languages ergonomic
enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

we would like to be able to use this from Swift like this (the comments
show the corresponding syntax you would use in Python):

  // import DogModule // import DogModule.Dog as Dog // an alternate let Dog = Python.import(“DogModule.Dog") // dog = Dog("Brianna") let dog = Dog("Brianna") // dog.add_trick("Roll over") dog.add_trick("Roll over") // dog2 = Dog("Kaylee").add_trick("snore") let dog2 = Dog("Kaylee").add_trick("snore")

Of course, this would also apply to standard Python APIs as well. Here is
an example working with the Python pickleAPI and the builtin Python
function open:

  // import pickle let pickle = Python.import("pickle")

  // file = open(filename) let file = Python.open(filename)

  // blob = file.read() let blob = file.read()

  // result = pickle.loads(blob) let result = pickle.loads(blob)

This can all be expressed today as library functionality written in Swift,
but without this proposal, the code required is unnecessarily verbose and
gross. Without it (but *with* the related DynamicCallable proposal
<https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> the
code would have explicit member lookups all over the place:

  // import pickle let pickle = Python.get(member: "import")("pickle")

  // file = open(filename) let file = Python.get(member: "open")(filename)

  // blob = file.read() let blob = file.get(member: "read")()

  // result = pickle.loads(blob) let result = pickle.get(member: "loads")(blob)

  // dog2 = Dog("Kaylee").add_trick("snore") let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")

While this is a syntactic sugar proposal, we believe that this expands
Swift to be usable in important new domains. In addition to dynamic
language interoperability, this sort of functionality is useful for other
APIs, e.g. when working with dynamically typed unstructured data like JSON,
which could provide an API like jsonValue?.jsonField1?.jsonField2where
each field is dynamically looked up.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#proposed-solution>Proposed
solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}

It also extends the language such that member lookup syntax (x.y) - when
it otherwise fails (because there is no member y defined on the type of x)
and when applied to a value which conforms to DynamicMemberLookupProtocol-
is accepted and transformed into a call to the subscript in the protocol.
This ensures that no member lookup on such a type ever fails.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#example-usage>Example
Usage

While there are many potential uses of this sort of API (e.g. resolving
JSON members to named results, producing optional bindings) a motivating
example comes from a prototype Python interoperability layer. There are
many ways to implement this, and the details are not particularly
important, but it is perhaps useful to know that this is directly useful to
address the motivation section described above. Given a currency type of
PyVal (and a conforming implementation named PyRef), an implementation
may look like:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result. }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#source-compatibility>Source
compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-abi-stability>Effect
on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-api-resilience>Effect
on API resilience

This has no impact on API resilience which is not already captured by
other language features.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternatives-considered>Alternatives
considered

A few alternatives were considered:

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#add-ability-to-provide-read-only-members>Add
ability to provide read-only members

The implementation above does not allow an implementation to statically
reject members that are read-only. If this was important to add, we could
add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol {
  associatedtype DynamicMemberLookupValue

  // gettable only subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get }
}
protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol {
  // gettable and settable. subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
  }

This would allow a type to implement one or the other based on their
capabilities. This proposal starts with a very simple design based on the
requirements of dynamic languages (which have no apparent immutability
model), but if there is demand for this (e.g. because we want input JSON
values to be gettable but not settalbe), the author is happy to switch to
this more general model.
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#naming>
Naming
There is a lot of grounds to debate naming of the protocol and methods in
this type. Suggestions (along with rationale to support them) are more than
welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Functional Programmer, iOS Developer, Surfs Poorly
http://twitter.com/n8gray

This looks great for Python, but let's talk about some other languages for a moment.

* Ruby and Perl don't have the "call a method by fetching a closure property and invoking it" behavior you're relying on here. Instead, Ruby has a syntax for settable "overloads" of methods (i.e. you can write `def someMember` and `def someMember= (newValue)`), while Perl supports lvalue methods (but sometimes uses getter and setter method pairs instead). How do you envision these behaviors being bridged to Swift? I worry that this protocol may not be sufficient, and that we may need a design which can distinguish between looking up methods and looking up properties.

* Ruby looks up members using symbols, which essentially play the same role as selectors in Objective-C—they're uniqued strings which are used for fast member dispatch. In some cases, you might see non-negligible speed improvements by only looking up the symbol once. Is there a way this design could accommodate that? For instance, could the type of the index be specified by an associated type, so Ruby could use a RbSymbol instead of a String? Or do you think that would be overkill?

* Generally, you've talked about properties (in this proposal) and methods (in the `DynamicCallable` proposal), but what about subscripts? Obviously you can just specify a `subscript(Pythonable) -> PyVal` on `PyVal` for the simple case, but what if the subscript takes multiple indices or has labels? Do we need a `DynamicSubscriptable` protocol?

* Let's step away from bridging entirely and just think about Swift for a moment. There are cases where we'd like to make *semi*-dynamic proxies which wrap another type and allow operations based on what's statically known about that type. Think, for example, of the appearance proxy in UIKit: This is an object attached to UIView subclasses which lets you (in essence) set default values for all instances. We currently just pretend it's an instance of `Self`, which mostly works because of Objective-C, but a Swift-native version would probably prefer to return a `UIAppearance<Self>` object which used its knowledge of `Self` to expose `Self`'s properties on itself. Is there a way we could design this feature, or a related feature, to cover that kind of use case? That is, to allow a limited set of keys—perhaps even key-path-based when you want static control—with a different type for each key, *or* to allow any key with some common type, depending on your type's needs?

* Actually, for that matter, let's talk about key paths. In principle, you can already think of member lookup in Swift—or at least property and subscript lookup—as though it always worked by constructing a key path and using `subscript(keyPath:)` to access it. Is there some way we could model this feature as extending the set of keys available on a given type—perhaps in a way that allowed compile-time-limited and strongly-typed sets of keys, like I mention with the `UIAppearance` example, in addition to the open-ended, type-erased sets you need—and then looking things up by key path? (Sorry if this is a little vague—I know very little about how key paths are implemented.)

* An implementation-level question about Swift: Internally, the compiler seems to be moving towards thinking of parameter labels as part of the identifier, rather than having them label the individual arguments. How do you see that jibing with what you're proposing here?

···

On Nov 14, 2017, at 11:29 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}

--
Brent Royal-Gordon
Architechies

Interesting! I like the spirit of this proposal a lot.

One question: presumably this behaves like Ruby’s method_missing in that any natively implemented members shadow the dynamic ones? i.e. Swift looks for a static match first, then uses the dynamicMember only as a fallback?

I found myself asking other questions as I read the proposal, and then agreeing with where you landed. Here are those questions in case they touch off useful thoughts:

• • •

Q: It seems like this fundamentally alters Swift’s aesthetic of “either an operation is type-safe, or it’s clear at the point of use that it’s not.” Should this use an operator other than a period, e.g. `pickle->loads(blob)`?

···

A: This shift in the language does give me pause, but using a different operator really undermines the interoperability story. This is probably a job for syntax highlighting, not syntax.

• • •

Q: Why the open-ended `associatedtype DynamicMemberLookupValue`? Seems like it will always just be Self in practice.

A: It would be Self in the Python and JSON examples in the proposal, but making it an associatedtype adds no implementation burden, does no apparent harm, and adds flexibility. Shifting types on traversal could be particularly useful in creating DSLs (e.g. sepia → Genus, sepia.officinalis → Species).

• • •

Q: OK, then shouldn’t individual dynamic methods be able to return different types?

A: And how precisely would that work?

Q: Good point. Question retracted.

• • •

Q: Is there any special handling for that member name string — to handle ruby method names like `blank?`, for example?

A: This should be the job of a language-specific interop layer.

• • •

Q: Should the subscript also take arg types and/or labels to allow overloading?

A: Ruby, Python, and JS all resolve members by name alone, and leave it to functions to untangle their own args. Obj-C is the lone oddball here. Relying on the companion proposal to handle the args makes sense.

Cheers, P

On Nov 15, 2017, at 1:29 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduction>Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

    a = someValue.someMember
    someValue.someMember = a
and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"]
  someValue[dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties <https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html> and underlying messaging infrastructure <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html>). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html>, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#motivation-and-context>Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities <https://pandas.pydata.org/> and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> the code would have explicit member lookups all over the place:

  // import pickle
  let pickle = Python.get(member: "import")("pickle")

  // file = open(filename)
  let file = Python.get(member: "open")(filename)

  // blob = file.read()
  let blob = file.get(member: "read")()

  // result = pickle.loads(blob)
  let result = pickle.get(member: "loads")(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}
It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#example-usage>Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternatives-considered>Alternatives considered

A few alternatives were considered:

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#add-ability-to-provide-read-only-members>Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol {
  associatedtype DynamicMemberLookupValue

  // gettable only
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get }
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol {
  // gettable and settable.
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
  }
This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#naming>Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Those two proposals are growing on me and they have the potential to make Swift even better at DSLs.

···

On 15 Nov 2017, at 08:29, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduction>Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

    a = someValue.someMember
    someValue.someMember = a
and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"]
  someValue[dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties <https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html> and underlying messaging infrastructure <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html>). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html>, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#motivation-and-context>Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities <https://pandas.pydata.org/> and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> the code would have explicit member lookups all over the place:

  // import pickle
  let pickle = Python.get(member: "import")("pickle")

  // file = open(filename)
  let file = Python.get(member: "open")(filename)

  // blob = file.read()
  let blob = file.get(member: "read")()

  // result = pickle.loads(blob)
  let result = pickle.get(member: "loads")(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}
It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#example-usage>Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternatives-considered>Alternatives considered

A few alternatives were considered:

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#add-ability-to-provide-read-only-members>Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol {
  associatedtype DynamicMemberLookupValue

  // gettable only
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get }
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol {
  // gettable and settable.
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
  }
This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#naming>Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Just a question about what it mean for API resilience. Actually, adding a property to an existing class is not a breaking change.

For a class that implements that protocol, as it will be possible to use property accessor with any non existing property. Adding a new property to this class will now change the class behavior in unexpected way as the old client will still call the dynamic lookup method, but new code will use the new accessor method. Is it something that should be mention in the « Effect on API resilience » section ?

···

Le 15 nov. 2017 à 08:29, Chris Lattner via swift-evolution <swift-evolution@swift.org> a écrit :

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD

I think this protocol and its subscript need a better name. From a user of the class point of view, the subscript looks like you can retrieve the members of the class, whereas in reality you'll only get the ones you have manually implemented. Even methods and properties having the `dynamic` attribute won't be available, even though the subscript name would suggest that.

I would propose adding the word `supplemental`to the subscript name and the name of the protocol to make it clearer that this is only for adding members that are not declared in the class (including the `dynamic` ones).

As in `SupplementalDynamicMemberLookupProtocol` and `supplementalDynamicMember` for the subscript.

···

Le 15 nov. 2017 à 2:29, Chris Lattner via swift-evolution <swift-evolution@swift.org> a écrit :

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}

--
Michel Fortin
https://michelf.ca

Hej Rick,

i definitely agree with you. +1

A big thing i like is that i could for example use already written projects in different languages which are maybe already in production and reuse their proven functionality without the requirement to transform the code to Swift.
The complexity of the Swift language itself isn't even touched in any area a “casual” developer uses, i personally see no argument of not integrating this feature for Swift.

I for myself can see a lot new possibilities and more reasons (and better arguments) for using Swift in other areas in my company like a normal phone app or some command-line tools.
Symbiosis is great lets have some fun with Swython, Swuby or SwHP in future.

···

From my personal experience i can say that a good working language interoperability gives us tons of new possibilities.

--
Stefan Mayer-Popp

On 15. Nov 2017, at 11:08, Rick Mann via swift-evolution <swift-evolution@swift.org> wrote:

+1 to both proposals. I don't know enough to comment further, but I know this would help in the adoption of Swift on my team(s) (we call a lot of Python, although currently via exec-ing some bash scripts; it's a mess, but works well in the CLI world the stuff was written for).

On Nov 14, 2017, at 23:29 , Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

  • Proposal: SE-NNNN
  • Author: Chris Lattner
  • Review Manager: TBD
  • Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

   a = someValue.someMember

   someValue.
someMember = a
and have it be interpreted by the compiler as:

a = someValue[dynamicMember: "someMember"
]
someValue[
dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties and underlying messaging infrastructure). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal

Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog
:

def __init__(self, name
):

self.name =
name

self.tricks = [] # creates a new empty list for each dog

def add_trick(self, trick
):

self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

// import DogModule
// import DogModule.Dog as Dog // an alternate
let Dog = Python.import(“DogModule.Dog")

// dog = Dog("Brianna")
let dog = Dog("Brianna")

// dog.add_trick("Roll over")
dog.add_trick("Roll over")

// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

// import pickle
let pickle = Python.import("pickle"
)

// file = open(filename)
let file = Python.open
(filename)

// blob = file.read()
let blob = file.read
()

// result = pickle.loads(blob)
let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal the code would have explicit member lookups all over the place:

// import pickle
let pickle = Python.get(member: "import")("pickle"
)

// file = open(filename)
let file = Python.get(member: "open"
)(filename)

// blob = file.read()
let blob = file.get(member: "read"
)()

// result = pickle.loads(blob)
let result = pickle.get(member: "loads"
)(blob)

// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol
{

associatedtype DynamicMemberLookupValue

subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
}
}

It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal
{

subscript(dynamicMember member: String) ->
PyVal {

get
{

let result = PyObject_GetAttrString(borrowedPyObject, member)!

return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
   }

set
{

PyObject_SetAttrString
(borrowedPyObject, member,
                            newValue.
toPython().borrowedPyObject
)
   }
}
}

Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol
{

associatedtype DynamicMemberLookupValue

// gettable only
subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get
}
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol
{

// gettable and settable.
subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
}
}

This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

How would this work for a property access that caused a python exception to be thrown? Are we assuming that only "good" non-throwing properties will be bridged in this way?

>>> x = "hello"
>>> x.foobar
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'foobar'

The prototype that I’m working on (which I’ll send out at some point soon) defaults to assuming that calls, member lookups, etc are all non-throwing. There are multiple ways to model this, and I’m not super attached to this approach, but here is how the current prototype works:

If you’d like to write code that is throwing, a modifier is used to get it reflected as a Swift optional or error. e.g.

  let x : PyVal = “hello”
  x.foobar // traps.
  x.checking.foobar // returns optional.

We could also use a postfix operator to make this nicer if common, e.g.:
  x^.foobar

But I’d like to avoid that if at all possible.

The way this works is that there are two types, “PyVal” is the currency type for python values. The “x.checking” invocation returns a “CheckedPyVal” wrapper around the PyVal. Both PyVal and CheckedPyVal conform to the two Dynamic* protocols: the former traps on error, the later produces an optional or throws (for calls).

-Chris

···

On Nov 15, 2017, at 1:12 PM, Nathan Gray via swift-evolution <swift-evolution@swift.org> wrote:

On Tue, Nov 14, 2017 at 11:29 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduction>Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

    a = someValue.someMember
    someValue.someMember = a
and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"]
  someValue[dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties <https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html> and underlying messaging infrastructure <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html>). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html>, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#motivation-and-context>Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities <https://pandas.pydata.org/> and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = [] # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> the code would have explicit member lookups all over the place:

  // import pickle
  let pickle = Python.get(member: "import")("pickle")

  // file = open(filename)
  let file = Python.get(member: "open")(filename)

  // blob = file.read()
  let blob = file.get(member: "read")()

  // result = pickle.loads(blob)
  let result = pickle.get(member: "loads")(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}
It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#example-usage>Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}
<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternatives-considered>Alternatives considered

A few alternatives were considered:

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#add-ability-to-provide-read-only-members>Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol {
  associatedtype DynamicMemberLookupValue

  // gettable only
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get }
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol {
  // gettable and settable.
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
  }
This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

<https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#naming>Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Functional Programmer, iOS Developer, Surfs Poorly
http://twitter.com/n8gray_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Just a question about what it mean for API resilience. Actually, adding a property to an existing class is not a breaking change.

For a class that implements that protocol, as it will be possible to use property accessor with any non existing property. Adding a new property to this class will now change the class behavior in unexpected way as the old client will still call the dynamic lookup method, but new code will use the new accessor method. Is it something that should be mention in the « Effect on API resilience » section ?

Right. Sure, I’ll add a mention of that.

-Chris

···

On Nov 15, 2017, at 2:52 PM, Jean-Daniel via swift-evolution <swift-evolution@swift.org> wrote:

Le 15 nov. 2017 à 08:29, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> a écrit :

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Swift has the backtick syntax for escaping keywords (`identifier`), but it doesn't allow non-identifier characters to be used in identifiers. We might explore loosening that.

···

On Nov 15, 2017, at 8:35 PM, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

Q: Is there any special handling for that member name string — to handle ruby method names like `blank?`, for example?

A: This should be the job of a language-specific interop layer.

--
Brent Royal-Gordon
Architechies

Key paths could be made to work by implementing the protocol in terms of retrieving key paths instead:

protocol SupplementalKeyPathDispatching {
  associatedType KP: PartialKeyPath<Self>
  static func supplementalKeyPath(for name: String) -> KP
}

and in your implementation of PyVal:

extension PyVal: SupplementalKeyPathDispatching {
  static func supplementalKeyPath(for name: String) -> KeyPath<PyVal, PyVal> {
    return \PyVal[dynamicMember: name]
  }
}

The compiler then implements two transforms:

  \PyVal.add_trick
  // becomes:
  PyVal.supplementalKeyPath(for: "add_trick")
  // returns key path \PyVal[dynamicMember: name]

and:

  let Dog = Python.import(“DogModule.Dog")
  let dog = Dog("Briana")

  dog.add_trick("snore")
  // becomes:
  dog[keyPath: \PyVal.add_trick]("snore")
  // or fully expanded:
  dog[keyPath: PyVal.supplementalKeypath(for: "add_trick")]("snore")
  // calls dog[dynamicMember: name]

You now have one more level of indirection which allows key paths. There's no compile time checking however: all imaginable key paths will work.

···

Le 16 nov. 2017 à 1:00, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> a écrit :

* Actually, for that matter, let's talk about key paths. In principle, you can already think of member lookup in Swift—or at least property and subscript lookup—as though it always worked by constructing a key path and using `subscript(keyPath:)` to access it. Is there some way we could model this feature as extending the set of keys available on a given type—perhaps in a way that allowed compile-time-limited and strongly-typed sets of keys, like I mention with the `UIAppearance` example, in addition to the open-ended, type-erased sets you need—and then looking things up by key path? (Sorry if this is a little vague—I know very little about how key paths are implemented.)

--
Michel Fortin
https://michelf.ca

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}

This looks great for Python, but let's talk about some other languages for a moment.

* Ruby and Perl don't have the "call a method by fetching a closure property and invoking it" behavior you're relying on here. Instead, Ruby has a syntax for settable "overloads" of methods (i.e. you can write `def someMember` and `def someMember= (newValue)`), while Perl supports lvalue methods (but sometimes uses getter and setter method pairs instead). How do you envision these behaviors being bridged to Swift? I worry that this protocol may not be sufficient, and that we may need a design which can distinguish between looking up methods and looking up properties.

* Ruby looks up members using symbols, which essentially play the same role as selectors in Objective-C—they're uniqued strings which are used for fast member dispatch. In some cases, you might see non-negligible speed improvements by only looking up the symbol once. Is there a way this design could accommodate that? For instance, could the type of the index be specified by an associated type, so Ruby could use a RbSymbol instead of a String? Or do you think that would be overkill?

And how would that works ? The actual proposal introduce a minor change in the compiler as it just has to be capable of generating a literal string from an identifier. To use something else (RbSymbol), the compiler would have to lean how to generate such object from the unresolved identifier.

One generic solution would be to make sure the key is buildable by string literal, but what would be the benefit here as it would not be better than simply create the RbSymbol directly in the specific subscript implementation.

···

Le 16 nov. 2017 à 07:00, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> a écrit :

On Nov 14, 2017, at 11:29 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

* Generally, you've talked about properties (in this proposal) and methods (in the `DynamicCallable` proposal), but what about subscripts? Obviously you can just specify a `subscript(Pythonable) -> PyVal` on `PyVal` for the simple case, but what if the subscript takes multiple indices or has labels? Do we need a `DynamicSubscriptable` protocol?

* Let's step away from bridging entirely and just think about Swift for a moment. There are cases where we'd like to make *semi*-dynamic proxies which wrap another type and allow operations based on what's statically known about that type. Think, for example, of the appearance proxy in UIKit: This is an object attached to UIView subclasses which lets you (in essence) set default values for all instances. We currently just pretend it's an instance of `Self`, which mostly works because of Objective-C, but a Swift-native version would probably prefer to return a `UIAppearance<Self>` object which used its knowledge of `Self` to expose `Self`'s properties on itself. Is there a way we could design this feature, or a related feature, to cover that kind of use case? That is, to allow a limited set of keys—perhaps even key-path-based when you want static control—with a different type for each key, *or* to allow any key with some common type, depending on your type's needs?

* Actually, for that matter, let's talk about key paths. In principle, you can already think of member lookup in Swift—or at least property and subscript lookup—as though it always worked by constructing a key path and using `subscript(keyPath:)` to access it. Is there some way we could model this feature as extending the set of keys available on a given type—perhaps in a way that allowed compile-time-limited and strongly-typed sets of keys, like I mention with the `UIAppearance` example, in addition to the open-ended, type-erased sets you need—and then looking things up by key path? (Sorry if this is a little vague—I know very little about how key paths are implemented.)

* An implementation-level question about Swift: Internally, the compiler seems to be moving towards thinking of parameter labels as part of the identifier, rather than having them label the individual arguments. How do you see that jibing with what you're proposing here?

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Arg, I keep sending these with the wrong from address and getting bounced! Excited for Discourse…]

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}

This looks great for Python, but let's talk about some other languages for a moment.

* Ruby and Perl don't have the "call a method by fetching a closure property and invoking it" behavior you're relying on here. Instead, Ruby has a syntax for settable "overloads" of methods (i.e. you can write `def someMember` and `def someMember= (newValue)`), while Perl supports lvalue methods (but sometimes uses getter and setter method pairs instead). How do you envision these behaviors being bridged to Swift? I worry that this protocol may not be sufficient, and that we may need a design which can distinguish between looking up methods and looking up properties.

I’ve never pried the lid of Ruby’s implementation of method dispatch, but I’m pretty sure that if foo defines a bar method, then

    foo.bar(…args…)

is fully equivalent to:

    foo.method(:bar).call(…args…)

IOW, there is an intermediate Method object that would fit the shape of the proposed callable protocol.

If foo instead doesn’t actually declare the bar method, but instead handles it via method_missing or __send__, then foo.method(:bar) raises an exception. However, it would be trivial to write a deferred invocation wrapper that also fits the shape of the proposed protocols and calls foo.send(“bar”, …args…) at the appropriate time.

In short, I don’t think there’s a problem here.

In the example you bring up:

you can write `def someMember` and `def someMember= (newValue)`)

…there is no overloading. The = is _part of the method name_, i.e. there is a `someMember` method and a `someMember=` method. The following are equivalent:

    foo.bar = 3 # just sugar
    foo.bar=(3)
    foo.send("bar=", 3)

Ruby allows ?, !, and = as the last char of method names, and AFAIK other than the special sugar around setters, they are just parts of the method name with no further semantic significance.

* Ruby looks up members using symbols, which essentially play the same role as selectors in Objective-C—they're uniqued strings which are used for fast member dispatch. In some cases, you might see non-negligible speed improvements by only looking up the symbol once. Is there a way this design could accommodate that? For instance, could the type of the index be specified by an associated type, so Ruby could use a RbSymbol instead of a String? Or do you think that would be overkill?

Good question. The language gets its symbol speedup by canonicalizing symbols at compile time when possible, and that failing, when symbolicated instead of when dispatched:

    one_zillion.times { foo.some_long_method_name } # fastest

    # only ~1.5x slower than prev, despite one extra method dispatch:
    one_zillion.times { foo.send(:some_long_method_name) }

    # performance identical to previous:
    symbol = string.to_sym
    one_zillion.times { foo.send(symbol) }

    # ~2x slower:
    one_zillion.times { foo.send(string) }

(Those are based on actual benchmarks.)

Swift should be able to do the same optimizations. Making the method name a string would preclude that.

* Generally, you've talked about properties (in this proposal) and methods (in the `DynamicCallable` proposal), but what about subscripts? Obviously you can just specify a `subscript(Pythonable) -> PyVal` on `PyVal` for the simple case, but what if the subscript takes multiple indices or has labels? Do we need a `DynamicSubscriptable` protocol?

* Let's step away from bridging entirely and just think about Swift for a moment. There are cases where we'd like to make *semi*-dynamic proxies which wrap another type and allow operations based on what's statically known about that type. Think, for example, of the appearance proxy in UIKit: This is an object attached to UIView subclasses which lets you (in essence) set default values for all instances. We currently just pretend it's an instance of `Self`, which mostly works because of Objective-C, but a Swift-native version would probably prefer to return a `UIAppearance<Self>` object which used its knowledge of `Self` to expose `Self`'s properties on itself. Is there a way we could design this feature, or a related feature, to cover that kind of use case? That is, to allow a limited set of keys—perhaps even key-path-based when you want static control—with a different type for each key, *or* to allow any key with some common type, depending on your type's needs?

Per my question about whether native methods shadow dynamic ones, one might be able to achieve some of this using a mix of statically typed, statically declared methods + dynamic members.

···

On Nov 16, 2017, at 12:00 AM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Nov 14, 2017, at 11:29 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

* Actually, for that matter, let's talk about key paths. In principle, you can already think of member lookup in Swift—or at least property and subscript lookup—as though it always worked by constructing a key path and using `subscript(keyPath:)` to access it. Is there some way we could model this feature as extending the set of keys available on a given type—perhaps in a way that allowed compile-time-limited and strongly-typed sets of keys, like I mention with the `UIAppearance` example, in addition to the open-ended, type-erased sets you need—and then looking things up by key path? (Sorry if this is a little vague—I know very little about how key paths are implemented.)

* An implementation-level question about Swift: Internally, the compiler seems to be moving towards thinking of parameter labels as part of the identifier, rather than having them label the individual arguments. How do you see that jibing with what you're proposing here?

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Interesting! I like the spirit of this proposal a lot.

One question: presumably this behaves like Ruby’s method_missing in that any natively implemented members shadow the dynamic ones? i.e. Swift looks for a static match first, then uses the dynamicMember only as a fallback?

Right. I added that to the proposal in the “API resilience section”.

Q: It seems like this fundamentally alters Swift’s aesthetic of “either an operation is type-safe, or it’s clear at the point of use that it’s not.” Should this use an operator other than a period, e.g. `pickle->loads(blob)`?

AnyObject dispatch is the closest relative to this feature that Swift currently has, and is neither type safe nor unsurprising. :-)

It is important to note that implementations of this protocol can definitely implement it in terms of optionals, so it is completely type safe. Consider a JSON implementation for example: the dynamic getter would return a type of “JSON?”. That design is fully type safe.

Q: Why the open-ended `associatedtype DynamicMemberLookupValue`? Seems like it will always just be Self in practice.

A: It would be Self in the Python and JSON examples in the proposal, but making it an associatedtype adds no implementation burden, does no apparent harm, and adds flexibility. Shifting types on traversal could be particularly useful in creating DSLs (e.g. sepia → Genus, sepia.officinalis → Species).

Not necessarily the same. Consider a bad use of this for sugaring a string to int dictionary. The string “keys” would be the “dynamicMember", but the DynamicMemberLookupValue would be the key type: Int.

Q: Is there any special handling for that member name string — to handle ruby method names like `blank?`, for example?

Q: Should the subscript also take arg types and/or labels to allow overloading?

A: Ruby, Python, and JS all resolve members by name alone, and leave it to functions to untangle their own args. Obj-C is the lone oddball here. Relying on the companion proposal to handle the args makes sense.

AFAIK, Swift’s subscript model is already general enough to do what we need it to do, but if there is some interesting language that isn’t served by it, we can talk about that when it comes up.

-Chris

···

On Nov 15, 2017, at 8:35 PM, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:
A: Use the standard Swift backquote mechanism: x.`blank?`()

Right.

-Chris

···

On Nov 15, 2017, at 9:23 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 15, 2017, at 8:35 PM, Paul Cantrell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Q: Is there any special handling for that member name string — to handle ruby method names like `blank?`, for example?

A: This should be the job of a language-specific interop layer.

Swift has the backtick syntax for escaping keywords (`identifier`), but it doesn't allow non-identifier characters to be used in identifiers. We might explore loosening that.

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}

This looks great for Python, but let's talk about some other languages for a moment.

* Ruby and Perl don't have the "call a method by fetching a closure property and invoking it" behavior you're relying on here. Instead, Ruby has a syntax for settable "overloads" of methods (i.e. you can write `def someMember` and `def someMember= (newValue)`), while Perl supports lvalue methods (but sometimes uses getter and setter method pairs instead). How do you envision these behaviors being bridged to Swift? I worry that this protocol may not be sufficient, and that we may need a design which can distinguish between looking up methods and looking up properties.

I’m not sure without more description, but it seems like these are both general property models which can be implemented with this protocol as is. The getter would call someMember, and the setter would call someMember=. What complication are you anticipating?

* Ruby looks up members using symbols, which essentially play the same role as selectors in Objective-C—they're uniqued strings which are used for fast member dispatch. In some cases, you might see non-negligible speed improvements by only looking up the symbol once. Is there a way this design could accommodate that?

Yes, I think that this is very much in scope for the DynamicCallable proposal. We talked about this in the context of Squeak (another smalltalky language), and it fits naturally with the design. I’ll add that when I have time to revise the pitch.

* Generally, you've talked about properties (in this proposal) and methods (in the `DynamicCallable` proposal), but what about subscripts? Obviously you can just specify a `subscript(Pythonable) -> PyVal` on `PyVal` for the simple case, but what if the subscript takes multiple indices or has labels? Do we need a `DynamicSubscriptable` protocol?

Subscripts can already be varargs, so the only reason we’d need a DynamicSubscriptable proposal is if there were another language that allowed keywords on subscript indices. If so, then yes, we’d need that.

* Let's step away from bridging entirely and just think about Swift for a moment. There are cases where we'd like to make *semi*-dynamic proxies which wrap another type and allow operations based on what's statically known about that type. Think, for example, of the appearance proxy in UIKit: This is an object attached to UIView subclasses which lets you (in essence) set default values for all instances. We currently just pretend it's an instance of `Self`, which mostly works because of Objective-C, but a Swift-native version would probably prefer to return a `UIAppearance<Self>` object which used its knowledge of `Self` to expose `Self`'s properties on itself. Is there a way we could design this feature, or a related feature, to cover that kind of use case? That is, to allow a limited set of keys—perhaps even key-path-based when you want static control—with a different type for each key, *or* to allow any key with some common type, depending on your type's needs?

I wouldn’t be at all surprised if this was possible, but given that this type erases everything that goes through the proxy, and given that it doesn’t provide type checking, it isn’t clear to me that you’d get a really great answer.

* Actually, for that matter, let's talk about key paths. In principle, you can already think of member lookup in Swift—or at least property and subscript lookup—as though it always worked by constructing a key path and using `subscript(keyPath:)` to access it. Is there some way we could model this feature as extending the set of keys available on a given type—perhaps in a way that allowed compile-time-limited and strongly-typed sets of keys, like I mention with the `UIAppearance` example, in addition to the open-ended, type-erased sets you need—and then looking things up by key path? (Sorry if this is a little vague—I know very little about how key paths are implemented.)

I don’t know.

* An implementation-level question about Swift: Internally, the compiler seems to be moving towards thinking of parameter labels as part of the identifier, rather than having them label the individual arguments. How do you see that jibing with what you're proposing here?

This is part of the DynamicCallable proposal that I will be revising as mentioned above, it isn’t related to the member lookup side of the equation.

-Chris

···

On Nov 15, 2017, at 10:00 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

On Nov 14, 2017, at 11:29 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I think this protocol and its subscript need a better name. From a user of the class point of view, the subscript looks like you can retrieve the members of the class, whereas in reality you'll only get the ones you have manually implemented. Even methods and properties having the `dynamic` attribute won't be available, even though the subscript name would suggest that.

I would propose adding the word `supplemental`to the subscript name and the name of the protocol to make it clearer that this is only for adding members that are not declared in the class (including the `dynamic` ones).

As in `SupplementalDynamicMemberLookupProtocol` and `supplementalDynamicMember` for the subscript.

I’m totally open to suggestions for a better name, but I don’t see what “Supplemental” is adding here. Recall that this protocol is compiler “magic” that is part of an implementation of a type, it isn’t really part of the user-exposed API of the type. I wouldn’t expect a user to ever write:

   pyVal[dynamicMember: “foo”]

Even though they could. Maybe we need to add private conformances to Swift or something :-)

-Chris

···

On Nov 16, 2017, at 4:49 AM, Michel Fortin via swift-evolution <swift-evolution@swift.org> wrote:

Le 15 nov. 2017 à 2:29, Chris Lattner via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> a écrit :

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}

--
Michel Fortin
https://michelf.ca <https://michelf.ca/>
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Really interesting proposal,
I don't really have the need for Python interop at the moment so I
would be more interesting on the potential usages of this changes to
improve Swift code like the proxy and json accessors that you mention.
The JSON part I can see it, as you are basically accessing with
strings a keyed data structure.
But I'm having a hard time imagining how a proxy example would work as
there doesn't seem to be a mechanism to access Swift members from the
information received on the subscript. I pressume you can call your
proxied object subscript but that would mean that the proxied objects
need to be also DynamicMemberLookupProtocol and in any case at some
point down the chain somebody will have to dispatch from the "string"
to a member lookup with some switch or similar.
Am I missing something or is this just completely unrelated to the
proposal and I just left my imagination go to far? :stuck_out_tongue:

···

On Wed, Nov 15, 2017 at 10:58 AM, Stefan Mayer-Popp via swift-evolution <swift-evolution@swift.org> wrote:

Hej Rick,

i definitely agree with you. +1

From my personal experience i can say that a good working language interoperability gives us tons of new possibilities.

A big thing i like is that i could for example use already written projects in different languages which are maybe already in production and reuse their proven functionality without the requirement to transform the code to Swift.
The complexity of the Swift language itself isn't even touched in any area a “casual” developer uses, i personally see no argument of not integrating this feature for Swift.

I for myself can see a lot new possibilities and more reasons (and better arguments) for using Swift in other areas in my company like a normal phone app or some command-line tools.
Symbiosis is great lets have some fun with Swython, Swuby or SwHP in future.

--
Stefan Mayer-Popp

On 15. Nov 2017, at 11:08, Rick Mann via swift-evolution <swift-evolution@swift.org> wrote:

+1 to both proposals. I don't know enough to comment further, but I know this would help in the adoption of Swift on my team(s) (we call a lot of Python, although currently via exec-ing some bash scripts; it's a mess, but works well in the CLI world the stuff was written for).

On Nov 14, 2017, at 23:29 , Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible. My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole. Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings. I’ll share them in the next day or two in case they are useful to provide context. It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability. The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.

Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438

A snapshot of the proposal is included below in case it is useful. Thanks in advance for help improving the proposal!

-Chris

Introduce User-defined "Dynamic Member Lookup" Types

     • Proposal: SE-NNNN
     • Author: Chris Lattner
     • Review Manager: TBD
     • Status: Awaiting implementation
Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

   a = someValue.someMember

   someValue.
someMember = a
and have it be interpreted by the compiler as:

a = someValue[dynamicMember: "someMember"
]
someValue[
dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties and underlying messaging infrastructure). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal

Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog
:

def __init__(self, name
):

self.name =
name

self.tricks = [] # creates a new empty list for each dog

def add_trick(self, trick
):

self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

// import DogModule
// import DogModule.Dog as Dog // an alternate
let Dog = Python.import(“DogModule.Dog")

// dog = Dog("Brianna")
let dog = Dog("Brianna")

// dog.add_trick("Roll over")
dog.add_trick("Roll over")

// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

// import pickle
let pickle = Python.import("pickle"
)

// file = open(filename)
let file = Python.open
(filename)

// blob = file.read()
let blob = file.read
()

// result = pickle.loads(blob)
let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal the code would have explicit member lookups all over the place:

// import pickle
let pickle = Python.get(member: "import")("pickle"
)

// file = open(filename)
let file = Python.get(member: "open"
)(filename)

// blob = file.read()
let blob = file.get(member: "read"
)()

// result = pickle.loads(blob)
let result = pickle.get(member: "loads"
)(blob)

// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol
{

associatedtype DynamicMemberLookupValue

subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
}
}

It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal
{

subscript(dynamicMember member: String) ->
PyVal {

get
{

let result = PyObject_GetAttrString(borrowedPyObject, member)!

return PyRef(owned: result) // PyObject_GetAttrString returns +1 result.
   }

set
{

PyObject_SetAttrString
(borrowedPyObject, member,
                            newValue.
toPython().borrowedPyObject
)
   }
}
}

Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol
{

associatedtype DynamicMemberLookupValue

// gettable only
subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get
}
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol
{

// gettable and settable.
subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
}
}

This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Alejandro Martinez
http://alejandromp.com

Terms of Service

Privacy Policy

Cookie Policy