Automatic Mutable Pointer Conversion

Evolution Pitches
21

Seems like the knub of this problem (as with the conversation with previous conversations about conversions) is concerns over there being any possibility of slowing the type check checker down. I don’t write compilers for a living but I'd like to re-iterate there is an interesting aspect of this particular PR in that I put the detection of the conversion in the routine ConstraintSystem::repairFailures() which if I understand correctly is called at a point where type checking fails and looks for potential fall backs. In this way I thought I could be reasonably confident that it couldn't possibly slow down the compilation of an already valid program. Correct me if I'm wrong on this.

As a suggestion, is it possible this approach could be used for implicit conversions i.e. they are only looked up on a type if all else fails? This reduces the flexibility of the feature in that you would be able to call methods on CGFloat for example if a value had Double type nor would you be able to access its conformances which is likely an excess of flexibility or ambiguity but it should mitigate the risk of a regression of type checker performance. You would however be able to pass a double to a function or operator expecting CGFloat which is what most people are after.

Would this result in a half-feature users would find difficult to anticipate the limitations of? Finding a limited scope for implicit conversions seems important if it is ever to be realised and we can begin to entertain other safe conversions such as Int8 -> Int.

22

To clarify, I am not concerned about type checker performance for this specific implicit conversion.

I don't necessarily think this is the main problem for a general feature, but it is a problem that could prevent a very attractive design from being feasible. We're not even at the point where we know what a design would look like, which is why I said that is a big discussion and it shouldn't hold up this pitch.

When working in the constraint system, it's very easy to have a narrow of a view of the piece of code you're working on, and miss the implications on the larger system. Speaking from experience :slightly_smiling_face:

Most (or all?) implicit conversions today don't actually introduce new disjunctions, so it might seem like the implicit conversion won't have an impact on existing code. For a particular overload that doesn't need the implicit conversion, that's true. But for some other overload that already exists that today fails immediately in the solver, the implicit conversion could actually make the expression type-check using that overload. Implicit conversions expand the search space even if they don't themselves introduce disjunctions, because they make other existing possibilities well-formed. This is also what leads to ambiguities. Maybe only one overload worked before, but now several overloads are valid, and the solver has to find all of them and determine which one is best. If the solution scores are the same and none of the overloads are more "specialized" than the others, that then becomes an ambiguity error.

After this explanation, you might think that the solver should short-circuit disjunctions if it has already found a solution. The solver already does this in some cases today in an attempt to improve performance, but it's a hack, and it leads to incorrect behavior and sometimes prevents valid code from compiling because the solver will miss a solution that ends up having a better score.

This is already how implicit conversions work today:

extension Optional {
  func someMethod() {}
}

func test(a: Int?, b: Int) {
  a.someMethod()
  b.someMethod() // error: Value of type 'Int' has no member 'someMethod'
  (b as Optional).someMethod() // okay
}
2 Likes
23

@hborla I think there is a typo on the last line, IIUC for (value-to-optional) implicit conversion that would be (b as Optional).someMethod() right?

1 Like
24

Oops, you’re right I meant to show calling someMethod on b. Fixed, thank you!

1 Like
25

I 100% agree with this argument! I do recognize that there are many precedents of poor implicit conversion implementations, but
That doesn’t necessarily mean that the whole concept of implicit conversion is a bad idea. It merely means that the implementation was bad. I fully support the idea of discussing this topic in depth and at least putting in some effort into designing a custom subtyping mechanism before deeming the whole concept as a lost cause.

26

It would be great if the current optional promotion rules could be documented formally. Implicit promotion of Optional so far each of the different subtyping rules seems to have different rules. I love the idea of a general DAG approach.

1 Like
27

Yes, I completely agree.

My sense is that this is too late to get into Swift 5.5, and there is no published plan for subsequent releases. If true, I think we have some time to discuss the alternatives now, and if there is a pressing reason to get it into some looming release, we can discuss that near the end of the cycle.

As I mentioned above, the simplest thing to do is to add a mapping table with an attribute to add to it. I think we can do better than this though.

-Chris

1 Like
28

I hope my post is on topic.

As a person who deals with C APIs, I look forward to such a feature. I am "on the fence" about implicit conversion, but I would love to have the ability to use an explicit conversion. My intended usage is having the ability to mimic the Foundation API.

Example with String:

let str = "Hello"
fAPI(str as NSString)
cfAPI(str as CFString)

Such conversion should have full support for generics and should support "omitting Optional".
I have met with C APIs, which have a convention, that nil/NULL is a valid "instance" of an empty collection. One such example may be GList (glib/glist.c · main · GNOME / GLib · GitLab).

It would be great if I could design a wrapper, what would allow me to express something like this:

func emptyList() -> Optional<UnsafeMutablePointer<linked_list_t>> {
    nil
}

// assuming hypothetical `linked_list_t` contains type information
let result = emptyList() as Array<String>
print(result) // "[]"

// or if it does not
let result = emptyList() as Array<UnsafeMutableRawPointer>
print(result) // "[]"

Even if it would be impossible to support generics and/or "omit optional", a must-have feature for me is, that it should not have the same restrictions as adding protocol conformance in extension.

Now, I am not able to pass a Swift array to an argument expecting instance of linked_list_t, because such linked_list_t can not be extended to conform to ExpressibleByArrayLiteral (since it was declared in the C API).

I would like to have the ability to convert (explicitly) array:

func passList(_ arg: UnsafeMutablePointer<linked_list_t>) { /* nop */ }

passList(["A", "B"])

Edit:

Essentially, I would like to have the ability to create a "bridging code" in Swift such as swift/Pointer.swift at 700b11daef05e58db050dcd57b0e6433faeca9d7 · apple/swift · GitHub .
Would it be possible to treat as as an operator and declare each conversion as a public global function?

29

As somebody who implemented SE-0307 I'd like to share a couple of insights about implicit conversions when it comes to a type-checker performance.

All of the following are empirically proven to lead to a bad performance:

  • Allowing (pure) bi-directional conversions. In case of Double/CGFloat it led to ambiguities even in relatively simple expressions.
  • Allowing multiple conversions in the same position e.g. converting from Double to CGFloat and back to Double.
  • Allowing conversions in argument positions, especially for homogeneous operators (which allow of a sudden became heterogeneous) because it leads to unexpected results. Double/CGFloat managed to (somewhat) work around that by preferring Double over CGFloat.

I'm not going to re-iterate what @hborla and @codafi already said (and with which I agree), just want to add that while working on SE-0307 I did consider possible ways to implement that particular conversion in a more general way that could be extended to widening of integers (this is also discussed in pitch for Double/CGFloat conversion as possible future direction) but the problem is that even widening has big performance implications although it's scoped down to just a handful of types.

I think if we were to consider a general feature here we'd have to put so many restrictions on its syntactic use that it couldn't be called "general" in a normal sense and wouldn't be worth the complexity.

Now let's consider one possible solution as implemented in the associated PR. The idea is to add a non-failable initializer for each possible conversion type could have e.g.

extension UnsafeRawPointer {
  init(implicit: UnsafeMutableRawPointer) { ... }
}

Could be used to implement uni-directional conversion from UnsafeMutableRawPointer to UnsafeRawPointer. Without bike-shedding to much on the syntax let's consider what the semantics might be:

  • Any struct, class, enum (or nominal for that matter) should be convertible to other type if it has an initializer in form of init(implicit: T) where T is the "from" conversion type.

Which means that every Conversion (of any type e.g. argument, operator argument, regular) or Subtype relationship in the type system now has to check whether right-hand side has an initializer that accepts left-hand type as an argument. This new rule could be expressed in a following set of constraints:

Starting from:

$From <convertible to> $To

Simplifies into:

$To[.member == init(implicit:)] == $Init
($From) -> $To <applicable function> $Init

Note that after lookup $Init could be bound to a number of choices with different types of arguments (important - including failable initializers) if $To is convertible to N different types.

After $Init is determined it's possible to simplify ($From) -> $To <applicable function> $Init into:

$From <argument conversion> $Init.$Arg0
$To <function result> $Init.$Result

Further simplification results in yet another [.member == init(implicit:)] ... check for $Init.$Arg0 and $From types if $To has multiple overloads of init(implicit:).

It could be reasonable to disable followup checks in "implicit conversion" context but that would restrict the generality of the feature.

This new rule impacts all expressions in the language non-trivially but having to perform at least one additional member lookup per any kind of conversion, even if a type doesn't support implicit conversions. It could be argued that we could use a declaration attribute to mark some of the types as @implicitlyConvertible to avoid having to pay the extra cost in the type-checker, but if we did that for some commonly used types e.g. to support widening conversions it would still impact a non-trivial amount of code and retroactively adding such an annotation might not be possible.

My suggestion would be to consider merit of an uni-directional conversion from UnsafeMutableRawPointer to UnsafeRawPointer just like we did Double/CGFloat and discuss general implicit conversions separately based on the knowledge gained from it and previous conversions.

8 Likes
30

Personally, I'm against adding this. Based on the discussion in this thread, it seems like we need to more carefully consider the space of options around implicit conversions. I don't like the idea of delving deeper down this rabbit hole without a clear long term vision for the language.

To play devil's advocate, one could argue that if we did add some general form of implicit conversions in the future (which for the record I'm also against, though I could perhaps be convinced otherwise) it would probably be source level compatible with this special case.

If you want a really "hot" take though, to the extent that all this boiler plate discourages interoperating with C code, that might actually be a good thing. It's extremely important that Swift can call C code, but C code should general be hidden inside an API with clear high level guarantees, since it is often trivial to invoke undefined behavior by calling C APIs directly.

31

I agree with you, except for:

If you want a really "hot" take though, to the extent that all this boiler plate discourages interoperating with C code, that might actually be a good thing. It's extremely important that Swift can call C code, but C code should general be hidden inside an API with clear high level guarantees, since it is often trivial to invoke undefined behavior by calling C APIs directly.

Using C from Swift is not an uncommon thing, even on the Apple platforms. If you control the C API you can take advantage of wide variety of macros and attributes that allow you to create a fluent Swift API for your C code. (Example)
If you "control" the compiler itself, you can take the whole thing into another level with bridging.

A lot of popular Swift Packages use C targets or dependencies. I think that making the C interop more fluent and readable is a good thing.

I am not a fan of making things difficult in order to discourage people from using it. If you (or a developer of a dependency) make the decision to use a C code from Swift, you probably have no other option. You're probably not going to re-implement sqlite3 in Swift. And if your program does not work, purposely complicated syntax won't help you in solving the issue.

6 Likes
32

Rather than see this pitch wither on the vine, I’d like to try to keep things moving. It took an unexpected twist when Chris intervened and raised the bar to “could a design be found where implicit conversions be expressed in Swift rather than a sequence ad-hoc changes to the compiler itself” all this without slowing type checking down. I have come to be strongly supportive of this goal.

To this end, I’ve produce a special “Friday the 13th” toolchain making a start on implementing this feature and been unable to record it slowing the compiler down benchmarking repeated builds of the fairly substantial GitHub - grpc/grpc-swift: The Swift language implementation of gRPC. project.

https://drive.google.com/file/d/1wUgTREO6LeMxdqXsoCeotTNF3O81Gnb-/view?usp=sharing

The toolchain is the “diff” of the original PR applied to tag swift-5.5-DEVELOPMENT-SNAPSHOT-2021-08-11-a which means it is most closely related to the Swift in recent Xcode betas.

Resistance to this feature seems to come from two quarters I’d characterise as “Should we” and “Could we”. Opinions vary widely about whether implicit conversions could be a good thing, the common criticism being that they affect the coders ability to anticipate what code is executed by a given expression but I would argue the ship has already sailed on this in Swift (I’m talking about you computed properties, not to mention willSet and didSet observing patterns or destructors which can fire at practically any time for that matter.) Like operator overloading introducing implicit conversions is a double edged sword and could certainly be open to abuse but it’s part of moving functionality out of the compiler developers hands into the Swift space. There are already implicit conversions and there are pitches for more about but they can/should be expressed in Swift.

The second objection “Could we” which I suspect colours the first objection in some people's eyes I hope to put to bed with the prototype toolchain. Although the toolchain takes a naive approach grafted onto the last phase of type checking this may in fact be a feature. The type checker and its internal constraint solver is a powerful abstraction at the heart of Swift but in this case it may be holding us back. Let’s face it, as a result Swift is a bit precious about strict typing when there are safe conversions it could be performing. This would make developers lives easier as in this pitch for example as it started out or recently for CGFloat/Double convertibility or even just up-typing smaller integer types to Int. if this feature can in fact be accommodated by modifying the constraint solver, all well and good but that is out of my competence.

And so to the “design” of the feature in the prototype. It is very simple. To create an implicit conversion you add an initialiser in an extension in the “to type” that takes the “from type” as an argument with a label implicit:. This design, using initialisers rather than conversion functions on the “from type” was a by-product of the implementation which re-uses the CGFloat<->Double code but it also seems to group conversions more intuitively. An example:

extension Int {
  init(implicit: Int16) {
    print("Int.init(implicit: Int16)")
    self.init(implicit)
  }
  init(implicit: Int32) {
    print("Int.init(implicit: Int32)")
    self.init(implicit)
  }
}
extension Int32 {
  init(implicit: Int8) {
    print("Int32.init(implicit: Int8)")
    self.init(implicit)
  }
  init(implicit: Int16) {
    print("Int32.init(implicit: Int16)")
    self.init(implicit)
  }
}

Note that the implementation does not chain conversions which constrains the search space. In this case Int8 is convertible to Int32 and Int32 is convertible to Int but Int8 is not convertible to Int as a result. As a result you can now make cyclic (i.e. non-DAG) conversions to make types interchangeable such as the new implementation of CFFloat<->Double:

extension Double {
  init(implicit: CGFloat) {
    print("Double.init(implicit: CGFloat)")
    self.init(implicit)
  }
}
extension CGFloat {
  init(implicit: Double) {
    print("CGFloat.init(implicit: Double)")
    self.init(implicit)
  }
}

A crinkle in the existing operation of the type checker is that to find a solution to coerce to use the new implicit constructor there needs to be an existing unlabelled conversion between the two types. If you want to make a completely new relationship between two types you need to provide such an initialiser. So, for example to make Doubles interchangeable with a Bool, you use the implicit label but in this case hidden:

extension Bool {
    init(_ implicit: Double) {
        self.init(implicit != 0.0)
    }
}

func smothingOperation(param: Double) {
    print("Smoothing \(param)")

}

let smoothingParam = 99.1
if smoothingParam {
    smothingOperation(param: smoothingParam)
}

This can all be bike-shedded and many will likely prefer a new @implicit annotation instead. I won’t pretend that crashing the toolchain isn’t possible as there is more work to do on handling generics but it serves its sole purpose: to prove that we should keep this conversation moving.

5 Likes
33

To summarise this long and rambling post all I wanted to say was let's not dismiss bringing a universal solution for bringing implicit conversions into the language on the grounds that "it can't be done without slowing down type checking". The solution may be a sort of lightweight post-processing stage outside of the type checker that can be optimised once. This is he point of the toolchain makes. Anyway, the conversation has moved to this thread if you want to continue reading.

34

Just FYI, I have started a topic in Evolution/Discussion to create a space to discuss generalization of implicit conversions - Generalization of Implicit Conversions.

3 Likes