Automatic Mutable Pointer Conversion

While it's true that full implicit conversions suffer from those pitfalls, I don't think that needs to exclude a narrower solution specifically for subtyping. In Swift, we already allow implicit conversions from a variable to an existential whose protocol it conforms to, or from a class to a superclass, or even from T to T?. For example, we allow this:

protocol UnsafePointerProtocol {}
extension UnsafeMutablePointer: UnsafePointerProtocol {}
let pointer: UnsafePointerProtocol = UnsafeMutablePointer<Int>.allocate(capacity: 1)

and equally, if UnsafePointer were a class, we'd allow:

class UnsafeMutablePointer: UnsafePointer {}
let pointer: UnsafePointer = UnsafeMutablePointer<Int>.allocate(capacity: 1)

Let's say that we hypothetically allowed something like:

struct UnsafeMutablePointer<Pointee>: @subtypeConvertible UnsafePointer<Pointee> {
    public func upcast() -> UnsafePointer<Pointee> {
        return UnsafePointer(self)
    }
}

I may be missing something, but I don't see how that's anything more harmful than the subtyping relationships that are already in the language. The issue I do see is that downcasts would not work as you'd expect; e.g.:

let pointer: UnsafePointer<Int> = UnsafeMutablePointer<Int>.allocate(capacity: 1)
let downcastPointer: UnsafeMutablePointer<Int>? = pointer as? UnsafeMutablePointer<Int> // nil

I don't know how to appropriately solve that for the UnsafePointer case, since it's not safe to convert an immutable pointer to a mutable pointer in general – I think the only reasonable answer is for that cast to always fail. For other cases, though, if type-checker and runtime performance allows it, we could optionally allow something like the following:

enum GPUResource {
    case texture(GPUTexture)
}

struct GPUTexture : @subtypeConvertible GPUResource {
    // required
    public func upcast() -> GPUResource {
        return .texture(self)
    }
 
    // optional, only used for `as?` casts
    public init?(downcastingFrom superType: GPUResource) -> GPUTexture? {
        switch superType { 
        case .texture(let texture):
            return texture
        default:
            return nil
        }
    }
}

I agree, user-defined conversions shouldn’t be a thing. We lose our ability to exercise local reasoning, which is one of the foundational pillars of Swift and one of the reasons for which we have value types and let bindings, and instead fall into the world of global reasoning, since we wouldn’t know in which file some contributor added some conversion.

I want to emphasize this. Personally, I think the additional axis of exponential search space is an unacceptable consequence of user-defined implicit conversions. It's important to realize that arbitrary implicit conversions would undermine many of the pruning heuristics that are in place in the constraint solver today to make simple cases fast.

For example, if there's no possibility that implicit conversions can add conformances to an argument (which is possible with an implicit conversion to a value type with different conformances, and is very rare today), conformance constraints on a parameter type can be transferred to the argument type directly, which often allows the constraint solver to prune search paths early. This heuristic allows the solver to fail before a generic argument type is attempted for generic overloads that "obviously" aren't satisfied by the given argument types. Generic arguments are only bound once the solver has attempted all disjunctions. So, in a case with several generic operators chained together, e.g. a + b + c + d, the solver has to first bind each + to an overload before any generic arguments are attempted, because only then does the solver have the complete set of possible bindings for those generic arguments. In the worst case, the solver attempts all permutations with repetition of the overloads. Without this pruning heuristic, expressions that use generic operators chained together (among other kinds of expressions) are subject to worst-case exponential behavior.

This is just one example. There are others. Another big issue is introducing more ambiguities. To resolve an ambiguity, you need to write explicit type information anyway.

Perhaps there are ways to make the constraint solver performance more immune to implicit conversions, but there is a lot of engineering work to be done before we get there. I don't think implicit conversions are feasible to implement with tolerable, let alone good type checker performance today.

8 Likes

If we were to enforce true subtyping – i.e. UnsafeMutablePointer<Pointee>: @subtypeConvertible UnsafePointer<Pointee> meaning that UnsafeMutablePointer inherits all of the conformances of UnsafePointer (and possibly conforms by default by upcast()ing to UnsafePointer) – how many of these issues still apply? Are there still more ambiguities introduced?
Likewise, would implementing integer widening by having a chain of UInt16: @subtypeConvertible UInt8, UInt32: @subtypeConvertible UInt16 etc. have a major impact on type-checker performance?

Right, Robert also makes a similar argument and points to challenges with C++ implicit conversions for example. I think we can all see the dangers of a poor design, and those of us who lived through the __conversion thing early on in Swift don't want to relive that.

However, I'm not suggesting we add C++ implicit conversions or __conversion to Swift. I'm observing several things:

  1. Swift has fully general user-defined conversions already in the form of protocol and class inheritance. These implicitly impose a DAG-based structure to the subtyping problem, among other restrictions.

  2. Swift already has a bunch of special case implicit conversions, e.g. T to T?, the CGFloat ones, and several others. We cannot remove these for source compatibility reasons.

  3. Because we have some of these in the language, people continue pointing out other cases where subtype relationships are useful to model, e.g. the latest is unsafe pointers.

  4. No one (that I'm aware of) has done a systemic study of what stdlib types should be implicitly convertible, and we don't have a principle guiding this. Why should UnsafePointer be implicitly convertible, but Int8 shouldn't convert to Int? Both are subtypes, so what should our principle be?

  5. One of the founding ideas of Swift is that much of the language is "in the library", and the reason for that is we want an expressive ecosystem of APIs that feel fluent and natural (e.g. all of "Float", "Complex" and "Quaternion" should be able to have consistent design approaches even if they are at different levels of the library stack). Continuing to privilege a few specific APIs like UnsafePointer breaks that.

  6. There are other use-cases we should address outside direct use by familiar APIs, e.g. in bridging of other languages, C++ interop, and more.

If Swift had a huge user base and did so without implicit ad-hoc conversions yet, then we'd have a strong argument that we don't them in the language. However, (for better or worse), we already have them, our users expect to not write silly casts in some cases, and we have people asking to add more special cases to the type checker. This doesn't make it simpler.

I see several possibilities looking ahead:

  1. We could draw the line where it is now forever and come up with reasons why CGFloat is ok but nothing else is.

  2. We could continue adding specific things like UnsafePointer in the standard library, building in more special cases into the constraint solver endlessly.

  3. We could provide a more general solution so this is extensible in a controlled way, and potentially subsume a bunch of the existing complexity in the type checker (the special cases, but also possibly other things like the magic argument promotions) into a more principled framework.

I think that approach 3 has a lot of merit, and there is tremendous design room around this. The most trivial thing would be to take the hardcoded mapping the constraint solver uses and making it more extensible (something that certainly wouldn't affect type checker performance more than adding other cases). It is also possible to add one step conversions. It is also possible to add something fancier like the DAG based precedence rules but for conversions.

What I am suggesting is that we explore this design space. I think that it makes sense to start by discussing requirements (e.g. on type system performance, problems we want to solve, limitations we want to impose, etc) before talking about solutions.

-Chris

10 Likes

Sure, I'd be happy to participate in that exploration. I do see the value in allowing a subtype relationship to be defined for certain types when it's "clearly" safe; I'm not opposed to the change pitched here, for example.

Another way to phrase my opinion is that constraint system performance has to be a critical consideration in any discussion about extensible implicit conversions, because a model that introduces another axis of exponential search space into the constraint system is just not practical. I'm also certainly open to being proven wrong about the impact that extensible implicit conversions would have on the constraint system, either with restrictions in the design or with a clever implementation strategy :slightly_smiling_face:

That is clearly a big discussion, though, and I don't think it should hold up this pitch. If mutable -> immutable pointer conversions are a useful addition to the language now, I don't think we should block it from moving forward based on a feature that may not even be feasible to ever add to Swift. I haven't seen evidence that these narrow implicit conversions are harmful to users (e.g. in their understanding of the type system) besides perhaps being mildly annoyed when they have to write an explicit initialization when it seems like it shouldn't be necessary, so I don't see a reason why we shouldn't continue using option 2 of blessing very narrow implicit conversions while option 3 is explored.

EDIT: (I suggest using an alternative term instead of 'whitelist', as there are more inclusive options. If anybody would like to discuss this further, please DM me)

4 Likes

Seems like the knub of this problem (as with the conversation with previous conversations about conversions) is concerns over there being any possibility of slowing the type check checker down. I don’t write compilers for a living but I'd like to re-iterate there is an interesting aspect of this particular PR in that I put the detection of the conversion in the routine ConstraintSystem::repairFailures() which if I understand correctly is called at a point where type checking fails and looks for potential fall backs. In this way I thought I could be reasonably confident that it couldn't possibly slow down the compilation of an already valid program. Correct me if I'm wrong on this.

As a suggestion, is it possible this approach could be used for implicit conversions i.e. they are only looked up on a type if all else fails? This reduces the flexibility of the feature in that you would be able to call methods on CGFloat for example if a value had Double type nor would you be able to access its conformances which is likely an excess of flexibility or ambiguity but it should mitigate the risk of a regression of type checker performance. You would however be able to pass a double to a function or operator expecting CGFloat which is what most people are after.

Would this result in a half-feature users would find difficult to anticipate the limitations of? Finding a limited scope for implicit conversions seems important if it is ever to be realised and we can begin to entertain other safe conversions such as Int8 -> Int.

To clarify, I am not concerned about type checker performance for this specific implicit conversion.

I don't necessarily think this is the main problem for a general feature, but it is a problem that could prevent a very attractive design from being feasible. We're not even at the point where we know what a design would look like, which is why I said that is a big discussion and it shouldn't hold up this pitch.

When working in the constraint system, it's very easy to have a narrow of a view of the piece of code you're working on, and miss the implications on the larger system. Speaking from experience :slightly_smiling_face:

Most (or all?) implicit conversions today don't actually introduce new disjunctions, so it might seem like the implicit conversion won't have an impact on existing code. For a particular overload that doesn't need the implicit conversion, that's true. But for some other overload that already exists that today fails immediately in the solver, the implicit conversion could actually make the expression type-check using that overload. Implicit conversions expand the search space even if they don't themselves introduce disjunctions, because they make other existing possibilities well-formed. This is also what leads to ambiguities. Maybe only one overload worked before, but now several overloads are valid, and the solver has to find all of them and determine which one is best. If the solution scores are the same and none of the overloads are more "specialized" than the others, that then becomes an ambiguity error.

After this explanation, you might think that the solver should short-circuit disjunctions if it has already found a solution. The solver already does this in some cases today in an attempt to improve performance, but it's a hack, and it leads to incorrect behavior and sometimes prevents valid code from compiling because the solver will miss a solution that ends up having a better score.

This is already how implicit conversions work today:

extension Optional {
  func someMethod() {}
}

func test(a: Int?, b: Int) {
  a.someMethod()
  b.someMethod() // error: Value of type 'Int' has no member 'someMethod'
  (b as Optional).someMethod() // okay
}
2 Likes

@hborla I think there is a typo on the last line, IIUC for (value-to-optional) implicit conversion that would be (b as Optional).someMethod() right?

1 Like

Oops, you’re right I meant to show calling someMethod on b. Fixed, thank you!

1 Like

I 100% agree with this argument! I do recognize that there are many precedents of poor implicit conversion implementations, but
That doesn’t necessarily mean that the whole concept of implicit conversion is a bad idea. It merely means that the implementation was bad. I fully support the idea of discussing this topic in depth and at least putting in some effort into designing a custom subtyping mechanism before deeming the whole concept as a lost cause.

It would be great if the current optional promotion rules could be documented formally. Implicit promotion of Optional so far each of the different subtyping rules seems to have different rules. I love the idea of a general DAG approach.

1 Like

Yes, I completely agree.

My sense is that this is too late to get into Swift 5.5, and there is no published plan for subsequent releases. If true, I think we have some time to discuss the alternatives now, and if there is a pressing reason to get it into some looming release, we can discuss that near the end of the cycle.

As I mentioned above, the simplest thing to do is to add a mapping table with an attribute to add to it. I think we can do better than this though.

-Chris

1 Like

I hope my post is on topic.

As a person who deals with C APIs, I look forward to such a feature. I am "on the fence" about implicit conversion, but I would love to have the ability to use an explicit conversion. My intended usage is having the ability to mimic the Foundation API.

Example with String:

let str = "Hello"
fAPI(str as NSString)
cfAPI(str as CFString)

Such conversion should have full support for generics and should support "omitting Optional".
I have met with C APIs, which have a convention, that nil/NULL is a valid "instance" of an empty collection. One such example may be GList (glib/glist.c · main · GNOME / GLib · GitLab).

It would be great if I could design a wrapper, what would allow me to express something like this:

func emptyList() -> Optional<UnsafeMutablePointer<linked_list_t>> {
    nil
}

// assuming hypothetical `linked_list_t` contains type information
let result = emptyList() as Array<String>
print(result) // "[]"

// or if it does not
let result = emptyList() as Array<UnsafeMutableRawPointer>
print(result) // "[]"

Even if it would be impossible to support generics and/or "omit optional", a must-have feature for me is, that it should not have the same restrictions as adding protocol conformance in extension.

Now, I am not able to pass a Swift array to an argument expecting instance of linked_list_t, because such linked_list_t can not be extended to conform to ExpressibleByArrayLiteral (since it was declared in the C API).

I would like to have the ability to convert (explicitly) array:

func passList(_ arg: UnsafeMutablePointer<linked_list_t>) { /* nop */ }

passList(["A", "B"])

Edit:

Essentially, I would like to have the ability to create a "bridging code" in Swift such as swift/Pointer.swift at 700b11daef05e58db050dcd57b0e6433faeca9d7 · apple/swift · GitHub .
Would it be possible to treat as as an operator and declare each conversion as a public global function?

As somebody who implemented SE-0307 I'd like to share a couple of insights about implicit conversions when it comes to a type-checker performance.

All of the following are empirically proven to lead to a bad performance:

  • Allowing (pure) bi-directional conversions. In case of Double/CGFloat it led to ambiguities even in relatively simple expressions.
  • Allowing multiple conversions in the same position e.g. converting from Double to CGFloat and back to Double.
  • Allowing conversions in argument positions, especially for homogeneous operators (which allow of a sudden became heterogeneous) because it leads to unexpected results. Double/CGFloat managed to (somewhat) work around that by preferring Double over CGFloat.

I'm not going to re-iterate what @hborla and @codafi already said (and with which I agree), just want to add that while working on SE-0307 I did consider possible ways to implement that particular conversion in a more general way that could be extended to widening of integers (this is also discussed in pitch for Double/CGFloat conversion as possible future direction) but the problem is that even widening has big performance implications although it's scoped down to just a handful of types.

I think if we were to consider a general feature here we'd have to put so many restrictions on its syntactic use that it couldn't be called "general" in a normal sense and wouldn't be worth the complexity.

Now let's consider one possible solution as implemented in the associated PR. The idea is to add a non-failable initializer for each possible conversion type could have e.g.

extension UnsafeRawPointer {
  init(implicit: UnsafeMutableRawPointer) { ... }
}

Could be used to implement uni-directional conversion from UnsafeMutableRawPointer to UnsafeRawPointer. Without bike-shedding to much on the syntax let's consider what the semantics might be:

  • Any struct, class, enum (or nominal for that matter) should be convertible to other type if it has an initializer in form of init(implicit: T) where T is the "from" conversion type.

Which means that every Conversion (of any type e.g. argument, operator argument, regular) or Subtype relationship in the type system now has to check whether right-hand side has an initializer that accepts left-hand type as an argument. This new rule could be expressed in a following set of constraints:

Starting from:

$From <convertible to> $To

Simplifies into:

$To[.member == init(implicit:)] == $Init
($From) -> $To <applicable function> $Init

Note that after lookup $Init could be bound to a number of choices with different types of arguments (important - including failable initializers) if $To is convertible to N different types.

After $Init is determined it's possible to simplify ($From) -> $To <applicable function> $Init into:

$From <argument conversion> $Init.$Arg0
$To <function result> $Init.$Result

Further simplification results in yet another [.member == init(implicit:)] ... check for $Init.$Arg0 and $From types if $To has multiple overloads of init(implicit:).

It could be reasonable to disable followup checks in "implicit conversion" context but that would restrict the generality of the feature.

This new rule impacts all expressions in the language non-trivially but having to perform at least one additional member lookup per any kind of conversion, even if a type doesn't support implicit conversions. It could be argued that we could use a declaration attribute to mark some of the types as @implicitlyConvertible to avoid having to pay the extra cost in the type-checker, but if we did that for some commonly used types e.g. to support widening conversions it would still impact a non-trivial amount of code and retroactively adding such an annotation might not be possible.

My suggestion would be to consider merit of an uni-directional conversion from UnsafeMutableRawPointer to UnsafeRawPointer just like we did Double/CGFloat and discuss general implicit conversions separately based on the knowledge gained from it and previous conversions.

8 Likes

Personally, I'm against adding this. Based on the discussion in this thread, it seems like we need to more carefully consider the space of options around implicit conversions. I don't like the idea of delving deeper down this rabbit hole without a clear long term vision for the language.

To play devil's advocate, one could argue that if we did add some general form of implicit conversions in the future (which for the record I'm also against, though I could perhaps be convinced otherwise) it would probably be source level compatible with this special case.

If you want a really "hot" take though, to the extent that all this boiler plate discourages interoperating with C code, that might actually be a good thing. It's extremely important that Swift can call C code, but C code should general be hidden inside an API with clear high level guarantees, since it is often trivial to invoke undefined behavior by calling C APIs directly.

I agree with you, except for:

If you want a really "hot" take though, to the extent that all this boiler plate discourages interoperating with C code, that might actually be a good thing. It's extremely important that Swift can call C code, but C code should general be hidden inside an API with clear high level guarantees, since it is often trivial to invoke undefined behavior by calling C APIs directly.

Using C from Swift is not an uncommon thing, even on the Apple platforms. If you control the C API you can take advantage of wide variety of macros and attributes that allow you to create a fluent Swift API for your C code. (Example)
If you "control" the compiler itself, you can take the whole thing into another level with bridging.

A lot of popular Swift Packages use C targets or dependencies. I think that making the C interop more fluent and readable is a good thing.

I am not a fan of making things difficult in order to discourage people from using it. If you (or a developer of a dependency) make the decision to use a C code from Swift, you probably have no other option. You're probably not going to re-implement sqlite3 in Swift. And if your program does not work, purposely complicated syntax won't help you in solving the issue.

6 Likes

Rather than see this pitch wither on the vine, I’d like to try to keep things moving. It took an unexpected twist when Chris intervened and raised the bar to “could a design be found where implicit conversions be expressed in Swift rather than a sequence ad-hoc changes to the compiler itself” all this without slowing type checking down. I have come to be strongly supportive of this goal.

To this end, I’ve produce a special “Friday the 13th” toolchain making a start on implementing this feature and been unable to record it slowing the compiler down benchmarking repeated builds of the fairly substantial GitHub - grpc/grpc-swift: The Swift language implementation of gRPC. project.

https://drive.google.com/file/d/1wUgTREO6LeMxdqXsoCeotTNF3O81Gnb-/view?usp=sharing

The toolchain is the “diff” of the original PR applied to tag swift-5.5-DEVELOPMENT-SNAPSHOT-2021-08-11-a which means it is most closely related to the Swift in recent Xcode betas.

Resistance to this feature seems to come from two quarters I’d characterise as “Should we” and “Could we”. Opinions vary widely about whether implicit conversions could be a good thing, the common criticism being that they affect the coders ability to anticipate what code is executed by a given expression but I would argue the ship has already sailed on this in Swift (I’m talking about you computed properties, not to mention willSet and didSet observing patterns or destructors which can fire at practically any time for that matter.) Like operator overloading introducing implicit conversions is a double edged sword and could certainly be open to abuse but it’s part of moving functionality out of the compiler developers hands into the Swift space. There are already implicit conversions and there are pitches for more about but they can/should be expressed in Swift.

The second objection “Could we” which I suspect colours the first objection in some people's eyes I hope to put to bed with the prototype toolchain. Although the toolchain takes a naive approach grafted onto the last phase of type checking this may in fact be a feature. The type checker and its internal constraint solver is a powerful abstraction at the heart of Swift but in this case it may be holding us back. Let’s face it, as a result Swift is a bit precious about strict typing when there are safe conversions it could be performing. This would make developers lives easier as in this pitch for example as it started out or recently for CGFloat/Double convertibility or even just up-typing smaller integer types to Int. if this feature can in fact be accommodated by modifying the constraint solver, all well and good but that is out of my competence.

And so to the “design” of the feature in the prototype. It is very simple. To create an implicit conversion you add an initialiser in an extension in the “to type” that takes the “from type” as an argument with a label implicit:. This design, using initialisers rather than conversion functions on the “from type” was a by-product of the implementation which re-uses the CGFloat<->Double code but it also seems to group conversions more intuitively. An example:

extension Int {
  init(implicit: Int16) {
    print("Int.init(implicit: Int16)")
    self.init(implicit)
  }
  init(implicit: Int32) {
    print("Int.init(implicit: Int32)")
    self.init(implicit)
  }
}
extension Int32 {
  init(implicit: Int8) {
    print("Int32.init(implicit: Int8)")
    self.init(implicit)
  }
  init(implicit: Int16) {
    print("Int32.init(implicit: Int16)")
    self.init(implicit)
  }
}

Note that the implementation does not chain conversions which constrains the search space. In this case Int8 is convertible to Int32 and Int32 is convertible to Int but Int8 is not convertible to Int as a result. As a result you can now make cyclic (i.e. non-DAG) conversions to make types interchangeable such as the new implementation of CFFloat<->Double:

extension Double {
  init(implicit: CGFloat) {
    print("Double.init(implicit: CGFloat)")
    self.init(implicit)
  }
}
extension CGFloat {
  init(implicit: Double) {
    print("CGFloat.init(implicit: Double)")
    self.init(implicit)
  }
}

A crinkle in the existing operation of the type checker is that to find a solution to coerce to use the new implicit constructor there needs to be an existing unlabelled conversion between the two types. If you want to make a completely new relationship between two types you need to provide such an initialiser. So, for example to make Doubles interchangeable with a Bool, you use the implicit label but in this case hidden:

extension Bool {
    init(_ implicit: Double) {
        self.init(implicit != 0.0)
    }
}

func smothingOperation(param: Double) {
    print("Smoothing \(param)")

}

let smoothingParam = 99.1
if smoothingParam {
    smothingOperation(param: smoothingParam)
}

This can all be bike-shedded and many will likely prefer a new @implicit annotation instead. I won’t pretend that crashing the toolchain isn’t possible as there is more work to do on handling generics but it serves its sole purpose: to prove that we should keep this conversation moving.

5 Likes

To summarise this long and rambling post all I wanted to say was let's not dismiss bringing a universal solution for bringing implicit conversions into the language on the grounds that "it can't be done without slowing down type checking". The solution may be a sort of lightweight post-processing stage outside of the type checker that can be optimised once. This is he point of the toolchain makes. Anyway, the conversation has moved to this thread if you want to continue reading.

Just FYI, I have started a topic in Evolution/Discussion to create a space to discuss generalization of implicit conversions - Generalization of Implicit Conversions.

3 Likes