Emitting redundant protocol conformances to support type lookup


(Luke Howard) #1

In the fix for [SR-381] (links at end) we implement an API for looking up classes by name by searching the protocol conformance table. The initial consumer of this is NSClassFromString() as used by NSKeyedUnarchiver in Foundation.

The limitation of this approach is that only classes that explicitly conform to protocols can be resolved. We’ll work around this in Foundation by having subclasses that otherwise inherit their protocol conformance explicitly conform to a dummy protocol. However, this behaviour is confusing and would be nice to fix.

One approach I’ve been playing with is for classes always to have an explicit conformance to AnyObject (at least, if they don’t explicitly conform to anything else). (I have a bit of a hacky patch to implement this but it’s failing at link time as there’s no symbol for “_TMps9AnyObject”.)

Before I proceed further down this path – is this an approach worth pursuing or would it be better not to abuse the conformance table for name lookups long term?

— Luke

https://bugs.swift.org/browse/SR-381
https://github.com/apple/swift/pull/834/files

···

--
www.lukehoward.com
soundcloud.com/lukehoward


(Zhao Xin) #2

According to Swift docs on AnyObject,

The protocol to which all classes implicitly conform.

​So it is not ​
an explicit conformance.

​http://swiftdoc.org/v2.1/protocol/AnyObject/

z
​haoxin​

···

On Fri, Jan 8, 2016 at 4:50 PM, Luke Howard via swift-dev < swift-dev@swift.org> wrote:

In the fix for [SR-381] (links at end) we implement an API for looking up
classes by name by searching the protocol conformance table. The initial
consumer of this is NSClassFromString() as used by NSKeyedUnarchiver in
Foundation.

The limitation of this approach is that only classes that explicitly
conform to protocols can be resolved. We’ll work around this in Foundation
by having subclasses that otherwise inherit their protocol conformance
explicitly conform to a dummy protocol. However, this behaviour is
confusing and would be nice to fix.

One approach I’ve been playing with is for classes always to have an
explicit conformance to AnyObject (at least, if they don’t explicitly
conform to anything else). (I have a bit of a hacky patch to implement this
but it’s failing at link time as there’s no symbol for “_TMps9AnyObject”.)

Before I proceed further down this path – is this an approach worth
pursuing or would it be better not to abuse the conformance table for name
lookups long term?

— Luke

https://bugs.swift.org/browse/SR-381
https://github.com/apple/swift/pull/834/files

--
www.lukehoward.com
soundcloud.com/lukehoward

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--

Owen Zhao


(Luke Howard) #3

Right, the point was to make it explicit so as to force a protocol conformance table entry to be emitted.

···

Sent from my iPhone

On 8 Jan 2016, at 21:09, 肇鑫 <owenzx@gmail.com> wrote:

According to Swift docs on AnyObject,

The protocol to which all classes implicitly conform.

​So it is not ​an explicit conformance.​

​http://swiftdoc.org/v2.1/protocol/AnyObject/

z​haoxin​

On Fri, Jan 8, 2016 at 4:50 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:
In the fix for [SR-381] (links at end) we implement an API for looking up classes by name by searching the protocol conformance table. The initial consumer of this is NSClassFromString() as used by NSKeyedUnarchiver in Foundation.

The limitation of this approach is that only classes that explicitly conform to protocols can be resolved. We’ll work around this in Foundation by having subclasses that otherwise inherit their protocol conformance explicitly conform to a dummy protocol. However, this behaviour is confusing and would be nice to fix.

One approach I’ve been playing with is for classes always to have an explicit conformance to AnyObject (at least, if they don’t explicitly conform to anything else). (I have a bit of a hacky patch to implement this but it’s failing at link time as there’s no symbol for “_TMps9AnyObject”.)

Before I proceed further down this path – is this an approach worth pursuing or would it be better not to abuse the conformance table for name lookups long term?

— Luke

https://bugs.swift.org/browse/SR-381
https://github.com/apple/swift/pull/834/files

--
www.lukehoward.com
soundcloud.com/lukehoward

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--

Owen Zhao


(Luke Howard) #4

The limitation of this approach is that only classes that explicitly conform to protocols can be resolved. We’ll work around this in Foundation by having subclasses that otherwise inherit their protocol conformance explicitly conform to a dummy protocol. However, this behaviour is confusing and would be nice to fix.

Sorry “we’ll work around this” sounds a bit presumptuous – what I meant was “it can be worked around”. :slight_smile:

One approach I’ve been playing with is for classes always to have an explicit conformance to AnyObject (at least, if they don’t explicitly conform to anything else). (I have a bit of a hacky patch to implement this but it’s failing at link time as there’s no symbol for “_TMps9AnyObject”.)

Having IRGenModule::emitProtocolConformances() special case AnyObject to emit a zero reference solves the linkage issue. (RelativeIndirectablePointer will then return a bogus pointer, but maybe a Flag in the conformance record could be used to indicate null.)

This approach means the protocol conformance tables could get very large, but it has a fairly low impact to the compiler and runtime. An alternative might be to emit explicit entries only for classes that inherit from, say, NSObjectProtocol or NSCoding, and don’t conform to anything explicitly. Or classes could be marked as resolvable-by-name with some magic attribute (however this runs counter the goal of source code portability between Darwin and other platforms).

— Luke


(John McCall) #5

I’m pretty opposed to wasting a bunch of space in the binary on formal protocol conformances that can be easy rederived by just checking whether the conforming type is a class type. AnyObject is a special case in the compiler; it makes sense for it to be a special case in the runtime conformance checking routines.

John.

···

On Jan 8, 2016, at 6:40 AM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

The limitation of this approach is that only classes that explicitly conform to protocols can be resolved. We’ll work around this in Foundation by having subclasses that otherwise inherit their protocol conformance explicitly conform to a dummy protocol. However, this behaviour is confusing and would be nice to fix.

Sorry “we’ll work around this” sounds a bit presumptuous – what I meant was “it can be worked around”. :slight_smile:

One approach I’ve been playing with is for classes always to have an explicit conformance to AnyObject (at least, if they don’t explicitly conform to anything else). (I have a bit of a hacky patch to implement this but it’s failing at link time as there’s no symbol for “_TMps9AnyObject”.)

Having IRGenModule::emitProtocolConformances() special case AnyObject to emit a zero reference solves the linkage issue. (RelativeIndirectablePointer will then return a bogus pointer, but maybe a Flag in the conformance record could be used to indicate null.)

This approach means the protocol conformance tables could get very large, but it has a fairly low impact to the compiler and runtime. An alternative might be to emit explicit entries only for classes that inherit from, say, NSObjectProtocol or NSCoding, and don’t conform to anything explicitly. Or classes could be marked as resolvable-by-name with some magic attribute (however this runs counter the goal of source code portability between Darwin and other platforms).


(Joe Groff) #6

Yeah, there's no reason to inject explicit AnyObject conformances. We could introduce a separate table in the image for all of the types in the image, or maybe just the types that aren't already involved in a conformance.

-Joe

···

On Jan 8, 2016, at 9:09 AM, John McCall via swift-dev <swift-dev@swift.org> wrote:

On Jan 8, 2016, at 6:40 AM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

The limitation of this approach is that only classes that explicitly conform to protocols can be resolved. We’ll work around this in Foundation by having subclasses that otherwise inherit their protocol conformance explicitly conform to a dummy protocol. However, this behaviour is confusing and would be nice to fix.

Sorry “we’ll work around this” sounds a bit presumptuous – what I meant was “it can be worked around”. :slight_smile:

One approach I’ve been playing with is for classes always to have an explicit conformance to AnyObject (at least, if they don’t explicitly conform to anything else). (I have a bit of a hacky patch to implement this but it’s failing at link time as there’s no symbol for “_TMps9AnyObject”.)

Having IRGenModule::emitProtocolConformances() special case AnyObject to emit a zero reference solves the linkage issue. (RelativeIndirectablePointer will then return a bogus pointer, but maybe a Flag in the conformance record could be used to indicate null.)

This approach means the protocol conformance tables could get very large, but it has a fairly low impact to the compiler and runtime. An alternative might be to emit explicit entries only for classes that inherit from, say, NSObjectProtocol or NSCoding, and don’t conform to anything explicitly. Or classes could be marked as resolvable-by-name with some magic attribute (however this runs counter the goal of source code portability between Darwin and other platforms).

I’m pretty opposed to wasting a bunch of space in the binary on formal protocol conformances that can be easy rederived by just checking whether the conforming type is a class type. AnyObject is a special case in the compiler; it makes sense for it to be a special case in the runtime conformance checking routines.


(Luke Howard) #7

Yeah, there's no reason to inject explicit AnyObject conformances. We could introduce a separate table in the image for all of the types in the image, or maybe just the types that aren't already involved in a conformance.

Cool, I’ll look at that then.

— Luke


(Luke Howard) #8

So would one define a putative TypeMetadataRecord that is identical to a ProtocolConformanceRecord but only contains the metadata reference (and reference type discriminant)?

Is emitNominalMetadataRef() the right place to create (not emit) these records?

— Luke


(Luke Howard) #9

Whoops, that emits references not definitions, seems like IRGenModule::emitXXX() is better.

— Luke

···

On 9 Jan 2016, at 11:25 AM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

Is emitNominalMetadataRef() the right place to create (not emit) these records?


(Luke Howard) #10

I made an experimental patch to IRGen to emit a table of type metadata, independent of the protocol conformance table.

The entry format is a subset of a protocol conformance record:

struct TypeMetadataRecord {
private:
  // Some description of the type that is resolvable at runtime.
  union {
    /// A direct reference to the metadata.
    RelativeIndirectablePointer<Metadata> DirectType;
    
    /// An indirect reference to the metadata.
    RelativeIndirectablePointer<const ClassMetadata *> IndirectClass;
  };

  /// Flags describing the type metadata record.
  TypeMetadataRecordFlags Flags;
}

Otherwise the implementation is pretty similar. They are emitted in irgen::emitXXXMetadata. Mappings for generic types are at runtime when swift_getGenericMetadata() is called.

It seems to work OK and removes the limitation that the conformance table approach had, where one could only resolve the names of types with explicit protocol conformances.

Haven’t tested on Linux yet.

— Luke


(Joe Groff) #11

Cool. We should also emit references to generic metadata pattern from this table. None of these pointers needs to be indirectable, since they'll always refer to types declared within the current object file.

-Joe

···

On Jan 10, 2016, at 3:37 AM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

I made an experimental patch to IRGen to emit a table of type metadata, independent of the protocol conformance table.

The entry format is a subset of a protocol conformance record:

struct TypeMetadataRecord {
private:
  // Some description of the type that is resolvable at runtime.
  union {
    /// A direct reference to the metadata.
    RelativeIndirectablePointer<Metadata> DirectType;
    
    /// An indirect reference to the metadata.
    RelativeIndirectablePointer<const ClassMetadata *> IndirectClass;
  };

  /// Flags describing the type metadata record.
  TypeMetadataRecordFlags Flags;
}

Otherwise the implementation is pretty similar. They are emitted in irgen::emitXXXMetadata. Mappings for generic types are at runtime when swift_getGenericMetadata() is called.

It seems to work OK and removes the limitation that the conformance table approach had, where one could only resolve the names of types with explicit protocol conformances.


(Luke Howard) #12

Tested on Linux & also updated to avoid emitting type metadata records when there is already a protocol conformance record for that type.

— Luke

···

On 10 Jan 2016, at 10:37 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

It seems to work OK and removes the limitation that the conformance table approach had, where one could only resolve the names of types with explicit protocol conformances.

Haven’t tested on Linux yet.


(Luke Howard) #13

Would it be possible to extend GenericParameterDescriptor to include a tail-emplaced set of references to ProtocolDescriptors for any parameter type constraints that require witness tables? This would be useful plumbing for a future API that can dynamically instantiate generic types.

See:

https://github.com/lhoward/swift/blob/SR-381/include/swift/Runtime/Metadata.h#L1199

https://github.com/lhoward/swift/blob/SR-381/stdlib/public/runtime/MetadataLookup.cpp#L338

— Luke


(Luke Howard) #14

Based on your most recent request I’ve completely removed the generic type record support out of the SR-381 branch. But if you want I can just separate out the runtime changes and leave the compiler changes in.

No-generic – https://github.com/apple/swift/pull/834/files
Generic – https://github.com/apple/swift/pull/959/files

— Luke

···

On 12 Jan 2016, at 5:21 AM, Joe Groff <jgroff@apple.com> wrote:

Cool. We should also emit references to generic metadata pattern from this table. None of these pointers needs to be indirectable, since they'll always refer to types declared within the current object file.


(Joe Groff) #15

That's something to consider. However, we will be systematically revamping these metadata structures soon, so I don't think there's much benefit to incrementally changing what we have.

-Joe

···

On Jan 12, 2016, at 7:46 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

Would it be possible to extend GenericParameterDescriptor to include a tail-emplaced set of references to ProtocolDescriptors for any parameter type constraints that require witness tables? This would be useful plumbing for a future API that can dynamically instantiate generic types.

See:

https://github.com/lhoward/swift/blob/SR-381/include/swift/Runtime/Metadata.h#L1199

https://github.com/lhoward/swift/blob/SR-381/stdlib/public/runtime/MetadataLookup.cpp#L338


(Luke Howard) #16

NB – presently the Protocols list is at the very end of GenericParameterDescriptor but it’s cleaner (if ABI disruptive) to put nest it inside Parameter.

···

On 13 Jan 2016, at 2:46 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

Would it be possible to extend GenericParameterDescriptor to include a tail-emplaced set of references to ProtocolDescriptors for any parameter type constraints that require witness tables? This would be useful plumbing for a future API that can dynamically instantiate generic types.

See:

https://github.com/lhoward/swift/blob/SR-381/include/swift/Runtime/Metadata.h#L1199

https://github.com/lhoward/swift/blob/SR-381/stdlib/public/runtime/MetadataLookup.cpp#L338

— Luke

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
www.lukehoward.com
soundcloud.com/lukehoward


(Luke Howard) #17

Also, what’s the best way of doing alignment when emitting C structs from IRGen?

For now I added this to ConstantBuilderBase:

    void alignToWordBoundary() {
      NextOffset.roundUpToAlignment(IGM.getPointerAlignment());
    }

— Luke

···

On 13 Jan 2016, at 6:18 PM, Luke Howard <lukeh@padl.com> wrote:

NB – presently the Protocols list is at the very end of GenericParameterDescriptor but it’s cleaner (if ABI disruptive) to put nest it inside Parameter.

On 13 Jan 2016, at 2:46 PM, Luke Howard via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Would it be possible to extend GenericParameterDescriptor to include a tail-emplaced set of references to ProtocolDescriptors for any parameter type constraints that require witness tables? This would be useful plumbing for a future API that can dynamically instantiate generic types.

See:

https://github.com/lhoward/swift/blob/SR-381/include/swift/Runtime/Metadata.h#L1199

https://github.com/lhoward/swift/blob/SR-381/stdlib/public/runtime/MetadataLookup.cpp#L338

— Luke

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

--
www.lukehoward.com <http://www.lukehoward.com/>
soundcloud.com/lukehoward

--
www.lukehoward.com
soundcloud.com/lukehoward


(Joe Groff) #18

Also, what’s the best way of doing alignment when emitting C structs from IRGen?

For now I added this to ConstantBuilderBase:

    void alignToWordBoundary() {
      NextOffset.roundUpToAlignment(IGM.getPointerAlignment());
    }

Why do you need to do this?

-Joe

···

On Jan 12, 2016, at 11:37 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

— Luke

On 13 Jan 2016, at 6:18 PM, Luke Howard <lukeh@padl.com <mailto:lukeh@padl.com>> wrote:

NB – presently the Protocols list is at the very end of GenericParameterDescriptor but it’s cleaner (if ABI disruptive) to put nest it inside Parameter.

On 13 Jan 2016, at 2:46 PM, Luke Howard via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Would it be possible to extend GenericParameterDescriptor to include a tail-emplaced set of references to ProtocolDescriptors for any parameter type constraints that require witness tables? This would be useful plumbing for a future API that can dynamically instantiate generic types.

See:

https://github.com/lhoward/swift/blob/SR-381/include/swift/Runtime/Metadata.h#L1199

https://github.com/lhoward/swift/blob/SR-381/stdlib/public/runtime/MetadataLookup.cpp#L338

— Luke

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

--
www.lukehoward.com <http://www.lukehoward.com/>
soundcloud.com/lukehoward <http://soundcloud.com/lukehoward>

--
www.lukehoward.com <http://www.lukehoward.com/>
soundcloud.com/lukehoward <http://soundcloud.com/lukehoward>
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


(Luke Howard) #19

Emitting this:

struct GenericParameterDescriptor {
  uint32_t Offset;
  uint32_t NumParams;
  uint32_t NumPrimaryParams;
  
  /// A type parameter.
  struct Parameter {
    /// The number of protocol witness tables required by this type parameter.
    size_t NumWitnessTables;
    /// The protocols required by this type parameter. If NumWitnessTables is
    /// zero, this is absent.
    ProtocolDescriptor *Protocols[1];
  };

  /// The parameter descriptors are in a tail-emplaced array of NumParams
  /// elements. Because Parameters are variable length, use getParameterAt()
  /// to access them.
  Parameter Parameters[1];
}

needed to align the start of Parameters correctly (and also Protocols but I changed NumWitnessTables to a size_t for now to make that simpler). But perhaps there’s a way to get LLVM to do the layout for you, I feel I’m doing something wrong here.

— Luke

···

On 14 Jan 2016, at 4:31 AM, Joe Groff <jgroff@apple.com> wrote:

On Jan 12, 2016, at 11:37 PM, Luke Howard via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Also, what’s the best way of doing alignment when emitting C structs from IRGen?

For now I added this to ConstantBuilderBase:

    void alignToWordBoundary() {
      NextOffset = NextOffset.roundUpToAlignment(IGM.getPointerAlignment());
    }

Why do you need to do this?


(John McCall) #20

You should use a relative reference here instead of an absolute pointer. We really, really don’t want relocations in reflective metadata.

John.

···

On Jan 13, 2016, at 2:08 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

On 14 Jan 2016, at 4:31 AM, Joe Groff <jgroff@apple.com <mailto:jgroff@apple.com>> wrote:

On Jan 12, 2016, at 11:37 PM, Luke Howard via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Also, what’s the best way of doing alignment when emitting C structs from IRGen?

For now I added this to ConstantBuilderBase:

    void alignToWordBoundary() {
      NextOffset = NextOffset.roundUpToAlignment(IGM.getPointerAlignment());
    }

Why do you need to do this?

Emitting this:

struct GenericParameterDescriptor {
  uint32_t Offset;
  uint32_t NumParams;
  uint32_t NumPrimaryParams;
  
  /// A type parameter.
  struct Parameter {
    /// The number of protocol witness tables required by this type parameter.
    size_t NumWitnessTables;
    /// The protocols required by this type parameter. If NumWitnessTables is
    /// zero, this is absent.
    ProtocolDescriptor *Protocols[1];
  };

  /// The parameter descriptors are in a tail-emplaced array of NumParams
  /// elements. Because Parameters are variable length, use getParameterAt()
  /// to access them.
  Parameter Parameters[1];
}

needed to align the start of Parameters correctly (and also Protocols but I changed NumWitnessTables to a size_t for now to make that simpler). But perhaps there’s a way to get LLVM to do the layout for you, I feel I’m doing something wrong here.