Linker/Compiler limitation on number of protocol extensions?

Using Swift
,
1

Hi all,

On our large iOS project, we have a generated module which bridges types together. That is to say: it analyzes the types contained within a different set of modules, it generates public protocols over the related types, and it generates protocol extensions.

This module is large, with a ton of generated protocols and protocol extensions (we have something on the order of ~35,000 protocol extensions and ~10,000 protocols). Recently, we've begun having issues. I believe we've begun to hit a limitation of the linker; we have so many protocols and extensions, that the extensions from this module are no longer being recognized. In fact, moving the extensions out of this module and into our main app target makes everything work again.

If anyone is aware:

  • Is there a numeric linker limitation?
  • How can we move forward? We aren't in a position to reduce the total count at the moment, but this has the potential to grind all development down to a halt.
1 Like
2

How many conformances do you have total? You can determine this with something like (from here):

otool -l path/to/binary | grep _swift5_proto -A 4

In that blog post it's mentioned the app they tested had >100k, I think we have a similarly large amount and have never heard of any hard limits. How is it failing specifically for you? Do you think you could come up with a repro project? Depending on how your library is being linked they could be being dropped because they appear unused, but that wouldn't have started happening based on the number of them.

1 Like
3

Hi Keith, thanks for the ideas there.

It's hard for me to say how many conformances we have; we are dynamically linking all of our internal libraries and frameworks. When I pipe the release build of our application to otool, I get 2,885 protocol conformances, however, that's not actually correct; since all of our internal dependencies are linked dynamically, they don't appear in this count. When I pull the counts for our two generated frameworks, I get: 19,967 and 14,872 conformances (which tracks with my rough estimate of ~35,000).

How is it failing specifically for you?

The protocol conformances aren't being recognized. Let's say our generated type is called MyDataType. I'll write a function that's something like this:

func transform<T: ExternalProtocol>(value: ExternalProtocol.AssociatedValue) -> MyObject<T> where ExternalProtocol.AssociatedValue: MyDataType { ... }

This will give the following error:

Instance method 'transform(value:)' requires that 'ConcreteType.AssociatedValue' conform to 'MyDataType'

In my generated framework, I have the following:

public protocol MyDataType { ... }
extension ConcreteTypeAssociatedValue: MyDataType { } // all properties being added are `public`

// Where the definition for `ConcreteType` in an external framework is such that its `AssociatedValue` is specifically of type `ConcreteTypeAssociatedValue`

Simply copying over the extension from the framework files to the main app target allows it to successfully build.

Depending on how your library is being linked they could be being dropped because they appear unused

I tried changing the linking from dynamic to static, and still arrive at the same error. In case of a symbol conflict, I also renamed the generated protocol names -- to no avail.

4

This sounds like it could be a build dependency issue? Is the main target correctly depending on the module that adds the conformances? Are you saying that if you remove some number of conformances that works?

5

The main target is correctly depending on the module that adds the conformance; it's the same module that declares the protocol.

Removing some generated protocols does cause this to start working again. The reason I think that this is something to do with the symbol table / linking is because of duplicate (but scoped) naming. For added context: We are using Apollo GraphQL to generate our queries, mutations, and so on for our schema. However, as we support enterprise deployments -- essentially frozen snapshots of our live schema at a given point in time -- we generate these outputs for each versioned schema. By and large, the generated code is the same for most of these, with the live version largely being additive.

As an effect of this, Apollo will generate nested objects with identical names within the same module, like Node (MyQuery.Data.MyConnection.Node and MyOtherQuery.Data.MyOtherConnection.Node). We generate n modules this way. Then, we use these generated outputs as inputs to our own generated layers that analyze this code and generates protocols.

For example, Apollo may generate:

struct MyQuery {
  struct Data {
    struct MyObject {
      struct Connection {
        struct Node {
          ...
        }
      }
    }
  }
}

Let's say we have this object generated across 6 schemas -- we would generate our own protocol, MyQueryDataMyObjectConnectionNodeType and generate conformance like so:

public protocol MyQueryDataMyObjectConnectionNodeType {
  public var whatever: String { get }
}

extension Schema1.MyQuery.Data.MyObject.Connection.Node: MyQueryDataMyObjectConnectionNodeType {
  public var whatever: String { ... }
}

extension Schema2.MyQuery.Data.MyObject.Connection.Node: MyQueryDataMyObjectConnectionNodeType {
  public var whatever: String { ... }
}

// etc

This has been working well enough for us, however now when we add more queries to generate over, it can no longer recognize the protocol conformances despite recognizing the protocol names.

6

If the error you're seeing is the one you mentioned above like:

Instance method 'transform(value:)' requires that 'ConcreteType.AssociatedValue' conform to 'MyDataType'

that is a compiler error not a linker error, so I don't think you'd be hitting anything on the linking side yet.

7

We also run into errors of the variety:

Value of type `ConcreteType.AssociatedValue` has no member `myValue`

where myValue is defined in the protocol extension -- though, I assume that's also a compiler error?

8

yea looks like it

9

I've figured this one out -- what a doozy.

In cases that were successful, the underlying types looked something like this (simplified for brevity):

protocol Operation {
  associatedtype Data: Selection
}

protocol Selection {
  ...
}

struct MyOperation: Operation { ... }

Whereas the cases that were giving me issues had definitions that looked something like this:

protocol Operation {
  associatedtype Data: Selection
}

protocol Selection: Hashable {
  ...
}

struct MyOperation: Operation { ... }

Effectively, the second definition forces a Self requirement onto the associated type. That means that MyOperation.Data cannot be resolved to be of type MyOperationDataType when the protocols and concrete definitions are used interchangeably in function signatures.

Therefore, I added a type conformance directly into the generated protocol definition:

protocol MyOperationType: Operation where Self.Data: MyOperationDataType { ... }

That allowed me to compile successfully. Bringing the extension over from the other file was giving the compiler enough information to bridge the pieces together, although I would sincerely argue that it perhaps shouldn't have; that extension was already active elsewhere and repeating it shouldn't have done anything.