Making a generic factory

I want to have a generic factory method where I create a specific subclass depending on what the generic type parameter is. I've had a hard time working out how to approach it, and my current angle isn't working.

Below is a simplified version of what I have. The full version is in this gist, and the previous working version of the file is here. Basically I have a generic sequence and iterator where I want the type parameter to be a more abstract type, and doing so comes down to solving this factory puzzle. Any suggestions?

protocol Branch {}
protocol LocalBranch: Branch {}
protocol RemoteBranch: Branch {}

// Here's the concrete type, where I need to store a pointer (from libgit2).
// The factory will know about the pointer situation,
// but it's not part of the protocol interface.
class GitBranch: Branch
{
  let pointer: OpaquePointer
  
  init(pointer: OpaquePointer)
  {
    self.pointer = pointer
  }
}
class GitLocalBranch: GitBranch, LocalBranch {}
class GitRemoteBranch: GitBranch, RemoteBranch {}

class BranchIterator<BranchType: Branch>: IteratorProtocol
{
  public func next() -> BranchType?
  {
    let pointer = OpaquePointer(bitPattern: 0) // figure out a real value
    
    return makeBranch(pointer)
  }
  
  // I expected this to get overridden by what's in the extension
  fileprivate func makeBranch(_ pointer: OpaquePointer?) -> BranchType?
  {
    print("oops")
    return nil
  }
}

/* Same-type constraint type 'LocalBranch' does not conform to required protocol 'Branch'
extension BranchIterator where BranchType == LocalBranch {}
*/

extension BranchIterator where BranchType: LocalBranch
{
  fileprivate func makeBranch(_ pointer: OpaquePointer) -> BranchType?
  {
    print("made local branch")
    return GitLocalBranch(pointer: pointer) as? BranchType
  }
}
// ...and similarly for remote branch

let localIterator: BranchIterator<GitLocalBranch> = BranchIterator()
let branch = localIterator.next()

// output: "oops"

What is the problem?

The next method is declared to return a BranchType?, and there exists an overload of makeBranch which returns that exact type, so of course the version with the exactly-matching type is called.

I guess the real goal is one level up from this. I want to have a protocol that looks something like this:

protocol BranchListing
{
  func localBranches() -> AnySequence<LocalBranch>
  func remoteBranches -> AnySequence<RemoteBranch>
}

But since my actual sequence is a custom type with a custom iterator, I'm having trouble figuring out how to wrap that in the more abstract AnySequence.

So if I have an object that conforms to Sequence where Element is GitLocalBranch, how can I return that as AnySequence<LocalBranch>? The compiler won't let me call AnySequence<LocalBranch>(myGitLocalBranchSequence) because the element types have to be exactly equivalent.

I think I'll start with the elephant in the room here: why are you using protocols for this?

Interface segregation. Testability. I have a class that represents a Git repository, which has a ton of functions. I'm using protocols to kind of categorize those functions so that clients can be specific about what aspects of the repository they care about.

My current approach for the branch-listing protocol uses associated types, and I'd like to get away from that, since you run into stuff like the "can only be used as a generic constraint" issue.

Any reason to not implement this with "views"? i.e. an object that refers to the parent (directly or indirectly) but has only restricted (or different) PoV on its contents.

Hm... that may be workable. Thanks.