Minimizing Memory Footprint of Swift Types <-> C Types

[xanderdunn]

Thanks everyone for your help!

@harlanhaskins

The first thing to do here would be to avoid using protocols as values in this way. A protocol used this way needs to allocate extra memory to potentially store an arbitrary payload under the hood, and needs to bring type metadata and dynamic dispatch tables around with it.

Thanks, I didn't know protocols carry additional memory baggage. This goes back to my question here where I'm struggling to contain these various different types (Double, Bool, Decimal, Date, Int, String) in a way that I can operate on all of them without writing if statements in every function where I work on them:

if Double
else if Bool
else if Date
...

It's my limited understanding of the solutions using protocols proposed over in that post that lead to the current state of things. However, I phrased the question in that post in terms of type erasure and that may have been a red herring.

@AlexanderM

Have you considered staying in unsafe api land (a.k.a. C land), and only bridging to/from Swift types on request?

E.g. materialize a Swift.Array<Int> of some small result set, only when queried. Have the backing store still be the C-allocated memory.

This is an interesting idea, thanks for suggesting. If the goal is to iterate across all of the data once, then we need keep in memory only the full dataset in C + the current small subset in Swift. In fact, if that's the goal we need keep in memory only the subset we're currently operating on. The Apache Arrow format can be memory mapped and we can simply iterate through the file as we operate on the subset we have available to us. But, I think this could turn into considerable overhead if we're doing many repeated random reads?

@dfsweeney

what you're trying to do is make an array of heterogenous arrays (like a data frame of the column vectors). I think that might be where things get weird--I could not get the code to compile like that

Yes that's absolutely the thinking. Yeah, the code snippet I provided wasn't enough to compile. Here's a more complete example for Double:

public protocol BaseArrowArrayElement: CustomStringConvertible {
}

protocol ArrowArrayElement: Equatable, BaseArrowArrayElement {
    static func toGArrowArray(array: [Self]) throws -> UnsafeMutablePointer<GArrowArray>?
    static func fromGArrowArray(_ gArray: UnsafeMutablePointer<GArrowArray>?) -> [Self]
}

extension Double: ArrowArrayElement {
    static func toGArrowArray(array: [Double]) throws -> UnsafeMutablePointer<GArrowArray>? {
    }

    static func fromGArrowArray(_ gArray: UnsafeMutablePointer<GArrowArray>?) -> [Double] {
    }
}

@dfsweeney

struct TwoArrays<T, U> {
  let i : Array<T>
  let j : Array<U>
}

As you noticed, the trouble with this is that I don't know how many columns there are at compile time, it's determined by the file being read in. Your solution with a struct that contains metadata is very interesting, I'll play around with this idea.

@dfsweeney

I think the ABI Stability Manifesto stuff is the place to start

Many thanks for this, I'll give this a read.