Our system has only low frequency connections to UI via GRPC - it’s a Siri for enterprise system that runs instantly on any recent mobile device - so it does ALOT of compute in 200ms. (For example when you say “play Madonna”, the word Madonna is in a model of tens of millions of artists- and that’s just one “domain”.
Our agent arch uniquely has a many minds architecture fusing knowledge , grammar, and vocabulary like a human brain - you can Google the words “pinker thinking machine” and look for books by professor pinker of Harvard to get an inkling.(I used to work on Siri)
Like Viv, our agent can handle sentences like “take me to a place with wine that goes well with Pasta with peas”, but without training the language model.
By using GRPC as our API, our clients connect from anywhere on the internet (including locally or in process) and can be almost any language or on almost any platform. And our public api is only what’s available in GRPC- so we don’t have the typical issues around versioning and public classes.
This also lets us use flutter instead of SwiftUI and have the same pretty UI on almost any platform. & Flutters code-compile-run loop is quite a bit faster than swifts so we like that combination. I like SwiftUI - but we want to do our work on apple platforms and deploy anywhere. I have been at Apple more than once and love what we Apple people do for innovation and progress !
Have fast arrays, and reusable pool of instances, and use immortalize, and fine grained concurrency - really just would like alloc instance from our memory.
We were using structs for more of our instances & fast arrays- but using protocol inheritance incurs unavoidable arc traffic that we can’t afford- which we solved with a reusable immortalized pool of class instances, where both the containers and elements are pooled. We don’t use iterators either - more arc traffic. And we are very careful with our use of actors and async - because using async tends to remove a lot of inlining which tends to exacerbate Arc traffic. We’ve found we can use structs only without protocols and only with no mutability (too much ARC traffic otherwise) - and use that in cases when we can.
The thing about pools is they let us do the same cheap alloc as on the stack - but with more control around generations and usage patterns - and without worrying about blowing the stack. And allocating our instances in our memory let’s us get more locality and performance.
We like performance ! Our engine is already very scalable - but we are building a next gen Ai system, where more performance means “smarter”.
The more nodes we walk, the better our thinking. The wider our Viterbi searches, the less aggressive our pruning, the better our robustness in noise.
We like our system ! And we love swift. And we want more performance !
Please send our custom allocator 
