Sorry to take so long to respond, I had taken a step back and wanted some time to recover from such work.
I think you're right, the way forward is to have a package meant to enable better Number types. Documentation and predictable performance are key and a clear path of other packages could follow like algebraic solving or geometry. I may even take this opportunity to make a package for each number type and then Number package to wrap them all up for more advanced packages.
@wadetregaskis and I had been brainstorming a solution to give UInt-like performance when that's all that's needed and a pointer to storage for larger numbers. This is a meaningful performance change and trying to do it without blowing up the memory footprint is a task.