Ah yes, its implementation is short & relatively readable but not efficient¹. Lots of temporaries, which means copying, which seems to be the biggest performance-killer in this sort of thing.
@oscbyspro wrote a pretty good version for decimal conversion, that's part of the new new Foundation. If you're curious about some of the optimisation details check out the pull request (and if you're tangentially curious about the functional-correctness aspects, check out my initial version - it's harder than you might think to implement this sort of thing even before you start to worry about performance).
If you look at the benchmark results in Oscar's pull request, in short "v2x" is Oscar's final version (that's now in Foundation) and "v0" is roughly equivalent to attaswift/BigInt's version. So you can see that using this version should render the cost of converting to a string basically irrelevant, in @axello's benchmark.
While this new version in Foundation doesn't do anything but decimal, it is fully localisation-aware (when used via IntegerFormatStyle) and it works with all BinaryIntegers.
I'm not sure if it's actually shipped in any Apple OS release yet, though.
This reminds me, @axello - Numberick is Oscar's own package containing (among other things) big-int implementations, and is probably more performant (it definitely is for string conversion, at least). Perhaps worth trying instead - it should be pretty trivial to switch to.
¹ Hypothetically it could be, but it'd require the optimiser to do much more dramatic transformations than it does currently.