It's an interesting idea, although a big departure from convention so it might require some getting used to.
Replacing incorrect uses of Int
I certainly do like the idea of fixing various APIs that use signed types when in fact negative values are absolutely invalid. And this Number type seems like it moots all the arguments for using signed types for values that are not in fact signed. So big positive energy there, as Casey Liss would say.
However, I'm not sure how plausible it is to actually change all those APIs, irrespective of how good the replacement is. Especially since many are in Foundation (distributed separately from the compiler & stdlib, and with the requirement to be compatible across even major compiler version bumps). I believe changing Int to UInt for collection indices and counts has already been unilaterally rejected by the Swift team, repeatedly.
"Policy" aside, from a technical perspective the replacement WholeNumber (or whatever) would need to be at least as fast and efficient (size-wise) as Int, at least in release builds, otherwise I think this particular application is untenable. Bounds checking is already a performance limiter for some applications, and counter-proposals like "don't use standard collections for performance sensitive code" or "use special APIs to bypass bounds checks" go against the current grain and Swift's larger philosophy of 'safety'.
Performance
It's the Swift team's policy that proposals don't have to generally address runtime performance, and may ignore known performance problems (on the presumption they can be fixed as implementation details). I don't think that's wise in general (see also: macros) but especially in this case, reasonable performance needs to be assured. Such a pervasive change, as redoing essentially the entire numerics system of the language, needs to be certain of good performance and to have that good performance from the outset (Swift is no longer a seedling language; noticeable regression of standard library performance is untenable).
As such, it'd be good to see benchmark results in the proposal, and performance examined more broadly.
Of course, if the proposal reduces its ambition somewhat (re. replacing Int etc) then performance expectations can be appropriately reduced. I would certainly hate to see something like this rejected from consideration just because it's not quite as fast as the primitive numeric types, because it obviously has great value otherwise and might well be worth the trade-off in other use-cases.
Vectorisation
I'd say this particular aspect of performance and implementation can be largely deferred, because it can't possibly be pivotal to acceptable performance. Most numbers in most programs are small, standalone scalars and don't benefit from SIMD except by loop unrolling - only that aspect (ensuring this doesn't interfere with or preclude existing compiler optimisations) need be addressed.
It also seems pretty straightforward to add SIMD enhancements to the underlying arbitrary precision number types, for [the relatively rare] cases where they are large enough to benefit (65 bits or larger, I'd guess). Therefore I'm not worried about that particular aspect being punted to a later time.
Async
Re. the proposal's vague mention of async, it's not clear to me how that could plausibly improve performance. Is it merely an attempt to make it easier to do numeric calculations in parallel by [ab]using async let? Because otherwise I wouldn't think numeric calculations contain any suspension points internally, so nominally have no need nor benefit from being async. And being async introduces significant overheads to basic execution.
Full safety (re. "Throwing Operations")
Re. cases like division by zero, it seems like the proposal as it stands already eliminates enough categories of errors (overflow & underflow etc) as to make throwing operations viable to introduce for the remainder. Swift has previously (repeatedly) rejected the notion of having basic arithmetic operators throw on the premise that it would make them too inconvenient to work with. But if throws only applied to a limited subset of operators, like division, then perhaps folks will find it more palatable (even without further assistance from the compiler, such as implicit conversion to non-throwing for known-good inputs).
And being able to say that Swift Numbers are completely safe (i.e. won't randomly crash your whole app!) would be a huge boon both for the language (PR-wise) and for all its users.
Protocol conformances
The proposal makes no mention of things like Codable. Those should be covered, at least insofar as there should be a clear roadmap even if the implementation details are omitted.
Precision Tracking
Adding 'precision tracking' seems like more trouble than it's worth, at least on first blush. I think it has to be taken as a given, and understood by all programmers, that the language is only as good as its inputs. e.g. if the user provides pre-rounded input (let-alone outright wrong input), that can't possibly be the responsibility of the basic numeric types. I can see how maybe them having a facility to track semantic precision could be helpful to end users, but I worry that it'd spill that complexity into use-cases that don't have a need for that. It might be better to leave that kind of metadata management to a higher layer, as it is today.
That said, it might just be that this is so far from typical programming language convention that I'm being a bit of a luddite about it. I do welcome more discussion on this topic, particularly from anyone that has actually had to deal with / implement this sort of thing in real-world code (I have but like twenty years ago, so it's all too faded from memory).
And if it did track that, presumably it would need more nuance than a boolean ("precise or not"). Precision varies differently depending on the arithmetic operations applied (re. additive vs compounding errors, etc).
Technically irrelevant
Kudos for using Monodraw (or at least, that's my intuition based on the style). Monodraw is one of the most delightful programs I've ever used. I can't believe someone actually made something so niche that's so bloody well done.