i am writing some kind of scientific or geometry tool or library
i originally write the whole thing in Swift’s default Double
now some graphics programmer wants it to be in Float, because that’s the maximum precision GPU’s are built for. Now they have to copy and paste all my source code into their own project and they have to do find and replace throughout the whole source code to swap out Double with Float.
now some scientist wants it to be in some arbitrary-precision superfloat type. Now they have to copy and paste all my source code into their own project, and replace Float with their own FloatingPoint type.
this is literally the same story that motivates all other generics. Floaty types are no exception.
So, I actually wrote such a scientific library. You’ll notice that the final code uses only concrete types. I started using Double everywhere, then moved to generic methods everywhere, then scrapped everything and used Float.
If you’re creating or using one of these libraries, what's a reasonable slowdown for a few values quickly becomes unacceptable for any significant amount of data. Optimized routines such as those in Accelerate simply don’t exist for arbitrary floating-point types, and (as you mention) preserving the option of using GPU acceleration also limits your options greatly.
Within your library, some very carefully written code might be agnostic to Float versus Double, but you don't have to go very far before you find that only one specific floating-point type can be used for this or that nontrivial feature and that the overhead of even the most optimized Float-to-Double conversion kills your performance.
So yes, floating-point types are exceptional in this way. Let me put it this way: knowing what I know now, I would not vend, nor would I use, a mathematical or scientific library that uses generic floating point; it's almost certainly not fit for purpose.
If you have to use @_specialize to make your library work, then you're doing something wrong: it's not a supported language feature and isn't meant for public consumption.
I’ve also written such a module (spherical voronoi tessellation), and I ended up writing it in Float because of lack of generic protocols. However at some point I want it to support statically generated hi-res maps (that you would save to a file and load at runtime, as opposed to computing low-res maps dynamically), which would at the minimum need Double as the base floating type. At the end of the day, we have different precision floating types for a reason, but without generic protocols to abstract over them, having those types is pointless, because you have to settle on “one canonical float format”.
So you've had the same experience as I have. What I'm saying to you is, it's not the state of generic floating-point protocols that's holding you back. There is nothing we can express in today's Swift that would allow the compiler to optimize generic floating-point code to the extent that it can for Float or Double (for example: GPU support, or taking advantage of SIMD instructions). It's in the nature of performance-sensitive work with floating-point types that you'll have to write algorithms that work with only "one canonical float format."