Add `Float16`


(Steve Canon) #21

Are you guys actually doing accumulation in BFloat16? Intel's extension does accumulation in Float, and so do all the other public proposals that I've seen so far.

Edit: for the curious, here's Intel's white paper. They only define three operations: an FMA that accumulates a bfloat16 x bfloat16 product to a float32, and conversions between bfloat16 and float. So all arithmetic is done in float. Every other public proposal I've seen follows exactly the same pattern.


(Chris Lattner) #22

I don't believe that it is publicly documented.


(Happy Human Pointer) #23

This seems like an interesting idea. However, I do not see the general users of Swift taking full advantage of it. I understand use of Float, but most people just use Double. If there are any cases to use Float16 then please correct me.

This would require a reimplementation of log and other functions. That would lead to a lot of code reusing. Is gyb used on these functions?


(Steve Canon) #24

As mentioned upthread, Float16 is used heavily in ML and graphics APIs both in Apple's SDKs and on Linux. We would like people to be able to use those APIs from Swift.

No, it doesn't. Those operations are not required for BinaryFloatingPoint conformance, and these types are useful without them, to interoperate with the aforementioned APIs (the type would even be useful without any arithmetic at all, but we can and should provide it.)

That said, we will eventually provide the usual set of math functions for these types.


(Happy Human Pointer) #25

In what cases? I followed the thread but why would you not use just Float32?


(Steve Canon) #26

Because being able to transfer or use twice as many weights buys you more accuracy improvement than additional bits to represent each weight does, and doing computation in Float16 consumes less energy so you can get more useful work done before you run out of battery. There are just a few of the reasons why this type gets used pretty heavily now.

Ultimately, the exact use cases barely matter. The fact is that there are common APIs that cannot currently be used from Swift, and adding this type makes them usable. That's reason enough to have them.


#27

I support this. I'm also on the Float16 camp (without Half).

What is the interop story with __fp16 and _Float16 values coming from C?


(Steve Canon) #28

They use the same underlying LLVM type, so we can make the importer map them both fairly easily.