Spelling of relaxed floating-point arithmetic operations

Related Projects Swift Numerics
1

Hi all--

I'm looking into adding "relaxed" floating-point operations in Swift Numerics to allow the compiler to loop-unroll and vectorize floating-point accumulations. See feature request and PR for more information.

At present, the implementation uses a straw-man spelling of _relaxedAdd and _relaxedMul. I would like to solicit ideas about how these should eventually be exposed as "real API". Some options:

  • &+ and &* are available for FloatingPoint. While "relaxed" operations are not the same as wrapping operations on integers, they are spiritually quite similar--they relax the semantics of the type to make the operation associative and unlock compiler optimizations. On the other hand, there are also important differences: &+ and + always either produce the same result on integers, or + traps. This is not true for the relaxed floating-point operations; they can simply produce different results than normal arithmetic does.

  • I could invent another "modified operator" spelling for these. I dunno what, but I'm sure we could come up with something.

  • I could add a module RelaxedFloatingPoint module that shadows on the standard library + and * with the relaxed operations when imported. This is nice in some ways, but (to me) unacceptable given that we don't have a mechanism to scope imports. The transformation would apply at file-scope, which is undesirable.

  • These could be placed in an enum namespace like the Augmented operations are: Relaxed.sum(a, b). This is nice and unambiguous, but I think too wordy for this use case. This might be acceptable given a sufficiently rich set of higher-level operations defined in terms of them, however. If users don't have to resort to the bare arithmetic operations very often, having a wordy spelling for them is OK.

  • ...

12 Likes
2

I don’t know if this is a good idea, but you could also have a RelaxedFloat or Relaxed<Float> type where the meaning of the default arithmetic operators changes. That assumes that you wouldn’t want to mix relaxed and non-relaxed arithmetic much (cause otherwise you’d be hopping in and out of this new type a lot) but I suspect that’s a valid assumption.

1 Like
3

I considered that, but in practice you do want to mix these operations pretty often (or at least, want to do relaxed accumulation on values produced via normal arithmetic, and then do normal arithmetic on the result). It's not a bad idea, but definitely introduces some friction I would prefer to avoid.

One in-between option for accumulation specifically would be to define +=(_:inout Relaxed<Float>,_:Float), which avoids the hassle on one side, but you still have to do something with the result, and it's not really clear how to generalize that nicely to multiplication.

2 Likes
4

I like that approach even if not identical semantically, mentally I’ve always thought of those operators as “please optimize this, I know what I’m doing here” - that is as you say spiritually close enough and would be concise.

7 Likes
5

It depends how much friction there is. If it was a simple wrapper view, moving between representations could be quite simple.

var value: Float = ...

value += .pi  // strict
value.relaxed += .pi
value.relaxed.multiply(0.5)

So then it's easy to do one-off calculations, and if you want to do a lot of them, you can just keep a wrapped value around as a variable, overriding its operators, and convert it back later with .strict or .base or something.

I don't think &+, etc would be appropriate. For me, they mean one very specific thing: overflow. And in particular, memory safety concerns if I'm using these near an UnsafeBufferPointer, as they could lead to my safety mechanisms not catching an invalid access. Floating points are used for numerical tasks, and as you say, the result of using them can even be more accurate numerically than the raw +, etc operators. We shouldn't make them look scary.

"relaxed" sounds to me like "oh, I don't really have any specific requirements about accuracy to a given precision, so make whatever optimisations you can".

4 Likes
6

How about |+ and |* ? Treat them as associative operations so they can be reordered (parallelized). Thus the |. :smile:

7

My two cents would be:

Using &+ for this purpose makes sense in the same way that << being originally given exponentiation precedence in Swift makes sense. It wouldn’t be unreasonable and is actually rather thoughtful, but could appear at first blush to be counterintuitive to users.

Relaxed.sum is clear and wordy. It feels in the spirit of truncatingRemainder(dividingBy:). There is zero cleverness but it will work, for sure. This is the safest bet I think.

I am not enamored of inventing yet another “alternative operator”—it has readability challenges and we could all do without the bikeshedding exercise. Shadowing the standard library operators is a bit too magical (and with the core team resolved never to overload functions on return value, I can’t imagine they would be enamored of this kind of overloading either if we were to try to incorporate these facilities into the standard library), and I agree the lack of scoped imports makes it a non-starter anyway.

5 Likes
8

While &+ and &* may currently be defined as Overflow Operators, I do too think that &+ and &* suit perfectly as the operators for relaxed floating-point arithmetic – if we were to define the guarantees of these operators mostly as:

The fact that "&+ and + always either produce the same result on integers, or + traps" is then a special case for integer arithmetic, that is not necessarily guaranteed by the operator.

As for the other options, although it may not be to ergonomic, I would prefer Relaxed.sum(a, b) over a modified operator.

1 Like
9

The whole point of &+ and other overflow operators is that they overflow or underflow instead of trapping, and this behavior is not unique to integer arithmetic; it applies to floating point arithmetic as well. Therefore, I don't understand what this "special case" is that you're referring to.

The &+ operator is guaranteed to overflow instead of trapping and the + operator is guaranteed to trap on overflow. What behavior is "not necessarily guaranteed by the operator"?

10

Sorry if this statement caused some confusion. I didn't mean to change the guarantees of the &+ or &* operator for integer arithmetic (nor + or * for that matter) - but rather lift this requirement for the &+ and &* operators on a generic numeric type (RelaxedArithmetic for example).

11

Also agree with this specific feedback. The readability challenge for new users of swift is stacking up imho (also an issue with many proposals for sugar overall as I see it…).

3 Likes
12

These operators do not overflow or underflow on integers—those terms refer to exceptional conditions. These operators wrap, which means that (unlike + and *) they never overflow. No integer operator underflows.

This behavior is unique to integer arithmetic. There is no notion of wrapping for (IEEE 754) floating-point arithmetic.

9 Likes
13

Oh, you wouldn't need _modify for a write-through wrapper :man_facepalming:. You'd only need that if you had to avoid COW, which obviously isn't a problem for scalars.

So we could do this kind of thing just with regular get/set.

1 Like
14

I read the motivation and the entire thread here. This is something I’d be very interested in and can imagine using quite a bit.

The best suggestion I’ve seen so far is to namespace the operations in either something like Relaxed.sum(a, b) - it’s clear and in my opinion / imagined use cases not too verbose. Namespacing / imports would work well at a fine-grained level. The only problem I see with this is discoverability - if I hadn’t read this thread I’d never know to look for it there.

Alternatively, taking the example of an API like .truncatingRemainder(…), I could also imagine something like myFloat.relaxedAdd(otherFloat) and the like. Assuming Numerics may eventually find it’s way into the standard library (?) this alternative would be preferable for discoverability IMHO.

Ah, I wanted to add: I’m not a fan of using a type to express this (including the variant with myFloat.relaxed). To me this is not a different type but rather a different kind of operation. I could be convinced with some comparable examples though. The closest I can think of is .lazy, but I consider .lazy collections to be composable in a different way. I assume we can’t add a .relaxed with a non-.relaxed float - to me it gets murky pretty quickly.

15

Your argument makes sense!

Looks like the Swift docs need updating in that case: Advanced Operators — The Swift Programming Language (Swift 5.7)

4 Likes
16

In addition to the conciseness, I think one advantage of operators over (non-operator) functions is that they flatten out long expressions to make them easier to read.

For example, calculating the area of a triangle using Heron's formula:

func area(a: Double, b: Double, c: Double) -> Double {
    ((a + b - c) * (b + c - a) * (c + a - b) * (a + b + c)).squareRoot() / 4
}

is long but readable. However, if each operator is used like a function, it becomes much less readable with many layers of parentheses:

func area(a: Double, b: Double, c: Double) -> Double {
    /(*(*(-(+(a, b), c),-(+(b, c), a)),*(-(+(c, a), b),+(+(a, b), c))).squareRoot(), 4)
}

If the functions are spelled out and placed in an enum, then you're more or less forced to write a biig pyramid:

func area(a: Double, b: Double, c: Double) -> Double {
    Relaxed.multiply(
        Relaxed.multiply(
            Relaxed.multiply(
                Relaxed.subtract(c, from: Relaxed.add(a, b)),
                Relaxed.subtract(a, from: Relaxed.add(b, c))
            ),
            Relaxed.multiply(
                Relaxed.subtract(b, from: Relaxed.add(c, a)),
                Relaxed.add(Relaxed.add(a, b), c)
            )
        ).squareRoot(),
        0.25
    )
}

Would ~+ and ~* (also ~- and ~/ if relaxed subtraction and division are added too) work? I feel "~" conveys the sense of "not strict, relaxed".

4 Likes