Operator syntax for vector and matrix arithmetic

Using Swift
1

I'm working on a Swift package where I need to define operators for performing various arithmetic operations on vector and matrix structures. The operations are implemented by various functions in Accelerate for doing element-wise addition, matrix multiplication, etc. Operators for vector and matrix arithmetic tend to vary between different languages and packages. For example, element-wise matrix multiplication is performed with .* or * depending on the language or package. In the table below I tried to summarize some of the operators for NumPy, Julia, and MATLAB. In the last column I listed my preference for implementing these operators in a Swift package. I personally don't like the "dot" syntax because it clutters up the code with a bunch of dots. I also need to consider compound assignment operators for vectors and matrices such as *= and += where I feel the dot syntax would be even more untidy, for example .+=. However, I would like to get other peoples thoughts on these operators. So for a Swift package, what operator syntax would Swift users prefer for vector and matrix arithmetic? Are there other languages and/or packages that I should look at for inspiration?

Description NumPy Julia MATLAB Swift
Element-wise addition + .+ + +
Element-wise subtraction - .- - -
Element-wise multiplication * .* .* *
Matrix multiplication @ * * **
Element-wise division / ./ ./ /
Element-wise power ** .^ .^ ^
2 Likes
2

The only thing I think Python got right is + and -, but I think it would be bad to let their choice of * and / infect other languages. I would follow MATLAB's or Julia's notation.

In my personal Swift matrix packages, I've allow both ⨀ and .*, but unfortunately ⊘ (for ./) looks terrible for some reason, and there is no circled ^ equivalent. (Many weird problems in the standard characters that I wish could get fixed.)

Using ** for multiplication is just asking for confusion.

4 Likes
3

I use these operators in one of my apps for Linear Algebra.

+  := add
-  := sub
*  := mul
/  := div
@  := transpose
~  := inverse
~| := determinant
~@ := cofactors
4

So you don't mind having a bunch of dots in your code? Something like
(a .* b).^2 .+ (c ./ d).^3
drives me crazy. It looks much cleaner like this
(a * b)^2 + (c / d)^3

1 Like
5

What do you use for element-wise matrix multiplication and matrix multiplication? Also, I tried using @ as an infix operator but Swift does not allow it.

7

Well, in matrix code, I rarely if ever do element-wise multiplication between two matrices, so an equation like that wouldn't come up. I could imagine creating a different object which is a bag of real numbers that used * and ^ as you describe, and then ok, bag₁ * bag₂ could be fine to let it be element-wise, but I like to keep these concepts separate.

2 Likes
8

Yes, that's the kind of scenario I'm dealing with. But there's still the issue of what to use for matrix multiplication (not element-wise). As you mentioned, ** is not a good choice and @ is not allowed as an operator in Swift. So beyond unicode characters, there's not any other options that I know of. What about the following as a compromise (see Swift column in table)?

Description NumPy Julia MATLAB Swift
Element-wise addition + .+ + +
Element-wise subtraction - .- - -
Element-wise multiplication * .* .* .*
Matrix multiplication @ * * *
Element-wise division / ./ ./ /
Element-wise power ** .^ .^ ^
9

Personally I like Julia’s consistency in using the . prefix for element-wise operations. And I think that’d be helpful in Swift because the operators aren’t built in, which makes overload resolution easier for the compiler.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

5 Likes
10

Swift’s SIMD uses plain operators (like + and /), using the . prefix only for operators that would otherwise conflict with protocol-defined operators (like .!= and .<).

1 Like
11

That would prevent vectors and matrices from conforming to AdditiveArithmetic.

I would go with:

Description NumPy Julia MATLAB Swift
Element-wise addition + .+ + +
Element-wise subtraction - .- - -
Element-wise multiplication * .* .* .*
Matrix multiplication @ * * *
Element-wise division / ./ ./ ./
Element-wise power ** .^ .^ .**
Element-wise xor bitwise_xor bitxor .^
1 Like
12

One thing I noticed about the dot syntax is it's obvious that vector or matrix arithmetic is being performed.

If element-wise matrix addition is performed with + as shown below, then it's not entirely obvious if a and b are scalars or matrices. The a + b may be defined somewhere in code that is not near the creation of the matrix variables so just looking at a + b could mean adding two floats, doubles, or integers or it could mean adding two matrices.

let a: Matrix<Double> = [[1, 2, 3], [4, 5, 6]]
let b: Matrix<Double> = [[2, 3, 4], [7, 8, 9]]
let c = a + b

If dot syntax is used such as .+, then it's obvious that a .+ b is adding two vectors or matrices element-by-element. And because .+ does not exist in Swift, then it's not likely to get confused with adding two floats, doubles, or integers.

let a: Matrix<Double> = [[1, 2, 3], [4, 5, 6]]
let b: Matrix<Double> = [[2, 3, 4], [7, 8, 9]]
let c = a .+ b

So I guess the Julia approach is best here. But Julia does support different operators for the same operation, such as + and .+ both perform element-wise matrix addition. I could do the same in Swift:

Description NumPy Julia MATLAB Swift
Element-wise addition + .+ + .+ or +
Element-wise subtraction - .- - .- or -
Element-wise multiplication * .* .* .*
Matrix multiplication @ * * *
Element-wise division / ./ ./ ./
Element-wise power ** .^ .^ .^
13

Some thoughts:

  • Inner product and matrix multiplication are more common than element-wise multiplication, I think.
  • Since there's no such thing as non-element-wise addition and subtraction, perhaps those operators should just be + and - so that vectors and matrices can conform to AdditiveArithmetic.
  • There is also the inner product (also known as dot product or scalar product) of two vectors.
  • Vector * scalar, Scalar * vector, Matrix * scalar, Scalar * matrix, Matrix * vector, vector * matrix all make sense as well.
  • Square matrices also have an exponentiation operation (T^n = T * T * T * ... n times, via the usual matrix multiplication).
  • Exponentiation (either the ring operation or element-wise) should not use ^, because that's bitwise XOR on integers.
3 Likes
14

Since there's no such thing as non-element-wise addition and subtraction, perhaps those operators should just be + and - so that vectors and matrices can conform to AdditiveArithmetic

I would rather use .+ and .- to be consistent with .* for element-wise operations.

There is also the inner product (also known as dot product or scalar product) of two vectors

For things like dot product, I just use a dot function or the ⋅ operator.

Vector * scalar , Scalar * vector , Matrix * scalar , Scalar * matrix

These are element-wise operations so I'm leaning towards .* for these as well. This would be consistent with element-wise matrix multiplication. Otherwise, you would have code like Scalar * Matrix and Matrix .* Matrix which are both element-wise operations. I'm not crazy about having different operators for the same kind of operation.

15

I agree with Slava that matrix multiplication is far more common than elementwise multiplication, so I would expect to see Scalar * Matrix and Matrix * Matrix together far more often than Scalar * Matrix and Matrix .* Matrix.

2 Likes
16

I was imagining Matrix * Vector and Vector * Matrix are not element-wise, but linear operator application.

When does element-wise multiplication of vectors and matrices come up?

1 Like
17

When does element-wise multiplication of vectors and matrices come up?

When a matrix represents the state of a system at a particular time step. For example, each element in a matrix could represent a chemical concentration in a 2D grid. At each time step that grid is updated to represent the change in chemical concentrations. The Gray-Scott model for reaction-diffussion is an example. This model uses element-wise multiplication for matrix-to-matrix and scalar-to-matrix operations. In the website I linked to, the A and B represent 2D matrices.

2 Likes
18

When it's not a matrix in the linear-algebra sense, but rather a 2D grid of values on which vector operations are to be performed (or even a 1D grid of vector values, like color data).

This confusion is more or less unavoidable when using a single type to represent both linear operators and 2d data. Some uses want the elementwise operations to be primary, some uses want the matrix ring structure to be primary. A library has to decide which it cares about more.

Elementwise addition and subtraction are the same as the ring-structure addition and subtraction for vectors and matrices, so there's no need to make this choice. You can have both, and have them mean the same thing, or just provide + and - since those are unambiguous (at least until the Array and String people come along).

For ML/data-processing focused libraries, making the elementwise operations be primary is quite normal and defensible. Making the ring multiplication primary for linear algebra-focused libraries makes some sense, but really only in the simpler cases--usually you want not just to multiply two matrices, but to update a result matrix in place, or make a linear combination of the product with some other matrix, or take advantage of some symmetry in one of the operands, or to provide scratch space or threading control arguments, and so you end up needing more parameters than an operator lets you have anyway. So just spelling out the ring operations as functions becomes pretty attractive. (e.g., even math-focused languages like Julia have matrix-multiply operators, but also provide explicit BLAS bindings as functions)

3 Likes
19

Is there a good argument for splitting the use-cases into two separate types? For example, if I just need a pair of floats, I wouldn't want to use a Complex type.

1 Like
20

It’s common to use a simd_float2 (or similar vec2) for this, though. Does that make your pair a vector?

1 Like
21

What do you mean by "ring multiplication"? Is this the same as matrix multiplication (not element-wise)? What do you mean by "spelling out ring operations as functions"? Is that like using a function named matmul to perform matrix multiplication?

I'm coming at this from a data processing/data science/ML perspective. So for me, element-wise operations are the primary focus. I do element-wise matrix multiplication more often than matrix multiplication. I honestly don't remember the last time I did a linear algebra matrix multiplication in code. I would personally like to use +, -, *, /, ^ for element-wise operations but what would I use for matrix multiplication? NumPy uses the @ symbol but that's not allowed in Swift. I could have a matmul function and use it like shown below. But it would be convenient to do this with A @ B.

let A: Matrix<Double> = [[1, 0, 1],
                         [2, 1, 1],
                         [0, 1, 1],
                         [1, 2, 3]]

let B: Matrix<Double> = [[1, 2, 1],
                         [2, 3, 1],
                         [4, 2, 2]]

let AB = matmul(A, B)

Oh, I just now discovered a Consortium for Python Data API Standards. In their Array API standard they use * for element-wise matrix multiplication and @ for matrix multiplication.