[Pitch] Revising certain behaviors of `Decimal`

Development Foundation
1

Hi all,

As promised, I've gathered some API- and policy-level tweaks that I think should be considered with the implementation-level renewal of Decimal. I'd like to pitch those here:

Revising certain behaviors of Decimal

  • Proposal: SF-NNNN
  • Authors: Xiaodi Wu
  • Review Manager: TBD
  • Status: Awaiting review
  • Bug:
  • Implementation: 1
  • Review: (pitch)

Introduction

This document proposes to specify and regularize certain behaviors of the Decimal type and to deprecate some APIs retained from an earlier stage of Swift's evolution. Explicitly omitted are any changes in the type's representation or new APIs.

Motivation

Foundation.Decimal is Swift's existing general-purpose decimal floating-point type. It pre-dates Swift's floating-point protocols and differs from standard library binary floating-point types in fundamental respects: for example, Decimal does not represent infinity or negative zero.

Recent implementation work has substantially rewritten Decimal arithmetic, formatting, conversion, and normalization for performance and correctness. With that, it is now possible to deliver on certain behavior guarantees that were previously inconsistently honored or underspecified.

Proposed solution

Arithmetic rounding mode

Specify that Decimal arithmetic operations use .bankers rounding mode (i.e., round to nearest, ties to even) instead of .plain rounding mode.

This would be in line with the rounding convention for ordinary binary floating-point arithmetic (and also implicitly used already in some Decimal APIs). It is also what some users have said they expect for a decimal floating-point type for financial systems.

The change would not affect legacy NSDecimal* functions, which explicitly accept a rounding mode and continue to permit callers to request .plain or another mode.

Changing a default generally deserves considerable caution. Here, however, existing behavior is a particularly weak compatibility constraint: many arithmetic operations did not in fact round as documented (and/or did not have the precision necessary to do so). With recent improvements in correctness and precision, now is a key time to establish a consistent, desirable default.

Conversions between Double and Decimal

Specify explicit semantics for conversion in each direction.

There are (at least) two reasonable ways to convert a Double to a Decimal: the result can be either the nearest to the Double (at the full precision afforded by Decimal), or it can be the shortest (i.e., one with the least precision necessary, ties to nearest or even) for which the original value is the nearest Double.

It's possible that we will eventually want explicit APIs for both ways of converting. However, irrespective of hypothetical new APIs, there's an existing unlabeled converting initializer that's not going anywhere. Its current implementation doesn't convert precisely enough to align with either of the desired behaviors. A revised implementation that would improve on it needs to wait for a policy decision (this proposal) on what the behavior should be:

Decimal.init(_ value: Double) should give the shortest Decimal representation of value, just as value.description gives the shortest decimal string representation. This preserves a useful and intuitive relationship: when a user expresses a Double by its shortest roundtripping decimal representation, converting that value to Decimaldoesn't result in additional decimal digits that arise solely from the binary representation:

let x = 0.1
Decimal(x) == Decimal(string: x.description)

Decimal's float literal initializer would adopt the same semantics so that near-equivalent ways of constructing a Decimaldo not unexpectedly produce different values.

In the other direction, Decimal.doubleValue should give the nearest Double representation, such that Double-to-Decimal roundtrip conversion (unless the result is overlarge, etc.) parallels the lossless behavior of Double-to-Stringroundtrip conversion.

The idea is that these behaviors would be the least astonishing for most users.

Constants and floating-point classification

Correct some existing constants and floating-point classification properties.

Currently, Decimal.leastNonzeroMagnitude has exponent -127 when it should have exponent -128 (because that's the exponent of the actual least nonzero magnitude). The current state of affairs arose as a typo; it was corrected for a time in swift-corelibs-foundation but never synchronized in the Apple overlay, and it has now regressed for all platforms in swift-foundation.

Currently, Decimal.pi is rounded toward zero, in line with prior semantics for FloatingPoint.pi; it should now be rounded to nearest in tandem with SE-0552 adopting a revised rounding requirement for FloatingPoint.pi.

Subnormal values

Currently, Decimal classifies every finite nonzero value as normal, since it doesn't use an IEEE format. But, just like subnormal values in IEEE formats, finite nonzero values with sufficiently small exponent can only be represented using fewer significant digits than fit in the type's significand (mantissa) capacity.

It would be more useful to classify values in that reduced-precision range as subnormal. leastNormalMagnitude would then change to be (UInt128.max / 10 + 1) * 1e-128, and isSubnormal, isNormal, and floatingPointClass would be updated consistently.

(If this notion of subnormals is not adopted and no value is to be subnormal, leastNormalMagnitude must still be changed so that it continues to be equal to leastNonzeroMagnitude.)

Canonicality and total ordering

Two existing APIs require either corrected semantics or deprecation: isCanonical and isTotallyOrdered(belowOrEqualTo:).

Currently, isCanonical is true unconditionally. That's difficult to reconcile with the fact that Decimal has compact and (possibly multiple) non-compact encodings of the same value. It's rather misleading for an API named isCanonical to be true for different encodings of the same value, even if it's not an IEEE format.

Meanwhile, isTotallyOrdered(belowOrEqualTo:) doesn't actually provide a true total ordering over the complete set of valid Decimal values. If the API is to be retained, its behavior must be defined for all valid Decimal values, and the treatment of noncanonical or malformed representations must also be considered. Otherwise, deprecation is preferable to an operation that promises semantics it doesn't provide.

Other deprecations

Parts of Decimal's public API surface were added to align with early drafts of Swift's floating-point protocols. The standard library subsequently changed those protocols, abandoning certain spellings, but Decimal continued to retain those abandoned forms.

Deprecate APIs with abandoned spellings that are redundant, misleading, or both:

  • quietNaN in favor of nan
  • isSignaling in favor of isSignalingNaN
  • leastFiniteMagnitude in favor of -Decimal.greatestFiniteMagnitude — the constant actually gives the most negative finite value, which is not a (non-negative) magnitude
  • isEqual(to:), isLess(than:), and isLessThanOrEqualTo(_:) in favor of ==, <, and <=
  • add, subtract, multiply(by:), and divide(by:) in favor of arithmetic operators

Source compatibility

Arithmetic results may differ where an operation requires rounding, but (as described above) users couldn't rely on accurate rounding in many cases. Except in midpoint cases, any results that were correctly rounded remain so, and all legacy NSDecimal* APIs retain their existing explicit rounding modes.

Conversion results would change where the old implementation failed to choose the shortest or nearest representation (as the case may be) accurately. The corrected constants and changes to subnormal classification would change for code that explicitly observes the relevant properties. These observable changes replace behavior that was insufficiently specified and/or incorrect with behavior that users can reason about and rely on.

It's unlikely that the APIs proposed for deprecation are in wide use, but in any case deprecation is source-compatible and directs users to more idiomatic spellings that are by now longstanding.

Implications on adoption

In that this proposal introduces no new features, implications on adoption are limited to the source compatibility concerns detailed above.

Future directions

API additions to expose additional facilities directly on the Decimal type, some not currently available at all and some only via legacy NSDecimal* APIs, can be the subject of future proposals. This might include, for example, arithmetic with explicit rounding mode and scale parameters, conversions to and from (U)Int128, a rounded(_:scale:) API, or additional rounding modes.

Alternatives considered

A broad alternative is to stick with .plain rounding as default, leave conversion behavior unspecified, and to maintain current constants and duplicative APIs, all on compatibility grounds. That is unattractive for Decimal because much of the behavior addressed here was never a coherent semantic contract: rounding was not implemented consistently, binary floating-point conversion behavior arose from implementation limitations, constants and floating-point classification APIs were inaccurate, and public members were retained from protocol designs subsequently abandoned.

11 Likes
2

What is the behavior when running code built against a newer foundation on an older OS; does the revised rounding behavior back deploy, so that we can get the same result on older systems, or would users see different results from the same binary?

Also, if someone wants to continue getting the existing behavior (possibly with bugs fixed), how should they do that?

(There isn't necessarily a right answer to these questions, but a proposal like this should be clear about what the change will be).

2 Likes
3

Given our experience of the decimal creation bug fix in iOS 18 causing financial apps to display incorrect amounts (Foundation #833) I think it's important that whatever changes are made to existing Decimal APIs that can in any way change the results, there needs to be backward binary compatibility for exact results.

Obviously since Decimal is performance-sensitive, that might need to be a renamed symbol, rather than an actual linked-on-or-after check.

4

Good points, which I’ll incorporate into the text—

None of the Decimal APIs are inlinable, unless I’m mistaken, so on a given ABI-stable older system behavior will be consistent among all apps, and behavior will be different when a given binary runs on systems of different vintages. This would not be limited to rounding mode but extend to all quirks of the current implementation, bugs and all.

They can use the NSDecimal* APIs that take an explicit rounding mode argument to get actual .plain rounding, and as a future direction the addition of methods to Decimal can be contemplated to make it ergonomic.

That said, .plain rounding is the intended but not actual “existing” behavior; most (nearly all) operations lack the precision as of 6.4 to be rounded meaningfully in any way at all, and a bunch arbitrarily truncate but only along some code paths.

3 Likes
5

Thanks @xwu for trying to clean this up -- not sure if it fits I this context, but would like to point out one missing API surface for Decimal also to make it useful for high performance use cases when it comes to serialization -- I know you saw this issue, but might not remember -- we are lacking efficient serialization/deserialization hooks (which is not Codable) -- here is one possible suggestion:

extension Decimal {
    public init(_ binaryRepresentation: RawSpan) {}
    public func withRepresentation<R>(_ body:(RawSpan) throws -> R) throws -> R
}

Also it would be tremendously useful if we could have either the size (or if we don't want to commit to a size, then the possible max size) of the Decimal representation to be documented and committed to, such that we can map this to a fixed-size data type for serialisation purposes if possible.

Background: Decimal: different access level for fields and initializer on MacOS and Linux · Issue #934 · swiftlang/swift-foundation · GitHub

This may fit if we are viewing as overall API surface cleanup.

1 Like
6

The issue you mention arose from a confluence of issues not implicated here, and fundamentally an organizational one where the Apple overlay and the Linux version fell out of sync for several years and then suddenly resynced in an unfortunate way. The consequence of that was also particularly unfortunate in that it caused differences in sign at runtime, which is clearly…not good and generally ought not to be possible for a published type that ships with the toolchain.

But improvements in precision and accuracy of arithmetic operations have always been on the table for numeric types, binary floating-point and otherwise (if sin can be more accurately rounded on a system then all the better). The changes proposed here are related to such improvements in that they mostly have to do with last decimal places (and in a type with a lot of precision, more than the IEEE decimal128 type).

I think most would agree that restricting any such changes forever is probably not the right tradeoff. We are, for example, discussing changing the value of Float.pi in the other thread. But point well taken that the effect on clients needs to be thought through here.

4 Likes
7

So, fun fact: the swift-foundation Decimal type declaration is not used on Apple platforms, where instead the bulk of the now shared swift-foundation code extends a type declared in C.

As it happens, the swift-foundation type declaration is not frozen. And, to my chagrin and now yours, while code comments indicate that the flags should be laid out in a certain way (which would then be identical to the C layout), the not-shared code then proceeds to use an unintentionally different layout.

Which is to say we have some work to do before we have “a” binary representation that we would want to publicly expose, a worthwhile future direction…

4 Likes
8

Reflecting on this point:

It would be quite reasonable—and the wiser move—to maintain binary compatibility entrypoints for legacy Decimal.init(_ value: Double) and doubleValue [doubleValue is not public API]. These are not well understood arithmetic operations with an obviously "correct" result, and the existing implementations could be load-bearing (quirks and all). Same reasoning if we decide to fix rather than deprecate isTotallyOrdered.

It is possible, though more far-fetched, to imagine load-bearing uses of isNormal as a synonym for != 0, and little reason to think anyone is relying on isSubnormal as a synonym for false or isCanonical as a synonym for true. But if it's felt wiser we can straightforwardly adopt binary compatibility entrypoints for those and floatingPointClass. Whatever the standard library ends up doing for Float.pi in terms of compatibility, we should do for Decimal.pi and the other constants.

1 Like
9

Thanks for the write-up! I think the deprecation all makes sense. While I also have concerns about compatibility I don't think there's a reason to treat them differently from other bugs; we'll just have to treat them with caution and revisit each of the decision as we find bugs.

It seems unfortunate that we'd have to ask those who want the .plain behavior to use NSDecimal* APIs that take an explicit rounding mode. Why don't we just introduce the proper Swift rounding function that takes a rounding mode here together?

10

My thinking here is that I'd limit this particular proposal to just existing APIs. There's a bunch of missing APIs that are obviously nice to add--including one like what you describe--which I would want to include as a follow-up to this proposal in short order (and ideally in the same release). However, just the API changes are substantial enough that I don't want to expand the scope of the proposal here unless we absolutely have to.

When it comes to introducing new APIs, unfortunately, there are caveats as to what it means to do that "together" with changes to existing APIs on this type--limitations that cause me to think that the benefit of simultaneous consideration in one proposal isn't gaining us much:

As you know, the naive approach would be to add new APIs with Swift Next availability, limiting their use in codebases that want to support older platforms. In that case, many would turn to NSDecimal* APIs with longstanding availability anyway. If (on the other hand) we'd like these API declarations to backdeploy, the implementation would need to call into an existing usable-from-inline entrypoint that takes a custom rounding mode, which (unless I'm missing something) could only be the NSDecimal* APIs. However, that'd be a layering violation as the NSDecimal* APIs themselves live in Apple's proprietary wrapper and in the open-source wrapper that's now swift-corelibs-foundation. You can tell me if sinking these NSDecimal* declarations into swift-foundation is something that's can be "just" done, but it can't be done by a non-Apple contributor :)

When it comes to arithmetic APIs that take custom rounding mode, specifically, there are a few open design questions worth exploring holistically (better discussed in its own pitch). Some thoughts to demonstrate what I mean:

  • Probably, these APIs should also accept a scale parameter (e.g., round to two decimal places, etc., rather than to the full precision of the type). There are NSDecimal* APIs that are advertised with such functionality but they (unfortunately) currently double round, when in fact the revised Decimal internals can now give correctly scaled-and-rounded results in one go. (Either as public API, or as SPI, the internal implementation should also be exposed so that swift-corelibs-foundation and Apple's proprietary wrapper can call into them when available.)

  • While we're at it, we should probably have these APIs report out calculation errors (inexactness, overflow, etc.) to allow users to decide how to handle them. The design question would be if that should all be found in one advanced do-it-all API or whether we should have yet another set of arithmetic APIs with a signature like func addingReportingInexact(_: Decimal, roundingMode: RoundingMode, scale: Int) throws(CalculationError) -> (value: Decimal, inexact: Bool).

  • Finally with all of these knobs now explicitly spelled out, we'd want to explore ways to improve the ergonomics of their use. It'd be nice if it were possible to specify (perhaps with a macro that would be the modern moral equivalent of NSDecimalNumberBehaviors?) the default rounding mode, scale, and calculation error handling for an entire lexical scope. Then, a user could write a * b and the compiler could rewrite that as a.multiplied(by: b, roundingMode: .plain, scale: 3). Is it feasible? I haven't prototyped this part yet to know.

Anyway, lots to think about such that I think it deserves its own discussion.

11

I should mention that I've slimmed down the list of deprecations based on implementation experience and expanded on source compatibility based on the feedback here in a revised text, now posted as a PR:

https://github.com/swiftlang/swift-foundation/pull/2266

I now propose to keep an explicit legacy entrypoint for clients compiled against prior versions to maintain the existing behavior of init(_: Double) and init(floatLiteral: Double).

12

It's also worth point out, I think, that among the "obvious" APIs we'll want to add in follow-on is an actual .round(...) API, which is not public currently.

So, yes, it's weird that getting .plain rounding for your arithmetic requires going to NSDecimal* APIs, but it's actually the status quo that even rounding the number requires NSDecimal* APIs.