Histogram (HDR Histogram native Swift port) package

We are happy to share Histogram with the community and hope it will be useful for many of you.

Histogram is a port of Gil Tene's High Dynamic Range (HDR) Histogram to native Swift. It provides recording and analyzing of sampled data value counts across a large, configurable value range with configurable precision within the range. The resulting "HDR" histogram allows for fast and accurate analysis of the extreme ranges of data with non-normal distributions, like latency.

Histogram supports the recording and analyzing of sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording, and provides control over value quantization behavior across the value range and the subsequent value resolution at any given level.

For example, a Histogram could be configured to track the counts of observed integer values between 0 and 3,600,000,000 while maintaining a value precision of 3 significant digits across that range. Value quantization within the range will thus be no larger than 1/1,000th (or 0.1%) of any value.

This example Histogram could be used to track and analyze the counts of observed response times ranging between 1 microsecond and 1 hour in magnitude, while maintaining a value resolution of 1 microsecond up to 1 millisecond, a resolution of 1 millisecond (or better) up to one second, and a resolution of 1 second (or better) up to 1,000 seconds. At its maximum tracked value (1 hour), it would still maintain a resolution of 3.6 seconds (or better).

Histogram is designed for recording histograms of value measurements in latency and performance sensitive applications.

Measurements show value recording times as low as 3-4 nanoseconds on Apple Silicon CPUs (M1).

The Histogram maintains a fixed cost in both space and time. A Histogram's memory footprint is constant, with no allocation operations involved in recording data values or in iterating through them.

The memory footprint is fixed regardless of the number of data value samples recorded, and depends solely on the dynamic range and precision chosen. The amount of work involved in recording a sample is constant, and directly computes storage index locations such that no iteration or searching is ever involved in recording data values.

For more details:

12 Likes

For background "why" to use this, I'd suggest (any of) Gil's presentations

Also, by printing the description of a histogram with just print("\(histogram)") and saving it to a file:

        Value     Percentile TotalCount 1/(1-Percentile)
------------ -------------- ---------- ----------------
       1.000 0.000385333041     500933             1.00
     260.000 0.100001190000  130001559             1.11
     517.000 0.200088646915  260115265             1.25
     738.000 0.300192748444  390250609             1.43
     959.000 0.400288279204  520374811             1.67
    2187.000 0.500014406922  650018789             2.00
    2919.000 0.550058224610  715075758             2.22
    3651.000 0.600093290761  780121350             2.50
    4247.000 0.650366496120  845476523             2.86
    4715.000 0.700366860735  910477003             3.34
    5503.000 0.750150457678  975195685             4.00
    6151.000 0.775066834609 1007586978             4.45
    6803.000 0.800133618449 1040173800             5.00
    7451.000 0.825051931534 1072567610             5.72
    8099.000 0.850154393832 1105200814             6.67
    8719.000 0.875132500757 1137672356             8.01
    9031.000 0.887705988827 1154017892             8.91
    9343.000 0.900270770744 1170352110            10.03
    9647.000 0.912515522691 1186270289            11.43
    9959.000 0.925086393838 1202612423            13.35
   15879.000 0.937509102691 1218761946            16.00
   19295.000 0.943765641345 1226895447            17.78
   22703.000 0.950009839230 1235012905            20.00
   26111.000 0.956251769423 1243127415            22.86
   29535.000 0.962525461921 1251283216            26.68
   32959.000 0.968791639035 1259429247            32.04
   34655.000 0.971899204132 1263469082            35.59
   36351.000 0.975004798461 1267506355            40.01
   38079.000 0.978169358938 1271620284            45.81
   39775.000 0.981276464805 1275659522            53.41
   41471.000 0.984380645288 1279694957            64.02
   42335.000 0.985961380527 1281749913            71.23
   43199.000 0.987544090381 1283807436            80.28
   44031.000 0.989068701009 1285789430            91.48
   44895.000 0.990648506248 1287843177           106.93
   45759.000 0.992230585333 1289899880           128.71
   46175.000 0.992991899878 1290889589           142.69
   46591.000 0.993754032115 1291880361           160.10
   47039.000 0.994573748193 1292945992           184.29
   47455.000 0.995336162738 1293937131           214.42
   47871.000 0.996099291899 1294929199           256.36
   48095.000 0.996509623399 1295462630           286.50
   48319.000 0.996920156438 1295996323           324.69
   48511.000 0.997271172560 1296452644           366.46
   48735.000 0.997682061752 1296986800           431.42
   48959.000 0.998092552484 1297520438           524.26
   49055.000 0.998268727083 1297749465           577.61
   49151.000 0.998444678605 1297978202           642.95
   49279.000 0.998678831660 1298282601           756.91
   49375.000 0.998854519337 1298510995           873.00
   49471.000 0.999030400090 1298739640          1031.35
   49535.000 0.999147479309 1298891843          1172.99
   49599.000 0.999265125452 1299044783          1360.78
   49631.000 0.999323931601 1299121231          1479.14
   49695.000 0.999440911590 1299273305          1788.63
   49759.000 0.999557967733 1299425478          2262.28
   49791.000 0.999616761574 1299501910          2609.34
   49791.000 0.999616761574 1299501910          2609.34
   49823.000 0.999675600799 1299578401          3082.62
   49855.000 0.999734285409 1299654691          3763.44
   49887.000 0.999792913865 1299730908          4828.91
   49887.000 0.999792913865 1299730908          4828.91
   49919.000 0.999851716937 1299807352          6743.86
   49919.000 0.999851716937 1299807352          6743.86
   49951.000 0.999910305393 1299883517         11148.94
   49951.000 0.999910305393 1299883517         11148.94
   49951.000 0.999910305393 1299883517         11148.94
   49951.000 0.999910305393 1299883517         11148.94
   49983.000 0.999968770772 1299959522         32021.28
   49983.000 0.999968770772 1299959522         32021.28
   49983.000 0.999968770772 1299959522         32021.28
   49983.000 0.999968770772 1299959522         32021.28
   49983.000 0.999968770772 1299959522         32021.28
   49983.000 0.999968770772 1299959522         32021.28
   49983.000 0.999968770772 1299959522         32021.28
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
   50015.000 0.999999923077 1300000020      13000001.20
  105535.000 0.999999929231 1300000028      14130436.08
  115903.000 0.999999940769 1300000043      16883118.45
  123135.000 0.999999946923 1300000051      18840581.44
  129727.000 0.999999953077 1300000059      21311477.40
  135935.000 0.999999958462 1300000066      24074076.31
  148095.000 0.999999964615 1300000074      28260872.17
  155007.000 0.999999970769 1300000082      34210529.54
  160767.000 0.999999973846 1300000086      38235297.62
  165503.000 0.999999976923 1300000090      43333337.16
  169983.000 0.999999979231 1300000093      48148152.76
  173439.000 0.999999982308 1300000097      56521744.33
  182527.000 0.999999985385 1300000101      68421058.56
  182783.000 0.999999986923 1300000103      76470595.24
  183295.000 0.999999989231 1300000106      92857150.86
  185215.000 0.999999990000 1300000107     100000008.38
  186879.000 0.999999991538 1300000109     118181828.92
  189439.000 0.999999993846 1300000112     162500013.17
  189439.000 0.999999993846 1300000112     162500013.17
  190719.000 0.999999994615 1300000113     185714299.80
  192511.000 0.999999995385 1300000114     216666686.83
  195583.000 0.999999996923 1300000116     325000038.06
  195583.000 0.999999996923 1300000116     325000038.06
  195583.000 0.999999996923 1300000116     325000038.06
  196479.000 0.999999997692 1300000117     433333384.08
  196479.000 0.999999997692 1300000117     433333384.08
  198271.000 0.999999998462 1300000118     650000029.21
  198271.000 0.999999998462 1300000118     650000029.21
  198271.000 0.999999998462 1300000118     650000029.21
  198911.000 0.999999999231 1300000119    1300000152.23
  198911.000 0.999999999231 1300000119    1300000152.23
  198911.000 0.999999999231 1300000119    1300000152.23
  198911.000 0.999999999231 1300000119    1300000152.23
  198911.000 0.999999999231 1300000119    1300000152.23
  199551.000 1.000000000000 1300000120                 
  199551.000 1.000000000000 1300000120                 
#[Mean    =     4884.998, StdDeviation   =     8192.428]
#[Max     =   199551.000, Total count    =   1300000120]
#[Buckets =           22, SubBuckets     =         2048]

One can use the online plotting functionality to get output such as:

4 Likes

Short update for any future goggles - it was now accepted as a port and thus moved to the proper upstream organization and is now renamed:

6 Likes

Very cool, I've seen the chatter in the repo :slight_smile:
Congratulations! What a small world it is... Very nice to see this become an official swift impl of the histogram :100:

2 Likes