Benchmarks game?

I'm not saying anyone here is doing this, but I think an instinctive reaction to this kind of thing is to put one's back up and say "they're using it wrong!" Be that as it may, if newcomers repeatedly write slow code it's much more productive to treat it as a performance issue to be fixed (by making fast code the obvious code) than to try and educate one person or fix one test.

People will write code with large constant tables and then (perhaps rightly) say that C compilers have been able to handle these kinds of tables for 50 years and therefore Swift can't be that advanced if a bunch of lists causes it to fall over. Or that to get an algorithm fast they have to write using pointers, effectively jettisoning memory safety and writing the equivalent C code, so why bother with Swift?

I think the only reasonable approach to solve these problems is to design the language and documentation in such a way that this trap is fell into less often. With regards to pointers, I think Span and Vector/FixedArray show progress on that front. I'm not sure how to stop people from writing large constant lists -- sometimes it's the obvious solution, and there doesn't seem to be a great alternative offerred.

4 Likes

There's no inherent reason large collection literals must be slow to type check, though. There are many possible improvements we could make in the fullness of time. It's a bug that it's sometimes slow today.

7 Likes

For what it's worth, "performance predictability" is an ongoing key focus area in Swift's development.

2 Likes

at some superficial level, the existence of Swift Performance Predictability Effort is a plausible rebuttal to the various criticisms aired in this thread. but the gritty reality is that “Performance Predictability” is about putting advanced tools in the hands of expert users who are already intimately familiar with Swift’s performance behavior. these are advanced features that beginners should not be expected to be aware of, or even understand why they are necessary in the first place.

1 Like

eh I mean OSSA should help everyone

2 Likes

this code may well be an entire lecture in my next swift class.

Moderator note: Communication has really broken down in this thread over the last few days. I think there's something that can be salvaged here, so I've decided to just move all of those posts out, in an effort to reset the discussion. I've also asked Isaac to try to be clearer in his future posts about what he means rather than just posting links as if they stand on their own. I'm re-opening the thread, and I'll keep an eye on it for a while.

11 Likes

plb2

kostya

I am not.

I am not, because the Swift code was transliterated from Dart.

And, in any case, the "#8" sub-project was already finished in December last year — without new crowd-sourced code.

And, in any case, binary-trees programs are not actually shown as-part of that "#8" sub-project.

And, I doubt that Swift "#8" binary-trees program has even been shown in the half-a-dozen other "#8" programs at the top of the binary-trees page.

I wonder that anyone would notice the program, by themselves.

Would you please clearly document on the website, for each transliteration, what the original source language was? If that information is available, I could not find it. Thank you.

One of my colleagues did, and wanted to know why the numbers were so bad, which was why this whole thread was started.

2 Likes

Why? To what purpose?

My mistake: the “#8” programs are shown on other pages. For example, Swift versus C clang binary-trees

How will knowing the original source language of the slowest of those Swift binarytrees programs help make Swift seem more performant on the benchmarks game website?

That information has been available for months, in the header comment of these "#8" programs:

fannkuch-redux
fasta
mandelbrot
n-body
spectral-norm

(not binary-trees, not k-nucleotide)

For example, those linked at the top of the home page and those linked here

That seems to be a frivolous request.

If the information is worth having, it’s fair to think about the best way to present it. In this case, knowing that a benchmark is transliterated does seem like important context for interpreting the results. For example, if I saw that language X generally lagged significantly behind C except on a couple of benchmarks, seeing that those were transliterated from C and the others were not would be very useful to my analysis — it suggests that language X can reach C performance, but that it requires writing code in an apparently unidiomatic style. Now, collecting that information and reorganizing the website sounds like a lot of work, and whoever maintains it is entitled to spend their time as they please. But I don’t think it’s at all unreasonable to suggest that it would be a better summary if it included this information.

8 Likes

The benchmarks game compares program measurements. That does not include support for whatever highly speculative analyses others may wish to pursue.

It would include adding a new Swift fasta program: completely original or "inspired" by a commented C program.

As I explained above, it:

Without having that kind of information available for all the benchmarks, it feels like trying to “play a game” with unwritten rules.

2 Likes

My impression is that you're concerned that you'll do the work to write a program and then it may not be accepted.

That's correct.

So if you don't think writing a few tiny programs will be fun for you — Don't do it!

But not one "inspired" by this program.

No, that's not my concern. It's that other people:

  1. don't waste time trying to achieve something in the wrong way.
  2. can better understand how to interpret what they see on the site.

It's a very simple request. Of course whether to oblige is entirely your choice.

Firstly:

Same answer.

Secondly:

These measurements of program A are larger / smaller than these measurements of program B.

These measurements of this group of programs are larger / smaller than these measurements of this group of programs.

For more than that we need to look at the source code.

From what you've said, you seem not to have realized that information was shown for:

fannkuch-redux
fasta
mandelbrot
n-body
spectral-norm

:so it's probably not "very simple".