Suitable form for discussing and optimizing small programs to help improving compiler/optimizer/stdlib?

Background (can be skipped)

While trying to optimize performance critical parts of a project we’re working on, I sometimes stumble into what turns out to be opportunities for improvement in the Swift compiler and/or the Standard Library (even though I’m sure my choice/design of algorithms and data structures could also be improved).

Isolating every potential issue and formulating a good Jira bug/improvement can be very time comsuming for non-native-english-speaking non-compiler-hackers like myself.

I’d find it much easier to create a forum post containing a minified toy-version of my original problem, eg a sufficiently commented self-contained little command line program with detailed compile instructions, etc.

Other forum members might then find it interesting to help identifying and formulating any potential compiler/stdlib bug/improvement, or they might point out that I’m simply misunderstanding something and the problem is only on my side, which means I (and perhaps others) have learned something new.


Proposal / Question

While trying to optimize Swift code as a single individual or a small team might easily lead to frustration and disappointment in the current state of Swift, maybe resulting in a decision to go back to C/C++ for performance critical code, I think that doing it in a community could instead lead to more people contributing to Swift in various important ways, for example:

  • Identifying (and ideally reporting) compiler/optimizer/stdlib bugs or improvements.

  • Providing ideas for (and ideally submitting) new performance tests for the compiler.

  • Spreading knowledge and best practices in the context of performance critical code, including information about what is and isn’t “Swift’s fault”.

Does anyone else see value in a group activity like this?
In what form?


My Thoughts on This

I’d say it’s not always easy to choose an existing category for this kind of posts, eg: Should I pick “Using Swift” or maybe “Development - Standard Library” since I suspect my code example might lead to the identification of an opportunity for improvement in the stdlib implementation, or maybe “Development - Compiler” since it might end up indicating an optimizer bug?

And I guess the effectiveness of this sort of group-activity depends heavily on the way it is organized.

For example, maybe there could be signal/noise enhancing rules like:

  • Each original post must contain a working single-file command line program.
  • The purpose and content of the program should be quickly understood, and it should be clear in what way the OP thinks it could be optimized.
  • The OP should provide an example of compiling and running the program from the command line, showing the version of the compiler and the flags used.

Other forum participants can then always expect to be able to just read, copy, paste and compile it. Best practices for measuring execution times etc could evolve or be decided, etc.

If enough interest is expressed, perhaps it could motivate a new category?

I feel that just a tag (and perhaps a set of rules that could be linked to somehow) would force us to choose one of the other categories and thereby polluting them with this kind of posts.

But maybe forums.swift.org is not the right place, or maybe there are reasons why the whole thing is not a good idea at all?

2 Likes

/cc @Ben_Cohen, @tkremenek, @Nicole_Jacque

Also @Erik_Eckstein.

OK, I guess I’m not gaining much traction here. : )

I’ll make a another attempt.

The following is a typical example of the sort of productivity hampering issue I’d like a more efficient way to deal with:

I reported it 30 Oct 2017, and it’s still one of many contributing sources of unproductivity. That is, taken together, issues like this make writing performance critical code in Swift an almost impossibly unpredictable and mysterious experience. They shift my focus from what I’m actually trying to achieve (writing an app) to that of investigating and trying to help fixing/improving the tools. Note that this bug report could probably be broken up into multiple more condensed and better formulated ones. But I lack the skills needed to do that efficiently.

Below are a couple of more recent similar issues. But for these, I have, despite my lack of the required skills to do it efficiently, tried even harder and spent even more time to try and isolate them and formulate better reports:

Note that I’m not complaining about the current situation, I’m impressed by everyone’s hard work on Swift and the speed at which the language is moving forward.

I’m just trying to start a discussion about whether there might be even better ways to channel my (and possibly others’) time, and sometimes frustration, into valuable contributions to the compiler/optimizer.

I’d rather not go back to using C/C++, because I’d miss the good parts of Swift so much, but I wish I could just post a small program (to a hypothetical category in this forum) that demonstrates some interesting possible performance/predictabilty issue(s). And then a group of community members more skilled than me would be there, digging into the threads of that category, isolating individual issues, formulating terse and effective bug reports, managing and keeping track of their progress, getting ideas for new performance tests based on them, and so on.

Perhaps a forum thread (in this hypothetical new category) could be seen as a kind of “higher level bug report”, a non-compiler-hacker-user-provided demo-program spawning a series of actual high quality bug reports, and for example new performance tests or ideas for improvements in the stdlib or whatever.

Perhaps I’m not the only one who would feel more comfortable and effective contributing at such a level, than through individual Jira reports?

As previously mentioned, I also think something like this could be a good way to get people into learning more about and working on the compiler.

I’m in the same situation. I was toying with elegant Mandelbrot Set generator from Haskell, when I stumbled on massive performance bottlenecks. That was a year ago. I have reduced the issue to inconsistent optimizations when composing sequence operations. I’m in the process of writing benchmarks to help pinpoint these performance traps, but I took a detour through improving measurement process for Swift Benchmark Suite.

With all that said, I think Jens is onto something with the suggestion for a category for discussing possible performance problems. What about sticking to the #swift-users with a performance tag? We could spin off a topic about “How to diagnose performance issues” and “How to reduce performance bugs” documenting these processes that can allow community to ease the burden for core developers…

FWIW we had a few in-person discussions on this topic, but apparently none of them surfaced here.

First of all:

Wouldn’t it make a great starting point to a hypothetical “How to reduce performance bugs” article!

There is an amazing document on how to measure compiler performance. Substitute “compiler” to “your program” there and a big part of the document still applies.

As for the discussions and the overall suggestion. I could not agree more that the performance problems should be reported and looked at. It is a matter of making these reports useful and used.

I can categorize reports into 3 categories:

  1. Potential performance problem that can also be just a misunderstanding of the complexity guarantees or compiler inter-workings.
  2. Confirmed performance problems that are however not reduced to a minimal case.
  3. Reduced cases where the performance problems manifests itself.

The most obvious choice for the 3rd category of problems would be to just add them as PRs to the Swift benchmark suite. It is believed (by me) to not be hard to integrate benchmarks into that suite. We are very interested in more benchmarks that cover real-world scenarios, therefore such PRs don’t have to be perfect, just something that shows the problem. We can improve/integrate that separately.

Next goes the non-reduced examples. I think these should be filed as bugs since they are confirmed. I would not mind at all if the new topic is started on swift-users with a special tag with a “call for reduction”. This would cover category number 2.

Category number 1 in my opinion is something that can be discussed on the forum, as it can be beneficial to many people. I would expect at least some of these problems to be due to misunderstanding, and as such, the forum post would become a kind of documentation.

Personally, I don’t think we really need a separate category, the tag would do.

Does it make sense? If so, we can ask @Nicole_Jacque to create this tag.

Thank you for your detailed reply!

Just a quick question: Which of the 3 categories would the following two issues (from my above list) be?

I’m not at all sure, but I’d guess they’d both be 3 (Reduced case where the performance problem manifests itself), and if so I should add them as PRs to the Swift Benchmark Suite.

Looking at the way those benchmarks are constructed (test is specified in a func, which seems to then be stored in a property, and later called from within a specific context), I fail to see how these two particular performance problems could/should be turned into such benchmarks (since they are about unexpected differences in performance depending on context, things like whether some code is wrapped within a nested func or not, or if a closure has been passed as an argument once or twice before it is heavily used etc.). A similar example would be if an issue manifested itself only in global scope, not otherwise, I can’t see how such an issue could be added as a benchmark in the current design of the Benchmark Suite.

In other words: A lot of the issues I’m seeing manifests themselves only in very specific contexts, and it seems like the Swift Benchmark Suite imposes a certain context on its benchmarks, ie it is not possible to reproduce any context in those benchmarks.

I’m probably just missing something obvious. It would certainly be obvious if I could just see SR-7023 and SR-6983 turned into benchmarks for the Swift Benchmark Suite. So if someone would feel like having a go at that, I’d really appreciate it!

Ahh… Did not “do my homework” again before replying. Sorry. I believe I have to mark these as non-benchmark-able special cases. There are other ways to test whether some optimizations happen (or not), but I would not expect anyone who reports the issue to be able/willing to do that.

Therefore even though these issues are technically category 3, I’d say a JIRA is fine for those.

Unless @Erik_Eckstein or @Arnold or @Michael_Gottesman disagree.

1 Like

I have no opinion on which forum group to use, since I don’t pay much attention to the categories. A performance tag that I could signup to “watch” might be helpful though.

I’m working on a reponse to the first bug report, SR-6254.
https://bugs.swift.org/browse/SR-6254

That bug report already has more than enough information. I was able to reproduce it right away. Spending more time on the bug report/benchmark probably won’t get it fixed any quicker, but it may give you more insight to work around the problem quicker next time.

I mainly want to assure you that filing bugs like this is extremely helpful, even if it seems like no one is looking at them. Posting to the forum also helps draw attention. If it’s a problem that hasn’t been solved in months and is still affecting you, it helps to update the bug or post to the forum again.

I know that reducing a performance problem to a standalone benchmark is aggravating when you don’t know what is causing fluctuations. But if you take the time to do that (as you did in SR-6264) that’s very helpful to someone debugging the problem. Whoever actually does the debugging will be able further reduce it to a regression benchmark and add it to the benchmark suite. The reporter does not need to add a benchmark.

2 Likes

Thanks again for your very informative response to SR-6254!

I will continue writing (most of) our performance sensitive code in Swift, even though it probably means I’ll also continue spending half the time on investigating and reporting issues.

When posting demonstrations of potential issues to the forum I will use a “performance” tag and category “Using Swift”.

3 Likes

Seeing such detailed perf bug reports from you and active conversation in there from e.g. @Andrew_Trick makes me so happy. Thank you for raising them, even if they’ll take a while to be addressed.

2 Likes

Agreed. But, @Andrew_Trick, can you teach us how to fish?

I know how to profile with Instruments, which shows me the assembly of the hot loops. How exactly does one get to the SIL you . I know there are some parameters to the compiler…

For anyone interested in such teaching, here’s an opportunity to guide us through a small practical example!

(I roughly know how to use Instruments (I think), using High Frequency recording in Timed profiling and/or Counters, but more often than not, I end up getting jumbled/confusing results, pointing to the wrong lines in the code even though I use -g, and I know there are swiftc -emit-sil and -emit-assembly etc, but I’m unable to make much sense of the output. I used to do some x86 asm programming as a kid and while studying computer science.)

You can see what I did with this bug:
https://bugs.swift.org/browse/SR-7150

When integer protocols are involved, even an innocent looking loop can be a massive expedition.

There’s also a good debugging the compiler document, written by others, here:

2 Likes

Wow that was quick, thanks!