The Benchmark Game has moved from Swift 4.2.1 to the 2019-02-06 build of 5.0-dev. The RegexRedux benchmark's already abysmal performance of around 74 seconds, has now increased to 115 seconds. Is this on anyone's radar?
Should this be in Core Libraries or Standard Library? The benchmark certainly looks like it's primarily exercising
NSRegularExpression, but if there was a regression I would expect it to be related to
String and its bridging, which are in the stdlib.
There’s a lot wrong with this benchmark, little of which has to do with
String. First and foremost, it’s constantly recompiling the same regex. That should be cached. Secondly, and somewhat bizarrely, it dispatches onto a
background queue to perform its work, which may have unpredictable overhead and system contention issues. Finally, there may still be optimization issues with global level code (though hopefully those have been fixed by now).
So there will likely need to be some substantial rework and analysis before being able to say this has anything to do with
My poor programming skills aside, the execution time of this program has gone from 74 seconds on 4.2.1 to 115 seconds on the 2019-02-06 development build of 5.0. While everything you say is valid, I'm more concerned with finding the source of the loss in performance between language versions.
I tested a regex-redux program (by macOS app with local build SwiftFoundation).
In the result, 50% of the cpu time spent by
getCharacters(_:range:) implementation calls
character(at:) for many times...
I'm getting roughly identical runtimes for
regex-redux compiled with Swift 4.2.1 (Xcode 10.1) and 5 (from Xcode 10.2 beta-2) on mac os 10.14.3. So I think somebody with access to Linux machine needs to duplicate the regression first…
FWIW, here's trace from Instruments (Main Thread)
|64.5%||closure #3 in|
|56.9%||icu::RegexMatcher::MatchAt(long long, signed char, UErrorCode&)|
...and it finishes in 96 seconds on the 2008 MBP (Core 2 Duo), which is nothing to brag about in the context of the Benchmarks Game, just pointing out how poor the Foundation on Linux performs in comparison. Looks like it doesn't delegate the regex processing to ICU?
Is the source of this available anywhere?
It's available at the first link in the original post.
I put up a (as-yet completely untested) draft PR that should go a long way towards addressing this https://github.com/apple/swift-corelibs-foundation/pull/1928
Another alternative we could consider is making StringStorage's NSString methods available on non-ObjC platforms, in which case NSString could just call directly into them.
I was seeing a similar regression for
Testing was being done on macOS 10.13, and I didn't retry it on 10.14
In general, while the need for bridging is much lower on Linux, there are situations in which we bridge and Linux bridging is done eagerly.
Thinking about it more, this is tricky: StringStorage isn't exposed publicly at all, it's only useful for ObjC bridging because we can rely on objc_msgSend just ignoring that. So we'd need to expose some underscored stuff for it to call…
String.UTF8View issues are unlikely to be related to the regexredux regression, since there's no NSString stuff involved there. I have a few guesses about what might be going on there, but Michael's the right person and it looks like he's already involved, so I'll leave it be.
A better version of this patch was merged today (thanks @millenomi !); I haven't tried it with the benchmark game test, but a getCharacters()-specific test suggests it should reduce the time spent in that function by around an order of magnitude.
(I should note in case there's anyone interested in tinkering with it, there's still a good deal of low hanging fruit in that function: ASCII fast paths, providing an override of it in _NSCFConstantString, and so on. I'll try to circle back and pick some of that up eventually, but my expectation is that most people on Linux are going to be using Swift String instead of NSString, and Darwin NSString has a different implementation, so it's lower on my list than some other speedups.)
I just tried the 02-26-2019 trunk build and can report a marked improvement in performance (on my Linux machine). RegexRedux took 45 seconds to complete on 4.2.1 and 64 seconds on 5.0-dev. Now it takes 31 seconds! Thanks @David_Smith!
The great thing about performance regressions is sometimes they expose already-inefficient code that had been slipping by unnoticed
Is there any plan this patch be merged into 5.0-branch?
This patch improves various
let url = URL(string: "http://swift.org")
I asked, and it sounds like we're still sorting out some stuff about what gets merged into 5.0 and when, but this will be a reasonable candidate for it once we do
I really hope this patch gets in. That said, I'm hoping for more patches in the future: I want to move my text processing routines off of Java and RegexRedux, using the same approach, takes 7 seconds on my Linux machine.