I was working over the Easter holiday weekend on script that reports the benchmark results comparison. I wasn’t always up-to-date with the master, but I have noticed a mysterious change in MAX_RSS value reported from benchmarks. The change went away again in later builds.
For example the AngryPhonebook benchmark usually reports MAX_RSS of 10 543 104 - so around 10 MB. But my local builds from 2017-04-15 had all similar tests drop to around 3 MB (AngryPhonebook reported 3 506 176).
That seems like a significant and positive change, but later builds returned to previous values. I have tried to look through the commits around that time, but nothing there strikes me as obvious cause.
Forgive my cluelessness, I lack proper visibility into the projects current development, but is there some work going on in this area, that got merged and reverted?
I was working over the Easter holiday weekend on script that reports the benchmark results comparison. I wasn’t always up-to-date with the master, but I have noticed a mysterious change in MAX_RSS value reported from benchmarks. The change went away again in later builds.
For example the AngryPhonebook benchmark usually reports MAX_RSS of 10 543 104 - so around 10 MB. But my local builds from 2017-04-15 had all similar tests drop to around 3 MB (AngryPhonebook reported 3 506 176).
That seems like a significant and positive change, but later builds returned to previous values. I have tried to look through the commits around that time, but nothing there strikes me as obvious cause.
Forgive my cluelessness, I lack proper visibility into the projects current development, but is there some work going on in this area, that got merged and reverted?
I can't think of anything. But what I can say is that this makes me want to track MAX_RSS over time ; ).
Michael
···
On Apr 21, 2017, at 3:57 AM, Pavol Vaskovic via swift-dev <swift-dev@swift.org> wrote:
The value of MAX_RSS depends on OS behavior. Other activity on the same machine may change MAX_RSS of the benchmark. The changes you saw might not be "real".
If you re-run the 2017-04-15 builds today, do you still see 3 MB instead of 10 MB?
···
On Apr 21, 2017, at 3:57 AM, Pavol Vaskovic via swift-dev <swift-dev@swift.org> wrote:
Hi swift-dev!
I was working over the Easter holiday weekend on script that reports the benchmark results comparison. I wasn’t always up-to-date with the master, but I have noticed a mysterious change in MAX_RSS value reported from benchmarks. The change went away again in later builds.
For example the AngryPhonebook benchmark usually reports MAX_RSS of 10 543 104 - so around 10 MB. But my local builds from 2017-04-15 had all similar tests drop to around 3 MB (AngryPhonebook reported 3 506 176).
That seems like a significant and positive change, but later builds returned to previous values. I have tried to look through the commits around that time, but nothing there strikes me as obvious cause.
Forgive my cluelessness, I lack proper visibility into the projects current development, but is there some work going on in this area, that got merged and reverted?
The value of MAX_RSS depends on OS behavior. Other activity on the same machine may change MAX_RSS of the benchmark.
Can you please describe the mechanism of how “other activity on the same machine may change MAX_RSS of the benchmark”.?
Not all memory used by the benchmark counts against its RSS. For example, paged-out and VM-compressed memory are both excluded. If there are other processes contending for memory while the benchmark runs then the benchmark's RSS will be artificially reduced.
Did you see any time difference between the 3MB and the 10MB runs?
The changes you saw might not be "real”.
How was that not real? I have logs that prove that. See the attachment in original post.
The possibility is that the benchmark's RSS did in fact change, but the benchmark's "real" memory usage did not.
If you re-run the 2017-04-15 builds today, do you still see 3 MB instead of 10 MB?
Nope. If I saw it now, I wouldn’t be searching for that, but congratulating the brave committer that gave us this improvement.
That tends to suggest a difference in the test environment rather than a change in Swift. If some Swift changes were responsible for the RSS decrease and subsequent increase then re-running the benchmark with that version of Swift ought to exhibit the same RSS behavior.
Measuring memory usage is good. RSS is a difficult value to use for such measurements.
···
On Apr 24, 2017, at 8:52 PM, Pavol Vaskovic <pali@pali.sk> wrote:
On 25 Apr 2017, at 01:28, Greg Parker <gparker@apple.com> wrote:
Thanks for the explanation and sorry for the report. I’m putting the attachment to dropbox, to work around the mailing list size cap.
>
> >
> > The value of MAX_RSS depends on OS behavior. Other activity on the same machine may change MAX_RSS of the benchmark.
> Can you please describe the mechanism of how “other activity on the same machine may change MAX_RSS of the benchmark”.?
Not all memory used by the benchmark counts against its RSS. For example, paged-out and VM-compressed memory are both excluded. If there are other processes contending for memory while the benchmark runs then the benchmark's RSS will be artificially reduced.
Did you see any time difference between the 3MB and the 10MB runs?
Yes. A negative inverse. The runs with 10 MB were mostly faster. 3MB were mostly slower, but it differed between various benchmark families - some were comparable speed.
> > The changes you saw might not be "real”.
> How was that not real? I have logs that prove that. See the attachment in original post.
The possibility is that the benchmark's RSS did in fact change, but the benchmark's "real" memory usage did not.
OK, I understand. Would you say its possible I did one of those measurements under extreme memory pressure on my machine and that’s what caused this? The so the Swift runtime gets compacted under low memory conditions with memory compression from 10 to 3 MB? And correspondingly tests that need to access compressed parts are slowed down by decompression and some lucky tests evade this penalty.
> > If you re-run the 2017-04-15 builds today, do you still see 3 MB instead of 10 MB?
> Nope. If I saw it now, I wouldn’t be searching for that, but congratulating the brave committer that gave us this improvement.
That tends to suggest a difference in the test environment rather than a change in Swift. If some Swift changes were responsible for the RSS decrease and subsequent increase then re-running the benchmark with that version of Swift ought to exhibit the same RSS behavior.
Measuring memory usage is good. RSS is a difficult value to use for such measurements.Please have a look at the Numbers file at https://db.tt/CnhRTCrqED\. The 2 tabs of interest are Release-3s-1 from 20170414 (ie. before) and Release-3s from 20170415 (the change). Later measurements returned to the before value. Last 4 columns are the differences between the max RSS and median time, with percentages (color coded).
Best regards
Pavol Vaskovic
···
On 25 Apr 2017, at 08:36, Greg Parker <gparker@apple.com (mailto:gparker@apple.com)> wrote:
> On Apr 24, 2017, at 8:52 PM, Pavol Vaskovic <pali@pali.sk (mailto:pali@pali.sk)> wrote:
> > On 25 Apr 2017, at 01:28, Greg Parker <gparker@apple.com (mailto:gparker@apple.com)> wrote: