I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example: https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?
I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example: https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?
I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?
Michael
···
On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example: https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?
I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?
This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.
Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores
Thanks,
Mishal Shah
···
On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:
On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
It still seems like, for a smoke test, we're doing way too much work. This appears to be much more than what I get from build-script -t when I run tests locally. Maybe I'm misunderstanding the intended role of our smoke tests, but since nobody is correcting me, I'm betting not.
Anyway, regardless of the explanation, what can be done about this? Between a spurious failure in LLDB and the length of the smoke test it took several hours to be able to merge something that could not possibly break the build, which seems absurd.
···
Sent from my iPad
On Oct 18, 2016, at 11:44 PM, mishal_shah <mishal_shah@apple.com> wrote:
On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:
On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example: https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?
I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?
This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.
Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores
It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.
Someone wrote to me privately:
"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".
This seems wrong to me. Can we fix it?
···
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org> wrote:
Anyway, regardless of the explanation, what can be done about this?
Between a spurious failure in LLDB and the length of the smoke test it
took several hours to be able to merge something that could not
possibly break the build, which seems absurd.
On Oct 18, 2016, at 11:44 PM, mishal_shah <mishal_shah@apple.com> wrote:
On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:
On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example: https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?
I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?
This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.
Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores
It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.
Even smoke tests should run the validation tests…
Someone wrote to me privately:
"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".
This seems wrong to me. Can we fix it?
…but I could see "long tests" going either way.
···
On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:
It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.
Someone wrote to me privately:
"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".
This seems wrong to me. Can we fix it?
Anyway, regardless of the explanation, what can be done about this?
Between a spurious failure in LLDB and the length of the smoke test it
took several hours to be able to merge something that could not
possibly break the build, which seems absurd.
On Oct 18, 2016, at 11:44 PM, mishal_shah <mishal_shah@apple.com> wrote:
On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:
On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example: https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?
I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?
This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.
Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores
It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.
Even smoke tests should run the validation tests…
What's the point of distinguishing validation from other tests if even
the smoke tests run them?
Someone wrote to me privately:
"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".
This seems wrong to me. Can we fix it?
…but I could see "long tests" going either way.
Well, what is “smoke” supposed to mean? If it requires running
validation tests and maybe even long tests, what is the “smoke test”
distinction *for*?
···
on Wed Oct 19 2016, Jordan Rose <swift-dev-AT-swift.org> wrote:
On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:
It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.
Even smoke tests should run the validation tests…
What's the point of distinguishing validation from other tests if even
the smoke tests run them?
I think the validation tests are tests that compiler developers don’t bother running locally before they move to a PR. I’d be very concerned about landing changes in master without having run the validation tests—that would get us back to the days of consistent failures because someone forgot to update compiler_crashers.
Someone wrote to me privately:
"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".
This seems wrong to me. Can we fix it?
…but I could see "long tests" going either way.
Well, what is “smoke” supposed to mean? If it requires running
validation tests and maybe even long tests, what is the “smoke test”
distinction *for*?
It only builds and runs one platform per builder (no iOS/watchOS/tvOS), and doesn’t necessarily build and test all the downstream projects.
I agree that if we are to have something called “smoke test”, 40 minutes is too long.
Jordan
···
On Oct 19, 2016, at 10:17, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Jordan Rose <swift-dev-AT-swift.org> wrote:
On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:
It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.
Even smoke tests should run the validation tests…
What's the point of distinguishing validation from other tests if even
the smoke tests run them?
I think the validation tests are tests that compiler developers don’t bother running locally before they move to a PR. I’d be very concerned about landing changes in master without having run the validation tests—that would get us back to the days of consistent failures because someone forgot to update compiler_crashers.
Someone wrote to me privately:
"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".
This seems wrong to me. Can we fix it?
…but I could see "long tests" going either way.
Well, what is “smoke” supposed to mean? If it requires running
validation tests and maybe even long tests, what is the “smoke test”
distinction *for*?
It only builds and runs one platform per builder (no iOS/watchOS/tvOS), and doesn’t necessarily build and test all the downstream projects.
I agree that if we are to have something called “smoke test”, 40 minutes is too long.