smoke test not very smoky?


(Dave Abrahams) #1

I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example:
https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?

···

--
-Dave


(Michael Gottesman) #2

I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example:
https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?

I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?

Michael

···

On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

--
-Dave
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Mishal Shah) #3

I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example:
https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?

I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?

This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.

Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores

Thanks,
Mishal Shah

···

On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:

On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

Michael

--
-Dave
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Dave Abrahams) #4

It still seems like, for a smoke test, we're doing way too much work. This appears to be much more than what I get from build-script -t when I run tests locally. Maybe I'm misunderstanding the intended role of our smoke tests, but since nobody is correcting me, I'm betting not.

Anyway, regardless of the explanation, what can be done about this? Between a spurious failure in LLDB and the length of the smoke test it took several hours to be able to merge something that could not possibly break the build, which seems absurd.

···

Sent from my iPad

On Oct 18, 2016, at 11:44 PM, mishal_shah <mishal_shah@apple.com> wrote:

On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:

On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example:
https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?

I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?

This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.

Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores

Thanks,
Mishal Shah

Michael

--
-Dave
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Dave Abrahams) #5

It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.

Someone wrote to me privately:

  "buildbot_linux_1404" preset used in Linux smoke test contains
  "--long-test".

This seems wrong to me. Can we fix it?

···

on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org> wrote:

Anyway, regardless of the explanation, what can be done about this?
Between a spurious failure in LLDB and the length of the smoke test it
took several hours to be able to merge something that could not
possibly break the build, which seems absurd.

On Oct 18, 2016, at 11:44 PM, mishal_shah <mishal_shah@apple.com> wrote:

On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:

On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example:
https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?

I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?

This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.

Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores

Thanks,
Mishal Shah

Michael

--
-Dave
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
-Dave


(Jordan Rose) #6

It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.

Even smoke tests should run the validation tests…

Someone wrote to me privately:

"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".

This seems wrong to me. Can we fix it?

…but I could see "long tests" going either way.

···

On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:


(Mishal Shah) #7

Yes, I will fix it today by creating new preset for smoke test.

Mishal Shah

···

On Oct 19, 2016, at 9:44 AM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:

It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.

Someone wrote to me privately:

"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".

This seems wrong to me. Can we fix it?

Anyway, regardless of the explanation, what can be done about this?
Between a spurious failure in LLDB and the length of the smoke test it
took several hours to be able to merge something that could not
possibly break the build, which seems absurd.

On Oct 18, 2016, at 11:44 PM, mishal_shah <mishal_shah@apple.com> wrote:

On Oct 18, 2016, at 8:29 PM, Michael Gottesman <mgottesman@apple.com> wrote:

On Oct 18, 2016, at 6:40 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

I thought the smoke test was supposed to be a fairly quick test
that just covered the basics, but it seems to be doing plenty of *really
long* tests, for example:
https://ci.swift.org/job/swift-PR-Linux-smoke-test/1916/console
which has already been running for over 30 minutes. Am I missing
something?

I think people have over time been adding more to the Linux tests. But I am not in the know. Mishal, do you know whats happening here?

This is most likely due to number cores we have for Linux bots, the new Linux bots are 12 cores vs 48 cores and we are running multiple executors on it.

Old - 48 cores / 4 executors = 12 cores
New - 12 cores / 2 executors = 6 cores

Thanks,
Mishal Shah

Michael

--
-Dave
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

--
-Dave

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


(Dave Abrahams) #8

It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.

Even smoke tests should run the validation tests…

What's the point of distinguishing validation from other tests if even
the smoke tests run them?

Someone wrote to me privately:

"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".

This seems wrong to me. Can we fix it?

…but I could see "long tests" going either way.

Well, what is “smoke” supposed to mean? If it requires running
validation tests and maybe even long tests, what is the “smoke test”
distinction *for*?

···

on Wed Oct 19 2016, Jordan Rose <swift-dev-AT-swift.org> wrote:

On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:

--
-Dave


(Jordan Rose) #9

It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.

Even smoke tests should run the validation tests…

What's the point of distinguishing validation from other tests if even
the smoke tests run them?

I think the validation tests are tests that compiler developers don’t bother running locally before they move to a PR. I’d be very concerned about landing changes in master without having run the validation tests—that would get us back to the days of consistent failures because someone forgot to update compiler_crashers.

Someone wrote to me privately:

"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".

This seems wrong to me. Can we fix it?

…but I could see "long tests" going either way.

Well, what is “smoke” supposed to mean? If it requires running
validation tests and maybe even long tests, what is the “smoke test”
distinction *for*?

It only builds and runs one platform per builder (no iOS/watchOS/tvOS), and doesn’t necessarily build and test all the downstream projects.

I agree that if we are to have something called “smoke test”, 40 minutes is too long.

Jordan

···

On Oct 19, 2016, at 10:17, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Jordan Rose <swift-dev-AT-swift.org> wrote:

On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:
on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:


(Mishal Shah) #10

Linux smoke test builds with new preset (buildbot_linux,smoketest) and it takes 39mins compare to 1hr 4min.

https://github.com/apple/swift/pull/5378

Thanks,
Mishal Shah

···

On Oct 20, 2016, at 9:33 AM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

On Oct 19, 2016, at 10:17, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

on Wed Oct 19 2016, Jordan Rose <swift-dev-AT-swift.org> wrote:

On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:

It still seems like, for a smoke test, we're doing way too much work.
This appears to be much more than what I get from build-script -t when
I run tests locally. Maybe I'm misunderstanding the intended role of
our smoke tests, but since nobody is correcting me, I'm betting not.

Even smoke tests should run the validation tests…

What's the point of distinguishing validation from other tests if even
the smoke tests run them?

I think the validation tests are tests that compiler developers don’t bother running locally before they move to a PR. I’d be very concerned about landing changes in master without having run the validation tests—that would get us back to the days of consistent failures because someone forgot to update compiler_crashers.

Someone wrote to me privately:

"buildbot_linux_1404" preset used in Linux smoke test contains
"--long-test".

This seems wrong to me. Can we fix it?

…but I could see "long tests" going either way.

Well, what is “smoke” supposed to mean? If it requires running
validation tests and maybe even long tests, what is the “smoke test”
distinction *for*?

It only builds and runs one platform per builder (no iOS/watchOS/tvOS), and doesn’t necessarily build and test all the downstream projects.

I agree that if we are to have something called “smoke test”, 40 minutes is too long.

Jordan

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev