ccache and/or distcc for swift


(Oscar Bonilla) #1

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar


(Flamedoge) #2

Just curious, are there scenarios other than simple local build that you
wish to speed up? I don't know if ccache offers much more than incremental
build by existing build systems.
-Kevin

ยทยทยท

On Tue, Sep 27, 2016 at 11:09 AM, Oscar Bonilla via swift-dev < swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Mishal Shah) #3

Hi Oscar,

I am not aware any distributed build system which builds swift files however, swift build-script has support for distcc to build cpp files.

https://github.com/apple/swift/blob/master/utils/build-script#L1005

Thanks,
Mishal Shah

ยทยทยท

On Sep 27, 2016, at 11:09 AM, Oscar Bonilla via swift-dev <swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Daniel Dunbar) #4

Hi Oscar,

Nothing exists like that for Swift yet -- it is non-trivial for several reasons, the big two are:
1. The Swift compilation model relies on global information for a module, and needs to see all the sources. This process is currently embedded in the `swiftc` driver which would need to be aware of how to manage this process (or factored into a library API that a client able to manage the distribution could interact with).
2. The Swift compilation model also heavily leverages the ability to import Clang modules, which means that it *also* needs to be able to see large amounts of C headers, in modular model not a header preprocessing model. The distribution mechanism would also need to be aware of this.

- Daniel

ยทยทยท

On Sep 27, 2016, at 11:09 AM, Oscar Bonilla via swift-dev <swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Oscar Bonilla) #5

Primarily many builds from scratch. Specifically for a CI pipeline. That's
why I was also wondering about distcc, so the build objects could be cached
and shared among many machines.

ยทยทยท

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Tue, Sep 27, 2016 at 11:52 AM, Kevin Choi <code.kchoi@gmail.com> wrote:

Just curious, are there scenarios other than simple local build that you
wish to speed up? I don't know if ccache offers much more than incremental
build by existing build systems.
-Kevin

On Tue, Sep 27, 2016 at 11:09 AM, Oscar Bonilla via swift-dev < > swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Oscar Bonilla) #6

Thanks! It seems that is for building swift itself, not for building
projects written in swift ;(

ยทยทยท

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Tue, Sep 27, 2016 at 12:04 PM, mishal_shah <mishal_shah@apple.com> wrote:

Hi Oscar,

I am not aware any distributed build system which builds swift files
however, swift build-script has support for distcc to build cpp files.

https://github.com/apple/swift/blob/master/utils/build-script#L1005

Thanks,
Mishal Shah

On Sep 27, 2016, at 11:09 AM, Oscar Bonilla via swift-dev < > swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Oscar Bonilla) #7

Hi Daniel,

Thanks for the reply. It seems that maybe this should be a collaboration
between the swift compiler and the build system. I'm thinking something
along the lines of the swift compiler being able to print out a list of
file-level dependencies that the build system can use to decide whether to
compile files or reuse compiled objects?

If the swift compiler could produce as a side effect of the compilation a
list of files (with checksums) that it used, in theory it would be possible
to checksum those files on another compilation (in a different machine) and
reuse the object files right?

Does that seem reasonable/doable?

Thanks!

-Oscar

ยทยทยท

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Tue, Sep 27, 2016 at 12:47 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

Hi Oscar,

Nothing exists like that for Swift yet -- it is non-trivial for several
reasons, the big two are:
1. The Swift compilation model relies on global information for a module,
and needs to see all the sources. This process is currently embedded in the
`swiftc` driver which would need to be aware of how to manage this process
(or factored into a library API that a client able to manage the
distribution could interact with).
2. The Swift compilation model also heavily leverages the ability to
import Clang modules, which means that it *also* needs to be able to see
large amounts of C headers, in modular model not a header preprocessing
model. The distribution mechanism would also need to be aware of this.

- Daniel

On Sep 27, 2016, at 11:09 AM, Oscar Bonilla via swift-dev < > swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Jordan Rose) #8

I just wrote up a document describing why this is tricky. :slight_smile: Check it out: https://github.com/apple/swift/blob/master/docs/Driver.md

Jordan

ยทยทยท

On Sep 28, 2016, at 14:05, Oscar Bonilla via swift-dev <swift-dev@swift.org> wrote:

Hi Daniel,

Thanks for the reply. It seems that maybe this should be a collaboration between the swift compiler and the build system. I'm thinking something along the lines of the swift compiler being able to print out a list of file-level dependencies that the build system can use to decide whether to compile files or reuse compiled objects?

If the swift compiler could produce as a side effect of the compilation a list of files (with checksums) that it used, in theory it would be possible to checksum those files on another compilation (in a different machine) and reuse the object files right?

Does that seem reasonable/doable?

Thanks!

-Oscar

--
Oscar Bonilla
Staff Software Engineer
Tools Group

obonilla@linkedin.com <mailto:obonilla@linkedin.com>
linkedin.com/in/ <http://linkedin.com/in/>seeob

On Tue, Sep 27, 2016 at 12:47 PM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:
Hi Oscar,

Nothing exists like that for Swift yet -- it is non-trivial for several reasons, the big two are:
1. The Swift compilation model relies on global information for a module, and needs to see all the sources. This process is currently embedded in the `swiftc` driver which would need to be aware of how to manage this process (or factored into a library API that a client able to manage the distribution could interact with).
2. The Swift compilation model also heavily leverages the ability to import Clang modules, which means that it *also* needs to be able to see large amounts of C headers, in modular model not a header preprocessing model. The distribution mechanism would also need to be aware of this.

- Daniel

On Sep 27, 2016, at 11:09 AM, Oscar Bonilla via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Daniel Dunbar) #9

Ah, if you are specifically asking about the case of speeding up builds from scratch using cached results, that *is* somewhat doable, but it is still true that I don't know of an existing system for managing it for you.

I want llbuild + swiftc to grow in the direction of being able to solve that problem, but no one is actively working on it. If it is something you were interested in contributing to (bearing in mind it is a very large project) or just pick up and adopt? Also, is this something you were looking for just in the context of Swift, or in the Xcode + Swift context?

- Daniel

ยทยทยท

On Sep 28, 2016, at 2:00 PM, Oscar Bonilla via swift-dev <swift-dev@swift.org> wrote:

Primarily many builds from scratch. Specifically for a CI pipeline. That's why I was also wondering about distcc, so the build objects could be cached and shared among many machines.

--
Oscar Bonilla
Staff Software Engineer
Tools Group

obonilla@linkedin.com <mailto:obonilla@linkedin.com>
linkedin.com/in/ <http://linkedin.com/in/>seeob

On Tue, Sep 27, 2016 at 11:52 AM, Kevin Choi <code.kchoi@gmail.com <mailto:code.kchoi@gmail.com>> wrote:
Just curious, are there scenarios other than simple local build that you wish to speed up? I don't know if ccache offers much more than incremental build by existing build systems.
-Kevin

On Tue, Sep 27, 2016 at 11:09 AM, Oscar Bonilla via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Oscar Bonilla) #10

Oh, thanks for that link. Very useful... I'm still digesting it to
understand the implications...

ยทยทยท

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Wed, Sep 28, 2016 at 2:10 PM, Jordan Rose <jordan_rose@apple.com> wrote:

I just wrote up a document describing why this is tricky. :slight_smile: Check it
out: https://github.com/apple/swift/blob/master/docs/Driver.md

Jordan

On Sep 28, 2016, at 14:05, Oscar Bonilla via swift-dev < > swift-dev@swift.org> wrote:

Hi Daniel,

Thanks for the reply. It seems that maybe this should be a collaboration
between the swift compiler and the build system. I'm thinking something
along the lines of the swift compiler being able to print out a list of
file-level dependencies that the build system can use to decide whether to
compile files or reuse compiled objects?

If the swift compiler could produce as a side effect of the compilation a
list of files (with checksums) that it used, in theory it would be possible
to checksum those files on another compilation (in a different machine) and
reuse the object files right?

Does that seem reasonable/doable?

Thanks!

-Oscar

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Tue, Sep 27, 2016 at 12:47 PM, Daniel Dunbar <daniel_dunbar@apple.com> > wrote:

Hi Oscar,

Nothing exists like that for Swift yet -- it is non-trivial for several
reasons, the big two are:
1. The Swift compilation model relies on global information for a module,
and needs to see all the sources. This process is currently embedded in the
`swiftc` driver which would need to be aware of how to manage this process
(or factored into a library API that a client able to manage the
distribution could interact with).
2. The Swift compilation model also heavily leverages the ability to
import Clang modules, which means that it *also* needs to be able to see
large amounts of C headers, in modular model not a header preprocessing
model. The distribution mechanism would also need to be aware of this.

- Daniel

On Sep 27, 2016, at 11:09 AM, Oscar Bonilla via swift-dev < >> swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Oscar Bonilla) #11

Yeah, speeding up builds from scratch is the case I'm trying to solve.

I'm willing to dedicate time to solve the problem but I'm unfamiliar with
the swift source base so I would need a lot of guidance :wink:

I am looking at Xcode + Swift. I'm currently using xcodebuild from the
command line for the builds. However, I would not mind moving away from
xcodebuild and using something else. I just figured the swift compiler
would need to help or I would just be chasing my own tail trying to deal
with dependencies from an external build system.

Thanks!

-Oscar

ยทยทยท

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Wed, Sep 28, 2016 at 2:16 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

Ah, if you are specifically asking about the case of speeding up builds
from scratch using cached results, that *is* somewhat doable, but it is
still true that I don't know of an existing system for managing it for you.

I want llbuild + swiftc to grow in the direction of being able to solve
that problem, but no one is actively working on it. If it is something you
were interested in contributing to (bearing in mind it is a very large
project) or just pick up and adopt? Also, is this something you were
looking for just in the context of Swift, or in the Xcode + Swift context?

- Daniel

On Sep 28, 2016, at 2:00 PM, Oscar Bonilla via swift-dev < > swift-dev@swift.org> wrote:

Primarily many builds from scratch. Specifically for a CI pipeline. That's
why I was also wondering about distcc, so the build objects could be cached
and shared among many machines.

--
*Oscar Bonilla*
Staff Software Engineer
Tools Group

obonilla@linkedin.com
linkedin.com/in/seeob

On Tue, Sep 27, 2016 at 11:52 AM, Kevin Choi <code.kchoi@gmail.com> wrote:

Just curious, are there scenarios other than simple local build that you
wish to speed up? I don't know if ccache offers much more than incremental
build by existing build systems.
-Kevin

On Tue, Sep 27, 2016 at 11:09 AM, Oscar Bonilla via swift-dev < >> swift-dev@swift.org> wrote:

Hello swift developers,

I was wondering if any of you knows anything about something
like ccache and/or distcc for swift.

Basically, what I want is to speed up compiles by caching the result
(like ccache does) and then reusing the compilation results across
multiple machines.

Does anything like that exist for swift? I looked at ccache but they
don't support swift and I couldn't find anything on distcc either.

Thanks!

-Oscar

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev