Swift incremental compile profiling

I have a large project (308 swift files, 441 objective c, 66k lines of
code) where incremental builds can be extremely slow. I'm trying to do some
profiling to figure out what type of things cause large scale recompiles.
The problem is that I can't find a good way of telling which files get
recompiled on an incremental build and which do not. It seems like files
that are not recompiled still get listed in xcode, but the compiler just
passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality? Or
is there some other way to get this info?

Thank you,
Sam

Hi, Sam. I don't think we currently have a good answer for this built into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would be fast enough that it wouldn't matter! That's obviously not where we are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share another one of our debugging flags, '-driver-show-incremental'. You can add this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered later

In this case, I took a version of the Adventure sample project and modified "Tree.swift"; that triggered recompilation of several other files. Unfortunately this view doesn't tell you how they're related, only which ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift" rather than just "trying to figure out why it's repeating work") would be to look at the "swiftdeps" files stored in your DerivedData folder. These are currently just YAML files describing what Swift thinks the file depends on, as well as what will trigger rebuilding of other files. This is intended to be a conservative estimate, since not recompiling something would result in an invalid binary. (Unfortunately I say "intended" because there are known bugs; fortunately, archive builds are always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's dependency analysis: https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst. The one thing that's not in there is the notion of changes that don't affect other files at all. This is accomplished by computing a hash of all the tokens that could affect other files, and seeing if that hash has changed.

We definitely have room for improvement here.

Jordan

···

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <swift-dev@swift.org> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of code) where incremental builds can be extremely slow. I'm trying to do some profiling to figure out what type of things cause large scale recompiles. The problem is that I can't find a good way of telling which files get recompiled on an incremental build and which do not. It seems like files that are not recompiled still get listed in xcode, but the compiler just passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality? Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile
flag that when turned on would require you to explicitly import any file
that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs to
see if this would be a feasible. If anyone has opinions or insights into
this I would love to hear from you.

Sam

···

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com> wrote:

Hi, Sam. I don't think we currently have a good answer for this built into
xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would
be fast enough that it wouldn't matter! That's obviously not where we are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share
another one of our debugging flags, '-driver-show-incremental'. You can add
this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies
discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies
discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered
later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered
later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered
later

In this case, I took a version of the Adventure sample project and
modified "Tree.swift"; that triggered recompilation of several other files.
Unfortunately this view doesn't tell you how they're related, only which
ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift" rather
than just "trying to figure out why it's repeating work") would be to look
at the "swiftdeps" files stored in your DerivedData folder. These are
currently just YAML files describing what Swift thinks the file depends on,
as well as what will trigger rebuilding of other files. This is intended to
be a conservative estimate, since *not* recompiling something would
result in an invalid binary. (Unfortunately I say "intended" because there
are known bugs; fortunately, archive builds are always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's
dependency analysis:
https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst.
The one thing that's *not* in there is the notion of changes that don't
affect other files at all. This is accomplished by computing a hash of all
the tokens that *could* affect other files, and seeing if that hash has
changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev < > swift-dev@swift.org> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of
code) where incremental builds can be extremely slow. I'm trying to do some
profiling to figure out what type of things cause large scale recompiles.
The problem is that I can't find a good way of telling which files get
recompiled on an incremental build and which do not. It seems like files
that are not recompiled still get listed in xcode, but the compiler just
passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality?
Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hey Sam,

One thought: if you have an app with mixed objc and swift code, then the app-bridge.h and app-swift.h files might be creating massive choke points in your dependency graph. I have no idea how optimized the bridging functionality is but it has always seemed like a potentially weak part of the dependency management. I imagine that with an objc half and a swift half of the code base, every time you make a change in a swift file you trigger recompilation of the objc half, and vice versa. I'd love to hear from the core team to what extent this is true!

George

···

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <swift-dev@swift.org> wrote:

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile flag that when turned on would require you to explicitly import any file that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs to see if this would be a feasible. If anyone has opinions or insights into this I would love to hear from you.

Sam

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:
Hi, Sam. I don't think we currently have a good answer for this built into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would be fast enough that it wouldn't matter! That's obviously not where we are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share another one of our debugging flags, '-driver-show-incremental'. You can add this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered later

In this case, I took a version of the Adventure sample project and modified "Tree.swift"; that triggered recompilation of several other files. Unfortunately this view doesn't tell you how they're related, only which ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift" rather than just "trying to figure out why it's repeating work") would be to look at the "swiftdeps" files stored in your DerivedData folder. These are currently just YAML files describing what Swift thinks the file depends on, as well as what will trigger rebuilding of other files. This is intended to be a conservative estimate, since not recompiling something would result in an invalid binary. (Unfortunately I say "intended" because there are known bugs; fortunately, archive builds are always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's dependency analysis: https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst. The one thing that's not in there is the notion of changes that don't affect other files at all. This is accomplished by computing a hash of all the tokens that could affect other files, and seeing if that hash has changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of code) where incremental builds can be extremely slow. I'm trying to do some profiling to figure out what type of things cause large scale recompiles. The problem is that I can't find a good way of telling which files get recompiled on an incremental build and which do not. It seems like files that are not recompiled still get listed in xcode, but the compiler just passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality? Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hi Jordan,

The thing that sticks out in the dependency analysis is the treatment of
external dependencies. The entire module has the same list of external
dependencies which causes a lot of needless recompiles- especially if you
start with a large swift project and slowly start to move things into
modules.

So to me the lowest hanging fruit would be to only mark files for
recompilation that explicitly import the external dependency. This seems
pretty safe since you can't compile unless the dependency is explicitly
imported. Has anyone on the list tried this before?

A second idea would be to consider a file as changed only if its build
artifact actually changes. Obviously, we'd have to actually build the file
to figure this out, so we wouldn't have the same level of parallelism
initially. Perhaps if it was an optional compiler flag this would be more
palatable? Also wondering if anyone has tried something along these lines.

Thanks!

George- The bridge between objective c and swift between is definitely a
choke point. We've been able to mitigate objective-c recompiles somewhat by
limiting our imports of swift into objective-c. We've even gone so far as
to make wrapper classes in objective c around some of our most commonly
used swift classes so as not to import swift.

It's also very true that changing an objective c .h file that is imported
into the bridging header will trigger massive recompiles. As more of our
app has transitioned to swift this has been less of an issue. Most of the
problems at this point have to do with recompiling a large portion of our
swift code due to small changes in unrelated parts of our other swift code.

Get the latest from Hopscotch!

Sign-up for our newsletter <http://eepurl.com/Ui0eX>

···

On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists@gmail.com> wrote:

Hey Sam,

One thought: if you have an app with mixed objc and swift code, then the
app-bridge.h and app-swift.h files might be creating massive choke points
in your dependency graph. I have no idea how optimized the bridging
functionality is but it has always seemed like a potentially weak part of
the dependency management. I imagine that with an objc half and a swift
half of the code base, every time you make a change in a swift file you
trigger recompilation of the objc half, and vice versa. I'd love to hear
from the core team to what extent this is true!

George

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev < > swift-dev@swift.org> wrote:

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile
flag that when turned on would require you to explicitly import any file
that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs to
see if this would be a feasible. If anyone has opinions or insights into
this I would love to hear from you.

Sam

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com> wrote:

Hi, Sam. I don't think we currently have a good answer for this built
into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds
would be fast enough that it wouldn't matter! That's obviously not where we
are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share
another one of our debugging flags, '-driver-show-incremental'. You can add
this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies
discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies
discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered
later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered
later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered
later

In this case, I took a version of the Adventure sample project and
modified "Tree.swift"; that triggered recompilation of several other files.
Unfortunately this view doesn't tell you how they're related, only which
ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift" rather
than just "trying to figure out why it's repeating work") would be to look
at the "swiftdeps" files stored in your DerivedData folder. These are
currently just YAML files describing what Swift thinks the file depends on,
as well as what will trigger rebuilding of other files. This is intended to
be a conservative estimate, since *not* recompiling something would
result in an invalid binary. (Unfortunately I say "intended" because there
are known bugs; fortunately, archive builds are always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's
dependency analysis:
https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst.
The one thing that's *not* in there is the notion of changes that don't
affect other files at all. This is accomplished by computing a hash of all
the tokens that *could* affect other files, and seeing if that hash has
changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev < >> swift-dev@swift.org> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of
code) where incremental builds can be extremely slow. I'm trying to do some
profiling to figure out what type of things cause large scale recompiles.
The problem is that I can't find a good way of telling which files get
recompiled on an incremental build and which do not. It seems like files
that are not recompiled still get listed in xcode, but the compiler just
passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality?
Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hello List (cc/Jordan),

At a high level: Brian and I are looking into contributing to incremental
compilation in Swift. Right now we're trying to do an incremental compile
by invoking swiftc.

Our understanding so far:
- swiftc takes a list of input files.
- If swiftc is invoked with '-incremental', it requires an
'-output-file-map' option to also be passed. We assume this is used to
determine where intermediate dependency files go.
- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of
files Xcode generates when building Swift projects, we determined
'-output-file-map' to be a JSON file (which is parsed by the llvm yaml
parser?
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100).
We tried writing our own, this is the shortest version that the compiler
didn't error on:

{
  "": {
    "swift-dependencies": "MyModule-main.swiftdeps"
  }
}

- When compiling with the above file ('swiftc -incremental -output-file-map
OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The
end result looks like this:

version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift
a2cf18ba2f)"
options: "9277a78155e85019ce36a3c52e9f3f02"
build_time: [514847765, 412105000]
inputs:
  "Class1.swift": [514841531, 0]
  "Class2.swift": [514844635, 0]
  "main.swift": [514841821, 0]

- Where the [xxx, yyy] are timestamps with the first number representing
seconds, the second nanoseconds. (
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221
)

- We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift
-parseable-output -save-temps' to show us the paths to the generated
.swiftdeps files. We assume that to get incremental compiles to work for
us, we'd need to pass these generated .swiftdeps files' paths to the
compiler somehow. We can't figure out how to do this, so this is the point
where our incremental compile fails:
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348

This is as far as we got. Would super appreciate if anyone on the list
could point us in the right direction from here, especially for the
following questions:

1. This would probably be easier if we could find the right test case. We
poked around in test/Driver/Dependencies which had a bunch of tests around
reading the .swiftdeps files, but we couldn't find tests that demonstrated
how incremental compilation worked at a high level.

2. '-output-file-map': Is this file meant to be written by hand, or is
there a part of the swift compiler that writes this for you?

3. Specifying paths for .swiftdeps: We had assumed this was done based on
the '-output-file-map', but writing the paths we wanted manually did not
seem to work. Any tips?

Thanks so much!
Sam and Brian

···

On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam@gethopscotch.com> wrote:

Hi Jordan,

The thing that sticks out in the dependency analysis is the treatment of
external dependencies. The entire module has the same list of external
dependencies which causes a lot of needless recompiles- especially if you
start with a large swift project and slowly start to move things into
modules.

So to me the lowest hanging fruit would be to only mark files for
recompilation that explicitly import the external dependency. This seems
pretty safe since you can't compile unless the dependency is explicitly
imported. Has anyone on the list tried this before?

A second idea would be to consider a file as changed only if its build
artifact actually changes. Obviously, we'd have to actually build the file
to figure this out, so we wouldn't have the same level of parallelism
initially. Perhaps if it was an optional compiler flag this would be more
palatable? Also wondering if anyone has tried something along these lines.

Thanks!

George- The bridge between objective c and swift between is definitely a
choke point. We've been able to mitigate objective-c recompiles somewhat by
limiting our imports of swift into objective-c. We've even gone so far as
to make wrapper classes in objective c around some of our most commonly
used swift classes so as not to import swift.

It's also very true that changing an objective c .h file that is imported
into the bridging header will trigger massive recompiles. As more of our
app has transitioned to swift this has been less of an issue. Most of the
problems at this point have to do with recompiling a large portion of our
swift code due to small changes in unrelated parts of our other swift code.

Get the latest from Hopscotch!

Sign-up for our newsletter <http://eepurl.com/Ui0eX>

On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists@gmail.com> wrote:

Hey Sam,

One thought: if you have an app with mixed objc and swift code, then the
app-bridge.h and app-swift.h files might be creating massive choke points
in your dependency graph. I have no idea how optimized the bridging
functionality is but it has always seemed like a potentially weak part of
the dependency management. I imagine that with an objc half and a swift
half of the code base, every time you make a change in a swift file you
trigger recompilation of the objc half, and vice versa. I'd love to hear
from the core team to what extent this is true!

George

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev < >> swift-dev@swift.org> wrote:

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile
flag that when turned on would require you to explicitly import any file
that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs to
see if this would be a feasible. If anyone has opinions or insights into
this I would love to hear from you.

Sam

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com> >> wrote:

Hi, Sam. I don't think we currently have a good answer for this built
into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds
would be fast enough that it wouldn't matter! That's obviously not where we
are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share
another one of our debugging flags, '-driver-show-incremental'. You can add
this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies
discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies
discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered
later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered
later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered
later

In this case, I took a version of the Adventure sample project and
modified "Tree.swift"; that triggered recompilation of several other files.
Unfortunately this view doesn't tell you how they're related, only which
ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift"
rather than just "trying to figure out why it's repeating work") would be
to look at the "swiftdeps" files stored in your DerivedData folder. These
are currently just YAML files describing what Swift thinks the file depends
on, as well as what will trigger rebuilding of other files. This is
intended to be a conservative estimate, since *not* recompiling
something would result in an invalid binary. (Unfortunately I say
"intended" because there are known bugs; fortunately, archive builds are
always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's
dependency analysis:
https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst.
The one thing that's *not* in there is the notion of changes that don't
affect other files at all. This is accomplished by computing a hash of all
the tokens that *could* affect other files, and seeing if that hash has
changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev < >>> swift-dev@swift.org> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of
code) where incremental builds can be extremely slow. I'm trying to do some
profiling to figure out what type of things cause large scale recompiles.
The problem is that I can't find a good way of telling which files get
recompiled on an incremental build and which do not. It seems like files
that are not recompiled still get listed in xcode, but the compiler just
passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality?
Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hello List (cc/Jordan),

At a high level: Brian and I are looking into contributing to incremental compilation in Swift. Right now we're trying to do an incremental compile by invoking swiftc.

Our understanding so far:
- swiftc takes a list of input files.
- If swiftc is invoked with '-incremental', it requires an '-output-file-map' option to also be passed. We assume this is used to determine where intermediate dependency files go.
- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of files Xcode generates when building Swift projects, we determined '-output-file-map' to be a JSON file (which is parsed by the llvm yaml parser? https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100). We tried writing our own, this is the shortest version that the compiler didn't error on:

{
  "": {
    "swift-dependencies": "MyModule-main.swiftdeps"
  }
}

- When compiling with the above file ('swiftc -incremental -output-file-map OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The end result looks like this:

version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift a2cf18ba2f)"
options: "9277a78155e85019ce36a3c52e9f3f02"
build_time: [514847765, 412105000]
inputs:
  "Class1.swift": [514841531, 0]
  "Class2.swift": [514844635, 0]
  "main.swift": [514841821, 0]

- Where the [xxx, yyy] are timestamps with the first number representing seconds, the second nanoseconds. (https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221)

- We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift -parseable-output -save-temps' to show us the paths to the generated .swiftdeps files. We assume that to get incremental compiles to work for us, we'd need to pass these generated .swiftdeps files' paths to the compiler somehow. We can't figure out how to do this, so this is the point where our incremental compile fails: https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348

You don't need to point the compiler at the swiftdeps files, it reuses the ones passed in the output file map if -incremental is passed.

You didn't mention it (I don't think) but you also need to pass -emit-dependencies, however it sounds like you are doing this.

This is as far as we got. Would super appreciate if anyone on the list could point us in the right direction from here, especially for the following questions:

1. This would probably be easier if we could find the right test case. We poked around in test/Driver/Dependencies which had a bunch of tests around reading the .swiftdeps files, but we couldn't find tests that demonstrated how incremental compilation worked at a high level.

2. '-output-file-map': Is this file meant to be written by hand, or is there a part of the swift compiler that writes this for you?

It is meant to be written by the build system which invokes Swift. See also:
  https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206
which is what the Swift package manager uses (which supports incremental compiles).

Your best bet is to take the exact command line used by xcodebuild, and then invoke swiftc with that to replicate the incremental build. If you run with -v you should be able to see exactly what files get built

HTH!
- Daniel

···

On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev <swift-dev@swift.org> wrote:

3. Specifying paths for .swiftdeps: We had assumed this was done based on the '-output-file-map', but writing the paths we wanted manually did not seem to work. Any tips?

Thanks so much!
Sam and Brian

On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam@gethopscotch.com <mailto:sam@gethopscotch.com>> wrote:
Hi Jordan,

The thing that sticks out in the dependency analysis is the treatment of external dependencies. The entire module has the same list of external dependencies which causes a lot of needless recompiles- especially if you start with a large swift project and slowly start to move things into modules.

So to me the lowest hanging fruit would be to only mark files for recompilation that explicitly import the external dependency. This seems pretty safe since you can't compile unless the dependency is explicitly imported. Has anyone on the list tried this before?

A second idea would be to consider a file as changed only if its build artifact actually changes. Obviously, we'd have to actually build the file to figure this out, so we wouldn't have the same level of parallelism initially. Perhaps if it was an optional compiler flag this would be more palatable? Also wondering if anyone has tried something along these lines.

Thanks!

George- The bridge between objective c and swift between is definitely a choke point. We've been able to mitigate objective-c recompiles somewhat by limiting our imports of swift into objective-c. We've even gone so far as to make wrapper classes in objective c around some of our most commonly used swift classes so as not to import swift.

It's also very true that changing an objective c .h file that is imported into the bridging header will trigger massive recompiles. As more of our app has transitioned to swift this has been less of an issue. Most of the problems at this point have to do with recompiling a large portion of our swift code due to small changes in unrelated parts of our other swift code.

Get the latest from Hopscotch!

Sign-up for our newsletter <http://eepurl.com/Ui0eX>

On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists@gmail.com <mailto:gwk.lists@gmail.com>> wrote:
Hey Sam,

One thought: if you have an app with mixed objc and swift code, then the app-bridge.h and app-swift.h files might be creating massive choke points in your dependency graph. I have no idea how optimized the bridging functionality is but it has always seemed like a potentially weak part of the dependency management. I imagine that with an objc half and a swift half of the code base, every time you make a change in a swift file you trigger recompilation of the objc half, and vice versa. I'd love to hear from the core team to what extent this is true!

George

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile flag that when turned on would require you to explicitly import any file that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs to see if this would be a feasible. If anyone has opinions or insights into this I would love to hear from you.

Sam

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:
Hi, Sam. I don't think we currently have a good answer for this built into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would be fast enough that it wouldn't matter! That's obviously not where we are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share another one of our debugging flags, '-driver-show-incremental'. You can add this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered later

In this case, I took a version of the Adventure sample project and modified "Tree.swift"; that triggered recompilation of several other files. Unfortunately this view doesn't tell you how they're related, only which ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift" rather than just "trying to figure out why it's repeating work") would be to look at the "swiftdeps" files stored in your DerivedData folder. These are currently just YAML files describing what Swift thinks the file depends on, as well as what will trigger rebuilding of other files. This is intended to be a conservative estimate, since not recompiling something would result in an invalid binary. (Unfortunately I say "intended" because there are known bugs; fortunately, archive builds are always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's dependency analysis: https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst. The one thing that's not in there is the notion of changes that don't affect other files at all. This is accomplished by computing a hash of all the tokens that could affect other files, and seeing if that hash has changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of code) where incremental builds can be extremely slow. I'm trying to do some profiling to figure out what type of things cause large scale recompiles. The problem is that I can't find a good way of telling which files get recompiled on an incremental build and which do not. It seems like files that are not recompiled still get listed in xcode, but the compiler just passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality? Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hello List (cc/Jordan),

At a high level: Brian and I are looking into contributing to incremental
compilation in Swift. Right now we're trying to do an incremental compile
by invoking swiftc.

Our understanding so far:
- swiftc takes a list of input files.
- If swiftc is invoked with '-incremental', it requires an
'-output-file-map' option to also be passed. We assume this is used to
determine where intermediate dependency files go.
- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of
files Xcode generates when building Swift projects, we determined
'-output-file-map' to be a JSON file (which is parsed by the llvm yaml
parser?
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100).
We tried writing our own, this is the shortest version that the compiler
didn't error on:

{
  "": {
    "swift-dependencies": "MyModule-main.swiftdeps"
  }
}

- When compiling with the above file ('swiftc -incremental
-output-file-map OurFileMap.json *.swift'), swiftc writes to
MyModule-main.swiftdeps. The end result looks like this:

version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift
a2cf18ba2f)"
options: "9277a78155e85019ce36a3c52e9f3f02"
build_time: [514847765, 412105000]
inputs:
  "Class1.swift": [514841531, 0]
  "Class2.swift": [514844635, 0]
  "main.swift": [514841821, 0]

- Where the [xxx, yyy] are timestamps with the first number representing
seconds, the second nanoseconds. (
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221
)

- We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift
-parseable-output -save-temps' to show us the paths to the generated
.swiftdeps files. We assume that to get incremental compiles to work for
us, we'd need to pass these generated .swiftdeps files' paths to the
compiler somehow. We can't figure out how to do this, so this is the point
where our incremental compile fails:
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348

You don't need to point the compiler at the swiftdeps files, it reuses the
ones passed in the output file map if -incremental is passed.

You didn't mention it (I don't think) but you also need to pass
-emit-dependencies, however it sounds like you are doing this.

This is as far as we got. Would super appreciate if anyone on the list
could point us in the right direction from here, especially for the
following questions:

1. This would probably be easier if we could find the right test case. We
poked around in test/Driver/Dependencies which had a bunch of tests around
reading the .swiftdeps files, but we couldn't find tests that demonstrated
how incremental compilation worked at a high level.

2. '-output-file-map': Is this file meant to be written by hand, or is
there a part of the swift compiler that writes this for you?

It is meant to be written by the build system which invokes Swift. See
also:

https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206
which is what the Swift package manager uses (which supports incremental
compiles).

This sounds perfect. I tried to install the llbuild but got stuck on
filecheck (and saw that you are trying to add it to the toolchain:
https://github.com/apple/swift-llbuild/pull/22/files#r61007286) Is there an
executable of llbuild I could download, or do you have any basic
instructions for what to do about filecheck?

Your best bet is to take the exact command line used by xcodebuild, and
then invoke swiftc with that to replicate the incremental build. If you run
with -v you should be able to see exactly what files get built

I tried running xcodebuild with the verbose flag but wasn't getting much
useful. Is that what you meant here? -v didn't seem to be an option.

Thanks so much,
Sam

···

On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev < > swift-dev@swift.org> wrote:

HTH!
- Daniel

3. Specifying paths for .swiftdeps: We had assumed this was done based on
the '-output-file-map', but writing the paths we wanted manually did not
seem to work. Any tips?

Thanks so much!
Sam and Brian

On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam@gethopscotch.com> > wrote:

Hi Jordan,

The thing that sticks out in the dependency analysis is the treatment of
external dependencies. The entire module has the same list of external
dependencies which causes a lot of needless recompiles- especially if you
start with a large swift project and slowly start to move things into
modules.

So to me the lowest hanging fruit would be to only mark files for
recompilation that explicitly import the external dependency. This seems
pretty safe since you can't compile unless the dependency is explicitly
imported. Has anyone on the list tried this before?

A second idea would be to consider a file as changed only if its build
artifact actually changes. Obviously, we'd have to actually build the file
to figure this out, so we wouldn't have the same level of parallelism
initially. Perhaps if it was an optional compiler flag this would be more
palatable? Also wondering if anyone has tried something along these lines.

Thanks!

George- The bridge between objective c and swift between is definitely a
choke point. We've been able to mitigate objective-c recompiles somewhat by
limiting our imports of swift into objective-c. We've even gone so far as
to make wrapper classes in objective c around some of our most commonly
used swift classes so as not to import swift.

It's also very true that changing an objective c .h file that is imported
into the bridging header will trigger massive recompiles. As more of our
app has transitioned to swift this has been less of an issue. Most of the
problems at this point have to do with recompiling a large portion of our
swift code due to small changes in unrelated parts of our other swift code.

Get the latest from Hopscotch!

Sign-up for our newsletter <http://eepurl.com/Ui0eX>

On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists@gmail.com> wrote:

Hey Sam,

One thought: if you have an app with mixed objc and swift code, then the
app-bridge.h and app-swift.h files might be creating massive choke points
in your dependency graph. I have no idea how optimized the bridging
functionality is but it has always seemed like a potentially weak part of
the dependency management. I imagine that with an objc half and a swift
half of the code base, every time you make a change in a swift file you
trigger recompilation of the objc half, and vice versa. I'd love to hear
from the core team to what extent this is true!

George

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev < >>> swift-dev@swift.org> wrote:

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile
flag that when turned on would require you to explicitly import any file
that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs
to see if this would be a feasible. If anyone has opinions or insights into
this I would love to hear from you.

Sam

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com> >>> wrote:

Hi, Sam. I don't think we currently have a good answer for this built
into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds
would be fast enough that it wouldn't matter! That's obviously not where we
are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share
another one of our debugging flags, '-driver-show-incremental'. You can add
this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies
discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies
discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies
discovered later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered
later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered
later

In this case, I took a version of the Adventure sample project and
modified "Tree.swift"; that triggered recompilation of several other files.
Unfortunately this view doesn't tell you how they're related, only which
ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift"
rather than just "trying to figure out why it's repeating work") would be
to look at the "swiftdeps" files stored in your DerivedData folder. These
are currently just YAML files describing what Swift thinks the file depends
on, as well as what will trigger rebuilding of other files. This is
intended to be a conservative estimate, since *not* recompiling
something would result in an invalid binary. (Unfortunately I say
"intended" because there are known bugs; fortunately, archive builds are
always clean builds anyway.)

There's a document in the Swift repo describing the logic behind
Swift's dependency analysis:
https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst.
The one thing that's *not* in there is the notion of changes that
don't affect other files at all. This is accomplished by computing a hash
of all the tokens that *could* affect other files, and seeing if that
hash has changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev < >>>> swift-dev@swift.org> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of
code) where incremental builds can be extremely slow. I'm trying to do some
profiling to figure out what type of things cause large scale recompiles.
The problem is that I can't find a good way of telling which files get
recompiled on an incremental build and which do not. It seems like files
that are not recompiled still get listed in xcode, but the compiler just
passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of
functionality? Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hello List (cc/Jordan),

At a high level: Brian and I are looking into contributing to incremental compilation in Swift. Right now we're trying to do an incremental compile by invoking swiftc.

Our understanding so far:
- swiftc takes a list of input files.
- If swiftc is invoked with '-incremental', it requires an '-output-file-map' option to also be passed. We assume this is used to determine where intermediate dependency files go.
- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of files Xcode generates when building Swift projects, we determined '-output-file-map' to be a JSON file (which is parsed by the llvm yaml parser? https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100). We tried writing our own, this is the shortest version that the compiler didn't error on:

{
  "": {
    "swift-dependencies": "MyModule-main.swiftdeps"
  }
}

- When compiling with the above file ('swiftc -incremental -output-file-map OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The end result looks like this:

version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift a2cf18ba2f)"
options: "9277a78155e85019ce36a3c52e9f3f02"
build_time: [514847765, 412105000]
inputs:
  "Class1.swift": [514841531, 0]
  "Class2.swift": [514844635, 0]
  "main.swift": [514841821, 0]

- Where the [xxx, yyy] are timestamps with the first number representing seconds, the second nanoseconds. (https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221)

- We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift -parseable-output -save-temps' to show us the paths to the generated .swiftdeps files. We assume that to get incremental compiles to work for us, we'd need to pass these generated .swiftdeps files' paths to the compiler somehow. We can't figure out how to do this, so this is the point where our incremental compile fails: https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348

You don't need to point the compiler at the swiftdeps files, it reuses the ones passed in the output file map if -incremental is passed.

You didn't mention it (I don't think) but you also need to pass -emit-dependencies, however it sounds like you are doing this.

This is as far as we got. Would super appreciate if anyone on the list could point us in the right direction from here, especially for the following questions:

1. This would probably be easier if we could find the right test case. We poked around in test/Driver/Dependencies which had a bunch of tests around reading the .swiftdeps files, but we couldn't find tests that demonstrated how incremental compilation worked at a high level.

2. '-output-file-map': Is this file meant to be written by hand, or is there a part of the swift compiler that writes this for you?

It is meant to be written by the build system which invokes Swift. See also:
  https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206
which is what the Swift package manager uses (which supports incremental compiles).

This sounds perfect. I tried to install the llbuild but got stuck on filecheck (and saw that you are trying to add it to the toolchain: https://github.com/apple/swift-llbuild/pull/22/files#r61007286) Is there an executable of llbuild I could download, or do you have any basic instructions for what to do about file check?

You shouldn't need to download llbuild, you probably want to go the xcodebuild route...

Your best bet is to take the exact command line used by xcodebuild, and then invoke swiftc with that to replicate the incremental build. If you run with -v you should be able to see exactly what files get built

I tried running xcodebuild with the verbose flag but wasn't getting much useful. Is that what you meant here? -v didn't seem to be an option.

I wasn't very clear here. What you need to do is run xcodebuild from the command line, and then find the top-level line which is compiling the module you are about (this will be the line with `swiftc ... -incremental ...` in it but not including `-frontend). That is the command you can run with `-v`.

- Daniel

···

On Apr 25, 2016, at 8:15 PM, Samantha John <sam@gethopscotch.com> wrote:
On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Thanks so much,
Sam

HTH!
- Daniel

3. Specifying paths for .swiftdeps: We had assumed this was done based on the '-output-file-map', but writing the paths we wanted manually did not seem to work. Any tips?

Thanks so much!
Sam and Brian

On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam@gethopscotch.com <mailto:sam@gethopscotch.com>> wrote:
Hi Jordan,

The thing that sticks out in the dependency analysis is the treatment of external dependencies. The entire module has the same list of external dependencies which causes a lot of needless recompiles- especially if you start with a large swift project and slowly start to move things into modules.

So to me the lowest hanging fruit would be to only mark files for recompilation that explicitly import the external dependency. This seems pretty safe since you can't compile unless the dependency is explicitly imported. Has anyone on the list tried this before?

A second idea would be to consider a file as changed only if its build artifact actually changes. Obviously, we'd have to actually build the file to figure this out, so we wouldn't have the same level of parallelism initially. Perhaps if it was an optional compiler flag this would be more palatable? Also wondering if anyone has tried something along these lines.

Thanks!

George- The bridge between objective c and swift between is definitely a choke point. We've been able to mitigate objective-c recompiles somewhat by limiting our imports of swift into objective-c. We've even gone so far as to make wrapper classes in objective c around some of our most commonly used swift classes so as not to import swift.

It's also very true that changing an objective c .h file that is imported into the bridging header will trigger massive recompiles. As more of our app has transitioned to swift this has been less of an issue. Most of the problems at this point have to do with recompiling a large portion of our swift code due to small changes in unrelated parts of our other swift code.

Get the latest from Hopscotch!

Sign-up for our newsletter <http://eepurl.com/Ui0eX>

On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists@gmail.com <mailto:gwk.lists@gmail.com>> wrote:
Hey Sam,

One thought: if you have an app with mixed objc and swift code, then the app-bridge.h and app-swift.h files might be creating massive choke points in your dependency graph. I have no idea how optimized the bridging functionality is but it has always seemed like a potentially weak part of the dependency management. I imagine that with an objc half and a swift half of the code base, every time you make a change in a swift file you trigger recompilation of the objc half, and vice versa. I'd love to hear from the core team to what extent this is true!

George

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Thank you Jordan! This is a great starting off point.

I'm thinking about proposing a "strict import" mode in swift: A compile flag that when turned on would require you to explicitly import any file that contained a dependency you needed (like in objective-c).

I'm going to spend more time looking over the docs and the output logs to see if this would be a feasible. If anyone has opinions or insights into this I would love to hear from you.

Sam

On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:
Hi, Sam. I don't think we currently have a good answer for this built into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would be fast enough that it wouldn't matter! That's obviously not where we are.)

Since '-debug-time-function-bodies' is now public knowledge, I'll share another one of our debugging flags, '-driver-show-incremental'. You can add this to your "Other Swift Flags". The output isn't very detailed, though:

Queuing Tree.swift (initial)
Queuing AdventureScene.swift (initial)
Queuing AdventureScene.swift because of dependencies discovered later
Queuing AppDelegate.swift because of dependencies discovered later
Queuing ChaseArtificialIntelligence.swift because of dependencies discovered later
Queuing Character.swift because of dependencies discovered later
Queuing SpawnArtificialIntelligence.swift because of dependencies discovered later
Queuing Goblin.swift because of dependencies discovered later
Queuing Cave.swift because of dependencies discovered later
Queuing AdventureSceneOSXEvents.swift because of dependencies discovered later
Queuing HeroCharacter.swift because of dependencies discovered later
Queuing EnemyCharacter.swift because of dependencies discovered later
Queuing Boss.swift because of dependencies discovered later
Queuing SharedAssetManagement.swift because of dependencies discovered later
Queuing Warrior.swift because of dependencies discovered later
Queuing Archer.swift because of dependencies discovered later
Queuing Player.swift because of dependencies discovered later
Queuing ArtificialIntelligence.swift because of dependencies discovered later

In this case, I took a version of the Adventure sample project and modified "Tree.swift"; that triggered recompilation of several other files. Unfortunately this view doesn't tell you how they're related, only which ones are actually getting rebuilt.

The next step (and moving into the territory of "working on Swift" rather than just "trying to figure out why it's repeating work") would be to look at the "swiftdeps" files stored in your DerivedData folder. These are currently just YAML files describing what Swift thinks the file depends on, as well as what will trigger rebuilding of other files. This is intended to be a conservative estimate, since not recompiling something would result in an invalid binary. (Unfortunately I say "intended" because there are known bugs; fortunately, archive builds are always clean builds anyway.)

There's a document in the Swift repo describing the logic behind Swift's dependency analysis: https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst. The one thing that's not in there is the notion of changes that don't affect other files at all. This is accomplished by computing a hash of all the tokens that could affect other files, and seeing if that hash has changed.

We definitely have room for improvement here.

Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

I have a large project (308 swift files, 441 objective c, 66k lines of code) where incremental builds can be extremely slow. I'm trying to do some profiling to figure out what type of things cause large scale recompiles. The problem is that I can't find a good way of telling which files get recompiled on an incremental build and which do not. It seems like files that are not recompiled still get listed in xcode, but the compiler just passes over them really fast.

Does anyone know if xctool or xcodebuild has this type of functionality? Or is there some other way to get this info?

Thank you,
Sam
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

Sorry to resurrect an old thread, but:
Jordan, I'm trying to learn more about swift/lib/Driver. I picked up a few tasks to get familiar with it:
- https://bugs.swift.org/browse/SR-1788- https://bugs.swift.org/browse/SR-2400- https://bugs.swift.org/browse/SR-656
I didn't find any tasks related to incremental builds, though. Are there any low hanging fruit or gardening tasks related to incremental builds? Maybe something you've been meaning to do, but haven't written a task for yet?
- Brian Gesiak

···

On Mon, Apr 25, 2016 at 10:17 PM -0500, "Daniel Dunbar via swift-dev" <swift-dev@swift.org> wrote:

On Apr 25, 2016, at 8:15 PM, Samantha John <sam@gethopscotch.com> wrote:
On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev <swift-dev@swift.org> wrote:
Hello List (cc/Jordan),
At a high level: Brian and I are looking into contributing to incremental compilation in Swift. Right now we’re trying to do an incremental compile by invoking swiftc.
Our understanding so far:- swiftc takes a list of input files. - If swiftc is invoked with ‘-incremental’, it requires an ‘-output-file-map’ option to also be passed. We assume this is used to determine where intermediate dependency files go.- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of files Xcode generates when building Swift projects, we determined ‘-output-file-map’ to be a JSON file (which is parsed by the llvm yaml parser? https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100). We tried writing our own, this is the shortest version that the compiler didn’t error on:
{ "": { "swift-dependencies": "MyModule-main.swiftdeps" }}

  • When compiling with the above file (‘swiftc -incremental -output-file-map OurFileMap.json *.swift’), swiftc writes to MyModule-main.swiftdeps. The end result looks like this:
    version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift a2cf18ba2f)"options: "9277a78155e85019ce36a3c52e9f3f02"build_time: [514847765, 412105000]inputs: "Class1.swift": [514841531, 0] "Class2.swift": [514844635, 0] "main.swift": [514841821, 0]
    - Where the [xxx, yyy] are timestamps with the first number representing seconds, the second nanoseconds. (https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221)
    - We invoked ‘swiftc -incremental -output-file-map OurFileMap.json *.swift -parseable-output -save-temps’ to show us the paths to the generated .swiftdeps files. We assume that to get incremental compiles to work for us, we’d need to pass these generated .swiftdeps files’ paths to the compiler somehow. We can’t figure out how to do this, so this is the point where our incremental compile fails: https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348
    You don’t need to point the compiler at the swiftdeps files, it reuses the ones passed in the output file map if -incremental is passed.
    You didn’t mention it (I don’t think) but you also need to pass -emit-dependencies, however it sounds like you are doing this.
    This is as far as we got. Would super appreciate if anyone on the list could point us in the right direction from here, especially for the following questions:
    1. This would probably be easier if we could find the right test case. We poked around in test/Driver/Dependencies which had a bunch of tests around reading the .swiftdeps files, but we couldn’t find tests that demonstrated how incremental compilation worked at a high level.
    2. ‘-output-file-map’: Is this file meant to be written by hand, or is there a part of the swift compiler that writes this for you?
    It is meant to be written by the build system which invokes Swift. See also: https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206which is what the Swift package manager uses (which supports incremental compiles).
    This sounds perfect. I tried to install the llbuild but got stuck on filecheck (and saw that you are trying to add it to the toolchain: https://github.com/apple/swift-llbuild/pull/22/files#r61007286) Is there an executable of llbuild I could download, or do you have any basic instructions for what to do about file check?
    You shouldn’t need to download llbuild, you probably want to go the xcodebuild route...

Your best bet is to take the exact command line used by xcodebuild, and then invoke swiftc with that to replicate the incremental build. If you run with -v you should be able to see exactly what files get built
I tried running xcodebuild with the verbose flag but wasn't getting much useful. Is that what you meant here? -v didn't seem to be an option.
I wasn't very clear here. What you need to do is run xcodebuild from the command line, and then find the top-level line which is compiling the module you are about (this will be the line with `swiftc ... -incremental ...` in it but not including `-frontend). That is the command you can run with `-v`.
- Daniel
Thanks so much,
Sam

HTH! - Daniel

3. Specifying paths for .swiftdeps: We had assumed this was done based on the '-output-file-map', but writing the paths we wanted manually did not seem to work. Any tips?
Thanks so much!Sam and Brian
On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam@gethopscotch.com> wrote:
Hi Jordan,
The thing that sticks out in the dependency analysis is the treatment of external dependencies. The entire module has the same list of external dependencies which causes a lot of needless recompiles- especially if you start with a large swift project and slowly start to move things into modules.
So to me the lowest hanging fruit would be to only mark files for recompilation that explicitly import the external dependency. This seems pretty safe since you can't compile unless the dependency is explicitly imported. Has anyone on the list tried this before?
A second idea would be to consider a file as changed only if its build artifact actually changes. Obviously, we'd have to actually build the file to figure this out, so we wouldn't have the same level of parallelism initially. Perhaps if it was an optional compiler flag this would be more palatable? Also wondering if anyone has tried something along these lines.
Thanks!

George- The bridge between objective c and swift between is definitely a choke point. We've been able to mitigate objective-c recompiles somewhat by limiting our imports of swift into objective-c. We've even gone so far as to make wrapper classes in objective c around some of our most commonly used swift classes so as not to import swift.
It's also very true that changing an objective c .h file that is imported into the bridging header will trigger massive recompiles. As more of our app has transitioned to swift this has been less of an issue. Most of the problems at this point have to do with recompiling a large portion of our swift code due to small changes in unrelated parts of our other swift code.

Get the latest from Hopscotch!

Sign-up for our newsletter
On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists@gmail.com> wrote:
Hey Sam,
One thought: if you have an app with mixed objc and swift code, then the app-bridge.h and app-swift.h files might be creating massive choke points in your dependency graph. I have no idea how optimized the bridging functionality is but it has always seemed like a potentially weak part of the dependency management. I imagine that with an objc half and a swift half of the code base, every time you make a change in a swift file you trigger recompilation of the objc half, and vice versa. I'd love to hear from the core team to what extent this is true!
George

On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <swift-dev@swift.org> wrote:
Thank you Jordan! This is a great starting off point.
I'm thinking about proposing a "strict import" mode in swift: A compile flag that when turned on would require you to explicitly import any file that contained a dependency you needed (like in objective-c).
I'm going to spend more time looking over the docs and the output logs to see if this would be a feasible. If anyone has opinions or insights into this I would love to hear from you.
Sam
On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose@apple.com> wrote:
Hi, Sam. I don't think we currently have a good answer for this built into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would be fast enough that it wouldn't matter! That's obviously not where we are.)
Since '-debug-time-function-bodies' is now public knowledge, I'll share another one of our debugging flags, '-driver-show-incremental'. You can add this to your "Other Swift Flags". The output isn't very detailed, though:
Queuing Tree.swift (initial)Queuing AdventureScene.swift (initial)Queuing AdventureScene.swift because of dependencies discovered laterQueuing AppDelegate.swift because of dependencies discovered laterQueuing ChaseArtificialIntelligence.swift because of dependencies discovered laterQueuing Character.swift because of dependencies discovered laterQueuing SpawnArtificialIntelligence.swift because of dependencies discovered laterQueuing Goblin.swift because of dependencies discovered laterQueuing Cave.swift because of dependencies discovered laterQueuing AdventureSceneOSXEvents.swift because of dependencies discovered laterQueuing HeroCharacter.swift because of dependencies discovered laterQueuing EnemyCharacter.swift because of dependencies discovered laterQueuing Boss.swift because of dependencies discovered laterQueuing SharedAssetManagement.swift because of dependencies discovered laterQueuing Warrior.swift because of dependencies discovered laterQueuing Archer.swift because of dependencies discovered laterQueuing Player.swift because of dependencies discovered laterQueuing ArtificialIntelligence.swift because of dependencies discovered later
In this case, I took a version of the Adventure sample project and modified "Tree.swift"; that triggered recompilation of several other files. Unfortunately this view doesn't tell you how they're related, only which ones are actually getting rebuilt.
The next step (and moving into the territory of "working on Swift" rather than just "trying to figure out why it's repeating work") would be to look at the "swiftdeps" files stored in your DerivedData folder. These are currently just YAML files describing what Swift thinks the file depends on, as well as what will trigger rebuilding of other files. This is intended to be a conservative estimate, since not recompiling something would result in an invalid binary. (Unfortunately I say "intended" because there are known bugs; fortunately, archive builds are always clean builds anyway.)
There's a document in the Swift repo describing the logic behind Swift's dependency analysis: https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst. The one thing that's not in there is the notion of changes that don't affect other files at all. This is accomplished by computing a hash of all the tokens that could affect other files, and seeing if that hash has changed.
We definitely have room for improvement here.
Jordan

On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <swift-dev@swift.org> wrote:
I have a large project (308 swift files, 441 objective c, 66k lines of code) where incremental builds can be extremely slow. I'm trying to do some profiling to figure out what type of things cause large scale recompiles. The problem is that I can't find a good way of telling which files get recompiled on an incremental build and which do not. It seems like files that are not recompiled still get listed in xcode, but the compiler just passes over them really fast.
Does anyone know if xctool or xcodebuild has this type of functionality? Or is there some other way to get this info?

Thank you,
Sam_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

[+swift-dev again, for posterity]

Let’s see, those two are really mostly “I haven’t thought about how to recover from this scenario.” I have an internal presentation on why dependency analysis is the way it is that I should probably throw up in the docs in some format, but mostly these are all the occasions where dependencies change during a build, and I haven’t convinced myself one way or another whether the swiftdeps file left over from the last compilation is sufficient to represent what happened to a file that’s been removed.

Tasks more directly related to incremental builds aren’t really represented in JIRA because there aren’t many in digestible chunks. The most obvious improvements are all wrapped up in other things:

- Moving operators into types has the potential to stop making “==“ a top-level dependency on everything that defines “==“, but we haven’t figured out how yet. (Being part of a constraint system instead of being looked up as a member makes that hard, even though if we pick a member operator it must operate on one of the types involved.)

- Vtable layout depends on the methods in a class, so even if file A only calls one method in a class, adding a completely different method forces file A to be recompiled. This is probably the right trade-off in release builds, but not debug builds.

- Dependencies only use base names rather than full names, but default arguments make it hard to specify which full names are affected when a particular method is changed.

There are also two Radars about places where the incremental build logic seems to be incorrect; unfortunately I can’t share those projects. Obviously a bad incremental experience is still preferable to behavior changing on a clean build.

Tasks related to incremental builds also aren’t represented so much in Driver because at this point a lot of it’s about breaking dependencies and not being overconservative. The Driver generally can’t help with that; it just deals with the dependencies that the frontend spits out. The one thing we could do here is improve -driver-show-incremental. I haven’t used it in a while so I don’t actually know where I left off with that.

One thing we have noticed is that the cost for each frontend task for a project with 1500 tiny, non-dependent Swift files and a bridging header is much higher than it is for Clang. People have blamed Swift having to open and parse every file in the project, but I don’t actually believe this is the case; time spent in Parse is tiny for these functions. It would be very interesting for someone to investigate where this time is going. (There’s a similar project in SR-2461 <https://bugs.swift.org/browse/SR-2461>; I haven’t checked whether the project we have in Radar is appropriately clean of user info to share.)

I guess that’s about it. Oh, and I keep meaning to write tools to manipulate the data in the swiftdeps YAML files, but that can happen at any time. You’re welcome to contribute something before I get to it.

Definitely thanks for helping out here!
Jordan

···

On Aug 26, 2016, at 11:35, Brian Gesiak <modocache@gmail.com> wrote:

Although I was hoping to do more work on this over the weekend, so I'd really appreciate any pointers you had before then.

I'm currently looking into the "file removed" case that causes a complete rebuild -- the "FIXME: Distinguish errors from "file removed", which is benign." and "FIXME: Can we do better?" comments in lib/Driver/Driver.cpp. So if you have any insights into that, or really anything else, I'd love to hear them!

- Brian

Thank you, Jordan!

I had guessed there wouldn't be much low-hanging fruit here -- I'm sure a
ton of work has been put into improving compilation time, after all. Thanks
for all the great explanations.

The one thing we *could* do here is improve -driver-show-incremental. I

haven’t used it in a while so I don’t actually know where I left off with
that.

I assume you mean showing more information when this option is specified.
Let me know if there's something more. Also, do you mind if I create a JIRA
task for this?

One thing we *have* noticed is that the cost for each frontend task for a
project with 1500 tiny, non-dependent Swift files and a bridging header is
much higher than it is for Clang. People have blamed Swift having to open
and parse every file in the project, but I don’t actually believe this is
the case; time spent in Parse is *tiny* for these functions. It would be
very interesting for someone to investigate where this time is going.
(There’s a similar project in SR-2461
<https://bugs.swift.org/browse/SR-2461>; I haven’t checked whether the
project we have in Radar is appropriately clean of user info to share.

I looked into this a bit. Compiling 1501 Swift files takes 38 seconds on my
Linux VM (https://gist.github.com/modocache/7b6e2f0c17f65e8418c2acf5e1c82154),
compiling 1501 C files using clang takes 22 seconds (
https://gist.github.com/modocache/ceeefaa76befd738e62e078dd2e186f1). These
numbers don't seem outrageous to me, so I guess the problem could be
specific to bridging headers. I don't have a macOS environment capable of
building Swift right now, but I'll try to look into this more once Xcode 8
is released.

I guess that’s about it. Oh, and I keep meaning to write tools to
manipulate the data in the swiftdeps YAML files, but that can happen at any
time. You’re welcome to contribute something before I get to it.

What do you have in mind? A command-line tool to add input files, update
their timestamps, that sort of thing?

- Brian Gesiak

I’ll refresh the thread a bit.

I’m trying to figure out what causes constant rebuild in our project, here what I found. I’m looking at the project now and what I see:

  1. swiftc -incremental -whole-module-optimization
    re-compiles single Swift file - the reason is unknown
  2. swiftc -incremental
    re-compiles 138 Swift files - the reason is unknown

I’m running the command found in Xcode.

Apple Swift version 4.1 (swiftlang-902.0.48 clang-902.0.37.1)

WMO rebuilds your entire project, always. Just because you see only one job doesn’t mean it’s only recompiling one file.

I found that -incremental is not compatible with -whole-module-optimization, but I don't think it's reported to the user when building (at least with Xcode, maybe there is a good reason, but still).

There's not a good reason. :-( In the early days of Swift Xcode passed -incremental all the time (it might even still do so), and so if we warned on -incremental plus -whole-module-optimization then Xcode users would get a warning they couldn't turn off whenever they tried to use WMO.

(Yes, Apple owned both Swift and Xcode, but they had different internal release cycles, and so not adding the warning was much simpler. Except that we should have gone back and fixed Xcode so that we could eventually add the warning.)