SwiftPM recompilation - Packages folder


(Ricardo N Olivieri) #1

Hello all,

I was wondering if someone could shed some light on how SwiftPM determines
if it needs to compile again any of the source code under the Packages
folder. I had thought that the modified timestamp attribute of any source
code files under the Packages folder was the key for determining if any of
the Swift package dependencies should be recompiled. However, it seems
SwiftPM uses additional criteria to make this determination. Here’s the
test we executed:

1) At T0, we executed swift build. As expected all Swift package
dependencies were cloned and compiled along with the application’s code.

2) At T1, we executed swift build again and nothing was recompiled, as
expected since no code changes were made.

3) At T2, we made a copy of the Packages folder and place this copy in a
location outside the repo’s directory. We then deleted the Packages folder
from the repo’s directory, and, finally, we copied the Packages folder
back to its original location in the repo’s directory.

4) At T3, we executed again swift build and noticed that SwiftPM
recompiled each one of the Swift package dependencies in the Packages
folder, even though no code changes were made. We also double checked that
the modified and creation timestamps for the Packages folder and folders
and files in it were not changed. After doing so, we concluded that though
copying the Packages folder back to the application’s repo did keep the
original creation and modified timestamp values of all files and folders,
SwiftPM still recompiled all of the dependencies in the Packages folder.

Does anyone know why we are seeing in SwiftPM the behavior described
above?

Regards,
   Ricardo Olivieri


(Daniel Dunbar) #2

swift-build-tool will currently rebuild any file if it detects a change to
the file inode or device -- my guess is that is what you are seeing here.

My intention is to eventually move to using content-based signatures to
determine the rebuild conditions (for tools where we know that is safe),
which would prevent the rebuild in this situation.

Is there a particularly reason it is important to you that this not rebuild?

- Daniel

···

On Mon, Nov 21, 2016 at 1:54 PM, Ricardo N Olivieri via swift-build-dev < swift-build-dev@swift.org> wrote:

Hello all,

I was wondering if someone could shed some light on how SwiftPM determines
if it needs to compile again any of the source code under the Packages
folder. I had thought that the modified timestamp attribute of any source
code files under the Packages folder was the key for determining if any of
the Swift package dependencies should be recompiled. However, it seems
SwiftPM uses additional criteria to make this determination. Here’s the
test we executed:

1) At T0, we executed swift build. As expected all Swift package
dependencies were cloned and compiled along with the application’s code.

2) At T1, we executed swift build again and nothing was recompiled, as
expected since no code changes were made.

3) At T2, we made a copy of the Packages folder and place this copy in a
location outside the repo’s directory. We then deleted the Packages folder
from the repo’s directory, and, finally, we copied the Packages folder back
to its original location in the repo’s directory.

4) At T3, we executed again swift build and noticed that SwiftPM
recompiled each one of the Swift package dependencies in the Packages
folder, even though no code changes were made. We also double checked that
the modified and creation timestamps for the Packages folder and folders
and files in it were not changed. After doing so, we concluded that though
copying the Packages folder back to the application’s repo did keep the
original creation and modified timestamp values of all files and folders,
SwiftPM still recompiled all of the dependencies in the Packages folder.

Does anyone know why we are seeing in SwiftPM the behavior described
above?

Regards,
  Ricardo Olivieri

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Ricardo N Olivieri) #3

Thanks Daniel for the response. That's probably the reason (i.e. metadata
changes in the file inode) for the behavior we are seeing in SwiftPM.

The use case for us is as follows:

We'd like to improve even further the performance of the IBM Bluemix
buildpack for Swift and one way we could accomplish this is by not
recompiling the Packages folder as long as the content of that folder has
not changed. Without getting into too many details about cloud foundry
buildpacks, for the buildpack to reuse the contents of the Packages folder
in future re-deployments of the same application to the cloud, the
buildpack has to copy the Packages folder a different folder outside the
repo's folder and then copy it back again when the developer redeploys
his/her application. This is when we noticed that despite the contents of
the Packages folder had not changed, SwiftPM still recompiled all of the
dependencies under the Packages folder.

Is there an ETA for using content-based signatures instead?

Regards,
   Ricardo Olivieri
   Software Engineer
   IBM Swift Engineering at IBM Cloud

···

From: Daniel Dunbar <daniel@zuster.org>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org
Date: 11/21/2016 03:58 PM
Subject: Re: [swift-build-dev] SwiftPM recompilation - Packages
folder
Sent by: daniel.dunbar@gmail.com

swift-build-tool will currently rebuild any file if it detects a change to
the file inode or device -- my guess is that is what you are seeing here.

My intention is to eventually move to using content-based signatures to
determine the rebuild conditions (for tools where we know that is safe),
which would prevent the rebuild in this situation.

Is there a particularly reason it is important to you that this not
rebuild?

- Daniel

On Mon, Nov 21, 2016 at 1:54 PM, Ricardo N Olivieri via swift-build-dev < swift-build-dev@swift.org> wrote:
Hello all,

I was wondering if someone could shed some light on how SwiftPM determines
if it needs to compile again any of the source code under the Packages
folder. I had thought that the modified timestamp attribute of any source
code files under the Packages folder was the key for determining if any of
the Swift package dependencies should be recompiled. However, it seems
SwiftPM uses additional criteria to make this determination. Here’s the
test we executed:

1) At T0, we executed swift build. As expected all Swift package
dependencies were cloned and compiled along with the application’s code.

2) At T1, we executed swift build again and nothing was recompiled, as
expected since no code changes were made.

3) At T2, we made a copy of the Packages folder and place this copy in a
location outside the repo’s directory. We then deleted the Packages folder
from the repo’s directory, and, finally, we copied the Packages folder
back to its original location in the repo’s directory.

4) At T3, we executed again swift build and noticed that SwiftPM
recompiled each one of the Swift package dependencies in the Packages
folder, even though no code changes were made. We also double checked that
the modified and creation timestamps for the Packages folder and folders
and files in it were not changed. After doing so, we concluded that though
copying the Packages folder back to the application’s repo did keep the
original creation and modified timestamp values of all files and folders,
SwiftPM still recompiled all of the dependencies in the Packages folder.

Does anyone know why we are seeing in SwiftPM the behavior described
above?

Regards,
  Ricardo Olivieri

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #4

Thanks Daniel for the response. That's probably the reason (i.e. metadata
changes in the file inode) for the behavior we are seeing in SwiftPM.

The use case for us is as follows:

We'd like to improve even further the performance of the IBM Bluemix
buildpack for Swift and one way we could accomplish this is by not
recompiling the Packages folder as long as the content of that folder has
not changed. Without getting into too many details about cloud foundry
buildpacks, for the buildpack to reuse the contents of the Packages folder
in future re-deployments of the same application to the cloud, the
buildpack has to copy the Packages folder a different folder outside the
repo's folder and then copy it back again when the developer redeploys
his/her application. This is when we noticed that despite the contents of
the Packages folder had not changed, SwiftPM still recompiled all of the
dependencies under the Packages folder.

Makes sense, thanks!

Is there an ETA for using content-based signatures instead?

No, unfortunately.

If it is important for your use case, I'd be ok with accepting a PR to
change swift-build-tool to only look at the time/size information to
workaround this in the meantime.

- Daniel

···

On Mon, Nov 21, 2016 at 2:37 PM, Ricardo N Olivieri < ricardo.olivieri@us.ibm.com> wrote:

Regards,
  Ricardo Olivieri
  Software Engineer
  IBM Swift Engineering at IBM Cloud

From: Daniel Dunbar <daniel@zuster.org>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org
Date: 11/21/2016 03:58 PM
Subject: Re: [swift-build-dev] SwiftPM recompilation - Packages
folder
Sent by: daniel.dunbar@gmail.com
------------------------------

swift-build-tool will currently rebuild any file if it detects a change to
the file inode or device -- my guess is that is what you are seeing here.

My intention is to eventually move to using content-based signatures to
determine the rebuild conditions (for tools where we know that is safe),
which would prevent the rebuild in this situation.

Is there a particularly reason it is important to you that this not
rebuild?

- Daniel

On Mon, Nov 21, 2016 at 1:54 PM, Ricardo N Olivieri via swift-build-dev < > *swift-build-dev@swift.org* <swift-build-dev@swift.org>> wrote:
Hello all,

I was wondering if someone could shed some light on how SwiftPM determines
if it needs to compile again any of the source code under the Packages
folder. I had thought that the modified timestamp attribute of any source
code files under the Packages folder was the key for determining if any of
the Swift package dependencies should be recompiled. However, it seems
SwiftPM uses additional criteria to make this determination. Here’s the
test we executed:

1) At T0, we executed swift build. As expected all Swift package
dependencies were cloned and compiled along with the application’s code.

2) At T1, we executed swift build again and nothing was recompiled, as
expected since no code changes were made.

3) At T2, we made a copy of the Packages folder and place this copy in a
location outside the repo’s directory. We then deleted the Packages folder
from the repo’s directory, and, finally, we copied the Packages folder back
to its original location in the repo’s directory.

4) At T3, we executed again swift build and noticed that SwiftPM
recompiled each one of the Swift package dependencies in the Packages
folder, even though no code changes were made. We also double checked that
the modified and creation timestamps for the Packages folder and folders
and files in it were not changed. After doing so, we concluded that though
copying the Packages folder back to the application’s repo did keep the
original creation and modified timestamp values of all files and folders,
SwiftPM still recompiled all of the dependencies in the Packages folder.

Does anyone know why we are seeing in SwiftPM the behavior described
above?

Regards,
  Ricardo Olivieri

_______________________________________________
swift-build-dev mailing list
*swift-build-dev@swift.org* <swift-build-dev@swift.org>
*https://lists.swift.org/mailman/listinfo/swift-build-dev*
<https://lists.swift.org/mailman/listinfo/swift-build-dev>


(Ricardo N Olivieri) #5

Got it, thanks Daniel. Just in case we get the cycles to work on a PR,
could you point us to the file (or set of files) in the
https://github.com/apple/swift-package-manager repo that we'd need to
modify so that the swift-build-tool only looks at the time/size metadata?
I must admit I am not quite familiar yet with the internals of SwiftPM.

Regards,
   Ricardo Olivieri
   Software Engineer
   IBM Swift Engineering at IBM Cloud

···

From: Daniel Dunbar <daniel@zuster.org>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org
Date: 11/21/2016 04:41 PM
Subject: Re: [swift-build-dev] SwiftPM recompilation - Packages
folder
Sent by: daniel.dunbar@gmail.com

On Mon, Nov 21, 2016 at 2:37 PM, Ricardo N Olivieri < ricardo.olivieri@us.ibm.com> wrote:
Thanks Daniel for the response. That's probably the reason (i.e. metadata
changes in the file inode) for the behavior we are seeing in SwiftPM.

The use case for us is as follows:

We'd like to improve even further the performance of the IBM Bluemix
buildpack for Swift and one way we could accomplish this is by not
recompiling the Packages folder as long as the content of that folder has
not changed. Without getting into too many details about cloud foundry
buildpacks, for the buildpack to reuse the contents of the Packages folder
in future re-deployments of the same application to the cloud, the
buildpack has to copy the Packages folder a different folder outside the
repo's folder and then copy it back again when the developer redeploys
his/her application. This is when we noticed that despite the contents of
the Packages folder had not changed, SwiftPM still recompiled all of the
dependencies under the Packages folder.

Makes sense, thanks!

Is there an ETA for using content-based signatures instead?

No, unfortunately.

If it is important for your use case, I'd be ok with accepting a PR to
change swift-build-tool to only look at the time/size information to
workaround this in the meantime.

- Daniel

Regards,
  Ricardo Olivieri
  Software Engineer
  IBM Swift Engineering at IBM Cloud

From: Daniel Dunbar <daniel@zuster.org>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org
Date: 11/21/2016 03:58 PM
Subject: Re: [swift-build-dev] SwiftPM recompilation - Packages
folder
Sent by: daniel.dunbar@gmail.com

swift-build-tool will currently rebuild any file if it detects a change to
the file inode or device -- my guess is that is what you are seeing here.

My intention is to eventually move to using content-based signatures to
determine the rebuild conditions (for tools where we know that is safe),
which would prevent the rebuild in this situation.

Is there a particularly reason it is important to you that this not
rebuild?

- Daniel

On Mon, Nov 21, 2016 at 1:54 PM, Ricardo N Olivieri via swift-build-dev < swift-build-dev@swift.org> wrote:
Hello all,

I was wondering if someone could shed some light on how SwiftPM determines
if it needs to compile again any of the source code under the Packages
folder. I had thought that the modified timestamp attribute of any source
code files under the Packages folder was the key for determining if any of
the Swift package dependencies should be recompiled. However, it seems
SwiftPM uses additional criteria to make this determination. Here’s the
test we executed:

1) At T0, we executed swift build. As expected all Swift package
dependencies were cloned and compiled along with the application’s code.

2) At T1, we executed swift build again and nothing was recompiled, as
expected since no code changes were made.

3) At T2, we made a copy of the Packages folder and place this copy in a
location outside the repo’s directory. We then deleted the Packages folder
from the repo’s directory, and, finally, we copied the Packages folder
back to its original location in the repo’s directory.

4) At T3, we executed again swift build and noticed that SwiftPM
recompiled each one of the Swift package dependencies in the Packages
folder, even though no code changes were made. We also double checked that
the modified and creation timestamps for the Packages folder and folders
and files in it were not changed. After doing so, we concluded that though
copying the Packages folder back to the application’s repo did keep the
original creation and modified timestamp values of all files and folders,
SwiftPM still recompiled all of the dependencies in the Packages folder.

Does anyone know why we are seeing in SwiftPM the behavior described
above?

Regards,
  Ricardo Olivieri

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev