Over the past few weeks I’ve encountered some subtle issues with SwiftPM and the system pkg-config tool disagreeing over what compiler and linker flags should be provided on a given Linux system, as shown in SR-6317 (https://bugs.swift.org/browse/SR-6317\). After some investigation on my part, I’d like to propose that we should replace SwiftPM’s hand-rolled pkgconfig parsing logic (in PkgConfig.swift) with a tool that simply shells out to the system pkg-config binary, if available.
In general, on Linux, any system capable of compiling a SwiftPM project is going to have pkg-config installed. It’s an extremely common dependency in build toolchains, and the likelihood is that if your project wants to link against anything more esoteric than libc and the various Swift shared libraries pkg-config will have been installed anyway. If for some reason pkg-config is not present on the $PATH on such a system, that is likely a signal that pkg-config is not a good tool to use to populate the linker and compile flags *anyway*.
Given that pkg-config is so widely deployed in Linux, it seems senseless to carry a reimplementation of that tool in SwiftPM. This is doubly true given that the pkg-config binary has a number of compile-time configurations applied by the system builder that give it special knowledge of the system’s default linker and include search paths. SwiftPM’s lack of that knowledge is what led to SR-6317, and rather than attempt to add complexity to SwiftPM’s implementation we would probably be better served by simply deferring to the implementation on the system that knows what to do best.
My proposal is to replace the current implementation with one that shells out to pkg-config. If pkg-config cannot be found, then we simply fall back to the rest of the user configuration, exactly as if there were no pkg-config support at all.
+1, we should use the right tool for the right job. Happy to still have the fallback code in SwiftPM.
···
On 30 Nov 2017, at 11:42 am, Cory Benfield <cbenfield@apple.com> wrote:
Hey all,
Over the past few weeks I’ve encountered some subtle issues with SwiftPM and the system pkg-config tool disagreeing over what compiler and linker flags should be provided on a given Linux system, as shown in SR-6317 (https://bugs.swift.org/browse/SR-6317\). After some investigation on my part, I’d like to propose that we should replace SwiftPM’s hand-rolled pkgconfig parsing logic (in PkgConfig.swift) with a tool that simply shells out to the system pkg-config binary, if available.
In general, on Linux, any system capable of compiling a SwiftPM project is going to have pkg-config installed. It’s an extremely common dependency in build toolchains, and the likelihood is that if your project wants to link against anything more esoteric than libc and the various Swift shared libraries pkg-config will have been installed anyway. If for some reason pkg-config is not present on the $PATH on such a system, that is likely a signal that pkg-config is not a good tool to use to populate the linker and compile flags *anyway*.
Given that pkg-config is so widely deployed in Linux, it seems senseless to carry a reimplementation of that tool in SwiftPM. This is doubly true given that the pkg-config binary has a number of compile-time configurations applied by the system builder that give it special knowledge of the system’s default linker and include search paths. SwiftPM’s lack of that knowledge is what led to SR-6317, and rather than attempt to add complexity to SwiftPM’s implementation we would probably be better served by simply deferring to the implementation on the system that knows what to do best.
My proposal is to replace the current implementation with one that shells out to pkg-config. If pkg-config cannot be found, then we simply fall back to the rest of the user configuration, exactly as if there were no pkg-config support at all.
Daniel and I talked a bit about this issue and we think it is best to drop
the current implementation of the pkg-config parser and always shell out.
In future, we would want to move towards a model where the pkg-config is
also expressed as dependencies in llbuild (using the llbuild APIs). This
would give us a lot of free things like caching, dependency management etc.
Moving towards that model will be a lot of work so right now lets just
implement the shelling out without any persistent cache. We can think about
the temporary cache if it seriously impacts the build performance.
···
On Thu, Nov 30, 2017 at 3:42 AM, Cory Benfield via swift-build-dev < swift-build-dev@swift.org> wrote:
Hey all,
Over the past few weeks I’ve encountered some subtle issues with SwiftPM
and the system pkg-config tool disagreeing over what compiler and linker
flags should be provided on a given Linux system, as shown in SR-6317 ( https://bugs.swift.org/browse/SR-6317\). After some investigation on my
part, I’d like to propose that we should replace SwiftPM’s hand-rolled
pkgconfig parsing logic (in PkgConfig.swift) with a tool that simply shells
out to the system pkg-config binary, if available.
In general, on Linux, any system capable of compiling a SwiftPM project is
going to have pkg-config installed. It’s an extremely common dependency in
build toolchains, and the likelihood is that if your project wants to link
against anything more esoteric than libc and the various Swift shared
libraries pkg-config will have been installed anyway. If for some reason
pkg-config is not present on the $PATH on such a system, that is likely a
signal that pkg-config is not a good tool to use to populate the linker and
compile flags *anyway*.
Given that pkg-config is so widely deployed in Linux, it seems senseless
to carry a reimplementation of that tool in SwiftPM. This is doubly true
given that the pkg-config binary has a number of compile-time
configurations applied by the system builder that give it special knowledge
of the system’s default linker and include search paths. SwiftPM’s lack of
that knowledge is what led to SR-6317, and rather than attempt to add
complexity to SwiftPM’s implementation we would probably be better served
by simply deferring to the implementation on the system that knows what to
do best.
My proposal is to replace the current implementation with one that shells
out to pkg-config. If pkg-config cannot be found, then we simply fall back
to the rest of the user configuration, exactly as if there were no
pkg-config support at all.
I've been consistently getting a warning on Heroku when building a Vapor app. I dunno if this is related, or if the issue has to do with Vapor's configuration (I don't have any experience with pkg-config).
warning: error while trying to use pkgConfig flags for CTLS: couldNotFindConfigFile
But a definite +1 if it would fix this warning.
···
On Nov 30, 2017, at 5:44 AM, Johannes Weiß via swift-build-dev <swift-build-dev@swift.org> wrote:
+1, we should use the right tool for the right job. Happy to still have the fallback code in SwiftPM.
On 30 Nov 2017, at 11:42 am, Cory Benfield <cbenfield@apple.com> wrote:
Hey all,
Over the past few weeks I’ve encountered some subtle issues with SwiftPM and the system pkg-config tool disagreeing over what compiler and linker flags should be provided on a given Linux system, as shown in SR-6317 (https://bugs.swift.org/browse/SR-6317\). After some investigation on my part, I’d like to propose that we should replace SwiftPM’s hand-rolled pkgconfig parsing logic (in PkgConfig.swift) with a tool that simply shells out to the system pkg-config binary, if available.
In general, on Linux, any system capable of compiling a SwiftPM project is going to have pkg-config installed. It’s an extremely common dependency in build toolchains, and the likelihood is that if your project wants to link against anything more esoteric than libc and the various Swift shared libraries pkg-config will have been installed anyway. If for some reason pkg-config is not present on the $PATH on such a system, that is likely a signal that pkg-config is not a good tool to use to populate the linker and compile flags *anyway*.
Given that pkg-config is so widely deployed in Linux, it seems senseless to carry a reimplementation of that tool in SwiftPM. This is doubly true given that the pkg-config binary has a number of compile-time configurations applied by the system builder that give it special knowledge of the system’s default linker and include search paths. SwiftPM’s lack of that knowledge is what led to SR-6317, and rather than attempt to add complexity to SwiftPM’s implementation we would probably be better served by simply deferring to the implementation on the system that knows what to do best.
My proposal is to replace the current implementation with one that shells out to pkg-config. If pkg-config cannot be found, then we simply fall back to the rest of the user configuration, exactly as if there were no pkg-config support at all.
Assuming ctls is this: GitHub - vapor-community/ctls: LibreSSL / OpenSSL module map for Swift, then this warning is because ctls has a pkgConfig directive that says “ctls”, but there is no ctls.pc file on the system. That’s hardly a surprise, given that ctls is a pure Swift package.
Step 1 in fixing this would be to get ctls change their .pc file from macos.pc to ctls.pc, but I don’t think that’s enough because the .pc file would still not be on the PKG_CONFIG_PATH. I don’t think the .pc file in ctls works at all, and they should just remove it and have the pkgConfig directive call for “openssl” instead. That will work fine with Homebrew and will achieve the result they’re looking for.
Cory
···
On 30 Nov 2017, at 17:12, Joe DeCapo <snoogansbc@gmail.com> wrote:
I've been consistently getting a warning on Heroku when building a Vapor app. I dunno if this is related, or if the issue has to do with Vapor's configuration (I don't have any experience with pkg-config).
warning: error while trying to use pkgConfig flags for CTLS: couldNotFindConfigFile
Removing the current pkg-config parser entirely could break compatibility for existing users who don't have pkg-config installed. It seems like it would be better to leave the old pkg-config parser in and just not use it for packages whose tools-version is SwiftPM 5 or later.
- Rick
···
On Nov 30, 2017, at 12:28 PM, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org> wrote:
Hey Cory,
Daniel and I talked a bit about this issue and we think it is best to drop the current implementation of the pkg-config parser and always shell out. In future, we would want to move towards a model where the pkg-config is also expressed as dependencies in llbuild (using the llbuild APIs). This would give us a lot of free things like caching, dependency management etc. Moving towards that model will be a lot of work so right now lets just implement the shelling out without any persistent cache. We can think about the temporary cache if it seriously impacts the build performance.
On Thu, Nov 30, 2017 at 3:42 AM, Cory Benfield via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Hey all,
Over the past few weeks I’ve encountered some subtle issues with SwiftPM and the system pkg-config tool disagreeing over what compiler and linker flags should be provided on a given Linux system, as shown in SR-6317 (https://bugs.swift.org/browse/SR-6317\). After some investigation on my part, I’d like to propose that we should replace SwiftPM’s hand-rolled pkgconfig parsing logic (in PkgConfig.swift) with a tool that simply shells out to the system pkg-config binary, if available.
In general, on Linux, any system capable of compiling a SwiftPM project is going to have pkg-config installed. It’s an extremely common dependency in build toolchains, and the likelihood is that if your project wants to link against anything more esoteric than libc and the various Swift shared libraries pkg-config will have been installed anyway. If for some reason pkg-config is not present on the $PATH on such a system, that is likely a signal that pkg-config is not a good tool to use to populate the linker and compile flags *anyway*.
Given that pkg-config is so widely deployed in Linux, it seems senseless to carry a reimplementation of that tool in SwiftPM. This is doubly true given that the pkg-config binary has a number of compile-time configurations applied by the system builder that give it special knowledge of the system’s default linker and include search paths. SwiftPM’s lack of that knowledge is what led to SR-6317, and rather than attempt to add complexity to SwiftPM’s implementation we would probably be better served by simply deferring to the implementation on the system that knows what to do best.
My proposal is to replace the current implementation with one that shells out to pkg-config. If pkg-config cannot be found, then we simply fall back to the rest of the user configuration, exactly as if there were no pkg-config support at all.
Thanks for the info! I hadn't really looked into it much since the app is working fine otherwise, but the warning still bugged me. I'll file an issue on GitHub with your feedback.
···
On Nov 30, 2017, at 11:45 AM, Cory Benfield <cbenfield@apple.com> wrote:
On 30 Nov 2017, at 17:12, Joe DeCapo <snoogansbc@gmail.com <mailto:snoogansbc@gmail.com>> wrote:
I've been consistently getting a warning on Heroku when building a Vapor app. I dunno if this is related, or if the issue has to do with Vapor's configuration (I don't have any experience with pkg-config).
warning: error while trying to use pkgConfig flags for CTLS: couldNotFindConfigFile
But a definite +1 if it would fix this warning.
Assuming ctls is this: GitHub - vapor-community/ctls: LibreSSL / OpenSSL module map for Swift, then this warning is because ctls has a pkgConfig directive that says “ctls”, but there is no ctls.pc file on the system. That’s hardly a surprise, given that ctls is a pure Swift package.
Step 1 in fixing this would be to get ctls change their .pc file from macos.pc to ctls.pc, but I don’t think that’s enough because the .pc file would still not be on the PKG_CONFIG_PATH. I don’t think the .pc file in ctls works at all, and they should just remove it and have the pkgConfig directive call for “openssl” instead. That will work fine with Homebrew and will achieve the result they’re looking for.