Draft Proposal SwiftPM System Module Search Paths


(Max Howell) #1

SwiftPM System Module Search Paths
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
Author: Max Howell <https://github.com/mxcl>
Status: Awaiting review
Review manager: Anders Bertelrud
<>Introduction
Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map file.

The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed.

<>Motivation
The current implementation of system module packages have a number of problems:

Install locations vary across platforms and modulemap files require absolute paths
/usr/lib:/usr/local/lib is not always a sufficient -L search path
/usr/include:/usr/local/include is not always a sufficient -I C compiler search path
Installing the system library is left up to the end-user to figure out
For example to import a module map representing the GTK library, the include search path must be supplemented with -I/usr/include/gtk so that a number of includes in the gtk.h header can be sourced for the complete modular definition of GTK.

For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for this system package is libgtk-3-0-dev which is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than /usr..modulemap files must specify headers with absolute paths. The standard we encourage with modulemaps is for the headers to be specified with an assumed prefix of /usr, but you will not find eg. jpeglib.h at /usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<>Proposed Solution
We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool so that it can query pkg-config for the missing path and flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool to identify when the system package is not installed to a /usr and in such a case preprocess the modulemap changing the prefix it uses.

We propose that Package.swift is supplemented with metadata that provides the package-install-name for specific platforms.

<>Detailed Design
<>Solving Path/Flags Issues
Some of our problems can be solved by using the cross platform tool: pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

Its install location
Supplementary C-flags that should be used when building against this library
If SwiftPM used the .pc file that comes with packages, this solves problems 1 through 3.

Of the tickets we currently have open describing issues using Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-config for by parsing the .modulemap file in the Swift package.

<>Providing Package Install Names
Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
)
Thus, in the event of build failure for modules that depend on this package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev
Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own platforms. It also means that if a specific packager requires additional parameters, they can be added on a per enum basis.

<>Impact on Existing Code
There will be no impact on existing code as this feature simply improves an existing feature making new code possible.

<>Alternatives Considered
A clear alternative is allowing additional flags to be specified in a system-module package’s Package.swift.

However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in Package.swift, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from being able to understand and control the build ensuring reliability and preventing “Dependency Hell”.

<>Future Directions
The build system could be made more reliable by having the specific packager provide the information that this proposal garners from pkg-config. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.

If this was done pkg-config could become just one option for providing this data, and be used only as a fallback.

We could add an additional flag so that deployment of Swift Packages could be made simpler and system dependencies be installed automatically.


(Colin Barrett) #2

Hi Max,

Thanks for sending out this proposal. I admit I’m somewhat ignorant of the current state of SwiftPM so feel free to point me at existing resources if I’m missing basic things.

You don’t cover it in your proposal, but does this mean pkg-config is now a dependency for SwiftPM? I may be mistaken but it doesn’t seem as though pkg-config is installed on the base OS X system. (This leads me to assume it’s an optional dependency.)

Your proposal suggests the set of supported package mangers will be closed. What about folks using new, or non-standard package managers? What about packages that are installed system-wide without a package manager (e.g. with ./configure && make && make install)?

This may be covered elsewhere, but about the case of a C package that is normally system-wide but is currently being hacked on by the user? (For example to while debugging it, or writing bindings for it.)

-Colin

···

On Mar 23, 2016, at 2:10 PM, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

SwiftPM System Module Search Paths
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
Author: Max Howell <https://github.com/mxcl>
Status: Awaiting review
Review manager: Anders Bertelrud
<>Introduction
Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map file.

The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed.

<>Motivation
The current implementation of system module packages have a number of problems:

Install locations vary across platforms and modulemap files require absolute paths
/usr/lib:/usr/local/lib is not always a sufficient -L search path
/usr/include:/usr/local/include is not always a sufficient -I C compiler search path
Installing the system library is left up to the end-user to figure out
For example to import a module map representing the GTK library, the include search path must be supplemented with -I/usr/include/gtk so that a number of includes in the gtk.h header can be sourced for the complete modular definition of GTK.

For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for this system package is libgtk-3-0-dev which is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than /usr..modulemap files must specify headers with absolute paths. The standard we encourage with modulemaps is for the headers to be specified with an assumed prefix of /usr, but you will not find eg. jpeglib.h at /usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<>Proposed Solution
We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool so that it can query pkg-config for the missing path and flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool to identify when the system package is not installed to a /usr and in such a case preprocess the modulemap changing the prefix it uses.

We propose that Package.swift is supplemented with metadata that provides the package-install-name for specific platforms.

<>Detailed Design
<>Solving Path/Flags Issues
Some of our problems can be solved by using the cross platform tool: pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

Its install location
Supplementary C-flags that should be used when building against this library
If SwiftPM used the .pc file that comes with packages, this solves problems 1 through 3.

Of the tickets we currently have open describing issues using Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-config for by parsing the .modulemap file in the Swift package.

<>Providing Package Install Names
Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
)
Thus, in the event of build failure for modules that depend on this package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev
Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own platforms. It also means that if a specific packager requires additional parameters, they can be added on a per enum basis.

<>Impact on Existing Code
There will be no impact on existing code as this feature simply improves an existing feature making new code possible.

<>Alternatives Considered
A clear alternative is allowing additional flags to be specified in a system-module package’s Package.swift.

However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in Package.swift, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from being able to understand and control the build ensuring reliability and preventing “Dependency Hell”.

<>Future Directions
The build system could be made more reliable by having the specific packager provide the information that this proposal garners from pkg-config. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.

If this was done pkg-config could become just one option for providing this data, and be used only as a fallback.

We could add an additional flag so that deployment of Swift Packages could be made simpler and system dependencies be installed automatically.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Ankit Agarwal) #3

It is a convention to name the .pc file after the library link-name, so
we can determine which .pc file to ask pkg-config for by parsing the
.modulemap file in the Swift package.

what about the cases where .pc file doesn't matches the link-name from
modulemap for eg : gtk+2 or 3 has these link-names: `link "gtk-2.0"`, `link
"gtk-3.0"` and .pc files are `gtk+-2.0.pc`, `gtk+-3.0.pc`

One option could be an optional in Package -> `pkgconfig: "gtk+-2.0"`

···

----

Probably not in scope of this proposal, I noticed that pkg-config can give
versions of the system libs, would it be a good idea for user to mention a
version range of system lib while creating the modulemap wrapper package.

On Wed, Mar 23, 2016 at 11:40 PM, Max Howell via swift-evolution < swift-evolution@swift.org> wrote:

SwiftPM System Module Search Paths

   - Proposal: SE-NNNN
   <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
   - Author: Max Howell <https://github.com/mxcl>
   - Status: Awaiting review
   - Review manager: Anders Bertelrud

Introduction

Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map
file.

The current system for using these module-map files with SwiftPM works,
but with a number of caveats that must be addressed.
Motivation

The current implementation of system module packages have a number of
problems:

   1. Install locations vary across platforms and modulemap files require
   absolute paths
   2. /usr/lib:/usr/local/lib is not always a sufficient -L search path
   3. /usr/include:/usr/local/include is not always a sufficient -I C
   compiler search path
   4. Installing the system library is left up to the end-user to figure
   out

For example to import a module map representing the GTK library, the
include search path must be supplemented with -I/usr/include/gtk so that
a number of includes in the gtk.h header can be sourced for the complete
modular definition of GTK.

For example to import a module map representing the GTK library a user
must first have a copy of GTK and its headers installed. On Debian based
systems the install name for this system package is libgtk-3-0-dev which
is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than
/usr..modulemap files must specify headers with absolute paths. The
standard we encourage with modulemaps is for the headers to be specified
with an assumed prefix of /usr, but you will not find eg. jpeglib.h at
/usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.
Proposed Solution

We propose that SwiftPM gains the ability to use the cross-platform
pkg-config tool so that it can query pkg-config for the missing path and
flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform
pkg-config tool to identify when the system package is not installed to a
/usr and in such a case preprocess the modulemap changing the prefix it
uses.

We propose that Package.swift is supplemented with metadata that provides
the package-install-name for specific platforms.
Detailed DesignSolving Path/Flags Issues

Some of our problems can be solved by using the cross platform tool:
pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

   1. Its install location
   2. Supplementary C-flags that should be used when building against
   this library

If SwiftPM used the .pc file that comes with packages, this solves
problems 1 through 3.

Of the tickets we currently have open describing issues using
Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so
we can determine which .pc file to ask pkg-config for by parsing the
.modulemap file in the Swift package.
Providing Package Install Names

Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
)

Thus, in the event of build failure for modules that depend on this
package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev

Since the syntax to provide this information uses an explicit enum we can
add code for each enum to detect which system packagers should be
recommended. The community will need to write the code for their own
platforms. It also means that if a specific packager requires additional
parameters, they can be added on a per enum basis.
Impact on Existing Code

There will be no impact on existing code as this feature simply improves
an existing feature making new code possible.
Alternatives Considered

A clear alternative is allowing additional flags to be specified in a
system-module package’s Package.swift.

However since these paths and flags will vary by platform this would
because a large matrix that is quite a maintenance burden. Really this
information is recorded already, in the system package itself, and in fact
almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in
Package.swift, this allows packages too much power to break a large
dependency graph with bad compiles. The only entity that understands the
whole graph and can manage the build without breakage is SwiftPM, and
allowing packages themselves to add arbitrary flags prevents SwiftPM from
being able to understand and control the build ensuring reliability and
preventing “Dependency Hell”.
Future Directions

The build system could be made more reliable by having the specific
packager provide the information that this proposal garners from
pkg-config. For example, Homebrew installs everything into independent
directories, using these directories instead of more general POSIX search
paths means there is no danger of edge-case search path collisions and the
wrong libraries being picked up.

If this was done pkg-config could become just one option for providing
this data, and be used only as a fallback.
------------------------------

We could add an additional flag so that deployment of Swift Packages could
be made simpler and system dependencies be installed automatically.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Ankit


(Max Howell) #4

Thanks for sending out this proposal. I admit I’m somewhat ignorant of the current state of SwiftPM so feel free to point me at existing resources if I’m missing basic things.

You don’t cover it in your proposal, but does this mean pkg-config is now a dependency for SwiftPM? I may be mistaken but it doesn’t seem as though pkg-config is installed on the base OS X system. (This leads me to assume it’s an optional dependency.)

This is a good point, I’ll revise the proposal.

My intentions were: if available, use it, if not, then things may not build. Long term we could parse pkg-config files ourselves since the format is very basic which would be preferable since “your package graph *may* or *may not* build” is hardly fun.

Your proposal suggests the set of supported package mangers will be closed. What about folks using new, or non-standard package managers?

They can add their package manager to the supported list via a Pull Request.

What about packages that are installed system-wide without a package manager (e.g. with ./configure && make && make install)?

Such packages should come with a .pc file.

This may be covered elsewhere, but about the case of a C package that is normally system-wide but is currently being hacked on by the user? (For example to while debugging it, or writing bindings for it.)

There is no convenient way to do this at this time, but proposals are welcome. It *is* possible currently, you can make a local module map package that encapsulates the C library, or if it is simple enough our C-target support may be sufficient.


(Max Howell) #5

Our decision here was that if you have CFoo then CFoo2 is the name of the package for the major version bump to 2.

The rationale for this is: this is how it basically works in the C library packaging system currently.

Major version bumps are much rarer and the approach taken by system packagers has been to make entirely new packages for them. Even the .pc files are versioned thus, at least, this is all I’ve ever seen.

It’s not clean, but we’re building on top of what is there.

···

On Mar 26, 2016, at 12:15 PM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-config for by parsing the .modulemap file in the Swift package.

what about the cases where .pc file doesn't matches the link-name from modulemap for eg : gtk+2 or 3 has these link-names: `link "gtk-2.0"`, `link "gtk-3.0"` and .pc files are `gtk+-2.0.pc`, `gtk+-3.0.pc`

One option could be an optional in Package -> `pkgconfig: "gtk+-2.0"`

----

Probably not in scope of this proposal, I noticed that pkg-config can give versions of the system libs, would it be a good idea for user to mention a version range of system lib while creating the modulemap wrapper package.


(Colin Barrett) #6

Thanks for sending out this proposal. I admit I’m somewhat ignorant of the current state of SwiftPM so feel free to point me at existing resources if I’m missing basic things.

You don’t cover it in your proposal, but does this mean pkg-config is now a dependency for SwiftPM? I may be mistaken but it doesn’t seem as though pkg-config is installed on the base OS X system. (This leads me to assume it’s an optional dependency.)

This is a good point, I’ll revise the proposal.

My intentions were: if available, use it, if not, then things may not build. Long term we could parse pkg-config files ourselves since the format is very basic which would be preferable since “your package graph *may* or *may not* build” is hardly fun

Great! Myself, I would just parse the pc files straight away. Avoiding dependencies seems highly desirable for something this low in the stack.

Your proposal suggests the set of supported package mangers will be closed. What about folks using new, or non-standard package managers?

They can add their package manager to the supported list via a Pull Request.

This is a surprising answer! Just to be clear―what functionality, specifically, is this enabling?

What about packages that are installed system-wide without a package manager (e.g. with ./configure && make && make install)?

Such packages should come with a .pc file.

This may be covered elsewhere, but about the case of a C package that is normally system-wide but is currently being hacked on by the user? (For example to while debugging it, or writing bindings for it.)

There is no convenient way to do this at this time, but proposals are welcome. It *is* possible currently, you can make a local module map package that encapsulates the C library, or if it is simple enough our C-target support may be sufficient

No proposal here, but this is more common than you'd think―it merits at least an FAQ in the final proposal.

Thanks Max!
-Colin

···

On Mar 24, 2016, at 5:55 PM, Max Howell <max.howell@apple.com> wrote:


(Max Howell) #7

I have updated the proposal with everyone’s feedback:

SwiftPM System Module Search Paths
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
Author: Max Howell <https://github.com/mxcl>
Status: Awaiting review
Review manager: Anders Bertelrud
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#introduction>Introduction

Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map file.

The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#motivation>Motivation

The current implementation of system module packages have a number of problems:

Install locations vary across platforms and modulemap files require absolute paths
/usr/lib:/usr/local/lib is not always a sufficient -L search path
/usr/include:/usr/local/include is not always a sufficient -I C compiler search path
Installing the system library is left up to the end-user to figure out
For example to import a module map representing the GTK library, the include search path must be supplemented with -I/usr/include/gtk so that a number of includes in the gtk.h header can be sourced for the complete modular definition of GTK.

For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for this system package is libgtk-3-0-dev which is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than /usr. .modulemap files must specify headers with absolute paths. The standard we encourage with modulemaps is for the headers to be specified with an assumed prefix of /usr, but you will not find eg. jpeglib.h at /usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#proposed-solution>Proposed Solution

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool so that it can query pkg-config for the missing path and flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool to identify when the system package is not installed to a /usr and in such a case preprocess the modulemap changing the prefix it uses.

We propose that Package.swift is supplemented with metadata that provides the package-install-name for specific platforms.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#detailed-design>Detailed Design

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#solving-pathflags-issues>Solving Path/Flags Issues

Some of our problems can be solved by using the cross platform tool: pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

Its install location
Supplementary C-flags that should be used when compiling against this library
Supplementary C-flags that should be used when linking against this library
If SwiftPM used the .pc file that comes with packages, this solves problems 1 through 3.

Of the tickets we currently have open describing issues using Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-configfor by parsing the .modulemap file in the Swift package. However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will make it possible to specify the .pc file name in Package.swift.

pkg-config is not currently a dependency of the Swift toolchain, and thus to avoid depending on it we will schedule work to interpret .pc files without requiring pkg-config to be installed. The file format for .pc files is simple and standard so despite reinventing the wheel, this is a low risk choice.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#providing-package-install-names>Providing Package Install Names

Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
          .PkgConfig("foo.pc"),
)
Thus, in the event of build failure for modules that depend on this package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev
Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own platforms. It also means that if a specific packager requires additional parameters, they can be added on a per enum basis.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#install-names-are-not-standard>Install-names are not standard

apt is used across multiple distirbutions and the install-names for tools vary. Even for the same distribution install-names may vary across releases (eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on ocassion at finer granularity.

We will not add explicit handling for this, but one can imagine the enums for different system packagers could be supplemented in a backwards compatible way to provide specific handling as real-world uses emerge, eg:

case Apt(installName: String)

// …could be adapted to:

struct Debian: Linux {}
struct Ubuntu: Debian {
    enum Variant {
        case Gubuntu
        case Kubuntu(Version)
    }
    enum Version {
        case v1510
        case v1504
    }
}
case Apt(installName: String, distribution: Linux? = nil)
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#impact-on-existing-code>Impact on Existing Code

There will be no impact on existing code as this feature simply improves an existing feature making new code possible.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#alternatives-considered>Alternatives Considered

A clear alternative is allowing additional flags to be specified in a system-module package’s Package.swift.

However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in Package.swift, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from being able to understand and control the build ensuring reliability and preventing “Dependency Hell”.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#unsolved-problems>Unsolved Problems

Some (usually more legacy) C libraries do not provide .pc files instead they may provide a tool named eg. foo-configthat can be queried for compile and link flags. We do not yet support these tools, and would prefer to take a wait and see approach to determine how important supporting them may be.

Some libraries on OS X do not come with .pc files. Again we'd like to see which libraries are affected before potentially offering a solution here.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#future-directions>Future Directions

The build system could be made more reliable by having the specific packager provide the information that this proposal garners from pkg-config. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.

If this was done pkg-config could become just one option for providing this data, and be used only as a fallback.

We do not wish to provide a flag to automatically install dependencies via the system packager. We feel this opens us up to security implications beyond the scope of this tool.

Instead we can provide JSON output that can be parsed and executed by some other tooling developed outside of Apple.


(Ankit Agarwal) #8

:+1:

···

On Friday 1 April 2016, Max Howell <max.howell@apple.com> wrote:

I have updated the proposal with everyone’s feedback:

SwiftPM System Module Search Paths

   - Proposal: SE-NNNN
   <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
   - Author: Max Howell <https://github.com/mxcl>
   - Status: *Awaiting review*
   - Review manager: Anders Bertelrud

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#introduction>
Introduction

Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map
file.

The current system for using these module-map files with SwiftPM works,
but with a number of caveats that must be addressed.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#motivation>
Motivation

The current implementation of system module packages have a number of
problems:

   1. Install locations vary across platforms and modulemap files require
   absolute paths
   2. /usr/lib:/usr/local/lib is not always a sufficient -L search path
   3. /usr/include:/usr/local/include is not always a sufficient -I C
   compiler search path
   4. Installing the system library is left up to the end-user to figure
   out

For example to import a module map representing the GTK library, the
include search path must be supplemented with -I/usr/include/gtk so that
a number of includes in the gtk.h header can be sourced for the complete
modular definition of GTK.

For example to import a module map representing the GTK library a user
must first have a copy of GTK and its headers installed. On Debian based
systems the install name for this system package is libgtk-3-0-dev which
is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than
/usr. .modulemap files must specify headers with absolute paths. The
standard we encourage with modulemaps is for the headers to be specified
with an assumed prefix of /usr, but you will not find eg. jpeglib.h at
/usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#proposed-solution>Proposed
Solution

We propose that SwiftPM gains the ability to use the cross-platform
pkg-config tool so that it can query pkg-config for the missing path and
flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform
pkg-config tool to identify when the system package is not installed to a
/usr and in such a case preprocess the modulemap changing the prefix it
uses.

We propose that Package.swift is supplemented with metadata that provides
the package-install-name for specific platforms.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#detailed-design>Detailed
Design
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#solving-pathflags-issues>Solving
Path/Flags Issues

Some of our problems can be solved by using the cross platform tool:
pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

   1. Its install location
   2. Supplementary C-flags that should be used when compiling against
   this library
   3. Supplementary C-flags that should be used when linking against this
   library

If SwiftPM used the .pc file that comes with packages, this solves
problems 1 through 3.

Of the tickets we currently have open describing issues using
Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so
we can determine which .pc file to ask pkg-configfor by parsing the
.modulemap file in the Swift package. However sometimes this is not true,
(eg. GTK-3 on Ubuntu), so we will make it possible to specify the .pc file
name in Package.swift.

pkg-config is not currently a dependency of the Swift toolchain, and thus
to avoid depending on it we will schedule work to interpret .pc files
without requiring pkg-config to be installed. The file format for .pc files
is simple and standard so despite reinventing the wheel, this is a low risk
choice.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#providing-package-install-names>Providing
Package Install Names

Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
          .PkgConfig("foo.pc"),
)

Thus, in the event of build failure for modules that depend on this
package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev

Since the syntax to provide this information uses an explicit enum we can
add code for each enum to detect which system packagers should be
recommended. The community will need to write the code for their own
platforms. It also means that if a specific packager requires additional
parameters, they can be added on a per enum basis.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#install-names-are-not-standard>Install-names
are not standard

apt is used across multiple distirbutions and the install-names for tools
vary. Even for the same distribution install-names may vary across releases
(eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on ocassion at finer
granularity.

We will not add explicit handling for this, but one can imagine the enums
for different system packagers could be supplemented in a backwards
compatible way to provide specific handling as real-world uses emerge, eg:

case Apt(installName: String)
// …could be adapted to:
struct Debian: Linux {}struct Ubuntu: Debian {
    enum Variant {
        case Gubuntu
        case Kubuntu(Version)
    }
    enum Version {
        case v1510
        case v1504
    }
}case Apt(installName: String, distribution: Linux? = nil)

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#impact-on-existing-code>Impact
on Existing Code

There will be no impact on existing code as this feature simply improves
an existing feature making new code possible.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#alternatives-considered>Alternatives
Considered

A clear alternative is allowing additional flags to be specified in a
system-module package’s Package.swift.

However since these paths and flags will vary by platform this would
because a large matrix that is quite a maintenance burden. Really this
information is recorded already, in the system package itself, and in fact
almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in
Package.swift, this allows packages too much power to break a large
dependency graph with bad compiles. The only entity that understands the
whole graph and can manage the build without breakage is SwiftPM, and
allowing packages themselves to add arbitrary flags prevents SwiftPM from
being able to understand and control the build ensuring reliability and
preventing “Dependency Hell”.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#unsolved-problems>Unsolved
Problems

Some (usually more legacy) C libraries do not provide .pc files instead
they may provide a tool named eg. foo-configthat can be queried for
compile and link flags. We do not yet support these tools, and would prefer
to take a wait and see approach to determine how important supporting them
may be.

Some libraries on OS X do not come with .pc files. Again we'd like to see
which libraries are affected before potentially offering a solution here.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#future-directions>Future
Directions

The build system could be made more reliable by having the specific
packager provide the information that this proposal garners from
pkg-config. For example, Homebrew installs everything into independent
directories, using these directories instead of more general POSIX search
paths means there is no danger of edge-case search path collisions and the
wrong libraries being picked up.

If this was done pkg-config could become just one option for providing
this data, and be used only as a fallback.
------------------------------

We do not wish to provide a flag to automatically install dependencies via
the system packager. We feel this opens us up to security implications
beyond the scope of this tool.
Instead we can provide JSON output that can be parsed and executed by some
other tooling developed outside of Apple.

--
Ankit


(Daniel Dunbar) #9

Hi Max,

The proposal refers to "the pkg-config specification", can you add a link to that? In particular, I am curious how SwiftPM will know where to look for those files.

- Daniel

···

On Mar 31, 2016, at 4:04 PM, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

I have updated the proposal with everyone’s feedback:

SwiftPM System Module Search Paths

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
Author: Max Howell <https://github.com/mxcl>
Status: Awaiting review
Review manager: Anders Bertelrud
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#introduction>Introduction

Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map file.

The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#motivation>Motivation

The current implementation of system module packages have a number of problems:

Install locations vary across platforms and modulemap files require absolute paths
/usr/lib:/usr/local/lib is not always a sufficient -L search path
/usr/include:/usr/local/include is not always a sufficient -I C compiler search path
Installing the system library is left up to the end-user to figure out
For example to import a module map representing the GTK library, the include search path must be supplemented with -I/usr/include/gtk so that a number of includes in the gtk.h header can be sourced for the complete modular definition of GTK.

For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for this system package is libgtk-3-0-dev which is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than /usr. .modulemap files must specify headers with absolute paths. The standard we encourage with modulemaps is for the headers to be specified with an assumed prefix of /usr, but you will not find eg. jpeglib.h at /usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#proposed-solution>Proposed Solution

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool so that it can query pkg-config for the missing path and flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool to identify when the system package is not installed to a /usr and in such a case preprocess the modulemap changing the prefix it uses.

We propose that Package.swift is supplemented with metadata that provides the package-install-name for specific platforms.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#detailed-design>Detailed Design

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#solving-pathflags-issues>Solving Path/Flags Issues

Some of our problems can be solved by using the cross platform tool: pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

Its install location
Supplementary C-flags that should be used when compiling against this library
Supplementary C-flags that should be used when linking against this library
If SwiftPM used the .pc file that comes with packages, this solves problems 1 through 3.

Of the tickets we currently have open describing issues using Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-configfor by parsing the .modulemap file in the Swift package. However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will make it possible to specify the .pc file name in Package.swift.

pkg-config is not currently a dependency of the Swift toolchain, and thus to avoid depending on it we will schedule work to interpret .pc files without requiring pkg-config to be installed. The file format for .pc files is simple and standard so despite reinventing the wheel, this is a low risk choice.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#providing-package-install-names>Providing Package Install Names

Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
          .PkgConfig("foo.pc"),
)
Thus, in the event of build failure for modules that depend on this package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev
Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own platforms. It also means that if a specific packager requires additional parameters, they can be added on a per enum basis.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#install-names-are-not-standard>Install-names are not standard

apt is used across multiple distirbutions and the install-names for tools vary. Even for the same distribution install-names may vary across releases (eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on ocassion at finer granularity.

We will not add explicit handling for this, but one can imagine the enums for different system packagers could be supplemented in a backwards compatible way to provide specific handling as real-world uses emerge, eg:

case Apt(installName: String)

// …could be adapted to:

struct Debian: Linux {}
struct Ubuntu: Debian {
    enum Variant {
        case Gubuntu
        case Kubuntu(Version)
    }
    enum Version {
        case v1510
        case v1504
    }
}
case Apt(installName: String, distribution: Linux? = nil)
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#impact-on-existing-code>Impact on Existing Code

There will be no impact on existing code as this feature simply improves an existing feature making new code possible.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#alternatives-considered>Alternatives Considered

A clear alternative is allowing additional flags to be specified in a system-module package’s Package.swift.

However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in Package.swift, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from being able to understand and control the build ensuring reliability and preventing “Dependency Hell”.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#unsolved-problems>Unsolved Problems

Some (usually more legacy) C libraries do not provide .pc files instead they may provide a tool named eg. foo-configthat can be queried for compile and link flags. We do not yet support these tools, and would prefer to take a wait and see approach to determine how important supporting them may be.

Some libraries on OS X do not come with .pc files. Again we'd like to see which libraries are affected before potentially offering a solution here.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#future-directions>Future Directions

The build system could be made more reliable by having the specific packager provide the information that this proposal garners from pkg-config. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.

If this was done pkg-config could become just one option for providing this data, and be used only as a fallback.

We do not wish to provide a flag to automatically install dependencies via the system packager. We feel this opens us up to security implications beyond the scope of this tool.

Instead we can provide JSON output that can be parsed and executed by some other tooling developed outside of Apple.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Max Howell) #10

There is no spec, but the man page has info on what you asked

http://linux.die.net/man/1/pkg-config

···

Hi Max,

The proposal refers to "the pkg-config specification", can you add a link to that? In particular, I am curious how SwiftPM will know where to look for those files.

- Daniel

On Mar 31, 2016, at 4:04 PM, Max Howell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I have updated the proposal with everyone’s feedback:

SwiftPM System Module Search Paths
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
Author: Max Howell <https://github.com/mxcl>
Status: Awaiting review
Review manager: Anders Bertelrud
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#introduction>Introduction

Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map file.

The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#motivation>Motivation

The current implementation of system module packages have a number of problems:

Install locations vary across platforms and modulemap files require absolute paths
/usr/lib:/usr/local/lib is not always a sufficient -L search path
/usr/include:/usr/local/include is not always a sufficient -I C compiler search path
Installing the system library is left up to the end-user to figure out
For example to import a module map representing the GTK library, the include search path must be supplemented with -I/usr/include/gtk so that a number of includes in the gtk.h header can be sourced for the complete modular definition of GTK.

For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for this system package is libgtk-3-0-dev which is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than /usr. .modulemap files must specify headers with absolute paths. The standard we encourage with modulemaps is for the headers to be specified with an assumed prefix of /usr, but you will not find eg. jpeglib.h at /usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#proposed-solution>Proposed Solution

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool so that it can query pkg-config for the missing path and flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool to identify when the system package is not installed to a /usr and in such a case preprocess the modulemap changing the prefix it uses.

We propose that Package.swift is supplemented with metadata that provides the package-install-name for specific platforms.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#detailed-design>Detailed Design

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#solving-pathflags-issues>Solving Path/Flags Issues

Some of our problems can be solved by using the cross platform tool: pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

Its install location
Supplementary C-flags that should be used when compiling against this library
Supplementary C-flags that should be used when linking against this library
If SwiftPM used the .pc file that comes with packages, this solves problems 1 through 3.

Of the tickets we currently have open describing issues using Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-configfor by parsing the .modulemap file in the Swift package. However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will make it possible to specify the .pc file name in Package.swift.

pkg-config is not currently a dependency of the Swift toolchain, and thus to avoid depending on it we will schedule work to interpret .pc files without requiring pkg-config to be installed. The file format for .pc files is simple and standard so despite reinventing the wheel, this is a low risk choice.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#providing-package-install-names>Providing Package Install Names

Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
          .PkgConfig("foo.pc"),
)
Thus, in the event of build failure for modules that depend on this package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev
Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own platforms. It also means that if a specific packager requires additional parameters, they can be added on a per enum basis.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#install-names-are-not-standard>Install-names are not standard

apt is used across multiple distirbutions and the install-names for tools vary. Even for the same distribution install-names may vary across releases (eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on ocassion at finer granularity.

We will not add explicit handling for this, but one can imagine the enums for different system packagers could be supplemented in a backwards compatible way to provide specific handling as real-world uses emerge, eg:

case Apt(installName: String)

// …could be adapted to:

struct Debian: Linux {}
struct Ubuntu: Debian {
    enum Variant {
        case Gubuntu
        case Kubuntu(Version)
    }
    enum Version {
        case v1510
        case v1504
    }
}
case Apt(installName: String, distribution: Linux? = nil)
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#impact-on-existing-code>Impact on Existing Code

There will be no impact on existing code as this feature simply improves an existing feature making new code possible.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#alternatives-considered>Alternatives Considered

A clear alternative is allowing additional flags to be specified in a system-module package’s Package.swift.

However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in Package.swift, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from being able to understand and control the build ensuring reliability and preventing “Dependency Hell”.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#unsolved-problems>Unsolved Problems

Some (usually more legacy) C libraries do not provide .pc files instead they may provide a tool named eg. foo-configthat can be queried for compile and link flags. We do not yet support these tools, and would prefer to take a wait and see approach to determine how important supporting them may be.

Some libraries on OS X do not come with .pc files. Again we'd like to see which libraries are affected before potentially offering a solution here.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#future-directions>Future Directions

The build system could be made more reliable by having the specific packager provide the information that this proposal garners from pkg-config. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.

If this was done pkg-config could become just one option for providing this data, and be used only as a fallback.

We do not wish to provide a flag to automatically install dependencies via the system packager. We feel this opens us up to security implications beyond the scope of this tool.

Instead we can provide JSON output that can be parsed and executed by some other tooling developed outside of Apple.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Daniel Dunbar) #11

I don't see that information in the man page (also, I am not familiar enough with pkg-config to know how results described in that man page translate to other systems).

Specifically, that man page does not seem to document where on disk the .pc files live. How are we going to know that?

For example, one on random VM I have lying about I see .pc files here:

···

--
/# find / -name \*.pc | xargs dirname | sort | uniq
/usr/lib/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/mit-krb5
/usr/share/pkgconfig
--
and the pkg-config tool appears to be able to find results from all of those. How would we know those directory names if we don't have a dependency on the actual tool?

- Daniel

On Apr 6, 2016, at 10:43 AM, Max Howell <max.howell@apple.com> wrote:

There is no spec, but the man page has info on what you asked

http://linux.die.net/man/1/pkg-config

Hi Max,

The proposal refers to "the pkg-config specification", can you add a link to that? In particular, I am curious how SwiftPM will know where to look for those files.

- Daniel

On Mar 31, 2016, at 4:04 PM, Max Howell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I have updated the proposal with everyone’s feedback:

SwiftPM System Module Search Paths

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md>
Author: Max Howell <https://github.com/mxcl>
Status: Awaiting review
Review manager: Anders Bertelrud
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#introduction>Introduction

Swift is able to import C libraries in the same manner as Swift libraries.

For this to occur the library must be represented by a clang module-map file.

The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#motivation>Motivation

The current implementation of system module packages have a number of problems:

Install locations vary across platforms and modulemap files require absolute paths
/usr/lib:/usr/local/lib is not always a sufficient -L search path
/usr/include:/usr/local/include is not always a sufficient -I C compiler search path
Installing the system library is left up to the end-user to figure out
For example to import a module map representing the GTK library, the include search path must be supplemented with -I/usr/include/gtk so that a number of includes in the gtk.h header can be sourced for the complete modular definition of GTK.

For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for this system package is libgtk-3-0-dev which is not entirely intuitive.

For example, Homebrew and MacPorts on OS X install to prefixes other than /usr. .modulemap files must specify headers with absolute paths. The standard we encourage with modulemaps is for the headers to be specified with an assumed prefix of /usr, but you will not find eg. jpeglib.h at /usr/include/jpeglib.h if it is installed with Homebrew or MacPorts.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#proposed-solution>Proposed Solution

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool so that it can query pkg-config for the missing path and flag arguments.

We propose that SwiftPM gains the ability to use the cross-platform pkg-config tool to identify when the system package is not installed to a /usr and in such a case preprocess the modulemap changing the prefix it uses.

We propose that Package.swift is supplemented with metadata that provides the package-install-name for specific platforms.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#detailed-design>Detailed Design

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#solving-pathflags-issues>Solving Path/Flags Issues

Some of our problems can be solved by using the cross platform tool: pkg-config.

A C package can provide a pkg-config file (.pc) which describes:

Its install location
Supplementary C-flags that should be used when compiling against this library
Supplementary C-flags that should be used when linking against this library
If SwiftPM used the .pc file that comes with packages, this solves problems 1 through 3.

Of the tickets we currently have open describing issues using Swift-system-module-packages, reading the .pc file would fix all of them.

It is a convention to name the .pc file after the library link-name, so we can determine which .pc file to ask pkg-configfor by parsing the .modulemap file in the Swift package. However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will make it possible to specify the .pc file name in Package.swift.

pkg-config is not currently a dependency of the Swift toolchain, and thus to avoid depending on it we will schedule work to interpret .pc files without requiring pkg-config to be installed. The file format for .pc files is simple and standard so despite reinventing the wheel, this is a low risk choice.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#providing-package-install-names>Providing Package Install Names

Package.swift would be supplemented like so:

let package = Package(
    name: "CFoo",
    providers: .Brew(installName: "foo"),
                .Apt(installName: "libfoo-dev"),
          .PkgConfig("foo.pc"),
)
Thus, in the event of build failure for modules that depend on this package we provide additional help to the user:

error: failed to build module `bar'
note: you may need to install `foo' using your system-packager:

    apt-get install libfoo-dev
Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own platforms. It also means that if a specific packager requires additional parameters, they can be added on a per enum basis.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#install-names-are-not-standard>Install-names are not standard

apt is used across multiple distirbutions and the install-names for tools vary. Even for the same distribution install-names may vary across releases (eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on ocassion at finer granularity.

We will not add explicit handling for this, but one can imagine the enums for different system packagers could be supplemented in a backwards compatible way to provide specific handling as real-world uses emerge, eg:

case Apt(installName: String)

// …could be adapted to:

struct Debian: Linux {}
struct Ubuntu: Debian {
    enum Variant {
        case Gubuntu
        case Kubuntu(Version)
    }
    enum Version {
        case v1510
        case v1504
    }
}
case Apt(installName: String, distribution: Linux? = nil)
<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#impact-on-existing-code>Impact on Existing Code

There will be no impact on existing code as this feature simply improves an existing feature making new code possible.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#alternatives-considered>Alternatives Considered

A clear alternative is allowing additional flags to be specified in a system-module package’s Package.swift.

However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a .pc pkg-config file.

Also we do not want to allow arbitrary flags to be specified in Package.swift, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from being able to understand and control the build ensuring reliability and preventing “Dependency Hell”.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#unsolved-problems>Unsolved Problems

Some (usually more legacy) C libraries do not provide .pc files instead they may provide a tool named eg. foo-configthat can be queried for compile and link flags. We do not yet support these tools, and would prefer to take a wait and see approach to determine how important supporting them may be.

Some libraries on OS X do not come with .pc files. Again we'd like to see which libraries are affected before potentially offering a solution here.

<https://github.com/mxcl/swift-evolution/blob/system-module-search-paths/proposals/NNNN-swiftpm-system-module-search-paths.md#future-directions>Future Directions

The build system could be made more reliable by having the specific packager provide the information that this proposal garners from pkg-config. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.

If this was done pkg-config could become just one option for providing this data, and be used only as a fallback.

We do not wish to provide a flag to automatically install dependencies via the system packager. We feel this opens us up to security implications beyond the scope of this tool.

Instead we can provide JSON output that can be parsed and executed by some other tooling developed outside of Apple.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Max Howell) #12

I don't see that information in the man page (also, I am not familiar enough with pkg-config to know how results described in that man page translate to other systems).

Specifically, that man page does not seem to document where on disk the .pc files live. How are we going to know that?

For example, one on random VM I have lying about I see .pc files here:
--
/# find / -name \*.pc | xargs dirname | sort | uniq
/usr/lib/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/mit-krb5
/usr/share/pkgconfig
--
and the pkg-config tool appears to be able to find results from all of those. How would we know those directory names if we don't have a dependency on the actual tool?

There is a config file and an environment variable.

Do you need me to document them completely so that we can move forward with this?


(Daniel Dunbar) #13

No, not at all, I think the proposal makes complete sense and should be moved along.

Nevertheless, I still want to understand how it will work... I might be missing some information from the man page but I just don't see where this is described. It *looks* to me like the default search list is hard coded into the tool.

- Daniel

···

On Apr 6, 2016, at 11:11 AM, Max Howell <max.howell@apple.com> wrote:

I don't see that information in the man page (also, I am not familiar enough with pkg-config to know how results described in that man page translate to other systems).

Specifically, that man page does not seem to document where on disk the .pc files live. How are we going to know that?

For example, one on random VM I have lying about I see .pc files here:
--
/# find / -name \*.pc | xargs dirname | sort | uniq
/usr/lib/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/mit-krb5
/usr/share/pkgconfig
--
and the pkg-config tool appears to be able to find results from all of those. How would we know those directory names if we don't have a dependency on the actual tool?

There is a config file and an environment variable.

Do you need me to document them completely so that we can move forward with this?


(Max Howell) #14

I don't see that information in the man page (also, I am not familiar enough with pkg-config to know how results described in that man page translate to other systems).

Specifically, that man page does not seem to document where on disk the .pc files live. How are we going to know that?

For example, one on random VM I have lying about I see .pc files here:
--
/# find / -name \*.pc | xargs dirname | sort | uniq
/usr/lib/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/mit-krb5
/usr/share/pkgconfig
--
and the pkg-config tool appears to be able to find results from all of those. How would we know those directory names if we don't have a dependency on the actual tool?

There is a config file and an environment variable.

Do you need me to document them completely so that we can move forward with this?

No, not at all, I think the proposal makes complete sense and should be moved along.

Nevertheless, I still want to understand how it will work... I might be missing some information from the man page but I just don't see where this is described. It *looks* to me like the default search list is hard coded into the tool.

Best docs I have found are: https://people.freedesktop.org/~dbn/pkg-config-guide.html

Essentially:

The std path is /usr/lib/pkg-config and /usr/share/pkg-config, this can be supplemented by an env var PKG_CONFIG_PATH

It is common for system packagers to add more paths. Eg. brew does this, I coded it myself.

This is fine, because the additional paths can only be queried by a custom configured pkg-config, and in such cases we can just ask the custom configured pkg-config, if it’s there, there may be more paths, if it’s not then there cannot.