[Draft] Package Manager Custom Targets Layout

[Aciid (Ankit Aggarwal)]

Hi,

We would love to get some feedback on a draft proposal for defining custom target layouts in the Package Manager. This proposal allows overriding the target layout rules set by the Package Manager and simplifies some complex "magic" behaviours.

You can find the proposal at the link below. The text is also included in the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

···

--

Package Manager Custom Targets Layout
Proposal: SE-NNNN <file:///Users/ankit/NNNN-package-manager-custom-targets-layout.md>
Author: Ankit Aggarwal <https://github.com/aciidb0mb3r&gt;
Review Manager: TBD
Status: Discussion
Bug: SR-29 <Issues · apple/swift-issues · GitHub;
Introduction
This proposal enhances the Package.swift manifest APIs to support custom target layouts, and removes a convention which allowed omission of targets from the manifest.

Motivation
The Package Manager uses a convention system to infer targets structure from disk layout. This works well for most packages, which can easily adopt the conventions, and frees users from needing to update their Package.swift file every time they add or remove sources. Adopting the conventions is more difficult for some packages, however – especially existing C libraries or large projects, which would be difficult to reorganize. We intend to give users a way to make such projects into packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets and source files by inferring them automatically from disk, but they also can be confusing, overly-implicit, and difficult to debug; for example, if the user does not follow the conventions correctly which determine their targets, they may wind up with targets they don't expect, or not having targets they did expect, and either way their clients can't easily see which targets are available by looking at the Package.swift manifest. We want to retain convenience where it really matters, such as easy addition of new source files, but require explicit declarations where being explicit adds significant value. We also want to make sure that the implicit conventions we keep are straightforward and easy to remember.

Proposed solution
We propose to stop inferring targets from disk. They must be explicitly declared in the manifest file. The inference was not very useful, as targets eventually need to be declared in order to use common features such as product and target dependencies, or build settings (which are planned for Swift 4). Explicit target declarations make a package easier to understand by clients, and allow us to provide good diagnostics when the layout on disk does not match the declarations.

We propose to remove the requirement that name of a test target must have suffix "Tests". Instead, test targets will be explicitly declared as such in the manifest file.

We propose a list of pre-defined search paths for declared targets.

When a target does not declare an explicit path, these directories will be used to search for the target. The name of the directory must match the name of the target. The search will be done in order and will be case-sensitive.

Regular targets: package root, Sources, Source, src, srcs. Test targets: Tests, package root, Sources, Source, src, srcs.

It is an error if a target is found in more than one of these paths. In such cases, the path should be explicitly declared using the path property proposed below.

We propose to add a factory method testTarget to the Target class, to define test targets.

.testTarget(name: "FooTests", dependencies: ["Foo"])
We propose to add three properties to the Target class: path, sources andexclude.

path: This property defines the path to the top-level directory containing the target's sources, relative to the package root. It is not legal for this path to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The default value of this property will be nil, which means the target will be searched for in the pre-defined paths. The empty string ("") or dot (".") implies that the target's sources are directly inside the package root.

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included. This can contain directories and individual source files. Directories will be searched recursively for valid source files. Paths specified are relative to the target path.

Each source file will be represented by String type. In future, we will consider upgrading this to its own type to allow per-file build settings. The new type would conform to CustomStringConvertible, so existing declarations would continue to work (except where the strings were constructed programatically).

exclude: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative to the target path. This property has more precedence than sources property.

Note: We plan to support globbing in future, but to keep this proposal short we are not proposing it right now.

It is an error if the paths of two targets overlap (unless resolved with exclude).

// This is an error:
.target(name: "Bar", path: "Sources/Bar"),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

// This works:
.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
For C family library targets, we propose to add a publicHeadersPath property.

This property defines the path to the directory containing public headers of a C target. This path is relative to the target path and default value of this property is include. This mechanism should be further improved in the future, but there are several behaviors, such as modulemap generation, which currently depend of having only one public headers directory. We will address those issues separately in a future proposal.

All existing rules related to custom and automatic modulemap remain intact.

Remove exclude from Package class.

This property is no longer required because of the above proposed per-target exclude property.

The templates provided by the swift package init subcommand will be updated according to the above rules, so that users do not need to manually add their first target to the manifest.

Examples:
Dummy manifest containing all Swift code.
let package = Package(
    name: "SwiftyJSON",
    targets: [
        .target(
            name: "Utility",
            path: "Sources/BasicCode"
        ),

        .target(
            name: "SwiftyJSON",
            dependencies: ["Utility"],
            path: "SJ",
            sources: ["SwiftyJSON.swift"]
        ),

        .testTarget(
            name: "AllTests",
            dependencies: ["Utility", "SwiftyJSON"],
            path: "Tests",
            exclude: ["Fixtures"]
        ),
    ]
)
LibYAML
let packages = Package(
    name: "LibYAML",
    targets: [
        .target(
            name: "libyaml",
            sources: ["src"]
        )
    ]
)
Node.js http-parser
let packages = Package(
    name: "http-parser",
    targets: [
        .target(
            name: "http-parser",
            publicHeaders: ".",
            sources: ["http_parser.c"]
        )
    ]
)
swift-build-tool
let packages = Package(
    name: "llbuild",
    targets: [
        .target(
            name: "swift-build-tool",
            path: ".",
            sources: [
                "lib/Basic",
                "lib/llvm/Support",
                "lib/Core",
                "lib/BuildSystem",
                "products/swift-build-tool/swift-build-tool.cpp",
            ]
        )
    ]
)
Impact on existing code
These enhancements will be added to the version 4 manifest API, which will release with Swift 4. There will be no impact on packages using the version 3 manifest API. When packages update their minimum tools version to 4.0, they will need to update the manifest according to the changes in this proposal.

There are two flat layouts supported in Swift 3:

Source files directly in the package root.
Source files directly inside a Sources/ directory.
If packages want to continue using either of these flat layouts, they will need to explicitly set a target path to the flat directory; otherwise, a directory named after the target is expected. For example, if a package Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift
The updated manifest will look like this:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        .target(name: "Foo", path: "Sources"),
    ]
)
Alternatives considered
We considered making a more minimal change which disabled the flat layouts by default, and provided a top-level property to allow opting back in to them. This would allow us to discourage these layouts – which we would like to do before the package ecosystem grows – without needing to add a fully customizable API. However, we think the fuller API we've proposed here is fairly straightforward and provides the ability to make a number of existing projects into packages, so we think this is worth doing at this time.

[Rien]

Sounds good on a first read through.

Regards,
Rien

Site: http://balancingrock.nl
Blog: http://swiftrien.blogspot.com
Github: Balancingrock (Rien) · GitHub
Project: http://swiftfire.nl

···

On 24 Mar 2017, at 21:26, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

We would love to get some feedback on a draft proposal for defining custom target layouts in the Package Manager. This proposal allows overriding the target layout rules set by the Package Manager and simplifies some complex "magic" behaviours.

You can find the proposal at the link below. The text is also included in the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

--

Package Manager Custom Targets Layout
  • Proposal: SE-NNNN
  • Author: Ankit Aggarwal
  • Review Manager: TBD
  • Status: Discussion
  • Bug: SR-29
Introduction
This proposal enhances the Package.swift manifest APIs to support custom target layouts, and removes a convention which allowed omission of targets from the manifest.

Motivation
The Package Manager uses a convention system to infer targets structure from disk layout. This works well for most packages, which can easily adopt the conventions, and frees users from needing to update their Package.swift file every time they add or remove sources. Adopting the conventions is more difficult for some packages, however – especially existing C libraries or large projects, which would be difficult to reorganize. We intend to give users a way to make such projects into packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets and source files by inferring them automatically from disk, but they also can be confusing, overly-implicit, and difficult to debug; for example, if the user does not follow the conventions correctly which determine their targets, they may wind up with targets they don't expect, or not having targets they did expect, and either way their clients can't easily see which targets are available by looking at the Package.swift manifest. We want to retain convenience where it really matters, such as easy addition of new source files, but require explicit declarations where being explicit adds significant value. We also want to make sure that the implicit conventions we keep are straightforward and easy to remember.

Proposed solution
  • We propose to stop inferring targets from disk. They must be explicitly declared in the manifest file. The inference was not very useful, as targets eventually need to be declared in order to use common features such as product and target dependencies, or build settings (which are planned for Swift 4). Explicit target declarations make a package easier to understand by clients, and allow us to provide good diagnostics when the layout on disk does not match the declarations.

  • We propose to remove the requirement that name of a test target must have suffix "Tests". Instead, test targets will be explicitly declared as such in the manifest file.

  • We propose a list of pre-defined search paths for declared targets.

When a target does not declare an explicit path, these directories will be used to search for the target. The name of the directory must match the name of the target. The search will be done in order and will be case-sensitive.

Regular targets: package root, Sources, Source, src, srcs. Test targets: Tests, package root, Sources, Source, src, srcs.

It is an error if a target is found in more than one of these paths. In such cases, the path should be explicitly declared using the path property proposed below.

  • We propose to add a factory method testTarget to the Target class, to define test targets.

.testTarget(name: "FooTests", dependencies: ["Foo"])
  • We propose to add three properties to the Target class: path, sources andexclude.

    • path: This property defines the path to the top-level directory containing the target's sources, relative to the package root. It is not legal for this path to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The default value of this property will be nil, which means the target will be searched for in the pre-defined paths. The empty string ("") or dot (".") implies that the target's sources are directly inside the package root.

    • sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included. This can contain directories and individual source files. Directories will be searched recursively for valid source files. Paths specified are relative to the target path.

Each source file will be represented by String type. In future, we will consider upgrading this to its own type to allow per-file build settings. The new type would conform to CustomStringConvertible, so existing declarations would continue to work (except where the strings were constructed programatically).

    • exclude: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative to the target path. This property has more precedence than sources property.

Note: We plan to support globbing in future, but to keep this proposal short we are not proposing it right now.

  • It is an error if the paths of two targets overlap (unless resolved with exclude).

// This is an error:
.target(name: "Bar", path: "Sources/Bar"),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

// This works:
.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
  • For C family library targets, we propose to add a publicHeadersPath property.

This property defines the path to the directory containing public headers of a C target. This path is relative to the target path and default value of this property is include. This mechanism should be further improved in the future, but there are several behaviors, such as modulemap generation, which currently depend of having only one public headers directory. We will address those issues separately in a future proposal.

All existing rules related to custom and automatic modulemap remain intact.

  • Remove exclude from Package class.

This property is no longer required because of the above proposed per-target exclude property.

  • The templates provided by the swift package init subcommand will be updated according to the above rules, so that users do not need to manually add their first target to the manifest.

Examples:
  • Dummy manifest containing all Swift code.
let package = Package(

    name
: "SwiftyJSON",

    targets
: [

.target(

            name
: "Utility",

            path
: "Sources/BasicCode"

),

.target(

            name
: "SwiftyJSON",

            dependencies
: ["Utility"],

            path
: "SJ",

            sources
: ["SwiftyJSON.swift"]

),

.testTarget(

            name
: "AllTests",

            dependencies
: ["Utility", "SwiftyJSON"],

            path
: "Tests",

            exclude
: ["Fixtures"]

),

]
)
  • LibYAML
let packages = Package(

    name
: "LibYAML",

    targets
: [

.target(

            name
: "libyaml",

            sources
: ["src"]

)

]
)
  • Node.js http-parser
let packages = Package(

    name
: "http-parser",

    targets
: [

.target(

            name
: "http-parser",

            publicHeaders
: ".",

            sources
: ["http_parser.c"]

)

]
)
  • swift-build-tool
let packages = Package(

    name
: "llbuild",

    targets
: [

.target(

            name
: "swift-build-tool",

            path
: ".",

            sources
: [

"lib/Basic",

"lib/llvm/Support",

"lib/Core",

"lib/BuildSystem",

"products/swift-build-tool/swift-build-tool.cpp",

]

)

]
)
Impact on existing code
These enhancements will be added to the version 4 manifest API, which will release with Swift 4. There will be no impact on packages using the version 3 manifest API. When packages update their minimum tools version to 4.0, they will need to update the manifest according to the changes in this proposal.

There are two flat layouts supported in Swift 3:

  • Source files directly in the package root.
  • Source files directly inside a Sources/ directory.
If packages want to continue using either of these flat layouts, they will need to explicitly set a target path to the flat directory; otherwise, a directory named after the target is expected. For example, if a package Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift

The updated manifest will look like this:

// swift-tools-version:4.0
import PackageDescription

let package = Package(

    name
: "Foo",

    targets
: [

.target(name: "Foo", path: "Sources"),

]
)
Alternatives considered
We considered making a more minimal change which disabled the flat layouts by default, and provided a top-level property to allow opting back in to them. This would allow us to discourage these layouts – which we would like to do before the package ecosystem grows – without needing to add a fully customizable API. However, we think the fuller API we've proposed here is fairly straightforward and provides the ability to make a number of existing projects into packages, so we think this is worth doing at this time.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Aaron_Crespo]

I recently experienced these pain points when trying to support SwiftPM on
some of our libraries. I think this proposal goes a long way to alleviate
most of the those I experienced.

···

On Fri, Mar 24, 2017 at 4:26 PM, Ankit Aggarwal via swift-evolution < swift-evolution@swift.org> wrote:

Hi,

We would love to get some feedback on a draft proposal for defining custom
target layouts in the Package Manager. This proposal allows overriding the
target layout rules set by the Package Manager and simplifies some complex
"magic" behaviours.

You can find the proposal at the link below. The text is also included in
the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/
proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

--

Package Manager Custom Targets Layout

   - Proposal: SE-NNNN
   - Author: Ankit Aggarwal <https://github.com/aciidb0mb3r&gt;
   - Review Manager: TBD
   - Status: *Discussion*
   - Bug: SR-29 <Issues · apple/swift-issues · GitHub;

Introduction

This proposal enhances the Package.swift manifest APIs to support custom
target layouts, and removes a convention which allowed omission of targets
from the manifest.
Motivation

The Package Manager uses a convention system to infer targets structure
from disk layout. This works well for most packages, which can easily adopt
the conventions, and frees users from needing to update their
Package.swift file every time they add or remove sources. Adopting the
conventions is more difficult for some packages, however – especially
existing C libraries or large projects, which would be difficult to
reorganize. We intend to give users a way to make such projects into
packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets
and source files by inferring them automatically from disk, but they also
can be confusing, overly-implicit, and difficult to debug; for example, if
the user does not follow the conventions correctly which determine their
targets, they may wind up with targets they don't expect, or not having
targets they did expect, and either way their clients can't easily see
which targets are available by looking at the Package.swift manifest. We
want to retain convenience where it really matters, such as easy addition
of new source files, but require explicit declarations where being explicit
adds significant value. We also want to make sure that the implicit
conventions we keep are straightforward and easy to remember.
Proposed solution

   -

   We propose to stop inferring targets from disk. They must be
   explicitly declared in the manifest file. The inference was not very
   useful, as targets eventually need to be declared in order to use common
   features such as product and target dependencies, or build settings (which
   are planned for Swift 4). Explicit target declarations make a package
   easier to understand by clients, and allow us to provide good diagnostics
   when the layout on disk does not match the declarations.
   -

   We propose to remove the requirement that name of a test target must
   have suffix "Tests". Instead, test targets will be explicitly declared as
   such in the manifest file.
   -

   We propose a list of pre-defined search paths for declared targets.

   When a target does not declare an explicit path, these directories
   will be used to search for the target. The name of the directory must match
   the name of the target. The search will be done in order and will be
   case-sensitive.

   Regular targets: package root, Sources, Source, src, srcs. Test
   targets: Tests, package root, Sources, Source, src, srcs.

   It is an error if a target is found in more than one of these paths.
   In such cases, the path should be explicitly declared using the path
   property proposed below.
   -

   We propose to add a factory method testTarget to the Target class, to
   define test targets.

   .testTarget(name: "FooTests", dependencies: ["Foo"])

   -

   We propose to add three properties to the Target class: path, sources
    andexclude.
   -

      path: This property defines the path to the top-level directory
      containing the target's sources, relative to the package root. It is not
      legal for this path to escape the package root, i.e., values like "../Foo",
      "/Foo" are invalid. The default value of this property will be nil,
      which means the target will be searched for in the pre-defined paths. The
      empty string ("") or dot (".") implies that the target's sources are
      directly inside the package root.
      -

      sources: This property defines the source files to be included in
      the target, relative to the target path. The default value of this property
      will be an empty array, which means all valid source files found in the
      target's path will be included. This can contain directories and individual
      source files. Directories will be searched recursively for valid source
      files. Paths specified are relative to the target path.

      Each source file will be represented by String type. In future, we
      will consider upgrading this to its own type to allow per-file build
      settings. The new type would conform to CustomStringConvertible, so
      existing declarations would continue to work (except where the strings were
      constructed programatically).
      -

      exclude: This property can be used to exclude certain files and
      directories from being picked up as sources. Exclude paths are relative to
      the target path. This property has more precedence than sources
       property.

      *Note: We plan to support globbing in future, but to keep this
      proposal short we are not proposing it right now.*
      -

   It is an error if the paths of two targets overlap (unless resolved
   with exclude).

   // This is an error:.target(name: "Bar", path: "Sources/Bar"),.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
   // This works:.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

   -

   For C family library targets, we propose to add a publicHeadersPath
    property.

   This property defines the path to the directory containing public
   headers of a C target. This path is relative to the target path and default
   value of this property is include. This mechanism should be further
   improved in the future, but there are several behaviors, such as modulemap
   generation, which currently depend of having only one public headers
   directory. We will address those issues separately in a future proposal.

   *All existing rules related to custom and automatic modulemap remain
   intact.*
   -

   Remove exclude from Package class.

   This property is no longer required because of the above proposed
   per-target exclude property.
   -

   The templates provided by the swift package init subcommand will be
   updated according to the above rules, so that users do not need to manually
   add their first target to the manifest.

Examples:

   - Dummy manifest containing all Swift code.

let package = Package(
    name: "SwiftyJSON",
    targets: [
        .target(
            name: "Utility",
            path: "Sources/BasicCode"
        ),

        .target(
            name: "SwiftyJSON",
            dependencies: ["Utility"],
            path: "SJ",
            sources: ["SwiftyJSON.swift"]
        ),

        .testTarget(
            name: "AllTests",
            dependencies: ["Utility", "SwiftyJSON"],
            path: "Tests",
            exclude: ["Fixtures"]
        ),
    ])

   - LibYAML

let packages = Package(
    name: "LibYAML",
    targets: [
        .target(
            name: "libyaml",
            sources: ["src"]
        )
    ])

   - Node.js http-parser

let packages = Package(
    name: "http-parser",
    targets: [
        .target(
            name: "http-parser",
            publicHeaders: ".",
            sources: ["http_parser.c"]
        )
    ])

   - swift-build-tool

let packages = Package(
    name: "llbuild",
    targets: [
        .target(
            name: "swift-build-tool",
            path: ".",
            sources: [
                "lib/Basic",
                "lib/llvm/Support",
                "lib/Core",
                "lib/BuildSystem",
                "products/swift-build-tool/swift-build-tool.cpp",
            ]
        )
    ])

Impact on existing code

These enhancements will be added to the version 4 manifest API, which will
release with Swift 4. There will be no impact on packages using the version
3 manifest API. When packages update their minimum tools version to 4.0,
they will need to update the manifest according to the changes in this
proposal.

There are two flat layouts supported in Swift 3:

   1. Source files directly in the package root.
   2. Source files directly inside a Sources/ directory.

If packages want to continue using either of these flat layouts, they will
need to explicitly set a target path to the flat directory; otherwise, a
directory named after the target is expected. For example, if a package
Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift

The updated manifest will look like this:

// swift-tools-version:4.0import PackageDescription
let package = Package(
    name: "Foo",
    targets: [
        .target(name: "Foo", path: "Sources"),
    ])

Alternatives considered

We considered making a more minimal change which disabled the flat layouts
by default, and provided a top-level property to allow opting back in to
them. This would allow us to discourage these layouts – which we would like
to do before the package ecosystem grows – without needing to add a fully
customizable API. However, we think the fuller API we've proposed here is
fairly straightforward and provides the ability to make a number of
existing projects into packages, so we think this is worth doing at this
time.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Karl]

Hi,

I like the part of the proposal which stops inferring targets from the directory layout; I’ve always found that stuff pretty annoying.

Two points:

1) I’m not sure about listing all of the test targets together with the product targets. I feel it may be better to have them as separate Arrays.

2) I think we need a way to specify directories non-recursively. For example:

- Sources

- Foo
> - MultipleImplementations
> > - Impl1
> > - Impl2

In this case, “MultipleImplementations” contains some common definitions, but has multiple potential implementations. Impl1 and Impl2 will each implement the functionality in their own way, perhaps making use of platform-specific features or optional dependencies. Basically, I want to import “MultipleImplementations” non-recursively, then decide which subdirectory/implementation to use based on some build condition:

.Target(“Foo”, sources: [“Sources/Foo”, “[NR]Sources/Foo/MultipleImplementations”])

if someCondition {
  package.targets[0].sources.append(“Sources/Foo/MultipleImplementations/Impl1”)
else {
  package.targets[0].sources.append(“Sources/Foo/MultipleImplementations/Impl2”)
}

The workaround today is to test for the inverse and add every other directory to “exclude”, but that’s not an elegant long-term solution.

- Karl

···

On 24 Mar 2017, at 21:26, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

We would love to get some feedback on a draft proposal for defining custom target layouts in the Package Manager. This proposal allows overriding the target layout rules set by the Package Manager and simplifies some complex "magic" behaviours.

You can find the proposal at the link below. The text is also included in the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

--

Package Manager Custom Targets Layout
Proposal: SE-NNNN <file:///Users/ankit/NNNN-package-manager-custom-targets-layout.md>
Author: Ankit Aggarwal <https://github.com/aciidb0mb3r&gt;
Review Manager: TBD
Status: Discussion
Bug: SR-29 <Issues · apple/swift-issues · GitHub;
Introduction
This proposal enhances the Package.swift manifest APIs to support custom target layouts, and removes a convention which allowed omission of targets from the manifest.

Motivation
The Package Manager uses a convention system to infer targets structure from disk layout. This works well for most packages, which can easily adopt the conventions, and frees users from needing to update their Package.swift file every time they add or remove sources. Adopting the conventions is more difficult for some packages, however – especially existing C libraries or large projects, which would be difficult to reorganize. We intend to give users a way to make such projects into packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets and source files by inferring them automatically from disk, but they also can be confusing, overly-implicit, and difficult to debug; for example, if the user does not follow the conventions correctly which determine their targets, they may wind up with targets they don't expect, or not having targets they did expect, and either way their clients can't easily see which targets are available by looking at the Package.swift manifest. We want to retain convenience where it really matters, such as easy addition of new source files, but require explicit declarations where being explicit adds significant value. We also want to make sure that the implicit conventions we keep are straightforward and easy to remember.

Proposed solution
We propose to stop inferring targets from disk. They must be explicitly declared in the manifest file. The inference was not very useful, as targets eventually need to be declared in order to use common features such as product and target dependencies, or build settings (which are planned for Swift 4). Explicit target declarations make a package easier to understand by clients, and allow us to provide good diagnostics when the layout on disk does not match the declarations.

We propose to remove the requirement that name of a test target must have suffix "Tests". Instead, test targets will be explicitly declared as such in the manifest file.

We propose a list of pre-defined search paths for declared targets.

When a target does not declare an explicit path, these directories will be used to search for the target. The name of the directory must match the name of the target. The search will be done in order and will be case-sensitive.

Regular targets: package root, Sources, Source, src, srcs. Test targets: Tests, package root, Sources, Source, src, srcs.

It is an error if a target is found in more than one of these paths. In such cases, the path should be explicitly declared using the path property proposed below.

We propose to add a factory method testTarget to the Target class, to define test targets.

.testTarget(name: "FooTests", dependencies: ["Foo"])
We propose to add three properties to the Target class: path, sources andexclude.

path: This property defines the path to the top-level directory containing the target's sources, relative to the package root. It is not legal for this path to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The default value of this property will be nil, which means the target will be searched for in the pre-defined paths. The empty string ("") or dot (".") implies that the target's sources are directly inside the package root.

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included. This can contain directories and individual source files. Directories will be searched recursively for valid source files. Paths specified are relative to the target path.

Each source file will be represented by String type. In future, we will consider upgrading this to its own type to allow per-file build settings. The new type would conform to CustomStringConvertible, so existing declarations would continue to work (except where the strings were constructed programatically).

exclude: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative to the target path. This property has more precedence than sources property.

Note: We plan to support globbing in future, but to keep this proposal short we are not proposing it right now.

It is an error if the paths of two targets overlap (unless resolved with exclude).

// This is an error:
.target(name: "Bar", path: "Sources/Bar"),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

// This works:
.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
For C family library targets, we propose to add a publicHeadersPath property.

This property defines the path to the directory containing public headers of a C target. This path is relative to the target path and default value of this property is include. This mechanism should be further improved in the future, but there are several behaviors, such as modulemap generation, which currently depend of having only one public headers directory. We will address those issues separately in a future proposal.

All existing rules related to custom and automatic modulemap remain intact.

Remove exclude from Package class.

This property is no longer required because of the above proposed per-target exclude property.

The templates provided by the swift package init subcommand will be updated according to the above rules, so that users do not need to manually add their first target to the manifest.

Examples:
Dummy manifest containing all Swift code.
let package = Package(
    name: "SwiftyJSON",
    targets: [
        .target(
            name: "Utility",
            path: "Sources/BasicCode"
        ),

        .target(
            name: "SwiftyJSON",
            dependencies: ["Utility"],
            path: "SJ",
            sources: ["SwiftyJSON.swift"]
        ),

        .testTarget(
            name: "AllTests",
            dependencies: ["Utility", "SwiftyJSON"],
            path: "Tests",
            exclude: ["Fixtures"]
        ),
    ]
)
LibYAML
let packages = Package(
    name: "LibYAML",
    targets: [
        .target(
            name: "libyaml",
            sources: ["src"]
        )
    ]
)
Node.js http-parser
let packages = Package(
    name: "http-parser",
    targets: [
        .target(
            name: "http-parser",
            publicHeaders: ".",
            sources: ["http_parser.c"]
        )
    ]
)
swift-build-tool
let packages = Package(
    name: "llbuild",
    targets: [
        .target(
            name: "swift-build-tool",
            path: ".",
            sources: [
                "lib/Basic",
                "lib/llvm/Support",
                "lib/Core",
                "lib/BuildSystem",
                "products/swift-build-tool/swift-build-tool.cpp",
            ]
        )
    ]
)
Impact on existing code
These enhancements will be added to the version 4 manifest API, which will release with Swift 4. There will be no impact on packages using the version 3 manifest API. When packages update their minimum tools version to 4.0, they will need to update the manifest according to the changes in this proposal.

There are two flat layouts supported in Swift 3:

Source files directly in the package root.
Source files directly inside a Sources/ directory.
If packages want to continue using either of these flat layouts, they will need to explicitly set a target path to the flat directory; otherwise, a directory named after the target is expected. For example, if a package Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift
The updated manifest will look like this:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        .target(name: "Foo", path: "Sources"),
    ]
)
Alternatives considered
We considered making a more minimal change which disabled the flat layouts by default, and provided a top-level property to allow opting back in to them. This would allow us to discourage these layouts – which we would like to do before the package ecosystem grows – without needing to add a fully customizable API. However, we think the fuller API we've proposed here is fairly straightforward and provides the ability to make a number of existing projects into packages, so we think this is worth doing at this time.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Joseph_Heck]

I like the explicitness and flexibility that providing these inputs via declaration in the package manager format.

For the C target classes, I'd like to also see linker flags defined for those packages, as currently that has to be included in additional flags and maintained entirely outside the package structure with scripting, Makefiles, or the like. That may be quite a bit beyond the current proposal of making the directories explicit rather than based on convention, but since this addresses the C target header search paths as well, it seems relevant.

- joe

···

On Mar 24, 2017, at 1:26 PM, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi,

We would love to get some feedback on a draft proposal for defining custom target layouts in the Package Manager. This proposal allows overriding the target layout rules set by the Package Manager and simplifies some complex "magic" behaviours.

You can find the proposal at the link below. The text is also included in the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

--

Package Manager Custom Targets Layout
Proposal: SE-NNNN
Author: Ankit Aggarwal
Review Manager: TBD
Status: Discussion
Bug: SR-29
Introduction
This proposal enhances the Package.swift manifest APIs to support custom target layouts, and removes a convention which allowed omission of targets from the manifest.

Motivation
The Package Manager uses a convention system to infer targets structure from disk layout. This works well for most packages, which can easily adopt the conventions, and frees users from needing to update their Package.swift file every time they add or remove sources. Adopting the conventions is more difficult for some packages, however – especially existing C libraries or large projects, which would be difficult to reorganize. We intend to give users a way to make such projects into packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets and source files by inferring them automatically from disk, but they also can be confusing, overly-implicit, and difficult to debug; for example, if the user does not follow the conventions correctly which determine their targets, they may wind up with targets they don't expect, or not having targets they did expect, and either way their clients can't easily see which targets are available by looking at the Package.swift manifest. We want to retain convenience where it really matters, such as easy addition of new source files, but require explicit declarations where being explicit adds significant value. We also want to make sure that the implicit conventions we keep are straightforward and easy to remember.

Proposed solution
We propose to stop inferring targets from disk. They must be explicitly declared in the manifest file. The inference was not very useful, as targets eventually need to be declared in order to use common features such as product and target dependencies, or build settings (which are planned for Swift 4). Explicit target declarations make a package easier to understand by clients, and allow us to provide good diagnostics when the layout on disk does not match the declarations.

We propose to remove the requirement that name of a test target must have suffix "Tests". Instead, test targets will be explicitly declared as such in the manifest file.

We propose a list of pre-defined search paths for declared targets.

When a target does not declare an explicit path, these directories will be used to search for the target. The name of the directory must match the name of the target. The search will be done in order and will be case-sensitive.

Regular targets: package root, Sources, Source, src, srcs. Test targets: Tests, package root, Sources, Source, src, srcs.

It is an error if a target is found in more than one of these paths. In such cases, the path should be explicitly declared using the path property proposed below.

We propose to add a factory method testTarget to the Target class, to define test targets.

.testTarget(name: "FooTests", dependencies: ["Foo"])
We propose to add three properties to the Target class: path, sources andexclude.

path: This property defines the path to the top-level directory containing the target's sources, relative to the package root. It is not legal for this path to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The default value of this property will be nil, which means the target will be searched for in the pre-defined paths. The empty string ("") or dot (".") implies that the target's sources are directly inside the package root.

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included. This can contain directories and individual source files. Directories will be searched recursively for valid source files. Paths specified are relative to the target path.

Each source file will be represented by String type. In future, we will consider upgrading this to its own type to allow per-file build settings. The new type would conform to CustomStringConvertible, so existing declarations would continue to work (except where the strings were constructed programatically).

exclude: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative to the target path. This property has more precedence than sources property.

Note: We plan to support globbing in future, but to keep this proposal short we are not proposing it right now.

It is an error if the paths of two targets overlap (unless resolved with exclude).

// This is an error:
.target(name: "Bar", path: "Sources/Bar"),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

// This works:
.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
For C family library targets, we propose to add a publicHeadersPath property.

This property defines the path to the directory containing public headers of a C target. This path is relative to the target path and default value of this property is include. This mechanism should be further improved in the future, but there are several behaviors, such as modulemap generation, which currently depend of having only one public headers directory. We will address those issues separately in a future proposal.

All existing rules related to custom and automatic modulemap remain intact.

Remove exclude from Package class.

This property is no longer required because of the above proposed per-target exclude property.

The templates provided by the swift package init subcommand will be updated according to the above rules, so that users do not need to manually add their first target to the manifest.

Examples:
Dummy manifest containing all Swift code.
let package = Package(
    name: "SwiftyJSON",
    targets: [
        .target(
            name: "Utility",
            path: "Sources/BasicCode"
        ),

        .target(
            name: "SwiftyJSON",
            dependencies: ["Utility"],
            path: "SJ",
            sources: ["SwiftyJSON.swift"]
        ),

        .testTarget(
            name: "AllTests",
            dependencies: ["Utility", "SwiftyJSON"],
            path: "Tests",
            exclude: ["Fixtures"]
        ),
    ]
)
LibYAML
let packages = Package(
    name: "LibYAML",
    targets: [
        .target(
            name: "libyaml",
            sources: ["src"]
        )
    ]
)
Node.js http-parser
let packages = Package(
    name: "http-parser",
    targets: [
        .target(
            name: "http-parser",
            publicHeaders: ".",
            sources: ["http_parser.c"]
        )
    ]
)
swift-build-tool
let packages = Package(
    name: "llbuild",
    targets: [
        .target(
            name: "swift-build-tool",
            path: ".",
            sources: [
                "lib/Basic",
                "lib/llvm/Support",
                "lib/Core",
                "lib/BuildSystem",
                "products/swift-build-tool/swift-build-tool.cpp",
            ]
        )
    ]
)
Impact on existing code
These enhancements will be added to the version 4 manifest API, which will release with Swift 4. There will be no impact on packages using the version 3 manifest API. When packages update their minimum tools version to 4.0, they will need to update the manifest according to the changes in this proposal.

There are two flat layouts supported in Swift 3:

Source files directly in the package root.
Source files directly inside a Sources/ directory.
If packages want to continue using either of these flat layouts, they will need to explicitly set a target path to the flat directory; otherwise, a directory named after the target is expected. For example, if a package Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift
The updated manifest will look like this:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        .target(name: "Foo", path: "Sources"),
    ]
)
Alternatives considered
We considered making a more minimal change which disabled the flat layouts by default, and provided a top-level property to allow opting back in to them. This would allow us to discourage these layouts – which we would like to do before the package ecosystem grows – without needing to add a fully customizable API. However, we think the fuller API we've proposed here is fairly straightforward and provides the ability to make a number of existing projects into packages, so we think this is worth doing at this time.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Brent_Royal-Gordon]

Stupid little niggle: could this be an optional array with a default value of `nil`? It seems more natural to treat the absence of a `sources` list as meaning "auto-detect" than to treat an empty `sources` list that way.

···

On Mar 24, 2017, at 1:26 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included.

--
Brent Royal-Gordon
Architechies

[Anders]

Hi,

We would love to get some feedback on a draft proposal for defining custom target layouts in the Package Manager. This proposal allows overriding the target layout rules set by the Package Manager and simplifies some complex "magic" behaviours.

You can find the proposal at the link below. The text is also included in the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

--

Package Manager Custom Targets Layout
Proposal: SE-NNNN <file:///Users/ankit/NNNN-package-manager-custom-targets-layout.md>
Author: Ankit Aggarwal <https://github.com/aciidb0mb3r&gt;
Review Manager: TBD
Status: Discussion
Bug: SR-29 <Issues · apple/swift-issues · GitHub;
Introduction
This proposal enhances the Package.swift manifest APIs to support custom target layouts, and removes a convention which allowed omission of targets from the manifest.

Motivation
The Package Manager uses a convention system to infer targets structure from disk layout. This works well for most packages, which can easily adopt the conventions, and frees users from needing to update their Package.swift file every time they add or remove sources. Adopting the conventions is more difficult for some packages, however – especially existing C libraries or large projects, which would be difficult to reorganize. We intend to give users a way to make such projects into packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets and source files by inferring them automatically from disk, but they also can be confusing, overly-implicit, and difficult to debug; for example, if the user does not follow the conventions correctly which determine their targets, they may wind up with targets they don't expect, or not having targets they did expect, and either way their clients can't easily see which targets are available by looking at the Package.swift manifest. We want to retain convenience where it really matters, such as easy addition of new source files, but require explicit declarations where being explicit adds significant value. We also want to make sure that the implicit conventions we keep are straightforward and easy to remember.

Proposed solution
We propose to stop inferring targets from disk. They must be explicitly declared in the manifest file. The inference was not very useful, as targets eventually need to be declared in order to use common features such as product and target dependencies, or build settings (which are planned for Swift 4). Explicit target declarations make a package easier to understand by clients, and allow us to provide good diagnostics when the layout on disk does not match the declarations.

We propose to remove the requirement that name of a test target must have suffix "Tests". Instead, test targets will be explicitly declared as such in the manifest file.

We propose a list of pre-defined search paths for declared targets.

When a target does not declare an explicit path, these directories will be used to search for the target. The name of the directory must match the name of the target. The search will be done in order and will be case-sensitive.

Regular targets: package root, Sources, Source, src, srcs. Test targets: Tests, package root, Sources, Source, src, srcs.

It is an error if a target is found in more than one of these paths. In such cases, the path should be explicitly declared using the path property proposed below.

We propose to add a factory method testTarget to the Target class, to define test targets.

.testTarget(name: "FooTests", dependencies: ["Foo"])
We propose to add three properties to the Target class: path, sources andexclude.

path: This property defines the path to the top-level directory containing the target's sources, relative to the package root. It is not legal for this path to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The default value of this property will be nil, which means the target will be searched for in the pre-defined paths. The empty string ("") or dot (".") implies that the target's sources are directly inside the package root.

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included. This can contain directories and individual source files. Directories will be searched recursively for valid source files. Paths specified are relative to the target path.

Each source file will be represented by String type. In future, we will consider upgrading this to its own type to allow per-file build settings. The new type would conform to CustomStringConvertible, so existing declarations would continue to work (except where the strings were constructed programatically).

Is the default not recursively searching for source files? If not, it would be great to have ways that achieve that, e.g. `*.swift`.

exclude: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative to the target path. This property has more precedence than sources property.

Note: We plan to support globbing in future, but to keep this proposal short we are not proposing it right now.

It is an error if the paths of two targets overlap (unless resolved with exclude).

// This is an error:
.target(name: "Bar", path: "Sources/Bar"),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

// This works:
.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
For C family library targets, we propose to add a publicHeadersPath property.

This property defines the path to the directory containing public headers of a C target. This path is relative to the target path and default value of this property is include. This mechanism should be further improved in the future, but there are several behaviors, such as modulemap generation, which currently depend of having only one public headers directory. We will address those issues separately in a future proposal.

All existing rules related to custom and automatic modulemap remain intact.

Remove exclude from Package class.

This property is no longer required because of the above proposed per-target exclude property.

The templates provided by the swift package init subcommand will be updated according to the above rules, so that users do not need to manually add their first target to the manifest.

Examples:
Dummy manifest containing all Swift code.
let package = Package(
    name: "SwiftyJSON",
    targets: [
        .target(
            name: "Utility",
            path: "Sources/BasicCode"
        ),

        .target(
            name: "SwiftyJSON",
            dependencies: ["Utility"],
            path: "SJ",
            sources: ["SwiftyJSON.swift"]
        ),

        .testTarget(
            name: "AllTests",
            dependencies: ["Utility", "SwiftyJSON"],
            path: "Tests",
            exclude: ["Fixtures"]
        ),
    ]
)
LibYAML
let packages = Package(
    name: "LibYAML",
    targets: [
        .target(
            name: "libyaml",
            sources: ["src"]
        )
    ]
)
Node.js http-parser
let packages = Package(
    name: "http-parser",
    targets: [
        .target(
            name: "http-parser",
            publicHeaders: ".",
            sources: ["http_parser.c"]
        )
    ]
)
swift-build-tool
let packages = Package(
    name: "llbuild",
    targets: [
        .target(
            name: "swift-build-tool",
            path: ".",
            sources: [
                "lib/Basic",
                "lib/llvm/Support",
                "lib/Core",
                "lib/BuildSystem",
                "products/swift-build-tool/swift-build-tool.cpp",
            ]
        )
    ]
)
Impact on existing code
These enhancements will be added to the version 4 manifest API, which will release with Swift 4. There will be no impact on packages using the version 3 manifest API. When packages update their minimum tools version to 4.0, they will need to update the manifest according to the changes in this proposal.

There are two flat layouts supported in Swift 3:

Source files directly in the package root.
Source files directly inside a Sources/ directory.
If packages want to continue using either of these flat layouts, they will need to explicitly set a target path to the flat directory; otherwise, a directory named after the target is expected. For example, if a package Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift
The updated manifest will look like this:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        .target(name: "Foo", path: "Sources"),
    ]
)
Alternatives considered
We considered making a more minimal change which disabled the flat layouts by default, and provided a top-level property to allow opting back in to them. This would allow us to discourage these layouts – which we would like to do before the package ecosystem grows – without needing to add a fully customizable API. However, we think the fuller API we've proposed here is fairly straightforward and provides the ability to make a number of existing projects into packages, so we think this is worth doing at this time.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Regards,
Anders

···

On 25 Mar 2017, at 4:26 AM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

[codafi (Robert Widmann)]

A huge +1 for this proposal (and I’m definitely saying that partially because it folds in the test targets proposal). Having a flexible layout will make packages that need to support additional structure for things like alternative package managers or more exotic dependency trees a reality.

~Robert Widmann

···

On Mar 25, 2017, at 9:20 PM, Karl Wagner via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

I like the part of the proposal which stops inferring targets from the directory layout; I’ve always found that stuff pretty annoying.

Two points:

1) I’m not sure about listing all of the test targets together with the product targets. I feel it may be better to have them as separate Arrays.

2) I think we need a way to specify directories non-recursively. For example:

- Sources
> - Foo
> > - MultipleImplementations
> > > - Impl1
> > > - Impl2

In this case, “MultipleImplementations” contains some common definitions, but has multiple potential implementations. Impl1 and Impl2 will each implement the functionality in their own way, perhaps making use of platform-specific features or optional dependencies. Basically, I want to import “MultipleImplementations” non-recursively, then decide which subdirectory/implementation to use based on some build condition:

.Target(“Foo”, sources: [“Sources/Foo”, “[NR]Sources/Foo/MultipleImplementations”])

if someCondition {
  package.targets[0].sources.append(“Sources/Foo/MultipleImplementations/Impl1”)
else {
  package.targets[0].sources.append(“Sources/Foo/MultipleImplementations/Impl2”)
}

The workaround today is to test for the inverse and add every other directory to “exclude”, but that’s not an elegant long-term solution.

- Karl

On 24 Mar 2017, at 21:26, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi,

We would love to get some feedback on a draft proposal for defining custom target layouts in the Package Manager. This proposal allows overriding the target layout rules set by the Package Manager and simplifies some complex "magic" behaviours.

You can find the proposal at the link below. The text is also included in the email.

https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md

Thanks,
Ankit

--

Package Manager Custom Targets Layout
Proposal: SE-NNNN <file:///Users/ankit/NNNN-package-manager-custom-targets-layout.md>
Author: Ankit Aggarwal <https://github.com/aciidb0mb3r&gt;
Review Manager: TBD
Status: Discussion
Bug: SR-29 <Issues · apple/swift-issues · GitHub;
Introduction
This proposal enhances the Package.swift manifest APIs to support custom target layouts, and removes a convention which allowed omission of targets from the manifest.

Motivation
The Package Manager uses a convention system to infer targets structure from disk layout. This works well for most packages, which can easily adopt the conventions, and frees users from needing to update their Package.swift file every time they add or remove sources. Adopting the conventions is more difficult for some packages, however – especially existing C libraries or large projects, which would be difficult to reorganize. We intend to give users a way to make such projects into packages without needing to conform to our conventions.

The current convention rules make it very convenient to add new targets and source files by inferring them automatically from disk, but they also can be confusing, overly-implicit, and difficult to debug; for example, if the user does not follow the conventions correctly which determine their targets, they may wind up with targets they don't expect, or not having targets they did expect, and either way their clients can't easily see which targets are available by looking at the Package.swift manifest. We want to retain convenience where it really matters, such as easy addition of new source files, but require explicit declarations where being explicit adds significant value. We also want to make sure that the implicit conventions we keep are straightforward and easy to remember.

Proposed solution
We propose to stop inferring targets from disk. They must be explicitly declared in the manifest file. The inference was not very useful, as targets eventually need to be declared in order to use common features such as product and target dependencies, or build settings (which are planned for Swift 4). Explicit target declarations make a package easier to understand by clients, and allow us to provide good diagnostics when the layout on disk does not match the declarations.

We propose to remove the requirement that name of a test target must have suffix "Tests". Instead, test targets will be explicitly declared as such in the manifest file.

We propose a list of pre-defined search paths for declared targets.

When a target does not declare an explicit path, these directories will be used to search for the target. The name of the directory must match the name of the target. The search will be done in order and will be case-sensitive.

Regular targets: package root, Sources, Source, src, srcs. Test targets: Tests, package root, Sources, Source, src, srcs.

It is an error if a target is found in more than one of these paths. In such cases, the path should be explicitly declared using the path property proposed below.

We propose to add a factory method testTarget to the Target class, to define test targets.

.testTarget(name: "FooTests", dependencies: ["Foo"])
We propose to add three properties to the Target class: path, sources andexclude.

path: This property defines the path to the top-level directory containing the target's sources, relative to the package root. It is not legal for this path to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The default value of this property will be nil, which means the target will be searched for in the pre-defined paths. The empty string ("") or dot (".") implies that the target's sources are directly inside the package root.

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included. This can contain directories and individual source files. Directories will be searched recursively for valid source files. Paths specified are relative to the target path.

Each source file will be represented by String type. In future, we will consider upgrading this to its own type to allow per-file build settings. The new type would conform to CustomStringConvertible, so existing declarations would continue to work (except where the strings were constructed programatically).

exclude: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative to the target path. This property has more precedence than sources property.

Note: We plan to support globbing in future, but to keep this proposal short we are not proposing it right now.

It is an error if the paths of two targets overlap (unless resolved with exclude).

// This is an error:
.target(name: "Bar", path: "Sources/Bar"),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),

// This works:
.target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]),
.testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"),
For C family library targets, we propose to add a publicHeadersPath property.

This property defines the path to the directory containing public headers of a C target. This path is relative to the target path and default value of this property is include. This mechanism should be further improved in the future, but there are several behaviors, such as modulemap generation, which currently depend of having only one public headers directory. We will address those issues separately in a future proposal.

All existing rules related to custom and automatic modulemap remain intact.

Remove exclude from Package class.

This property is no longer required because of the above proposed per-target exclude property.

The templates provided by the swift package init subcommand will be updated according to the above rules, so that users do not need to manually add their first target to the manifest.

Examples:
Dummy manifest containing all Swift code.
let package = Package(
    name: "SwiftyJSON",
    targets: [
        .target(
            name: "Utility",
            path: "Sources/BasicCode"
        ),

        .target(
            name: "SwiftyJSON",
            dependencies: ["Utility"],
            path: "SJ",
            sources: ["SwiftyJSON.swift"]
        ),

        .testTarget(
            name: "AllTests",
            dependencies: ["Utility", "SwiftyJSON"],
            path: "Tests",
            exclude: ["Fixtures"]
        ),
    ]
)
LibYAML
let packages = Package(
    name: "LibYAML",
    targets: [
        .target(
            name: "libyaml",
            sources: ["src"]
        )
    ]
)
Node.js http-parser
let packages = Package(
    name: "http-parser",
    targets: [
        .target(
            name: "http-parser",
            publicHeaders: ".",
            sources: ["http_parser.c"]
        )
    ]
)
swift-build-tool
let packages = Package(
    name: "llbuild",
    targets: [
        .target(
            name: "swift-build-tool",
            path: ".",
            sources: [
                "lib/Basic",
                "lib/llvm/Support",
                "lib/Core",
                "lib/BuildSystem",
                "products/swift-build-tool/swift-build-tool.cpp",
            ]
        )
    ]
)
Impact on existing code
These enhancements will be added to the version 4 manifest API, which will release with Swift 4. There will be no impact on packages using the version 3 manifest API. When packages update their minimum tools version to 4.0, they will need to update the manifest according to the changes in this proposal.

There are two flat layouts supported in Swift 3:

Source files directly in the package root.
Source files directly inside a Sources/ directory.
If packages want to continue using either of these flat layouts, they will need to explicitly set a target path to the flat directory; otherwise, a directory named after the target is expected. For example, if a package Foo has following layout:

Package.swift
Sources/main.swift
Sources/foo.swift
The updated manifest will look like this:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        .target(name: "Foo", path: "Sources"),
    ]
)
Alternatives considered
We considered making a more minimal change which disabled the flat layouts by default, and provided a top-level property to allow opting back in to them. This would allow us to discourage these layouts – which we would like to do before the package ecosystem grows – without needing to add a fully customizable API. However, we think the fuller API we've proposed here is fairly straightforward and provides the ability to make a number of existing projects into packages, so we think this is worth doing at this time.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[David_Sweeris]

Agreed

···

Sent from my iPhone

On Mar 26, 2017, at 22:27, Brent Royal-Gordon via swift-build-dev <swift-build-dev@swift.org> wrote:

On Mar 24, 2017, at 1:26 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included.

Stupid little niggle: could this be an optional array with a default value of `nil`? It seems more natural to treat the absence of a `sources` list as meaning "auto-detect" than to treat an empty `sources` list that way.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[hartbit (David Hart)]

I second that.

···

On 27 Mar 2017, at 07:27, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 24, 2017, at 1:26 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included.

Stupid little niggle: could this be an optional array with a default value of `nil`? It seems more natural to treat the absence of a `sources` list as meaning "auto-detect" than to treat an empty `sources` list that way.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Aciid (Ankit Aggarwal)]

Hi Karl,

Thank you for the feedback, comments inline.

1) I’m not sure about listing all of the test targets together with the
product targets. I feel it may be better to have them as separate Arrays.

They both are similar entity so they stay together. Another example is
products, which can be an executable or a library. It is also easier to
scale if these are not separate. Otherwise, we need to keep adding new
properties to the top level Package structure every time we introduce a new
type of target.

2) I think we need a way to specify directories non-recursively. For

example:

- Sources
> - Foo
> > - MultipleImplementations
> > > - Impl1
> > > - Impl2

In this case, “MultipleImplementations” contains some common definitions,
but has multiple potential implementations. Impl1 and Impl2 will each
implement the functionality in their own way, perhaps making use of
platform-specific features or optional dependencies. Basically, I want to
import “MultipleImplementations” non-recursively, then decide which
subdirectory/implementation to use based on some build condition:

.Target(“Foo”, sources: [“Sources/Foo”, “[NR]Sources/Foo/
MultipleImplementations”])

if someCondition {
  package.targets[0].sources.append(“Sources/Foo/
MultipleImplementations/Impl1”)
else {
  package.targets[0].sources.append(“Sources/Foo/
MultipleImplementations/Impl2”)
}

The workaround today is to test for the inverse and add every other
directory to “exclude”, but that’s not an elegant long-term solution.

We agree we need to support more complex specifications. We plan to add
support for globbing and conditionals in future (there is a note about that
in the proposal). Both of these are additive features and this proposals
gives us a good base for designing that.

Thanks,
Ankit

[Aciid (Ankit Aggarwal)]

Hi Joesph,

Thank you for the feedback!

For the C target classes, I'd like to also see linker flags defined for
those packages, as currently that has to be included in additional flags
and maintained entirely outside the package structure with scripting,
Makefiles, or the like. That may be quite a bit beyond the current proposal
of making the directories explicit rather than based on convention, but
since this addresses the C target header search paths as well, it seems
relevant.

We do need to add support for that but linker flags falls under build
settings and not this proposal. This proposal allows customising the path
to the directory containing public headers which is currently hard coded to
"include".

[Jens_Nerup]

+1

···

On 27 Mar 2017, at 07.27, Brent Royal-Gordon via swift-build-dev <swift-build-dev@swift.org> wrote:

On Mar 24, 2017, at 1:26 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

sources: This property defines the source files to be included in the target, relative to the target path. The default value of this property will be an empty array, which means all valid source files found in the target's path will be included.

Stupid little niggle: could this be an optional array with a default value of `nil`? It seems more natural to treat the absence of a `sources` list as meaning "auto-detect" than to treat an empty `sources` list that way.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Aciid (Ankit Aggarwal)]

Is the default *not* recursively searching for source files? If not, it
would be great to have ways that achieve that, e.g. `*.swift`.

The default *is* searching recursively. There will be globbing support in
future that allows `*.swift` but for now we will (recursively) search for
valid sources, which could be either Swift or C-family (but not both).

[Aciid (Ankit Aggarwal)]

Oops, I had pressed reply instead reply all. The proposal
<https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md&gt;
is
updated with this change.

···

On Mon, Mar 27, 2017 at 12:32 PM, Ankit Aggarwal <ankit_aggarwal@apple.com> wrote:

Hi Brent,

I agree, this sounds like a reasonable thing to do. We can probably emit a
warning when its set to an empty array.

Thanks!

On Mon, Mar 27, 2017 at 10:57 AM, Brent Royal-Gordon via swift-build-dev < > swift-build-dev@swift.org> wrote:

On Mar 24, 2017, at 1:26 PM, Ankit Aggarwal via swift-evolution < >> swift-evolution@swift.org> wrote:

sources: This property defines the source files to be included in the
target, relative to the target path. The default value of this property
will be an empty array, which means all valid source files found in the
target's path will be included.

Stupid little niggle: could this be an optional array with a default
value of `nil`? It seems more natural to treat the absence of a `sources`
list as meaning "auto-detect" than to treat an empty `sources` list that
way.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[David_Sweeris]

What do we think about adding convenience inits that take a single parameter in place of the s, for when we're only passing one value?

The down side is that it makes it more complicated for 3rd party tools to parse. Dunno if that's a concern.

- Dave Sweeris

···

On Mar 29, 2017, at 10:19, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org> wrote:

The proposal is updated with this change.

[Aciid (Ankit Aggarwal)]

We discussed variadic overloads during the manifest redesign proposal but ultimately rejected <https://github.com/apple/swift-evolution/blob/master/proposals/0158-package-manager-manifest-api-redesign.md#alternatives-considered&gt; it. We could add an overload for a singular source file (instead of variadic) but I think its simpler to just have one option, and it is also easier when you want to extend your source list form one to two.

I think parsing is not a concern because tools are expected to ask for JSON representation and not directly parse the swift file.

···

On 29-Mar-2017, at 11:18 PM, David Sweeris <davesweeris@mac.com> wrote:

On Mar 29, 2017, at 10:19, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

The proposal <https://github.com/aciidb0mb3r/swift-evolution/blob/custom-targets-layout/proposals/NNNN-package-manager-custom-targets-layout.md&gt; is updated with this change.

What do we think about adding convenience inits that take a single parameter in place of the s, for when we're only passing one value?

The down side is that it makes it more complicated for 3rd party tools to parse. Dunno if that's a concern.

- Dave Sweeris

[David_Sweeris]

K, seems reasonable.

- Dave Sweeris

···

On Mar 29, 2017, at 10:56, Ankit Aggarwal <ankit_aggarwal@apple.com> wrote:

We discussed variadic overloads during the manifest redesign proposal but ultimately rejected it. We could add an overload for a singular source file (instead of variadic) but I think its simpler to just have one option, and it is also easier when you want to extend your source list form one to two.