[Pitch] New command for package creation, and package templates

Hey everyone!

I'd like to get your feedback on this evolution. In short the goal is to improve the experience of creating brand new packages when there isn't any pre-existing sources, and the introduction of package templates allowing for greater flexibility when creating Swift packages.

4 Likes

-1 as is

The proposal was well written and I enjoyed reading it. swift package create is a very pleasant sounding command, but the changes suggested don’t seem worth the complexity and added burden to the package managers maintainability.

Templates are clearly the star of the show here. I think templates in general sound like a good idea. However while I was reading the proposal I kept thinking it would be way easier to copy or clone the package you want and rename it, then to look up template names assuming you even have them installed and updated.

I can’t personally conceive of a situation where you’d be creating the same type of package so frequently that you’d need your own personal template. However my packages always require a per-project executable package and an intermediate library with dependencies. All my users will need to do this, so I can appreciate an automation of some kind for this situation. However the proposal requires my users to install a template locally, and to be perfectly frank; if this feature existed today I would simply tell my users to clone the sample project and rename it.

I think if a template system were to be added it should revolve around git and actual package directory structures. This way the package manager won’t require complex changes and templates would grow with package features. Renaming the clone and removing its .git directory in a single command would be nice.

So a swift package rename command would solve most of the suggested issues. There’s another proposal for modifying existing packages and I think it includes rename, Please forgive me I’m on mobile and cant find it atm.

If a revision is done, I would appreciate a focus on keeping things as similar to actual packages as possible. I would also like to see a method for sharing templates, preferably with git, and preferably allowing multiple templates in a repository.

After reading through the pitch proposal, I am +1 in the concept, but -1 on the approach.

The concept being proposed would be a boon for anyone looking to establish conventions and speed up onboarding, from organizations to framework authors. However, there are some aspects of the approach that miss the mark for me. A few questions, if you don’t mind.

Have you looked into the how other PM communities approach this problem, such as npm.js ? In general, it would be better if templates were invisibly downloaded and processed by users, and if they gave creators the ability to specify any permutation of filesystem structure and content using both static assets and dynamic executables. Additionally, I would’ve liked to have seen a section in the proposal that detailed which features from which PM communities where used to establish precedent.

Why JSON and not Swift? Again, I would like to see the filesystem alone declaring the template’s static structure and content. But for all things dynamic, I suggest following the precedent established by SPM itself in its use of Swift for Package.swift.

Hope this feedback helps refine the proposal. :slight_smile:

By request, I'm taking this over from Twitter. Please pardon the semi-coherent info-dump.

The motivation is strong but I don't like this particular approach. My most frequent use-cases for creation (omitting, for example, libraries with samples):

  • basic executable (about 10%)
  • Swift Argument Parser executable (about 65%)
  • libraries (about 15%)

The SAP executable is not addressed at all yet. So I have a lot of template folders and occasionally add building utilities to my command-line utils. I'm in the middle of breaking some of those out to their own SPM initializer util.

Every time I dive into to the SPM code, I throw up my hands in frustration as I'd prefer things to work better from that end. I'm not fond of how SPM currently handles hard-coded creation and agree strongly with the motivation.

A few more thoughts: Making Project.swift the primary end-user-facing point for project customization and control, especially outside the Xcode ecosystem, is confusing especially for new learners and those who create projects infrequently.

I particularly dislike the three "names" in the Project.swift file and the current set of comments. (The names roughly break down to project-name (often the same as a repo), module name, and the library or executable name that is produced and include one or more modules and/or dependencies.) These aren't explained in comments and the package init command uses the same name for each spot, which is particularly unhelpful.

Further, I know it is Applecentric to include comments about 5.1 being Mojave-friend, for example, but a link to recommended swift versions wouldn't be out of place and wouldn't have to be maintained from within the code (although it would outside).

Initialization does not include README.md, CHANGELOG.md, or LICENSE.txt. CHANGELOG in particular can be (but probably shouldn't be) generated with tags and a git history.

Initialization does not git init but generates a particularly unhelpful .gitignore, and does not check for a global gitignore. Any guiding json files (or yaml or whatever) should be able to be generated as a skeleton by SPM and then ignored with gitignore after. Hiding it down in five levels is unhelpful.

There's more but that's all I can think of at the moment.

6 Likes

I read two interesting points here:

  1. the template should be based on a full package directory that is copied & transformed (aka "renamed" above) instead of configuration driven, as mentioned in the alternatives considered section

imo, there is allot of of appeal to the "transform a directory" approach as it allows for more flexibility. the challange I can see would be getting the transformation step right. afaict, it would not be enough to simply copy a directory and rename it, as the template package could be arbitrarily complex with many products, targets, supporting files, etc. going down this path also requires some placeholder substitution logic, for example taking the requested package name from the --name flag and applying it to the directory base template. all this to say that this approach is more flexible but may be difficult to achieve with a high level of correctness given the open ended complexity of such template.

  1. how to make sharing and installing a template easier.

I think this points to a missing piece in the pitch/proposal. I suggest adding a swift package add-template (or similar name) command that takes a URL, and downloads the template to the right location so its easy to share and "install" templates.

one use case for this could be a large development team, or corporate department where they want to enforce standards across packages, such as sources are always in the "src" directory instead of SwiftPM's current "Sources" directory.

1 Like

Are you suggesting to prioritize the idea mentioned in the the future direction section? Or did you have something else in mind on this?

Could you elaborate more on what you are missing in the proposed approach to help cover your use case?

+1 to make the code comments in the generated Package.swift more useful. I think this is something that can be fixed now, regardless of this proposal.

adopting a "transform a directory" approach (see above), could help solve the supporting files aspect. even the current configuration based approach could potentially do that, either by embedding such files in the JSON, or making it into a more complex zip file with the JSON pointing to addition files included in the zip file.

personally, I question if SwiftPM should create a .gitignore at all, and instead leave it to the users to decide.

It might be worth exploring the idea of leveraging spm itself for this. For example templates could use commands suggested in the package update pitch I mentioned earlier and expand upon that. Though expanding it should probably be it's own pitch.

Templates could be constructed by transforming a package using a sequence of spm commands from a manifest. The approach would be similar to the JSON manifest suggested in the proposal, but the manifest contents would map in some way to spm command line arguments. Then those commands could also be used in custom scripts or automation if exposed for command line. The result could be two powerful feature sets with not too much additional cost or maintenance.

Templates could also expand easily with new command line arguments in the future. The complexity would depend on the chosen style for the command list manifest. I'd personally like something along the lines of a new top level variable in Package.swift, let template = Template(), or a Template.swift that can only exist without a Package.swift. That would be a familiar approach for users as @ryansobol mentioned.

This could theoretically result in templates that make templates that make packages. But I'm not entirely sure if that's useful or not; but it sounds powerful and interesting.

I personally very much like having the gitignore setup for me, it would be nice to also setup a git repo and maybe provide an escape hatch like --no-git to disable git related setup.

I have a lot of use cases that both overlap the proposal and do not. So that's a little hard to answer.

  • There's no consideration of injection into an Xcode project rather than folders of Swift code files. I have a lot of projects set up side-by-side, which I have had to do by hand, and there's an issue of the source tree not matching the SPM default. There's no consideration of creating a workspace with a build/test scheme either.

  • SPM does not have any awareness of SAP.

  • Yes, I would like to see the .gitignore gone and I would like to see the very basics of readme, changelog, and license added as practical realities.

init is a term of art for creation (see git) and shouldn't be replaced with create or redefined for injection. I would personally like to see a required third item swift package init lib and swift package init exe to clarify that the tool is intentionally building one form or another rather than always building libraries. Trying to get help on swift package init --type is unduly burdensome indicating poor design.

I agree that that's a separate problem and should be a separate evolution thread. I also think that SPM injection into an existing project and SPM template reform are two issues so widely separated that they should be separate proposals.

I find the JSON approach inelegant. I think a lot more work needs to be done in fixing SPM before considering end-user templating the way that Xcode supports end-user templating, especially for SPM multi-platform deployment. I think the "right" answer for project creation, project upgrade/integration, and so forth will feel more elegant.

Yes. Though whether SwiftPM needs new APIs, as mentioned in the future direction section, or a filesystem-based contract that executes custom Swift scripts that adhere to a file name convention, or a hybrid approach of both, template authors will need a well-defined plug-in system for dynamically generating templates. And I question the use of JSON for this need. Fundamentally, it’s a leaky abstraction for doing branching computation because it’s not a Turing-complete language.

I can offer some perspective on this, since it's what we do with the Vapor Toolbox. We've changed the approach over the years as the toolbox has gotten better. Originally we did the simple thing of cloning a template, changing the Package name in the manifest and resetting the .git file. This served us pretty well for a number of years and was fairly basic. You can see the code for that in the 3.1.3 tag. (It also interfaced with a few cloud providers and wrapped swift test and swift build etc, but vapor new does as talked about in this pitch.)

The current toolbox is much more advanced and more 'template-y'. This parses a manifest file in our template to generate a project that works with different databases (or none at all) and Leaf or no Leaf (our templating engine for HTML). vapor new asks a number of questions and then builds a Package.swift, relevant sources (including model files if a database is chosen), a Docker compose file and Leaf templates if required. It's great and works really well for us (and you can specify your own template) but it is complex.

IMO I don't think that behaviour belongs in the package manager. I can see the justification for a basic "clone this Git project and reset .git" but beyond that feels like putting too much complexity for building packages into a package manager.

Happy to answer any questions about our toolbox and templates though!

4 Likes

Using the proposed package editing machinery for package construction has a lot of appeal, since the latter is just essentially a transformation starting with an empty state. In the future, such package transformers could perhaps be deployed as plugins, giving them access to a richer API than just command line options.

The section discussing future directions spells out a very nice future in this area, I think. For me the question is whether to wait for that future, or whether to start with something simpler that does have limitations but that can provide a lot of fairly immediate improvement. To me it seems very appealing to make short-term ergonomic improvements along the lines of this pitch, even if there are ideas for how to do something more advanced in the future.

2 Likes

I've been brainstorming with @leogdion about the initialization problem. I thought I'd drop some of my thoughts into this convo in a fairly rough form:

A package manager's role is managing packages. swift package init doesn't manage packages. Divesting an init tool may be a better solution than adding new commands for package creation.

The package manager shouldn't be stepping on, reproducing, or messing with git. A swift package init tool should do its own job, not git's job. It should not create, manage, or customize (.gitignore) git repositories.

A well-designed package init tool should be able to set-up new repos as well as inject SwiftPM support into an existing repo. It should be able to do so for both standard projects (libraries, executables, SAP executables) and customized projects. It should also support a broader understanding of what makes up a core baseline package.

Any init tool that hard codes content, as SwiftPM currently does, is fundamentally broken from the start. A better tool separates content from the tool itself, which probably means some level of served, cached, upgradable content for off-line use.

Templating and project customization can be presented to the project authors in a better and more understandable form than "Project.swift". Templates are needed for both file types and project types and should allow end-user customization stored in a standard location, for example ~/Library/Developer/SwiftPM/Templates/Files and .../Projects.

3 Likes

Noted. :blush:

I hear what you’re saying. Though there are a lot of folks who work in other language ecosystems that might disagree.

All of the above package managers can initialize new packages as well build packages for distribution. As a member in many of these ecosystems, I’ve watched first-hand how their project initialization tools have received continued investments in functionality over time. In short, you want to propose a divestment in swift package init, know that you’ll be swimming upstream against a pretty mighty current. I highly recommend everyone participating in this conversation to spend some time learning the ins and outs of the tools from other languages.

Agreed. While Git may be the de facto SCM, it would probably be irresponsible for SwiftPM’s default project template to assume every project will use it, or any SCM for that matter. However, I fully support the ability for project template authors to override the default project content and structure, customizing their templates for their needs.

If by repo you mean project, then agreed. Though, personally, I would gladly take support for applying custom templates to new projects over applying default templates to existing projects, if a choice has to be made in the short-term.

Not sure what “broader understanding” means, but it sounds cool!

3 Likes

Continued…

While the word broken feels a little too harsh, I hear what you’re saying. Agreed that there are lots of unrealized architectural benefits from decoupling the default project template from swift package init, especially when considering the implementation of custom project templates.

When it comes to served, cached, and upgradable content, I recommend taking a close look at npm init <initializer>. Since it’s introduction a few years ago, it has become widely used in the Node.js ecosystem.

For static content and structure, I agree that there’s no better abstraction and data model for project templates than the filesystem itself. In addition, there will be times when template authors will need to offer choices to their users. For example, choose from a list of platforms the ones this project will run on. Choose which binary targets this project will need. Choose which dependencies, code linter, code formatter, style guide, SCM, etc. this project will need.

For all things involving user choice, template authors will want to bundle dynamic (Swift) code with their static filesystem assets to be run at initialization time. As mentioned in an earlier post of mine, functionally like this would be a boon for anyone looking to establish conventions and speed up onboarding, from organizations to framework authors.

1 Like

I think I'd be happy if it were divested even if it does not appear to be so from the user's POV (as it can be argued that a dev might think that swift and swift build are the same project).. I really don't like adding create.

Agreed.

A little hobbyhorse. Standard repo elements including readme/changelog/license. Things that make sense no matter where you use SPM and assume that the package will be hosted somewhere, whether publicly or privately. While I know that GitHub has added README and License support, git has not, and it feels to me as if this is inherently part of the initialized package, and not the role of version control.

I tried to come up with another word but "broken" seems industry standard so I went with it. Wasn't going for harsh and apologies it came across too strong

Sorry for the randomness of this but there are also a few things that I haven't come up with good ways to deal with, such as author/organization. I ended up with /usr/bin/git config user.name for my own stuff. I'm not sure how Xcode pulls into its own templates.

Finally, these templates and their supporting files (and code) are themselves are potentially sharable/distributable artifacts.

1 Like

After taking everyone's feedback we've decided that it would be best to transform a directory instead of using JSON for package templates.

Here's the revised proposal:

2 Likes