[GSoC-2023] Customizable(and interactive) package templates proposal

[Howard_Chu]

Hey everyone, I'd like to work on the customizable package templates project.

Abstract

One of the fastest ways to create a swift project is to use a swift package init command in swiftPM commandline tool. User can then select from a handful of hardcoded templates to start with. Because the templates are hardcoded into swiftPM’s source code, they are very limited in terms of variety.

Users have many different needs for such templates, so it’s best to figure out a way to customize these templates and share them with others.

The goals are to 1.define a format for templates, 2.design an API for their customization, and 3.make initialization of a project(with templates) interactive.

Here is my full proposal: proposal

I draw inspiration from this post: Update to swift package init templates

Design Ideas

Introducing three commands:

swift package init --template <path/to/your/template/file>
---- initialize a project with template file

swift package init
---- initialize a project interactively

swift package template <path/to/your/project>
---- create a template file using existing projects

In my opinion, the project can be split into 3 parts: Use(user-friendly CLI), Share(file format) and Customize(API).

[Use (user-friendly CLI) ]

For a user-friendly CLI, I think frontend people know it best. In Vite(a frontend tool for initialize a react or vue project), the CLI is very user-friendly.

Using arrow-keys and return to guide the initialization of the project has always been interactive, intuitive and easy-to-use to me.

With the current version of swiftPM, if we enter the swift package init command, a library package is created by default.

It would be better if after entering this command, we could interactively select the type of package we want to create.

[Share (file format) ]

As for sharing, I think a .stmpl (i.e. swift template, or anything you want it to be) format could be introduced. A program template consists mainly of directories and text files, and I think the xml format would be better for very large pure code templates.

With the .stmpl file in hand, we can initialize a project, with a customized myTemplate.stmpl, fast and easy, using --template option like this:

Or just swift package init:

And because .stmpl file itself contains interactable information, we can select a set of dependencies to include, and pick between including tests or not.

[Customize (API) ]

Now, for users who want to create their own templates, it’s good to have an API to empower that. Instead of an API in the form of functions, I think we can create a Template.swift just like Package.swift, where users configure attributes(for example, include specific directories by default, interactability) of a template, just like Package.swift.

With a simple swift package template </path/to/your/project> command, swiftPM reads the configurations users make, including which directories to include, which directories can be included interatively and which dependencies to include(essencially writing Package.swift file). Users can then create their customized templates(.stmpl), and share them with people.

In Template.swift, for example, you can define:

1. [name] Name a template using name attribute
2. [testDirectory] Select a directory for testing, specify its relative path using “.path: /Your/Path”. Use “.toggle” to make including it interactable like this:
   
   

3. [directories] In directories array, determine whether to include a directory by default using “defaultInclude:”, again, use “toggle:” to let user include it interactively

4. [dependencies] In dependencies array, use “url:” to specify which dependencies to include, use “defaultInclude” to determine include it by default, use “toggle:” to let user include this dependency interactively.

Current progress

I started with the simplest and implemented the interactive swift package init command (Very rough implementation, the code structure will be optimized in the future)

You can watch a video demo here:

[GSoC-2023]SwiftPM interactive init demo

Sweet swift forum, please give me feedback

Please do not hesitate to give me some advice on how to name the commands, the file format, the properties that Template.swift should support, and the features that you would like to see added. Thank you very much for your participation and for seeing the end of the article.

[bitjammer (Ashley Garland)]

Hello, @Howard_Chu! Very cool. I have been prototyping something like what you have proposed and would love to work with you on this as I've been spending a good bit of my time thinking about the command line UX of the tools. I love the general direction. Here's some feedback on what you've proposed here.

swift package init --template <path/to/your/template/file>
---- initialize a project with template file

This seems pretty reasonable. In keeping the user experience in mind, especially of those new to the language, who are incoming all the time: I would encourage you to think of ways where someone does not need to pass a template description at all. That is, once a template is installed, the user shouldn't have to think about where it is installed unless something has gone wrong. I could see passing a template as a power-user ability or for testing template development, though!

The interaction for the package type might use some tweaking visually, but overall looks reasonable. I would personally lean on that and make it fast to get through with reasonable and stable defaults, like not including tests by default for some package types.

@Howard_Chu

As for sharing, I think a .stmpl (i.e. swift template, or anything you want it to be) format could be introduced. A program template consists mainly of directories and text files, and I think the xml format would be better for very large pure code templates.

I would personally not want to see another swift-based description format and especially not for this. If you require structured data for the description, I would suggest just making it JSON as it's easier to read and encode/decode.

@Howard_Chu

And because .stmpl file itself contains interactable information, we can select a set of dependencies to include, and pick between including tests or not.

What ultimately drives the list of possible dependencies and why would you say it is customizable? What are the most common packages that would be included and in what scenarios?

@Howard_Chu

Now, for users who want to create their own templates, it’s good to have an API to empower that. Instead of an API in the form of functions, I think we can create a Template.swift just like Package.swift, where users configure attributes(for example, include specific directories by default, interactability) of a template, just like Package.swift.

Again, unless there needs to be code execution in the template itself, I don't see a need to use Swift source code to define this. Whether it's source files, the resulting Package.swift manifest, or even empty directories, can the template's contents simply be a directory of files, plus a JSON description, that can be modified before being copied into place? I think it's great to leave the door open for more complex needs in the future, but better to start this off humbly in terms of complexity if we can. For example, perhaps we can utilize installable SwiftPM plugins in the future for more complex functionality .

Instead of a PDF, in the future would you please share a Markdown file, perhaps in a GitHub gist or something, with more details with what you're thinking? It makes it easier for folks to access.

Nice job prototyping this! Looking forward to seeing further refinements.

[Howard_Chu]

Thank you so much for your reply, Ashley. First of all, I would love to work with you!

@bitjammer

I would encourage you to think of ways where someone does not need to pass a template description at all. That is, once a template is installed, the user shouldn't have to think about where it is installed unless something has gone wrong. I could see passing a template as a power-user ability or for testing template development, though!

I fully agree that the swift package template command should be used to store the template in a fixed location, so that it can be used directly when initializing in the future (e.g. swift package init). And the --template option seems completely unnecessary, after all it does the same thing as --type. I am sorry that the design is so complicated and unnecessary.

@bitjammer

The interaction for the package type might use some tweaking visually, but overall looks reasonable. I would personally lean on that and make it fast to get through with reasonable and stable defaults, like not including tests by default for some package types.

Sure, I think tweaking is definitely needed, this is only the simplest version implemented for demonstration purposes.

@bitjammer

I would personally not want to see another swift-based description format and especially not for this. If you require structured data for the description, I would suggest just making it JSON as it's easier to read and encode/decode.

After thinking about it, I also think that such a format is not necessary. The reason I thought so is that in the official idea list (probably written by boris), it says "In this project, we will define a format for templates", which I thought was a way to package all the code and folders in a project into a single file. In fact, a zip file with a template.json file describing the interactivity (e.g., specifying the location of the test folder) would solve the problem perfectly, and would be much simpler to use.

I am sorry for this design mistake, as I started writing the proposal very late due to health issues, so I did not get enough feedback.

If that's so, there is only one use for this command, swift package template </path/to/your/project>, that is to copy the project as a template into a pre-defined fixed location for the user to call init later. As for sharing, users can just send zip files to each other directly. (just like downloading a template from some iOS App Templates websites) The only thing that is worth mentioning is that the root path of the project can contain a template.json file for possible interactions.

@bitjammer

What ultimately drives the list of possible dependencies and why would you say it is customizable? What are the most common packages that would be included and in what scenarios?

I'm sorry to have to say that I don't know. I don't think it's so difficult to write Package.swift files by hand that you need to include dependencies interactively. Moreover, when users can design their own templates, they can modify the Package.swift file first and then use the swift package template to store it for future use. If you say that writing Package.swift is too difficult for newcomers, it is better to include dependencies interactively, but how will newcomers know which dependencies to include, and which not to include? So I don't think interactive include dependencies is a necessary feature. Sorry, I'm kinda limited by the description in the idea list.

@bitjammer

Again, unless there needs to be code execution in the template itself, I don't see a need to use Swift source code to define this. Whether it's source files, the resulting Package.swift manifest, or even empty directories, can the template's contents simply be a directory of files, plus a JSON description, that can be modified before being copied into place? I think it's great to leave the door open for more complex needs in the future, but better to start this off humbly in terms of complexity if we can. For example, perhaps we can utilize installable SwiftPM plugins in the future for more complex functionality .

Definitely, I think this is the best approach. Customize is entirely the user's job and swiftPM only helps store it in a fixed place for quick initialization, with some interactivity like whether to add the test folder or not.

@bitjammer

Instead of a PDF, in the future would you please share a Markdown file, perhaps in a GitHub gist or something, with more details with what you're thinking? It makes it easier for folks to access.

Of course, I'll write up a markdown file and make it available soon.

@bitjammer

Nice job prototyping this! Looking forward to seeing further refinements.

Thank you very much, although I think most of my proposals are too complex and lack practicality, I am sorry for not thinking about the problem properly, and I promise to iterate quickly.

Also, I don't really understand what real practical interactivity there is, other than whether to include the test folder or not. I'll create a post asking for opinions on the forum.

[bitjammer (Ashley Garland)]

@Howard_Chu

If that's so, there is only one use for this command, swift package template </path/to/your/project>, that is to copy the project as a template into a pre-defined fixed location for the user to call init later.

It's possible you might get away with not needing this command either, as long as the location(s) for installing a template are well known. Something to think about.

@Howard_Chu

So I don't think interactive include dependencies is a necessary feature.

Even so, it doesn't hurt to think about it now! I would say keep thinking about the packages that are often used. Also, consider Swift Algorithms, Swift Numerics, etc. If there is an objective class of packages that you can reference, it might work in the future. For example, perhaps in the future we could ask a question like, "Would you like to add a dependency on one of Swift's 'core libraries'?" or something like that. If we can identify what those kinds of packages are in a fair way, it could make sense.

@Howard_Chu

Also, I don't really understand what real practical interactivity there is, other than whether to include the test folder or not. I'll create a post asking for opinions on the forum.

I think a great place to start is the command line usage itself. For example, if someone passed a very complete and perfect set of arguments, then interactivity would be unnecessary. So, it helps to think of the interactivity as generating a list of arguments that will then ultimately be passed to the tool, a reminder of what's possible for those that either don't know or forgot.

@Howard_Chu

Thank you very much, although I think most of my proposals are too complex and lack practicality, I am sorry for not thinking about the problem properly, and I promise to iterate quickly.

No need to apologize! On the contrary, you're already thinking a lot about this and it's great to see. Whether it's coming up with ideas or scoping things to a small incremental step, we're all learning together here. We want to see you succeed. All the best.

[sebsto (Sébastien Stormacq)]

Thank you for the proposal. This would be very convenient indeed to be able to scaffold different types of projects from the command line.

I am actually considering doing so for AWS Lambda functions, where a simple Lambda function requires some dependency but also a code skeleton. Different code skeletons might be needed depending on the use case (is the Lambda function triggered by a URL call, a REST API, or a message in queue?)

Unless I missed it, the template should also contain code skeleton that would allow the developer to start with a Hello Wolrd code without effort.

One approach I saw in many similar tools is to host the project skeleton / template on a public git repositories and let the CLI clone a project from a know URL. That approach would simplify the creation of templates (a template is a git repo or a directory withing a git repo) and can be easily maintained independently of the plugin / spm itself

The challenge is to find mechanisms to scale : who / how de we do the gatekeeping on these templates (should we?) , how to discover them ?

The package repo might be a solution to this problem too.

[Howard_Chu]

Sorry for replying late this week due to schoolwork.

@bitjammer

It's possible you might get away with not needing this command either, as long as the location(s) for installing a template are well known.

It's entirely possible. For newcomers, I would think it may not be too easy to find the location(s) (finding such location(s) most likely requires reading the documentation, or print it out using the --help command option). This is where I think @sebsto's idea may come in handy. If swift package init can support cloning a remote project using URL, it can be quite easy to use. If so, for ease of understanding, do you think we should add a --url option for using remote templates, or stick with --type? The difference between an init command and git clone will be that init command can decode the config file and enters interactive mode.

@bitjammer

I would say keep thinking about the packages that are often used.
If we can identify what those kinds of packages are in a fair way, it could make sense.

I will definitely look into it, i think it's a great idea. I could also imagine power users creating highly customized templates that can quickly include different types of dependencies, for it can be faster interactively than to rewrite Package.swift every time. What do you think?

@bitjammer

I think a great place to start is the command line usage itself.
So, it helps to think of the interactivity as generating a list of arguments that will then ultimately be passed to the tool, a reminder of what's possible for those that either don't know or forgot.

It's a great way of thinking about it. Although some of the arguments may go beyond the scope of this projects(customizable templates), I think it's definitely worth consideration.

Starting with the basic swift package init we got these arguments:

--type <type>           Package type: (default: interact)
        library           - A package with a library.
        executable        - A package with an executable.
        tool              - A package with an executable that uses
                            Swift Argument Parser. Use this template if you
                            plan to have a rich set of command-line arguments.
        build-tool-plugin - A package that vends a build tool plugin.
        command-plugin    - A package that vends a command plugin.
        macro             - A package that vends a macro.
        empty             - An empty package with a Package.swift manifest.
  --name <name>           Provide custom package name
  --version               Show the version.
  -h, -help, --help       Show help information.

Obviously, no one has interest in interacting with --version or --help, which leaves us with --type and --name. Making --type interactive is a must-have, but --name is interesting. In the past, it was customary to create a folder manually and then cd and init. The process of creating a folder can be omitted by interacting with --name, which is also the approach of many project initialization tools. This idea was mentioned by @Diggory and @hassila in Update to swift package init templates. I personally like this idea too, it would be a nice experience if swiftPM automatically creates the project folder with users only have to enter the project name.

Another potential interaction is adding plugins like swift-docc-plugin or custom plugins by specifying path, which has also been mentioned in the post above and by @filip-sakel, so I'm only recapping it.

I will continue to write a markdown file to elaborate on the details while updating my proposal. Thank you for your response.

[Howard_Chu]

Just wrote a markdown file to changed the content in the proposal, and described some details of the interactive cli implementation. If you are interested, please take a look at it.

[Alejandro_Martinez]

Hello,
I'm curious if there is any plan to have the interactive part in a separate library so other CLIs can benefit from it.

As I mentioned in another post the Swift ecosystem lacks a powerful library for interactive CLIs that's is as easy to use as libs in other ecosystems.

We have an amazing library for CLI arguments with GitHub - apple/swift-argument-parser: Straightforward, type-safe argument parsing for Swift, I would love to have access to a such high-quality lib but for interactive clis. ^^

[Howard_Chu]

I think that's honestly the best way to do it.

SwiftPM also uses swift-argument-parser to parse arguments, it obviously makes more sense to add interactive API in swift-argument-parser than to customize one specifically for swiftPM.

Although this project is about customizable package templates, contributing to swift-argument-parser is potentially beyond the scope of the project, I will definitely discuss this with my mentor if I am lucky enough to be selected to participate in GSoC-2023.

Thank you very much for your advice!

[Howard_Chu]

The results just came in and I'm sorry to say I was rejected. Please be supportive of the people who were chosen to implement this feature. I will contribute to swift if I get a chance. Thank you all for your support.

[Rajveer100 (Rajveer Singh Bharadwaj)]

Same here, even I got rejected for the C++/Swift Interoperability project.

I was mainly excited that I could get a chance to contribute for large scale projects, while I am already an active contributor.

[ktoso (Konrad 'ktoso' Malawski 🐟🏴‍☠️)]

Hi there and thank you for your great proposals! It’s always though selecting the projects and as usual it was a though competition — thank you all for taking the time and I hope you’ll remain active in the community regardless of the gsoc results here. We’ll also try to improve the contributor experience for people outside gsoc, so please don’t hesitate to reach out at any time

We’ll post an official summary of the accepted projects shortly. Thanks!

[Rajveer100 (Rajveer Singh Bharadwaj)]

Thanks for the message @ktoso!