Supporting SwiftSyntax on more targets

Currently, SwiftSyntax is only supported on Darwin. Given that this library is written using Swift, it does take advantage of Foundation for some functionality (and thus, indirectly, on libdispatch and CoreFoundation).

Having the library be available on other platforms makes it possible to write tools that can work with swift code on other targets (e.g. Linux). This means that it is possible to be able to develop across Linux or Darwin.

There exists a swift implementation of Foundation (swift-corelibs-foundation) and it would be possible to use that to provide Foundation on targets that do not have an Objective-C implementation. It may be interesting to see if it is possible to use that implementation everywhere in the future.

Another option would be to remove the dependency on Foundation from SwiftSyntax. While certainly an option, I think that Foundation is meant to abstract the differences of the underlying platform, and as such, is a useful tool to have at that level. It protects SwiftSyntax from having to implement things like process launching for each target.

I think that having an option to control if SwiftSyntax is built would be a good way to introduce a soft dependency on Foundation. The option would default to true on Darwin (maintaining the status quo). On Linux, if the swift-corelibs-foundation tree is present, we would build it, otherwise we would not. If the tree is present, it would allow us to enable the SwiftSyntax build by default. Having the option exposed means that the user can make an explicit call if they like.

This would allow us to make progress on enabling SwiftSyntax on other platforms, while also allowing a bit more time to determine the best path forward longer term (remove the dependency on Foundation or embrace it). It could be that using Foundation really helps simplify the SwiftSyntax source tree. In the mean time, it also would serve as a test for the swift implementation, helping expose latent issues (as it did with NSFIleHandle and NSPipe).

Another option may be to split SwiftSyntax out of the core swift repository. It may make sense to do that as the library is not directly integrated into the compiler build.

With the option to control if the library is built, we can continue to make progress in building and enabling SwiftSyntax on other hosts. It allows for different sets of developers to work on different aspects simultaneously, which would be a boon to productivity.

CC: @Xi_Ge, @tkremenek, @Michael_Gottesman, @DaveZ

Hi @compnerd,

If you do go down the "soft dependency" path, all I ask is that you be careful when changing the build system. Specifically:

  1. The CMake files should be insensitive to build-script changes.
  2. The build-script should not duplicate logic that CMake provides or can provide.

In practice, we're fairly good about #1 and somewhat bad about #2. I'd like to see the latter improve, but I don't have any time to work on it. At the very least, I don't want #2 to regress more than needed.

@DaveZ,

I completely agree that we should avoid duplicating the logic in the build-script for this would be bad :-). The way that I plan to implement this is to actually only teach the build-script an additional bit of the tree layout. It would pass to CMake the location for the checkout (as we already do for libdispatch). Everything else would be taken care of in the CMake, which would allow us to re-use existing CMake support and even track dependencies.

Foundation is our official library for stuff like that, so SwiftSyntax should use it too if possible IMO (like SwiftPM does).

Perhaps that is the best option - again, it could be handled like SwiftPM (another compiler tool, written in Swift).

1 Like

Hey @compnerd,
Thank you for proposing this! Here're some of my thoughts:

  • Building SwiftSyntax on other platforms is an important goal and we should make it happen.

  • SwiftSyntax's depending on Foundation isn't a strict requirement. SwiftSyntax provides a structured way of viewing/modifying Swift code by using Swift. It doesn't necessarily required Foundation from any logic point of view. In other words, avoiding depending on Foundation is acceptable. IMO, the benefit of doing so (self-contained Swift module) outweighs the benefits (using URL in API, etc.).

  • I may miss something in your proposal. But i think working towards dropping Foundation dependency doesn't necessarily require a build flag controlling SwiftSyntax. Could you elaborate how this flag can make the work easier?

@Xi_Ge,

The build flag controlling SwiftSyntax allows some people to move forward with building and testing SwiftSyntax on Linux. In parallel, someone else can work on removing the Foundation dependency. It just allows multiple parallel items of work to be done. If you feel that it is a small enough thing to remove the dependency on Foundation and you can just get that committed in a day or so, then the calculus changes and it would not make as much sense to have the additional option.

It's probably worth noting that a solution that substitutes Foundation for Unix/Posix APIs is less than ideal. Eventually, SwiftSyntax should be able to run on Windows as well; having all of the platform abstraction logic contained in Foundation minimises the number of changes that have to be made to support it (and other non-Unix platforms).

On the other hand cutting a few of Foundation's tentacles may be a good thing long term. In my opinion leveraging Foundation was great for bootstrapping, but as Swift matures it becomes more of a drag.

It may be time to start moving essential stuff from Foundation, into the standard library (see also Move Foundation.Data in to the standard library)

2 Likes

OK, thank you for the explanation. If this build flag helps us to incrementally finish the task of building SwiftSyntax on linux, please go ahead and add it (with the default to be true). Please keep in mind that we need to remove this flag once SwiftSyntax builds on linux. Quoting from @harlanhaskins's previous investigation, we need to remove the following Foundation types to remove the dependency:

  • URL , for file paths and loading swiftc
    • We could accept string paths, which is an unfortunate tradeoff IMO.
  • Process and related communications APIs
    • We could use posix_spawn directly, bypassing Foundation
  • Codable
    • We can use #18497 instead of Codable, which I think would be a mistake because the JSON output enables external clients to consume the Syntax tree in a structured way without having to implement a hand-written binary deserializer.

Are you planning to tackle these tasks as well?

IMHO, it would be bad if SwiftSyntax drops dependency on Foundation as it makes things difficult for its clients (and also for its potential contributors). Interfacing directly with C-APIs like posix_spawn is difficult in Swift and we would essentially end up re-implementing some of the functionality provided by Foundation. Also, given it is a Swift project, it should fully embrace Foundation OR Foundation should really be a part of the stdlib. I've only skimmed the thread but it sounds like its a build order problem. If thats the case, we should be able to solve it similar to what we've done for other Swift-based projects.

1 Like

I agree that we probably shouldn't re-implement some platform-independent APIs that Foundation can provide us, like Process. On the other hand, SwiftSyntax uses Process to invoke the compiler to generate an internal representation of the tree, which is arguably a proper responsibility of the SwiftSyntax module at all.

IMHO, SwiftSyntax should keep lean and be only responsible for interpreting SyntaxTree raw data. And we should have another higher-level library, which depends on Foundation and other cool libraries, to invoke compiler/sourcekit to generate the raw data. The client side can combine the high-level library with SwiftSyntax to implement user cases.

1 Like

I never even realized SwiftSyntax only supported a limited subset of platforms.

Since it still isn’t available in any stable binary release, I just dragged the source from the corresponding release commit into my own package as a temporary shim. It more or less just worked, and I have been using it on Linux.

I hope it will soon be an integral part of Swift on all platforms (much like Foundation is now).

(I don’t particularly care what SwiftSyntax does or does not depend on, but I’m not convinced there is any value to avoiding Foundation.)

For handling paths, we will need to have some abstraction over path component separators, drives, etc. The paths on Windows are different, and due to the limitation of the default APIs, it is preferable to use some of the lower level APIs which life some of the restrictions on the path limit.

I don't think just using posix_spawn works. Windows doesn't support that, but does have CreateProcessExW. So, we will need a more comprehensive replacement for it.

No strong objection to using a custom serialization format, but will point out that it does make it harder to write things to interact with it.

I think that dropping the Foundation dependency from swiftSyntax, while possible, would be fairly unwise.

URL , for file paths and loading swiftc

This is probably the easiest change, but I think swiftSyntax as a native Swift library should set a good example of the usage of APIs. As such, clients would expect it to use URL for file paths and moving back to String feels like a step into old Objective-C days to me.

Process and related communications APIs

Reimplementing process spawning is likely to become fragile and it's a duplication of the work that has already been done for Foundation. Especially if we consider supporting more platforms (e.g. Windows) in the future, we would always need to duplicate our efforts and test them once again in swiftSyntax.

Codable

While we could use the binary transfer format of #18497 to transfer the syntax tree, I feel like having JSON as the transfer format may turn out to be really useful in the future.
a) If we continue supporting JSON in swiftSyntax, it is continually tested in CI and clients who want to use the tree but not swiftSyntax will be able to consume the JSON emitted by the compiler
b) For debugging purposes: Should anything in the binary format ever break, it's good to have a secondary human-readable format at hand that we can inspect to see if the bug is in the binary format or the structure of the syntax tree
c) We currently only support serialising a syntax tree that only exists in swiftSyntax to JSON. Thus if we switch to the binary format, we would need to either support serialising the syntax tree to that format or drop support for saving syntax trees that were generated using the factory methods in swiftSyntax.

1 Like

Hey all, after some discussions we believe it will be beneficial to split SwiftSyntax in a separate repo out of the core swift one. Specifically:

  • The scripts that specify the nodes will stay in the swift repo and will be used to generate the C++ part of libSyntax
  • The swift-syntax repo will contain the handwritten swift code and depend on the swift repo's scripts for generating the code-generated SwiftSyntax files. Main development for SwiftSyntax will happen on this repo, unless you need to modify the nodes or the C++ gyb files
  • The swift-syntax repo will also contain tags that will include the code-generated swift files so that a SwiftPM package can point to it and be able to build SwiftSyntax without the swift repo being present

We'll post more details in the forums once they are available.

12 Likes

Do you think it will be beneficial to split the standard library in another repo too ?

Does the points you raise apply to other swift libraries that are in the main swift repo ?

I think that's a question for Standard Library - Swift Forums.
Note that one of the primary motivations to split SwiftSyntax is so that SwiftPM packages can depend on it, this doesn't apply to standard library.

SwiftSyntax is in its own repo, more details here: SwiftSyntax is now a SwiftPM project

1 Like