Shipping sourcekitd-test/repl with Swift

Hey everyone,

Recently I've been working on making Swift autocomplete outside of Xcode
(specifically vim). Of course to do this, I've been using [SourceKitten][0],
which is a great bridge for interacting with `sourcekitd`.

While working on this, I also ran across `sourcekitd-test` and `sourcekitd-repl`
from the Swift repo. These tools are also awesome for working with `sourcekitd`.
`sourcekitd-test` even has practically the same command line interface as
SourceKitten's complete command.

With `sourcekitd-test`:

$ sourcekitd-test -req=complete -offset=x file.swift -- [compiler args]

With SourceKitten:

$ sourcekitten complete --offset x --file file.swift -- [compiler args]

These 2 commands of course call through to `sourcekitd` in the same way, so this
ends up with the same output as well.

All of this is just to show that I think these tools would be extremely valuable
to have shipped with whichever Swift toolchains are bundled with Xcode, so users
would automatically have tools for completion installed.

I'd love to hear some thoughts on this, and also if it's even a feasible thing
to ask for. Also let me know if this post would be better suited for another
list. I didn't feel like this was particularly appropriate for swift-evolution
since there aren't really any implementation details in question here.

[0]: https://github.com/jpsim/SourceKitten

Thanks for reading!

ยทยทยท

--
Keith Smiley

2 Likes

Sorry to bump this thread, just wondering if anyone has any thoughts on
this now that (some) of the Swift 3.0 craziness is done.

ยทยทยท

--
Keith Smiley

On Sun, Jul 31, 2016, at 21:46, Keith Smiley via swift-dev wrote:

Hey everyone,

Recently I've been working on making Swift autocomplete outside of Xcode
(specifically vim). Of course to do this, I've been using
[SourceKitten][0],
which is a great bridge for interacting with `sourcekitd`.

While working on this, I also ran across `sourcekitd-test` and
`sourcekitd-repl`
from the Swift repo. These tools are also awesome for working with
`sourcekitd`.
`sourcekitd-test` even has practically the same command line interface as
SourceKitten's complete command.

With `sourcekitd-test`:

$ sourcekitd-test -req=complete -offset=x file.swift -- [compiler args]

With SourceKitten:

$ sourcekitten complete --offset x --file file.swift -- [compiler args]

These 2 commands of course call through to `sourcekitd` in the same way,
so this
ends up with the same output as well.

All of this is just to show that I think these tools would be extremely
valuable
to have shipped with whichever Swift toolchains are bundled with Xcode,
so users
would automatically have tools for completion installed.

I'd love to hear some thoughts on this, and also if it's even a feasible
thing
to ask for. Also let me know if this post would be better suited for
another
list. I didn't feel like this was particularly appropriate for
swift-evolution
since there aren't really any implementation details in question here.

[0]: https://github.com/jpsim/SourceKitten

Thanks for reading!

--
Keith Smiley

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Before SourceKit-related functionality can be considered for inclusion in
release packages for Linux, it needs to be able to be built during "normal"
invocation of the build script. I say normal because it's possible to build
SourceKit with two (simple) invocations of the build script at the moment:
https://github.com/apple/swift/pull/3594#issuecomment-234169759

The reason SourceKit can't currently be built in a single pass is that it
depends on libdispatch, which has a dependency on Swift for its overlay and
so is built _after_ Swift, but SourceKit is part of the Swift project. This
is only a problem on Linux because SourceKit can use the system's
libdispatch rather than the corelibs one when building on OS X.

Here are different ideas for addressing this circular dependency:

1. Break SourceKit out into a different CMake "project". This way, the
build script would build Swift -> libdispatch -> SourceKit. This is
probably the most elegant solution.
2. Build libdispatch twice: once before building Swift (without its
overlay) and again with its overlay. This is probably the easiest and most
naive way to solve this, but is potentially wasteful since we're building
it twice. Actually probably not, assuming the build cache is reused
effectively. But it's certainly not pretty.
3. Remove SourceKit's dependency on libdispatch. I dislike this option
because rewriting the libdispatch parts of SK with libpthread or C++14
builtins would be a sizable undertaking and the codebase would suffer for
it.

I'm sure there are other approaches that could work but I'm not aware of
them.

This is being tracked as SR-1676, which is currently unassigned:
https://bugs.swift.org/browse/SR-1676

ยทยทยท

On Mon, 29 Aug 2016 at 20:25 Keith Smiley via swift-dev <swift-dev@swift.org> wrote:

Sorry to bump this thread, just wondering if anyone has any thoughts on
this now that (some) of the Swift 3.0 craziness is done.

--
Keith Smiley

On Sun, Jul 31, 2016, at 21:46, Keith Smiley via swift-dev wrote:
> Hey everyone,
>
> Recently I've been working on making Swift autocomplete outside of Xcode
> (specifically vim). Of course to do this, I've been using
> [SourceKitten][0],
> which is a great bridge for interacting with `sourcekitd`.
>
> While working on this, I also ran across `sourcekitd-test` and
> `sourcekitd-repl`
> from the Swift repo. These tools are also awesome for working with
> `sourcekitd`.
> `sourcekitd-test` even has practically the same command line interface as
> SourceKitten's complete command.
>
> With `sourcekitd-test`:
>
> ```
> $ sourcekitd-test -req=complete -offset=x file.swift -- [compiler args]
> ```
>
> With SourceKitten:
>
> ```
> $ sourcekitten complete --offset x --file file.swift -- [compiler args]
> ```
>
> These 2 commands of course call through to `sourcekitd` in the same way,
> so this
> ends up with the same output as well.
>
> All of this is just to show that I think these tools would be extremely
> valuable
> to have shipped with whichever Swift toolchains are bundled with Xcode,
> so users
> would automatically have tools for completion installed.
>
> I'd love to hear some thoughts on this, and also if it's even a feasible
> thing
> to ask for. Also let me know if this post would be better suited for
> another
> list. I didn't feel like this was particularly appropriate for
> swift-evolution
> since there aren't really any implementation details in question here.
>
>
> [0]: https://github.com/jpsim/SourceKitten
>
>
> Thanks for reading!
>
> --
> Keith Smiley
>
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Thanks for the context JP! Maybe we should circle back once that has been
figured out then.

ยทยทยท

--
Keith Smiley

On 08/30, Jean-Pierre Simard wrote:

Before SourceKit-related functionality can be considered for inclusion in
release packages for Linux, it needs to be able to be built during "normal"
invocation of the build script. I say normal because it's possible to build
SourceKit with two (simple) invocations of the build script at the moment:
https://github.com/apple/swift/pull/3594#issuecomment-234169759

The reason SourceKit can't currently be built in a single pass is that it
depends on libdispatch, which has a dependency on Swift for its overlay and
so is built _after_ Swift, but SourceKit is part of the Swift project. This
is only a problem on Linux because SourceKit can use the system's
libdispatch rather than the corelibs one when building on OS X.

Here are different ideas for addressing this circular dependency:

1. Break SourceKit out into a different CMake "project". This way, the
build script would build Swift -> libdispatch -> SourceKit. This is
probably the most elegant solution.
2. Build libdispatch twice: once before building Swift (without its
overlay) and again with its overlay. This is probably the easiest and most
naive way to solve this, but is potentially wasteful since we're building
it twice. Actually probably not, assuming the build cache is reused
effectively. But it's certainly not pretty.
3. Remove SourceKit's dependency on libdispatch. I dislike this option
because rewriting the libdispatch parts of SK with libpthread or C++14
builtins would be a sizable undertaking and the codebase would suffer for
it.

I'm sure there are other approaches that could work but I'm not aware of
them.

This is being tracked as SR-1676, which is currently unassigned:
https://bugs.swift.org/browse/SR-1676

On Mon, 29 Aug 2016 at 20:25 Keith Smiley via swift-dev <swift-dev@swift.org> > wrote:

> Sorry to bump this thread, just wondering if anyone has any thoughts on
> this now that (some) of the Swift 3.0 craziness is done.
>
> --
> Keith Smiley
>
> On Sun, Jul 31, 2016, at 21:46, Keith Smiley via swift-dev wrote:
> > Hey everyone,
> >
> > Recently I've been working on making Swift autocomplete outside of Xcode
> > (specifically vim). Of course to do this, I've been using
> > [SourceKitten][0],
> > which is a great bridge for interacting with `sourcekitd`.
> >
> > While working on this, I also ran across `sourcekitd-test` and
> > `sourcekitd-repl`
> > from the Swift repo. These tools are also awesome for working with
> > `sourcekitd`.
> > `sourcekitd-test` even has practically the same command line interface as
> > SourceKitten's complete command.
> >
> > With `sourcekitd-test`:
> >
> > ```
> > $ sourcekitd-test -req=complete -offset=x file.swift -- [compiler args]
> > ```
> >
> > With SourceKitten:
> >
> > ```
> > $ sourcekitten complete --offset x --file file.swift -- [compiler args]
> > ```
> >
> > These 2 commands of course call through to `sourcekitd` in the same way,
> > so this
> > ends up with the same output as well.
> >
> > All of this is just to show that I think these tools would be extremely
> > valuable
> > to have shipped with whichever Swift toolchains are bundled with Xcode,
> > so users
> > would automatically have tools for completion installed.
> >
> > I'd love to hear some thoughts on this, and also if it's even a feasible
> > thing
> > to ask for. Also let me know if this post would be better suited for
> > another
> > list. I didn't feel like this was particularly appropriate for
> > swift-evolution
> > since there aren't really any implementation details in question here.
> >
> >
> > [0]: https://github.com/jpsim/SourceKitten
> >
> >
> > Thanks for reading!
> >
> > --
> > Keith Smiley
> >
> > _______________________________________________
> > swift-dev mailing list
> > swift-dev@swift.org
> > https://lists.swift.org/mailman/listinfo/swift-dev
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev
>

It seems that the build-toolchain script at least does build SourceKit now and includes libsourcekitdInProc.so in the resulting zip archive on Linux (as tested on Ubuntu 18.04 with the current master).

Including sourcekitd-test or something like it would therefore still be very useful.

3 Likes