RFC: Removing the integrated REPL

Development Compiler
1

Hi all,

Swift currently has two REPL (read-eval-print-loop) implementations:

  • The LLDB REPL, which is what you normally see when you download a toolchain from Swift.org, or install Xcode on a Mac, and run "swift".

  • The so-called "integrated REPL", which is what you see if you run "swift" and LLDB is not available, most likely because you built Swift yourself without also building LLDB.

We would like the remove the integrated REPL. Unless you are aware of its existence, you're probably not using it and it won't affect you.

If you are using it, we would love to hear why so that we can understand the use cases not accommodated by the LLDB REPL.

20 Likes
2

I wasn’t even aware that there was more than one implementation. I’ve never gone out of my way to not build LLDB, so I guess I‘m not relying on the integrated REPL.

Out of curiosity, what does the integrated version do that the LLDB version doesn’t? I mean, if they‘re feature-identical, is there even a reason to debate this? Is the only user-facing difference just that one doesn’t require LLDB to be installed?

3

Isn’t the integrated REPL used in macOS Catalina for #!/bin/swift execution? Swift as a scripting language is available in Catalina without installing the Swift compiler. This is expected to be used by IT/DevOps since it does not require Xcode to be installed. This is likely to supplant Python somewhat as an IT scripting language since Python will no longer be installed by default in the future. It is already being used for example scripts in some of Apple’s IT documentation.

EDIT: Even if it is not required for Swift as a scripting language, I think it is still useful to have a REPL built in to Catalina without Xcode installed.

4 Likes
4

Hi,

I sometimes use integrated REPL explicitly as swift -frontend -repl because it works in a docker container launched without the --privileged option.
If integrated REPL is removed, it would be nice if LLDB REPL could be invoked in a docker container without the --privileged option.

FYI, how to try integrated REPL on macOS:

$ swift -frontend -repl -sdk $(xcrun -sdk macosx -show-sdk-path)
***  You are running Swift's integrated REPL,  ***
***  intended for compiler and stdlib          ***
***  development and testing purposes only.    ***
***  The full REPL is built as part of LLDB.   ***
***  Type ':help' for assistance.              ***
(swift) :help
Available commands:
  :quit - quit the interpreter (you can also use :exit or Control+D or exit(0))
  :autoindent (on|off) - turn on/off automatic indentation of bracketed lines
  :constraints debug (on|off) - turn on/off the debug output for the constraint-based type checker
  :dump_ir - dump the LLVM IR generated by the REPL
  :dump_decl <name> - dump the AST representation of the named declarations
  :dump_source - dump the user input (ignoring lines with errors)
  :print_decl <name> - print the AST representation of the named declarations
  :print_module <name> - print the decls in the given module, but not submodules
API documentation etc. will be here eventually.
(swift)

Changed command line syntax from fish to zsh

11 Likes
5

How (if at all) will this affect the swift test suite when LLDB is not built/available?

6

What you're describing is called "immediate mode" and is distinct from the integrated REPL. Removing the REPL will not affect the ability to run scripts.

Edit: Just saw your edit :) When you run "/usr/bin/swift", it will either prompt to download the command-line developer tools, or use xcrun to find swift from an Xcode install. Even the command line tools ship LLDB so to the best of my knowledge you will always get the LLDB REPL.

3 Likes
7

Other than a handful of tests that specifically test the REPL, it should have no effect.

2 Likes
8

I would love this simplification! When I started learning about how the REPL worked, it was confusing to have two of them.

Will you keep the "-repl-code-completion" tests? They test REPLCodeCompletion, which is used by the LLDB REPL, so it would be good to keep them.

9

I would expect we keep these tests. There's nothing in there that should be specific to the integrated repl.

10

That's an excellent question. I have no idea. I think @dcci should know the answer here.

(The LLDB REPL uses multiple processes, with the main process JITting code that is run in the child Swift process. There is also a debugserver running. So I suspect some level of special privileges are necessary, but maybe it's possible to do something more fine-grained)

1 Like
11

It has a slightly different set of built-in commands available, more geared toward dumping AST declarations (it's quite incomplete though). Mostly I think LLDB offers a superset of the functionality, but with a different presentation.

1 Like
12

Even if it is not required for Swift as a scripting language, I think it is still useful to have a REPL built in to Catalina without Xcode installed.

Note that by default you do not have a real install of /usr/bin/swift. That dummy executable will kick xcrun and prompt you to install the command line tools package. At which point you will be running scripts in immediate mode.

1 Like
13

You're right. I just verified on a vanilla test system. Command line tool installation is required for both REPL and immediate mode. I was thinking immediate mode was built-in based on a conference session I attended and lack of mention of this requirement.

1 Like
14

It's not possible to run the lldb REPL without --privileged , as the REPL is simply a wrapper around lldb. So it uses ptrace() and all the debugging mechanisms provided by the OS that require privileges.

3 Likes
15

Possibly relevant article by Julia Evans: a new enough version of Docker may get around this. Why strace doesn't work in Docker

3 Likes
16

Thank you for the information.
I used that blog post as a reference to check with Docker 19.03, but it didn't work. Apparently, REPL requires privileges other than ptrace().

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ docker run -it --rm norionomura/swift:5.2.3 uname -a
Linux 191cfb9666e4 4.14.154-boot2docker #1 SMP Thu Nov 14 19:19:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ docker run -it --rm norionomura/swift:5.2.3 swift
error: failed to launch REPL process: process launch failed: 'A' packet returned an error: 8
$ docker run --cap-add=SYS_PTRACE -it --rm norionomura/swift:5.2.3 swift
error: failed to launch REPL process: process launch failed: 'A' packet returned an error: 8
$ docker run --privileged -it --rm norionomura/swift:5.2.3 swift
Welcome to Swift version 5.2.3 (swift-5.2.3-RELEASE).
Type :help for assistance.
  1> ^D
1 Like
17

The combination that I've needed in Docker - for REPL or LLDB - are:

  • --cap-add=SYS_PTRACE
    (allows the launched process to trace system calls from)
  • --security-opt seccomp=unconfined
    (runs the container without the default security profile)
  • --security-opt apparmor=unconfined
    (runs the container without the AppArmor security profile)
5 Likes
18

Just ran into an issue where I was testing some stuff after building swift and noticed this was removed. I didn't realize that my constant bin/swift usage to test stuff was part of the integrated REPL. Little disappointed that I can't do that anymore, but I understand that I'm a minority that uses this (also helps with stack traces when I run into runtime errors). Cheers!

19

@Slava_Pestov shouldn‘t this thread be in #evolution:discuss to reach more people? I don‘t think many of us observe the #development:compiler threads.

20

No, this has purely to do with developing the compiler; there should be no user-facing change.

1 Like