Better integration with UNIX tools

Swift is a great shell scripting language except for it's lack of any
API to execute UNIX commands. Compare these two shell scripts:

···

#!/usr/bin/php
<?

$files = `find ~/Desktop -name *.png`;

foreach (explode("\n", $files) as $file) {
  // do something with $file
}

-

#!/usr/bin/swift

import Foundation

let process = Process()
process.launchPath = "/usr/bin/find"
process.arguments = [
  NSString(string:"~/Desktop").expandingTildeInPath,
  "-name",
  "*.png"
]

let output = Pipe()
process.standardOutput = output

process.launch()

let files: String
if let filesUtf8 = NSString(data:
output.fileHandleForReading.readDataToEndOfFile(), encoding:
String.Encoding.utf8.rawValue) {> files = filesUtf8 as String
} else {
  files = NSString(data:
  output.fileHandleForReading.readDataToEndOfFile(), encoding:
  String.Encoding.isoLatin1.rawValue) as NSString! as String> }

files.enumerateLines { file, _ in
  // do something with file
}

It's a contrived example, I could have used NSFileManager, but I
run into this all the time integrating with more complex tools
such as rsync.
Adding my own high level wrapper around the Process command isn't an
option since there is no good way to import code from another file when
executing swift asa shell script. All your code needs to be in one file.
- Abhi

Hi Abhi,

It does seem like there is a possibility of some better convenience API here.

Any ideas on what form it would take? A class method on Process that returns the output, maybe?

- Tony

···

On Nov 16, 2017, at 3:34 PM, Abhi Beckert via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Swift is a great shell scripting language except for it's lack of any API to execute UNIX commands. Compare these two shell scripts:

#!/usr/bin/php
<?

$files = `find ~/Desktop -name *.png`;

foreach (explode("\n", $files) as $file) {
  // do something with $file
}

-

#!/usr/bin/swift

import Foundation

let process = Process()
process.launchPath = "/usr/bin/find"
process.arguments = [
  NSString(string:"~/Desktop").expandingTildeInPath,
  "-name",
  "*.png"
]

let output = Pipe()
process.standardOutput = output

process.launch()

let files: String
if let filesUtf8 = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.utf8.rawValue) {
  files = filesUtf8 as String
} else {
  files = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.isoLatin1.rawValue) as NSString! as String
}

files.enumerateLines { file, _ in
  // do something with file
}

It's a contrived example, I could have used NSFileManager, but I run into this all the time integrating with more complex tools such as rsync.

Adding my own high level wrapper around the Process command isn't an option since there is no good way to import code from another file when executing swift asa shell script. All your code needs to be in one file.

- Abhi
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

I, to, have wished for such an API.

I think that would be a very welcome addition.

···

On Nov 17, 2017, at 11:34 AM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi Abhi,

It does seem like there is a possibility of some better convenience API here.

Any ideas on what form it would take? A class method on Process that returns the output, maybe?

- Tony

On Nov 16, 2017, at 3:34 PM, Abhi Beckert via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Swift is a great shell scripting language except for it's lack of any API to execute UNIX commands. Compare these two shell scripts:

#!/usr/bin/php
<?

$files = `find ~/Desktop -name *.png`;

foreach (explode("\n", $files) as $file) {
  // do something with $file
}

-

#!/usr/bin/swift

import Foundation

let process = Process()
process.launchPath = "/usr/bin/find"
process.arguments = [
  NSString(string:"~/Desktop").expandingTildeInPath,
  "-name",
  "*.png"
]

let output = Pipe()
process.standardOutput = output

process.launch()

let files: String
if let filesUtf8 = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.utf8.rawValue) {
  files = filesUtf8 as String
} else {
  files = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.isoLatin1.rawValue) as NSString! as String
}

files.enumerateLines { file, _ in
  // do something with file
}

It's a contrived example, I could have used NSFileManager, but I run into this all the time integrating with more complex tools such as rsync.

Adding my own high level wrapper around the Process command isn't an option since there is no good way to import code from another file when executing swift asa shell script. All your code needs to be in one file.

- Abhi
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

`Process.run(_:arguments:terminationHandler:)` is not a bad basis for this, other than the first argument being a URL. I might add a variant which does a $PATH search and expands tildes:

  extension Process {
    class func runInPath(_ commandName: String, with arguments: [String], terminationHandler: ((Process) -> Void)? = nil) -> Process
  }

(I would *not* add a variant which simply shells out, or at least I wouldn't make it the only option. Every scripting language I can think of has this feature, and every one of them discourages its use and pushes people towards something else with pre-split arguments and no shell attack surface.)

And then add a method which gathers all stdout output and returns it, along with the termination status (throwing if it's a signal):

  extension Process {
    func outputAfterExit() throws -> (output: String, status: Int32)
  }

The result is not *as* convenient as PHP, but it's a lot safe than running things through a shell, too.

  let (output, _) = try Task.runInPath("find", with: ["~/Desktop", "-name", "*.png"]).outputAfterExit()

The biggest problem I see with this design is that expanding tildes in the arguments, but *not* globbing them, happens to be right for this case, but may not be in the general case. One interesting possibility would be to go the custom operator route:

  /// Returns an absolute path placing `relativePath` in the user's home directory.
  prefix func ~ (relativePath: String) -> String {
    return FileManager.default.homeDirectoryForCurrentUser.appendingPathComponent(relativePath).path
  }

  let (output, _) = try Task.runInPath("find", with: [~"Desktop", "-name", "*.png"]).outputAfterExit()

But I'm not sure we'd want that available in million-line Mac apps.

On the other hand, maybe Foundation's APIs in general are too verbose for scripting. I could imagine a "Foundation.Script" module which added some unprincipled but convenient shortcuts:

  extension URL: ExpressibleByStringLiteral { // Should be interpolatable too, but we need to redesign that.
    public init(stringLiteral value: String) {
      self.init(fileURLWithPath: value)
    }
  }
  public var FS { return FileManager.default }
  public var ENV { return ProcessInfo.processInfo.environment }

That might be a good place for a leading tilde operator, too.

···

On Nov 17, 2017, at 11:34 AM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

It does seem like there is a possibility of some better convenience API here.

Any ideas on what form it would take? A class method on Process that returns the output, maybe?

--
Brent Royal-Gordon
Architechies

Looking at what Python (subprocess) and Go (os.exec) do, it looks like they
agree on the following:
- executable and arguments are merged in one array
- they don't require full path for the executable
- they don't expand tildes
- blocking calls are the default
- they are more explicit about stdin, stdout, stderr

Some example scenarios based on that, with possible swift code:

1) Run a command and ignore output

Python:
    subprocess.run(["sleep", "1"], stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)

Go:
    cmd := exec.Command("sleep", "1")
    err := cmd.Run()

Possible Swift:
    try Process.run(["sleep", "1"])

2) Run a command and capture stdout

Python:
    out = subprocess.check_output(["ls", "-l"])

Go:

    cmd := exec.Command("ls", "-l")
    out, err := cmd.Output()

Possible Swift:

    let proc = try Process.run(["ls", "-l"])
    let out = proc.stdout.read() // proc.stdout is OutputStream, assumes
read() exists and returns Data
    // stderr available at proc.stderr

3) Run a command and capture both stdout and stder together

Python:
    out = subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT)

Go:
    cmd := exec.Command("ls", "-l")
    out, err := cmd.CombinedOutput()

Possible Swift:
    let proc = try Process.run(["ls", "-l"], combinedOutput: true)
    let out = proc.stdout.read()

4) Shell out

Python:
    subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT,
shell=True)

Go:
    cmd := exec.Command("sh", "-c", "ls -l")
    out, err := cmd.CombinedOutput()

Possible Swift:
    let proc = try Process.run(["sh", "-c", "ls -l"], combinedOutput: true)
    let out = proc.stdout.read()

5) Pipe to stdin

Python:
    p = subprocess.Popen(["wc"], stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
    p.stdin.write(b'blah')
    p.stdin.close()
    out = p.stdout.read()

Go:
    cmd := exec.Command("wc")
    stdin, err := cmd.StdinPipe()
    io.WriteString(stdin, "blah")
    out = cmd.CombinedOutput()

Possible Swift:
    let stdin = InputStream(data: "blah".data(using: .utf8))
    let proc = try Process.run(["wc"], stdin: stdin, combinedOutput: true)
    let out = proc.stdout.read()

6) Async

Python:
    p = subprocess.Popen(["sleep", "5"])
    p.wait()

Go:
    cmd := exec.Command("sleep", "5")
    err := cmd.Start()
    err2 := cmd.Wait()

Possible Swift:
    let proc = Process(["sleep", "5"])
    try proc.start()
    try proc.wait()

···

On Fri, Nov 17, 2017 at 9:34 PM, Tony Parker via swift-corelibs-dev < swift-corelibs-dev@swift.org> wrote:

Hi Abhi,

It does seem like there is a possibility of some better convenience API
here.

Any ideas on what form it would take? A class method on Process that
returns the output, maybe?

- Tony

On Nov 16, 2017, at 3:34 PM, Abhi Beckert via swift-corelibs-dev < > swift-corelibs-dev@swift.org> wrote:

Swift is a great shell scripting language except for it's lack of any API
to execute UNIX commands. Compare these two shell scripts:

#!/usr/bin/php
<?

$files = `find ~/Desktop -name *.png`;

foreach (explode("\n", $files) as $file) {
  // do something with $file
}

-

#!/usr/bin/swift

import Foundation

let process = Process()
process.launchPath = "/usr/bin/find"
process.arguments = [
  NSString(string:"~/Desktop").expandingTildeInPath,
  "-name",
  "*.png"
]

let output = Pipe()
process.standardOutput = output

process.launch()

let files: String
if let filesUtf8 = NSString(data: output.fileHandleForReading.readDataToEndOfFile(),
encoding: String.Encoding.utf8.rawValue) {
  files = filesUtf8 as String
} else {
  files = NSString(data: output.fileHandleForReading.readDataToEndOfFile(),
encoding: String.Encoding.isoLatin1.rawValue) as NSString! as String
}

files.enumerateLines { file, _ in
  // do something with file
}

It's a contrived example, I could have used NSFileManager, but I run into
this all the time integrating with more complex tools such as rsync.

Adding my own high level wrapper around the Process command isn't an
option since there is no good way to import code from another file when
executing swift asa shell script. All your code needs to be in one file.

- Abhi
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

I cosign all of this. Nice work, Brent.

···

On Nov 17, 2017, at 4:47 PM, Brent Royal-Gordon via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Nov 17, 2017, at 11:34 AM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

It does seem like there is a possibility of some better convenience API here.

Any ideas on what form it would take? A class method on Process that returns the output, maybe?

`Process.run(_:arguments:terminationHandler:)` is not a bad basis for this, other than the first argument being a URL. I might add a variant which does a $PATH search and expands tildes:

  extension Process {
    class func runInPath(_ commandName: String, with arguments: [String], terminationHandler: ((Process) -> Void)? = nil) -> Process
  }

(I would *not* add a variant which simply shells out, or at least I wouldn't make it the only option. Every scripting language I can think of has this feature, and every one of them discourages its use and pushes people towards something else with pre-split arguments and no shell attack surface.)

And then add a method which gathers all stdout output and returns it, along with the termination status (throwing if it's a signal):

  extension Process {
    func outputAfterExit() throws -> (output: String, status: Int32)
  }

The result is not *as* convenient as PHP, but it's a lot safe than running things through a shell, too.

  let (output, _) = try Task.runInPath("find", with: ["~/Desktop", "-name", "*.png"]).outputAfterExit()

The biggest problem I see with this design is that expanding tildes in the arguments, but *not* globbing them, happens to be right for this case, but may not be in the general case. One interesting possibility would be to go the custom operator route:

  /// Returns an absolute path placing `relativePath` in the user's home directory.
  prefix func ~ (relativePath: String) -> String {
    return FileManager.default.homeDirectoryForCurrentUser.appendingPathComponent(relativePath).path
  }

  let (output, _) = try Task.runInPath("find", with: [~"Desktop", "-name", "*.png"]).outputAfterExit()

But I'm not sure we'd want that available in million-line Mac apps.

On the other hand, maybe Foundation's APIs in general are too verbose for scripting. I could imagine a "Foundation.Script" module which added some unprincipled but convenient shortcuts:

  extension URL: ExpressibleByStringLiteral { // Should be interpolatable too, but we need to redesign that.
    public init(stringLiteral value: String) {
      self.init(fileURLWithPath: value)
    }
  }
  public var FS { return FileManager.default }
  public var ENV { return ProcessInfo.processInfo.environment }

That might be a good place for a leading tilde operator, too.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Any ideas on what form it would take? A class method on Process that
returns the output, maybe?

I think there are two solutions, and I'd like to see both of them taken.
First of all, I disagree with Brent's assessment that we shouldn't
have a variant that "simply shells out". Even though this use case is
"discouraged" in other languages, I think in practice it definitely
has a place.
I would like swift to have something similar to PHP's backticks, with
language syntax to simply execute a shell command - maybe
somethinglike this:

let output = #!"ls ~/Desktop -name *.png"

In the above example, the user's shell PATH variable would be used to
find the `ls` command, the tilde in in the path would be expanded, and
arguments would be split on whitespace, and output would be string with
automatically detected encoding. While this is terrible for complicated
use cases, it is appropriate for simple ones.
For a second solution to handle complex cases, the current API just
needs a lot of cleanup work – especially:
* expanding a tilde should not require working with NSString
   (perhaps NSURL could have an option to expand tildes when created
   from a string) * It should be possible to capture the entire output of a process
   without working with NSPipe and NSFileHandle. Personally I'd prefer a
   block that is executed on completion with NSData arguments for stdout
   and stderr, but there are other options. * converting NSData to a swift string should not require working with
   NSString and it should be possible to do this with automatically
   detected encoding (NSString can do this when reading from a file,
   and CFString can do it reading from NSData. Swift strings should
   have it as well).
- Abhi

···

On Sat, Nov 18, 2017, at 10:47, Brent Royal-Gordon wrote:

On Nov 17, 2017, at 11:34 AM, Tony Parker via swift-corelibs-dev <swift-corelibs- >> dev@swift.org> wrote:

It does seem like there is a possibility of some better convenience
API here.>>
Any ideas on what form it would take? A class method on Process that
returns the output, maybe?>

`Process.run(_:arguments:terminationHandler:)` is not a bad basis for
this, other than the first argument being a URL. I might add a variant
which does a $PATH search and expands tildes:>
extension Process {
class func runInPath(_ commandName: String, with arguments: [String],
terminationHandler: ((Process) -> Void)? = nil) -> Process> }

(I would *not* add a variant which simply shells out, or at least I
wouldn't make it the only option. Every scripting language I can think
of has this feature, and every one of them discourages its use and
pushes people towards something else with pre-split arguments and no
shell attack surface.)>
And then add a method which gathers all stdout output and returns it,
along with the termination status (throwing if it's a signal):>
extension Process {
func outputAfterExit() throws -> (output: String, status: Int32)
}

The result is not *as* convenient as PHP, but it's a lot safe than
running things through a shell, too.>
let (output, _) = try Task.runInPath("find", with: ["~/Desktop", "-
name", "*.png"]).outputAfterExit()>
The biggest problem I see with this design is that expanding tildes in
the arguments, but *not* globbing them, happens to be right for this
case, but may not be in the general case. One interesting possibility
would be to go the custom operator route:>
/// Returns an absolute path placing `relativePath` in the user's home
directory.> prefix func ~ (relativePath: String) -> String {
return FileManager.default.homeDirectoryForCurrentUser.appendingPathC-
omponent(relativePath).path> }

let (output, _) = try Task.runInPath("find", with: [~"Desktop", "-
name", "*.png"]).outputAfterExit()>
But I'm not sure we'd want that available in million-line Mac apps.

On the other hand, maybe Foundation's APIs in general are too verbose
for scripting. I could imagine a "Foundation.Script" module which
added some unprincipled but convenient shortcuts:>
extension URL: ExpressibleByStringLiteral { // Should be
interpolatable too, but we need to redesign that.> public init(stringLiteral value: String) {
self.init(fileURLWithPath: value)
}
}
public var FS { return FileManager.default }
public var ENV { return ProcessInfo.processInfo.environment }

That might be a good place for a leading tilde operator, too.

--
Brent Royal-Gordon
Architechies

Hi Nick,

Thanks for this survey of other languages, it’s very useful.

I think if we were to add something, I would prefer to keep it simple. Just one class method on Process which is a fairly straightforward wrapper of the other functionality already there.

I was thinking of perhaps a completion handler version, with the expectation that once async/await style completions land it would be pretty easy to use in a kind of straight-line mechanism.

- Tony

···

On Nov 20, 2017, at 4:37 AM, Nick Keets <nick.keets@gmail.com> wrote:

Looking at what Python (subprocess) and Go (os.exec) do, it looks like they agree on the following:
- executable and arguments are merged in one array
- they don't require full path for the executable
- they don't expand tildes
- blocking calls are the default
- they are more explicit about stdin, stdout, stderr

Some example scenarios based on that, with possible swift code:

1) Run a command and ignore output

Python:
    subprocess.run(["sleep", "1"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

Go:
    cmd := exec.Command("sleep", "1")
    err := cmd.Run()

Possible Swift:
    try Process.run(["sleep", "1"])

2) Run a command and capture stdout

Python:
    out = subprocess.check_output(["ls", "-l"])

Go:

    cmd := exec.Command("ls", "-l")
    out, err := cmd.Output()

Possible Swift:

    let proc = try Process.run(["ls", "-l"])
    let out = proc.stdout.read() // proc.stdout is OutputStream, assumes read() exists and returns Data
    // stderr available at proc.stderr

3) Run a command and capture both stdout and stder together

Python:
    out = subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT)

Go:
    cmd := exec.Command("ls", "-l")
    out, err := cmd.CombinedOutput()

Possible Swift:
    let proc = try Process.run(["ls", "-l"], combinedOutput: true)
    let out = proc.stdout.read()

4) Shell out

Python:
    subprocess.check_output(["ls", "-l"], stderr=subprocess.STDOUT, shell=True)

Go:
    cmd := exec.Command("sh", "-c", "ls -l")
    out, err := cmd.CombinedOutput()

Possible Swift:
    let proc = try Process.run(["sh", "-c", "ls -l"], combinedOutput: true)
    let out = proc.stdout.read()

5) Pipe to stdin

Python:
    p = subprocess.Popen(["wc"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    p.stdin.write(b'blah')
    p.stdin.close()
    out = p.stdout.read()

Go:
    cmd := exec.Command("wc")
    stdin, err := cmd.StdinPipe()
    io.WriteString(stdin, "blah")
    out = cmd.CombinedOutput()

Possible Swift:
    let stdin = InputStream(data: "blah".data(using: .utf8))
    let proc = try Process.run(["wc"], stdin: stdin, combinedOutput: true)
    let out = proc.stdout.read()

6) Async

Python:
    p = subprocess.Popen(["sleep", "5"])
    p.wait()

Go:
    cmd := exec.Command("sleep", "5")
    err := cmd.Start()
    err2 := cmd.Wait()

Possible Swift:
    let proc = Process(["sleep", "5"])
    try proc.start()
    try proc.wait()

On Fri, Nov 17, 2017 at 9:34 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:
Hi Abhi,

It does seem like there is a possibility of some better convenience API here.

Any ideas on what form it would take? A class method on Process that returns the output, maybe?

- Tony

On Nov 16, 2017, at 3:34 PM, Abhi Beckert via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Swift is a great shell scripting language except for it's lack of any API to execute UNIX commands. Compare these two shell scripts:

#!/usr/bin/php
<?

$files = `find ~/Desktop -name *.png`;

foreach (explode("\n", $files) as $file) {
  // do something with $file
}

-

#!/usr/bin/swift

import Foundation

let process = Process()
process.launchPath = "/usr/bin/find"
process.arguments = [
  NSString(string:"~/Desktop").expandingTildeInPath,
  "-name",
  "*.png"
]

let output = Pipe()
process.standardOutput = output

process.launch()

let files: String
if let filesUtf8 = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.utf8.rawValue) {
  files = filesUtf8 as String
} else {
  files = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.isoLatin1.rawValue) as NSString! as String
}

files.enumerateLines { file, _ in
  // do something with file
}

It's a contrived example, I could have used NSFileManager, but I run into this all the time integrating with more complex tools such as rsync.

Adding my own high level wrapper around the Process command isn't an option since there is no good way to import code from another file when executing swift asa shell script. All your code needs to be in one file.

- Abhi
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

+1 from me.

I ended up making a wrapper objc->swift around system() since it was removed from swift recently, however, it behaves (from a user standpoint) exactly how I’d like. If my command line util jumps out to another program and it needs stdin/out/err, it all just passes through which is what I usually want. I found trying to pipe everything together via the other mechanisms available was just too much and seemed a little error prone. I can’t speak to system()’s security or general implementation, just that it seems to do what I want 100% of the time.

Brandon

···

From: <swift-corelibs-dev-bounces@swift.org> on behalf of William Dillon via swift-corelibs-dev <swift-corelibs-dev@swift.org>
Reply-To: William Dillon <william@housedillon.com>
Date: Friday, November 17, 2017 at 11:39 AM
To: Tony Parker <anthony.parker@apple.com>
Cc: "swift-corelibs-dev@swift.org" <swift-corelibs-dev@swift.org>
Subject: Re: [swift-corelibs-dev] Better integration with UNIX tools

I, to, have wished for such an API.

I think that would be a very welcome addition.

On Nov 17, 2017, at 11:34 AM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org<mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Abhi,

It does seem like there is a possibility of some better convenience API here.

Any ideas on what form it would take? A class method on Process that returns the output, maybe?

- Tony

On Nov 16, 2017, at 3:34 PM, Abhi Beckert via swift-corelibs-dev <swift-corelibs-dev@swift.org<mailto:swift-corelibs-dev@swift.org>> wrote:

Swift is a great shell scripting language except for it's lack of any API to execute UNIX commands. Compare these two shell scripts:

#!/usr/bin/php
<?

$files = `find ~/Desktop -name *.png`;

foreach (explode("\n", $files) as $file) {
  // do something with $file
}

-

#!/usr/bin/swift

import Foundation

let process = Process()
process.launchPath = "/usr/bin/find"
process.arguments = [
  NSString(string:"~/Desktop").expandingTildeInPath,
  "-name",
  "*.png"
]

let output = Pipe()
process.standardOutput = output

process.launch()

let files: String
if let filesUtf8 = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.utf8.rawValue) {
  files = filesUtf8 as String
} else {
  files = NSString(data: output.fileHandleForReading.readDataToEndOfFile(), encoding: String.Encoding.isoLatin1.rawValue) as NSString! as String
}

files.enumerateLines { file, _ in
  // do something with file
}

It's a contrived example, I could have used NSFileManager, but I run into this all the time integrating with more complex tools such as rsync.

Adding my own high level wrapper around the Process command isn't an option since there is no good way to import code from another file when executing swift asa shell script. All your code needs to be in one file.

- Abhi
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org<mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev&lt;https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.swift.org%2Fmailman%2Flistinfo%2Fswift-corelibs-dev&data=02|01|brsneed%40ebay.com|6eddf25bfd6742c1971508d52df2e751|46326bff992841a0baca17c16c94ea99|0|0|636465443639795145&sdata=pp4hSxpPPGDCB4JtwKo7HB3UIbklF5kD%2B7iZB2kRZdA%3D&reserved=0&gt;

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org<mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

If you really want to do this, you can do it through the array-based API by calling `sh` with the arguments `["-c", yourCommand]`. This makes it slightly less convenient than the safe API, which is a *feature*, because we want people to reach for the safer API.

···

On Nov 19, 2017, at 7:01 PM, Abhi Beckert <me@abhibeckert.com> wrote:

First of all, I disagree with Brent's assessment that we shouldn't have a variant that "simply shells out". Even though this use case is "discouraged" in other languages, I think in practice it definitely has a place.

--
Brent Royal-Gordon
Architechies

A completion handler only makes sense if we're running under the control of a runloop. We aren't today, and I don't believe the most recent async/await stuff I've seen included mechanisms to start a runloop if needed.

···

On Nov 27, 2017, at 3:50 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

I was thinking of perhaps a completion handler version, with the expectation that once async/await style completions land it would be pretty easy to use in a kind of straight-line mechanism.

--
Brent Royal-Gordon
Architechies

Sorry, this part of the sentence got mangled in editing. What I meant to say is, script-style programs currently don't run under a runloop, so if this API is being designed for scripting use, a completion handler—even with async/await—might not be the right design.

···

On Nov 27, 2017, at 11:01 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

We aren't today

--
Brent Royal-Gordon
Architechies

Why does it imply a run loop rather than one of many multithreading possibilities (dispatch queue, starting one more thread, etc)? And even if it did use run loops, why is that a problem?

In general we have been discouraging the use of synchronous and blocking API for many years now. It’s convenient for a script like this, but the API under discussion would be available to all kinds of apps. We must look for a solution that is general enough to be useful everywhere.

- Tony

···

On Nov 27, 2017, at 11:50 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

On Nov 27, 2017, at 11:01 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

We aren't today

Sorry, this part of the sentence got mangled in editing. What I meant to say is, script-style programs currently don't run under a runloop, so if this API is being designed for scripting use, a completion handler—even with async/await—might not be the right design.

--
Brent Royal-Gordon
Architechies

The problem is simply that we're discussing using this feature in Swift *scripts*. Swift scripts don't typically invoke `RunLoop.run()` or `dispatch_main()`, and without changing the way they generate main() functions to intrinsically do so (and then schedule main.swift's body on the main runloop/queue), I don't see a good way for it to do so. So an async API would be inconvenient to use from a Swift script.

···

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com> wrote:

Why does it imply a run loop rather than one of many multithreading possibilities (dispatch queue, starting one more thread, etc)? And even if it did use run loops, why is that a problem?

--
Brent Royal-Gordon
Architechies

Of course, Foundation API has no way of distinguishing if the caller is considered a script or not.

If the API is a bad idea for other kinds of apps then we simply wouldn’t add it. So, I think this proposed convenience API needs to consider all of the varied clients of Foundation.

- Tony

···

On Nov 28, 2017, at 12:24 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:

Why does it imply a run loop rather than one of many multithreading possibilities (dispatch queue, starting one more thread, etc)? And even if it did use run loops, why is that a problem?

The problem is simply that we're discussing using this feature in Swift *scripts*. Swift scripts don't typically invoke `RunLoop.run()` or `dispatch_main()`, and without changing the way they generate main() functions to intrinsically do so (and then schedule main.swift's body on the main runloop/queue), I don't see a good way for it to do so. So an async API would be inconvenient to use from a Swift script.

--
Brent Royal-Gordon
Architechies

In that case I think the discussion is kind of moot, scripts are
fundamentally different than apps, being terse is important and almost
always you want to block.

If better scripting support is a non-goal for Foundation then `Process` is
mostly fine as it is. My only wish would be to somehow make it easier to
read and write `Data` to stdin/stdout/stderr.

···

On Tue, Nov 28, 2017 at 11:30 PM, Tony Parker <anthony.parker@apple.com> wrote:

Of course, Foundation API has no way of distinguishing if the caller is
considered a script or not.

If the API is a bad idea for other kinds of apps then we simply wouldn’t
add it. So, I think this proposed convenience API needs to consider all of
the varied clients of Foundation.

- Tony

On Nov 28, 2017, at 12:24 PM, Brent Royal-Gordon <brent@architechies.com> > wrote:

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com> wrote:

Why does it imply a run loop rather than one of many multithreading
possibilities (dispatch queue, starting one more thread, etc)? And even if
it did use run loops, why is that a problem?

The problem is simply that we're discussing using this feature in Swift
*scripts*. Swift scripts don't typically invoke `RunLoop.run()` or
`dispatch_main()`, and without changing the way they generate main()
functions to intrinsically do so (and then schedule main.swift's body on
the main runloop/queue), I don't see a good way for it to do so. So an
async API would be inconvenient to use from a Swift script.

--
Brent Royal-Gordon
Architechies

Why did we give up so easily? I did not say that scripting support is somehow a non-goal. In fact, it is exactly the opposite.

I do not believe that somehow API that is useful for scripts is mutually exclusive with API that is useful for apps.

- Tony

···

On Nov 30, 2017, at 9:26 AM, Jacob Williams <ponyboy47@gmail.com> wrote:

Since this isn’t going to be coming to Foundation any time…ever, I’d take a look at the SwiftShell framework. It makes running command line utilities from swift pretty easy. Chaining outputs from previous commands into subsequent commands is also rather simple. It’s based on Process but gives a lot of sugar that may be useful for you. I’ve been using it in one of my projects on an Ubuntu machine and it is easy to work with.

GitHub - kareman/SwiftShell: A Swift framework for shell scripting.

On Nov 30, 2017, at 1:43 AM, Nick Keets via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

In that case I think the discussion is kind of moot, scripts are fundamentally different than apps, being terse is important and almost always you want to block.

If better scripting support is a non-goal for Foundation then `Process` is mostly fine as it is. My only wish would be to somehow make it easier to read and write `Data` to stdin/stdout/stderr.

On Tue, Nov 28, 2017 at 11:30 PM, Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:
Of course, Foundation API has no way of distinguishing if the caller is considered a script or not.

If the API is a bad idea for other kinds of apps then we simply wouldn’t add it. So, I think this proposed convenience API needs to consider all of the varied clients of Foundation.

- Tony

On Nov 28, 2017, at 12:24 PM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:

Why does it imply a run loop rather than one of many multithreading possibilities (dispatch queue, starting one more thread, etc)? And even if it did use run loops, why is that a problem?

The problem is simply that we're discussing using this feature in Swift *scripts*. Swift scripts don't typically invoke `RunLoop.run()` or `dispatch_main()`, and without changing the way they generate main() functions to intrinsically do so (and then schedule main.swift's body on the main runloop/queue), I don't see a good way for it to do so. So an async API would be inconvenient to use from a Swift script.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Since this isn’t going to be coming to Foundation any time…ever, I’d take a look at the SwiftShell framework. It makes running command line utilities from swift pretty easy. Chaining outputs from previous commands into subsequent commands is also rather simple. It’s based on Process but gives a lot of sugar that may be useful for you. I’ve been using it in one of my projects on an Ubuntu machine and it is easy to work with.

···

On Nov 30, 2017, at 1:43 AM, Nick Keets via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

In that case I think the discussion is kind of moot, scripts are fundamentally different than apps, being terse is important and almost always you want to block.

If better scripting support is a non-goal for Foundation then `Process` is mostly fine as it is. My only wish would be to somehow make it easier to read and write `Data` to stdin/stdout/stderr.

On Tue, Nov 28, 2017 at 11:30 PM, Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:
Of course, Foundation API has no way of distinguishing if the caller is considered a script or not.

If the API is a bad idea for other kinds of apps then we simply wouldn’t add it. So, I think this proposed convenience API needs to consider all of the varied clients of Foundation.

- Tony

On Nov 28, 2017, at 12:24 PM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:

Why does it imply a run loop rather than one of many multithreading possibilities (dispatch queue, starting one more thread, etc)? And even if it did use run loops, why is that a problem?

The problem is simply that we're discussing using this feature in Swift *scripts*. Swift scripts don't typically invoke `RunLoop.run()` or `dispatch_main()`, and without changing the way they generate main() functions to intrinsically do so (and then schedule main.swift's body on the main runloop/queue), I don't see a good way for it to do so. So an async API would be inconvenient to use from a Swift script.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

1 Like

Hi Nick,

You might be interested in the new Utility project that the Package
Manager team have published. It has a bunch of Foundation-esque
features including subprocess support, temporary file, progress bars
and more.

There's a good blog post about it here:

I hope it gets more publicity as there's some great functionality
here, which has already been used "for real" by the package manager.

Thanks,
Ian Partridge

···

On 30 November 2017 at 08:43, Nick Keets via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

In that case I think the discussion is kind of moot, scripts are
fundamentally different than apps, being terse is important and almost
always you want to block.

If better scripting support is a non-goal for Foundation then `Process` is
mostly fine as it is. My only wish would be to somehow make it easier to
read and write `Data` to stdin/stdout/stderr.

On Tue, Nov 28, 2017 at 11:30 PM, Tony Parker <anthony.parker@apple.com> > wrote:

Of course, Foundation API has no way of distinguishing if the caller is
considered a script or not.

If the API is a bad idea for other kinds of apps then we simply wouldn’t
add it. So, I think this proposed convenience API needs to consider all of
the varied clients of Foundation.

- Tony

On Nov 28, 2017, at 12:24 PM, Brent Royal-Gordon <brent@architechies.com> >> wrote:

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com> wrote:

Why does it imply a run loop rather than one of many multithreading
possibilities (dispatch queue, starting one more thread, etc)? And even if
it did use run loops, why is that a problem?

The problem is simply that we're discussing using this feature in Swift
*scripts*. Swift scripts don't typically invoke `RunLoop.run()` or
`dispatch_main()`, and without changing the way they generate main()
functions to intrinsically do so (and then schedule main.swift's body on the
main runloop/queue), I don't see a good way for it to do so. So an async API
would be inconvenient to use from a Swift script.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

--
Ian Partridge

1 Like

Hi Nick,

You might be interested in the new Utility project that the Package
Manager team have published. It has a bunch of Foundation-esque
features including subprocess support, temporary file, progress bars
and more.

There's a good blog post about it here:
https://www.hackingwithswift.com/articles/44/apple-s-new-utility-library-will-power-up-command-line-apps

I hope it gets more publicity as there's some great functionality
here, which has already been used "for real" by the package manager.

Just to be clear, the APIs in that product are developed for SwiftPM’s own use. They are definitely useful to us, but they are also unstable and unsupported… you are welcome to reuse them if you find them helpful, but they are very much designed specifically for SwiftPM and may change unexpectedly. Use at your own risk!

- Daniel

···

On Dec 1, 2017, at 6:28 AM, Ian Partridge via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Thanks,
Ian Partridge

On 30 November 2017 at 08:43, Nick Keets via swift-corelibs-dev > <swift-corelibs-dev@swift.org> wrote:

In that case I think the discussion is kind of moot, scripts are
fundamentally different than apps, being terse is important and almost
always you want to block.

If better scripting support is a non-goal for Foundation then `Process` is
mostly fine as it is. My only wish would be to somehow make it easier to
read and write `Data` to stdin/stdout/stderr.

On Tue, Nov 28, 2017 at 11:30 PM, Tony Parker <anthony.parker@apple.com> >> wrote:

Of course, Foundation API has no way of distinguishing if the caller is
considered a script or not.

If the API is a bad idea for other kinds of apps then we simply wouldn’t
add it. So, I think this proposed convenience API needs to consider all of
the varied clients of Foundation.

- Tony

On Nov 28, 2017, at 12:24 PM, Brent Royal-Gordon <brent@architechies.com> >>> wrote:

On Nov 28, 2017, at 8:00 AM, Tony Parker <anthony.parker@apple.com> wrote:

Why does it imply a run loop rather than one of many multithreading
possibilities (dispatch queue, starting one more thread, etc)? And even if
it did use run loops, why is that a problem?

The problem is simply that we're discussing using this feature in Swift
*scripts*. Swift scripts don't typically invoke `RunLoop.run()` or
`dispatch_main()`, and without changing the way they generate main()
functions to intrinsically do so (and then schedule main.swift's body on the
main runloop/queue), I don't see a good way for it to do so. So an async API
would be inconvenient to use from a Swift script.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

--
Ian Partridge
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev