[Swift3][Linux][URLSession]: Core dump when trying to make simple HTTP request


(Steven Harms) #1

Hello Swift,

I've been trying out Swift as a general utility language on the server
(replacing Ruby or Node). I'm trying to write a simple image retriever via
HTTP as a learning project.

Inspired by this [gist][], I've written the following:

let queue = DispatchQueue.global(qos: .background)

let sessionConfiguration = URLSessionConfiguration.default
let session = URLSession(configuration: sessionConfiguration)

print("staring sync")
queue.async(execute: {
    print(">>> [\(queue.label)]: At the queue start")
    print("x")
    let task = session.dataTask(with: URL(string: "http://google.com")!,
completionHandler: {
        (data, response, error) in
        print("Task ran!")
    })
    print("y")
    task.resume()
    print("z")
})
print("ending sync")

With output:

staring sync
ending sync

or...

staring sync
ending sync

[com.apple.root.background-qos]: At the queue start

x
y
y
z

Whoa.

At this point I'm going to have to confess that I need some help clarifying
my model.

1. How did point "y" get fired twice? Or how did it happen not at all?

2. How did my callback for dataTask *never* fire? Even if the connection
task *failed* the handler ought have fired, no?

3. I didn't want to bring this whole queue business into the picture, but
it appears that without it the program exits before the handler has a
chance to fire.

4. Changing to a queue.sync from queue.async consistently produces the
output I expect, but does not fire my completionHandler still.

5. Is there a canonical reference for doing this simplest of tasks?

Thanks for the help,

Steven

References:

[gist]: https://gist.github.com/stinger/71000c4d5fb4a25fb7686bae116df0a5


(Dave Abrahams) #2

Pasting a reply on behalf of Matt Wright (cc'd):

From: Steven Harms via swift-users <swift-users@swift.org>
Subject: [Swift3][Linux][URLSession]: Core dump when trying to make simple HTTP request

...<schnipp 40>...

Date: October 27, 2016 at 5:01:58 AM PDT
To: swift-users@swift.org
Reply-To: Steven Harms <sgharms@stevengharms.com>

Hello Swift,

I've been trying out Swift as a general utility language on the server (replacing Ruby or Node). I'm trying to write a simple image retriever via HTTP as a learning project.

Inspired by this [gist][], I've written the following:

let queue = DispatchQueue.global(qos: .background)

let sessionConfiguration = URLSessionConfiguration.default
let session = URLSession(configuration: sessionConfiguration)

print("staring sync")
queue.async(execute: {
    print(">>> [\(queue.label)]: At the queue start")
    print("x")
    let task = session.dataTask(with: URL(string: "http://google.com")!, completionHandler: {
        (data, response, error) in
        print("Task ran!")
    })
    print("y")
    task.resume()
    print("z")
})
print("ending sync")

With output:

staring sync
ending sync

or...

staring sync
ending sync
>>> [com.apple.root.background-qos]: At the queue start
x
y
y
z

Whoa.

At this point I'm going to have to confess that I need some help clarifying my model.

1. How did point "y" get fired twice? Or how did it happen not at all?

The double-fire is something I can’t immediately come up with an
explanation. It might be specific to the Linux dispatch
implementation, which I’m not well versed on but it’s very strange
that it would print twice. Not firing at all is likely because his
snippet of code exited before the work was run on another thread. The
“main” thread here either has to wait for the session work to finish
explicitly, or call dispatch_main() to cede control of the main thread
to the system (and subsequently exit out of the tool somewhere else).

2. How did my callback for dataTask *never* fire? Even if the connection task *failed* the handler

ought have fired, no?

Likely similar to (1), though if URLSession uses runloops (Linux?)
then it could also be because the runloop was never run anywhere. On
Darwin I’d suspect the runloop for URLSession wasn’t running, I don’t
know how it works on Linux.

3. I didn't want to bring this whole queue business into the
picture, but it appears that without it the program exits before the
handler has a chance to fire.

Probably similar to the issues in (1) and (2). If the snippet exits after running then a lot of this
asynchronously scheduled work will never happen.

4. Changing to a queue.sync from queue.async consistently produces the output I expect, but does

not fire my completionHandler still.

This forces the queue.sync to run in-line (on the current thread) but
the snippet will then likely still exit before anything happens to
cause the URLSession to finish.

···

On Oct 27, 2016, at 1:57 PM, Dave Abrahams <dabrahams@apple.com> wrote:

--
-Dave


(Evtim Papushev) #3

Hi Steven,

Do you run your project in Xcode Playgrounds or under Linux?

Generally processes are associated with a single thread, the main thread. Once it exits the process and all executing threads are terminated.

1. Queue.async only schedules a task to be executed, then continues with the rest of the statements. In your case it prints "ending sync”, then it reaches the end and the process terminates.

2. Queue.sync schedules the block for execution, but it also does not return until this block gets executed. So, it effectively pauses the main thread waiting for the task to complete.

In both cases, however, you have an implicit asynchrony as all URLSession tasks run asynchronously. Imagine that it internally calls someQueue.async(…)

There are a few different approaches that you might want to investigate.

1. Make explicit synchronization:

print("staring sync")

let group = DispatchGroup()

group.enter()
queue.async(execute: {
    print(">>> [\(queue.label)]: At the queue start")
    print("x")
    let task = session.dataTask(with: URL(string: "http://google.com")!, completionHandler: {
        (data, response, error) in
        print("Task ran!")
        group.leave()
    })
    print("y")
    task.resume()
    print("z")
})
print("ending sync")
group.wait()

I would go further and note, that since all URLSession tasks are asynchronous, there is absolutely no need to use additional queues:
(I took the liberty to add a set of URLs)

let sessionConfiguration = URLSessionConfiguration.default
let session = URLSession(configuration: sessionConfiguration)

let urls = ["https://www.google.com", "https://www.apple.com"]

let group = DispatchGroup()

print("staring sync")

urls.forEach { url in
    group.enter()
    session.dataTask(with: URL(string: url)!, completionHandler: {
        (data, response, error) in
        print("Task ran!")
        group.leave()
    }).resume()
}

print("ending sync")
group.wait()

2. Investigate the use of dispatch_main()
https://developer.apple.com/reference/dispatch/1452860-dispatch_main

Best,
Evtim

···

On Oct 27, 2016, at 15:01, Steven Harms via swift-users <swift-users@swift.org> wrote:

Hello Swift,

I've been trying out Swift as a general utility language on the server (replacing Ruby or Node). I'm trying to write a simple image retriever via HTTP as a learning project.

Inspired by this [gist][], I've written the following:

let queue = DispatchQueue.global(qos: .background)

let sessionConfiguration = URLSessionConfiguration.default
let session = URLSession(configuration: sessionConfiguration)

print("staring sync")
queue.async(execute: {
    print(">>> [\(queue.label)]: At the queue start")
    print("x")
    let task = session.dataTask(with: URL(string: "http://google.com <http://google.com/>")!, completionHandler: {
        (data, response, error) in
        print("Task ran!")
    })
    print("y")
    task.resume()
    print("z")
})
print("ending sync")

With output:

staring sync
ending sync

or...

staring sync
ending sync
>>> [com.apple.root.background-qos]: At the queue start
x
y
y
z

Whoa.

At this point I'm going to have to confess that I need some help clarifying my model.

1. How did point "y" get fired twice? Or how did it happen not at all?

2. How did my callback for dataTask *never* fire? Even if the connection task *failed* the handler ought have fired, no?

3. I didn't want to bring this whole queue business into the picture, but it appears that without it the program exits before the handler has a chance to fire.

4. Changing to a queue.sync from queue.async consistently produces the output I expect, but does not fire my completionHandler still.

5. Is there a canonical reference for doing this simplest of tasks?

Thanks for the help,

Steven

References:

[gist]: https://gist.github.com/stinger/71000c4d5fb4a25fb7686bae116df0a5_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users


(Quinn “The Eskimo!”) #4

At this point I'm going to have to confess that I need some help clarifying my model.

Quite. There two real issues here:

* You don’t need to create a queue just to start a request. It’s fine to start the request from any context. Thus, your `queue.async(…)` call is not needed.

* URLSession will run requests asynchronously, thus you need to prevent your process from terminating until the relevant requests are done. In a real program the framework you’re using typically does this for you. In a test project like this you have two options:

- dispatchMain() — That is, just add a line containing `dispatchMain()` to the bottom of your ‘main’ code.

- synthetic synchronous — Traditionally this is done using dispatch semaphores; I won’t show it here because a) you’ll find lots of example of it, and b) it’s not recommended, for obscure reasons related to QoS propagation.

1. How did point "y" get fired twice? Or how did it happen not at all?

I ran your code verbatim (except that I added a `dispatchMain()` at the end) on my Mac running 10.11.6, built with Xcode 8, and I didn’t see the second copy of `y`.

2. How did my callback for dataTask *never* fire? Even if the connection task *failed* the handler ought have fired, no?

Because the process terminated before the request finished.

3. I didn't want to bring this whole queue business into the picture, but it appears that without it the program exits before the handler has a chance to fire.

Correct.

4. Changing to a queue.sync from queue.async consistently produces the output I expect, but does not fire my completionHandler still.

Right, because that’s about how the request /starts/, not about how the request’s completion handler runs.

5. Is there a canonical reference for doing this simplest of tasks?

Here’s a minimal example of a command line tool that fetches a URL.

···

On 27 Oct 2016, at 13:01, Steven Harms via swift-users <swift-users@swift.org> wrote:

---------------------------------------------------------------------------
import Foundation

URLSession.shared.dataTask(with: URL(string: "http://www.example.com")!) { (data, response, error) in
    if let error = error {
        print("error: \(error)")
        exit(1)
    } else {
        let response = response as! HTTPURLResponse
        let data = data!
        print("status: \(response.statusCode)")
        for (key, value) in response.allHeaderFields {
            print("header: \(key) = \(value)")
        }
        print("body: \(data as NSData)")
        exit(0)
    }
}.resume()
dispatchMain()
---------------------------------------------------------------------------

When I run this (same environment as above) I get this:

status: 200
header: Date = Fri, 28 Oct 2016 08:02:04 GMT
header: Content-Length = 606
header: Etag = "359670651"
header: x-ec-custom-error = 1
header: Last-Modified = Fri, 09 Aug 2013 23:54:35 GMT
header: Accept-Ranges = bytes
header: Vary = Accept-Encoding
header: X-Cache = HIT
header: Content-Type = text/html
header: Expires = Fri, 04 Nov 2016 08:02:04 GMT
header: Server = ECS (ewr/15BD)
header: Content-Encoding = gzip
header: Cache-Control = max-age=604800
body: <3c21646f 63747970 65206874 6d6c3e0a 3c68746d …>

The only weird thing in the code above is that `key` is of type `AnyHashable` rather than `String`. That’s because the type of `allHeaderFields` is wrong <rdar://problem/27805863>.

Alas, I’m not set up to run this on Linux )-:

Share and Enjoy
--
Quinn "The Eskimo!" <http://www.apple.com/developer/>
Apple Developer Relations, Developer Technical Support, Core OS/Hardware


(Steven Harms) #5

All,

Thanks for all your help thus far. I'll have some time this weekend to try
things out but wanted to let you know that I'm only using Linux-native
tools: vim, swift binary from the command line.

To this end: meaningful error messages, friendly API documentation,
comparability to Python / Node / Ruby for interacting with the web etc. are
all the things I'm investigating. Apparently a task that's come up is
friendliness around coding around the asynchrony model :wink:

Thanks for the suggestions, I'll work through them soon and report back.

···

On Thu, Oct 27, 2016 at 6:32 PM, Dave Abrahams via swift-users < swift-users@swift.org> wrote:

Pasting a reply on behalf of Matt Wright (cc'd):

> On Oct 27, 2016, at 1:57 PM, Dave Abrahams <dabrahams@apple.com> wrote:
>
>
> From: Steven Harms via swift-users <swift-users@swift.org>
> Subject: [Swift3][Linux][URLSession]: Core dump when trying to make
simple HTTP request
...<schnipp 40>...
> Date: October 27, 2016 at 5:01:58 AM PDT
> To: swift-users@swift.org
> Reply-To: Steven Harms <sgharms@stevengharms.com>
>
>
> Hello Swift,
>
> I've been trying out Swift as a general utility language on the server
(replacing Ruby or Node). I'm trying to write a simple image retriever via
HTTP as a learning project.
>
> Inspired by this [gist][], I've written the following:
>
> let queue = DispatchQueue.global(qos: .background)
>
> let sessionConfiguration = URLSessionConfiguration.default
> let session = URLSession(configuration: sessionConfiguration)
>
> print("staring sync")
> queue.async(execute: {
> print(">>> [\(queue.label)]: At the queue start")
> print("x")
> let task = session.dataTask(with: URL(string: "http://google.com")!,
completionHandler: {
> (data, response, error) in
> print("Task ran!")
> })
> print("y")
> task.resume()
> print("z")
> })
> print("ending sync")
>
> With output:
>
> staring sync
> ending sync
>
> or...
>
> staring sync
> ending sync
> >>> [com.apple.root.background-qos]: At the queue start
> x
> y
> y
> z
>
> Whoa.
>
> At this point I'm going to have to confess that I need some help
clarifying my model.
>
> 1. How did point "y" get fired twice? Or how did it happen not at all?

The double-fire is something I can’t immediately come up with an
explanation. It might be specific to the Linux dispatch
implementation, which I’m not well versed on but it’s very strange
that it would print twice. Not firing at all is likely because his
snippet of code exited before the work was run on another thread. The
“main” thread here either has to wait for the session work to finish
explicitly, or call dispatch_main() to cede control of the main thread
to the system (and subsequently exit out of the tool somewhere else).

> 2. How did my callback for dataTask *never* fire? Even if the connection
task *failed* the handler
ought have fired, no?

Likely similar to (1), though if URLSession uses runloops (Linux?)
then it could also be because the runloop was never run anywhere. On
Darwin I’d suspect the runloop for URLSession wasn’t running, I don’t
know how it works on Linux.

> 3. I didn't want to bring this whole queue business into the
> picture, but it appears that without it the program exits before the
> handler has a chance to fire.

Probably similar to the issues in (1) and (2). If the snippet exits after
running then a lot of this
asynchronously scheduled work will never happen.

> 4. Changing to a queue.sync from queue.async consistently produces the
output I expect, but does
not fire my completionHandler still.

This forces the queue.sync to run in-line (on the current thread) but
the snippet will then likely still exit before anything happens to
cause the URLSession to finish.

--
-Dave

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users